BERT-Syntax

Assesing the syntactic abilities of BERT.

What

Evaluate Google's BERT-Base and BERT-Large models on the syntactic agreement datasets from Linzen, Goldberg and Dupoux 2016 and Marvin and Linzen 2018 and Gulordava et al 2018.

This is quite messy, as I hacked it together between things here and there. But I also believe it is accurate. This lists the data files and shows how to run the evaluation. For more details and results, see the arxiv report.

Data Files

Data taken from the github repos of Linzen, Goldberg and Dupoux (LGD), Marvin and Linzen (ML), and Gulordava et al.

File	Description
`marvin_linzen_dataset.tsv`	stimuli from Marvin and Linzen. I dumped it from the pickle files of ML
`wiki.vocab`	from LGD, used for verb inflections (wiki.vocab)
`lgd_dataset.tsv`	processed data from LGD
`generated.tab`	data from Gulordava et al (generated.tab)

lgd_dataset.tsv is created by

wget http://tallinzen.net/media/rnn_agreement/agr_50_mostcommon_10K.tsv.gz
gunzip agr_50_mostcommon_10K.tsv.gz
python make_linzen_goldberg_testset.py > lgd_dataset.tsv

Obtaining the results

pip install pytorch_pretrained_bert

python eval_bert.py > results/lgd_results_large.txt
python eval_bert.py base > results/lgd_results_base.txt
python eval_bert.py marvin > results/marvin_results_large.txt
python eval_bert.py marvin base > results/marvin_results_base.txt
python eval_bert.py gul > results/gulordava_results_large.txt
python eval_bert.py gul base > results/gulordava_results_base.txt

Generating tables (for the PDF)

python gen_marvin_tbl.py 
python gen_lgd_tbl.py
python gen_gul_tbl.py

Assessing syntactic abilities of BERT

Related tags

Overview

BERT-Syntax

What

Data Files

Obtaining the results

Generating tables (for the PDF)

Owner

Yoav Goldberg

4st place solution for the PBVS 2022 Multi-modal Aerial View Object Classification Challenge - Track 1 (SAR) at PBVS2022

End-to-End Dense Video Captioning with Parallel Decoding (ICCV 2021)

Implementation of Uformer, Attention-based Unet, in Pytorch

Code release for NeX: Real-time View Synthesis with Neural Basis Expansion

Official Repsoitory for "Activate or Not: Learning Customized Activation." [CVPR 2021]

Learning Confidence for Out-of-Distribution Detection in Neural Networks

High-Fidelity Pluralistic Image Completion with Transformers (ICCV 2021)

An implementation of Deep Graph Infomax (DGI) in PyTorch

Official PyTorch implementation of "Rapid Neural Architecture Search by Learning to Generate Graphs from Datasets" (ICLR 2021)

Java and SHACL code commented in the paper "Towards compliance checking in reified I/O logic via SHACL" submitted to ICAIL 2021

Repo público onde postarei meus estudos de Python, buscando aprender por meio do compartilhamento do aprendizado!

Python implementation of Bayesian optimization over permutation spaces.

A Comprehensive Analysis of Weakly-Supervised Semantic Segmentation in Different Image Domains (IJCV submission)

Train a state-of-the-art yolov3 object detector from scratch!

Code for the Population-Based Bandits Algorithm, presented at NeurIPS 2020.

A list of Machine Learning Art Colabs

Code for the paper "Jukebox: A Generative Model for Music"

This is the source code for generating the ASL-Skeleton3D and ASL-Phono datasets. Check out the README.md for more details.

TaCL: Improving BERT Pre-training with Token-aware Contrastive Learning

Improving Transferability of Representations via Augmentation-Aware Self-Supervision