BERT-Syntax

Assesing the syntactic abilities of BERT.

What

Evaluate Google's BERT-Base and BERT-Large models on the syntactic agreement datasets from Linzen, Goldberg and Dupoux 2016 and Marvin and Linzen 2018 and Gulordava et al 2018.

This is quite messy, as I hacked it together between things here and there. But I also believe it is accurate. This lists the data files and shows how to run the evaluation. For more details and results, see the arxiv report.

Data Files

Data taken from the github repos of Linzen, Goldberg and Dupoux (LGD), Marvin and Linzen (ML), and Gulordava et al.

File	Description
`marvin_linzen_dataset.tsv`	stimuli from Marvin and Linzen. I dumped it from the pickle files of ML
`wiki.vocab`	from LGD, used for verb inflections (wiki.vocab)
`lgd_dataset.tsv`	processed data from LGD
`generated.tab`	data from Gulordava et al (generated.tab)

lgd_dataset.tsv is created by

wget http://tallinzen.net/media/rnn_agreement/agr_50_mostcommon_10K.tsv.gz
gunzip agr_50_mostcommon_10K.tsv.gz
python make_linzen_goldberg_testset.py > lgd_dataset.tsv

Obtaining the results

pip install pytorch_pretrained_bert

python eval_bert.py > results/lgd_results_large.txt
python eval_bert.py base > results/lgd_results_base.txt
python eval_bert.py marvin > results/marvin_results_large.txt
python eval_bert.py marvin base > results/marvin_results_base.txt
python eval_bert.py gul > results/gulordava_results_large.txt
python eval_bert.py gul base > results/gulordava_results_base.txt

Generating tables (for the PDF)

python gen_marvin_tbl.py 
python gen_lgd_tbl.py
python gen_gul_tbl.py

Assessing syntactic abilities of BERT

Related tags

Overview

BERT-Syntax

What

Data Files

Obtaining the results

Generating tables (for the PDF)

Owner

Yoav Goldberg

Official Pytorch implementation of Meta Internal Learning

This is the face keypoint train code of project face-detection-project

I decide to sync up this repo and self-critical.pytorch. (The old master is in old master branch for archive)

Just playing with getting CLIP Guided Diffusion running locally, rather than having to use colab.

Omniscient Video Super-Resolution

[CVPR 2022] Structured Sparse R-CNN for Direct Scene Graph Generation

A system for quickly generating training data with weak supervision

Welcome to The Eigensolver Quantum School, a quantum computing crash course designed by students for students.

[ICME 2021 Oral] CORE-Text: Improving Scene Text Detection with Contrastive Relational Reasoning

A scikit-learn compatible neural network library that wraps PyTorch

DenseNet Implementation in Keras with ImageNet Pretrained Models

Unsupervised Learning of Probably Symmetric Deformable 3D Objects from Images in the Wild

📚 A collection of all the Deep Learning Metrics that I came across which are not accuracy/loss.

Modeling Category-Selective Cortical Regions with Topographic Variational Autoencoders

AutoDeeplab / auto-deeplab / AutoML for semantic segmentation, implemented in Pytorch

Tensorflow implementation of ID-Unet: Iterative Soft and Hard Deformation for View Synthesis.

Learning to Map Large-scale Sparse Graphs on Memristive Crossbar

Vision-and-Language Navigation in Continuous Environments using Habitat

Monocular 3D pose estimation. OpenVINO. CPU inference or iGPU (OpenCL) inference.

PyTorch implementation of paper “Unbiased Scene Graph Generation from Biased Training”