BERT-Syntax

Assesing the syntactic abilities of BERT.

What

Evaluate Google's BERT-Base and BERT-Large models on the syntactic agreement datasets from Linzen, Goldberg and Dupoux 2016 and Marvin and Linzen 2018 and Gulordava et al 2018.

This is quite messy, as I hacked it together between things here and there. But I also believe it is accurate. This lists the data files and shows how to run the evaluation. For more details and results, see the arxiv report.

Data Files

Data taken from the github repos of Linzen, Goldberg and Dupoux (LGD), Marvin and Linzen (ML), and Gulordava et al.

File	Description
`marvin_linzen_dataset.tsv`	stimuli from Marvin and Linzen. I dumped it from the pickle files of ML
`wiki.vocab`	from LGD, used for verb inflections (wiki.vocab)
`lgd_dataset.tsv`	processed data from LGD
`generated.tab`	data from Gulordava et al (generated.tab)

lgd_dataset.tsv is created by

wget http://tallinzen.net/media/rnn_agreement/agr_50_mostcommon_10K.tsv.gz
gunzip agr_50_mostcommon_10K.tsv.gz
python make_linzen_goldberg_testset.py > lgd_dataset.tsv

Obtaining the results

pip install pytorch_pretrained_bert

python eval_bert.py > results/lgd_results_large.txt
python eval_bert.py base > results/lgd_results_base.txt
python eval_bert.py marvin > results/marvin_results_large.txt
python eval_bert.py marvin base > results/marvin_results_base.txt
python eval_bert.py gul > results/gulordava_results_large.txt
python eval_bert.py gul base > results/gulordava_results_base.txt

Generating tables (for the PDF)

python gen_marvin_tbl.py 
python gen_lgd_tbl.py
python gen_gul_tbl.py

Assessing syntactic abilities of BERT

Related tags

Overview

BERT-Syntax

What

Data Files

Obtaining the results

Generating tables (for the PDF)

Owner

Yoav Goldberg

PyTorch code for our paper "Attention in Attention Network for Image Super-Resolution"

This repository contains the official code of the paper Equivariant Subgraph Aggregation Networks (ICLR 2022)

This folder contains the python code of UR5E's advanced forward kinematics model.

Fast Style Transfer in TensorFlow

⚡️Optimizing einsum functions in NumPy, Tensorflow, Dask, and more with contraction order optimization.

Lacmus is a cross-platform application that helps to find people who are lost in the forest using computer vision and neural networks.

gACSON software for visualization, processing and analysis of three-dimensional electron microscopy images

OneFlow is a performance-centered and open-source deep learning framework.

Preprossing-loan-data-with-NumPy - In this project, I have cleaned and pre-processed the loan data that belongs to an affiliate bank based in the United States.

Official code repository for "Exploring Neural Models for Query-Focused Summarization"

[CVPR'21] Learning to Recommend Frame for Interactive Video Object Segmentation in the Wild

Neural network-based build time estimation for additive manufacturing

Datasets and source code for our paper Webly Supervised Fine-Grained Recognition: Benchmark Datasets and An Approach

Self-Supervised Pre-Training for Transformer-Based Person Re-Identification

NeRViS: Neural Re-rendering for Full-frame Video Stabilization

Using pretrained language models for biomedical knowledge graph completion.

A fast Protein Chain / Ligand Extractor and organizer.

Official implementation of ETH-XGaze dataset baseline

An imperfect information game is a type of game with asymmetric information

PyTorch implementation for "Mining Latent Structures with Contrastive Modality Fusion for Multimedia Recommendation"