Variance-Aware Machine Translation Test Sets

Update: If you are looking for the huggingface Datasets version of these test sets, Gabriele Sarti has created a repository here.

See LICENSE. We follow the data licensing plan as the same as the WMT benchmark.

We release 70 lightweight and discriminative test sets for machine translation evaluation, covering 35 translation directions from WMT16 to WMT20 competitions. See VAT_data folder for detailed information.

For each translation direction of a specific year, both source and reference are provided for different types of evaluation metrics. For example,

VAT_data/
├── wmt20
    ├── ...
    ├── vat_newstest2020-zhen-ref.en.txt
    └── vat_newstest2020-zhen-src.zh.txt

We also provide the meta-inforamtion of reserved data. Each json file contains the IDs of retained data in the original test set. For instance, file wmt20/bert-r_filter-std60.json describes:

{
	...
	"en-de": [4, 6, 10, 13, 14, 15, ...],
	"de-en": [0, 3, 4, 5, 7, 9, ...],
	...
}

The reported results in the paper were produced by single NVIDIA GeForce 1080Ti card.

We will keep updating the code and related documentation after the response.

• sacreBLEU version >= 1.4.14
• BLEURT version >= 0.0.2
• COMET version >= 0.1.0
• BERTScore version >= 0.3.7 (hug_trans==4.2.1)
• PyTorch version >= 1.5.1
• Python version >= 3.8
• CUDA & cudatoolkit >= 10.1

Note: the minimal version is for reproducing the results

1. Use score_xxx.py to generate the CSV files that stores the sentence-level scores evaluated by the corresponding metrics. For example, evaluating all the WMT20 submissions of all the language pairs using BERTScore:

   CUDA_VISIBLE_DEVICES=0 python score_bert.py -b 128 -s -r dummy -c dummy --rescale_with_baseline \
   	--hypos-dir ${WMT_DATA_PATH}/system-outputs \
   	--refs-dir ${WMT_DATA_PATH}/references \
   	--scores-dir ${WMT_DATA_PATH}/results/system-level/scores_ALL \
   	--testset-name newstest2020 --score-dump wmt20-bertscore.csv

   (Alternative Option) You can use your implementation for dumping the scores given by the metrics. But the CSV header should contain:

   ,TESTSET,LP,ID,METRIC,SYS,SCORE
   

2. Use cal_filtering.py to filter the test set based on the score warehouse calculated in the last step. For example,

   python cal_filtering.py --score-dump wmt20-bertscore.csv --output VAT_meta/wmt20-test/ --filter-per 60

   It will produce the json files which contain the IDs of reserved sentences.