REval is a simple framework for probing sentence-level representations of Relation Extraction models.
REval is tested with:
- Python 3.7
<TBD>git clone https://github.com/DFKI-NLP/REval
cd REval
pip install -r requirements.txt| Task | SemEval 2010 | TACRED |
|---|---|---|
| ArgTypeHead | ✔️ | ✔️ |
| ArgTypeTail | ✔️ | ✔️ |
| Length | ✔️ | ✔️ |
| EntityDistance | ✔️ | ✔️ |
| ArgumentOrder | ✔️ | |
| EntityExistsBetweenHeadTail | ✔️ | ✔️ |
| PosTagHeadLeft | ✔️ | ✔️ |
| PosTagHeadRight | ✔️ | ✔️ |
| PosTagTailLeft | ✔️ | ✔️ |
| PosTagTailRight | ✔️ | ✔️ |
| TreeDepth | ✔️ | ✔️ |
| SDPTreeDepth | ✔️ | ✔️ |
| ArgumentHeadGrammaticalRole | ✔️ | ✔️ |
| ArgumentTailGrammaticalRole | ✔️ | ✔️ |
Step 1: create the probing task datasets from the original datasets.
python reval.py generate-all-from-semeval \
--train-file <SEMEVAL DIR>/train.json \
--validation-file <SEMEVAL DIR>/dev.json \
--test-file <SEMEVAL DIR>/test.json \
--output-dir ./data/semeval/python reval.py generate-all-from-tacred \
--train-file <TACRED DIR>/train.json \
--validation-file <TACRED DIR>/dev.json \
--test-file <TACRED DIR>/test.json \
--output-dir ./data/tacred/For example, download a Relation Extraction model trained with RelEx, e.g., the CNN trained on SemEval.
mkdir -p models/cnn_semeval
wget --content-disposition https://cloud.dfki.de/owncloud/index.php/s/F3gf9xkeb2foTFe/download -P models/cnn_semevalpython probing_task_evaluation.py \
--model-dir ./models/cnn_semeval/ \
--data-dir ./data/semeval/ \
--dataset semeval2010 \
--cuda-device 0 \
--batch-size 64 \
--cache-representationsAfter the run is completed, the results are stored to probing_task_results.json in the model-dir.
{
"ArgTypeHead": {
"acc": 75.82,
"devacc": 78.96,
"ndev": 670,
"ntest": 2283
},
"ArgTypeTail": {
"acc": 75.4,
"devacc": 78.79,
"ndev": 627,
"ntest": 2130
},
[...]
}If you use REval, please consider citing the following paper:
@inproceedings{alt-etal-2020-probing,
title={Probing Linguistic Features of Sentence-level Representations in Neural Relation Extraction},
author={Christoph Alt and Aleksandra Gabryszak and Leonhard Hennig},
year={2020},
booktitle={Proceedings of ACL},
url={https://arxiv.org/abs/2004.08134}
}
REval is released under the terms of the MIT License.