Repo for EMNLP 2021 paper "Beyond Preserved Accuracy: Evaluating Loyalty and Robustness of BERT Compression"
We recommend the off-the-shelf implementation in scipy.
from scipy.spatial import distance
distance.jensenshannon([0.75, 0.2, 0.05], [0.8, 0.1, 0.1]) # softmax prediction of the teacher and studentWe recommend the off-the-shelf implementation in sklearn.
from sklearn.metrics import accuracy_score
accuracy_score([0, 2, 1, 3], [0, 1, 2, 3]) # predicted labels of the teacher and studentWe use TextFooler, please follow the instructions there.
@inproceedings{beyond-preserved-accuracy,
title = "Beyond Preserved Accuracy: Evaluating Loyalty and Robustness of BERT Compression",
author = "Canwen Xu and Wangchunshu Zhou and Tao Ge and Ke Xu and Julian McAuley and Furu Wei",
booktitle = {{EMNLP}},
year = "2021",
}