This repo contains our codes for the paper "Super Tickets in Pre-Trained Language Models: From Model Compression to Improving Generalization" (ACL 2021).

1. python3.6
   Reference to download and install : https://www.python.org/downloads/release/python-360/
2. install requirements
   > pip install -r requirements.txt

1. Download data
   sh download.sh
   Please refer to download GLUE dataset: https://gluebenchmark.com/
2. Preprocess data
   > sh experiments/glue/prepro.sh
   For more data processing details, please refer to this repo.

1. Fine-tune a pre-trained BERT model with single task data, compute importance scores, and generate one-shot structured pruning masks at multiple sparsity levels. E.g., for MNLI, run
   

   ./scripts/train_mnli.sh GPUID
   

2. Rewind and evaluate the winning, random, and losing tickets at multiple sparsity levels. E.g., for MNLI, run
   

   ./scripts/rewind_mnli.sh GPUID
   

You may try tasks with smaller sizes (e.g., SST, MRPC, RTE) to see a more pronounced phase transition.

1. Identify a set of super tickets for each individual task.

   • Identify winning tickets at multiple sparsity levels for each individual task. E.g., for MTDNN-base, run

     ./scripts/prepare_mtdnn_base.sh GPUID
     

     We recommend to use the same optimization settings, e.g., learning rate, optimizer and random seed, in both the ticket identification procedures and the MTL. We empirically observe that the super tickets perform better in MTL in such a case.

   • [Optional] For each individual task, identify a set of super tickets from the winning tickets at multiple sparsity levels. You can skip this step if you wish to directly use the set of super tickets identified by us. If you wish to identify super tickets on your own (This is recommended if you use a different optimization settings, e.g., learning rate, optimizer and random seed, from those in our scripts. These factors may affect the candidacy of super tickets.), we provide the template scripts

     ./scripts/rewind_mnli_winning.sh GPUID
     ./scripts/rewind_qnli_winning.sh GPUID
     ./scripts/rewind_qqp_winning.sh GPUID
     ./scripts/rewind_sst_winning.sh GPUID
     ./scripts/rewind_mrpc_winning.sh GPUID
     ./scripts/rewind_cola_winning.sh GPUID
     ./scripts/rewind_stsb_winning.sh GPUID
     ./scripts/rewind_rte_winning.sh GPUID
     

     These scripts rewind the winning tickets at multiple sparsity levels. You can manually identify the set of super tickets as the set of winning tickets that perform the best among all sparsity levels.

2. Construct multi-task super tickets by aggregating the identified sets of super tickets of all tasks. E.g., to use the super tickets identified by us, run

   python construct_mtl_mask.py
   

   You can modify the script to use the super tickets identified by yourself.

3. MTL with tickets sharing. Run

   ./scripts/train_mtdnn.sh GPUID
   

MTL evaluation results on GLUE dev set averaged over 5 random seeds.

@article{liang2021super,
  title={Super Tickets in Pre-Trained Language Models: From Model Compression to Improving Generalization},
  author={Liang, Chen and Zuo, Simiao and Chen, Minshuo and Jiang, Haoming and Liu, Xiaodong and He, Pengcheng and Zhao, Tuo and Chen, Weizhu},
  journal={arXiv preprint arXiv:2105.12002},
  year={2021}
}

@article{liu2020mtmtdnn,
  title={The Microsoft Toolkit of Multi-Task Deep Neural Networks for Natural Language Understanding},
  author={Liu, Xiaodong and Wang, Yu and Ji, Jianshu and Cheng, Hao and Zhu, Xueyun and Awa, Emmanuel and He, Pengcheng and Chen, Weizhu and Poon, Hoifung and Cao, Guihong and Jianfeng Gao},
  journal={arXiv preprint arXiv:2002.07972},
  year={2020}
}

For help or issues related to this package, please submit a GitHub issue. For personal questions related to this paper, please contact Chen Liang (cliang73@gatech.edu).