LPAQA: Language model Prompt And Query Archive
This repository contains data and code for the paper How Can We Know What Language Models Know?
Install
Our repository is based on LAMA. Please download our fork from here and follow the instructions to set up the environment and download pre-trained language models.
# clone LPAQA
git clone https://github.com/jzbjyb/LPAQA.git LPAQA
# clone LAMA
git clone https://github.com/jzbjyb/LAMA.git LAMA
pushd LAMA && git reset --hard b6b1885c64de5981f249a8b65de25cb0802b4bd5 && rm -rf .git && popd
mv LAMA/* LPAQA/ && rm -rf LAMA
# follow the instructions to install
cd LPAQA
./setup.sh
Retrieve factual knowledge from LMs
For example, to query the owner of MSN (Microsoft is the answer), you can either use manually created prompts (x is owned by y):
python lama/eval_generation.py --lm bert --t "MSN is owned by [MASK]."
or use LPAQA that ensembles a diversity of prompts:
# mined prompts
python lama/eval_ensemble.py --lm bert --subject MSN --relation P127 --prompts prompt/mine
# paraphrased prompts
python lama/eval_ensemble.py --lm bert --subject MSN --relation P127 --prompts prompt/paraphrase
2.1k Jan 02, 2023
81 Dec 08, 2022
27 Dec 15, 2022
227 Dec 31, 2022
2 Sep 27, 2022
89 Dec 30, 2022
81 Dec 03, 2022
2 Oct 31, 2022
146 Dec 29, 2022
121 Dec 21, 2022
33 Dec 06, 2022
15 Dec 22, 2022
70 Nov 22, 2022
15 Nov 03, 2022
76 Dec 12, 2022
313 Dec 22, 2022
76 Sep 05, 2022
4 Jan 24, 2022
4 Jun 18, 2022
3.1k Jan 04, 2023