This repo is the implementation of the following paper:
SPARQLing Database Queries from Intermediate Question Decompositions
Irina Saparina and Anton Osokin
In proceedings of EMNLP'21
[31.05.2022]: We fixed several bugs in the decoding process, usage of the GraPPa tokenization (affect our model) and SQL-SQL comparison (affect on baselines). The current code reproduces the results from the updated version of the paper (arXiv:2109.06162v2).
This software is released under the MIT license, which means that you can use the code in any way you want.
Create conda env with pytorch 1.9 and many other packages upgraded: conda_env_with_pytorch1.9.yaml:
conda env create -n env-torch1.9 -f conda_env_with_pytorch1.9.yaml
conda activate env-torch1.9Download some nltk resourses, Bert and GraPPa:
mkdir -p third_party
pip install -r requirements.txt && \
pip install entmax && \
python -c "import nltk; nltk.download('stopwords'); nltk.download('punkt')"
python -c "from transformers import AutoModel; AutoModel.from_pretrained('bert-large-uncased-whole-word-masking'); AutoModel.from_pretrained('Salesforce/grappa_large_jnt')"
mkdir -p third_party && \
cd third_party && \
curl https://nlp.stanford.edu/software/stanford-corenlp-full-2018-10-05.zip | jar xvWe currently provide both Spider and Break inside our repos. Note that datasets differ from original ones as we fixed some annotation errors. Download databases:
bash ./utils/wget_gdrive.sh spider_temp.zip 11icoH_EA-NYb0OrPTdehRWm_d7-DIzWX
unzip spider_temp.zip -d spider_temp
cp -r spider_temp/spider/database ./data/spider
rm -rf spider_temp/
python ./qdmr2sparql/fix_databases.py --spider_path ./data/spiderTo reproduce our annotation procedure see qdmr2sparql/README.md.
For testing qdmr2sparql translator run qdmr2sparql/test_qdmr2sparql.py
Every experiment has its own config file in text2qdmr/configs/experiments.
The pipeline of working with any model version or dataset is:
python run_text2qdmr.py preprocess experiment_config_file # preprocess the data
python run_text2qdmr.py train experiment_config_file # train a model
python run_text2qdmr.py eval experiment_config_file # evaluate the results
# multiple GPUs on one machine:
export NGPUS=4 # set $NGPUS manually
python -m torch.distributed.launch --nproc_per_node=$NGPUS --use_env --master_port `./utils/get_free_port.sh` run_text2qdmr.py train experiment_config_fileNote that preprocessing and evaluation use execution and take some time. To speed up the evaluation, you can install Virtuoso server (see qdmr2sparql/README_Virtuoso.md).
The dev and test examples of model output are in model_samples/.
Checkpoints of our best models:
| Model name | Dev | Test | Config | Link |
|---|---|---|---|---|
| grappa-aug | 82.0 | 62.4 | text2qdmr/configs/eval-checkpoints/grappa_qdmr_aug.jsonnet | Google Drive |
| grappa-full_break-aug | 81.6 | 65.3 | text2qdmr/configs/eval-checkpoints/grappa_full_break_aug.jsonnet | Google Drive |
To reproduce, firstly download the checkpoints and put them into new folders:
mkdir logdir/grappa-aug/bs=6,lr=7.4e-04,bert_lr=3.0e-06,end_lr=0e0,att=1
mv grappa-aug logdir/grappa-aug/bs=6,lr=7.4e-04,bert_lr=3.0e-06,end_lr=0e0,att=1/model_checkpoint-00080000
mkdir logdir/grappa-full_break-aug/bs=6,lr=7.4e-04,bert_lr=3.0e-06,end_lr=0e0,att=1
mv grappa-full_break-aug logdir/grappa-full_break-aug/bs=6,lr=7.4e-04,bert_lr=3.0e-06,end_lr=0e0,att=1/model_checkpoint-00081000Then use the corresponding config_file for evaluation:
python run_text2qdmr.py preprocess path_to_config_file
python run_text2qdmr.py eval path_to_config_fileText2qdmr module is based on RAT-SQL code, the implementation of ACL'20 paper "RAT-SQL: Relation-Aware Schema Encoding and Linking for Text-to-SQL Parsers" by Wang et al.
Spider dataset was proposed by Yi et al. in EMNLP'18 paper "Spider: A Large-Scale Human-Labeled Dataset for Complex and Cross-Domain Semantic Parsing and Text-to-SQL Task".
Break dataset was proposed by Wolfson et al. in TACL paper "Break It Down: A Question Understanding Benchmark".