D-VQA
We provide the PyTorch implementation for Debiased Visual Question Answering from Feature and Sample Perspectives (NeurIPS 2021).
Dependencies
- Python 3.6
- PyTorch 1.1.0
- dependencies in requirements.txt
- We train and evaluate all of the models based on one TITAN Xp GPU
Getting Started
Installation
-
Clone this repository:
git clone https://github.com/Zhiquan-Wen/D-VQA.git cd D-VQA -
Install PyTorch and other dependencies:
pip install -r requirements.txt
Download and preprocess the data
cd data
bash download.sh
python preprocess_features.py --input_tsv_folder xxx.tsv --output_h5 xxx.h5
python feature_preprocess.py --input_h5 xxx.h5 --output_path trainval
python create_dictionary.py --dataroot vqacp2/
python preprocess_text.py --dataroot vqacp2/ --version v2
cd ..
Training
- Train our model
CUDA_VISIBLE_DEVICES=0 python main.py --dataroot data/vqacp2/ --img_root data/coco/trainval_features --output saved_models_cp2/ --self_loss_weight 3 --self_loss_q 0.7
- Train the model with 80% of the original training set
CUDA_VISIBLE_DEVICES=0 python main.py --dataroot data/vqacp2/ --img_root data/coco/trainval_features --output saved_models_cp2/ --self_loss_weight 3 --self_loss_q 0.7 --ratio 0.8
Evaluation
- A json file of results from the test set can be produced with:
CUDA_VISIBLE_DEVICES=0 python test.py --dataroot data/vqacp2/ --img_root data/coco/trainval_features --checkpoint_path saved_models_cp2/best_model.pth --output saved_models_cp2/result/
- Compute detailed accuracy for each answer type:
python comput_score.py --input saved_models_cp2/result/XX.json --dataroot data/vqacp2/
Pretrained model
A well-trained model can be found here. The test results file produced by it can be found here and its performance is as follows:
Overall score: 61.91
Yes/No: 88.93 Num: 52.32 other: 50.39
Reference
If you found this code is useful, please cite the following paper:
@inproceedings{D-VQA,
title = {Debiased Visual Question Answering from Feature and Sample Perspectives},
author = {Zhiquan Wen,
Guanghui Xu,
Mingkui Tan,
Qingyao Wu,
Qi Wu},
booktitle = {NeurIPS},
year = {2021}
}
Acknowledgements
This repository contains code modified from SSL-VQA, thank you very much!
Besides, we thank Yaofo Chen for providing MIO library to accelerate the data loading.
下面是我的代码
def plot_rect(image, boxes):
img = Image.fromarray(np.uint8(image))
draw = ImageDraw.Draw(img)
for k in range(2):
box = boxes[k,:]
print(box)
drawrect(draw, box, outline='green', width=3)
img = np.asarray(img)
return img
def drawrect(drawcontext, xy, outline=None, width=0):
x1, y1, x2, y2 = xy
points = (x1, y1), (x2, y1), (x2, y2), (x1, y2), (x1, y1)
drawcontext.line(points, fill=outline, width=width)
1 Dec 19, 2021
20 Jul 18, 2022
7 Nov 11, 2022
184 Dec 12, 2022
83 Dec 23, 2022
128 Dec 30, 2022
19 Oct 11, 2022
3 Dec 31, 2021
523 Jan 04, 2023
100 Nov 30, 2022
31 Oct 17, 2022
4 Oct 17, 2022
81 Dec 27, 2022
82 Sep 15, 2022
26 Oct 13, 2022
26 Dec 13, 2022
276 Dec 12, 2022
310 Nov 07, 2022
444 Jan 03, 2023
40 Sep 19, 2022