Pytorch Implementation of Zero-Shot Image-to-Text Generation for Visual-Semantic Arithmetic [CVPR 2022]
Check out our follow-up work - Zero-Shot Video Captioning with Evolving Pseudo-Tokens!
[Paper] [Notebook] [Caption Demo] [Arithmetic Demo] [Visual Relations Dataset]
⭐ New: Run captioning configuration it in the browser using replicate.ai UI.
$ python run.py
--reset_context_delta
--caption_img_path "example_images/captions/COCO_val2014_000000097017.jpg"$ python run.py
--reset_context_delta
--end_factor 1.06
--fusion_factor 0.95
--grad_norm_factor 0.95
--run_type arithmetics
--arithmetics_imgs "example_images/arithmetics/woman2.jpg" "example_images/arithmetics/king2.jpg" "example_images/arithmetics/man2.jpg"
--arithmetics_weights 1 1 -1$ python run.py
--reset_context_delta --cond_text "Image of"
--end_factor 1.04
--caption_img_path "example_images/real_world/simpsons.jpg"$ python run.py
--reset_context_delta --cond_text "Image of text that says"
--end_factor 1.04
--caption_img_path "example_images/OCR/welcome_sign.jpg"$ CUDA_VISIBLE_DEVICES=0,1,2,3,4 python run.py
--reset_context_delta
--caption_img_path "example_images/captions/COCO_val2014_000000097017.jpg"
--multi_gpuPlease cite our work if you use it in your research:
@article{tewel2021zero,
title={Zero-Shot Image-to-Text Generation for Visual-Semantic Arithmetic},
author={Tewel, Yoad and Shalev, Yoav and Schwartz, Idan and Wolf, Lior},
journal={arXiv preprint arXiv:2111.14447},
year={2021}
}