Learning to compose soft prompts for compositional zero-shot learning.

Overview

Compositional Soft Prompting (CSP)

Compositional soft prompting (CSP), a parameter-efficient learning technique to improve the zero-shot compositionality of large-scale pretrained vision-language models (VLMs) without the overhead of fine-tuning the entire model.

Reference Paper: Learning to Compose Soft Prompts for Compositional Zero-Shot Learning

alt text

If you find CSP helpful, please cite our paper:

@article{csp2022,
  author = {Nayak, Nihal V. and Yu, Peilin and Bach, Stephen H.},
  title = {Learning to Compose Soft Prompts for Compositional Zero-Shot Learning},
  volume = {arXiv:2204.03574 [cs.LG]},
  year = {2022},
}

Setup

conda create --name clip python=3.7
conda activate clip
pip3 install torch torchvision torchaudio --extra-index-url https://download.pytorch.org/whl/cu113
pip3 install ftfy regex tqdm scipy pandas
pip3 install git+https://github.com/openai/CLIP.git

Alternatively, you can use pip install -r requirements.txt to install all the dependencies.

Download Dataset

We experiment with three datasets: MIT-States, UT-Zappos, and C-GQA.

sh download_data.sh

If you already have setup the datasets, you can use symlink and ensure the following paths exist: data/<dataset> where <datasets> = {'mit-states', 'ut-zappos', 'cgqa'}.

Training

python -u train.py \
  --dataset mit-states \
  --model ViT-L/14 \
  --experiment_name csp \
  --seed 0 \
  --epochs 20 \
  --lr 5e-05 \
  --attr_dropout 0.3 \
  --weight_decay 0.00001 \
  --train_batch_size 64 \
  --gradient_accumulation_steps 2 \
  --context_length 8 \
  --save_path data/model/mit-states/sample_model \
  --save_every_n 1

You can replace --dataset with {mit-states, ut-zappos, cgqa}. The best hyperparameters are included in the paper.

Evaluation

We evaluate our models in two settings: closed-world and open-world.

Closed-World Evaluation

python -u evaluate.py \
  --dataset mit-states \
  --soft_embeddings data/model/mit-states/sample_model/soft_embeddings_epoch_20.pt \
  --context_length 16 \
  --text_encoder_batch_size 36 \
  --eval_batch_size 16 \
  --experiment_name csp

Open-World Evaluation

For our open-world evaluation, we compute the feasbility calibration and then evaluate on the dataset.

Feasibility Calibration

We use GloVe embeddings to compute the similarities between objects and attributes. Download the GloVe embeddings in the data directory:

cd data
wget https://nlp.stanford.edu/data/glove.6B.zip

Move glove.6B.300d.txt into data/glove.6B.300d.txt.

To compute feasibility calibration for each dataset, run the following command:

python -u datasets/feasibility.py --dataset mit-states

The feasibility similarities are saved at data/feasibility_<dataset>.pt.

Evaluation

The open-world evaluation with the thresholds (feasibility calibration).

python -u evaluate.py \
  --dataset mit-states \
  --soft_embeddings data/model/mit-states/sample_model/soft_embeddings_epoch_5.pt \
  --context_length 16 \
  --text_encoder_batch_size 36 \
  --eval_batch_size 256 \
  --experiment_name czsl \
  --threshold <threshold> \
  --open_world

If <threshold> is None, then the model picks the best threshold on the validation set. We use the following thresholds:

Dataset Threshold
mit-states 0.4069159426
ut-zappos 0.5299109123
cgqa 0.49937106273612186

Note: We use 256GB of cpu memory to evaluate cgqa.

Generalization to Higher-Order Compositions

Evaluate the trained CSP vocabulary on the new AAO-MIT-States dataset.

python aao/evaluate_att_att_obj.py \
  --experiment_name csp \
  --soft_embeddings data/model/mit-states/sample_model/soft_embeddings_epoch_20.pt

We thank Andrew Delworth and Elise Carman for helping us annotate this dataset.

Generalization to Mixed Pretrained and Fine-Tuned Vocabulary

Ablation experiment to train and evaluate CSP with reduced fine-tuned vocabulary. We run experiment on the ut-zappos dataset.

Training

python -u mix/mix_train.py \
  --dataset ut-zappos \
  --model ViT-L/14 \
  --experiment_name mix_csp \
  --seed 0 \
  --epochs 20 \
  --lr 5e-04 \
  --attr_dropout 0.2 \
  --weight_decay 0.00001 \
  --train_batch_size 64 \
  --context_length 8 \
  --save_path data/model/ut-zappos/mix_train_model_0.25 \
  --save_every_n 5 \
  --attr_keep_ratio 0.25 \
  --gradient_accumulation_steps 2

We change the --attr_keep_ratio to {0.25, 0.50, 0.75}.

Evaluation

python -u mix/evaluate_mix_train.py \
  --dataset ut-zappos \
  --soft_embeddings data/model/ut-zappos/mix_train_model_0.25/soft_embeddings.pt \
  --context_length 16 \
  --text_encoder_batch_size 36 \
  --eval_batch_size 256 \
  --experiment_name csp

Credits

The project uses openly available model, code, and datasets. Please see the credits.

Owner
Bats Research
Bats Research
Natas teaches the basics of serverside web-security.

over-the-wire-natas Natas teaches the basics of serverside web-security. Each level of natas consists of its own website located at http://natasX.nata

Siddhant Chouhan 1 Nov 27, 2021
Cobalt Strike < 4.4 dos CVE-2021-36798

CVE-2021-36798 CVE-2021-36798 Cobalt Strike 4.3 dos 用法 python3 CVE-2021-36798.py BeaconURL 打瘫Cobalt Strike 只需要一个包 已测试 4.3 4.2 参考: https://labs.sent

37 Nov 09, 2022
If you are worried about being found perhaps try taking cover under a blanket. Pure Python PowerShell Obfuscator

If you are worried about being found perhaps try taking cover under a blanket. Pure Python PowerShell Obfuscator

Ph0tonz 3 Jun 07, 2022
SeaSurf is a Flask extension for preventing cross-site request forgery (CSRF).

Flask-SeaSurf SeaSurf is a Flask extension for preventing cross-site request forgery (CSRF). CSRF vulnerabilities have been found in large and popular

Max Countryman 183 Dec 28, 2022
exchange-ssrf-rce

Usage python3 .\exchange-exp.py -------------------------------------------------------------------------------- |

Jen 76 Nov 09, 2022
Multi-Process Vulnerability Tool

Multi-Process Vulnerability Tool

Baris Dincer 1 Dec 22, 2021
A simple linux keylogger project.

The project This project is a simple linux keylogger. When activated, it registers all the actions made with the keyboard. The log files are registere

1 Oct 24, 2021
Apache Solr SSRF(CVE-2021-27905)

Solr-SSRF Apache Solr SSRF #Use [-] Apache Solr SSRF漏洞 (CVE-2021-27905) [-] Options: -h or --help : 方法说明 -u or --url

Henry4E36 70 Nov 09, 2022
Uses Sharphound, Bloodhound and Neo4j to produce an actionable list of attack paths for targeted remediation.

GoodHound ______ ____ __ __ / ____/___ ____ ____/ / / / /___ __ ______ ____/ / / / __/ __ \/ __ \/ __

idna 352 Jan 02, 2023
Um keylogger que se disfarça de um app que tira print da tela.

Keylogger_ Um keylogger que se disfarça de um app que tira print da tela. Este programa captura o print da tela e salva ,normalmente, na pasta Picture

Marcus Vinícius Ribeiro Andrade 1 Dec 03, 2021
Credit Card And SK Checker Written In Python

💳 Credit Card Checker (CC Checker) & Mass SK Checker & Generator 💳

Rimuru Tempest 53 Dec 31, 2022
Delta Sharing: An Open Protocol for Secure Data Sharing

Delta Sharing: An Open Protocol for Secure Data Sharing Delta Sharing is an open protocol for secure real-time exchange of large datasets, which enabl

Delta Lake 497 Jan 02, 2023
Abusing Microsoft 365 OAuth Authorization Flow for Phishing Attack

O365DevicePhish Microsoft365_devicePhish Abusing Microsoft 365 OAuth Authorization Flow for Phishing Attack This is a simple proof-of-concept script t

Trewis [work] Scotch 4 Sep 23, 2022
Password-Manager GUI

PASSWORD-MANAGER This repo contains all the project files. Project Description A Tkinter GUI that allows you to store website info like website name,

David .K. Danso 1 Dec 08, 2021
Trainspotting - Python Dependency Injector based on interface binding

Choose dependency injection Friendly with MyPy Supports lazy injections Supports

avito.tech 3 Jan 26, 2022
Scan publicly accessible assets on your AWS cloud environment

poro Description Scan for publicly accessible assets on your AWS environment Services covered by this tool: AWS ELB API Gateway S3 Buckets RDS Databas

9rnt 134 Dec 16, 2022
Burp Suite extension for encoding/decoding EVM calldata

unblocker Burp Suite extension for encoding/decoding EVM calldata 0x00_prerequisites Burp Suite Java 8+ Python 2.7 0x01_installation clone this reposi

Halborn 16 Aug 30, 2022
Undetectable Keylogger that reports to Discord

FUD Keylogger That Reports To Discord This python script will capture all of the keystrokes within a given time frame and report them to a Discord Ser

Dimitris Kalopisis 36 Dec 20, 2022
A kAFL based hypervisor fuzzer which fully supports nested VMs

hAFL2 hAFL2 is a kAFL-based hypervisor fuzzer. It is the first open-source fuzzer which is able to target hypervisors natively (including Hyper-V), as

SafeBreach Labs 115 Dec 07, 2022
This respository contains the source code of the printjack and phonejack attacks.

Printjack-Phonejack This repository contains the source code of the printjack and phonejack attacks. The Printjack directory contains the script to ca

pietrobiondi 2 Feb 12, 2022