This repo contains the code for our NeurIPS 2021 paper "Realistic evaluation of transductive few-shot learning". This is a framework that regroups all methods evaluated in our paper except for SIB and LR-ICI. Results provided in the paper can be reproduced with this repo. Code was developed under python 3.8.3 and pytorch 1.4.0.
To download datasets and pre-trained models (checkpoints), follow instructions 1.1.1 to 1.1.2 of NeurIPS 2020 paper "TIM: Transductive Information Maximization" public implementation (https://github.com/mboudiaf/TIM)
Make sure to place the downloaded datasets (data/ folder) at the root of the directory.
Make sure to place the downloaded pre-trained models (checkpoints/ folder) at the root of the directory.
Follow instruction 1.2 of NeurIPS 2020 paper "TIM: Transductive Information Maximization" public implementation (https://github.com/mboudiaf/TIM) if facing issues with previous steps. Make sure to place data/ and checkpoints/ folders at the root of the directory.
To install requirements:
conda create --name <env> --file requirements.txtWhere <env> is the name of your environment
Before anything, activate the environment:
source activate <env>| Method | 1-shot | 5-shot | 10-shot | 20-shot |
|---|---|---|---|---|
| SimpleShot | 63.0 | 80.1 | 84.0 | 86.1 |
| PT-MAP | 60.1 (↓16.8) | 67.1 (↓18.2) | 68.8 (↓18.0) | 70.4 (↓17.4) |
| LaplacianShot | 65.4 (↓4.7) | 81.6 (↓0.5) | 84.1 (↓0.2) | 86.0 (↑0.5) |
| BDCSPN | 67.0 (↓2.4) | 80.2 (↓1.8) | 82.7 (↓1.4) | 84.6 (↓1.1) |
| TIM | 67.3 (↓4.5) | 79.8 (↓4.1) | 82.3 (↓3.8) | 84.2 (↓3.7) |
| α-TIM | 67.4 | 82.5 | 85.9 | 87.9 |
To reproduce the results from Table 1. and 2. in the paper, from the root of the directory execute this python command.
python3 -m src.main --base_config <path_to_base_config_file> --method_config <path_to_method_config_file> The <path_to_base_config_file> follows this hierarchy:
config/<balanced or dirichlet>/base_config/<resnet18 or wideres>/<mini or tiered or cub>/base_config.yamlThe <path_to_method_config_file> follows this hierarchy:
config/<balanced or dirichlet>/methods_config/<alpha_tim or baseline or baseline_pp or bdcspn or entropy_min or laplacianshot or protonet or pt_map or simpleshot or tim>.yamlFor instance, if you want to reproduce the results in the balanced setting on mini-Imagenet, using ResNet-18, with alpha-TIM method go to the root of the directory and execute:
python3 -m src.main --base_config config/balanced/base_config/resnet18/mini/base_config.yaml --method_config config/balanced/methods_config/alpha_tim.yamlIf you want to reproduce the results in the randomly balanced setting on mini-Imagenet, using ResNet-18, with alpha-TIM method go to the root of the directory and execute:
python3 -m src.main --base_config config/dirichlet/base_config/resnet18/mini/base_config.yaml --method_config config/dirichlet/methods_config/alpha_tim.yamlOne of our main contribution is our realistic task sampling method following Dirichlet's distribution.
Our realistic sampler can be found in sampler.py file. The sampler has been implemented following Pytorch's norms and in a way that it can be easily reused and integrated in other projects.
The following notebook exemple_realistic_sampler.ipynb is an exemple that shows how to initialize and use our realistic category sampler.
For further questions or details, reach out to Olivier Veilleux (olivier.veilleux.2@ens.etsmtl.ca)
Special thanks to the authors of NeurIPS 2020 paper "TIM: Transductive Information Maximization" (TIM) (https://github.com/mboudiaf/TIM) for publicly sharing their pre-trained models and their source code from which this repo was inspired from.