https://arxiv.org/abs/2110.09246
This is a fast and scalable approach for detecting out-of-distribution test samples. It can be applied to any pretrained model.
# Assuming to have: trainloader, testloader, and model with model.backbone() and model.classifer() methods
# Extract the training set features. Dimensions 2D matrix: training_size x num_features
features = torch.vstack([model.backbone(images) for images, label in trainloader])
norm = torch.linalg.norm(features, dim=-1, keepdim=True)
features = features / norm
# Compute the training data empirical correlation matrix inverse
x_t_x_inv = torch.linalg.inv(features.T @ features)
# Calculate the regret: Large regret means out-of-distribution sample
for images, labels in testloader:
features = model.backbone(images)
# Normalize
norm = torch.linalg.norm(features, dim=-1, keepdim=True)
features = features / norm
# Get the probabilities of the normalized features
probs = torch.softmax(model.classifier(features), dim=-1)
# Calc projection
x_proj = features @ x_t_x_inv @ features.T
xt_g = x_proj / (1 + x_proj)
# Equation 20
n_classes = probs.shape[-1]
nf = torch.sum(probs / (probs + (1 - probs) * (probs ** x_t_g)), dim=-1)
regrets = torch.log(nf) / torch.log(torch.tensor(n_classes))
Install requirements
# Create env
conda create -n pnml_ood python=3.8.0 --yes
conda activate pnml_ood
# Install pip for fallback
conda install --yes pip
# Pytorch with GPU
conda install pytorch==1.8.0 torchvision==0.9.0 cudatoolkit=10.2 -c pytorch --yes
# All other: Install with conda. If package installation fails, install with pip.
while read requirement; do conda install --yes $requirement || pip install $requirement; done < requirements.txt Download data and models
# Download OOD data
cd bash_scripts
chmod 777 ./download_data.sh
./download_data.sh
# Download pretrained models
chmod 777 ./download_models.sh
./download_models.shFollow https://github.com/alinlab/CSI
Imagenet30 training set: https://drive.google.com/file/d/1B5c39Fc3haOPzlehzmpTLz6xLtGyKEy4/view
Imagenet30 testing set: https://drive.google.com/file/d/13xzVuQMEhSnBRZr-YaaO08coLU2dxAUq/view
Put and untar under ./data/Imagenet30
.
├── README.md
├── data
│ ├── Imagenet30
│ │ ├── one_class_test
│ │ ├── one_class_test.tar
│ │ ├── one_class_train
│ │ └── one_class_train.tar
Execute a single method. Our pNML method runs on-top of the executed method
cd src
python main_execute_method.py model=densenet trainset=cifar100 method=baselineExecute all methods
cd bash_scripts
chmod 777 ./execute_methods.sh
./execute_methods.shCreate paper's tables
cd src
python main_create_tables.pyIf you use this code in your research or wish to refer to the baseline results, please use the following BibTeX entry.
@inproceedings{bibas2021single,
title={Single Layer Predictive Normalized Maximum Likelihood for Out-of-Distribution Detection},
author={Bibas, Koby and Feder, Meir and Hassner, Tal},
booktitle={Advances in Neural Information Processing Systems},
year={2021}
}