Pytorch implementation of the unsupervised object discovery method LOST.

Last update: Dec 25, 2022

Related tags

Overview

LOST

Pytorch implementation of the unsupervised object discovery method LOST. More details can be found in the paper:

Localizing Objects with Self-Supervised Transformers and no Labels [arXiv]
by Oriane Siméoni, Gilles Puy, Huy V. Vo, Simon Roburin, Spyros Gidaris, Andrei Bursuc, Patrick Pérez, Renaud Marlet and Jean Ponce

If you use the LOST code or framework in your research, please consider citing:

@article{LOST,
   title = {Localizing Objects with Self-Supervised Transformers and no Labels},
   author = {Oriane Sim\'eoni and Gilles Puy and Huy V. Vo and Simon Roburin and Spyros Gidaris and Andrei Bursuc and Patrick P\'erez and Renaud Marlet and Jean Ponce},
   journal = {arXiv preprint arXiv:2109.14279},
   month = {09},
   year = {2021}
}

Installation

Dependencies

This code was implemented with python 3.7, PyTorch 1.7.1 and CUDA 10.2. Please install PyTorch. In order to install the additionnal dependencies, please launch the following command:

pip install -r requirements.txt

Install DINO

This method is based on DINO paper. The framework can be installed using the following commands:

> __init__.py; cd ../; ">

git clone https://github.com/facebookresearch/dino.git
cd dino; 
touch __init__.py
echo -e "import sys\nfrom os.path import dirname, join\nsys.path.insert(0, join(dirname(__file__), '.'))" >> __init__.py; cd ../;

The code was made using the commit ba9edd1 of DINO repo (please rebase if breakage).

Apply LOST to one image

Following are scripts to apply LOST to an image defined via the image_path parameter and visualize the predictions (pred), the maps of the Figure 2 in the paper (fms) and the visulization of the seed expansion (seed_expansion). Box predictions are also stored in the output directory given by parameter output_dir.

python main_lost.py --image_path examples/VOC07_000236.jpg --visualize pred
python main_lost.py --image_path examples/VOC07_000236.jpg --visualize fms
python main_lost.py --image_path examples/VOC07_000236.jpg --visualize seed_expansion

Launching on datasets

Following are the different steps to reproduce the results of LOST presented in the paper.

PASCAL-VOC

Please download the PASCAL VOC07 and PASCAL VOC12 datasets (link) and put the data in the folder datasets. There should be the two subfolders: datasets/VOC2007 and datasets/VOC2012. In order to apply lost and compute corloc results (VOC07 61.9, VOC12 64.0), please launch:

python main_lost.py --dataset VOC07 --set trainval
python main_lost.py --dataset VOC12 --set trainval

COCO

Please download the COCO dataset and put the data in datasets/COCO. Results are provided given the 2014 annotations following previous works. The following command line allows you to get results on the subset of 20k images of the COCO dataset (corloc 50.7), following previous litterature. To be noted that the 20k images are a subset of the train set.

python main_lost.py --dataset COCO20k --set train

Different models

We have tested the method on different setups of the VIT model, corloc results are presented in the following table (more can be found in the paper).

arch	pre-training	dataset
		VOC07	VOC12	COCO20k
ViT-S/16	DINO	61.9	64.0	50.7
ViT-S/8	DINO	55.5	57.0	49.5
ViT-B/16	DINO	60.1	63.3	50.0
ResNet50	DINO	36.8	42.7	26.5
ResNet50	Imagenet	33.5	39.1	25.5

Previous results on the dataset VOC07 can be obtained by launching:

python main_lost.py --dataset VOC07 --set trainval #VIT-S/16
python main_lost.py --dataset VOC07 --set trainval --patch_size 8 #VIT-S/8
python main_lost.py --dataset VOC07 --set trainval --arch vit_base #VIT-B/16
python main_lost.py --dataset VOC07 --set trainval --arch resnet50 #Resnet50/DINO
python main_lost.py --dataset VOC07 --set trainval --arch resnet50_imagenet #Resnet50/imagenet

Comments

Is LOST designed to perform well with DINO features specifically?

I've replaced LOST's backbone (basically the dino weights) with the ones in CLIP, and it did not work well. But when switching back to dino weights, both ViT and ResNet50 backbone could generate good feature maps. Why would this happen?
question

opened by zengyuy 3
Error in evaluation with Detectron2

Hi @osimeoni,

Thank you for making the code available!

When evaluating Detectron2 on VOC12 with the obtained pseudolables. I obtain the following error: AttributeError: "int object has no attribute 'value'. It seems that the coco_style_file is not registered by 'register_coco_instances' (see image underneath). Any idea how this can be fixed? Thanks.

opened by MarcVisions 2
Class-aware detection

Do you plan on releasing code for class-aware detection (i.e., to produce the results in Table 3 of https://arxiv.org/pdf/2109.14279.pdf)? I don't believe I see any of the necessary code for assigning object categories to boxes, but please correct me if I'm wrong.

opened by gholste 2
Multi-object discovery

HI, I have a confusion about the interesting work. How to perform multi-target discovery in the figure 1 (middle) of your paper？ Any advice is greatly appreciated.
question

opened by rgbd-zml 1
Lost not performing well using DINO with fine-tuning

I’ve trained DINO’s model with my own Dataset, doing a finetuning on the ViT’s pre trained models of DINO. After a feel experiments I noticed that, every time that a epoch of the DINO’s finetune ran, the loss of the training reduce, however the IoU (the validation metric that we are using) of the bounding boxes generated by the LOST algorithm gets worse. Can anyone explain me why this is happening and how can I fix it?

opened by ericyoshida 1
corLoc evaluation

Hi @osimeoni . I am suspicious about the corLoc evaluation part in the code. The corLoc for each image is true whenever one of the ground truth objects is hit! https://github.com/valeoai/LOST/blob/2b678aca89c18aa79c56ec3f6d4a0b979a91608d/main_lost.py#L311 What about other objects? Is it right?

opened by Mirsadeghi 1

Pytorch implementation of the unsupervised object discovery method LOST.

Related tags

Overview

LOST

Installation

Dependencies

Install DINO

Apply LOST to one image

Launching on datasets

PASCAL-VOC

COCO

Different models

Comments

Is LOST designed to perform well with DINO features specifically?

Error in evaluation with Detectron2

Class-aware detection

Multi-object discovery

Lost not performing well using DINO with fine-tuning

corLoc evaluation

Releases(models)

models(Feb 28, 2022)

Owner

Valeo.ai

TensorFlow implementation of Style Transfer Generative Adversarial Networks: Learning to Play Chess Differently.

Data Augmentation with Variational Autoencoders

Semi-Supervised Semantic Segmentation with Cross-Consistency Training (CCT)

Official Code for ICML 2021 paper "Revisiting Point Cloud Shape Classification with a Simple and Effective Baseline"

This is a yolo3 implemented via tensorflow 2.7

A concise but complete implementation of CLIP with various experimental improvements from recent papers

Repository for the COLING 2020 paper "Explainable Automated Fact-Checking: A Survey."

salabim - discrete event simulation in Python

Learning To Have An Ear For Face Super-Resolution

A generalist algorithm for cell and nucleus segmentation.

Preprossing-loan-data-with-NumPy - In this project, I have cleaned and pre-processed the loan data that belongs to an affiliate bank based in the United States.

PyTorch code for ICPR 2020 paper Future Urban Scene Generation Through Vehicle Synthesis

PyTorch Code of "Memory In Memory: A Predictive Neural Network for Learning Higher-Order Non-Stationarity from Spatiotemporal Dynamics"

ROCKET: Exceptionally fast and accurate time series classification using random convolutional kernels

A Neural Net Training Interface on TensorFlow, with focus on speed + flexibility

This repository contains code for the paper "Disentangling Label Distribution for Long-tailed Visual Recognition", published at CVPR' 2021

PyTorch implementation of the wavelet analysis from Torrence & Compo

Image Completion with Deep Learning in TensorFlow

PyTorch implementation of the YOLO (You Only Look Once) v2

Compositional Sketch Search