Visual Memorability for Robotic Interestingness via Unsupervised Online Learning (ECCV 2020 Oral and TRO)

Last update: Sep 08, 2022

Overview

Visual Interestingness

Refer to the project description for more details.
This code based on the following paper.

Chen Wang, Yuheng Qiu, Wenshan Wang, Yafei Hu, Seungchan Kim and Sebastian Scherer, Unsupervised Online Learning for Robotic Interestingness with Visual Memory, IEEE Transactions on Robotics (T-RO), 2021.
It is an extended version of the conference paper:

Chen Wang, Wenshan Wang, Yuheng Qiu, Yafei Hu, and Sebastian Scherer, Visual Memorability for Robotic Interestingness via Unsupervised Online Learning, European Conference on Computer Vision (ECCV), 2020.
If you want the original version, go to the ECCV Branch instead.
We also provide ROS wrapper for this project, you may go to interestingness_ros.

Install Dependencies

This version is tested in PyTorch 1.7

  pip3 install -r requirements.txt

Long-term Learning

You may skip this step, if you download the pre-trained vgg16.pt into folder "saves".

Download coco dataset into folder [data-root]:

bash download_coco.sh [data-root] # replace [data-root] by your desired location

The dataset will be look like:

data-root
├──coco
   ├── annotations
   │   ├── annotations_trainval2017
   │   └── image_info_test2017
   └── images
       ├── test2017
       ├── train2017
       └── val2017

Run

python3 longterm.py --data-root [data-root] --model-save saves/vgg16.pt

# This requires a long time for training on single GPU.
# Create a folder "saves" manually and a model named "ae.pt" will be saved.

Short-term Learning

Dowload the SubT front camera data (SubTF) and put into folder "data-root", so that it looks like:

data-root
├──SubTF
   ├── 0817-ugv0-tunnel0
   ├── 0817-ugv1-tunnel0
   ├── 0818-ugv0-tunnel1
   ├── 0818-ugv1-tunnel1
   ├── 0820-ugv0-tunnel1
   ├── 0821-ugv0-tunnel0
   ├── 0821-ugv1-tunnel0
   ├── ground-truth
   └── train

Run

python3 shortterm.py --data-root [data-root] --model-save saves/vgg16.pt --dataset SubTF --memory-size 100 --save-flag n100usage

# This will read the previous model "ae.pt".
# A new model "ae.pt.SubTF.n1000.mse" will be generated.

You may skip this step, if you download the pre-trained vgg16.pt.SubTF.n100usage.mse into folder "saves".

On-line Learning

Run

  python3 online.py --data-root [data-root] --model-save saves/vgg16.pt.SubTF.n100usage.mse --dataset SubTF --test-data 0 --save-flag n100usage

  # --test-data The sequence ID in the dataset SubTF, [0-6] is avaiable
  # This will read the trained model "vgg16.pt.SubTF.n100usage.mse" from short-term learning.

Alternatively, you may test all sequences by running
```
  bash test.sh
```
This will generate results files in folder "results".
You may skip this step, if you download our generated results.

Evaluation

We follow the SubT tutorial for evaluation, simply run

python performance.py --data-root [data-root] --save-flag n100usage --category normal --delta 1 2 3
# mean accuracy: [0.64455275 0.8368784  0.92165116 0.95906876]

python performance.py --data-root [data-root] --save-flag n100usage --category difficult --delta 1 2 4
# mean accuracy: [0.42088688 0.57836163 0.67878168 0.75491805]

This will generate performance figures and create data curves for two categories in folder "performance".

Citation

      @inproceedings{wang2020visual,
        title={Visual memorability for robotic interestingness via unsupervised online learning},
        author={Wang, Chen and Wang, Wenshan and Qiu, Yuheng and Hu, Yafei and Scherer, Sebastian},
        booktitle={European Conference on Computer Vision (ECCV)},
        year={2020},
        organization={Springer}
      }
      
      @article{wang2021unsupervised,
        title={Unsupervised Online Learning for Robotic Interestingness with Visual Memory},
        author={Wang, Chen and  Qiu, Yuheng and Wang, Wenshan and Hu, Yafei anad Kim, Seungchan and Scherer, Sebastian},
        journal={IEEE Transactions on Robotics (T-RO)},
        year={2021},
        publisher={IEEE}
      }

Download conferencec version paper.
Download journal version paper.

You may watch the following video to catch the idea of this work.

Code for the paper "Improving Vision-and-Language Navigation with Image-Text Pairs from the Web" (ECCV 2020)

Improving Vision-and-Language Navigation with Image-Text Pairs from the Web Arjun Majumdar, Ayush Shrivastava, Stefan Lee, Peter Anderson, Devi Parikh

44 Dec 14, 2022

Code for ECCV 2020 paper "Contacts and Human Dynamics from Monocular Video".

Contact and Human Dynamics from Monocular Video This is the official implementation for the ECCV 2020 spotlight paper by Davis Rempe, Leonidas J. Guib

207 Jan 5, 2023

Repository for Traffic Accident Benchmark for Causality Recognition (ECCV 2020)

Causality In Traffic Accident (Under Construction) Repository for Traffic Accident Benchmark for Causality Recognition (ECCV 2020) Overview Data Prepa

21 Nov 20, 2022

Code for our paper at ECCV 2020: Post-Training Piecewise Linear Quantization for Deep Neural Networks

PWLQ Updates 2020/07/16 - We are working on getting permission from our institution to release our source code. We will release it once we are granted

54 Dec 15, 2022

dataset for ECCV 2020 "Motion Capture from Internet Videos"

Motion Capture from Internet Videos Motion Capture from Internet Videos Junting Dong*, Qing Shuai*, Yuanqing Zhang, Xian Liu, Xiaowei Zhou, Hujun Bao

98 Dec 7, 2022

Code for the paper: Adversarial Training Against Location-Optimized Adversarial Patches. ECCV-W 2020.

Adversarial Training Against Location-Optimized Adversarial Patches arXiv | Paper | Code | Video | Slides Code for the paper: Sukrut Rao, David Stutz,

32 Dec 13, 2022

SNE-RoadSeg in PyTorch, ECCV 2020

SNE-RoadSeg Introduction This is the official PyTorch implementation of SNE-RoadSeg: Incorporating Surface Normal Information into Semantic Segmentati

242 Dec 20, 2022

[ECCV 2020] Gradient-Induced Co-Saliency Detection

Gradient-Induced Co-Saliency Detection Zhao Zhang*, Wenda Jin*, Jun Xu, Ming-Ming Cheng ⭐ Project Home » The official repo of the ECCV 2020 paper Grad

35 Nov 25, 2022

Code for Towards Streaming Perception (ECCV 2020) :car:

sAP — Code for Towards Streaming Perception ECCV Best Paper Honorable Mention Award Feb 2021: Announcing the Streaming Perception Challenge (CVPR 2021

85 Dec 22, 2022

Comments

Variable

https://github.com/wang-chen/interestingness/blob/6994d50bd47d14b617f34f5c36c1beaba03acfdc/test_interest.py#L94

I think using Variable() will just return a tensor object in the new pytorch version.

opened by haleqiu 2

Visual Memorability for Robotic Interestingness via Unsupervised Online Learning (ECCV 2020 Oral and TRO)

Related tags

Overview

Visual Interestingness

Install Dependencies

Long-term Learning

Short-term Learning

On-line Learning

Evaluation

Citation

You might also like...

Code for the paper "Improving Vision-and-Language Navigation with Image-Text Pairs from the Web" (ECCV 2020)

Code for ECCV 2020 paper "Contacts and Human Dynamics from Monocular Video".

Repository for Traffic Accident Benchmark for Causality Recognition (ECCV 2020)

Code for our paper at ECCV 2020: Post-Training Piecewise Linear Quantization for Deep Neural Networks

dataset for ECCV 2020 "Motion Capture from Internet Videos"

Code for the paper: Adversarial Training Against Location-Optimized Adversarial Patches. ECCV-W 2020.

SNE-RoadSeg in PyTorch, ECCV 2020

[ECCV 2020] Gradient-Induced Co-Saliency Detection

Code for Towards Streaming Perception (ECCV 2020) :car:

Comments

Variable

Releases(v2.0)

v2.0(Apr 12, 2021)

v1.0(Jun 19, 2020)

Owner

Chen Wang

ICON: Implicit Clothed humans Obtained from Normals (CVPR 2022)

[ECCV2020] Content-Consistent Matching for Domain Adaptive Semantic Segmentation

Mmdet benchmark with python

I decide to sync up this repo and self-critical.pytorch. (The old master is in old master branch for archive)

PyTorch Implementation of Daft-Exprt: Robust Prosody Transfer Across Speakers for Expressive Speech Synthesis

NU-Wave: A Diffusion Probabilistic Model for Neural Audio Upsampling

[EMNLP 2021] Distantly-Supervised Named Entity Recognition with Noise-Robust Learning and Language Model Augmented Self-Training

Gender Classification Machine Learning Model using Sk-learn in Python with 97%+ accuracy and deployment

MG-GCN: Scalable Multi-GPU GCN Training Framework

tensorflow code for inverse face rendering

​TextWorld is a sandbox learning environment for the training and evaluation of reinforcement learning (RL) agents on text-based games.

Classical OCR DCNN reproduction based on PaddlePaddle framework.

DeepMetaHandles: Learning Deformation Meta-Handles of 3D Meshes with Biharmonic Coordinates

A Multi-modal Model Chinese Spell Checker Released on ACL2021.

Learning nonlinear operators via DeepONet

official Pytorch implementation of ICCV 2021 paper FuseFormer: Fusing Fine-Grained Information in Transformers for Video Inpainting.

[ICCV2021] Official Pytorch implementation for SDGZSL (Semantics Disentangling for Generalized Zero-Shot Learning)

Implementation of SegNet: A Deep Convolutional Encoder-Decoder Architecture for Semantic Pixel-Wise Labelling

This repository implements Douzero's interface to IGCA.

Propose a principled and practically effective framework for unsupervised accuracy estimation and error detection tasks with theoretical analysis and state-of-the-art performance.

TextWorld is a sandbox learning environment for the training and evaluation of reinforcement learning (RL) agents on text-based games.