DeFMO: Deblurring and Shape Recovery of Fast Moving Objects (CVPR 2021)

Last update: Dec 26, 2022

Overview

Evaluation, Training, Demo, and Inference of DeFMO

DeFMO: Deblurring and Shape Recovery of Fast Moving Objects (CVPR 2021)

Denys Rozumnyi, Martin R. Oswald, Vittorio Ferrari, Jiri Matas, Marc Pollefeys

Qualitative results: https://www.youtube.com/watch?v=pmAynZvaaQ4

Pre-trained models

The pre-trained DeFMO model as reported in the paper is available here: https://polybox.ethz.ch/index.php/s/M06QR8jHog9GAcF. Put them into ./saved_models sub-folder.

Inference

For generating video temporal super-resolution:

python run.py --video example/falling_pen.avi

For generating temporal super-resolution of a single frame with the given background:

python run.py --im example/im.png --bgr example/bgr.png

Evaluation

After downloading the pre-trained models and downloading the evaluation datasets, you can run

python eval_dataset.py

Synthetic dataset generation

For the dataset generation, please download:

ShapeNetCore.v2 dataset: https://www.shapenet.org/.
Textures from the DTD dataset: https://www.robots.ox.ac.uk/~vgg/data/dtd/. The exact split used in DeFMO is from the "Neural Voxel Renderer: Learning an Accurate and Controllable Rendering Tool" model and can be downloaded here: https://polybox.ethz.ch/index.php/s/9Abv3QRm0ZgPzhK.
Backgrounds for the training dataset from the VOT dataset: https://www.votchallenge.net/vot2018/dataset.html.
Backgrounds for the testing dataset from the Sports1M dataset: https://cs.stanford.edu/people/karpathy/deepvideo/.
Blender 2.79b with Python enabled.

Then, insert your paths in renderer/settings.py file. To generate the dataset, run in renderer sub-folder:

python run_render.py

Note that the full training dataset with 50 object categories, 1000 objects per category, and 24 timestamps takes up to 1 TB of storage memory. Due to this and also the ShapeNet licence, we cannot make the pre-generated dataset public - please generate it by yourself using the steps above.

Training

Set up all paths in main_settings.py and run

python train.py

Evaluation on real-world datasets

All evaluation datasets can be found at http://cmp.felk.cvut.cz/fmo/. We provide a download_datasets.sh script to download the Falling Objects, the TbD-3D, and the TbD datasets.

Reference

If you use this repository, please cite the following publication ( https://arxiv.org/abs/2012.00595 ):

@inproceedings{defmo,
  author = {Denys Rozumnyi and Martin R. Oswald and Vittorio Ferrari and Jiri Matas and Marc Pollefeys},
  title = {DeFMO: Deblurring and Shape Recovery of Fast Moving Objects},
  booktitle = {CVPR},
  address = {Nashville, Tennessee, USA},
  month = jun,
  year = {2021}
}

Comments

Question about training set

Hi, thanks for your generous sharing.

I have a question about training set generating in your work. I generated a training set following your codes. Its size is about 100GB, far less than 1TB. Is there anything wrong?

Thanks.

opened by fan-hd 11
Apply your model on custom longer video clips

Hi thank you for releasing your code,

Can your model be applied on custom videos about high speed train crossing? Video clips last from 3 to 10 seconds, my idea was to preprocess them with your code in order to keep the same frame rate and have a better video quality for later object detection. This is an example frame from original video clip:

I tried to run your code on a video about 6 seconds and the result was a longer video (about 13min) with a lower level of detail, probably I'm doing something wrong. This is an example frame from output video clip:

How can I correctly reconstruct the quality of single frames usin all the information contained in the video?

opened by fabiozappo 4
Question about comparison with Jin et al.'s work (CVPR2018)

Hi, thank you for your interesting work! I have a question about the comparison of methods in your work. When making comparisons, did you retrain Jin et al.'s model ("Learning to Extract a Video Sequence from a Single Motion-Blurred Image" from CVPR 2018), or did you just use their pre-trained checkpoints? I couldn't find the training code on their github page.

opened by zzh-tech 2
Padding in Time-Consistency Loss

Hi,

Congratulations!

I found that "padding = tuple(side // 10 for side in sh[:2]) + (0,)" for normalized cross-correlation. Does it only implement padding to the height axis, since the padding tuple will be of size (4//10, H//10, 0)?

Thanks a lot.

opened by JLiu-Edinburgh 1
run on google colab!

I'm confused! and need to run the code on google colab or more explanation about how to implement that code in vscode or something else .if it know someone please help me

opened by ganikas 3

Releases(v1.0)

v1.0(Feb 7, 2021)

The first release of the method reported in the arxiv paper.
Source code(tar.gz)
Source code(zip)

Owner

Denys Rozumnyi

PhD student at ETH Zurich.

GitHub Repository https://github.com/rozumden/DeFMO

Efficient Sparse Attacks on Videos using Reinforcement Learning

EARL This repository provides a simple implementation of the work "Efficient Sparse Attacks on Videos using Reinforcement Learning" Example: Demo: Her

12 Dec 05, 2021

A chemical analysis of lipophilicities & molecule drawings including ML

A chemical analysis of lipophilicity & molecule drawings including a bit of ML analysis. This is a simple project that includes two Jupyter files (one

7 Nov 22, 2022

Tensorflow implementation of "Learning Deep Features for Discriminative Localization"

Weakly_detector Tensorflow implementation of "Learning Deep Features for Discriminative Localization" B. Zhou, A. Khosla, A. Lapedriza, A. Oliva, and

363 Jun 29, 2022

The project is an official implementation of our CVPR2019 paper "Deep High-Resolution Representation Learning for Human Pose Estimation"

Deep High-Resolution Representation Learning for Human Pose Estimation (CVPR 2019) News [2020/07/05] A very nice blog from Towards Data Science introd

3.9k Jan 05, 2023

A python script to convert images to animated sus among us crewmate twerk jifs as seen on r/196

img_sussifier A python script to convert images to animated sus among us crewmate twerk jifs as seen on r/196 Examples How to use install python pip i

41 Sep 30, 2022

Differentiable architecture search for convolutional and recurrent networks

Differentiable Architecture Search Code accompanying the paper DARTS: Differentiable Architecture Search Hanxiao Liu, Karen Simonyan, Yiming Yang. arX

3.7k Jan 09, 2023

Code for "ATISS: Autoregressive Transformers for Indoor Scene Synthesis", NeurIPS 2021

ATISS: Autoregressive Transformers for Indoor Scene Synthesis This repository contains the code that accompanies our paper ATISS: Autoregressive Trans

138 Dec 22, 2022

Implementation detail for paper "Multi-level colonoscopy malignant tissue detection with adversarial CAC-UNet"

Multi-level-colonoscopy-malignant-tissue-detection-with-adversarial-CAC-UNet Implementation detail for our paper "Multi-level colonoscopy malignant ti

[email protected]"> 84 Nov 22, 2022

Wandb-predictions - WANDB Predictions With Python

WANDB API CI/CD Below we capture the CI/CD scenarios that we would expect with o

6 Oct 07, 2022

Tianshou - An elegant PyTorch deep reinforcement learning library.

Tianshou (天授) is a reinforcement learning platform based on pure PyTorch. Unlike existing reinforcement learning libraries, which are mainly based on

5.5k Jan 05, 2023

Official PyTorch Implementation for "Recurrent Video Deblurring with Blur-Invariant Motion Estimation and Pixel Volumes"

PVDNet: Recurrent Video Deblurring with Blur-Invariant Motion Estimation and Pixel Volumes This repository contains the official PyTorch implementatio

98 Nov 06, 2022

Nicely is a real-time Feedback and Intervention Program Depression is a prevalent issue across all age groups, socioeconomic classes, and cultural identities.

1 Jan 16, 2022

DeFMO: Deblurring and Shape Recovery of Fast Moving Objects (CVPR 2021)

Related tags

Overview

Evaluation, Training, Demo, and Inference of DeFMO

DeFMO: Deblurring and Shape Recovery of Fast Moving Objects (CVPR 2021)

Denys Rozumnyi, Martin R. Oswald, Vittorio Ferrari, Jiri Matas, Marc Pollefeys

Qualitative results: https://www.youtube.com/watch?v=pmAynZvaaQ4

Pre-trained models

Inference

Evaluation

Synthetic dataset generation

Training

Evaluation on real-world datasets

Reference

Comments

Question about training set

Apply your model on custom longer video clips

Question about comparison with Jin et al.'s work (CVPR2018)

Padding in Time-Consistency Loss

run on google colab!

Releases(v1.0)

v1.0(Feb 7, 2021)

Owner

Denys Rozumnyi

Efficient Sparse Attacks on Videos using Reinforcement Learning

A chemical analysis of lipophilicities & molecule drawings including ML

Tensorflow implementation of "Learning Deep Features for Discriminative Localization"

The project is an official implementation of our CVPR2019 paper "Deep High-Resolution Representation Learning for Human Pose Estimation"

A python script to convert images to animated sus among us crewmate twerk jifs as seen on r/196

Differentiable architecture search for convolutional and recurrent networks

Code for "ATISS: Autoregressive Transformers for Indoor Scene Synthesis", NeurIPS 2021

Implementation detail for paper "Multi-level colonoscopy malignant tissue detection with adversarial CAC-UNet"

Wandb-predictions - WANDB Predictions With Python

Tianshou - An elegant PyTorch deep reinforcement learning library.

Official PyTorch Implementation for "Recurrent Video Deblurring with Blur-Invariant Motion Estimation and Pixel Volumes"

Nicely is a real-time Feedback and Intervention Program Depression is a prevalent issue across all age groups, socioeconomic classes, and cultural identities.

Project ArXiv Citation Network

The official implementation of Equalization Loss v1 & v2 (CVPR 2020, 2021) based on MMDetection.

A modular, primitive-first, python-first PyTorch library for Reinforcement Learning.

TF Image Segmentation: Image Segmentation framework

Convert dog pictures into various painting styles. Try LimnPet

Breaking the Curse of Space Explosion: Towards Efficient NAS with Curriculum Search

Text and code for the forthcoming second edition of Think Bayes, by Allen Downey.

Roadmap to becoming a machine learning engineer in 2020