Official repository of "BasicVSR++: Improving Video Super-Resolution with Enhanced Propagation and Alignment"

Overview

BasicVSR_PlusPlus (CVPR 2022)

[Paper] [Project Page] [Code]

This is the official repository for BasicVSR++. Please feel free to raise issue related to BasicVSR++! If you are also interested in RealBasicVSR, which is also accepted to CVPR 2022, please don't hesitate to star!

Authors: Kelvin C.K. Chan, Shangchen Zhou, Xiangyu Xu, Chen Change Loy, Nanyang Technological University

Acknowedgement: Our work is built upon MMEditing. Please follow and star this repository and MMEditing!

News

  • 2 Dec 2021: Colab demo released google colab logo
  • 18 Apr 2022: Code released. Also merged into MMEditing

TODO

  • Add data processing scripts
  • Add checkpoints for deblur and denoise
  • Add configs for deblur and denoise
  • Add Colab demo

Pre-trained Weights

You can find the pre-trained weights for deblurring and denoising in this link. For super-resolution and compressed video enhancement, please refer to MMEditing.

Installation

  1. Install PyTorch
  2. pip install openmim
  3. mim install mmcv-full
  4. git clone https://github.com/ckkelvinchan/BasicVSR_PlusPlus.git
  5. cd BasicVSR_PlusPlus
  6. pip install -v -e .

Inference a Video

  1. Download pre-trained weights
  2. python demo/restoration_video_demo.py ${CONFIG} ${CHKPT} ${IN_PATH} ${OUT_PATH}

For example, you can download the VSR checkpoint here to chkpts/basicvsr_plusplus_reds4.pth, then run

python demo/restoration_video_demo.py configs/basicvsr_plusplus_reds4.py chkpts/basicvsr_plusplus_reds4.pth data/demo_000 results/demo_000

You can also replace ${IN_PATH} ${OUT_PATH} by your video path (e.g., xxx/yyy.mp4) to input/output videos.

Training Models

  1. Put the dataset in the designated locations specified in the configuration file.
  2. sh tools/dist_train.sh ${CONFIG} ${NGPUS}

Data Preprocessing

To be added...

Related Work

Our BasicVSR series:

  1. BasicVSR: The Search for Essential Components in Video Super-Resolution and Beyond, CVPR 2021
  2. Investigating Tradeoffs in Real-World Video Super-Resolution, CVPR 2022

More about deformable alignment:

Citations

@inproceedings{chan2022basicvsrpp,
  author = {Chan, Kelvin C.K. and Zhou, Shangchen and Xu, Xiangyu and Loy, Chen Change},
  title = {{BasicVSR++}: Improving video super-resolution with enhanced propagation and alignment},
  booktitle = {IEEE Conference on Computer Vision and Pattern Recognition},
  year = {2022}
}
@article{chan2022generalization,
  title={On the Generalization of {BasicVSR++} to Video Deblurring and Denoising},
  author={Chan, Kelvin CK and Zhou, Shangchen and Xu, Xiangyu and Loy, Chen Change},
  journal={arXiv preprint arXiv:2204.05308},
  year={2022}
}
Issues
  • denoise pretrained model  didn't work

    denoise pretrained model didn't work

    Hi @ckkelvinchan , I have tried to run the denoise pretrianed model with DAVIS-test dataset, but the result images has no difference between the input images. My experiment is as below: pretrained model: basicvsr_plusplus_denoise-28f6920c.pth optical flow model: I download it to my local path dataset: I use the the tractor folder in DAIVS-test dataset which is used in your paper. I add the sigma 50 gaussian noise to the pictures as input. result: image

    It seem that it run successfully, but the result has no change between the input. Is their any parameter or option error in my attempt? By the way, basicvsr_plusplus_c64n7_8x1_600k_reds4.pth can run successfully and the result is excellent!

    opened by bjiale 8
  • Question for the computation of cond__n1

    Question for the computation of cond__n1

    why the cond_n1 is the out of flow_wrap(feat_prop, flow_n1.permute(0, 2, 3, 1)) while feat_prop is zeros? why not flow_wrap(feat_n1,flow_n1.permute(0, 2, 3, 1) and feat_n1 = feats[module_name][-1]

    opened by jiang-xingbp 4
  • Encountered error while trying to install package mmcv-full

    Encountered error while trying to install package mmcv-full

    I installed pytorch using the following command conda install pytorch torchvision torchaudio cudatoolkit=11.3 -c pytorch I am trying to install mmcv-full. I am facing the error

    opened by purna821 1
  • Data Preparation

    Data Preparation

    Hi, thank you very much for your excellent work. I wonder could you please show us the folders of the dataset like the following forms? So that I can recollect the dataset into the same form as yours. image

    opened by linjing7 0
  •  CUDA out of memory

    CUDA out of memory

    Hi @ckkelvinchan , I have tried running inference on RTX2060 with 6gb of vram but it doesn't perform inference on low memory. I tried using video.mp4 as input. Is their any parameter or option to increase the inference time by some tile or batch size limit but perform inference ?

    opened by Muhammad-Ahmad-Ghani 1
Owner
Kelvin C.K. Chan
Kelvin C.K. Chan
Official repository for "PAIR: Planning and Iterative Refinement in Pre-trained Transformers for Long Text Generation"

pair-emnlp2020 Official repository for the paper: Xinyu Hua and Lu Wang: PAIR: Planning and Iterative Refinement in Pre-trained Transformers for Long

Xinyu Hua 24 Jun 13, 2022
Official repository for Few-shot Image Generation via Cross-domain Correspondence (CVPR '21)

Few-shot Image Generation via Cross-domain Correspondence Utkarsh Ojha, Yijun Li, Jingwan Lu, Alexei A. Efros, Yong Jae Lee, Eli Shechtman, Richard Zh

Utkarsh Ojha 226 Jun 18, 2022
Official repository for Jia, Raghunathan, Göksel, and Liang, "Certified Robustness to Adversarial Word Substitutions" (EMNLP 2019)

Certified Robustness to Adversarial Word Substitutions This is the official GitHub repository for the following paper: Certified Robustness to Adversa

Robin Jia 36 Apr 20, 2022
Official repository for the ICLR 2021 paper Evaluating the Disentanglement of Deep Generative Models with Manifold Topology

Official repository for the ICLR 2021 paper Evaluating the Disentanglement of Deep Generative Models with Manifold Topology Sharon Zhou, Eric Zelikman

Stanford Machine Learning Group 33 Dec 14, 2021
The repository offers the official implementation of our paper in PyTorch.

Cloth Interactive Transformer (CIT) Cloth Interactive Transformer for Virtual Try-On Bin Ren1, Hao Tang1, Fanyang Meng2, Runwei Ding3, Ling Shao4, Phi

Bingoren 45 May 9, 2022
Official code repository of the paper Learning Associative Inference Using Fast Weight Memory by Schlag et al.

Learning Associative Inference Using Fast Weight Memory This repository contains the offical code for the paper Learning Associative Inference Using F

Imanol Schlag 16 Dec 12, 2021
Official repository for "Action-Based Conversations Dataset: A Corpus for Building More In-Depth Task-Oriented Dialogue Systems"

Action-Based Conversations Dataset (ABCD) This respository contains the code and data for ABCD (Chen et al., 2021) Introduction Whereas existing goal-

ASAPP Research 45 Jun 7, 2022
Official repository for HOTR: End-to-End Human-Object Interaction Detection with Transformers (CVPR'21, Oral Presentation)

Official PyTorch Implementation for HOTR: End-to-End Human-Object Interaction Detection with Transformers (CVPR'2021, Oral Presentation) HOTR: End-to-

Kakao Brain 100 Jun 27, 2022
Official repository for "Intriguing Properties of Vision Transformers" (2021)

Intriguing Properties of Vision Transformers Muzammal Naseer, Kanchana Ranasinghe, Salman Khan, Munawar Hayat, Fahad Shahbaz Khan, & Ming-Hsuan Yang P

Muzammal Naseer 131 Jun 24, 2022
Competitive Programming Club, Clinify's Official repository for CP problems hosting by club members.

Clinify-CPC_Programs This repository holds the record of the competitive programming club where the competitive coding aspirants are thriving hard and

Clinify Open Sauce 3 May 26, 2021
Official repository for "On Improving Adversarial Transferability of Vision Transformers" (2021)

Improving-Adversarial-Transferability-of-Vision-Transformers Muzammal Naseer, Kanchana Ranasinghe, Salman Khan, Fahad Khan, Fatih Porikli arxiv link A

Muzammal Naseer 37 May 8, 2022
This is the official repository of XVFI (eXtreme Video Frame Interpolation)

XVFI This is the official repository of XVFI (eXtreme Video Frame Interpolation), https://arxiv.org/abs/2103.16206 Last Update: 20210607 We provide th

Jihyong Oh 177 Jun 29, 2022
The official repository for BaMBNet

BaMBNet-Pytorch Paper

Junjun Jiang 16 Apr 1, 2022
CVPR 2021 - Official code repository for the paper: On Self-Contact and Human Pose.

selfcontact This repo is part of our project: On Self-Contact and Human Pose. [Project Page] [Paper] [MPI Project Page] It includes the main function

Lea Müller 61 Jun 11, 2022
CVPR 2021 - Official code repository for the paper: On Self-Contact and Human Pose.

SMPLify-XMC This repo is part of our project: On Self-Contact and Human Pose. [Project Page] [Paper] [MPI Project Page] License Software Copyright Lic

Lea Müller 77 Jun 22, 2022
Official repository for the paper "Going Beyond Linear Transformers with Recurrent Fast Weight Programmers"

Recurrent Fast Weight Programmers This is the official repository containing the code we used to produce the experimental results reported in the pape

IDSIA 34 Jun 22, 2022
Official repository for the paper "Can You Learn an Algorithm? Generalizing from Easy to Hard Problems with Recurrent Networks"

Easy-To-Hard The official repository for the paper "Can You Learn an Algorithm? Generalizing from Easy to Hard Problems with Recurrent Networks". Gett

Avi Schwarzschild 48 Apr 4, 2022
Official repository for the CVPR 2021 paper "Learning Feature Aggregation for Deep 3D Morphable Models"

Deep3DMM Official repository for the CVPR 2021 paper Learning Feature Aggregation for Deep 3D Morphable Models. Requirements This code is tested on Py

null 31 Jun 11, 2022