Open Source Light Field Toolbox for Super-Resolution

Overview

BasicLFSR

BasicLFSR is an open-source and easy-to-use Light Field (LF) image Super-Ressolution (SR) toolbox based on PyTorch, including a collection of papers on LF image SR and a benchmark to comprehensively evaluate the performance of existing methods. We also provided simple pipelines to train/valid/test state-of-the-art methods to get started quickly, and you can transform your methods into the benchmark.

Note: This repository will be updated on a regular basis, and the pretrained models of existing methods will be open-sourced one after another. So stay tuned!

Methods

Methods Paper Repository
LFSSR Light Field Spatial Super-Resolution Using Deep Efficient Spatial-Angular Separable Convolution. TIP2018 spatialsr/
DeepLightFieldSSR
resLF Residual Networks for Light Field Image Super-Resolution. CVPR2019 shuozh/resLF
HDDRNet High-Dimensional Dense Residual Convolutional Neural Network for Light Field Reconstruction. TPAMI2019 monaen/
LightFieldReconstruction
LF-InterNet Spatial-Angular Interaction for Light Field Image Super-Resolution. ECCV2019 YingqianWang/
LF-InterNet
LFSSR-ATO Light field spatial super-resolution via deep combinatorial geometry embedding and structural consistency regularization. CVPR2020 jingjin25/
LFSSR-ATO
LF-DFnet Light field image super-resolution using deformable convolution. TIP2020 YingqianWang/
LF-DFnet
MEG-Net End-to-End Light Field Spatial Super-Resolution Network using Multiple Epipolar Geometry. TIP2021 shuozh/MEG-Net

Datasets

We used the EPFL, HCInew, HCIold, INRIA and STFgantry datasets for both training and test. Please first download our datasets via Baidu Drive (key:7nzy) or OneDrive, and place the 5 datasets to the folder ./datasets/.

  • After downloading, you should find following structure:

    ├──./datasets/
    │    ├── EPFL
    │    │    ├── training
    │    │    │    ├── Bench_in_Paris.mat
    │    │    │    ├── Billboards.mat
    │    │    │    ├── ...
    │    │    ├── test
    │    │    │    ├── Bikes.mat
    │    │    │    ├── Books__Decoded.mat
    │    │    │    ├── ...
    │    ├── HCI_new
    │    ├── ...
    
  • Run Generate_Data_for_Training.m to generate training data. The generated data will be saved in ./data_for_train/ (SR_5x5_2x, SR_5x5_4x).

  • Run Generate_Data_for_Test.m to generate test data. The generated data will be saved in ./data_for_test/ (SR_5x5_2x, SR_5x5_4x).

Benchmark

We benchmark several methods on above datasets, and PSNR and SSIM metrics are used for quantitative evaluation.

PSNR and SSIM values achieved by different methods for 2xSR:

Method Scale #Params. EPFL HCInew HCIold INRIA STFgantry Average
Bilinear x2 -- 28.479949/0.918006 30.717944/0.919248 36.243278/0.970928 30.133901/0.945545 29.577468/0.931030 31.030508/0.936951
Bicubic x2 -- 29.739509/0.937581 31.887011/0.935637 37.685776/0.978536 31.331483/0.957731 31.062631/0.949769 32.341282/0.951851
VDSR x2
EDSR x2 33.088922/0.962924 34.828374/0.959156 41.013989/0.987400 34.984982/0.976397 36.295865/0.981809
RCSN x2
resLF x2
LFSSR x2 33.670594/0.974351 36.801555/0.974910 43.811050/0.993773 35.279443/0.983202 37.943969/0.989818
LF-ATO x2 34.271635/0.975711 37.243620/0.976684 44.205264/0.994202 36.169943/0.984241 39.636445/0.992862
LF-InterNet x2
LF-DFnet x2
MEG-Net x2
LFT x2

PSNR and SSIM values achieved by different methods for 4xSR:

Method Scale #Params. EPFL HCInew HCIold INRIA STFgantry Average
Bilinear x4 -- 24.567490/0.815793 27.084949/0.839677 31.688225/0.925630 26.226265/0.875682 25.203262/0.826105 26.954038/0.856577
Bicubic x4 -- 25.264206/0.832389 27.714905/0.851661 32.576315/0.934428 26.951718/0.886740 26.087451/0.845230 27.718919/0.870090
VDSR x4
EDSR x4
RCSN x4
resLF x4
LFSSR x4
LF-ATO x4
LF-InterNet x4
LF-DFnet x4
MEG-Net x4
LFT x4

Train

  • Run train.py to perform network training. Example for training [model_name] on 5x5 angular resolution for 2x/4x SR:
    $ python train.py --model_name [model_name] --angRes 5 --scale_factor 2 --batch_size 8
    $ python train.py --model_name [model_name] --angRes 5 --scale_factor 4 --batch_size 4
    
  • Checkpoints and Logs will be saved to ./log/, and the ./log/ has following structure:
    ├──./log/
    │    ├── SR_5x5_2x
    │    │    ├── [dataset_name]
    │    │         ├── [model_name]
    │    │         │    ├── [model_name]_log.txt
    │    │         │    ├── checkpoints
    │    │         │    │    ├── [model_name]_5x5_2x_epoch_01_model.pth
    │    │         │    │    ├── [model_name]_5x5_2x_epoch_02_model.pth
    │    │         │    │    ├── ...
    │    │         │    ├── results
    │    │         │    │    ├── VAL_epoch_01
    │    │         │    │    ├── VAL_epoch_02
    │    │         │    │    ├── ...
    │    │         ├── [other_model_name]
    │    │         ├── ...
    │    ├── SR_5x5_4x
    

Test

  • Run test.py to perform network inference. Example for test [model_name] on 5x5 angular resolution for 2x/4xSR:

    $ python test.py --model_name [model_name] --angRes 5 --scale_factor 2  
    $ python test.py --model_name [model_name] --angRes 5 --scale_factor 4 
    
  • The PSNR and SSIM values of each dataset will be saved to ./log/, and the ./log/ is following structure:

    ├──./log/
    │    ├── SR_5x5_2x
    │    │    ├── [dataset_name]
    │    │        ├── [model_name]
    │    │        │    ├── [model_name]_log.txt
    │    │        │    ├── checkpoints
    │    │        │    │   ├── ...
    │    │        │    ├── results
    │    │        │    │    ├── Test
    │    │        │    │    │    ├── evaluation.xls
    │    │        │    │    │    ├── [dataset_1_name]
    │    │        │    │    │    │    ├── [scene_1_name]
    │    │        │    │    │    │    │    ├── [scene_1_name]_CenterView.bmp
    │    │        │    │    │    │    │    ├── [scene_1_name]_SAI.bmp
    │    │        │    │    │    │    │    ├── views
    │    │        │    │    │    │    │    │    ├── [scene_1_name]_0_0.bmp
    │    │        │    │    │    │    │    │    ├── [scene_1_name]_0_1.bmp
    │    │        │    │    │    │    │    │    ├── ...
    │    │        │    │    │    │    │    │    ├── [scene_1_name]_4_4.bmp
    │    │        │    │    │    │    ├── [scene_2_name]
    │    │        │    │    │    │    ├── ...
    │    │        │    │    │    ├── [dataset_2_name]
    │    │        │    │    │    ├── ...
    │    │        │    │    ├── VAL_epoch_01
    │    │        │    │    ├── ...
    │    │        ├── [other_model_name]
    │    │        ├── ...
    │    ├── SR_5x5_4x
    

Recources

We provide some original super-resolved images and useful resources to facilitate researchers to reproduce the above results.

Other Recources

Contact

Any question regarding this work can be addressed to [email protected].

Owner
Squidward
Squidward
3D AffordanceNet is a 3D point cloud benchmark consisting of 23k shapes from 23 semantic object categories, annotated with 56k affordance annotations and covering 18 visual affordance categories.

3D AffordanceNet This repository is the official experiment implementation of 3D AffordanceNet benchmark. 3D AffordanceNet is a 3D point cloud benchma

49 Dec 01, 2022
Talk covering the features of skorch

Skorch Talk Skorch - A Union of Scikit-learn and PyTorch Presentation The slides can be downloaded at: download link. Google Colab Part One - MNIST Pa

Thomas J. Fan 3 Oct 20, 2020
An implementation of chunked, compressed, N-dimensional arrays for Python.

Zarr Latest Release Package Status License Build Status Coverage Downloads Gitter Citation What is it? Zarr is a Python package providing an implement

Zarr Developers 1.1k Dec 30, 2022
Pytorch and Keras Implementations of Hyperspectral Image Classification -- Traditional to Deep Models: A Survey for Future Prospects.

The repository contains the implementations for Hyperspectral Image Classification -- Traditional to Deep Models: A Survey for Future Prospects. Model

Ankur Deria 115 Jan 06, 2023
PyTorch implementation of Weak-shot Fine-grained Classification via Similarity Transfer

SimTrans-Weak-Shot-Classification This repository contains the official PyTorch implementation of the following paper: Weak-shot Fine-grained Classifi

BCMI 60 Dec 02, 2022
A basic implementation of Layer-wise Relevance Propagation (LRP) in PyTorch.

Layer-wise Relevance Propagation (LRP) in PyTorch Basic unsupervised implementation of Layer-wise Relevance Propagation (Bach et al., Montavon et al.)

Kai Fabi 28 Dec 26, 2022
Anatomy of Matplotlib -- tutorial developed for the SciPy conference

Introduction This tutorial is a complete re-imagining of how one should teach users the matplotlib library. Hopefully, this tutorial may serve as insp

Matplotlib Developers 1.1k Dec 29, 2022
Facebook AI Image Similarity Challenge: Descriptor Track

Facebook AI Image Similarity Challenge: Descriptor Track This repository contains the code for our solution to the Facebook AI Image Similarity Challe

Sergio MP 17 Dec 14, 2022
A JAX-based research framework for writing differentiable numerical simulators with arbitrary discretizations

jaxdf - JAX-based Discretization Framework Overview | Example | Installation | Documentation ⚠️ This library is still in development. Breaking changes

UCL Biomedical Ultrasound Group 65 Dec 23, 2022
Denoising images with Fourier Ring Correlation loss

Denoising images with Fourier Ring Correlation loss The python code accompanies the working manuscript Image quality measurements and denoising using

2 Mar 12, 2022
Pytorch implementation of CVPR2021 paper "MUST-GAN: Multi-level Statistics Transfer for Self-driven Person Image Generation"

MUST-GAN Code | paper The Pytorch implementation of our CVPR2021 paper "MUST-GAN: Multi-level Statistics Transfer for Self-driven Person Image Generat

TianxiangMa 46 Dec 26, 2022
High-level library to help with training and evaluating neural networks in PyTorch flexibly and transparently.

TL;DR Ignite is a high-level library to help with training and evaluating neural networks in PyTorch flexibly and transparently. Click on the image to

4.2k Jan 01, 2023
Predicting lncRNA–protein interactions based on graph autoencoders and collaborative training

Predicting lncRNA–protein interactions based on graph autoencoders and collaborative training Code for our paper "Predicting lncRNA–protein interactio

zhanglabNKU 1 Nov 29, 2022
CS550 Machine Learning course project on CNN Detection.

CNN Detection (CS550 Machine Learning Project) Team Members (Tensor) : Yadava Kishore Chodipilli (11940310) Thashmitha BS (11941250) This is a work do

yaadava_kishore 2 Jan 30, 2022
MAg: a simple learning-based patient-level aggregation method for detecting microsatellite instability from whole-slide images

MAg Paper Abstract File structure Dataset prepare Data description How to use MAg? Why not try the MAg_lib! Trained models Experiment and results Some

Calvin Pang 3 Apr 08, 2022
CVPR 2021 Official Pytorch Code for UC2: Universal Cross-lingual Cross-modal Vision-and-Language Pre-training

UC2 UC2: Universal Cross-lingual Cross-modal Vision-and-Language Pre-training Mingyang Zhou, Luowei Zhou, Shuohang Wang, Yu Cheng, Linjie Li, Zhou Yu,

Mingyang Zhou 28 Dec 30, 2022
A pytorch implementation of Pytorch-Sketch-RNN

Pytorch-Sketch-RNN A pytorch implementation of https://arxiv.org/abs/1704.03477 In order to draw other things than cats, you will find more drawing da

Alexis David Jacq 172 Dec 12, 2022
Code for our ICCV 2021 Paper "OadTR: Online Action Detection with Transformers".

Code for our ICCV 2021 Paper "OadTR: Online Action Detection with Transformers".

66 Dec 15, 2022
An open source Jetson Nano baseboard and tools to design your own.

My Jetson Nano Baseboard This basic baseboard gives the user the foundation and the flexibility to design their own baseboard for the Jetson Nano. It

NVIDIA AI IOT 57 Dec 29, 2022