Image Super-Resolution by Neural Texture Transfer

Related tags

Deep LearningSRNTT
Overview

SRNTT: Image Super-Resolution by Neural Texture Transfer

Tensorflow implementation of the paper Image Super-Resolution by Neural Texture Transfer accepted in CVPR 2019. This is a simplified version, where the reference images are used without augmentation, e.g., rotation and scaling.

Project Page

Pytorch Implementation

Contents

Pre-requisites

  • Python 3.6
  • TensorFlow 1.13.1
  • requests 2.21.0
  • pillow 5.4.1
  • matplotlib 3.0.2

Tested on MacOS (Mojave).

Dataset

This repo only provides a small training set of ten input-reference pairs for demo purpose. The input images and reference images are stored in data/train/CUFED/input and data/train/CUFED/ref, respectively. Corresponding input and refernece images are with the same file name. To speed up the training process, patch matching and swapping are performed offline, and the swapped feature maps will be saved to data/train/CUFED/map_321 (see offline_patchMatch_textureSwap.py for more details). If you want to train your own model, please prepare your own training set or download either of the following demo training sets:

11,485 input-reference pairs (size 320x320) extracted from DIV2K.

Each pair is extracted from the same image without overlap but considering scaling and rotation.

$ python download_dataset.py --dataset_name DIV2K
11,871 input-reference pairs (size 160x160) extracted from CUFED.

Each pair is extracted from the similar images, including five degrees of similarity.

$ python download_dataset.py --dataset_name CUFED

This repo includes one grounp of samples from the CUFED5 dataset, where each input image corresponds to five reference images (different from the paper) with different degrees of similarity to the input image. Please download the full dataset by

$ python download_dataset.py --dataset_name CUFED5

Easy Testing

$ sh test.sh

The results will be save to the folder demo_testing_srntt, including the following 6 images:

  • [1/6] HR.png, the original image.

    Original image

  • [2/6] LR.png, the low-resolution (LR) image, downscaling factor 4x.

    LR image

  • [3/6] Bicubic.png, the upscaled image by bicubic interpolation, upscaling factor 4x.

    Bicubic image

  • [4/6] Ref_XX.png, the reference images, indexed by XX.

    Reference image

  • [5/6] Upscale.png, the upscaled image by a pre-trained SR network, upscaling factor 4x.

    Upscaled image

  • [6/6] SRNTT.png, the SR result by SRNTT, upscaling factor 4x.

    Upscaled image

Custom Testing

$ python main.py 
    --is_train              False 
    --input_dir             path/to/input/image/file
    --ref_dir               path/to/ref/image/file
    --result_dir            path/to/result/folder
    --ref_scale             default 1, expected_ref_scale divided by original_ref_scale
    --is_original_image     default True, whether input is original 
    --use_init_model_only   default False, whether use init model, trained with reconstruction loss only
    --use_weight_map        defualt False, whether use weighted model, trained with the weight map.
    --save_dir              path/to/a/specified/model if it exists, otherwise ignor this parameter

Please note that this repo provides two types of pre-trained SRNTT models in SRNTT/models/SRNTT:

  • srntt.npz is trained by all losses, i.e., reconstruction loss, perceptual loss, texture loss, and adversarial loss.
  • srntt_init.npz is trained by only the reconstruction loss, corresponding to SRNTT-l2 in the paper.

To switch between the demo models, please set --use_init_model_only to decide whether use srntt_init.npz.

Easy Training

$ sh train.sh

The CUFED training set will be downloaded automatically. To speed up the training process, patch matching and swapping are conducted to get the swapped feature maps in an offline manner. The models will be saved to demo_training_srntt/model, and intermediate samples will be saved to demo_training_srntt/sample. Parameter settings are save to demo_training_srntt/arguments.txt.

Custom Training

Please first prepare the input and reference images which are squared patches in the same size. In addition, input and reference images should be stored in separated folders, and the correspoinding input and reference images are with the same file name. Please refer to the data/train/CUFED folder for examples. Then, use offline_patchMatch_textureSwap.py to generate the feature maps in ahead.

$ python main.py
    --is_train True
    --save_dir folder/to/save/models
    --input_dir path/to/input/image/folder
    --ref_dir path/to/ref/image/folder
    --map_dir path/to/feature_map/folder
    --batch_size default 9
    --num_epochs default 100
    --input_size default 40, the size of LR patch, i.e., 1/4 of the HR image, set to 80 for the DIV2K dataset
    --use_weight_map defualt False, whether use the weight map that reduces negative effect 
                     from the reference image but may also decrease the sharpness.  

Please refer to main.py for more parameter settings for training.

Test on the custom training model

$ python main.py 
    --is_train              False 
    --input_dir             path/to/input/image/file
    --ref_dir               path/to/ref/image/file
    --result_dir            path/to/result/folder
    --ref_scale             default 1, expected_ref_scale divided by original_ref_scale
    --is_original_image     default True, whether input is original 
    --save_dir              the same as save_dir in training

Acknowledgement

Thanks to Tensorlayer for facilitating the implementation of this demo code. We have include the Tensorlayer 1.5.0 in SRNTT/tensorlayer.

Contact

Zhifei Zhang

Owner
Zhifei Zhang
Zhifei Zhang
Code for our CVPR 2022 Paper "GEN-VLKT: Simplify Association and Enhance Interaction Understanding for HOI Detection"

GEN-VLKT Code for our CVPR 2022 paper "GEN-VLKT: Simplify Association and Enhance Interaction Understanding for HOI Detection". Contributed by Yue Lia

Yue Liao 47 Dec 04, 2022
Viperdb - A tiny log-structured key-value database written in pure Python

ViperDB 🐍 ViperDB is a lightweight embedded key-value store written in pure Pyt

17 Oct 17, 2022
SuMa++: Efficient LiDAR-based Semantic SLAM (Chen et al IROS 2019)

SuMa++: Efficient LiDAR-based Semantic SLAM This repository contains the implementation of SuMa++, which generates semantic maps only using three-dime

Photogrammetry & Robotics Bonn 701 Dec 30, 2022
Providing the solutions for high-frequency trading (HFT) strategies using data science approaches (Machine Learning) on Full Orderbook Tick Data.

Modeling High-Frequency Limit Order Book Dynamics Using Machine Learning Framework to capture the dynamics of high-frequency limit order books. Overvi

Chang-Shu Chung 1.3k Jan 07, 2023
PyTorch implementation of Lip to Speech Synthesis with Visual Context Attentional GAN (NeurIPS2021)

Lip to Speech Synthesis with Visual Context Attentional GAN This repository contains the PyTorch implementation of the following paper: Lip to Speech

6 Nov 02, 2022
Tool for working with Y-chromosome data from YFull and FTDNA

ycomp ycomp is a tool for working with Y-chromosome data from YFull and FTDNA. Run ycomp -h for information on how to use the program. Installation Th

Alexander Regueiro 2 Jun 18, 2022
Uses OpenCV and Python Code to detect a face on the screen

Simple-Face-Detection This code uses OpenCV and Python Code to detect a face on the screen. This serves as an example program. Important prerequisites

Denis Woolley (CreepyD) 1 Feb 12, 2022
Implementation of U-Net and SegNet for building segmentation

Specialized project Created by Katrine Nguyen and Martin Wangen-Eriksen as a part of our specialized project at Norwegian University of Science and Te

Martin.w-e 3 Dec 07, 2022
Julia and Matlab codes to simulated all problems in El-Hachem, McCue and Simpson (2021)

Substrate_Mediated_Invasion Julia and Matlab codes to simulated all problems in El-Hachem, McCue and Simpson (2021) 2DSolver.jl reproduces the simulat

Matthew Simpson 0 Nov 09, 2021
The Illinois repository for Climatehack (https://climatehack.ai/). We won 1st place!

Climatehack This is the repository for Illinois's Climatehack Team. We earned first place on the leaderboard with a final score of 0.87992. An overvie

Jatin Mathur 20 Jun 09, 2022
ExCon: Explanation-driven Supervised Contrastive Learning

ExCon: Explanation-driven Supervised Contrastive Learning Link to the paper: https://arxiv.org/pdf/2111.14271.pdf Contributors of this repo: Zhibo Zha

Zhibo (Darren) Zhang 18 Nov 01, 2022
RLBot Python bindings for the Rust crate rl_ball_sym

RLBot Python bindings for rl_ball_sym 0.6 Prerequisites: Rust & Cargo Build Tools for Visual Studio RLBot - Verify that the file %localappdata%\RLBotG

Eric Veilleux 2 Nov 25, 2022
Implementation of Barlow Twins paper

barlowtwins PyTorch Implementation of Barlow Twins paper: Barlow Twins: Self-Supervised Learning via Redundancy Reduction This is currently a work in

IgorSusmelj 86 Dec 20, 2022
Volumetric parameterization of the placenta to a flattened template

placenta-flattening A MATLAB algorithm for volumetric mesh parameterization. Developed for mapping a placenta segmentation derived from an MRI image t

Mazdak Abulnaga 12 Mar 14, 2022
A resource for learning about deep learning techniques from regression to LSTM and Reinforcement Learning using financial data and the fitness functions of algorithmic trading

A tour through tensorflow with financial data I present several models ranging in complexity from simple regression to LSTM and policy networks. The s

195 Dec 07, 2022
Bib-parser - Convenient script to parse .bib files with the ACM Digital Library like metadata

Bib Parser Convenient script to parse .bib files with the ACM Digital Library li

Mehtab Iqbal (Shahan) 1 Jan 26, 2022
EMNLP 2021 - Frustratingly Simple Pretraining Alternatives to Masked Language Modeling

Frustratingly Simple Pretraining Alternatives to Masked Language Modeling This is the official implementation for "Frustratingly Simple Pretraining Al

Atsuki Yamaguchi 31 Nov 18, 2022
Learning Representations that Support Robust Transfer of Predictors

Transfer Risk Minimization (TRM) Code for Learning Representations that Support Robust Transfer of Predictors Prepare the Datasets Preprocess the Scen

Yilun Xu 15 Dec 07, 2022
Monify: an Expense tracker Program implemented in a Graphical User Interface that allows users to keep track of their expenses

💳 MONIFY (EXPENSE TRACKER PRO) 💳 Description Monify is an Expense tracker Program implemented in a Graphical User Interface allows users to add inco

Moyosore Weke 1 Dec 14, 2021
Translate darknet to tensorflow. Load trained weights, retrain/fine-tune using tensorflow, export constant graph def to mobile devices

Intro Real-time object detection and classification. Paper: version 1, version 2. Read more about YOLO (in darknet) and download weight files here. In

Trieu 6.1k Jan 04, 2023