Practical Blind Denoising via Swin-Conv-UNet and Data Synthesis

Last update: Jun 08, 2022

Related tags

Overview

Practical Blind Denoising via Swin-Conv-UNet and Data Synthesis

The following results are obtained by our SCUNet with purely synthetic training data! We did not use the paired noisy/clean data by DND and SIDD during training!

Swin-Conv-UNet (SCUNet) denoising network

The architecture of the proposed Swin-Conv-UNet (SCUNet) denoising network. SCUNet exploits the swin-conv (SC) block as the main building block of a UNet backbone. In each SC block, the input is first passed through a 1×1 convolution, and subsequently is split evenly into two feature map groups, each of which is then fed into a swin transformer (SwinT) block and residual 3×3 convolutional (RConv) block, respectively; after that, the outputs of SwinT block and RConv block are concatenated and then passed through a 1×1 convolution to produce the residual of the input. “SConv” and “TConv” denote 2×2 strided convolution with stride 2 and 2×2 transposed convolution with stride 2, respectively.

New data synthesis pipeline for real image denoising

Schematic illustration of the proposed paired training patches synthesis pipeline. For a high quality image, a randomly shuffled degradation sequence is performed to produce a noisy image. Meanwhile, the resizing and reverse-forward tone mapping are performed to produce a corresponding clean image. A paired noisy/clean training patches are then cropped for training deep blind denoising model. Note that, since Poisson noise is signal-dependent, the dashed arrow for “Poisson” means the clean image is used to generate the Poisson noise. To tackle with the color shift issue, the dashed arrow for “Camera Sensor” means the reverse-forward tone mapping is performed on the clean image.

Synthesized noisy/clean patch pairs via our proposed training data synthesis pipeline. The size of the high quality image patch is 544×544. The size of the noisy/clean patches is 128×128.

Web Demo

Try Replicate web demo for SCUNet models here

Codes

Download SCUNet models

python main_download_pretrained_models.py --models "SCUNet" --model_dir "model_zoo"

Gaussian denoising

grayscale images

python main_test_scunet_gray_gaussian.py --model_name scunet_gray_25 --noise_level_img 25 --testset_name set12

color images

python main_test_scunet_color_gaussian.py --model_name scunet_color_25 --noise_level_img 25 --testset_name bsd68

Blind real image denoising

python main_test_scunet_real_application.py --model_name scunet_color_real_psnr --testset_name real3

Results on Gaussian denoising

Results on real image denoising

@article{zhang2022practical,
title={Practical Blind Denoising via Swin-Conv-UNet and Data Synthesis},
author={Zhang, Kai and Li, Yawei and Liang, Jingyun and Cao, Jiezhang and Zhang, Yulun and Tang, Hao and Timofte, Radu and Van Gool, Luc},
journal={arXiv preprint},
year={2022}
}

Issues

Will there be more models?

Very impressed with the results! However at times even the lowest noise reduction "15" model can remove too much detail. Will there be a "5" or "10" noise level model in the future?

Thank you for all the hard work!

opened by AIisCool 2
Add Web Demo & Docker environment

Hey @cszn! 👋

This pull request makes it possible to run your model inside a Docker environment, which makes it easier for other people to run it. We're using an open source tool called Cog to make this process easier.

This also means we can make a web page where other people can try out your model! View it here: https://replicate.com/cszn/scunet. The docker file can be found under the tab ‘run model with docker’.

We have added some examples, but do claim the page so you can own the page, customise the Example gallery as you like, push any future update to the web demo, and we'll feature it on our website and tweet about it too. Moreover, when the page is claimed, it will be automatically linked to the arXiv website as well (under “Demos”)! You can find the 'Claim this model' button on the top of the page ~

In case you're wondering who I am, I'm from Replicate, where we're trying to make machine learning reproducible. We got frustrated that we couldn't run all the really interesting ML work being done. So, we're going round implementing models we like. 😊

opened by chenxwh 1
Fantastic project, great results, but...
File "C:\Users\Miki\SCUNet\models\network_scunet.py", line 88, in forward if self.type!='W': output = torch.roll(output, shifts=(self.window_size//2, self.window_size//2), dims=(1,2)) RuntimeError: CUDA out of memory. Tried to allocate 486.00 MiB (GPU 0; 12.00 GiB total capacity; 10.76 GiB already allocated; 0 bytes free; 11.15 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation. See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF

@cszn Please, add minor changes to code, to support --tile.... in before opened issue, I asked for same question, followed your instructions, uncommented line # number 99th but same problem appears again.... No way to denoise photos larger than 3 Mpx...due cuda oom errors. It is pain doing in graphic editor manualy cropping & after denoise joinning big photo gallery I kindly ask for help... Cause this project has realy great potencial. Onothers like swinir, restormer etc... doesnt do joob like your, in terms on real denoising.... tnx very much in advance
opened by zelenooki87 1
Control sigma for scunet_color_real_psnr?

Results look very good, better than the "scunet_color" at times. But it would be nice to control the noise reduction like the grayscale and color ones.

Thank you.

opened by AIisCool 1

Owner

Kai Zhang

Image Restoration; Inverse Problems

GitHub Repository

SymmetryNet: Learning to Predict Reflectional and Rotational Symmetries of 3D Shapes from Single-View RGB-D Images

SymmetryNet SymmetryNet: Learning to Predict Reflectional and Rotational Symmetries of 3D Shapes from Single-View RGB-D Images ACM Transactions on Gra

23 Nov 21, 2021

Introduction to CPM

CPM CPM is an open-source program on large-scale pre-trained models, which is conducted by Beijing Academy of Artificial Intelligence and Tsinghua Uni

100 Jan 28, 2022

🔮 A refreshing functional take on deep learning, compatible with your favorite libraries

Thinc: A refreshing functional take on deep learning, compatible with your favorite libraries From the makers of spaCy, Prodigy and FastAPI Thinc is a

2.4k Jan 26, 2022

一套完整的微博舆情分析流程代码，包括微博爬虫、LDA主题分析和情感分析。

已经将项目的关键文件上传，包含微博爬虫、LDA主题分析和情感分析三个部分。 1.微博爬虫实现微博评论爬取和微博用户信息爬取，一天大概十万条。 2.LDA主题分析实现文档主题抽取，包括数据清洗及分词、主题数的确定（主题一致性和困惑度）和最优主题模型的选择（暴力搜索）。 3.情感分析实现评论文本的

24 Jan 18, 2022

thundernet ncnn

MMDetection_Lite 基于mmdetection 实现一些轻量级检测模型，安装方式和mmdeteciton相同 voc0712 voc 0712训练 voc2007测试 coco预训练 thundernet_voc_shufflenetv2_1.5 input shape mAP 320

35 Jan 02, 2022

Syllabic Quantity Patterns as Rhythmic Features for Latin Authorship Attribution

Syllabic Quantity Patterns as Rhythmic Features for Latin Authorship Attribution Abstract Within the Latin (and ancient Greek) production, it is well

2 Oct 22, 2021

Supplementary code for the paper "Meta-Solver for Neural Ordinary Differential Equations" https://arxiv.org/abs/2103.08561

Meta-Solver for Neural Ordinary Differential Equations Towards robust neural ODEs using parametrized solvers. Main idea Each Runge-Kutta (RK) solver w

25 Aug 11, 2021

MetaAvatar: Learning Animatable Clothed Human Models from Few Depth Images

MetaAvatar: Learning Animatable Clothed Human Models from Few Depth Images This repository contains the implementation of our paper MetaAvatar: Learni

46 Jan 21, 2022

Combining Latent Space and Structured Kernels for Bayesian Optimization over Combinatorial Spaces

This repository contains source code for the paper Combining Latent Space and Structured Kernels for Bayesian Optimization over Combinatorial Spaces a

4 Jan 03, 2022

Official PyTorch implementation of Retrieve in Style: Unsupervised Facial Feature Transfer and Retrieval.

Retrieve in Style: Unsupervised Facial Feature Transfer and Retrieval PyTorch This is the PyTorch implementation of Retrieve in Style: Unsupervised Fa

52 Jan 02, 2022

The repo for the paper "I3CL: Intra- and Inter-Instance Collaborative Learning for Arbitrary-shaped Scene Text Detection".

8 May 14, 2022

Practical Blind Denoising via Swin-Conv-UNet and Data Synthesis

Related tags

Overview

Practical Blind Denoising via Swin-Conv-UNet and Data Synthesis

Swin-Conv-UNet (SCUNet) denoising network

New data synthesis pipeline for real image denoising

Web Demo

Codes

Results on Gaussian denoising

Results on real image denoising

Issues

Will there be more models?

Add Web Demo & Docker environment

Fantastic project, great results, but...

Control sigma for scunet_color_real_psnr?

Owner

Kai Zhang

SymmetryNet: Learning to Predict Reflectional and Rotational Symmetries of 3D Shapes from Single-View RGB-D Images

Introduction to CPM

🔮 A refreshing functional take on deep learning, compatible with your favorite libraries

一套完整的微博舆情分析流程代码，包括微博爬虫、LDA主题分析和情感分析。

thundernet ncnn

Syllabic Quantity Patterns as Rhythmic Features for Latin Authorship Attribution

Supplementary code for the paper "Meta-Solver for Neural Ordinary Differential Equations" https://arxiv.org/abs/2103.08561

MetaAvatar: Learning Animatable Clothed Human Models from Few Depth Images

Combining Latent Space and Structured Kernels for Bayesian Optimization over Combinatorial Spaces

Official PyTorch implementation of Retrieve in Style: Unsupervised Facial Feature Transfer and Retrieval.

The repo for the paper "I3CL: Intra- and Inter-Instance Collaborative Learning for Arbitrary-shaped Scene Text Detection".

Air Pollution Prediction System using Linear Regression and ANN

WTTE-RNN a framework for churn and time to event prediction

Coursera - Quiz & Assignment of Coursera

Structure Information is the Key: Self-Attention RoI Feature Extractor in 3D Object Detection

A highly efficient and modular implementation of Gaussian Processes in PyTorch

BESS: Balanced Evolutionary Semi-Stacking for Disease Detection via Partially Labeled Imbalanced Tongue Data

Active window border replacement for window managers.

Continuous Time LiDAR odometry

Classify the disease status of a plant given an image of a passion fruit