I try to reimplement the training part. But I encounter the following problem when I set the param 'use_checkpoint_attn' in self.residual_group2 as True. Could you provide solutions to solve it?

I am getting this error on my own test data (with task 008_VRT_videodenoising_DAVIS)

RuntimeError: Given groups=1, weight of size [96, 28, 1, 3, 3], expected input[1, 27, 32, 128, 128] to have 28 channels, but got 27 channels instead

Full stack: File "C:\Dev\VRT\models\network_vrt.py", line 1395, in forward x = self.conv_first(x.transpose(1, 2)) File "C:\tools\miniconda3\envs\pt\lib\site-packages\torch\nn\modules\module.py", line 1102, in _call_impl return forward_call(*input, **kwargs) File "C:\tools\miniconda3\envs\pt\lib\site-packages\torch\nn\modules\conv.py", line 590, in forward return self._conv_forward(input, self.weight, self.bias) File "C:\tools\miniconda3\envs\pt\lib\site-packages\torch\nn\modules\conv.py", line 585, in _conv_forward return F.conv3d( RuntimeError: Given groups=1, weight of size [96, 28, 1, 3, 3], expected input[1, 27, 32, 128, 128] to have 28 channels, but got 27 channels instead

VRT testing command

CUDA_VISIBLE_DEVICES=9 \
python main_test_vrt.py --task 002_VRT_videosr_bi_REDS_16frames \
                        --folder_lq /home/liao/cjj/dataset/test/LR \
                        --folder_gt /home/liao/cjj/dataset/test/GT \
                        --tile 10 128 128 \
                        --tile_overlap 2 20 20

SwinIR model: 001_classicalSR_DIV2K_s48w8_SwinIR-M_x4

Video for testing: https://cowtransfer.com/s/1739646a86874e

Result

SwinIR |   |   |   |   |   -- | -- | -- | -- | -- | --   | 1 | 2 | 3 | 4 | 平均 PSNR | 26.9603 | 31.9831 | 33.0922 | 33.2781 | 31.32843 SSIM | 0.7353 | 0.9022 | 0.8842 | 0.9233 | 0.86125 VRT |   |   |   |   |     | 1 | 2 | 3 | 4 | 平均 PSNR | 26.7961 | 31.7153 | 30.7655 | 34.3461 | 30.90575 SSIM | 0.7272 | 0.8931 | 0.8724 | 0.9385 | 0.8578

Hola! Thanks for the great work with VRT. I wanted to know if you have any tips and recommendations to how we can run your evaluation code against our own higher resolution frames. It seems from my tests that anything above 180p just runs OOM in a K80 (12G) and a T4 (16G) regardless of the tile size that I use for all models (REDS, Vimeo, etc.). Do you have any advice? Thanks!

I ran into the same error as #14 , and verified that self.nonblind_denoising was set to True here, but still receive the error:

line 585, in _conv_forward
    return F.conv3d(
RuntimeError: Given groups=1, weight of size [96, 28, 1, 3, 3], expected input[1, 27, 40, 128, 128] to have 28 channels, but got 27 channels instead

This is using the dataset VRT/testsets/REDS4/sharp_bicubic via the call python main_test_vrt.py --task 008_VRT_videodenoising_DAVIS --folder_lq testsets/REDS4/sharp_bicubic --tile 40 128 128 --tile_overlap 2 20 20. I ultimately want to run this on my own folder of PNGs from a video.

If I only have 2/4 3090s and want to train a model for ×4 VSR, how can I set training parameters effectively? That is no OOM, no large performance drop, mild training time.

For example, there are two parameters of using checkpoint to save Cuda memory, use_checkpoint_attn and use_checkpoint_ffn, which one is the most influence one for training time/memory consumption?

Looking forward to your reply, thank you.

Hi, I've been trying to use this code in combination with the github://cszn/KAIR for training a VRT model using my data and a custom dataloader I wrote for my data. Unfortunately, I'm running into an error in the testing phase of the get_flow_4frames because the shape of the forward_flows[0] is: torch.Size([1, 0, 2, 64, 64])

The X input into forward is: torch.Size([1, 1, 3, 64, 64]) The X input into get_flows is: torch.Size([1, 1, 3, 64, 64]) The X input into get_flow_2frames: torch.Size([1, 1, 3, 64, 64]) The forward_flows[0] is as previously specified: torch.Size([1, 0, 2, 64, 64])

def get_flow_4frames(self, flows_forward, flows_backward):
        '''Get flow between t and t+2 from (t,t+1) and (t+1,t+2).'''

        # backward
        d = flows_forward[0].shape[1]
        flows_backward2 = []
        for flows in flows_backward:
            flow_list = []
            for i in range(d - 1, 0, -1):
                flow_n1 = flows[:, i - 1, :, :, :]  # flow from i+1 to i
                flow_n2 = flows[:, i, :, :, :]  # flow from i+2 to i+1
                flow_list.insert(0, flow_n1 + flow_warp(flow_n2, flow_n1.permute(0, 2, 3, 1)))  # flow from i+2 to i
            if len(flow_list) != 0:
                flows_backward2.append(torch.stack(flow_list, 1))

The training is working without any issues.

Is this the anticipated behavior within the code or is there something regarding the test settings that I'm missing?

Hi, congrats on this cool work! I'm trying to train your model but I only have 2 A100 GPUs so the memory is limited, I wonder how much space do you need to train models like "003_VRT_videosr_bi_Vimeo_7frames.pth" and "006_VRT_videodeblurring_GoPro.pth"?

According to the paper "The runtime is 2.2s per frame on 1280×720 blurred videos". What gpu are you guys used to measure runtime?
Also i have question about model size, Did you guys try smaller model sizes (popular in modern transformers something like VRT-S VRT-L with different parameter size etc.) or architecture is limited and don't coverage with custom sizes?

And ofc. congrats on cool paper 📦

Hello, thank you for the code. I meet an error when I train with 005_train_vrt_videodeblurring_dvd.json

Fix keys: ['spynet', 'deform'] for the first 20000 iters. Fix keys: ['spynet', 'deform'] for the first 20000 iters. 22-09-01 02:31:11.512 : <epoch: 0, iter: 400, lr:4.000e-04> G_loss: 7.544e-02 22-09-01 02:48:36.264 : <epoch: 0, iter: 600, lr:4.000e-04> G_loss: 1.637e-02 22-09-01 03:06:01.631 : <epoch: 0, iter: 800, lr:4.000e-04> G_loss: 7.941e-02 WARNING:torch.distributed.elastic.agent.server.api:Received 1 death signal, shutting down workers WARNING:torch.distributed.elastic.multiprocessing.api:Sending process 2704351 closing signal SIGHUP WARNING:torch.distributed.elastic.multiprocessing.api:Sending process 2704352 closing signal SIGHUP Traceback (most recent call last): File "/home/ET/huiyuxiang/miniconda3/envs/deblur/lib/python3.7/runpy.py", line 193, in _run_module_as_main "__main__", mod_spec) File "/home/ET/huiyuxiang/miniconda3/envs/deblur/lib/python3.7/runpy.py", line 85, in _run_code exec(code, run_globals) File "/home/ET/huiyuxiang/miniconda3/envs/deblur/lib/python3.7/site-packages/torch/distributed/launch.py", line 193, in <module> main() File "/home/ET/huiyuxiang/miniconda3/envs/deblur/lib/python3.7/site-packages/torch/distributed/launch.py", line 189, in main launch(args) File "/home/ET/huiyuxiang/miniconda3/envs/deblur/lib/python3.7/site-packages/torch/distributed/launch.py", line 174, in launch run(args) File "/home/ET/huiyuxiang/miniconda3/envs/deblur/lib/python3.7/site-packages/torch/distributed/run.py", line 755, in run )(*cmd_args) File "/home/ET/huiyuxiang/miniconda3/envs/deblur/lib/python3.7/site-packages/torch/distributed/launcher/api.py", line 131, in __call__ return launch_agent(self._config, self._entrypoint, list(args)) File "/home/ET/huiyuxiang/miniconda3/envs/deblur/lib/python3.7/site-packages/torch/distributed/launcher/api.py", line 236, in launch_agent result = agent.run() File "/home/ET/huiyuxiang/miniconda3/envs/deblur/lib/python3.7/site-packages/torch/distributed/elastic/metrics/api.py", line 125, in wrapper result = f(*args, **kwargs) File "/home/ET/huiyuxiang/miniconda3/envs/deblur/lib/python3.7/site-packages/torch/distributed/elastic/agent/server/api.py", line 709, in run result = self._invoke_run(role) File "/home/ET/huiyuxiang/miniconda3/envs/deblur/lib/python3.7/site-packages/torch/distributed/elastic/agent/server/api.py", line 850, in _invoke_run time.sleep(monitor_interval) File "/home/ET/huiyuxiang/miniconda3/envs/deblur/lib/python3.7/site-packages/torch/distributed/elastic/multiprocessing/api.py", line 60, in _terminate_process_handler raise SignalException(f"Process {os.getpid()} got signal: {sigval}", sigval=sigval) torch.distributed.elastic.multiprocessing.api.SignalException: Process 2704341 got signal: 1

and I use the python=3.7.13, pytorch=1.12.1

I was wondering if the authors have any suggestions for finetuning the VRT model to do a 2x upscale instead of a 4x upscale. I removed some layers from the Upsample module to support 2x upscale, however the forward/backward pass is consuming too much VRAM. Which layers do you suggest to remove from the model to reduce the model complexity and also achieve good results for a 2x upscale?

Currently, I have tried 2x upscale training with 1 GPU, batch size =1, low quality frames crop size = 64x64, and high quality frames crop size = 128x128. The maximum VRAM usage in the forward pass/backward pass is 23GB.

Great research, thank you! I want to test the denoising effect on my own dataset which has only noisy input and no corresponding GT, how can I do this? Thanks!

After 4 successfull cells,

FileNotFoundError                         Traceback (most recent call last)
[<ipython-input-5-b2c3ee9af109>](https://localhost:8080/#) in <module>
      4   os.remove(zip_filename)
      5 os.system(f"zip -r -j {zip_filename} results/*")
----> 6 files.download(zip_filename)

[/usr/local/lib/python3.7/dist-packages/google/colab/files.py](https://localhost:8080/#) in download(filename)
    207   if not _os.path.exists(filename):
    208     msg = 'Cannot find file: {}'.format(filename)
--> 209     raise FileNotFoundError(msg)  # pylint: disable=undefined-variable
    210 
    211   comm_manager = _IPython.get_ipython().kernel.comm_manager

FileNotFoundError: Cannot find file: VRT_result.zip

hi, i do to training VRT model currently. but reached to memory lack problem.

i used to NVIDIA RTX 2080 Ti which device memory amount is 11GB for training. so i was adjusted batch-size from 8 to 4 and torch.utils.checkpoint used. (use_checkpoint_attn: true, use_checkpoint_ffn: true, gt_size=256)

but arise to memory lack (cannot allocate memory).

is it possible training using RTX 2080 Ti?

Hello,

Thank you for your awesome code!

I am hoping you might open-source the log files you have from training. Maybe the training and validation loss as a function of epoch (and/or batch) with an estimate of the runtime?

I am trying to deblur a 150 frames video using a machine having two NVIDIA RTX A5000 GPUs using the the GoPro delur model and I reduced the tile value. But this operation is taking forever. How to solve this ? Is NVIDIA RTX A5000 enough to make ineferece ?