[Paper] [arXiv]

IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2021

An expanded version of the conference paper is released at arXiv, which reveals technical details regarding model compression based on layer-wise sparsity information obtained via optimization. That part of code is available at the sub-directory extended_version.

We propose a Compression-Driven network design for Frame Interpolation (CDFI), that leverages model compression to significantly reduce the model size (allows a better understanding of the current architecture) while making room for further improvements and achieving superior performance in the end. Concretely, we first compress AdaCoF and show that a 10X compressed AdaCoF performs similarly as its original counterpart; then we improve upon this compressed model with simple modifications. Note that typically it is prohibitive to implement the same improvements on the original heavy model.

• We achieve a significant performance gain with only a quarter in size compared with the original AdaCoF

  	Vimeo-90K	Middlebury	UCF101-DVF	#Params
  	PSNR, SSIM, LPIPS	PSNR, SSIM, LPIPS	PSNR, SSIM, LPIPS
  AdaCoF	34.35, 0.956, 0.019	35.72, 0.959, 0.019	35.16, 0.950, 0.019	21.84M
  Compressed AdaCoF	34.10, 0.954, 0.020	35.43, 0.957, 0.018	35.10, 0.950, 0.019	2.45M
  AdaCoF+	34.56, 0.959, 0.018	36.09, 0.962, 0.017	35.16, 0.950, 0.019	22.93M
  Compressed AdaCoF+	34.44, 0.958, 0.019	35.73, 0.960, 0.018	35.13, 0.950, 0.019	2.56M
  Our Final Model	35.17, 0.964, 0.010	37.14, 0.966, 0.007	35.21, 0.950, 0.015	4.98M

• Our final model also performs favorably against other state-of-the-arts (details refer to our paper)

  

• The proposed framework is generic and can be easily transferred to other DNN-based frame interpolation method

The above GIF is a demo of using our method to generate slow motion video, which increases the FPS from 5 to 160. We also provide a long video demonstration here (redirect to YouTube).

• CUDA 11.0

• python 3.8.3

• torch 1.8.1+cu111

• torchvision 0.9.1+cu111

• cupy 7.7.0

• scipy 1.7.3

• numpy 1.21.5

• Pillow 7.2.0

• scikit-image 0.19.2

• lpips

conda create -n cdfi python==3.8.3
conda activate cdfi
conda install -c conda-forge cupy==7.7.0
pip install torch==1.8.1+cu111 torchvision -f https://download.pytorch.org/whl/lts/1.8/torch_lts.html
pip install opencv-python lpips
conda install matplotlib scikit-image

Download repository:

$ git clone https://github.com/tding1/CDFI.git
$ cd CDFI/

For user convenience, we already provide the Middlebury and UCF101-DVF test datasets in our repository, which can be found under directory test_data/.

We use the built-in functions in skimage.metrics to compute the PSNR and SSIM, for which the higher the better. We also use LPIPS to measure perceptual similarity, for which the smaller the better.

Note: We are using squeeze net in calculating LPIPS, while other work (Softsplat, EDSC, etc) might use different methods in their original implementations, e.g., alex net. Although we manually test AdaCoF, EDSC, CAIN under the same setting and demonstrate the results in the paper, there may be discrepancies from their original results, see also the discussion here.

$ python mytest.py --gpu_id 0

By default, it will load our pre-trained model checkpoints/CDFI_adacof.pth. It will print the quantitative results on both Middlebury and UCF101-DVF, and the interpolated images will be saved under test_output/cdfi_adacof/.

$ python mytest.py --gpu_id 0 --model compressed_adacof --kernel_size 5 --dilation 1

By default, it will load the compressed AdaCoF model checkpoints/compressed_adacof_F_5_D_1.pth. It will print the quantitative results on both Middlebury and UCF101-DVF, and the interpolated images will be saved under test_output/compressed_adacof_F_5_D_1/.

$ python mytest.py --gpu_id 0 --model compressed_adacof --kernel_size 11 --dilation 2

By default, it will load the compressed AdaCoF+ model checkpoints/compressed_adacof_F_11_D_2.pth. It will print the quantitative results on both Middlebury and UCF101-DVF, and the interpolated images will be saved under test_output/compressed_adacof_F_11_D_2/.

$ python interpolate_twoframe.py --gpu_id 0 --first_frame imgs/0.png --second_frame imgs/1.png --output_frame ./output.png

By default, it will load our pre-trained model checkpoints/CDFI_adacof.pth, and generate the intermediate frame output.png given two consecutive frames in a sequence.

$ python interpolate_video.py --gpu_id 0 --input_video imgs/img_seq/ --output_video ./interpolated_video

This script will interpolate a video sequence using our pre-trained model checkpoints/CDFI_adacof.pth, thus increasing the FPS by a factor of 2. You may want to repeat the procedure on the interpolated video if a higher FPS is desired.

We use the Vimeo-90K triplet dataset for video frame interpolation task, which is relatively large (>32 GB).

$ wget http://data.csail.mit.edu/tofu/dataset/vimeo_triplet.zip
$ unzip vimeo_triplet.zip
$ rm vimeo_triplet.zip

$ python train.py --gpu_id 0 --data_dir path/to/vimeo_triplet/ --batch_size 8

It will generate an unique ID for each training, and all the intermediate results/records will be saved under model_weights/<training id>/ (or you can specify your experiment ID using --uid). For a GPU device with memory around 10GB, the --batch_size can take a value as large as 3, otherwise CUDA may be out of memory. There are many other training options, e.g., --lr, --epochs, --loss and so on, can be found in train.py.

One nice thing about CDFI is that the framework can be easily applied to other (heavy) DNN models and potentially boost their performance. The key to CDFI is the optimization-based compression that compresses a model via fine-grained pruning. In particular, we use the efficient and easy-to-use sparsity-inducing optimizer OBPROXSG (see also paper), and summarize the compression procedure for any other model in the following. For details, we recommend checking our long version of the paper at arXiv and the additional code at extended_version.

• Copy the OBPROXSG optimizer, which is already implemented as torch.optim.optimizer, to your working directory
• Starting from a pre-trained model, finetune its weights by using the OBPROXSG optimizer, like using any standard PyTorch built-in optimizer such as SGD or Adam
  • It is not necessarily to use the full dataset for this finetuning process
• The parameters for the OBPROXSG optimizer
  • lr: learning rate
  • lambda_: coefficient of the L1 regularization term
  • epochSize: number of batches in a epoch
  • Np: number of proximal steps, which is set to be 2 for pruning AdaCoF
  • No: number of orthant steps (key step to promote sparsity), for which we recommend using the default setting
  • eps: threshold for trimming zeros, which is set to be 0.0001 for pruning AdaCoF
• After the optimization is done (either by reaching a maximum number of epochs or achieving a high sparsity), use the layer density as the compression ratio for that layer, as described in the paper
• As an example, compare the architectures in models/adacof.py and model/compressed_adacof.py for compressing AdaCoF with the above procedure

Now it's ready to make further improvements/modifications on the compressed model, based on the understanding of its flaws/drawbacks.

@article{ding2021cdfi,
  title={CDFI: Compression-Driven Network Design for Frame Interpolation},
  author={Ding, Tianyu and Liang, Luming and Zhu, Zhihui and Zharkov, Ilya},
  journal={arXiv preprint arXiv:2103.10559},
  year={2021}
}

or

@inproceedings{ding2021cdfi,
  title={CDFI: Compression-Driven Network Design for Frame Interpolation},
  author={Ding, Tianyu and Liang, Luming and Zhu, Zhihui and Zharkov, Ilya},
  booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  pages={8001--8011},
  year={2021}
}

The code is largely based on HyeongminLEE/AdaCoF-pytorch and baowenbo/DAIN.