Using VapourSynth with super resolution models and speeding them up with TensorRT.

Overview

VSGAN-tensorrt-docker

Using image super resolution models with vapoursynth and speeding them up with TensorRT. Using NVIDIA/Torch-TensorRT combined with rlaphoenix/VSGAN. This repo makes the usage of tiling and ESRGAN models very easy. Models can be found on the wiki page. Further model architectures are planned to be added later on.

Currently working:

  • ESRGAN
  • RealESRGAN (adjust model load manually in inference.py, settings wont be adjusted automatically currently)

Usage:

# install docker, command for arch
yay -S docker nvidia-docker nvidia-container-toolkit
# Put the dockerfile in a directory and run that inside that directory
docker build -t vsgan_tensorrt:latest .
# run with a mounted folder
docker run --privileged --gpus all -it --rm -v /home/Desktop/tensorrt:/workspace/tensorrt vsgan_tensorrt:latest
# you can use it in various ways, ffmpeg example
vspipe --y4m inference.py - | ffmpeg -i pipe: example.mkv

If docker does not want to start, try this before you use docker:

# fixing docker errors
systemctl start docker
sudo chmod 666 /var/run/docker.sock

Windows is mostly similar, but the path needs to be changed slightly:

Example for C://path
docker run --privileged --gpus all -it --rm -v //c/path:/workspace/tensorrt vsgan_tensorrt:latest

If you don't want to use docker, vapoursynth install commands are here and a TensorRT example is here.

Set the input video path in inference.py and access videos with the mounted folder.

It is also possible to directly pipe the video into mpv, but you most likely wont be able to archive realtime speed. Change the mounted folder path to your own videofolder and use the mpv dockerfile instead. If you use a very efficient model, it may be possible on a very good GPU. Only tested in Manjaro.

yay -S pulseaudio

# i am not sure if it is needed, but go into pulseaudio settings and check "make pulseaudio network audio devices discoverable in the local network" and reboot

# start docker
docker run --rm -i -t \
    --network host \
    -e DISPLAY \
    -v /home/Schreibtisch/test/:/home/mpv/media \
    --ipc=host \
    --privileged \
    --gpus all \
    -e PULSE_COOKIE=/run/pulse/cookie \
    -v ~/.config/pulse/cookie:/run/pulse/cookie \
    -e PULSE_SERVER=unix:${XDG_RUNTIME_DIR}/pulse/native \
    -v ${XDG_RUNTIME_DIR}/pulse/native:${XDG_RUNTIME_DIR}/pulse/native \
    vsgan_tensorrt:latest
    
# run mpv
vspipe --y4m inference.py - | mpv -
Comments
  • Invalid data found when processing input

    Invalid data found when processing input

    Hey when i start the inference.py script this happen :

    someone can help me ?

    
    > ffmpeg version N-62110-g4d45f5acbd-static https://johnvansickle.com/ffmpeg/  Copyright (c) 2000-2022 the FFmpeg developers
    >   built with gcc 8 (Debian 8.3.0-6)
    >   configuration: --enable-gpl --enable-version3 --enable-static --disable-debug --disable-ffplay --disable-indev=sndio --disable-outdev=sndio --cc=gcc --enable-fontconfig --enable-frei0r --enable-gnutls --enable-gmp --enable-libgme --enable-gray --enable-libaom --enable-libfribidi --enable-libass --enable-libvmaf --enable-libfreetype --enable-libmp3lame --enable-libopencore-amrnb --enable-libopencore-amrwb --enable-libopenjpeg --enable-librubberband --enable-libsoxr --enable-libspeex --enable-libsrt --enable-libvorbis --enable-libopus --enable-libtheora --enable-libvidstab --enable-libvo-amrwbenc --enable-libvpx --enable-libwebp --enable-libx264 --enable-libx265 --enable-libxml2 --enable-libdav1d --enable-libxvid --enable-libzvbi --enable-libzimg
    >   libavutil      57. 26.100 / 57. 26.100
    >   libavcodec     59. 33.100 / 59. 33.100
    >   libavformat    59. 24.100 / 59. 24.100
    >   libavdevice    59.  6.100 / 59.  6.100
    >   libavfilter     8. 40.100 /  8. 40.100
    >   libswscale      6.  6.100 /  6.  6.100
    >   libswresample   4.  6.100 /  4.  6.100
    >   libpostproc    56.  5.100 / 56.  5.100
    > Information: Generating grammar tables from /usr/lib/python3.8/lib2to3/Grammar.txt
    > Information: Generating grammar tables from /usr/lib/python3.8/lib2to3/PatternGrammar.txt
    > Script evaluation failed:
    > Python exception: libtorch_cuda_cu.so: cannot open shared object file: No such file or directory
    > 
    > Traceback (most recent call last):
    >   File "src\cython\vapoursynth.pyx", line 2890, in vapoursynth._vpy_evaluate
    >   File "src\cython\vapoursynth.pyx", line 2891, in vapoursynth._vpy_evaluate
    >   File "inference.py", line 85, in <module>
    >     clip = ESRGAN_inference(clip=clip, model_path="/workspace/RealESRGAN_x4plus_anime_6B.pth", tile_x=480, tile_y=480, tile_pad=16, fp16=False, tta=False, tta_mode=1)
    >   File "/workspace/tensorrt/src/esrgan.py", line 680, in ESRGAN_inference
    >     import torch_tensorrt
    >   File "/usr/local/lib/python3.8/dist-packages/torch_tensorrt/__init__.py", line 11, in <module>
    >     from torch_tensorrt._compile import *
    >   File "/usr/local/lib/python3.8/dist-packages/torch_tensorrt/_compile.py", line 2, in <module>
    >     from torch_tensorrt import _enums
    >   File "/usr/local/lib/python3.8/dist-packages/torch_tensorrt/_enums.py", line 1, in <module>
    >     from torch_tensorrt._C import dtype, DeviceType, EngineCapability, TensorFormat
    > ImportError: libtorch_cuda_cu.so: cannot open shared object file: No such file or directory
    > 
    > pipe:: Invalid data found when processing input
    
    
    opened by NeoBurgerYT 10
  • Module not found 'scipy'

    Module not found 'scipy'

    I can't run my inference.py without getting this error message. Can someone direct me to where I can get the repo?

    File "/usr/local/lib/python3.8/dist-packages/mmedit/core/evaluation/metrics.py", line 7, in from scipy.ndimage import convolve ModuleNotFoundError: No module named 'scipy'

    pipe:: Invalid data found when processing input

    opened by terminatedkhla 8
  • Tutorial?

    Tutorial?

    Hi! This is amazing technology! I’m blown away. I’d love to contact you directly on how to use it in colab, I’m quite confused with the process. I’ve tried running it but not sure I’m running it correctly. Thanks in advance!

    opened by AIManifest 6
  • Trying On A M1 Mac

    Trying On A M1 Mac

    So I followed this tutorial https://www.youtube.com/watch?v=B134jvhO8yk&t=0s But when docker run --privileged --gpus all -it --rm -v /home/vsgan_path/:/workspace/tensorrt styler00dollar/vsgan_tensorrt:latest it just gives me an error that it doesn't find the right amd64 or somthing and I rage quit deleted it without seeing the full error. PLS HELP ME :(

    opened by Ghostkwebb 6
  • Crash when using RIFE ensemble models in vsmlrt

    Crash when using RIFE ensemble models in vsmlrt

    I get this error

    vapoursynth.Error: operator (): expects 8 input planes
    

    from this

    import vapoursynth as vs
    from vapoursynth import core
    core = vs.core
    import vsmlrt
    
    clip = core.lsmas.LWLibavSource(source=r"R:\output.mkv",cache=1, prefer_hw=1)
    clip = core.resize.Bicubic(clip, matrix_in_s="709", transfer_in_s='709', format=vs.RGBS)
    clip = vsmlrt.RIFE(clip, multi=4, model=46, backend=vsmlrt.Backend.TRT(fp16=True), tilesize=[1920,1088])
    clip = core.std.AssumeFPS(clip=clip, fpsnum=60, fpsden=1)
    clip = core.resize.Bicubic(clip, format=vs.RGB24, matrix_in_s="709")
    clip.set_output()
    
    opened by banjaminicc 4
  • Support for AITemplate?

    Support for AITemplate?

    There is something that came out recently and it's look promising in terms of performance/speed. Would it be possible to implement it for ESERGAN mode? https://github.com/facebookincubator/AITemplate

    opened by kodxana 4
  • CUDA out of Memory

    CUDA out of Memory

    System Specs: Ryzen 9 5900HX, NVidia 3070 Mobile, Arch Linux (EndeavorOS) on Kernel 5.17.2

    Whenever I try to run a model that is relying on CUDA, for example cugan, the program exits with

    Error: Failed to retrieve frame 0 with error: CUDA out of memory. Tried to allocate 148.00 MiB (GPU 0; 7.80 GiB total capacity; 5.53 GiB already allocated; 68.56 MiB free; 5.69 GiB reserved in total by PyTorch)

    and stops after having output 4 frames.

    However, TensorRT works fine for models that support it (like RealESRGAN for example).

    Edit: Running nvidia-smi while the command is executed reveals that vspipe is allocating GPU Memory, but <2 GiB of VRAM, far from the 8GiB my model has.

    opened by mmkzer0 4
  • No module named 'vsbasicvsrpp'

    No module named 'vsbasicvsrpp'

    Traceback (most recent call last): File "src\cython\vapoursynth.pyx", line 2832, in vapoursynth._vpy_evaluate File "src\cython\vapoursynth.pyx", line 2833, in vapoursynth._vpy_evaluate File "inference.py", line 12, in from vsbasicvsrpp import BasicVSRPP ModuleNotFoundError: No module named 'vsbasicvsrpp'

    opened by xt851231 4
  • Google colab request?

    Google colab request?

    I recently stumbled upon this VSGAN-tensorrt-docker and found it so incredible! Could anyone make a google colab notebook that features everything from this VSGAN-tensorrt-docker, so that we could experience the speed of TensorRT! Thanks in advance!

    opened by mikebilly 3
  • model conversion from onnx to trt

    model conversion from onnx to trt

    @styler00dollar this is not issue but a question, I read the scripts in inference.py and found real-esrgan 2x is loaded from trt engine file, since real-2x uses dynamic shapes as input, could you share any ideas how to convert this model to trt, thanks!

    opened by deism 3
  • ESRGAN with full episode

    ESRGAN with full episode

    Hello,

    I'm trying to upscale MKV files of full episodes with ESRGAN. I tried using vspipe -c y4m inference.py - | ffmpeg -i pipe: example.mkv, and it seems to run up to the point where it starts to give an ETA. Once there the time doesn't move and eventually, it says it was killed.

    Can you give me some tips on how to make this work better? I'm not familiar with most of the tools I've been given.

    opened by Ultramonte 2
  • [SUGGESTION] per-scene processing

    [SUGGESTION] per-scene processing

    Hi there, this project is awesome so thanks for your - voluntary - work !

    Since GANs-based processing is quite heavy computing task, it could be very useful to split it into multiple "segments" to allow parallel/scalable/collaborative/resumable instances.

    We suggest you to check @master-of-zen's Av1an framework, wich implements it.

    Hope that inspires.

    opened by forart 1
Releases(models)
Owner
I like Google Colab and Python.
Code for paper "ASAP-Net: Attention and Structure Aware Point Cloud Sequence Segmentation"

ASAP-Net This project implements ASAP-Net of paper ASAP-Net: Attention and Structure Aware Point Cloud Sequence Segmentation (BMVC2020). Overview We i

Hanwen Cao 26 Aug 25, 2022
PyTorch implementation for NED. It can be used to manipulate the facial emotions of actors in videos based on emotion labels or reference styles.

Neural Emotion Director (NED) - Official Pytorch Implementation Example video of facial emotion manipulation while retaining the original mouth motion

Foivos Paraperas 89 Dec 23, 2022
A certifiable defense against adversarial examples by training neural networks to be provably robust

DiffAI v3 DiffAI is a system for training neural networks to be provably robust and for proving that they are robust. The system was developed for the

SRI Lab, ETH Zurich 202 Dec 13, 2022
Speckle-free Holography with Partially Coherent Light Sources and Camera-in-the-loop Calibration

Speckle-free Holography with Partially Coherent Light Sources and Camera-in-the-loop Calibration Project Page | Paper Yifan Peng*, Suyeon Choi*, Jongh

Stanford Computational Imaging Lab 19 Dec 11, 2022
SAMO: Streaming Architecture Mapping Optimisation

SAMO: Streaming Architecture Mapping Optimiser The SAMO framework provides a method of optimising the mapping of a Convolutional Neural Network model

Alexander Montgomerie-Corcoran 20 Dec 10, 2022
pytorch implementation for PointNet

PointNet.pytorch This repo is implementation for PointNet in pytorch. The model is in pointnet/model.py. It is teste

Fei Xia 1.7k Dec 30, 2022
Python code for the paper How to scale hyperparameters for quickshift image segmentation

How to scale hyperparameters for quickshift image segmentation Python code for the paper How to scale hyperparameters for quickshift image segmentatio

0 Jan 25, 2022
The official PyTorch code for NeurIPS 2021 ML4AD Paper, "Does Thermal data make the detection systems more reliable?"

MultiModal-Collaborative (MMC) Learning Framework for integrating RGB and Thermal spectral modalities This is the official code for NeurIPS 2021 Machi

NeurAI 12 Nov 02, 2022
A library for answering questions using data you cannot see

A library for computing on data you do not own and cannot see PySyft is a Python library for secure and private Deep Learning. PySyft decouples privat

OpenMined 8.5k Jan 02, 2023
A Japanese Medical Information Extraction Toolkit

JaMIE: a Japanese Medical Information Extraction toolkit Joint Japanese Medical Problem, Modality and Relation Recognition The Train/Test phrases requ

7 Dec 12, 2022
Video Representation Learning by Recognizing Temporal Transformations. In ECCV, 2020.

Video Representation Learning by Recognizing Temporal Transformations [Project Page] Simon Jenni, Givi Meishvili, and Paolo Favaro. In ECCV, 2020. Thi

Simon Jenni 46 Nov 14, 2022
Official Pytorch implementation of Meta Internal Learning

Official Pytorch implementation of Meta Internal Learning

10 Aug 24, 2022
DCSAU-Net: A Deeper and More Compact Split-Attention U-Net for Medical Image Segmentation

DCSAU-Net: A Deeper and More Compact Split-Attention U-Net for Medical Image Segmentation By Qing Xu, Wenting Duan and Na He Requirements pytorch==1.1

Qing Xu 20 Dec 09, 2022
Contextualized Perturbation for Textual Adversarial Attack, NAACL 2021

Contextualized Perturbation for Textual Adversarial Attack Introduction This is a PyTorch implementation of Contextualized Perturbation for Textual Ad

cookielee77 30 Jan 01, 2023
Code release for "MERLOT Reserve: Neural Script Knowledge through Vision and Language and Sound"

merlot_reserve Code release for "MERLOT Reserve: Neural Script Knowledge through Vision and Language and Sound" MERLOT Reserve (in submission) is a mo

Rowan Zellers 92 Dec 11, 2022
Multispectral Object Detection with Yolov5

Multispectral-Object-Detection Intro Official Code for Cross-Modality Fusion Transformer for Multispectral Object Detection. Multispectral Object Dete

Richard Fang 121 Jan 01, 2023
PyTorch wrapper for Taichi data-oriented class

Stannum PyTorch wrapper for Taichi data-oriented class PRs are welcomed, please see TODOs. Usage from stannum import Tin import torch data_oriented =

86 Dec 23, 2022
transfer attack; adversarial examples; black-box attack; unrestricted Adversarial Attacks on ImageNet; CVPR2021 天池黑盒竞赛

transfer_adv CVPR-2021 AIC-VI: unrestricted Adversarial Attacks on ImageNet CVPR2021 安全AI挑战者计划第六期赛道2:ImageNet无限制对抗攻击 介绍 : 深度神经网络已经在各种视觉识别问题上取得了最先进的性能。

25 Dec 08, 2022
Some bravo or inspiring research works on the topic of curriculum learning.

Towards Scalable Unpaired Virtual Try-On via Patch-Routed Spatially-Adaptive GAN Official code for NeurIPS 2021 paper "Towards Scalable Unpaired Virtu

131 Jan 07, 2023
Simulated garment dataset for virtual try-on

Simulated garment dataset for virtual try-on This repository contains the dataset used in the following papers: Self-Supervised Collision Handling via

33 Dec 20, 2022