Neural Residual Flow Fields for Efficient Video Representations

1. Download MPI sintel dataset

Download MPI sintel dataset from here

2. GMA optical flow estimator

To obtain optical flow estimations for pretraining, we are using GMA from here. Note that it dose not have to do with our identity.

3. Training

Training neural residual flow fields (NRFF)

# frame 0 - 6
python train_video_flow_midkey.py --use-estimator --lr 0.0005 --training-step 30000 --data-dir {sintel dataset training directory} --video-name alley_1 --start-frame 0 --num-frames 7 --jpeg-quality 98 --hidden-features 96 --use-estimator --tag start0_jq98_hf96
# frame 7 - 13
python train_video_flow_midkey.py --use-estimator --lr 0.0005 --training-step 30000 --data-dir {sintel dataset training directory} --video-name alley_1 --start-frame 7 --num-frames 7 --jpeg-quality 98 --hidden-features 96 --use-estimator --tag start7_jq98_hf96
# frame 14 - 20
python train_video_flow_midkey.py --use-estimator --lr 0.0005 --training-step 30000 --data-dir {sintel dataset training directory} --video-name alley_1 --start-frame 14 --num-frames 7 --jpeg-quality 98 --hidden-features 96 --use-estimator --tag start14_jq98_hf96
# frame 21 - 27
python train_video_flow_midkey.py --use-estimator --lr 0.0005 --training-step 30000 --data-dir {sintel dataset training directory} --video-name alley_1 --start-frame 21 --num-frames 7 --jpeg-quality 98 --hidden-features 96 --use-estimator --tag start21_jq98_hf96

Training baseline (SIREN)

python train_video.py --data-dir {sintel dataset training directory} --video-name alley_1 --hidden-features 256 --num-frames 28 --lr 0.001 --training-step 30000 --tag baseline_siren_hf256

4. Examples

alley_2.mp4

HoneyBee.mp4

Eff video representation - Efficient video representation through neural fields

Related tags

Overview

Neural Residual Flow Fields for Efficient Video Representations

1. Download MPI sintel dataset

2. GMA optical flow estimator

3. Training

4. Examples

Owner

PyTorch implementations of Generative Adversarial Networks.

Self-Supervised Monocular DepthEstimation with Internal Feature Fusion(arXiv), BMVC2021

The Unreasonable Effectiveness of Random Pruning: Return of the Most Naive Baseline for Sparse Training

Pynomial - a lightweight python library for implementing the many confidence intervals for the risk parameter of a binomial model

DC3: A Learning Method for Optimization with Hard Constraints

Open-sourcing the Slates Dataset for recommender systems research

Geometry-Free View Synthesis: Transformers and no 3D Priors

pytorch implementation for Photo-Realistic Single Image Super-Resolution Using a Generative Adversarial Network arXiv:1609.04802

Target Propagation via Regularized Inversion

Spectral Temporal Graph Neural Network (StemGNN in short) for Multivariate Time-series Forecasting

Using deep actor-critic model to learn best strategies in pair trading

Back to Basics: Efficient Network Compression via IMP

[CVPR 2020] Transform and Tell: Entity-Aware News Image Captioning

RCD: Relation Map Driven Cognitive Diagnosis for Intelligent Education Systems

🤗 Transformers: State-of-the-art Natural Language Processing for Pytorch, TensorFlow, and JAX.

TagLab: an image segmentation tool oriented to marine data analysis

Unified Pre-training for Self-Supervised Learning and Supervised Learning for ASR

Local Multi-Head Channel Self-Attention for FER2013

A 3D Dense mapping backend library of SLAM based on taichi-Lang designed for the aerial swarm.

HiFi++: a Unified Framework for Neural Vocoding, Bandwidth Extension and Speech Enhancement