Neural Residual Flow Fields for Efficient Video Representations

1. Download MPI sintel dataset

Download MPI sintel dataset from here

2. GMA optical flow estimator

To obtain optical flow estimations for pretraining, we are using GMA from here. Note that it dose not have to do with our identity.

3. Training

Training neural residual flow fields (NRFF)

# frame 0 - 6
python train_video_flow_midkey.py --use-estimator --lr 0.0005 --training-step 30000 --data-dir {sintel dataset training directory} --video-name alley_1 --start-frame 0 --num-frames 7 --jpeg-quality 98 --hidden-features 96 --use-estimator --tag start0_jq98_hf96
# frame 7 - 13
python train_video_flow_midkey.py --use-estimator --lr 0.0005 --training-step 30000 --data-dir {sintel dataset training directory} --video-name alley_1 --start-frame 7 --num-frames 7 --jpeg-quality 98 --hidden-features 96 --use-estimator --tag start7_jq98_hf96
# frame 14 - 20
python train_video_flow_midkey.py --use-estimator --lr 0.0005 --training-step 30000 --data-dir {sintel dataset training directory} --video-name alley_1 --start-frame 14 --num-frames 7 --jpeg-quality 98 --hidden-features 96 --use-estimator --tag start14_jq98_hf96
# frame 21 - 27
python train_video_flow_midkey.py --use-estimator --lr 0.0005 --training-step 30000 --data-dir {sintel dataset training directory} --video-name alley_1 --start-frame 21 --num-frames 7 --jpeg-quality 98 --hidden-features 96 --use-estimator --tag start21_jq98_hf96

Training baseline (SIREN)

python train_video.py --data-dir {sintel dataset training directory} --video-name alley_1 --hidden-features 256 --num-frames 28 --lr 0.001 --training-step 30000 --tag baseline_siren_hf256

4. Examples

alley_2.mp4

HoneyBee.mp4

Eff video representation - Efficient video representation through neural fields

Related tags

Overview

Neural Residual Flow Fields for Efficient Video Representations

1. Download MPI sintel dataset

2. GMA optical flow estimator

3. Training

4. Examples

Owner

SAS output to EXCEL converter for Cornell/MIT Language and acquisition lab

Oriented Response Networks, in CVPR 2017

Implementation of momentum^2 teacher

LSTM Neural Networks for Spectroscopic Studies of Type Ia Supernovae

LLVIP: A Visible-infrared Paired Dataset for Low-light Vision

Code for IntraQ, PyTorch implementation of our paper under review

Spearmint Bayesian optimization codebase

Parris, the automated infrastructure setup tool for machine learning algorithms.

This is a collection of our NAS and Vision Transformer work.

Official repository for "PAIR: Planning and Iterative Refinement in Pre-trained Transformers for Long Text Generation"

An unofficial styleguide and best practices summary for PyTorch

CVPR 2021: "Generating Diverse Structure for Image Inpainting With Hierarchical VQ-VAE"

This is the official code for the paper "Learning with Nested Scene Modeling and Cooperative Architecture Search for Low-Light Vision"

Codebase for Time-series Generative Adversarial Networks (TimeGAN)

G-NIA model from "Single Node Injection Attack against Graph Neural Networks" (CIKM 2021)

An adaptive hierarchical energy management strategy for hybrid electric vehicles

A Python-based development platform for automated trading systems - from backtesting to optimisation to livetrading.

[BMVC2021] "TransFusion: Cross-view Fusion with Transformer for 3D Human Pose Estimation"

This repo provides function call to track multi-objects in videos

The PyTorch implementation of Directed Graph Contrastive Learning (DiGCL), NeurIPS-2021