Neural Residual Flow Fields for Efficient Video Representations

1. Download MPI sintel dataset

Download MPI sintel dataset from here

2. GMA optical flow estimator

To obtain optical flow estimations for pretraining, we are using GMA from here. Note that it dose not have to do with our identity.

3. Training

Training neural residual flow fields (NRFF)

# frame 0 - 6
python train_video_flow_midkey.py --use-estimator --lr 0.0005 --training-step 30000 --data-dir {sintel dataset training directory} --video-name alley_1 --start-frame 0 --num-frames 7 --jpeg-quality 98 --hidden-features 96 --use-estimator --tag start0_jq98_hf96
# frame 7 - 13
python train_video_flow_midkey.py --use-estimator --lr 0.0005 --training-step 30000 --data-dir {sintel dataset training directory} --video-name alley_1 --start-frame 7 --num-frames 7 --jpeg-quality 98 --hidden-features 96 --use-estimator --tag start7_jq98_hf96
# frame 14 - 20
python train_video_flow_midkey.py --use-estimator --lr 0.0005 --training-step 30000 --data-dir {sintel dataset training directory} --video-name alley_1 --start-frame 14 --num-frames 7 --jpeg-quality 98 --hidden-features 96 --use-estimator --tag start14_jq98_hf96
# frame 21 - 27
python train_video_flow_midkey.py --use-estimator --lr 0.0005 --training-step 30000 --data-dir {sintel dataset training directory} --video-name alley_1 --start-frame 21 --num-frames 7 --jpeg-quality 98 --hidden-features 96 --use-estimator --tag start21_jq98_hf96

Training baseline (SIREN)

python train_video.py --data-dir {sintel dataset training directory} --video-name alley_1 --hidden-features 256 --num-frames 28 --lr 0.001 --training-step 30000 --tag baseline_siren_hf256

4. Examples

alley_2.mp4

HoneyBee.mp4

Eff video representation - Efficient video representation through neural fields

Related tags

Overview

Neural Residual Flow Fields for Efficient Video Representations

1. Download MPI sintel dataset

2. GMA optical flow estimator

3. Training

4. Examples

Owner

A tool to estimate time varying instantaneous reproduction number during epidemics

Code for Estimating Multi-cause Treatment Effects via Single-cause Perturbation (NeurIPS 2021)

Experiments for Fake News explainability project

The Empirical Investigation of Representation Learning for Imitation (EIRLI)

This repository contains the segmentation user interface from the OpenSurfaces project, extracted as a lightweight tool

Learning Saliency Propagation for Semi-supervised Instance Segmentation

To provide 100 JAX exercises over different sections structured as a course or tutorials to teach and learn for beginners, intermediates as well as experts

Wider-Yolo Kütüphanesi ile Yüz Tespit Uygulamanı Yap

Local-Global Stratified Transformer for Efficient Video Recognition

Loopy belief propagation for factor graphs on discrete variables, in JAX!

Autonomous Ground Vehicle Navigation and Control Simulation Examples in Python

Demonstration of transfer of knowledge and generalization with distillation

This is an open source library implementing hyperbox-based machine learning algorithms

some academic posters as references. May we have in-person poster session soon!

Code repository for Semantic Terrain Classification for Off-Road Autonomous Driving

Official code for CVPR2022 paper: Depth-Aware Generative Adversarial Network for Talking Head Video Generation

Instance-based label smoothing for improving deep neural networks generalization and calibration

LIVECell - A large-scale dataset for label-free live cell segmentation

Bayesian inference for Permuton-induced Chinese Restaurant Process (NeurIPS2021).

Repo for code associated with Modeling the Mitral Valve.