Putting NeRF on a Diet: Semantically Consistent Few-Shot View Synthesis

Last update: Dec 27, 2022

Related tags

Overview

Putting NeRF on a Diet: Semantically Consistent Few-Shot View Synthesis

This repository contains the official implementation of DietNeRF, a system that reconstructs 3D scenes from a few posed photos.

Setup

We use the following folder structure:

dietnerf/
  logs/ (images, videos, checkpoints)
  data/
    nerf_synthetic/
  configs/ (run configuration files)
CLIP/ (Fork of OpenAI's clip repository with a wrapper)

Create conda environment:

conda create -n dietnerf python=3.9
conda activate dietnerf

Set up requirements and our fork of CLIP:

pip install -r requirements.txt
cd CLIP
pip install -e .

wandb login

Experiments on the Realistic Synthetic dataset

Realistic Synthetic experiments are implemented in the ./dietnerf subdirectory.

You need to download datasets from NeRF's Google Drive folder. The dataset was used in the original NeRF paper by Mildenhall et al. For example,

mkdir dietnerf/logs/ dietnerf/data/
cd dietnerf/data
pip install gdown
gdown --id 18JxhpWD-4ZmuFKLzKlAw-w5PpzZxXOcG -O nerf_synthetic.zip
unzip nerf_synthetic.zip
rm -r __MACOSX

Then, shrink images to 400x400:

python dietnerf/scripts/bulk_shrink_images.py "dietnerf/data/nerf_synthetic/*/*/*.png" dietnerf/data/nerf_synthetic_400_rgb/ True

These images are used for FID/KID computation. The dietnerf/run_nerf.py training and evaluation code automatically shrinks images with the --half_res argument.

Each experiment has a config file stored in dietnerf/configs/. Scripts in dietnerf/scripts/ can be run to train and evaluate models. Run these scripts from ./dietnerf. The scripts assume you are running one script at a time on a server with 8 NVIDIA GPUs.

cd dietnerf
export WANDB_ENTITY=
   
    

# NeRF baselines
sh scripts/run_synthetic_nerf_100v.sh
sh scripts/run_synthetic_nerf_8v.sh
sh scripts/run_synthetic_simplified_nerf_8v.sh

# DietNeRF with 8 observed views
sh scripts/run_synthetic_dietnerf_8v.sh
sh scripts/run_synthetic_dietnerf_ft_8v.sh

# NeRF and DietNeRF with partial observability
sh scripts/run_synthetic_unseen_side_14v.sh

Experiments on the DTU dataset

Coming soon. Our paper also fine-tunes pixelNeRF on DTU scenes for 1-shot view synthesis.

Citation and acknowledgements

If DietNeRF is relevant to your project, please cite our associated paper:

@InProceedings{Jain_2021_ICCV,
    author    = {Jain, Ajay and Tancik, Matthew and Abbeel, Pieter},
    title     = {Putting NeRF on a Diet: Semantically Consistent Few-Shot View Synthesis},
    booktitle = {Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)},
    month     = {October},
    year      = {2021},
    pages     = {5885-5894}
}

This code is based on Yen-Chen Lin's PyTorch implementation of NeRF and the official pixelNeRF code.

Putting NeRF on a Diet: Semantically Consistent Few-Shot View Synthesis

Related tags

Overview

Putting NeRF on a Diet: Semantically Consistent Few-Shot View Synthesis

Setup

Experiments on the Realistic Synthetic dataset

Experiments on the DTU dataset

Citation and acknowledgements

Owner

Ajay Jain

Public implementation of "Learning from Suboptimal Demonstration via Self-Supervised Reward Regression" from CoRL'21

Towards Calibrated Model for Long-Tailed Visual Recognition from Prior Perspective

[ICCV'21] UNISURF: Unifying Neural Implicit Surfaces and Radiance Fields for Multi-View Reconstruction

A Pytorch implementation of the multi agent deep deterministic policy gradients (MADDPG) algorithm

Preprocessed Datasets for our Multimodal NER paper

PixelPick This is an official implementation of the paper "All you need are a few pixels: semantic segmentation with PixelPick."

Sky Computing: Accelerating Geo-distributed Computing in Federated Learning

Self-supervised Deep LiDAR Odometry for Robotic Applications

AOT-GAN for High-Resolution Image Inpainting (codebase for image inpainting)

Playable Video Generation

Node for thenewboston digital currency network.

Official code for paper Exemplar Based 3D Portrait Stylization.

Official code for Score-Based Generative Modeling through Stochastic Differential Equations

[CVPR 2022] Thin-Plate Spline Motion Model for Image Animation.

Generative vs Discriminative: Rethinking The Meta-Continual Learning (NeurIPS 2021)

TorchIO is a Medical image preprocessing and augmentation toolkit for deep learning. Part of the PyTorch Ecosystem.

Automatic learning-rate scheduler

Official implementation of AAAI-21 paper "Label Confusion Learning to Enhance Text Classification Models"

Codes for AAAI 2022 paper: Context-aware Health Event Prediction via Transition Functions on Dynamic Disease Graphs

Official implementation of "Generating 3D Molecules for Target Protein Binding"