Official PyTorch Implementation of paper "NeLF: Neural Light-transport Field for Single Portrait View Synthesis and Relighting", EGSR 2021.

Related tags

Text Data & NLPnelf
Overview

NeLF: Neural Light-transport Field for Single Portrait View Synthesis and Relighting

Official PyTorch Implementation of paper "NeLF: Neural Light-transport Field for Single Portrait View Synthesis and Relighting", EGSR 2021.

Tiancheng Sun1*, Kai-En Lin1*, Sai Bi2, Zexiang Xu2, Ravi Ramamoorthi1

1University of California, San Diego, 2Adobe Research

*Equal contribution

Project Page | Paper | Pretrained models | Validation data | Rendering script

Requirements

Install required packages

Make sure you have up-to-date NVIDIA drivers supporting CUDA 11.1 (10.2 could work but need to change cudatoolkit package accordingly)

Run

conda env create -f environment.yml
conda activate pixelnerf

The following packages are used:

  • PyTorch (1.7 & 1.9.0 Tested)

  • OpenCV-Python

  • matplotlib

  • numpy

  • tqdm

OS system: Ubuntu 20.04

Download CelebAMask-HQ dataset link

  1. Download the dataset

  2. Remove background with the provided masks in the dataset

  3. Downsample the dataset to 512x512

  4. Store the resulting data in [path_to_data_directory]/CelebAMask

    Following this data structure

    [path_to_data_directory] --- data --- CelebAMask --- 0.jpg
                                       |              |- 1.jpg
                                       |              |- 2.jpg
                                       |              ...
                                       |- blender_both --- sub001
                                       |                |- sub002
                                       |                ...
    
    

(Optional) Download and render FaceScape dataset link

Due to FaceScape's license, we cannot release the full dataset. Instead, we will release our rendering script.

  1. Download the dataset

  2. Install Blender link

  3. Run rendering script link

Usage

Testing

  1. Download our pretrained checkpoint and testing data. Extract the content to [path_to_data_directory]. The data structure should look like this:

    [path_to_data_directory] --- data --- CelebAMask
                              |        |- blender_both
                              |        |- blender_view
                              |        ...
                              |- data_results --- nelf_ft
                              |- data_test --- validate_0
                                            |- validate_1
                                            |- validate_2
    
  2. In arg/__init__.py, setup data path by changing base_path

  3. Run python run_test.py nelf_ft [validation_data_name] [#iteration_for_the_model]

    e.g. python run_test.py nelf_ft validate_0 500000

  4. The results are stored in [path_to_data_directory]/data_test/[validation_data_name]/results

Training

Due to FaceScape's license, we are not allowed to release the full dataset. We will use validation data to run the following example.

  1. Download our validation data. Extract the content to [path_to_data_directory]. The data structure should look like this:

    [path_to_data_directory] --- data --- CelebAMask
                              |        |- blender_both
                              |        |- blender_view
                              |        ...
                              |- data_results --- nelf_ft
                              |- data_test --- validate_0
                                            |- validate_1
                                            |- validate_2
    

    (Optional) Run rendering script and render your own data.

    Remember to change line 35~42 and line 45, 46 in arg/config_nelf_ft.py accordingly.

  2. In arg/__init__.py, setup data path by changing base_path

  3. Run python run_train.py nelf_ft

  4. The intermediate results and model checkpoints are saved in [path_to_data_directory]/data_results/nelf_ft

Configs

The following config files can be found inside arg folder

Citation

@inproceedings {sun2021nelf,
    booktitle = {Eurographics Symposium on Rendering},
    title = {NeLF: Neural Light-transport Field for Portrait View Synthesis and Relighting},
    author = {Sun, Tiancheng and Lin, Kai-En and Bi, Sai and Xu, Zexiang and Ramamoorthi, Ravi},
    year = {2021},
}
Owner
Ken Lin
Ken Lin
基于pytorch+bert的中文事件抽取

pytorch_bert_event_extraction 基于pytorch+bert的中文事件抽取,主要思想是QA(问答)。 要预先下载好chinese-roberta-wwm-ext模型,并在运行时指定模型的位置。

西西嘛呦 31 Nov 30, 2022
Visual Automata is a Python 3 library built as a wrapper for Caleb Evans' Automata library to add more visualization features.

Visual Automata Copyright 2021 Lewi Lie Uberg Released under the MIT license Visual Automata is a Python 3 library built as a wrapper for Caleb Evans'

Lewi Uberg 55 Nov 17, 2022
Pytorch implementation of winner from VQA Chllange Workshop in CVPR'17

2017 VQA Challenge Winner (CVPR'17 Workshop) pytorch implementation of Tips and Tricks for Visual Question Answering: Learnings from the 2017 Challeng

Mark Dong 166 Dec 11, 2022
無料で使える中品質なテキスト読み上げソフトウェア、VOICEVOXの音声合成エンジン

VOICEVOX ENGINE VOICEVOXの音声合成エンジン。 実態は HTTP サーバーなので、リクエストを送信すればテキスト音声合成できます。 API ドキュメント VOICEVOX ソフトウェアを起動した状態で、ブラウザから

Hiroshiba 3 Jul 05, 2022
Turn clang-tidy warnings and fixes to comments in your pull request

clang-tidy pull request comments A GitHub Action to post clang-tidy warnings and suggestions as review comments on your pull request. What platisd/cla

Dimitris Platis 30 Dec 13, 2022
Revisiting Pre-trained Models for Chinese Natural Language Processing (Findings of EMNLP 2020)

This repository contains the resources in our paper "Revisiting Pre-trained Models for Chinese Natural Language Processing", which will be published i

Yiming Cui 463 Dec 30, 2022
Easy Language Model Pretraining leveraging Huggingface's Transformers and Datasets

Easy Language Model Pretraining leveraging Huggingface's Transformers and Datasets What is LASSL • How to Use What is LASSL LASSL은 LAnguage Semi-Super

LASSL: LAnguage Self-Supervised Learning 116 Dec 27, 2022
基于pytorch_rnn的古诗词生成

pytorch_peot_rnn 基于pytorch_rnn的古诗词生成 说明 config.py里面含有训练、测试、预测的参数,更改后运行: python main.py 预测结果 if config.do_predict: result = trainer.generate('丽日照残春')

西西嘛呦 3 May 26, 2022
Paradigm Shift in NLP - "Paradigm Shift in Natural Language Processing".

Paradigm Shift in NLP Welcome to the webpage for "Paradigm Shift in Natural Language Processing". Some resources of the paper are constantly maintaine

Tianxiang Sun 41 Dec 30, 2022
TEACh is a dataset of human-human interactive dialogues to complete tasks in a simulated household environment.

TEACh is a dataset of human-human interactive dialogues to complete tasks in a simulated household environment.

Alexa 98 Dec 09, 2022
Blazing fast language detection using fastText model

Luga A blazing fast language detection using fastText's language models Luga is a Swahili word for language. fastText provides a blazing fast language

Prayson Wilfred Daniel 18 Dec 20, 2022
AI and Machine Learning workflows on Anthos Bare Metal.

Hybrid and Sovereign AI on Anthos Bare Metal Table of Contents Overview Terraform as IaC Substrate ABM Cluster on GCE using Terraform TensorFlow ResNe

Google Cloud Platform 8 Nov 26, 2022
A Fast Sequence Transducer Implementation with PyTorch Bindings

transducer A Fast Sequence Transducer Implementation with PyTorch Bindings. The corresponding publication is Sequence Transduction with Recurrent Neur

Awni Hannun 184 Dec 18, 2022
Deep learning for NLP crash course at ABBYY.

Deep NLP Course at ABBYY Deep learning for NLP crash course at ABBYY. Suggested textbook: Neural Network Methods in Natural Language Processing by Yoa

Dan Anastasyev 597 Dec 18, 2022
The NewSHead dataset is a multi-doc headline dataset used in NHNet for training a headline summarization model.

This repository contains the raw dataset used in NHNet [1] for the task of News Story Headline Generation. The code of data processing and training is available under Tensorflow Models - NHNet.

Google Research Datasets 31 Jul 15, 2022
🚀Clone a voice in 5 seconds to generate arbitrary speech in real-time

English | 中文 Features 🌍 Chinese supported mandarin and tested with multiple datasets: aidatatang_200zh, magicdata, aishell3, data_aishell, and etc. ?

Vega 25.6k Dec 31, 2022
一个基于Nonebot2和go-cqhttp的娱乐性qq机器人

Takker - 一个普通的QQ机器人 此项目为基于 Nonebot2 和 go-cqhttp 开发,以 Sqlite 作为数据库的QQ群娱乐机器人 关于 纯兴趣开发,部分功能借鉴了大佬们的代码,作为Q群的娱乐+功能性Bot 声明 此项目仅用于学习交流,请勿用于非法用途 这是开发者的第一个Pytho

风屿 79 Dec 29, 2022
NeuTex: Neural Texture Mapping for Volumetric Neural Rendering

NeuTex: Neural Texture Mapping for Volumetric Neural Rendering Paper: https://arxiv.org/abs/2103.00762 Running Run on the provided DTU scene cd run ba

Fanbo Xiang 68 Jan 06, 2023
A minimal Conformer ASR implementation adapted from ESPnet.

Conformer ASR A minimal Conformer ASR implementation adapted from ESPnet. Introduction I want to use the pre-trained English ASR model provided by ESP

Niu Zhe 3 Jan 24, 2022
Stanford CoreNLP provides a set of natural language analysis tools written in Java

Stanford CoreNLP Stanford CoreNLP provides a set of natural language analysis tools written in Java. It can take raw human language text input and giv

Stanford NLP 8.8k Jan 07, 2023