Official PyTorch Implementation of paper "NeLF: Neural Light-transport Field for Single Portrait View Synthesis and Relighting", EGSR 2021.

Tiancheng Sun^1*, Kai-En Lin^1*, Sai Bi², Zexiang Xu², Ravi Ramamoorthi¹

¹University of California, San Diego, ²Adobe Research

^*Equal contribution

Project Page | Paper | Pretrained models | Validation data | Rendering script

Make sure you have up-to-date NVIDIA drivers supporting CUDA 11.1 (10.2 could work but need to change cudatoolkit package accordingly)

Run

conda env create -f environment.yml
conda activate nelf

The following packages are used:

• PyTorch (1.7 & 1.9.0 Tested)

• OpenCV-Python

• matplotlib

• numpy

• tqdm

OS system: Ubuntu 20.04

1. Download the dataset

2. Remove background with the provided masks in the dataset

3. Downsample the dataset to 512x512

4. Store the resulting data in [path_to_data_directory]/CelebAMask

   Following this data structure

   [path_to_data_directory] --- data --- CelebAMask --- 0.jpg
                                      |              |- 1.jpg
                                      |              |- 2.jpg
                                      |              ...
                                      |- blender_both --- sub001
                                      |                |- sub002
                                      |                ...
   
   

Due to FaceScape's license, we cannot release the full dataset. Instead, we will release our rendering script.

1. Download the dataset

2. Install Blender link

3. Run rendering script link

0. Download our pretrained checkpoint and testing data. Extract the content to [path_to_data_directory]. The data structure should look like this:

   [path_to_data_directory] --- data --- CelebAMask
                             |        |- blender_both
                             |        |- blender_view
                             |        ...
                             |- data_results --- nelf_ft
                             |- data_test --- validate_0
                                           |- validate_1
                                           |- validate_2
   

1. In arg/__init__.py, setup data path by changing base_path

2. Run python run_test.py nelf_ft [validation_data_name] [#iteration_for_the_model]

   e.g. python run_test.py nelf_ft validate_0 500000

3. The results are stored in [path_to_data_directory]/data_test/[validation_data_name]/results

Due to FaceScape's license, we are not allowed to release the full dataset. We will use validation data to run the following example.

0. Download our validation data. Extract the content to [path_to_data_directory]. The data structure should look like this:

   [path_to_data_directory] --- data --- CelebAMask
                             |        |- blender_both
                             |        |- blender_view
                             |        ...
                             |- data_results --- nelf_ft
                             |- data_test --- validate_0
                                           |- validate_1
                                           |- validate_2
   

   (Optional) Run rendering script and render your own data.

   Remember to change line 35~42 and line 45, 46 in arg/config_nelf_ft.py accordingly.

1. In arg/__init__.py, setup data path by changing base_path

2. Run python run_train.py nelf_ft

3. The intermediate results and model checkpoints are saved in [path_to_data_directory]/data_results/nelf_ft

The following config files can be found inside arg folder

• nelf_ft is our main model described in the paper

• ibr is our reimplementation of IBRNet

• sipr is our reimplementation of Single Image Portrait Relighting

@inproceedings {sun2021nelf,
    booktitle = {Eurographics Symposium on Rendering},
    title = {NeLF: Neural Light-transport Field for Portrait View Synthesis and Relighting},
    author = {Sun, Tiancheng and Lin, Kai-En and Bi, Sai and Xu, Zexiang and Ramamoorthi, Ravi},
    year = {2021},
}