Differentiable rasterization applied to 3D model simplification tasks

Related tags

Deep Learningnvdiffmodeling
Overview

nvdiffmodeling

Teaser image

Differentiable rasterization applied to 3D model simplification tasks, as described in the paper:

Appearance-Driven Automatic 3D Model Simplification
Jon Hasselgren, Jacob Munkberg, Jaakko Lehtinen, Miika Aittala and Samuli Laine
https://research.nvidia.com/publication/2021-04_Appearance-Driven-Automatic-3D
https://arxiv.org/abs/2104.03989

License

Copyright © 2021, NVIDIA Corporation. All rights reserved.

This work is made available under the Nvidia Source Code License.

For business inquiries, please contact [email protected]

Installation

Requires VS2019+, Cuda 10.2+ and PyTorch 1.6+

Tested in Anaconda3 with Python 3.6 and PyTorch 1.8

One time setup (Windows)

Install the Cuda toolkit (required to build the PyTorch extensions). We support Cuda 10.2 and above. Pick the appropriate version of PyTorch compatible with the installed Cuda toolkit. Below is an example with Cuda 11.1

conda create -n dmodel python=3.6
activate dmodel
conda install pytorch torchvision torchaudio cudatoolkit=11.1 -c pytorch -c conda-forge
conda install imageio
pip install PyOpenGL glfw

Install nvdiffrast: https://github.com/NVlabs/nvdiffrast in the dmodel conda env.

Every new command prompt

activate dmodel

Examples

Sphere to cow example:

python train.py --config configs/spot.json

The results will be stored in the out folder. The Spot model was created and released into the public domain by Keenan Crane.

Additional assets can be downloaded here [205MB]. Unzip and place the subfolders in the project data folder, e.g., data\skull. All assets are copyright of their respective authors, see included license files for further details.

Included examples

  • skull.json - Joint normal map and shape optimization on a skull
  • ewer.json - Ewer model from a reduced mesh as initial guess
  • gardenina.json - Aggregate geometry example
  • hibiscus.json - Aggregate geometry example
  • figure_brushed_gold_64.json - LOD example, trained against a supersampled reference
  • figure_displacement.json - Joint shape, normal map, and displacement map example

The json files that end in _paper.json are configs with the settings used for the results in the paper. They take longer and require a GPU with sufficient memory.

Server usage (through Docker)

  • Build docker image (run the command from the code root folder). docker build -f docker/Dockerfile -t diffmod:v1 . Requires a driver that supports Cuda 10.1 or newer.

  • Start an interactive docker container: docker run --gpus device=0 -it --rm -v /raid:/raid -it diffmod:v1 bash

  • Detached docker: docker run --gpus device=1 -d -v /raid:/raid -w=[path to the code] diffmod:v1 python train.py --config configs/spot.json

Comments
  • Proposition for more comprehensive Readme

    Proposition for more comprehensive Readme

    First of all, thank you for making this freely available for everyone! This is really a great move!

    I wanted to suggest a few things for the readme: 1.) Add "Microsoft Visual C++" to the requirements. It should be clear that you have to tick the C++ development environment while installing VS2019.

    2.) Install Git via "conda install git" (This makes it easier to install nvdiffrast and clone the repo)

    3.) Change "Install NVDiffRast" to something more descriptive: pip install -e git+https://github.com/NVlabs/[email protected]#egg=nvdiffrast

    4.) Add another step to the installation, detailing to move to a desired folder via the Anaconda Prompt and cloning this repo: cd YOUR/PATH/HERE git clone https://github.com/NVlabs/nvdiffmodeling.git and afterwards to move into the folder cd nvdiffmodeling

    5.) Installing CUDA via Anaconda only deploys the bare essential files. The CUDA path is not set when doing it this way. So I downloaded the CUDA toolkit from nvidia and installed it afterwards, to fix that.

    I am sure that all of these things are obvious for professionals, but it took me a while to figure all of this out, so I would like to save other people's time.

    opened by morph3us-net 7
  • About the raymarch fitting case

    About the raymarch fitting case

    Thanks for the marvelous work! I have some questions related to the shadertoy fitting case: It seems figure 22 & figure 23 use some special uv un-wrap methods, can you elaborate the algorithm under the hood? image

    opened by outlawever 4
  • RuntimeError: Error building extension 'renderutils_plugin'

    RuntimeError: Error building extension 'renderutils_plugin'

    I am trying to get this to work to test the workflow but ran into the error I have installed "Cuda" 11.1 and everything else as in the tutorial Graphics accelerator 3080t

    After trying to run a test command, it gives me the following text Do you have any ideas how to fix this? I am more of a technical artist and a cursory search did not return much

    
(dmodel) I:\Python\Nvidia_Lod\nvdiffmodeling-main\nvdiffmodeling-main>python train.py --config configs/spot.json
Using C:\Users\alexp\AppData\Local\torch_extensions\torch_extensions\Cache as PyTorch extensions root...
C:\ProgramData\Anaconda3\envs\dmodel\lib\site-packages\torch\utils\cpp_extension.py:305: UserWarning: Error checking compiler version for cl: 'utf-8' codec can't decode byte 0x8e in position 0: invalid start byte
  warnings.warn(f'Error checking compiler version for {compiler}: {error}')
Detected CUDA files, patching ldflags
Emitting ninja build file C:\Users\alexp\AppData\Local\torch_extensions\torch_extensions\Cache\renderutils_plugin\build.ninja...
Building extension module renderutils_plugin...
Allowing ninja to set a default number of workers... (overridable by setting the environment variable MAX_JOBS=N)
[1/7] cl /showIncludes -DTORCH_EXTENSION_NAME=renderutils_plugin -DTORCH_API_INCLUDE_EXTENSION_H -IC:\ProgramData\Anaconda3\envs\dmodel\lib\site-packages\torch\include -IC:\ProgramData\Anaconda3\envs\dmodel\lib\site-packages\torch\include\torch\csrc\api\include -IC:\ProgramData\Anaconda3\envs\dmodel\lib\site-packages\torch\include\TH -IC:\ProgramData\Anaconda3\envs\dmodel\lib\site-packages\torch\include\THC "-IC:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.1\include" -IC:\ProgramData\Anaconda3\envs\dmodel\Include -D_GLIBCXX_USE_CXX11_ABI=0 /MD /wd4819 /wd4251 /wd4244 /wd4267 /wd4275 /wd4018 /wd4190 /EHsc -c I:\Python\Nvidia_Lod\nvdiffmodeling-main\nvdiffmodeling-main\src\renderutils\c_src\common.cpp /Focommon.o
FAILED: common.o
cl /showIncludes -DTORCH_EXTENSION_NAME=renderutils_plugin -DTORCH_API_INCLUDE_EXTENSION_H -IC:\ProgramData\Anaconda3\envs\dmodel\lib\site-packages\torch\include -IC:\ProgramData\Anaconda3\envs\dmodel\lib\site-packages\torch\include\torch\csrc\api\include -IC:\ProgramData\Anaconda3\envs\dmodel\lib\site-packages\torch\include\TH -IC:\ProgramData\Anaconda3\envs\dmodel\lib\site-packages\torch\include\THC "-IC:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.1\include" -IC:\ProgramData\Anaconda3\envs\dmodel\Include -D_GLIBCXX_USE_CXX11_ABI=0 /MD /wd4819 /wd4251 /wd4244 /wd4267 /wd4275 /wd4018 /wd4190 /EHsc -c I:\Python\Nvidia_Lod\nvdiffmodeling-main\nvdiffmodeling-main\src\renderutils\c_src\common.cpp /Focommon.o
Оптимизирующий компилятор Microsoft (R) C/C++ версии 19.25.28611 для x64
(C) Корпорация Майкрософт (Microsoft Corporation).  Все права защищены.

Примечание: включение файла:  C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.1\include\cuda_runtime.h
Примечание: включение файла:   C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.1\include\crt/host_config.h
Примечание: включение файла:   C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.1\include\builtin_types.h
Примечание: включение файла:    C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.1\include\device_types.h
Примечание: включение файла:     C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.1\include\crt/host_defines.h
Примечание: включение файла:    C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.1\include\driver_types.h
Примечание: включение файла:     C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.1\include\crt/host_defines.h
Примечание: включение файла:     C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.1\include\vector_types.h
Примечание: включение файла:      C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.1\include\crt/host_defines.h
Примечание: включение файла:     C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.25.28610\include\limits.h
Примечание: включение файла:      C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.25.28610\include\vcruntime.h
Примечание: включение файла:       C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.25.28610\include\sal.h
Примечание: включение файла:        C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.25.28610\include\concurrencysal.h
Примечание: включение файла:       C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.25.28610\include\vadefs.h
C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.1\include\driver_types.h(78): fatal error C1083: Не удается открыть файл включение: stddef.h: No such file or directory,
[2/7] cl /showIncludes -DTORCH_EXTENSION_NAME=renderutils_plugin -DTORCH_API_INCLUDE_EXTENSION_H -IC:\ProgramData\Anaconda3\envs\dmodel\lib\site-packages\torch\include -IC:\ProgramData\Anaconda3\envs\dmodel\lib\site-packages\torch\include\torch\csrc\api\include -IC:\ProgramData\Anaconda3\envs\dmodel\lib\site-packages\torch\include\TH -IC:\ProgramData\Anaconda3\envs\dmodel\lib\site-packages\torch\include\THC "-IC:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.1\include" -IC:\ProgramData\Anaconda3\envs\dmodel\Include -D_GLIBCXX_USE_CXX11_ABI=0 /MD /wd4819 /wd4251 /wd4244 /wd4267 /wd4275 /wd4018 /wd4190 /EHsc -c I:\Python\Nvidia_Lod\nvdiffmodeling-main\nvdiffmodeling-main\src\renderutils\c_src\torch_bindings.cpp /Fotorch_bindings.o
FAILED: torch_bindings.o
cl /showIncludes -DTORCH_EXTENSION_NAME=renderutils_plugin -DTORCH_API_INCLUDE_EXTENSION_H -IC:\ProgramData\Anaconda3\envs\dmodel\lib\site-packages\torch\include -IC:\ProgramData\Anaconda3\envs\dmodel\lib\site-packages\torch\include\torch\csrc\api\include -IC:\ProgramData\Anaconda3\envs\dmodel\lib\site-packages\torch\include\TH -IC:\ProgramData\Anaconda3\envs\dmodel\lib\site-packages\torch\include\THC "-IC:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.1\include" -IC:\ProgramData\Anaconda3\envs\dmodel\Include -D_GLIBCXX_USE_CXX11_ABI=0 /MD /wd4819 /wd4251 /wd4244 /wd4267 /wd4275 /wd4018 /wd4190 /EHsc -c I:\Python\Nvidia_Lod\nvdiffmodeling-main\nvdiffmodeling-main\src\renderutils\c_src\torch_bindings.cpp /Fotorch_bindings.o
Оптимизирующий компилятор Microsoft (R) C/C++ версии 19.25.28611 для x64
(C) Корпорация Майкрософт (Microsoft Corporation).  Все права защищены.

Примечание: включение файла:  C:\ProgramData\Anaconda3\envs\dmodel\lib\site-packages\torch\include\torch/extension.h
Примечание: включение файла:   C:\ProgramData\Anaconda3\envs\dmodel\lib\site-packages\torch\include\torch\csrc\api\include\torch/all.h
Примечание: включение файла:    C:\ProgramData\Anaconda3\envs\dmodel\lib\site-packages\torch\include\torch\csrc\api\include\torch/cuda.h
Примечание: включение файла:     C:\ProgramData\Anaconda3\envs\dmodel\lib\site-packages\torch\include\torch/csrc/WindowsTorchApiMacro.h
Примечание: включение файла:      C:\ProgramData\Anaconda3\envs\dmodel\lib\site-packages\torch\include\c10/macros/Export.h
Примечание: включение файла:       C:\ProgramData\Anaconda3\envs\dmodel\lib\site-packages\torch\include\c10/macros/cmake_macros.h
Примечание: включение файла:     C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.25.28610\include\cstdint
Примечание: включение файла:      C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.25.28610\include\yvals.h
Примечание: включение файла:       C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.25.28610\include\yvals_core.h
Примечание: включение файла:        C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.25.28610\include\vcruntime.h
Примечание: включение файла:         C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.25.28610\include\sal.h
Примечание: включение файла:          C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.25.28610\include\concurrencysal.h
Примечание: включение файла:         C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.25.28610\include\vadefs.h
Примечание: включение файла:        C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.25.28610\include\xkeycheck.h
C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.25.28610\include\yvals.h(12): fatal error C1083: Не удается открыть файл включение: crtdbg.h: No such file or directory,
[3/7] C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.1\bin\nvcc --generate-dependencies-with-compile --dependency-output mesh.cuda.o.d -Xcudafe --diag_suppress=dll_interface_conflict_dllexport_assumed -Xcudafe --diag_suppress=dll_interface_conflict_none_assumed -Xcudafe --diag_suppress=field_without_dll_interface -Xcudafe --diag_suppress=base_class_has_different_dll_interface -Xcompiler /EHsc -Xcompiler /wd4190 -Xcompiler /wd4018 -Xcompiler /wd4275 -Xcompiler /wd4267 -Xcompiler /wd4244 -Xcompiler /wd4251 -Xcompiler /wd4819 -Xcompiler /MD -DTORCH_EXTENSION_NAME=renderutils_plugin -DTORCH_API_INCLUDE_EXTENSION_H -IC:\ProgramData\Anaconda3\envs\dmodel\lib\site-packages\torch\include -IC:\ProgramData\Anaconda3\envs\dmodel\lib\site-packages\torch\include\torch\csrc\api\include -IC:\ProgramData\Anaconda3\envs\dmodel\lib\site-packages\torch\include\TH -IC:\ProgramData\Anaconda3\envs\dmodel\lib\site-packages\torch\include\THC "-IC:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.1\include" -IC:\ProgramData\Anaconda3\envs\dmodel\Include -D_GLIBCXX_USE_CXX11_ABI=0 -D__CUDA_NO_HALF_OPERATORS__ -D__CUDA_NO_HALF_CONVERSIONS__ -D__CUDA_NO_BFLOAT16_CONVERSIONS__ -D__CUDA_NO_HALF2_OPERATORS__ --expt-relaxed-constexpr -gencode=arch=compute_86,code=compute_86 -gencode=arch=compute_86,code=sm_86 -c I:\Python\Nvidia_Lod\nvdiffmodeling-main\nvdiffmodeling-main\src\renderutils\c_src\mesh.cu -o mesh.cuda.o
FAILED: mesh.cuda.o
C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.1\bin\nvcc --generate-dependencies-with-compile --dependency-output mesh.cuda.o.d -Xcudafe --diag_suppress=dll_interface_conflict_dllexport_assumed -Xcudafe --diag_suppress=dll_interface_conflict_none_assumed -Xcudafe --diag_suppress=field_without_dll_interface -Xcudafe --diag_suppress=base_class_has_different_dll_interface -Xcompiler /EHsc -Xcompiler /wd4190 -Xcompiler /wd4018 -Xcompiler /wd4275 -Xcompiler /wd4267 -Xcompiler /wd4244 -Xcompiler /wd4251 -Xcompiler /wd4819 -Xcompiler /MD -DTORCH_EXTENSION_NAME=renderutils_plugin -DTORCH_API_INCLUDE_EXTENSION_H -IC:\ProgramData\Anaconda3\envs\dmodel\lib\site-packages\torch\include -IC:\ProgramData\Anaconda3\envs\dmodel\lib\site-packages\torch\include\torch\csrc\api\include -IC:\ProgramData\Anaconda3\envs\dmodel\lib\site-packages\torch\include\TH -IC:\ProgramData\Anaconda3\envs\dmodel\lib\site-packages\torch\include\THC "-IC:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.1\include" -IC:\ProgramData\Anaconda3\envs\dmodel\Include -D_GLIBCXX_USE_CXX11_ABI=0 -D__CUDA_NO_HALF_OPERATORS__ -D__CUDA_NO_HALF_CONVERSIONS__ -D__CUDA_NO_BFLOAT16_CONVERSIONS__ -D__CUDA_NO_HALF2_OPERATORS__ --expt-relaxed-constexpr -gencode=arch=compute_86,code=compute_86 -gencode=arch=compute_86,code=sm_86 -c I:\Python\Nvidia_Lod\nvdiffmodeling-main\nvdiffmodeling-main\src\renderutils\c_src\mesh.cu -o mesh.cuda.o
C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.25.28610\include\crtdefs.h(10): fatal error C1083: ═х єфрхЄё  юЄъЁ√Є№ Їрщы тъы■ўхэшх: corecrt.h: No such file or directory,
mesh.cu
[4/7] C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.1\bin\nvcc --generate-dependencies-with-compile --dependency-output loss.cuda.o.d -Xcudafe --diag_suppress=dll_interface_conflict_dllexport_assumed -Xcudafe --diag_suppress=dll_interface_conflict_none_assumed -Xcudafe --diag_suppress=field_without_dll_interface -Xcudafe --diag_suppress=base_class_has_different_dll_interface -Xcompiler /EHsc -Xcompiler /wd4190 -Xcompiler /wd4018 -Xcompiler /wd4275 -Xcompiler /wd4267 -Xcompiler /wd4244 -Xcompiler /wd4251 -Xcompiler /wd4819 -Xcompiler /MD -DTORCH_EXTENSION_NAME=renderutils_plugin -DTORCH_API_INCLUDE_EXTENSION_H -IC:\ProgramData\Anaconda3\envs\dmodel\lib\site-packages\torch\include -IC:\ProgramData\Anaconda3\envs\dmodel\lib\site-packages\torch\include\torch\csrc\api\include -IC:\ProgramData\Anaconda3\envs\dmodel\lib\site-packages\torch\include\TH -IC:\ProgramData\Anaconda3\envs\dmodel\lib\site-packages\torch\include\THC "-IC:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.1\include" -IC:\ProgramData\Anaconda3\envs\dmodel\Include -D_GLIBCXX_USE_CXX11_ABI=0 -D__CUDA_NO_HALF_OPERATORS__ -D__CUDA_NO_HALF_CONVERSIONS__ -D__CUDA_NO_BFLOAT16_CONVERSIONS__ -D__CUDA_NO_HALF2_OPERATORS__ --expt-relaxed-constexpr -gencode=arch=compute_86,code=compute_86 -gencode=arch=compute_86,code=sm_86 -c I:\Python\Nvidia_Lod\nvdiffmodeling-main\nvdiffmodeling-main\src\renderutils\c_src\loss.cu -o loss.cuda.o
FAILED: loss.cuda.o
C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.1\bin\nvcc --generate-dependencies-with-compile --dependency-output loss.cuda.o.d -Xcudafe --diag_suppress=dll_interface_conflict_dllexport_assumed -Xcudafe --diag_suppress=dll_interface_conflict_none_assumed -Xcudafe --diag_suppress=field_without_dll_interface -Xcudafe --diag_suppress=base_class_has_different_dll_interface -Xcompiler /EHsc -Xcompiler /wd4190 -Xcompiler /wd4018 -Xcompiler /wd4275 -Xcompiler /wd4267 -Xcompiler /wd4244 -Xcompiler /wd4251 -Xcompiler /wd4819 -Xcompiler /MD -DTORCH_EXTENSION_NAME=renderutils_plugin -DTORCH_API_INCLUDE_EXTENSION_H -IC:\ProgramData\Anaconda3\envs\dmodel\lib\site-packages\torch\include -IC:\ProgramData\Anaconda3\envs\dmodel\lib\site-packages\torch\include\torch\csrc\api\include -IC:\ProgramData\Anaconda3\envs\dmodel\lib\site-packages\torch\include\TH -IC:\ProgramData\Anaconda3\envs\dmodel\lib\site-packages\torch\include\THC "-IC:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.1\include" -IC:\ProgramData\Anaconda3\envs\dmodel\Include -D_GLIBCXX_USE_CXX11_ABI=0 -D__CUDA_NO_HALF_OPERATORS__ -D__CUDA_NO_HALF_CONVERSIONS__ -D__CUDA_NO_BFLOAT16_CONVERSIONS__ -D__CUDA_NO_HALF2_OPERATORS__ --expt-relaxed-constexpr -gencode=arch=compute_86,code=compute_86 -gencode=arch=compute_86,code=sm_86 -c I:\Python\Nvidia_Lod\nvdiffmodeling-main\nvdiffmodeling-main\src\renderutils\c_src\loss.cu -o loss.cuda.o
C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.25.28610\include\crtdefs.h(10): fatal error C1083: ═х єфрхЄё  юЄъЁ√Є№ Їрщы тъы■ўхэшх: corecrt.h: No such file or directory,
loss.cu
[5/7] C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.1\bin\nvcc --generate-dependencies-with-compile --dependency-output bsdf.cuda.o.d -Xcudafe --diag_suppress=dll_interface_conflict_dllexport_assumed -Xcudafe --diag_suppress=dll_interface_conflict_none_assumed -Xcudafe --diag_suppress=field_without_dll_interface -Xcudafe --diag_suppress=base_class_has_different_dll_interface -Xcompiler /EHsc -Xcompiler /wd4190 -Xcompiler /wd4018 -Xcompiler /wd4275 -Xcompiler /wd4267 -Xcompiler /wd4244 -Xcompiler /wd4251 -Xcompiler /wd4819 -Xcompiler /MD -DTORCH_EXTENSION_NAME=renderutils_plugin -DTORCH_API_INCLUDE_EXTENSION_H -IC:\ProgramData\Anaconda3\envs\dmodel\lib\site-packages\torch\include -IC:\ProgramData\Anaconda3\envs\dmodel\lib\site-packages\torch\include\torch\csrc\api\include -IC:\ProgramData\Anaconda3\envs\dmodel\lib\site-packages\torch\include\TH -IC:\ProgramData\Anaconda3\envs\dmodel\lib\site-packages\torch\include\THC "-IC:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.1\include" -IC:\ProgramData\Anaconda3\envs\dmodel\Include -D_GLIBCXX_USE_CXX11_ABI=0 -D__CUDA_NO_HALF_OPERATORS__ -D__CUDA_NO_HALF_CONVERSIONS__ -D__CUDA_NO_BFLOAT16_CONVERSIONS__ -D__CUDA_NO_HALF2_OPERATORS__ --expt-relaxed-constexpr -gencode=arch=compute_86,code=compute_86 -gencode=arch=compute_86,code=sm_86 -c I:\Python\Nvidia_Lod\nvdiffmodeling-main\nvdiffmodeling-main\src\renderutils\c_src\bsdf.cu -o bsdf.cuda.o
FAILED: bsdf.cuda.o
C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.1\bin\nvcc --generate-dependencies-with-compile --dependency-output bsdf.cuda.o.d -Xcudafe --diag_suppress=dll_interface_conflict_dllexport_assumed -Xcudafe --diag_suppress=dll_interface_conflict_none_assumed -Xcudafe --diag_suppress=field_without_dll_interface -Xcudafe --diag_suppress=base_class_has_different_dll_interface -Xcompiler /EHsc -Xcompiler /wd4190 -Xcompiler /wd4018 -Xcompiler /wd4275 -Xcompiler /wd4267 -Xcompiler /wd4244 -Xcompiler /wd4251 -Xcompiler /wd4819 -Xcompiler /MD -DTORCH_EXTENSION_NAME=renderutils_plugin -DTORCH_API_INCLUDE_EXTENSION_H -IC:\ProgramData\Anaconda3\envs\dmodel\lib\site-packages\torch\include -IC:\ProgramData\Anaconda3\envs\dmodel\lib\site-packages\torch\include\torch\csrc\api\include -IC:\ProgramData\Anaconda3\envs\dmodel\lib\site-packages\torch\include\TH -IC:\ProgramData\Anaconda3\envs\dmodel\lib\site-packages\torch\include\THC "-IC:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.1\include" -IC:\ProgramData\Anaconda3\envs\dmodel\Include -D_GLIBCXX_USE_CXX11_ABI=0 -D__CUDA_NO_HALF_OPERATORS__ -D__CUDA_NO_HALF_CONVERSIONS__ -D__CUDA_NO_BFLOAT16_CONVERSIONS__ -D__CUDA_NO_HALF2_OPERATORS__ --expt-relaxed-constexpr -gencode=arch=compute_86,code=compute_86 -gencode=arch=compute_86,code=sm_86 -c I:\Python\Nvidia_Lod\nvdiffmodeling-main\nvdiffmodeling-main\src\renderutils\c_src\bsdf.cu -o bsdf.cuda.o
C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.25.28610\include\crtdefs.h(10): fatal error C1083: ═х єфрхЄё  юЄъЁ√Є№ Їрщы тъы■ўхэшх: corecrt.h: No such file or directory,
bsdf.cu
[6/7] C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.1\bin\nvcc --generate-dependencies-with-compile --dependency-output normal.cuda.o.d -Xcudafe --diag_suppress=dll_interface_conflict_dllexport_assumed -Xcudafe --diag_suppress=dll_interface_conflict_none_assumed -Xcudafe --diag_suppress=field_without_dll_interface -Xcudafe --diag_suppress=base_class_has_different_dll_interface -Xcompiler /EHsc -Xcompiler /wd4190 -Xcompiler /wd4018 -Xcompiler /wd4275 -Xcompiler /wd4267 -Xcompiler /wd4244 -Xcompiler /wd4251 -Xcompiler /wd4819 -Xcompiler /MD -DTORCH_EXTENSION_NAME=renderutils_plugin -DTORCH_API_INCLUDE_EXTENSION_H -IC:\ProgramData\Anaconda3\envs\dmodel\lib\site-packages\torch\include -IC:\ProgramData\Anaconda3\envs\dmodel\lib\site-packages\torch\include\torch\csrc\api\include -IC:\ProgramData\Anaconda3\envs\dmodel\lib\site-packages\torch\include\TH -IC:\ProgramData\Anaconda3\envs\dmodel\lib\site-packages\torch\include\THC "-IC:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.1\include" -IC:\ProgramData\Anaconda3\envs\dmodel\Include -D_GLIBCXX_USE_CXX11_ABI=0 -D__CUDA_NO_HALF_OPERATORS__ -D__CUDA_NO_HALF_CONVERSIONS__ -D__CUDA_NO_BFLOAT16_CONVERSIONS__ -D__CUDA_NO_HALF2_OPERATORS__ --expt-relaxed-constexpr -gencode=arch=compute_86,code=compute_86 -gencode=arch=compute_86,code=sm_86 -c I:\Python\Nvidia_Lod\nvdiffmodeling-main\nvdiffmodeling-main\src\renderutils\c_src\normal.cu -o normal.cuda.o
FAILED: normal.cuda.o
C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.1\bin\nvcc --generate-dependencies-with-compile --dependency-output normal.cuda.o.d -Xcudafe --diag_suppress=dll_interface_conflict_dllexport_assumed -Xcudafe --diag_suppress=dll_interface_conflict_none_assumed -Xcudafe --diag_suppress=field_without_dll_interface -Xcudafe --diag_suppress=base_class_has_different_dll_interface -Xcompiler /EHsc -Xcompiler /wd4190 -Xcompiler /wd4018 -Xcompiler /wd4275 -Xcompiler /wd4267 -Xcompiler /wd4244 -Xcompiler /wd4251 -Xcompiler /wd4819 -Xcompiler /MD -DTORCH_EXTENSION_NAME=renderutils_plugin -DTORCH_API_INCLUDE_EXTENSION_H -IC:\ProgramData\Anaconda3\envs\dmodel\lib\site-packages\torch\include -IC:\ProgramData\Anaconda3\envs\dmodel\lib\site-packages\torch\include\torch\csrc\api\include -IC:\ProgramData\Anaconda3\envs\dmodel\lib\site-packages\torch\include\TH -IC:\ProgramData\Anaconda3\envs\dmodel\lib\site-packages\torch\include\THC "-IC:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.1\include" -IC:\ProgramData\Anaconda3\envs\dmodel\Include -D_GLIBCXX_USE_CXX11_ABI=0 -D__CUDA_NO_HALF_OPERATORS__ -D__CUDA_NO_HALF_CONVERSIONS__ -D__CUDA_NO_BFLOAT16_CONVERSIONS__ -D__CUDA_NO_HALF2_OPERATORS__ --expt-relaxed-constexpr -gencode=arch=compute_86,code=compute_86 -gencode=arch=compute_86,code=sm_86 -c I:\Python\Nvidia_Lod\nvdiffmodeling-main\nvdiffmodeling-main\src\renderutils\c_src\normal.cu -o normal.cuda.o
C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.25.28610\include\crtdefs.h(10): fatal error C1083: ═х єфрхЄё  юЄъЁ√Є№ Їрщы тъы■ўхэшх: corecrt.h: No such file or directory,
normal.cu
ninja: build stopped: subcommand failed.
Traceback (most recent call last):
  File "C:\ProgramData\Anaconda3\envs\dmodel\lib\site-packages\torch\utils\cpp_extension.py", line 1672, in _run_ninja_build
    env=env)
  File "C:\ProgramData\Anaconda3\envs\dmodel\lib\subprocess.py", line 438, in run
    output=stdout, stderr=stderr)
subprocess.CalledProcessError: Command '['ninja', '-v']' returned non-zero exit status 1.

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "train.py", line 20, in <module>
    import src.renderutils as ru
  File "I:\Python\Nvidia_Lod\nvdiffmodeling-main\nvdiffmodeling-main\src\renderutils\__init__.py", line 9, in <module>
    from .ops import xfm_points, xfm_vectors, image_loss, prepare_shading_normal, lambert, pbr_specular, pbr_bsdf, _fresnel_shlick, _ndf_ggx, _lambda_ggx, _masking_smith
  File "I:\Python\Nvidia_Lod\nvdiffmodeling-main\nvdiffmodeling-main\src\renderutils\ops.py", line 61, in <module>
    torch.utils.cpp_extension.load(name='renderutils_plugin', sources=source_paths, extra_ldflags=ldflags, with_cuda=True, verbose=True)
  File "C:\ProgramData\Anaconda3\envs\dmodel\lib\site-packages\torch\utils\cpp_extension.py", line 1092, in load
    keep_intermediates=keep_intermediates)
  File "C:\ProgramData\Anaconda3\envs\dmodel\lib\site-packages\torch\utils\cpp_extension.py", line 1303, in _jit_compile
    is_standalone=is_standalone)
  File "C:\ProgramData\Anaconda3\envs\dmodel\lib\site-packages\torch\utils\cpp_extension.py", line 1408, in _write_ninja_file_and_build_library
    error_prefix=f"Error building extension '{name}'")
  File "C:\ProgramData\Anaconda3\envs\dmodel\lib\site-packages\torch\utils\cpp_extension.py", line 1682, in _run_ninja_build
    raise RuntimeError(message) from e
RuntimeError: Error building extension 'renderutils_plugin'
    opened by TheAlexPG 4
  • about effect Fig.22 and Fig 23

    about effect Fig.22 and Fig 23

    I read your paper and video in detail，about LEARNING MESH AND MATERIALS FROM IMPLICIT SURFACES,How to improve the effect？ Can you give me some suggestions on setting super parameters? I used my own model as the target, started from the sphere you provided, and carried out experiments. I found that the feet and face were always unsatisfactory? There is a big gap with the rendering of snails and elephants mentioned in your paper. can you give me some adivce. image

    opened by Lucklycat 2
  • Texture maps have a lot of noise

    Texture maps have a lot of noise

    Thank you very much for your work！ When I tried the ewer example, I found the following textures： image When you zoom in, you will find many red, yellow and green noises image This is still the case when I import the output obj into blender, but the contrast image rendered from nvdiffast does not show this situation. What is the reason? image

    opened by Lucklycat 2
  • How to improve the optimization effect

    How to improve the optimization effect

    Your work is great. Thank you very much for your openness. I tried to use rendering on my own model, but I encountered the following problems and would like to consult: 1.In the process of rendering, I found that the face of the character, especially the eyes, was not poorly optimized, and the color could not be reflected. Are there any parameters that need to be adjusted? image 2.During rendering, I encountered that. mtl contains a map_ The texture of d cannot be represented in the renderer. How can I solve this problem? image

    Thank you for your reply

    opened by Lucklycat 2
  • Lower training resolution => heavier texture pixelation. But why?

    Lower training resolution => heavier texture pixelation. But why?

    Hi, and thanks for great paper and comprehensive code! Could you please help me understand one thing:

    I'm running configs/dancer_displacement.json. Despite "texture_res" being 2048, these 2048 x 2048 textures "learn" huge pixels at lower "train_res":

    image

    And this isn't just poor interpolation in Blender, but these are actual large solid blocks (of different scale!) in normal_map_opt and kd_map_opt! Here is a part of texture_n.png from the top left experiment, 1:1 scale:

    image

    Why do these maps get more pixelated instead of getting more blurry? I've spent couple days searching the code for the source of this behaviour, and I believe there are only three relevant lines (for now let's consider texture only):

    Interpolation doesn't use 'nearest' method so it shouldn't be the source of big pixels. And because views are always different, rasterization should be a problem too. But probably I'm just missing some rendering subtleties?

    Thanks again!

    opened by shrubb 2
  • msvcrt.lib(loadcfg.obj) : error LNK2001: unresolved external symbol __enclave_config

    msvcrt.lib(loadcfg.obj) : error LNK2001: unresolved external symbol __enclave_config

    hi, Thanks for your project. When I try to run sphere to cow demo, I will report this error: msvcrt.lib(loadcfg.obj) : error LNK2001: unresolved external symbol __enclave_config renderutils_plugin.pyd : fatal error LNK1120: Unresolved external command ninja: build stopped: subcommand failed.

    Is it because of my Nanjia version? my current version is: nanjia:1.10.2 pytorch:1.8.1 torchvision:0.9.1 cudnn:7.6.5 cuda:10.2

    opened by myshiop 2
  • matrix gradients does not flow back when using renderutils xfm_points

    matrix gradients does not flow back when using renderutils xfm_points

    hi, when using renderutils.xfm_points (e.g. in render.py v_pos_clip = ru.xfm_points(mesh.v_pos[None, ...], mtx_in) - and trying to optimize camera pose in mtx_in - gradients does not flow back. Only when using use_python=True it worked.

    opened by maorp 1
  • using camera calibration matrices

    using camera calibration matrices

    Hi, first thanks for sharing this work! I'm trying to solve inverse rendering problem using a photogrammetry dataset which comes with accurate camera calibration mats. i have started from the spot example and i need to convert my intrinsic\extrinsic params to be used in the rastering and cant find an example or a guide to do it. the only hint is that I know it should be converted to openGL and NDC according to nvdiffrast, ill be happy to get an advice, thanks.

    opened by maorp 1
  • some question about the complex material, such as that ray marching snail

    some question about the complex material, such as that ray marching snail

    Hi, Thank you very much for your research. I'm trying to use your framework to approximate complex materials, but the blender complex material dataset I made with my own is not very good, because I encountered many problems with camera and light settings in my own rendering, Is the ray marching snail used in this paper directly rendered with shadertoy, or is the ray marching process written in Python?

    opened by myshiop 1
  • where can i find the Animation and Skinning demo

    where can i find the Animation and Skinning demo

    hi, i am read the papaer--Appearance-Driven Automatic 3D Model Simplification, think it's great. But in the demo,i can not find the Animation and Skinning demo, where can i find it ? I want to study how to do it. please help ! thanks

    opened by Lucklycat 2
  • CUDA out of memory.

    CUDA out of memory.

    RuntimeError: CUDA out of memory. Tried to allocate 130.00 MiB (GPU 0; 8.00 GiB total capacity; 7.11 GiB already allocated; 0 bytes free; 7.30 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation. See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF

    opened by d-sharafeldeen 2
  • How to test on custom .obj file?

    How to test on custom .obj file?

    I am trying to test model simplification on my own .obj. I have created a .mtl file for the .obj but I got an error. How to export .obj file to train the model?

    opened by sanbuddhacharyas 0
  • Unable to execute the program, error during cpp_extension.py running at start

    Unable to execute the program, error during cpp_extension.py running at start 

    (dmodel) D:\NVIDIATools\nvdiffmodeling-main>python train.py --config configs/spot.json
No CUDA runtime is found, using CUDA_HOME='C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.5'
Using C:\Users\redacted\AppData\Local\torch_extensions\torch_extensions\Cache\py36_cpu as PyTorch extensions root...
Detected CUDA files, patching ldflags
Emitting ninja build file C:\Users\redacted\AppData\Local\torch_extensions\torch_extensions\Cache\py36_cpu\renderutils_plugin\build.ninja...
Traceback (most recent call last):
  File "train.py", line 20, in <module>
    import src.renderutils as ru
  File "D:\NVIDIATools\nvdiffmodeling-main\src\renderutils\__init__.py", line 9, in <module>
    from .ops import xfm_points, xfm_vectors, image_loss, prepare_shading_normal, lambert, pbr_specular, pbr_bsdf, _fresnel_shlick, _ndf_ggx, _lambda_ggx, _masking_smith
  File "D:\NVIDIATools\nvdiffmodeling-main\src\renderutils\ops.py", line 61, in <module>
    torch.utils.cpp_extension.load(name='renderutils_plugin', sources=source_paths, extra_ldflags=ldflags, with_cuda=True, verbose=True)
  File "C:\Users\redacted\anaconda3\envs\dmodel\lib\site-packages\torch\utils\cpp_extension.py", line 1136, in load
    keep_intermediates=keep_intermediates)
  File "C:\Users\redacted\anaconda3\envs\dmodel\lib\site-packages\torch\utils\cpp_extension.py", line 1347, in _jit_compile
    is_standalone=is_standalone)
  File "C:\Users\redacted\anaconda3\envs\dmodel\lib\site-packages\torch\utils\cpp_extension.py", line 1445, in _write_ninja_file_and_build_library
    is_standalone=is_standalone)
  File "C:\Users\redacted\anaconda3\envs\dmodel\lib\site-packages\torch\utils\cpp_extension.py", line 1834, in _write_ninja_file_to_build_library
    cuda_flags = common_cflags + COMMON_NVCC_FLAGS + _get_cuda_arch_flags()
  File "C:\Users\redacted\anaconda3\envs\dmodel\lib\site-packages\torch\utils\cpp_extension.py", line 1606, in _get_cuda_arch_flags
    arch_list[-1] += '+PTX'
IndexError: list index out of range

(dmodel) D:\NVIDIATools\nvdiffmodeling-main>

    After installing the prerequisites, only difference to the instruction is CUDA 11.5 (and i changed the version to the pip install line to that same version) Getting the above when attempting to run the example spot cow training.

    Something i'm doing wrong, incompatibility with versions or something else entirely?

    opened by RiversJohn 3
Releases(v1.0)
Owner
NVIDIA Research Projects
NVIDIA Research Projects
GitHub Repository
Pytorch Lightning Distributed Accelerators using Ray

Distributed PyTorch Lightning Training on Ray This library adds new PyTorch Lightning accelerators for distributed training using the Ray distributed

166 Dec 27, 2022
A unofficial pytorch implementation of PAN(PSENet2): Efficient and Accurate Arbitrary-Shaped Text Detection with Pixel Aggregation Network

Efficient and Accurate Arbitrary-Shaped Text Detection with Pixel Aggregation Network Requirements pytorch 1.1+ torchvision 0.3+ pyclipper opencv3 gcc

zhoujun 400 Dec 26, 2022
Trading Strategies for Freqtrade

Freqtrade Strategies Strategies for Freqtrade, developed primarily in a partnership between @werkkrew and @JimmyNixx from the Freqtrade Discord. Use t

Bryan Chain 242 Jan 07, 2023
A Semantic Segmentation Network for Urban-Scale Building Footprint Extraction Using RGB Satellite Imagery

A Semantic Segmentation Network for Urban-Scale Building Footprint Extraction Using RGB Satellite Imagery This repository is the official implementati

Aatif Jiwani 42 Dec 08, 2022
Data-driven reduced order modeling for nonlinear dynamical systems

SSMLearn Data-driven Reduced Order Models for Nonlinear Dynamical Systems This package perform data-driven identification of reduced order model based

Haller Group, Nonlinear Dynamics 27 Dec 13, 2022
Implementation of Online Label Smoothing in PyTorch

Online Label Smoothing Pytorch implementation of Online Label Smoothing (OLS) presented in Delving Deep into Label Smoothing. Introduction As the abst

83 Dec 14, 2022
Learning Skeletal Articulations with Neural Blend Shapes

This repository provides an end-to-end library for automatic character rigging and blend shapes generation as well as a visualization tool. It is based on our work Learning Skeletal Articulations wit

Peizhuo 504 Dec 30, 2022
JASS: Japanese-specific Sequence to Sequence Pre-training for Neural Machine Translation

JASS: Japanese-specific Sequence to Sequence Pre-training for Neural Machine Translation This the repository for this paper. Find extensions of this w

Zhuoyuan Mao 14 Oct 26, 2022
Snapchat-filters-app-opencv-python - Here we used opencv and other inbuilt python modules to create filter application like snapchat

Snapchat like filter App using opencv python Backend : opencv and python Library

2 Jul 19, 2022
Python scripts form performing stereo depth estimation using the HITNET model in ONNX.

ONNX-HITNET-Stereo-Depth-estimation Python scripts form performing stereo depth estimation using the HITNET model in ONNX. Stereo depth estimation on

Ibai Gorordo 30 Nov 08, 2022
Self-Supervised Learning with Data Augmentations Provably Isolates Content from Style

Self-Supervised Learning with Data Augmentations Provably Isolates Content from Style [NeurIPS 2021] Official code to reproduce the results and data p

Yash Sharma 27 Sep 19, 2022
PyTorch Implementation of DiffGAN-TTS: High-Fidelity and Efficient Text-to-Speech with Denoising Diffusion GANs

DiffGAN-TTS - PyTorch Implementation PyTorch implementation of DiffGAN-TTS: High

Keon Lee 157 Jan 01, 2023
Crowd-sourced Annotation of Human Motion.

Motion Annotation Tool Live: https://motion-annotation.humanoids.kit.edu Paper: The KIT Motion-Language Dataset Installation Start by installing all P

Matthias Plappert 4 May 25, 2020
A different spin on dataclasses.

dataklasses Dataklasses is a library that allows you to quickly define data classes using Python type hints. Here's an example of how you use it: from

David Beazley 752 Nov 18, 2022
Deep High-Resolution Representation Learning for Human Pose Estimation

Deep High-Resolution Representation Learning for Human Pose Estimation (accepted to CVPR2019) News If you are interested in internship or research pos

HRNet 167 Dec 27, 2022
DataCLUE: 国内首个以数据为中心的AI测评（含模型分析报告）

DataCLUE: A Benchmark Suite for Data-centric NLP You can get the english version of README. 以数据为中心的AI测评(DataCLUE) 内容导引 章节 描述 简介 介绍以数据为中心的AI测评(DataCLUE

CLUE benchmark 135 Dec 22, 2022
Behind the Curtain: Learning Occluded Shapes for 3D Object Detection

Behind the Curtain: Learning Occluded Shapes for 3D Object Detection Acknowledgement We implement our model, BtcDet, based on [OpenPcdet 0.3.0]. Insta

Qiangeng Xu 163 Dec 19, 2022
pyspark🍒🥭 is delicious，just eat it!😋😋

如何用10天吃掉pyspark？ 🔥 🔥 《10天吃掉那只pyspark》 🚀

lyhue1991 578 Dec 30, 2022
Official PyTorch Code of GrooMeD-NMS: Grouped Mathematically Differentiable NMS for Monocular 3D Object Detection (CVPR 2021)

GrooMeD-NMS: Grouped Mathematically Differentiable NMS for Monocular 3D Object Detection GrooMeD-NMS: Grouped Mathematically Differentiable NMS for Mo

Abhinav Kumar 76 Jan 02, 2023
PyTorch implementation of Pay Attention to MLPs

gMLP PyTorch implementation of Pay Attention to MLPs. Quickstart Clone this repository. git clone https://github.com/jaketae/g-mlp.git Navigate to th

Jake Tae 34 Dec 13, 2022