ADOP: Approximate Differentiable One-Pixel Point Rendering

Related tags

Deep LearningADOP
Overview

ADOP: Approximate Differentiable One-Pixel Point Rendering

Abstract: We present a novel point-based, differentiable neural rendering pipeline for scene refinement and novel view synthesis. The input are an initial estimate of the point cloud and the camera parameters. The output are synthesized images from arbitrary camera poses. The point cloud rendering is performed by a differentiable renderer using multi-resolution one-pixel point rasterization. Spatial gradients of the discrete rasterization are approximated by the novel concept of ghost geometry. After rendering, the neural image pyramid is passed through a deep neural network for shading calculations and hole-filling. A differentiable, physically-based tonemapper then converts the intermediate output to the target image. Since all stages of the pipeline are differentiable, we optimize all of the scene's parameters i.e. camera model, camera pose, point position, point color, environment map, rendering network weights, vignetting, camera response function, per image exposure, and per image white balance. We show that our system is able to synthesize sharper and more consistent novel views than existing approaches because the initial reconstruction is refined during training. The efficient one-pixel point rasterization allows us to use arbitrary camera models and display scenes with well over 100M points in real time.

  • The source code will be published after the paper has been accepted to a conference.

[Full Paper]

Video

Watch the video

Comments
  • Guide on exporting scene from COLMAP

    Guide on exporting scene from COLMAP

    Hi! I have some troubles exporting point cloud from COLMAP to compatible with ADOP format. Here is what I did:

    1. Autoreconstruction in COLMAP (both sparse and dense)
    2. File > Export As > as .ply
    3. preprocessed it with colmap2adop tool
    4. started training
    5. got an error that my point cloud file does not contain normal data
    6. I checked that your .ply files have normal data in headers, but I struggle to make COLMAP include it in mine
    opened by DenShlk 22
  • RAM usage when training dataset

    RAM usage when training dataset

    Through a lot of trial and error I managed to get both the ADOP viewer and trainer set up on Ubuntu 20.04.

    SETUP I'm trying to run my own dataset through the trainer to see how it is. I only have an Nvidia GTX 1070 card with 8GB of VRAM, 16GB of physical RAM but plenty of hard disk space. I've gotten to the point that the trainer runs without crashing. My dataset is 175 images, downscaled to 1000x562, converted from a ~7M point-cloud project in COLMAP.

    PROBLEM However, running the trainer seems to take up over 1GB per epoch on top of the base amount. This quickly sends my RAM usage to near full. Luckily I have a large swap partition but that too fills up over hundreds of epochs. Is this expected behaviour or a memory leak? Or could it be because my install or conda environment might be faulty? Or is it my incapable hardware? In any case, once all my swap space gets filled the program is unfortunately killed on its own.

    If tweaking any of the training config or dataset parameters could potentially fix or mitigate this issue, please do let me know. Thank you for your work on this, I find this to be a very interesting project.

    opened by ShinyLuxray 19
  • How to optimize for rendering

    How to optimize for rendering

    Hi I have a question regarding the adop_viewer. l tried the adop_viewer with pretrained models on tanks and temples dataset, it works perfectly on a laptop with a 3080 GPU (8GB VRAM) on all scenes.

    I guess you are using the 40GB A100 ini files for training these models, so I also tried training my data on a GPU with similar VRAM using exact settings from train_tank_and_temples_multi.ini. However, when I attempt to visualize my checkpoints on the 3080 GPU it shows CUDA out of memory for all my scenes.

    My question is: for rendering, did you make some optimizations to make the trained models fit in a GPU with less VRAM? Or is there certain configurations/arguments that allows the adop_viewer to take a model trained with high-end GPUs and render on low-end GPUs? Thanks in advance!

    opened by qiaosongwang 12
  • Issue with multi-camera COLMAP dataset

    Issue with multi-camera COLMAP dataset

    I have successfully trained with ADOP on scenes that use a single camera. I am having difficulty, however, with a larger dataset comprised of 882 cameras, 2/3 of which are from drone footage, the other 1/3 being a hand-held DSLR.

    COLMAP successfully generates the camera registrations and point cloud, and running colmap2adop correctly creates all the necessary files for training. I am noticing, however, that COLMAP resizes every single image, effectively generating 882 individual camera profiles. I thought at first this wouldn't necessarily be an issue, as adop_train can parse multiple cameras. I am running into this error, though:

    Assertion 'cam.w == image_size_input(0)' failed!
  File: /home/visionarymind/Downloads/ADOP/src/lib/data/Dataset.cpp:49
  Function: SceneDataTrainSampler::SceneDataTrainSampler(std::shared_ptr<SceneData>, std::vector<int>, bool, Saiga::ivec2, int, bool)
Aborted (core dumped)

    I have looked at each camera.ini file, and the dimensions match each image. Would you have any idea why this error would be thrown and any way to work around it? Technically, there were only two cameras, but COLMAP resizes the images during registration.

    opened by VisionaryMind 10
  • Something worry in PointRenderer.cu & PointRenderer.h

    Something worry in PointRenderer.cu & PointRenderer.h

    2021-11-10 15-59-34屏幕截图 Can not Build CUDA object src/lib/CMakeFiles/NeuralPoints.dir/rendering/PointRenderer.cu.o

    /home/xgenie/projectfold/ADOP/src/lib/./config.h(25): error: inline specifier allowed on function declarations only

    /home/xgenie/projectfold/ADOP/src/lib/./config.h(25): error: inline specifier allowed on function declarations only

    /home/xgenie/projectfold/ADOP/src/lib/rendering/PointRenderer.h(37): error: qualified name is not allowed

    /home/xgenie/projectfold/ADOP/src/lib/rendering/PointRenderer.h(39): error: Function is not a template

    /home/xgenie/projectfold/ADOP/src/lib/rendering/PointRenderer.h(39): error: not a class or struct name

    /home/xgenie/projectfold/ADOP/src/lib/rendering/PointRenderer.h(42): error: identifier "variable_list" is undefined

    /home/xgenie/projectfold/ADOP/src/lib/rendering/PointRenderer.h(42): error: identifier "AutogradContext" is undefined

    /home/xgenie/projectfold/ADOP/src/lib/rendering/PointRenderer.h(42): error: identifier "Variable" is undefined

    /home/xgenie/projectfold/ADOP/src/lib/rendering/PointRenderer.h(42): error: identifier "Variable" is undefined

    /home/xgenie/projectfold/ADOP/src/lib/rendering/PointRenderer.h(42): error: identifier "Variable" is undefined

    /home/xgenie/projectfold/ADOP/src/lib/rendering/PointRenderer.h(43): error: identifier "Variable" is undefined

    /home/xgenie/projectfold/ADOP/src/lib/rendering/PointRenderer.h(43): error: identifier "Variable" is undefined

    /home/xgenie/projectfold/ADOP/src/lib/rendering/PointRenderer.h(43): error: identifier "IValue" is undefined

    /home/xgenie/projectfold/ADOP/src/lib/rendering/PointRenderer.h(45): error: identifier "variable_list" is undefined

    /home/xgenie/projectfold/ADOP/src/lib/rendering/PointRenderer.h(45): error: identifier "AutogradContext" is undefined

    /home/xgenie/projectfold/ADOP/src/lib/rendering/PointRenderer.h(45): error: identifier "variable_list" is undefined

    /home/xgenie/projectfold/ADOP/src/lib/rendering/PointRenderer.cu(422): error: expected an identifier

    /home/xgenie/projectfold/ADOP/src/lib/rendering/PointRenderer.cu(422): error: identifier "K" is undefined

    /home/xgenie/projectfold/ADOP/src/lib/rendering/PointRenderer.cu(422): error: expected a "]"

    /home/xgenie/projectfold/ADOP/src/lib/rendering/PointRenderer.cu(422): warning: nonstandard use of "auto" to both deduce the type from an initializer and to announce a trailing return type

    /home/xgenie/projectfold/ADOP/src/lib/rendering/PointRenderer.cu(422): error: cannot deduce "auto" type (initializer required)

    /home/xgenie/projectfold/ADOP/src/lib/rendering/PointRenderer.cu(422): error: expected a ";"

    /home/xgenie/projectfold/ADOP/src/lib/rendering/PointRenderer.cu(429): error: expected an identifier

    /home/xgenie/projectfold/ADOP/src/lib/rendering/PointRenderer.cu(429): error: identifier "aff" is undefined

    /home/xgenie/projectfold/ADOP/src/lib/rendering/PointRenderer.cu(429): error: expected a "]"

    /home/xgenie/projectfold/ADOP/src/lib/rendering/PointRenderer.cu(429): warning: nonstandard use of "auto" to both deduce the type from an initializer and to announce a trailing return type

    /home/xgenie/projectfold/ADOP/src/lib/rendering/PointRenderer.cu(429): error: cannot deduce "auto" type (initializer required)

    /home/xgenie/projectfold/ADOP/src/lib/rendering/PointRenderer.cu(429): error: expected a ";"

    /home/xgenie/projectfold/ADOP/src/lib/rendering/PointRenderer.cu(621): error: expected an identifier

    /home/xgenie/projectfold/ADOP/src/lib/rendering/PointRenderer.cu(621): error: identifier "K" is undefined

    /home/xgenie/projectfold/ADOP/src/lib/rendering/PointRenderer.cu(621): error: expected a "]"

    /home/xgenie/projectfold/ADOP/src/lib/rendering/PointRenderer.cu(621): warning: nonstandard use of "auto" to both deduce the type from an initializer and to announce a trailing return type

    /home/xgenie/projectfold/ADOP/src/lib/rendering/PointRenderer.cu(621): error: cannot deduce "auto" type (initializer required)

    /home/xgenie/projectfold/ADOP/src/lib/rendering/PointRenderer.cu(621): error: expected a ";"

    /home/xgenie/projectfold/ADOP/src/lib/rendering/PointRenderer.cu(622): error: expected an identifier

    /home/xgenie/projectfold/ADOP/src/lib/rendering/PointRenderer.cu(622): error: identifier "g_point" is undefined

    /home/xgenie/projectfold/ADOP/src/lib/rendering/PointRenderer.cu(622): error: expected a "]"

    /home/xgenie/projectfold/ADOP/src/lib/rendering/PointRenderer.cu(622): warning: nonstandard use of "auto" to both deduce the type from an initializer and to announce a trailing return type

    /home/xgenie/projectfold/ADOP/src/lib/rendering/PointRenderer.cu(622): error: cannot deduce "auto" type (initializer required)

    /home/xgenie/projectfold/ADOP/src/lib/rendering/PointRenderer.cu(622): error: expected a ";"

    /home/xgenie/projectfold/ADOP/src/lib/rendering/PointRenderer.cu(686): error: expected an identifier

    /home/xgenie/projectfold/ADOP/src/lib/rendering/PointRenderer.cu(686): error: identifier "aff" is undefined

    /home/xgenie/projectfold/ADOP/src/lib/rendering/PointRenderer.cu(686): error: expected a "]"

    /home/xgenie/projectfold/ADOP/src/lib/rendering/PointRenderer.cu(686): warning: nonstandard use of "auto" to both deduce the type from an initializer and to announce a trailing return type

    /home/xgenie/projectfold/ADOP/src/lib/rendering/PointRenderer.cu(686): error: cannot deduce "auto" type (initializer required)

    /home/xgenie/projectfold/ADOP/src/lib/rendering/PointRenderer.cu(686): error: expected a ";"

    /home/xgenie/projectfold/ADOP/src/lib/rendering/PointRenderer.cu(687): error: expected an identifier

    /home/xgenie/projectfold/ADOP/src/lib/rendering/PointRenderer.cu(687): error: identifier "g_point" is undefined

    /home/xgenie/projectfold/ADOP/src/lib/rendering/PointRenderer.cu(687): error: expected a "]"

    /home/xgenie/projectfold/ADOP/src/lib/rendering/PointRenderer.cu(687): warning: nonstandard use of "auto" to both deduce the type from an initializer and to announce a trailing return type

    /home/xgenie/projectfold/ADOP/src/lib/rendering/PointRenderer.cu(687): error: cannot deduce "auto" type (initializer required)

    /home/xgenie/projectfold/ADOP/src/lib/rendering/PointRenderer.cu(687): error: expected a ";"

    /home/xgenie/projectfold/ADOP/src/lib/rendering/PointRenderer.cu(527): warning: variable "iw" was declared but never referenced

    /home/xgenie/projectfold/ADOP/src/lib/rendering/PointRenderer.cu(615): warning: variable "cam2" was set but never used

    opened by XGenietony 10
  • Unable to get output in the neural view

    Unable to get output in the neural view

    Hello, thanks for releasing your code!

    I tried running the program (the compilation was successful) but I am unable to obtain any output in the neural view.

    image

    This is the output I get from the terminal. I am using Ubuntu 20.04.3 LTS.

    (base) [email protected]:~/Documents/ADOP$ ./build/bin/adop_viewer --scene_dir scenes/tt_lighthouse/
register neural render info
Ref. 
=============== Saiga ===============
| Saiga Version     1.3.2           |
| Eigen Version     3.3.91          |
| Compiler          GNU             |
|   -> Version      9.3.0           |
| Build Type        RelWithDebInfo  |
| Debug             0               |
| Eigen Debug       0               |
| ASAN              0               |
| Asserts           1               |
| Optimizations     0               |
=====================================
Initializing GLFW.
Initializing GLFW sucessfull!
Creating GLFW Window. 1920x1080 Mode=3 Fullscreen=1 Borderless=0
=========================== OpenGL ===========================
| OpenGL Version    3.2.0 NVIDIA 495.29.05                   | 
| GLSL Version      1.50 NVIDIA via Cg compiler              | 
| Renderer          NVIDIA GeForce GTX 1660 SUPER/PCIe/SSE2  | 
| Vendor            NVIDIA Corporation                       | 
==============================================================
[Renderer] Target resized to 1848x1016
loading shader /home/jeremy/Documents/ADOP/External/saiga/shader/imgui_gl.glsl
loading shader /home/jeremy/Documents/ADOP/External/saiga/shader/tone_map.glsl
loading shader /home/jeremy/Documents/ADOP/External/saiga/shader/tone_map_linear.glsl
loading shader /home/jeremy/Documents/ADOP/External/saiga/shader/compute/bloom_extract_bright.glsl
loading shader /home/jeremy/Documents/ADOP/External/saiga/shader/compute/bloom_downsample.glsl
loading shader /home/jeremy/Documents/ADOP/External/saiga/shader/compute/bloom_upsample.glsl
loading shader /home/jeremy/Documents/ADOP/External/saiga/shader/compute/bloom_combine_simple.glsl
loading shader /home/jeremy/Documents/ADOP/External/saiga/shader/compute/copy_image.glsl
loading shader /home/jeremy/Documents/ADOP/External/saiga/shader/compute/compute_blur.glsl
loading shader /home/jeremy/Documents/ADOP/External/saiga/shader/compute/compute_blur.glsl
loading shader /home/jeremy/Documents/ADOP/External/saiga/shader/lighting/light_directional.glsl
loading shader /home/jeremy/Documents/ADOP/External/saiga/shader/lighting/light_directional.glsl
loading shader /home/jeremy/Documents/ADOP/External/saiga/shader/lighting/light_point.glsl
loading shader /home/jeremy/Documents/ADOP/External/saiga/shader/lighting/light_point.glsl
loading shader /home/jeremy/Documents/ADOP/External/saiga/shader/lighting/light_spot.glsl
loading shader /home/jeremy/Documents/ADOP/External/saiga/shader/lighting/light_spot.glsl
loading shader /home/jeremy/Documents/ADOP/External/saiga/shader/lighting/light_point.glsl
loading shader /home/jeremy/Documents/ADOP/External/saiga/shader/lighting/light_spot.glsl
loading shader /home/jeremy/Documents/ADOP/External/saiga/shader/lighting/stenciltest.glsl
loading shader /home/jeremy/Documents/ADOP/External/saiga/shader/post_processing/post_processing.glsl
loading shader /home/jeremy/Documents/ADOP/External/saiga/shader/lighting/blitDepth.glsl
Deferred Renderer initialized. Render resolution: 1848x1016
loading shader /home/jeremy/Documents/ADOP/External/saiga/shader/post_processing/imagedisplay.glsl
Program Initialized!
Loading Scene scenes/tt_lighthouse/
====================================
Scene Loaded
  Name       tt_lighthouse
  Path       /home/jeremy/Documents/ADOP/scenes/tt_lighthouse
  Image Size 2048x1080
  Aspect     1.8963
  K          1666.68 1661.18 1024    540     0      
  ocam       2048x1080 affine(1, 0, 0, 0, 0) cam2world() world2cam()
  ocam cut   1
  normalized center 0 0
  dist       -0.126731   0.0217503   0           0           0           0           0.000406465 0.000123456
  Points     12313620
  Colors     1
  Normals    1
  Num Images 309
  Num Cameras 1
====================================
loading shader /home/jeremy/Documents/ADOP/External/saiga/shader/asset/ColoredAsset.glsl
loading shader /home/jeremy/Documents/ADOP/External/saiga/shader/asset/ColoredAsset.glsl
loading shader /home/jeremy/Documents/ADOP/External/saiga/shader/asset/ColoredAsset.glsl
loading shader /home/jeremy/Documents/ADOP/External/saiga/shader/asset/LineVertexColoredAsset.glsl
loading shader /home/jeremy/Documents/ADOP/External/saiga/shader/asset/LineVertexColoredAsset.glsl
loading shader /home/jeremy/Documents/ADOP/External/saiga/shader/asset/LineVertexColoredAsset.glsl
loading shader /home/jeremy/Documents/ADOP/External/saiga/shader/tone_map.glsl
loading shader /home/jeremy/Documents/ADOP/External/saiga/shader/tone_map_linear.glsl
loading shader /home/jeremy/Documents/ADOP/External/saiga/shader/compute/bloom_extract_bright.glsl
loading shader /home/jeremy/Documents/ADOP/External/saiga/shader/compute/bloom_downsample.glsl
loading shader /home/jeremy/Documents/ADOP/External/saiga/shader/compute/bloom_upsample.glsl
loading shader /home/jeremy/Documents/ADOP/External/saiga/shader/compute/bloom_combine_simple.glsl
loading shader /home/jeremy/Documents/ADOP/External/saiga/shader/compute/copy_image.glsl
loading shader /home/jeremy/Documents/ADOP/External/saiga/shader/compute/compute_blur.glsl
loading shader /home/jeremy/Documents/ADOP/External/saiga/shader/compute/compute_blur.glsl
Found experiment experiments//2021-10-30_09:17:38_multi_scene_pose/ with 1 epochs
Found experiment experiments//2021-10-30_09:14:22_boat/ with 1 epochs
loading checkpoint 2021-10-30_09:17:38_multi_scene_pose -> ep0400
Load Checkpoint render
Total Model Params: 574651
> Reset cuda cache
GPU memory - Point Cloud: 295.527MB
Pinhole Intrinsics:
Tensor [1, 13] float cuda:0 Min/Max -0.126731 1666.68 Mean 376.289 Sum 4891.75 req-grad 1
GPU memory - Texture: 197.018MB
Load Checkpoint points 12313620 max uv: 12313619
Load Checkpoint texture. Texels: 12313620 Channels: 4
Load Checkpoint pose
First pose before SE3(Quatwxyz(0.990142,-0.0887136,0.107975,0.00948205),Vec3(0.115149,0.198961,4.53537))
First pose after SE3(Quatwxyz(0.990146,-0.0887041,0.107948,0.00948975),Vec3(0.114891,0.198873,4.53539))
Load Checkpoint intrinsics
Load Checkpoint environment_map
Load Checkpoint vignette
Load Checkpoint response
Load Checkpoint exposures_values
Using Adam texture optimzier
optimizing texture with lr 0.08/0.004
optimizing environment_map with lr 0.02
optimizing response with lr 0.001
optimizing exposure with lr 0.0005
optimizing vignette with lr 5e-06
optimizing poses with lr 0.005
GPU memory - Texture: 197.018MB
Current Best (img,cam) = (0,0) EV: -0.239136
[Renderer] Target resized to 733x354
[DeferredRenderer] Resize 733x354
loading shader shader//point_render.glsl
[Renderer] Target resized to 545x249
[DeferredRenderer] Resize 545x249
loading checkpoint 2021-10-30_09:17:38_multi_scene_pose -> ep0400
Load Checkpoint render
Total Model Params: 574651
> Reset cuda cache
GPU memory - Point Cloud: 295.527MB
Pinhole Intrinsics:
Tensor [1, 13] float cuda:0 Min/Max -0.126731 1666.68 Mean 376.289 Sum 4891.75 req-grad 1
GPU memory - Texture: 197.018MB
Load Checkpoint points 12313620 max uv: 12313619
Load Checkpoint texture. Texels: 12313620 Channels: 4
Load Checkpoint pose
First pose before SE3(Quatwxyz(0.990146,-0.0887041,0.107948,0.00948975),Vec3(0.114891,0.198873,4.53539))
First pose after SE3(Quatwxyz(0.990146,-0.0887041,0.107948,0.00948975),Vec3(0.114891,0.198873,4.53539))
Load Checkpoint intrinsics
Load Checkpoint environment_map
Load Checkpoint vignette
Load Checkpoint response
Load Checkpoint exposures_values
Using Adam texture optimzier
optimizing texture with lr 0.08/0.004
optimizing environment_map with lr 0.02
optimizing response with lr 0.001
optimizing exposure with lr 0.0005
optimizing vignette with lr 5e-06
optimizing poses with lr 0.005
GPU memory - Texture: 197.018MB
Current Best (img,cam) = (0,0) EV: -0.239136

    Any help would be greatly appreciated! Thank you!

    opened by mickey1356 9
  • A question about CmakeList.txt

    A question about CmakeList.txt

    Hello, I am a beginner of neural network. I want to run your program on my computer, but when I Compiling ADOP,in the command "cmake -DCMAKE_PREFIX_PATH="${CONDA}/lib/python3.8/site-packages/torch/;${CONDA}" ..",there was an error,it was "The source directory XXXXdoes not appear to contain CMakeLists.txt."How can i solve this problem?Should i write the CmakeList on myself ?Is there sth wrong in other parts ?this may easy for most people,but i sincerely hope you can answer me,thank you!

    opened by Lu-pin-an 9
  • Getting

    Getting "Loss not finite :(".

    Hello, I'm preparing a dataset of the interior of an appartment. I checked and COLMAP seems to work well, despite that there is a fair amount of white walls. After a random number of epochs, I'm getting the following (pasting only the tail of the output):

    === Epoch 81 ===

Train 81 |   0% |                              |    0/1016 [00:00:0000] [0.00 e/s] 
Train 81 |   0% |                              |    0/1016 [00:00:0000] [0.00 e/s] 
Train 81 |   4% |#                             |   40/1016 [00:05:0000] [8.00 e/s]  Cur=212.233490 Avg=302.391327
Train 81 |   9% |##                            |   88/1016 [00:10:0000] [8.80 e/s]  Cur=333.448730 Avg=287.600220
Train 81 |  13% |####                          |  136/1016 [00:15:0000] [9.07 e/s]  Cur=441.840668 Avg=295.512512
Train 81 |  18% |#####                         |  184/1016 [00:20:0000] [9.20 e/s]  Cur=379.657135 Avg=317.960571
Train 81 |  22% |######                        |  224/1016 [00:25:0000] [8.96 e/s]  Cur=277.145172 Avg=296.512878
Train 81 |  27% |########                      |  272/1016 [00:30:0000] [9.07 e/s]  Cur=310.707367 Avg=302.186951
Train 81 |  31% |#########                     |  320/1016 [00:35:0000] [9.14 e/s]  Cur=34.233173 Avg=287.412201
Train 81 |  36% |##########                    |  368/1016 [00:40:0001] [9.20 e/s]  Cur=33.885960 Avg=282.051544
Train 81 |  41% |############                  |  416/1016 [00:45:0001] [9.24 e/s]  Cur=267.568176 Avg=272.566742
Train 81 |  46% |#############                 |  464/1016 [00:50:0001] [9.28 e/s]  Cur=376.509857 Avg=253.497742
Train 81 |  50% |###############               |  512/1016 [00:55:0001] [9.31 e/s]  Cur=347.692017 Avg=255.870377
Train 81 |  55% |################              |  560/1016 [01:00:0001] [9.33 e/s]  Cur=222.076523 Avg=251.570724
Train 81 |  60% |#################             |  608/1016 [01:05:0001] [9.35 e/s]  Cur=184.181763 Avg=259.685944
Train 81 |  65% |###################           |  656/1016 [01:10:0002] [9.37 e/s]  Cur=30.161480 Avg=255.755127
Train 81 |  69% |####################          |  704/1016 [01:15:0002] [9.39 e/s]  Cur=34.554646 Avg=257.362579
Train 81 |  74% |######################        |  752/1016 [01:20:0002] [9.40 e/s]  Cur=239.453049 Avg=248.403091
Train 81 |  78% |#######################       |  792/1016 [01:25:0002] [9.32 e/s]  Cur=33.262161 Avg=248.473404
Train 81 |  83% |########################      |  840/1016 [01:30:0002] [9.33 e/s]  Cur=348.131836 Avg=255.551346
Train 81 |  87% |##########################    |  888/1016 [01:35:0003] [9.35 e/s]  Cur=547.986633 Avg=261.241058
Train 81 |  92% |###########################   |  936/1016 [01:40:0003] [9.36 e/s]  Cur=268.251221 Avg=261.505127
Tensor [8, 4, 512, 512] float cuda:0 Min/Max 1.3125e-06 20599.7 Mean 0.520595 Sum 4.36707e+06 sdev 7.16297 req-grad 1
Tensor [8, 4, 256, 256] float cuda:0 Min/Max 3.05836e-05 34.5411 Mean 0.535766 Sum 1.12358e+06 sdev 0.771836 req-grad 1
Tensor [8, 4, 128, 128] float cuda:0 Min/Max 1.22896e-05 33.5968 Mean 0.552693 Sum 289770 sdev 0.658228 req-grad 1
Tensor [8, 4, 64, 64] float cuda:0 Min/Max 0.000194362 36.972 Mean 0.557636 Sum 73090.4 sdev 0.61885 req-grad 1

Tensor [8, 2, 512, 512] float cuda:0 Min/Max -1 0.999414 Mean 0.083806 Sum 351508 sdev 0.473772 req-grad 0
Tensor [1] float cuda:0 Min/Max nan nan Mean nan Sum nan sdev nan req-grad 1
Tensor [8, 3, 512, 512] float cuda:0 Min/Max nan nan Mean nan Sum nan sdev nan req-grad 1
Tensor [1, 1, 512, 512] float cuda:0 Min/Max 0 1 Mean 0.878906 Sum 230400 sdev 0.326237 req-grad 0
Tensor [8, 3, 512, 512] float cuda:0 Min/Max 0 0.956863 Mean 0.653581 Sum 4.11197e+06 sdev 0.274293 req-grad 0

Scene:
Scene Log - Texture: Tensor [4, 11655018] float cuda:0 Min/Max -22.6205 15.4725 Mean -0.8064 Sum -3.75944e+07 sdev 1.10892 req-grad 1
Background Desc:  -0.970885 -0.863956 -0.900406 -0.882389 
Environment map: Tensor [1, 4, 1024, 512] float cuda:0 Min/Max -6.2707 3.55126 Mean -0.982767 Sum -2.06101e+06 sdev 0.274843 req-grad 1
Poses: Tensor [254, 8] double cuda:0 Min/Max -6.83462 8.02766 Mean 0.196939 Sum 400.181 sdev 1.62054 req-grad 0
Vignette params:  -0.0149647 -0.00850714 -0.00492687 |   0.00124693 -0.000540384
terminate called after throwing an instance of 'std::runtime_error'
  what():  Loss not finite :(

    Could it be due to the lack of enough points on the white walls? If so, how would you go about it? I'm willing to provide any data if required to better assess this.

    opened by cduguet 8
  • Something wrong with the neural view using my own data

    Something wrong with the neural view using my own data

    After using the ADOP viewer with 2 kinds of my trained data, I got the following results.

    2022-06-29 10-50-44 的屏幕截图

    2022-06-29 11-08-36 的屏幕截图

    All neural views of them are definitely wrong. Is there the same problem with me?

    By the way, when I used the ADOP viewer with the scene of "boat", there is a correct colored display in the neural view.

    opened by HandsomeFYM 5
  • Strange random color when training

    Strange random color when training

    Hi，I have some doubt of my train result on other dataset. It looks like below:( left picture is the train result, right is ground trurh) image

    I can not find out why it happens, is it caused by the rasterization method?

    Below show some of the rasterization result:( left: train render result, medium: rgb ground truth, right: rasterization mask) image

    opened by Sylvia6 4
  • How to generate a video from a given trajectory?

    How to generate a video from a given trajectory?

    Hi, I saw you provided some videos for the trained models on the different scenes. I was wondering how you generated them? Is it possible to generate them with the viewer?

    opened by vahidEttehadi 4
  • How to use the mask function？

    How to use the mask function？

    Hi, I'm wondering to use the mask to filter out the dynamic objects in my scene. But how to use the mask function？I have set the mask_dir in the dataset.ini? Should I also specify the classifaction_dir in the dataset.ini？

    opened by Gatsby23 0
  • can't move in vr_viewer using oculus quest 2

    can't move in vr_viewer using oculus quest 2

    Thank you for the excellent work. VR is being tested.

    The build is successful, but moving the VR headset does not change the rendering screen.

    If there is a solution, please share it with me.

    opened by minsoo5403 1
  • fail to run ./build/bin/adop_viewer --scene_dir scenes/boat

    fail to run ./build/bin/adop_viewer --scene_dir scenes/boat

    optimizing texture with lr 0.01/0.0002 optimizing environment_map with lr 0.002 optimizing response with lr 0.001 optimizing vignette with lr 5e-06 optimizing poses with lr 0.005 signal 11 (Segmentation fault), address is 0x31f from 0x7fbbb8c33eb5 [bt]: (0) /host/home//G/1workspace/NeRF-space/ADOP/build/External/saiga/lib/libsaiga_core.so(_ZN5Saiga17printCurrentStackEv+0x38) [0x7fbbb7c3b468] [bt]: (1) /host/home//G/1workspace/NeRF-space/ADOP/build/External/saiga/lib/libsaiga_core.so(+0xe5871) [0x7fbbb7c3b871] [bt]: (2) /usr/lib/x86_64-linux-gnu/libpthread.so.0(+0x14420) [0x7fbba6312420] [bt]: (3) /lib64/ld-linux-x86-64.so.2(+0x17eb5) [0x7fbbb8c33eb5] [bt]: (4) /usr/lib/x86_64-linux-gnu/libc.so.6(_dl_catch_exception+0x88) [0x7fbb96bf0928] [bt]: (5) /usr/lib/x86_64-linux-gnu/libc.so.6(_dl_catch_error+0x33) [0x7fbb96bf09f3] [bt]: (6) /usr/lib/x86_64-linux-gnu/libdl.so.2(+0x1b59) [0x7fbb96947b59] [bt]: (7) /usr/lib/x86_64-linux-gnu/libdl.so.2(dlclose+0x28) [0x7fbb96947468] [bt]: (8) /lib/x86_64-linux-gnu/libcuda.so(+0x217ec1) [0x7fbb31383ec1] [bt]: (9) /lib/x86_64-linux-gnu/libcuda.so(+0x217f33) [0x7fbb31383f33] [bt]: (10) /lib/x86_64-linux-gnu/libcuda.so(+0x2f42a2) [0x7fbb314602a2] [bt]: (11) /opt/conda/envs/adop/lib/libcudart.so.11.0(+0x1b408) [0x7fbbb6553408] [bt]: (12) /opt/conda/envs/adop/lib/libcudart.so.11.0(cudaGraphicsGLRegisterImage+0x1d1) [0x7fbbb65a34c1] [bt]: (13) /host/home//G/1workspace/NeRF-space/ADOP/build/bin/libNeuralPoints.so(_ZN5Saiga4CUDA7Interop9initImageEjN2gl6GLenumE+0x32) [0x7fbbb8b126b2] [bt]: (14) /host/home//G/1workspace/NeRF-space/ADOP/build/bin/libNeuralPoints.so(_ZN16RealTimeRenderer6RenderE9ImageInfo+0x78b) [0x7fbbb8affd2b] [bt]: (15) /host/home//G/1workspace/NeRF-space/ADOP/build/bin/libNeuralPoints.so(_ZN16RealTimeRenderer7ForwardEPN5Saiga6CameraE9ImageInfo+0x32b) [0x7fbbb8b03f5b] [bt]: (16) ./build/bin/adop_viewer(+0x21e07) [0x55e456e21e07] [bt]: (17) /host/home//G/1workspace/NeRF-space/ADOP/build/External/saiga/lib/libsaiga_opengl.so(_ZN5Saiga14OpenGLRenderer12PrepareImguiEb+0x160) [0x7fbbb87e7510] [bt]: (18) /host/home//G/1workspace/NeRF-space/ADOP/build/External/saiga/lib/libsaiga_opengl.so(_ZN5Saiga14OpenGLRenderer6renderERKNS_10RenderInfoE+0x5b) [0x7fbbb87e834b] [bt]: (19) /host/home//G/1workspace/NeRF-space/ADOP/build/External/saiga/lib/libsaiga_core.so(_ZN5Saiga10WindowBase6renderEv+0x4c) [0x7fbbb7c1745c] [bt]: (20) /host/home//G/1workspace/NeRF-space/ADOP/build/External/saiga/lib/libsaiga_core.so(_ZN5Saiga8MainLoop6renderEff+0x7a) [0x7fbbb7c1561a] [bt]: (21) /host/home//G/1workspace/NeRF-space/ADOP/build/External/saiga/lib/libsaiga_core.so(_ZN5Saiga8MainLoop13startMainLoopENS_18MainLoopParametersE+0x25c) [0x7fbbb7c159fc] [bt]: (22) /host/home//G/1workspace/NeRF-space/ADOP/build/External/saiga/lib/libsaiga_core.so(_ZN5Saiga10WindowBase13startMainLoopENS_18MainLoopParametersE+0x2d) [0x7fbbb7c1788d] [bt]: (23) ./build/bin/adop_viewer(+0x1adfa) [0x55e456e1adfa] [bt]: (24) /usr/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf3) [0x7fbb96ab4083] [bt]: (25) ./build/bin/adop_viewer(+0x1b1de) [0x55e456e1b1de] Assertion '0' failed! File: /host/home//G/1workspace/NeRF-space/ADOP/External/saiga/src/saiga/core/util/crash.cpp:109 Function: void Saiga::crit_err_hdlr(int, siginfo_t*, void*) Aborted (core dumped)

    opened by hongdayu 1
  • python setup.py install gives us an error

    python setup.py install gives us an error

    Building wheel torch-1.10.0a0+git302ee7b Traceback (most recent call last): File "setup.py", line 309, in cmake = CMake() File "/root/autodl-tmp/ADOP/External/pytorch/tools/setup_helpers/cmake.py", line 102, in init self._cmake_command = CMake._get_cmake_command() File "/root/autodl-tmp/ADOP/External/pytorch/tools/setup_helpers/cmake.py", line 126, in _get_cmake_command elif cmake is not None and CMake._get_version(cmake) >= distutils.version.LooseVersion("3.10.0"): File "/root/autodl-tmp/ADOP/External/pytorch/tools/setup_helpers/cmake.py", line 137, in _get_version return distutils.version.LooseVersion(line.strip().split(' ')[2]) AttributeError: module 'distutils' has no attribute 'version'

    opened by CHenYYff 0
  • make -j10 Failed

    make -j10 Failed

    image Thanks for your excellent work! But when I compile the adop project using "make -j10", it seems like meet errors. I'm following the steps of "compile instructions". Please help me out! Thanks~

    opened by Yukun66 0
Releases(v1.0)
Owner
Darius Rückert
Darius Rückert
GitHub Repository
Codes accompanying the paper "Believe What You See: Implicit Constraint Approach for Offline Multi-Agent Reinforcement Learning" (NeurIPS 2021 Spotlight

Implicit Constraint Q-Learning This is a pytorch implementation of ICQ on Datasets for Deep Data-Driven Reinforcement Learning (D4RL) and ICQ-MA on SM

42 Dec 23, 2022
SSL_SLAM2: Lightweight 3-D Localization and Mapping for Solid-State LiDAR (mapping and localization separated) ICRA 2021

SSL_SLAM2 Lightweight 3-D Localization and Mapping for Solid-State LiDAR (Intel Realsense L515 as an example) This repo is an extension work of SSL_SL

Wang Han 王晗 1.3k Jan 08, 2023
Prototypical Networks for Few shot Learning in PyTorch

Prototypical Networks for Few shot Learning in PyTorch Simple alternative Implementation of Prototypical Networks for Few Shot Learning (paper, code)

Orobix 835 Jan 08, 2023
Python-kafka-reset-consumergroup-offset-example - Python Kafka reset consumergroup offset example

Python Kafka reset consumergroup offset example This is a simple example of how

Willi Carlsen 1 Feb 16, 2022
Official pytorch code for SSAT: A Symmetric Semantic-Aware Transformer Network for Makeup Transfer and Removal

SSAT: A Symmetric Semantic-Aware Transformer Network for Makeup Transfer and Removal This is the official pytorch code for SSAT: A Symmetric Semantic-

ForeverPupil 57 Dec 13, 2022
This is the official Pytorch implementation of "Lung Segmentation from Chest X-rays using Variational Data Imputation", Raghavendra Selvan et al. 2020

README This is the official Pytorch implementation of "Lung Segmentation from Chest X-rays using Variational Data Imputation", Raghavendra Selvan et a

Raghav 42 Dec 15, 2022
Post-training Quantization for Neural Networks with Provable Guarantees

Post-training Quantization for Neural Networks with Provable Guarantees Authors: Jinjie Zhang ( Yixuan Zhou 2 Nov 29, 2022

A web application that provides real time temperature and humidity readings of a house.

About A web application which provides real time temperature and humidity readings of a house. If you're interested in the data collected so far click

Ben Thompson 3 Jan 28, 2022
This code is the implementation of the paper "Coherence-Based Distributed Document Representation Learning for Scientific Documents".

Introduction This code is the implementation of the paper "Coherence-Based Distributed Document Representation Learning for Scientific Documents". If

tsc 0 Jan 11, 2022
Automatically Build Multiple ML Models with a Single Line of Code. Created by Ram Seshadri. Collaborators Welcome. Permission Granted upon Request.

Auto-ViML Automatically Build Variant Interpretable ML models fast! Auto_ViML is pronounced "auto vimal" (autovimal logo created by Sanket Ghanmare) N

AutoViz and Auto_ViML 397 Dec 30, 2022
Space Time Recurrent Memory Network - Pytorch

Space Time Recurrent Memory Network - Pytorch (wip) Implementation of Space Time Recurrent Memory Network, recurrent network competitive with attentio

Phil Wang 50 Nov 07, 2021
PyTorch implementation of DreamerV2 model-based RL algorithm

PyDreamer Reimplementation of DreamerV2 model-based RL algorithm in PyTorch. The official DreamerV2 implementation can be found here. Features ... Run

118 Dec 15, 2022
Official implementation of the RAVE model: a Realtime Audio Variational autoEncoder

RAVE: Realtime Audio Variational autoEncoder Official implementation of RAVE: A variational autoencoder for fast and high-quality neural audio synthes

ACIDS 587 Jan 01, 2023
Official Python implementation of the FuzionCoin protocol

PyFuzc Official Python implementation of the FuzionCoin protocol WARNING: Under construction. Use at your own risk. Some functions may not work. Setup

FuzionCoin 3 Jul 07, 2022
Plenoxels: Radiance Fields without Neural Networks

Plenoxels: Radiance Fields without Neural Networks Alex Yu*, Sara Fridovich-Keil*, Matthew Tancik, Qinhong Chen, Benjamin Recht, Angjoo Kanazawa UC Be

Sara Fridovich-Keil 81 Dec 25, 2022
The source code and dataset for the RecGURU paper (WSDM 2022)

RecGURU About The Project Source code and baselines for the RecGURU paper "RecGURU: Adversarial Learning of Generalized User Representations for Cross

Chenglin Li 17 Jan 07, 2023
A Next Generation ConvNet by FaceBookResearch Implementation in PyTorch(Original) and TensorFlow.

ConvNeXt A Next Generation ConvNet by FaceBookResearch Implementation in PyTorch(Original) and TensorFlow. A FacebookResearch Implementation on A Conv

Raghvender 2 Feb 14, 2022
《LightXML: Transformer with dynamic negative sampling for High-Performance Extreme Multi-label Text Classification》(AAAI 2021) GitHub:

LightXML: Transformer with dynamic negative sampling for High-Performance Extreme Multi-label Text Classification

76 Dec 05, 2022
Automatic Differentiation Multipole Moment Molecular Forcefield

Automatic Differentiation Multipole Moment Molecular Forcefield Performance notes On a single gpu, using waterbox_31ang.pdb example from MPIDplugin wh

4 Jan 07, 2022
Using fully convolutional networks for semantic segmentation with caffe for the cityscapes dataset

Using fully convolutional networks for semantic segmentation (Shelhamer et al.) with caffe for the cityscapes dataset How to get started Download the

Simon Guist 27 Jun 06, 2022