If I want to train my datasets(texture-less), which has 4 classes.

First, how should I train the 2D detector? And MaskRCNN or Retinanet?

Second, in surfemb, What parameters should I modify?

Can you give me some advice? Thanks~

I downloaded the tless on bop and put it under data/bop/tless, the code can load the cad models , i use python -m surfemb.scripts.train tless --gpus 0 to run, but it runs to After trainer.fit, the following error will occur： TypeError: default_collate: batch must contain tensors, numpy arrays, numbers, dicts or lists; found <class 'trimesh.caching.TrackedArray'> Sorry, am I setting it wrong?

Question 1:I used ycbv-jwpvdij1.compact.ckpt(a trained model that you provided) to infer test datasets in ycbv(python -m surfemb.scripts.infer), then python -m surfemb.scripts.misc.format_results_for_eval, the score in results all is negative, for example，-0.339 , -0.401.Is that normal？ A:scene_id B:img_id C:est_obj_id D: score.

when i want to train the tless python -m surfemb.scripts.train tless:

Traceback (most recent call last):
  File "/home/zzz/miniconda3/envs/surfemb/lib/python3.8/runpy.py", line 194, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/home/zzz/miniconda3/envs/surfemb/lib/python3.8/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/home/zzz/github/surfemb/surfemb/scripts/train.py", line 121, in <module>
    main()
  File "/home/zzz/github/surfemb/surfemb/scripts/train.py", line 61, in main
    model = SurfaceEmbeddingModel(n_objs=len(obj_ids), **vars(args))
  File "/home/zzz/github/surfemb/surfemb/surface_embedding.py", line 48, in __init__
    n_class=(emb_dim + 1) if separate_decoders else n_objs * (emb_dim + 1),
TypeError: unsupported operand type(s) for +: 'NoneType' and 'int'

It Seems the emb_dim is None. But it has the default value emb_dim=12.

Hi, i run 'python -m surfemb.scripts.train ycbv', and ./data/bop/ycbv/models have 21 .ply files, and 80 folder in ./data/bop/ycbv/train_real. I did not use synth imgs, but the program always 0%. Is the program preprocessing image information,crop object from img?I waited a dozen hours and it was still 0%. like Epoch 0: 0%. The python is still running.Do I have to wait a long time before train model?Are you in a similar situation?

But if I just have 3 .ply files in ./data/bop/ycba/models,and 1 folder in ./data/bop/ycbv/train_real,it will soon(3 - 4 minutes) train cnn .Finally, the trained model is obtained. For example, obj_000008.ply, obj_000014.ply, obj_000021.ply in ./data/bop/ycba/models, 000000 in ./data/bop/ycbv/train_real(imgs in 000000 only have 3 types of objects, obj_8, obj_14, obj_21).

If I want to train all the objects in ycbv at once, Do I have to wait longer?I'm using a server, CPU performance is not weak.

{ "os": "Linux-4.15.0-175-generic-x86_64-with-debian-buster-sid", "python": "3.7.11", "heartbeatAt": "2022-04-09T09:34:05.311715", "startedAt": "2022-04-09T09:34:02.496814", "docker": null, "gpu": "GeForce RTX 3090", "gpu_count": 8, "cpu_count": 40, "cuda": null, "args": [], "state": "running", "program": "-m surfemb.scripts.train", "git": { "remote": "https://github.com/rasmushaugaard/surfemb.git", "commit": "46f46ddc5670848d696968dc8ec65c8ce62b16a8" }, "email": "[email protected]", "root": "/home/aa/prjs/surfemb", "host": "sddx-PR4908P", "username": "aa", "executable": "/home/aa/anaconda3/envs/d2_1.10/bin/python" }

logs: 2022-04-09 11:02:46,213 INFO MainThread:16337 [wandb_setup.py:_flush():75] Loading settings from /home/aa/.config/wandb/settings 2022-04-09 11:02:46,214 INFO MainThread:16337 [wandb_setup.py:_flush():75] Loading settings from /home/aa/prjs/bcnet/pose/surfemb/wandb/settings 2022-04-09 11:02:46,214 INFO MainThread:16337 [wandb_setup.py:_flush():75] Loading settings from environment variables: {'api_key': 'REDACTED', 'mode': 'offline', '_require_service': 'True'} 2022-04-09 11:02:46,214 WARNING MainThread:16337 [wandb_setup.py:_flush():75] Could not find program at -m surfemb.scripts.train 2022-04-09 11:02:46,214 INFO MainThread:16337 [wandb_setup.py:_flush():75] Inferring run settings from compute environment: {'program_relpath': None, 'program': '-m surfemb.scripts.train'} 2022-04-09 11:02:46,214 INFO MainThread:16337 [wandb_init.py:_log_setup():405] Logging user logs to /home/aa/prjs/bcnet/pose/surfemb/wandb/offline-run-20220409_110246-3fewafz3/logs/debug.log 2022-04-09 11:02:46,214 INFO MainThread:16337 [wandb_init.py:_log_setup():406] Logging internal logs to /home/aa/prjs/bcnet/pose/surfemb/wandb/offline-run-20220409_110246-3fewafz3/logs/debug-internal.log 2022-04-09 11:02:46,215 INFO MainThread:16337 [wandb_init.py:init():439] calling init triggers 2022-04-09 11:02:46,215 INFO MainThread:16337 [wandb_init.py:init():443] wandb.init called with sweep_config: {} config: {} 2022-04-09 11:02:46,215 INFO MainThread:16337 [wandb_init.py:init():492] starting backend 2022-04-09 11:02:46,228 INFO MainThread:16337 [backend.py:_multiprocessing_setup():101] multiprocessing start_methods=fork,spawn,forkserver, using: spawn 2022-04-09 11:02:46,232 INFO MainThread:16337 [wandb_init.py:init():501] backend started and connected 2022-04-09 11:02:46,238 INFO MainThread:16337 [wandb_init.py:init():565] updated telemetry 2022-04-09 11:02:46,578 INFO MainThread:16337 [wandb_init.py:init():625] starting run threads in backend 2022-04-09 11:02:49,104 INFO MainThread:16337 [wandb_run.py:_console_start():1733] atexit reg 2022-04-09 11:02:49,106 INFO MainThread:16337 [wandb_run.py:_redirect():1606] redirect: SettingsConsole.WRAP 2022-04-09 11:02:49,107 INFO MainThread:16337 [wandb_run.py:_redirect():1643] Wrapping output streams. 2022-04-09 11:02:49,108 INFO MainThread:16337 [wandb_run.py:_redirect():1667] Redirects installed. 2022-04-09 11:02:49,109 INFO MainThread:16337 [wandb_init.py:init():664] run started, returning control to user process 2022-04-09 11:02:49,130 INFO MainThread:16337 [wandb_run.py:_config_callback():992] config_cb None None {'n_objs': 21, 'emb_dim': 12, 'n_pos': 1024, 'n_neg': 1024, 'lr_cnn': 0.0003, 'lr_mlp': 3e-05, 'mlp_name': 'siren', 'mlp_hidden_features': 256, 'mlp_hidden_layers': 2, 'key_noise': 0.001, 'warmup_steps': 2000, 'separate_decoders': True, 'pa_sigma': 0.0, 'align_corners': False, 'dataset': 'ycbv', 'n_valid': 200, 'res_data': 256, 'res_crop': 224, 'batch_size': 16, 'num_workers': 'None', 'min_visib_fract': 0.1, 'max_steps': 500000, 'gpus': 2, 'debug': False, 'ckpt': 'None', 'synth': False, 'real': True} 2022-04-09 11:07:50,141 WARNING MsgRouterThr:16337 [router.py:message_loop():76] message_loop has been closed

Hi, thank you for your great work! I notice that your implementation of Unet is a little different from the original one. in this line you feed the data from "contracting path" into a convrelu before catenating it with current data from "expansive path", which you do in the next line. But the author of Unet merely copy it, with no furthur process. So what's the reason you do in such manner?

when i try to run python -m surfemb.scripts.misc.surface_samples_remesh_visible tless,

it cause: ImportError: /usr/lib/x86_64-linux-gnu/libstdc++.so.6: version `GLIBCXX_3.4.26' not found (required by /home/robot/miniconda3/envs/surfemb/lib/python3.8/site-packages/cv2.cpython-38-x86_64-linux-gnu.so)

i install opencv-python==4.1.2.30 to fix it.

But it cause another problem: Traceback (most recent call last): File "/home/zzz/miniconda3/envs/surfemb-test/lib/python3.8/runpy.py", line 194, in _run_module_as_main return _run_code(code, main_globals, None, File "/home/zzz/miniconda3/envs/surfemb-test/lib/python3.8/runpy.py", line 87, in _run_code exec(code, run_globals) File "/home/zzz/github/surfemb/surfemb/scripts/misc/surface_samples_remesh_visible.py", line 29, in ms.repair_non_manifold_edges_by_removing_faces() AttributeError: 'pymeshlab.pmeshlab.MeshSet' object has no attribute 'repair_non_manifold_edges_by_removing_faces'

Hi, authors!

Thank you for your great work! The visualization results in your paper and website are amazing, so would you please offer us the visualization code additionally?

Thanks for sharing your code.

Some source files may be missing under the folder of ./data, such as config, instance, detector_crops, ObjCoordRenderer.

I would like to know the value of loss after 500,000(default config) iterations of training in YCBV pbr dataset and real image dataset.Because I get worse recall than in the paper.I just need a rough range of loss. Thank you.

Hi~ Thanks for your sharing.

Ubuntu 16.04 2070 super

when i try to run python -m surfemb.scripts.misc.surface_samples_remesh_visible tless, it has the following problems:

Traceback (most recent call last):
  File "surfemb/scripts/misc/surface_samples_remesh_visible.py", line 4, in <module>
    import pymeshlab
  File "/home/zzz/miniconda3/envs/surfemb_test/lib/python3.8/site-packages/pymeshlab/__init__.py", line 11, in <module>
    from .pmeshlab import *
ImportError: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.25' not found (required by /home/zzz/miniconda3/envs/surfemb_test/lib/python3.8/site-packages/pymeshlab/lib/libpython3.8.so.1.0)

I cannot solve it. So i came to ask for help.

Hello, thanks for your great work and patient replyment. I find that the inputs of the proposed nn surfemb are cropped images where there're only one instance, whenever do training or inference, aren't they? So can you deal with a image which has multiple instances whose bboxs are not known?

I'm trying to use your render code for depth image generation. Although I have set 'OPENLAS_NUM_THREADS' etc, segmentation fault encounter in work always occurs if num_worker>0, do you have any idea about this?

I have tried to run the Inference Inspection code in my windows machine with the given inference data as proposed in the README but I got the error:

$ python -m surfemb.scripts.infer_debug data/models/tless-2rs64lwh.compact.ckpt --device cpu
loading objects: 0it [00:00, ?it/s]
Traceback (most recent call last):
  File "C:\Users\39331\anaconda3\envs\surfemb\lib\runpy.py", line 194, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "C:\Users\39331\anaconda3\envs\surfemb\lib\runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "C:\Users\39331\Documenti\Final Year Project\surfemb\surfemb\scripts\infer_debug.py", line 43, in <module>
    renderer = ObjCoordRenderer(objs, res_crop)
  File "C:\Users\39331\Documenti\Final Year Project\surfemb\surfemb\data\renderer.py", line 43, in __init__
    self.ctx = moderngl.create_context(standalone=True, backend='egl', device_index=device_idx)
  File "C:\Users\39331\anaconda3\envs\surfemb\lib\site-packages\moderngl\context.py", line 1619, in create_context
    ctx.mglo, ctx.version_code = mgl.create_context(glversion=require, mode=mode, **settings)
  File "C:\Users\39331\anaconda3\envs\surfemb\lib\site-packages\glcontext\__init__.py", line 49, in get_backend_by_name
    return _egl()
  File "C:\Users\39331\anaconda3\envs\surfemb\lib\site-packages\glcontext\__init__.py", line 106, in _egl
    from glcontext import egl
ImportError: cannot import name 'egl' from 'glcontext' (C:\Users\39331\anaconda3\envs\surfemb\lib\site-packages\glcontext\__init__.py)

I have tried installing OpenGL again but it did not solve the problem, I cannot find any sources for solving the dependency. How would you suggest me to solve it?

I run python -m surfemb.scripts.misc.surface_samples_remesh_visible AttributeError: 'pymeshlab.pmeshlab.MeshSet' object has no attribute 'repair_non_manifold_edges_by_removing_faces'

I install pymeshlab==0.2.1 to fix it. But it cause another problem:

0%| | 0/21 [00:00<?, ?it/s] /home/c/surfemb/data/bop/ycbv/models/obj_000013.ply

----------AngleRad 0.523599 Angledeg 30.000000 ratio 0.066987 vn 256 vn2 3821 asked 3821 got 3821 (expecting 255 instead of 256) 0%| | 0/21 [00:02<?, ?it/s] Traceback (most recent call last): File "/home/c/surfemb/surfemb/scripts/misc/surface_samples_remesh_visible.py", line 42, in area_reduction = trimesh.load_mesh(remesh_fp).area / trimesh.load_mesh(mesh_fp).area File "/home/c/anaconda3/envs/surfemb/lib/python3.8/site-packages/trimesh/constants.py", line 153, in timed result = method(*args, **kwargs) File "/home/c/anaconda3/envs/surfemb/lib/python3.8/site-packages/trimesh/exchange/load.py", line 209, in load_mesh results = mesh_loaders[file_type](file_obj, File "/home/c/anaconda3/envs/surfemb/lib/python3.8/site-packages/trimesh/exchange/ply.py", line 106, in load_ply ply_binary(elements, file_obj) File "/home/c/anaconda3/envs/surfemb/lib/python3.8/site-packages/trimesh/exchange/ply.py", line 881, in ply_binary raise ValueError('File is unexpected length!') ValueError: File is unexpected length!