If I want to train my datasets(texture-less), which has 4 classes.

First, how should I train the 2D detector? And MaskRCNN or Retinanet?

Second, in surfemb, What parameters should I modify?

Can you give me some advice? Thanks~

For some strange reason wget does not download data to the correct place on my machine, so I downloaded the inference_data.zip file manually. I am now unsure where to extract its contents? Would I do this in the root directory or perhaps in /data or /data/models. Thank you.

I downloaded the tless on bop and put it under data/bop/tless, the code can load the cad models , i use python -m surfemb.scripts.train tless --gpus 0 to run, but it runs to After trainer.fit, the following error will occur： TypeError: default_collate: batch must contain tensors, numpy arrays, numbers, dicts or lists; found <class 'trimesh.caching.TrackedArray'> Sorry, am I setting it wrong?

I'm trying to train surfemb on a custom dataset. But when I do inference, I find some files/dirs are musts(they're detection_results, surface_samples, surface_samples_normals). For dectection_results, I've already known it's from CosyPose, but I don't know how to generate it in detail. For surface_samples, I followed your guidance that run command $ python -m surfemb.scripts.misc.surface_samples_remesh_visible clip first. But I encountered this error:

Traceback (most recent call last):
  File "/root/miniconda3/envs/surfemb/lib/python3.8/runpy.py", line 194, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/root/miniconda3/envs/surfemb/lib/python3.8/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/root/surfemb/surfemb/scripts/misc/surface_samples_remesh_visible.py", line 32, in <module>
    ms.compute_scalar_ambient_occlusion(occmode='per-Face (deprecated)', reqviews=256)
AttributeError: 'pymeshlab.pmeshlab.MeshSet' object has no attribute 'compute_scalar_ambient_occlusion'

Could you please give me any suggestions on these two problems?

Hi, thank you again.

I'm comparing the pose score you proposed in paper and implemented in codes such that I encounter some questions.

1. In this line you calculate neg_mask_log_prob by inversing mask_lgts before feeding it into the logsigmoid. why do you inverse mask_lgts? Does it actually mean anything?
2. Can I think of the pose score as confidence as long as I map its value to [0, 1] via a kind of mono-increasing function?

Question 1:I used ycbv-jwpvdij1.compact.ckpt(a trained model that you provided) to infer test datasets in ycbv(python -m surfemb.scripts.infer), then python -m surfemb.scripts.misc.format_results_for_eval, the score in results all is negative, for example，-0.339 , -0.401.Is that normal？ A:scene_id B:img_id C:est_obj_id D: score.

when i want to train the tless python -m surfemb.scripts.train tless:

Traceback (most recent call last):
  File "/home/zzz/miniconda3/envs/surfemb/lib/python3.8/runpy.py", line 194, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/home/zzz/miniconda3/envs/surfemb/lib/python3.8/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/home/zzz/github/surfemb/surfemb/scripts/train.py", line 121, in <module>
    main()
  File "/home/zzz/github/surfemb/surfemb/scripts/train.py", line 61, in main
    model = SurfaceEmbeddingModel(n_objs=len(obj_ids), **vars(args))
  File "/home/zzz/github/surfemb/surfemb/surface_embedding.py", line 48, in __init__
    n_class=(emb_dim + 1) if separate_decoders else n_objs * (emb_dim + 1),
TypeError: unsupported operand type(s) for +: 'NoneType' and 'int'

It Seems the emb_dim is None. But it has the default value emb_dim=12.

Hi, thanks for your great work, I hope I will make it work and be able to use it on my custom dataset. my problem is this; I have one 2080ti and I am trying to train the tless pbr dataset but I get an error "cuda out of memory" . I have used smaller batch size which is 8, I have decreased the number of workers to 0. but it keeps giving the error ( ok now it gives the error later than before but it still gives the error)

it only works if I decrease the scenes from 50 to 1 in train_pbr folder. otherwise no chance.

is this normal behavior with this one gpu , or I am missing something

thanks in advance

I have tried to run the Inference Inspection code in my windows machine with the given inference data as proposed in the README but I got the error:

$ python -m surfemb.scripts.infer_debug data/models/tless-2rs64lwh.compact.ckpt --device cpu
loading objects: 0it [00:00, ?it/s]
Traceback (most recent call last):
  File "C:\Users\39331\anaconda3\envs\surfemb\lib\runpy.py", line 194, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "C:\Users\39331\anaconda3\envs\surfemb\lib\runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "C:\Users\39331\Documenti\Final Year Project\surfemb\surfemb\scripts\infer_debug.py", line 43, in <module>
    renderer = ObjCoordRenderer(objs, res_crop)
  File "C:\Users\39331\Documenti\Final Year Project\surfemb\surfemb\data\renderer.py", line 43, in __init__
    self.ctx = moderngl.create_context(standalone=True, backend='egl', device_index=device_idx)
  File "C:\Users\39331\anaconda3\envs\surfemb\lib\site-packages\moderngl\context.py", line 1619, in create_context
    ctx.mglo, ctx.version_code = mgl.create_context(glversion=require, mode=mode, **settings)
  File "C:\Users\39331\anaconda3\envs\surfemb\lib\site-packages\glcontext\__init__.py", line 49, in get_backend_by_name
    return _egl()
  File "C:\Users\39331\anaconda3\envs\surfemb\lib\site-packages\glcontext\__init__.py", line 106, in _egl
    from glcontext import egl
ImportError: cannot import name 'egl' from 'glcontext' (C:\Users\39331\anaconda3\envs\surfemb\lib\site-packages\glcontext\__init__.py)

I have tried installing OpenGL again but it did not solve the problem, I cannot find any sources for solving the dependency. How would you suggest me to solve it?

Hi, i run 'python -m surfemb.scripts.train ycbv', and ./data/bop/ycbv/models have 21 .ply files, and 80 folder in ./data/bop/ycbv/train_real. I did not use synth imgs, but the program always 0%. Is the program preprocessing image information,crop object from img?I waited a dozen hours and it was still 0%. like Epoch 0: 0%. The python is still running.Do I have to wait a long time before train model?Are you in a similar situation?

But if I just have 3 .ply files in ./data/bop/ycba/models,and 1 folder in ./data/bop/ycbv/train_real,it will soon(3 - 4 minutes) train cnn .Finally, the trained model is obtained. For example, obj_000008.ply, obj_000014.ply, obj_000021.ply in ./data/bop/ycba/models, 000000 in ./data/bop/ycbv/train_real(imgs in 000000 only have 3 types of objects, obj_8, obj_14, obj_21).

If I want to train all the objects in ycbv at once, Do I have to wait longer?I'm using a server, CPU performance is not weak.

{ "os": "Linux-4.15.0-175-generic-x86_64-with-debian-buster-sid", "python": "3.7.11", "heartbeatAt": "2022-04-09T09:34:05.311715", "startedAt": "2022-04-09T09:34:02.496814", "docker": null, "gpu": "GeForce RTX 3090", "gpu_count": 8, "cpu_count": 40, "cuda": null, "args": [], "state": "running", "program": "-m surfemb.scripts.train", "git": { "remote": "https://github.com/rasmushaugaard/surfemb.git", "commit": "46f46ddc5670848d696968dc8ec65c8ce62b16a8" }, "email": "[email protected]", "root": "/home/aa/prjs/surfemb", "host": "sddx-PR4908P", "username": "aa", "executable": "/home/aa/anaconda3/envs/d2_1.10/bin/python" }

logs: 2022-04-09 11:02:46,213 INFO MainThread:16337 [wandb_setup.py:_flush():75] Loading settings from /home/aa/.config/wandb/settings 2022-04-09 11:02:46,214 INFO MainThread:16337 [wandb_setup.py:_flush():75] Loading settings from /home/aa/prjs/bcnet/pose/surfemb/wandb/settings 2022-04-09 11:02:46,214 INFO MainThread:16337 [wandb_setup.py:_flush():75] Loading settings from environment variables: {'api_key': 'REDACTED', 'mode': 'offline', '_require_service': 'True'} 2022-04-09 11:02:46,214 WARNING MainThread:16337 [wandb_setup.py:_flush():75] Could not find program at -m surfemb.scripts.train 2022-04-09 11:02:46,214 INFO MainThread:16337 [wandb_setup.py:_flush():75] Inferring run settings from compute environment: {'program_relpath': None, 'program': '-m surfemb.scripts.train'} 2022-04-09 11:02:46,214 INFO MainThread:16337 [wandb_init.py:_log_setup():405] Logging user logs to /home/aa/prjs/bcnet/pose/surfemb/wandb/offline-run-20220409_110246-3fewafz3/logs/debug.log 2022-04-09 11:02:46,214 INFO MainThread:16337 [wandb_init.py:_log_setup():406] Logging internal logs to /home/aa/prjs/bcnet/pose/surfemb/wandb/offline-run-20220409_110246-3fewafz3/logs/debug-internal.log 2022-04-09 11:02:46,215 INFO MainThread:16337 [wandb_init.py:init():439] calling init triggers 2022-04-09 11:02:46,215 INFO MainThread:16337 [wandb_init.py:init():443] wandb.init called with sweep_config: {} config: {} 2022-04-09 11:02:46,215 INFO MainThread:16337 [wandb_init.py:init():492] starting backend 2022-04-09 11:02:46,228 INFO MainThread:16337 [backend.py:_multiprocessing_setup():101] multiprocessing start_methods=fork,spawn,forkserver, using: spawn 2022-04-09 11:02:46,232 INFO MainThread:16337 [wandb_init.py:init():501] backend started and connected 2022-04-09 11:02:46,238 INFO MainThread:16337 [wandb_init.py:init():565] updated telemetry 2022-04-09 11:02:46,578 INFO MainThread:16337 [wandb_init.py:init():625] starting run threads in backend 2022-04-09 11:02:49,104 INFO MainThread:16337 [wandb_run.py:_console_start():1733] atexit reg 2022-04-09 11:02:49,106 INFO MainThread:16337 [wandb_run.py:_redirect():1606] redirect: SettingsConsole.WRAP 2022-04-09 11:02:49,107 INFO MainThread:16337 [wandb_run.py:_redirect():1643] Wrapping output streams. 2022-04-09 11:02:49,108 INFO MainThread:16337 [wandb_run.py:_redirect():1667] Redirects installed. 2022-04-09 11:02:49,109 INFO MainThread:16337 [wandb_init.py:init():664] run started, returning control to user process 2022-04-09 11:02:49,130 INFO MainThread:16337 [wandb_run.py:_config_callback():992] config_cb None None {'n_objs': 21, 'emb_dim': 12, 'n_pos': 1024, 'n_neg': 1024, 'lr_cnn': 0.0003, 'lr_mlp': 3e-05, 'mlp_name': 'siren', 'mlp_hidden_features': 256, 'mlp_hidden_layers': 2, 'key_noise': 0.001, 'warmup_steps': 2000, 'separate_decoders': True, 'pa_sigma': 0.0, 'align_corners': False, 'dataset': 'ycbv', 'n_valid': 200, 'res_data': 256, 'res_crop': 224, 'batch_size': 16, 'num_workers': 'None', 'min_visib_fract': 0.1, 'max_steps': 500000, 'gpus': 2, 'debug': False, 'ckpt': 'None', 'synth': False, 'real': True} 2022-04-09 11:07:50,141 WARNING MsgRouterThr:16337 [router.py:message_loop():76] message_loop has been closed

Thanks again for this great work. I am still trying to make it work for our case. In order to run the model with our custom dataset, should we need to have bbox and mask in our bop formatted custom dataset? we can provide bbox_visible but not bbox. and we can provide mask_visible but not mask? how important are they? would be bbox_visible and mask_visible enough? if we need them for this model, do you have any idea how to generate that ? thanks thanks in advance

In the training script, an additional environment variable needs to be set in the worker_init_fn function

os.environ['OMP_NUM_THREADS'] = 1

See torch comment for additional information

Thank you for the wonderful work. Both the paper and the code are a pleasure to read.

I have tried the approach on a different dataset and would like to ask for your expert opinion, if I may. A fraction of the predictions (~60%) are very good even with only RGB refinement, but the remaining pose predictions are far away from the actual pose (about 1m in l1 distance) and could be ruled out by calculating the xyz boundaries of the crop. The input pose from the PNP between both fraction almost equally good.

• Do you have an idea how to discipline the refinement?
• How would you analyze the quality of the incoming query image or the sampled keys?
• Do you have any other suggestions what to look for in these cases?

Thanks again for the wonderful work.

whenever i try to resume from a previous checkpoint, i get this error: File "/home/moritz/anaconda3/envs/surfemb/lib/python3.8/runpy.py", line 194, in _run_module_as_main return _run_code(code, main_globals, None, File "/home/moritz/anaconda3/envs/surfemb/lib/python3.8/runpy.py", line 87, in _run_code exec(code, run_globals) File "/home/moritz/surfemb/surfemb/surfemb/scripts/train.py", line 123, in <module> main() File "/home/moritz/surfemb/surfemb/surfemb/scripts/train.py", line 119, in main trainer.fit(model, loader_train, loader_valid) File "/home/moritz/anaconda3/envs/surfemb/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 768, in fit self._call_and_handle_interrupt( File "/home/moritz/anaconda3/envs/surfemb/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 721, in _call_and_handle_interrupt return trainer_fn(*args, **kwargs) File "/home/moritz/anaconda3/envs/surfemb/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 809, in _fit_impl results = self._run(model, ckpt_path=self.ckpt_path) File "/home/moritz/anaconda3/envs/surfemb/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 1234, in _run results = self._run_stage() File "/home/moritz/anaconda3/envs/surfemb/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 1321, in _run_stage return self._run_train() File "/home/moritz/anaconda3/envs/surfemb/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 1351, in _run_train self.fit_loop.run() File "/home/moritz/anaconda3/envs/surfemb/lib/python3.8/site-packages/pytorch_lightning/loops/base.py", line 204, in run self.advance(*args, **kwargs) File "/home/moritz/anaconda3/envs/surfemb/lib/python3.8/site-packages/pytorch_lightning/loops/fit_loop.py", line 268, in advance self._outputs = self.epoch_loop.run(self._data_fetcher) File "/home/moritz/anaconda3/envs/surfemb/lib/python3.8/site-packages/pytorch_lightning/loops/base.py", line 204, in run self.advance(*args, **kwargs) File "/home/moritz/anaconda3/envs/surfemb/lib/python3.8/site-packages/pytorch_lightning/loops/epoch/training_epoch_loop.py", line 208, in advance batch_output = self.batch_loop.run(batch, batch_idx) File "/home/moritz/anaconda3/envs/surfemb/lib/python3.8/site-packages/pytorch_lightning/loops/base.py", line 204, in run self.advance(*args, **kwargs) File "/home/moritz/anaconda3/envs/surfemb/lib/python3.8/site-packages/pytorch_lightning/loops/batch/training_batch_loop.py", line 88, in advance outputs = self.optimizer_loop.run(split_batch, optimizers, batch_idx) File "/home/moritz/anaconda3/envs/surfemb/lib/python3.8/site-packages/pytorch_lightning/loops/base.py", line 204, in run self.advance(*args, **kwargs) File "/home/moritz/anaconda3/envs/surfemb/lib/python3.8/site-packages/pytorch_lightning/loops/optimization/optimizer_loop.py", line 203, in advance result = self._run_optimization( File "/home/moritz/anaconda3/envs/surfemb/lib/python3.8/site-packages/pytorch_lightning/loops/optimization/optimizer_loop.py", line 256, in _run_optimization self._optimizer_step(optimizer, opt_idx, batch_idx, closure) File "/home/moritz/anaconda3/envs/surfemb/lib/python3.8/site-packages/pytorch_lightning/loops/optimization/optimizer_loop.py", line 369, in _optimizer_step self.trainer._call_lightning_module_hook( File "/home/moritz/anaconda3/envs/surfemb/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 1593, in _call_lightning_module_hook output = fn(*args, **kwargs) File "/home/moritz/anaconda3/envs/surfemb/lib/python3.8/site-packages/pytorch_lightning/core/lightning.py", line 1644, in optimizer_step optimizer.step(closure=optimizer_closure) File "/home/moritz/anaconda3/envs/surfemb/lib/python3.8/site-packages/pytorch_lightning/core/optimizer.py", line 168, in step step_output = self._strategy.optimizer_step(self._optimizer, self._optimizer_idx, closure, **kwargs) File "/home/moritz/anaconda3/envs/surfemb/lib/python3.8/site-packages/pytorch_lightning/strategies/strategy.py", line 193, in optimizer_step return self.precision_plugin.optimizer_step(model, optimizer, opt_idx, closure, **kwargs) File "/home/moritz/anaconda3/envs/surfemb/lib/python3.8/site-packages/pytorch_lightning/plugins/precision/precision_plugin.py", line 155, in optimizer_step return optimizer.step(closure=closure, **kwargs) File "/home/moritz/anaconda3/envs/surfemb/lib/python3.8/site-packages/torch/optim/lr_scheduler.py", line 65, in wrapper return wrapped(*args, **kwargs) File "/home/moritz/anaconda3/envs/surfemb/lib/python3.8/site-packages/torch/optim/optimizer.py", line 109, in wrapper return func(*args, **kwargs) File "/home/moritz/anaconda3/envs/surfemb/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context return func(*args, **kwargs) File "/home/moritz/anaconda3/envs/surfemb/lib/python3.8/site-packages/torch/optim/adam.py", line 157, in step adam(params_with_grad, File "/home/moritz/anaconda3/envs/surfemb/lib/python3.8/site-packages/torch/optim/adam.py", line 213, in adam func(params, File "/home/moritz/anaconda3/envs/surfemb/lib/python3.8/site-packages/torch/optim/adam.py", line 255, in _single_tensor_adam assert not step_t.is_cuda, "If capturable=False, state_steps should not be CUDA tensors." AssertionError: If capturable=False, state_steps should not be CUDA tensors.

Any idea how to resolve this?

also i cant get the training to run with standard settings. i get an outofmemory error on a rtx3070ti (8gb) if i dont run at n-valid = 2 and batch-size = 1

When I run the surface_samples_remesh_visible script for tless objects the processing fails with

Aborted (core dumped)

did anyone face the same issue or did I miss some steps before running the script?