In the traffic light and traffic sign detection issue, you've mentioned that we just have to change the obj_list in project.yml by adding the classes needed. Does that apply to seg_list as well?

If I want to detect lane color and lane type, can I change seg_list as follows

seg_list: ['road', 'double white', 'double yellow', 'single white', 'single yellow', 'solid', 'dashed']

Actually, I need classes like double solid yellow, double solid white, single solid yellow, single solid white, single dashed yellow, single dashed white, double dashed yellow, double dashed white, but BDD100K already has labeled double white, double yellow, single white, single yellow classes under Lane Categories and solid, dashed classes under Lane Styles

Will the change in seg_list as shown above work, if not, how to do it

Hi,

This link is required access permission. Could you help with it? https://drive.google.com/drive/folders/1iW9Darrars4xc9uHq2ZnRsWoJolCcyzg?usp=sharing

Hello. First of all,thank you for this work.

I noted you mistake the code about the inf_time and fps.

So I think maybe you calculation the inference time incorrectly in the article , your article show that YOLOP have 52ms the infercence time per frame(batch size 1), which mean 20fps? (although 41 fps show in the YOLOP's article).

And sadly in the hybridnets_test.py , i try calculate the HYBRIDNET's inference time but only get 0.06s(only model(x) ) ,which means 17-20fps. （Tesla v100 ）but get 0.021s(only model(x) ) in YOLOP, which means 48 fps（Tesla v100 ）

Sadly , it may not faster than YOLOP and not reach the real-time.

code as follows, but export nothing:

weight_path = 'weights/hybridnets.pth'
device = 'cuda' if torch.cuda.is_available() else 'cpu'
params = Params(os.path.join(Path(__file__).resolve().parent, "projects/bdd100k.yml"))
model = HybridNetsBackbone(num_classes=len(params.obj_list), compound_coef=3,
                           ratios=eval(params.anchors_ratios), scales=eval(params.anchors_scales),
                           seg_classes=len(params.seg_list), backbone_name=None)
model.load_state_dict(torch.load(weight_path, map_location=device))
model.eval()
inputs = torch.randn(1, 3, 384, 640)
print("begin to convert onnx")
torch.onnx.export(model, inputs, 'HybridNetsBackbone.onnx',
                  verbose=False, opset_version=12, input_names=['images'])
print("done")

shell log:

HybridNets/utils/utils.py:673: TracerWarning: torch.from_numpy results are registered as constants in the trace. You can safely ignore this warning if you use this function to create tensors out of constant variables that would be the same every time you call this function. In any other case, this might cause the trace to be incorrect.
  anchor_boxes = torch.from_numpy(anchor_boxes.astype(dtype)).to(image.device)
Warning: Constant folding - Only steps=1 can be constant folded for opset >= 10 onnx::Slice op. Constant folding not applied.
Warning: Constant folding - Only steps=1 can be constant folded for opset >= 10 onnx::Slice op. Constant folding not applied.
Warning: Constant folding - Only steps=1 can be constant folded for opset >= 10 onnx::Slice op. Constant folding not applied.
Warning: Constant folding - Only steps=1 can be constant folded for opset >= 10 onnx::Slice op. Constant folding not applied.
Warning: Constant folding - Only steps=1 can be constant folded for opset >= 10 onnx::Slice op. Constant folding not applied.
Warning: Constant folding - Only steps=1 can be constant folded for opset >= 10 onnx::Slice op. Constant folding not applied.
Warning: Constant folding - Only steps=1 can be constant folded for opset >= 10 onnx::Slice op. Constant folding not applied.
Warning: Constant folding - Only steps=1 can be constant folded for opset >= 10 onnx::Slice op. Constant folding not applied.
Warning: Constant folding - Only steps=1 can be constant folded for opset >= 10 onnx::Slice op. Constant folding not applied.

...
ONNX export failed: Couldn't export Python operator SwishImplementation

Hi, I've been trying to recreate your results from the Hybridnets paper. I've run the eval code on the 100k dataset but the results I'm getting are nowhere close to the actual results you presented in the paper. I'm sure I'm missing something here, could you please tell me what could possibly be going wrong here. I think the root locations that I'm giving are inaccurate.

data root - raw images from the 100k dataset label root - I saved separate json file for each image from the "bdd100k_labels_images_val.json" that I downloaded from the bdd100k dataset. So in total I saved 10,000 separate json (one for each image) into another folder named "val". Road labels in Seg_list - I used the drivable masks Lane labels in Seg_list - I used the lane masks

The output that I got after evaluation of 100 images:

The iou values are inconsistent with the ones mentioned in the paper, they are nowhere near them. I was also wondering why the precision is so low, is there a specific reason as to why there are so many false positives. I hope you can help me out, thanks.

Hello, I tried to follow your suggestion to train the model. So accordingly, at first I freeze the segmentation and trained for some epoch.

python train.py -p bdd100k -c 3 -n 4 -b 8 --freeze_seg True --lr 1e-5 --optim adamw --num_epochs 75 --val_interval 1 --log_path D:\HybridNets\rgb-clean --saved_path D:\HybridNets\rgb-clean --save_interval 500 --verbose True --num_gpus 1 --plots True

After that I am freezing the backbone and detection head.

python train.py -p bdd100k -c 3 -n 4 -b 8 --freeze_backbone True --freeze_det True --lr 1e-5 --optim adamw --num_epochs 12 --val_interval 1 --log_path D:\HybridNets\rgb_clean --saved_path D:\HybridNets\rgb_clean --save_interval 500 --verbose True --num_gpus 1 --plots True -w D:\HybridNets\rgb-clean\bdd100k\hybridnets-d3_74_129225_best.pth

But I am getting the error

Can you please suggest how to solve this issue?

Thank you in advance

if i just want to seg one class，such as seg_list only have ’road‘. Then i run train.py , in loss.py line 538, in soft_tversky_score assert output.size() == target.size() AssertionError

then i debug code,find output.size() = torch.Size([2, 1, 245760]) target.size() = torch.Size([2, 1, 491520])

How to fix that???

Hello , when i put a image of size 1920*1080 for test,there is the following error. Can you please help me resolve this issue? Thank you! "IndexError: boolean index did not match indexed array along dimension 0; dimension is 1080 but corresponding boolean dimension is 720"

Back when we were toying with mosaic, we removed the segmentation head completely from the model and dataloader. Now that we try to add mosaic augmentation officially, we have to make a decision of not using it for segmentation training.

hybridnets/dataset.py

if self.use_mosaic:
    # honestly, mosaic is not for road and lane segmentation anyway
    # you cant expect road and lane to be split up in 4 separate corners in an image, do you?
    # only use mosaic with freeze_seg :)
    img, labels, seg_label, lane_label, (h0, w0), (h, w), path = self.load_mosaic(idx)

Only images and object annotations are mosaic, while segmentation annotations are kept intact, which produces incorrect segmentation loss but that doesn't matter because we froze segmentation head anyway, thinking that requires_grad=False makes the segmentation head disappear from backprop graph. But that is wrong, the backbone is still affected by segmentation loss.

Check this colab for interactive stuffs.

So we've been planning to just straight ahead set the losses to 0 when you --freeze_head like this:

cls_loss, reg_loss, seg_loss, regression, classification, anchors, segmentation = model(imgs, annot, seg_annot, obj_list=params.obj_list)
cls_loss = cls_loss.mean() if not opt.freeze_det else 0
reg_loss = reg_loss.mean() if not opt.freeze_det else 0
seg_loss = seg_loss.mean() if not opt.freeze_seg else 0

Is this approach too naive? Are there any recommendation regarding this matter? Or should we also mosaic the segmentation labels?

model_to_quantize = copy.deepcopy(model)

qconfig_dict = {"": torch.quantization.get_default_qconfig('qnnpack')}

model_to_quantize.eval()

# prepare model_prepared = quantize_fx.prepare_fx(model_to_quantize, qconfig_dict)

# calibrate (not shown) # quantize model_quantized = quantize_fx.convert_fx(model_prepared)

When using Pytorch quantization example for your model, I get this following error:

~/Documents/DL_course_project/HybridNets/backbone.py in forward(self, inputs) 100 101 # p1, p2, p3, p4, p5 = self.backbone_net(inputs) 102 --> p2, p3, p4, p5 = self.encoder(inputs)[-4:] # self.backbone_net(inputs) 103 104 features = (p3, p4, p5) NameError: module is not installed as a submodule

How can I avoid this error?

@datvuthanh thanks for sharing the code based i have following queries

1. since bdd100k has different types of lanes eg double yellow, single white lane , dashed, can we use current source code to train for different lane types? is so what are the modifications need to be made int he code based
2. Can we similar train the current source code with the driveable area and alternate drivable area labels ? if so what are the changes to be made

Please share your thoughts Thanks in advance

I know you shared a drive link for bdd100k drivable area and lane masks for the dataloader but I want to replicate it for understanding what to do for my custom dataset. I looked for bdd repo and there is "to_mask.py" which has some scripts to do it. It passes its own tests but I cannot get similar results as you shared. Can you please explain how to generate those masks? Thanks in advance.

Hello. First of all, great work.

While running the hybridnets_test_videos.py I found some issues with the FPS calculation part. In the hybridnets_test_videos.py script, the FPS is calculated as: (t2-t1)/frame_count)

But it seems that the above code will give the inference time per frame and not the FPS. Most probably it should be: 1/((t2-t1)/frame_count)) That is, we need to divide it by 1.

Please let me know if any updates happen on this front.

I changed the backbone to Efficientnet-b0 and reduce the number of BiFPN layers from 6 to 1, in order to cut down the runtime of inference. After training 200 epochs with segmentation head frozen, I tried to train the model freezing backbone and detection head. But I found that the train loss of segmentation head dose not seem to converge. And the loss of valuation and mIOU are reducing at the same time, which doesn't make sense. Apart from that, I also found that when freezing segmentation head, the segmentation loss is not set to 0 in the code. which can affect the updating of weights in backbone, I suppose.