E2EC: An End-to-End Contour-based Method for High-Quality High-Speed Instance Segmentation
Tao Zhang, Shiqing Wei, Shunping Ji
CVPR 2022

Any questions or discussions are welcomed!

An e2ec implementation using mmdetection is complete: e2ec-mmdet. The performance on coco-val is as follows:

Please see INSTALL.md.

We re-tested the speed on a single RTX3090.

The accuracy and inference speed of the contours at different stages on SBD val set. We also re-tested the speed on a single RTX3090.

The accuracy and inference speed of the contours at different stages on coco val set.

1. Download the pretrained model here or Baiduyun(password is e2ec).

2. Prepared the COCO dataset according to the INSTALL.md.

3. Test:

   # testing segmentation accuracy on coco val set
   python test.py coco --checkpoint /path/to/model_coco.pth --with_nms True
   # testing detection accuracy on coco val set
   python test.py coco --checkpoint /path/to/model_coco.pth --with_nms True --eval bbox
   # testing the speed
   python test.py coco --checkpoint /path/to/model_coco.pth --with_nms True --type speed
   # testing the contours of specified stage(init/coarse/final/final-dml)
   python test.py coco --checkpoint /path/to/model_coco.pth --with_nms True --stage coarse
   # testing on coco test-dev set, run and submit data/result/results.json
   python test.py coco --checkpoint /path/to/model_coco.pth --with_nms True --dataset coco_test
   

1. Download the pretrained model here or Baiduyun(password is e2ec).

2. Prepared the SBD dataset according to the INSTALL.md.

3. Test:

   # testing segmentation accuracy on SBD
   python test.py sbd --checkpoint /path/to/model_sbd.pth
   # testing detection accuracy on SBD
   python test.py sbd --checkpoint /path/to/model_sbd.pth --eval bbox
   # testing the speed
   python test.py sbd --checkpoint /path/to/model_sbd.pth --type speed
   # testing the contours of specified stage(init/coarse/final/final-dml)
   python test.py sbd --checkpoint /path/to/model_sbd.pth --stage coarse
   

1. Download the pretrained model here or Baiduyun(password is e2ec).

2. Prepared the KINS dataset according to the INSTALL.md.

3. Test:

   Maybe you will find some troules, such as object of type <class 'numpy.float64'> cannot be safely interpreted as an integer. Please modify the /path/to/site-packages/pycocotools/cooceval.py. Replace np.round((0.95 - .5) / .05) in lines 506 and 507 with int(np.round((0.95 - .5) / .05)).

   # testing segmentation accuracy on KINS
   python test.py kitti --checkpoint /path/to/model_kitti.pth
   # testing detection accuracy on KINS
   python test.py kitti --checkpoint /path/to/model_kitti.pth --eval bbox
   # testing the speed
   python test.py kitti --checkpoint /path/to/model_kitti.pth --type speed
   # testing the contours of specified stage(init/coarse/final/final-dml)
   python test.py kitti --checkpoint /path/to/model_kitti.pth --stage coarse
   

1. Download the pretrained model here or Baiduyun(password is e2ec).

2. Prepared the KINS dataset according to the INSTALL.md.

3. Test:

   We will soon release the code for e2ec with multi component detection. Currently only supported for testing e2ec performance on cityscapes dataset.

   # testing segmentation accuracy on Cityscapes with coco evaluator
   python test.py cityscapesCoco --checkpoint /path/to/model_cityscapes.pth
   # with cityscapes official evaluator
   python test.py cityscapes --checkpoint /path/to/model_cityscapes.pth
   # testing the detection accuracy
   python test.py cityscapesCoco \
   --checkpoint /path/to/model_cityscapes.pth --eval bbox
   # testing the speed
   python test.py cityscapesCoco \
   --checkpoint /path/to/model_cityscapes.pth --type speed
   # testing the contours of specified stage(init/coarse/final/final-dml)
   python test.py cityscapesCoco \
   --checkpoint /path/to/model_cityscapes.pth --stage coarse
   # testing on test set, run and submit the result file
   python test.py cityscapes --checkpoint /path/to/model_cityscapes.pth \
   --dataset cityscapes_test
   

1. Install the Boundary IOU API according boundary iou.

2. Testing segmentation accuracy with coco evaluator.

3. Using offline evaluation pipeline.

   python /path/to/boundary_iou_api/tools/coco_instance_evaluation.py \
       --gt-json-file /path/to/annotation_file.json \
       --dt-json-file data/result/result.json \
       --iou-type boundary
   

1. Download the pretrained model.

2. Visualize:

   # inference and visualize the images with coco pretrained model
   python visualize.py coco /path/to/images \
   --checkpoint /path/to/model_coco.pth --with_nms True
   # you can using other pretrained model, such as cityscapes 
   python visualize.py cityscapesCoco /path/to/images \
   --checkpoint /path/to/model_cityscapes.pth
   # if you want to save the visualisation, please specify --output_dir
   python visualize.py coco /path/to/images \
   --checkpoint /path/to/model_coco.pth --with_nms True \
   --output_dir /path/to/output_dir
   # visualize the results at different stage
   python visualize.py coco /path/to/images \
   --checkpoint /path/to/model_coco.pth --with_nms True --stage coarse
   # you can reset the score threshold, default is 0.3
   python visualize.py coco /path/to/images \
   --checkpoint /path/to/model_coco.pth --with_nms True --ct_score 0.1
   # if you want to filter some of the jaggedness caused by dml 
   # please using post_process
   python visualize.py coco /path/to/images \
   --checkpoint /path/to/model_coco.pth --with_nms True \
   --with_post_process True
   

We have released the code for multi GPU training with ddp.

CUDA_VISIBLE_DEVICES=${gpu_ids} python -m torch.distributed.launch \
--nproc_per_node ${gpu_nums} \
train_net_ddp.py \
--config_file ${dataset} \
--bs ${bs_per_gpu} \
--gpus ${gpu_nums}
# the example of training sbd dataset using 2 gpus
CUDA_VISIBLE_DEVICES=0,1 python -m torch.distributed.launch \
--nproc_per_node 2 \
train_net_ddp.py \
--config_file sbd \
--bs 12 \
--gpus 2

python train_net.py sbd --bs $batch_size
# if you do not want to use dinamic matching loss (significantly improves 
# contour detail but introduces jaggedness), please set --dml as False
python train_net.py sbd --bs $batch_size --dml False

python train_net.py kitti --bs $batch_size

python train_net.py cityscapesCoco --bs $batch_size

In fact it is possible to achieve the same accuracy without training so many epochs.

# first to train with adam
python train_net.py coco --bs $batch_size
# then finetune with sgd
python train_net.py coco_finetune --bs $batch_size \
--type finetune --checkpoint data/model/139.pth

If the annotations is in coco style:

1. Add dataset information to dataset/info.py.

2. Modify the configs/coco.py, reset the train.dataset , model.heads['ct_hm'] and test.dataset. Maybe you also need to change the train.epochs, train.optimizer['milestones'] and so on.

3. Train the network.

   python train_net.py coco --bs $batch_size
   

If the annotations is not in coco style:

1. Prepare dataset/train/your_dataset.py and dataset/test/your_dataset.py by referring to dataset/train/base.py and dataset/test/base.py.

2. Prepare evaluator/your_dataset/snake.py by referring to evaluator/coco/snake.py.

3. Prepare configs/your_dataset.py and by referring to configs/base.py.

4. Train the network.

   python train_net.py your_dataset --bs $batch_size
   

If you find this project helpful for your research, please consider citing using BibTeX below:

@inproceedings{zhang2022e2ec,
  title={E2EC: An End-to-End Contour-based Method for High-Quality High-Speed Instance Segmentation},
  author={Zhang, Tao and Wei, Shiqing and Ji, Shunping},
  booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  pages={4443--4452},
  year={2022}
}

Code is largely based on Deep Snake. Thanks for their wonderful works.