E2EC: An End-to-End Contour-based Method for High-Quality High-Speed Instance Segmentation

Related tags

Deep Learninge2ec
Overview

E2EC: An End-to-End Contour-based Method for High-Quality High-Speed Instance Segmentation

city

E2EC: An End-to-End Contour-based Method for High-Quality High-Speed Instance Segmentation
Tao Zhang, Shiqing Wei, Shunping Ji
CVPR 2022

Any questions or discussions are welcomed!

Installation

Please see INSTALL.md.

Performances

We re-tested the speed on a single RTX3090.

Dtataset AP Image size FPS
SBD val 59.2 512×512 59.60
COCO test-dev 33.8 original size 35.25
KINS val 34.0 768×2496 12.39
Cityscapes val 34.0 1216×2432 8.58

The accuracy and inference speed of the contours at different stages on SBD val set. We also re-tested the speed on a single RTX3090.

stage init coarse final final-dml
AP 51.4 55.9 58.8 59.2
FPS 101.73 91.35 67.48 59.6

The accuracy and inference speed of the contours at different stages on coco val set.

stage init coarse final final-dml
AP 27.8 31.6 33.5 33.6
FPS 80.97 72.81 42.55 35.25

Testing

Testing on COCO

  1. Download the pretrained model here or Baiduyun(password is e2ec).

  2. Prepared the COCO dataset according to the INSTALL.md.

  3. Test:

    # testing segmentation accuracy on coco val set
    python test.py coco --checkpoint /path/to/model_coco.pth --with_nms True
    # testing detection accuracy on coco val set
    python test.py coco --checkpoint /path/to/model_coco.pth --with_nms True --eval bbox
    # testing the speed
    python test.py coco --checkpoint /path/to/model_coco.pth --with_nms True --type speed
    # testing the contours of specified stage(init/coarse/final/final-dml)
    python test.py coco --checkpoint /path/to/model_coco.pth --with_nms True --stage coarse
    # testing on coco test-dev set, run and submit data/result/results.json
    python test.py coco --checkpoint /path/to/model_coco.pth --with_nms True --dataset coco_test
    

Testing on SBD

  1. Download the pretrained model here or Baiduyun(password is e2ec).

  2. Prepared the SBD dataset according to the INSTALL.md.

  3. Test:

    # testing segmentation accuracy on SBD
    python test.py sbd --checkpoint /path/to/model_sbd.pth
    # testing detection accuracy on SBD
    python test.py sbd --checkpoint /path/to/model_sbd.pth --eval bbox
    # testing the speed
    python test.py sbd --checkpoint /path/to/model_sbd.pth --type speed
    # testing the contours of specified stage(init/coarse/final/final-dml)
    python test.py sbd --checkpoint /path/to/model_sbd.pth --stage coarse
    

Testing on KINS

  1. Download the pretrained model here or Baiduyun(password is e2ec).

  2. Prepared the KINS dataset according to the INSTALL.md.

  3. Test:

    Maybe you will find some troules, such as object of type <class 'numpy.float64'> cannot be safely interpreted as an integer. Please modify the /path/to/site-packages/pycocotools/cooceval.py. Replace np.round((0.95 - .5) / .05) in lines 506 and 507 with int(np.round((0.95 - .5) / .05)).

    # testing segmentation accuracy on KINS
    python test.py kitti --checkpoint /path/to/model_kitti.pth
    # testing detection accuracy on KINS
    python test.py kitti --checkpoint /path/to/model_kitti.pth --eval bbox
    # testing the speed
    python test.py kitti --checkpoint /path/to/model_kitti.pth --type speed
    # testing the contours of specified stage(init/coarse/final/final-dml)
    python test.py kitti --checkpoint /path/to/model_kitti.pth --stage coarse
    

Testing on Cityscapes

  1. Download the pretrained model here or Baiduyun(password is e2ec).

  2. Prepared the KINS dataset according to the INSTALL.md.

  3. Test:

    We will soon release the code for e2ec with multi component detection. Currently only supported for testing e2ec performance on cityscapes dataset.

    # testing segmentation accuracy on Cityscapes with coco evaluator
    python test.py cityscapesCoco --checkpoint /path/to/model_cityscapes.pth
    # with cityscapes official evaluator
    python test.py cityscapes --checkpoint /path/to/model_cityscapes.pth
    # testing the detection accuracy
    python test.py cityscapesCoco \
    --checkpoint /path/to/model_cityscapes.pth --eval bbox
    # testing the speed
    python test.py cityscapesCoco \
    --checkpoint /path/to/model_cityscapes.pth --type speed
    # testing the contours of specified stage(init/coarse/final/final-dml)
    python test.py cityscapesCoco \
    --checkpoint /path/to/model_cityscapes.pth --stage coarse
    # testing on test set, run and submit the result file
    python test.py cityscapes --checkpoint /path/to/model_cityscapes.pth \
    --dataset cityscapes_test
    

Evaluate boundary AP

  1. Install the Boundary IOU API according boundary iou.

  2. Testing segmentation accuracy with coco evaluator.

  3. Using offline evaluation pipeline.

    python /path/to/boundary_iou_api/tools/coco_instance_evaluation.py \
        --gt-json-file /path/to/annotation_file.json \
        --dt-json-file data/result/result.json \
        --iou-type boundary
    

Visualization

  1. Download the pretrained model.

  2. Visualize:

    # inference and visualize the images with coco pretrained model
    python visualize.py coco /path/to/images \
    --checkpoint /path/to/model_coco.pth --with_nms True
    # you can using other pretrained model, such as cityscapes 
    python visualize.py cityscapesCoco /path/to/images \
    --checkpoint /path/to/model_cityscapes.pth
    # if you want to save the visualisation, please specify --output_dir
    python visualize.py coco /path/to/images \
    --checkpoint /path/to/model_coco.pth --with_nms True \
    --output_dir /path/to/output_dir
    # visualize the results at different stage
    python visualize.py coco /path/to/images \
    --checkpoint /path/to/model_coco.pth --with_nms True --stage coarse
    # you can reset the score threshold, default is 0.3
    python visualize.py coco /path/to/images \
    --checkpoint /path/to/model_coco.pth --with_nms True --ct_score 0.1
    # if you want to filter some of the jaggedness caused by dml 
    # please using post_process
    python visualize.py coco /path/to/images \
    --checkpoint /path/to/model_coco.pth --with_nms True \
    --with_post_process True
    

Training

We have only released the code for single GPU training, multi GPU training with ddp will be released soon.

Training on SBD

python train_net.py sbd --bs $batch_size
# if you do not want to use dinamic matching loss (significantly improves 
# contour detail but introduces jaggedness), please set --dml as False
python train_net.py sbd --bs $batch_size --dml False

Training on KINS

python train_net.py kitti --bs $batch_size

Training on Cityscapes

python train_net.py cityscapesCoco --bs $batch_size

Training on COCO

In fact it is possible to achieve the same accuracy without training so many epochs.

# first to train with adam
python train_net.py coco --bs $batch_size
# then finetune with sgd
python train_net.py coco_finetune --bs $batch_size \
--type finetune --checkpoint data/model/139.pth

Training on the other dataset

If the annotations is in coco style:

  1. Add dataset information to dataset/info.py.

  2. Modify the configs/coco.py, reset the train.dataset , model.heads['ct_hm'] and test.dataset. Maybe you also need to change the train.epochs, train.optimizer['milestones'] and so on.

  3. Train the network.

    python train_net.py coco --bs $batch_size
    

If the annotations is not in coco style:

  1. Prepare dataset/train/your_dataset.py and dataset/test/your_dataset.py by referring to dataset/train/base.py and dataset/test/base.py.

  2. Prepare evaluator/your_dataset/snake.py by referring to evaluator/coco/snake.py.

  3. Prepare configs/your_dataset.py and by referring to configs/base.py.

  4. Train the network.

    python train_net.py your_dataset --bs $batch_size
    

Citation

If you find this project helpful for your research, please consider citing using BibTeX below:

@article{zhang2022e2ec,
  title={E2EC: An End-to-End Contour-based Method for High-Quality High-Speed Instance Segmentation},
  author={Zhang, Tao and Wei, Shiqing and Ji, Shunping},
  journal={arXiv preprint arXiv:2203.04074},
  year={2022}
}

Acknowledgement

Code is largely based on Deep Snake. Thanks for their wonderful works.

Owner
zhangtao
zhangtao
Paddle implementation for "Cross-Lingual Word Embedding Refinement by ℓ1 Norm Optimisation" (NAACL 2021)

L1-Refinement Paddle implementation for "Cross-Lingual Word Embedding Refinement by ℓ1 Norm Optimisation" (NAACL 2021) 🙈 A more detailed readme is co

Lincedo Lab 4 Jun 09, 2021
Official Pytorch implementation of MixMo framework

MixMo: Mixing Multiple Inputs for Multiple Outputs via Deep Subnetworks Official PyTorch implementation of the MixMo framework | paper | docs Alexandr

79 Nov 07, 2022
Code for our CVPR2021 paper coordinate attention

Coordinate Attention for Efficient Mobile Network Design (preprint) This repository is a PyTorch implementation of our coordinate attention (will appe

Qibin (Andrew) Hou 726 Jan 05, 2023
This is the official repository of the paper Stocastic bandits with groups of similar arms (NeurIPS 2021). It contains the code that was used to compute the figures and experiments of the paper.

Experiments How to reproduce experimental results of Stochastic bandits with groups of similar arms submitted paper ? Section 5 of the paper To reprod

Fabien 0 Oct 25, 2021
ESP32 python application to read data from a Tilt™ Hydrometer for homebrewing

TitlESP32 ESP32 MicroPython application to read and log data from a Tilt™ Hydrometer. Requirements A board with an ESP32 chip USB cable - USB A / micr

IoBeer 5 Dec 01, 2022
Neural Network Libraries

Neural Network Libraries Neural Network Libraries is a deep learning framework that is intended to be used for research, development and production. W

Sony 2.6k Dec 30, 2022
PyTorch implementation of "Dataset Knowledge Transfer for Class-Incremental Learning Without Memory" (WACV2022)

Dataset Knowledge Transfer for Class-Incremental Learning Without Memory [Paper] [Slides] Summary Introduction Installation Reproducing results Citati

Habib Slim 5 Dec 05, 2022
Probabilistic Cross-Modal Embedding (PCME) CVPR 2021

Probabilistic Cross-Modal Embedding (PCME) CVPR 2021 Official Pytorch implementation of PCME | Paper Sanghyuk Chun1 Seong Joon Oh1 Rafael Sampaio de R

NAVER AI 87 Dec 21, 2022
Clustergram - Visualization and diagnostics for cluster analysis in Python

Clustergram Visualization and diagnostics for cluster analysis Clustergram is a diagram proposed by Matthias Schonlau in his paper The clustergram: A

Martin Fleischmann 96 Dec 26, 2022
Official PyTorch implementation of the paper "Graph-based Generative Face Anonymisation with Pose Preservation" in ICIAP 2021

Contents AnonyGAN Installation Dataset Preparation Generating Images Using Pretrained Model Train and Test New Models Evaluation Acknowledgments Citat

Nicola Dall'Asen 10 May 24, 2022
This project deploys a yolo fastest model in the form of tflite on raspberry 3b+. The model is from another repository of mine called -Trash-Classification-Car

Deploy-yolo-fastest-tflite-on-raspberry 觉得有用的话可以顺手点个star嗷 这个项目将垃圾分类小车中的tflite模型移植到了树莓派3b+上面。 该项目主要是为了记录在树莓派部署yolo fastest tflite的流程 (之后有时间会尝试用C++部署来提升

7 Aug 16, 2022
PyKale is a PyTorch library for multimodal learning and transfer learning as well as deep learning and dimensionality reduction on graphs, images, texts, and videos

PyKale is a PyTorch library for multimodal learning and transfer learning as well as deep learning and dimensionality reduction on graphs, images, texts, and videos. By adopting a unified pipeline-ba

PyKale 370 Dec 27, 2022
This package proposes simplified exporting pytorch models to ONNX and TensorRT, and also gives some base interface for model inference.

PyTorch Infer Utils This package proposes simplified exporting pytorch models to ONNX and TensorRT, and also gives some base interface for model infer

Alex Gorodnitskiy 11 Mar 20, 2022
Repository for tackling Kaggle Ultrasound Nerve Segmentation challenge using Torchnet.

Ultrasound Nerve Segmentation Challenge using Torchnet This repository acts as a starting point for someone who wants to start with the kaggle ultraso

Qure.ai 46 Jul 18, 2022
📚 A collection of all the Deep Learning Metrics that I came across which are not accuracy/loss.

📚 A collection of all the Deep Learning Metrics that I came across which are not accuracy/loss.

Rahul Vigneswaran 1 Jan 17, 2022
Augmenting Physical Models with Deep Networks for Complex Dynamics Forecasting

Official code of APHYNITY Augmenting Physical Models with Deep Networks for Complex Dynamics Forecasting (ICLR 2021, Oral) Yuan Yin*, Vincent Le Guen*

Yuan Yin 24 Oct 24, 2022
torchbearer: A model fitting library for PyTorch

Note: We're moving to PyTorch Lightning! Read about the move here. From the end of February, torchbearer will no longer be actively maintained. We'll

631 Jan 04, 2023
Learning Dynamic Network Using a Reuse Gate Function in Semi-supervised Video Object Segmentation.

Training Script for Reuse-VOS This code implementation of CVPR 2021 paper : Learning Dynamic Network Using a Reuse Gate Function in Semi-supervised Vi

HYOJINPARK 22 Jan 01, 2023
Official implementation of "Not only Look, but also Listen: Learning Multimodal Violence Detection under Weak Supervision" ECCV2020

XDVioDet Official implementation of "Not only Look, but also Listen: Learning Multimodal Violence Detection under Weak Supervision" ECCV2020. The proj

peng 64 Dec 12, 2022
BackgroundRemover lets you Remove Background from images and video with a simple command line interface

BackgroundRemover BackgroundRemover is a command line tool to remove background from video and image, made by nadermx to power https://BackgroundRemov

Johnathan Nader 1.7k Dec 30, 2022