ICCV2021 - Mining Contextual Information Beyond Image for Semantic Segmentation

Related tags

Deep Learningmcibi
Overview

Introduction

The official repository for "Mining Contextual Information Beyond Image for Semantic Segmentation". Our full code has been merged into sssegmentation.

Abstract

This paper studies the context aggregation problem in semantic image segmentation. The existing researches focus on improving the pixel representations by aggregating the contextual information within individual images. Though impressive, these methods neglect the significance of the representations of the pixels of the corresponding class beyond the input image. To address this, this paper proposes to mine the contextual information beyond individual images to further augment the pixel representations. We first set up a feature memory module, which is updated dynamically during training, to store the dataset-level representations of various categories. Then, we learn class probability distribution of each pixel representation under the supervision of the ground-truth segmentation. At last, the representation of each pixel is augmented by aggregating the dataset-level representations based on the corresponding class probability distribution. Furthermore, by utilizing the stored dataset-level representations, we also propose a representation consistent learning strategy to make the classification head better address intra-class compactness and inter-class dispersion. The proposed method could be effortlessly incorporated into existing segmentation frameworks (e.g., FCN, PSPNet, OCRNet and DeepLabV3) and brings consistent performance improvements. Mining contextual information beyond image allows us to report state-of-the-art performance on various benchmarks: ADE20K, LIP, Cityscapes and COCO-Stuff.

Framework

img

Performance

COCOStuff-10k

Model Backbone Crop Size Schedule Train/Eval Set mIoU/mIoU (ms+flip) Download
DeepLabV3 R-50-D8 512x512 LR/POLICY/BS/EPOCH: 0.001/poly/16/110 train/test 38.84%/39.68% model | log
DeepLabV3 R-101-D8 512x512 LR/POLICY/BS/EPOCH: 0.001/poly/16/110 train/test 39.84%/41.49% model | log
DeepLabV3 S-101-D8 512x512 LR/POLICY/BS/EPOCH: 0.001/poly/32/150 train/test 41.18%/42.15% model | log
DeepLabV3 HRNetV2p-W48 512x512 LR/POLICY/BS/EPOCH: 0.001/poly/16/110 train/test 39.77%/41.35% model | log
DeepLabV3 ViT-Large 512x512 LR/POLICY/BS/EPOCH: 0.001/poly/16/110 train/test 44.01%/45.23% model | log

ADE20k

Model Backbone Crop Size Schedule Train/Eval Set mIoU/mIoU (ms+flip) Download
DeepLabV3 R-50-D8 512x512 LR/POLICY/BS/EPOCH: 0.01/poly/16/130 train/val 44.39%/45.95% model | log
DeepLabV3 R-101-D8 512x512 LR/POLICY/BS/EPOCH: 0.01/poly/16/130 train/val 45.66%/47.22% model | log
DeepLabV3 S-101-D8 512x512 LR/POLICY/BS/EPOCH: 0.004/poly/16/180 train/val 46.63%/47.36% model | log
DeepLabV3 HRNetV2p-W48 512x512 LR/POLICY/BS/EPOCH: 0.004/poly/16/180 train/val 45.79%/47.34% model | log
DeepLabV3 ViT-Large 512x512 LR/POLICY/BS/EPOCH: 0.01/poly/16/130 train/val 49.73%/50.99% model | log

CityScapes

Model Backbone Crop Size Schedule Train/Eval Set mIoU (ms+flip) Download
DeepLabV3 R-50-D8 512x1024 LR/POLICY/BS/EPOCH: 0.01/poly/16/440 trainval/test 79.90% model | log
DeepLabV3 R-101-D8 512x1024 LR/POLICY/BS/EPOCH: 0.01/poly/16/440 trainval/test 82.03% model | log
DeepLabV3 S-101-D8 512x1024 LR/POLICY/BS/EPOCH: 0.01/poly/16/500 trainval/test 81.59% model | log
DeepLabV3 HRNetV2p-W48 512x1024 LR/POLICY/BS/EPOCH: 0.01/poly/16/500 trainval/test 82.55% model | log

LIP

Model Backbone Crop Size Schedule Train/Eval Set mIoU/mIoU (flip) Download
DeepLabV3 R-50-D8 473x473 LR/POLICY/BS/EPOCH: 0.01/poly/32/150 train/val 53.73%/54.08% model | log
DeepLabV3 R-101-D8 473x473 LR/POLICY/BS/EPOCH: 0.01/poly/32/150 train/val 55.02%/55.42% model | log
DeepLabV3 S-101-D8 473x473 LR/POLICY/BS/EPOCH: 0.007/poly/40/150 train/val 56.21%/56.34% model | log
DeepLabV3 HRNetV2p-W48 473x473 LR/POLICY/BS/EPOCH: 0.007/poly/40/150 train/val 56.40%/56.99% model | log

Citation

If this code is useful for your research, please consider citing:

@article{jin2021mining,
  title={Mining Contextual Information Beyond Image for Semantic Segmentation},
  author={Jin, Zhenchao and Gong, Tao and Yu, Dongdong and Chu, Qi and Wang, Jian and Wang, Changhu and Shao, Jie},
  journal={arXiv preprint arXiv:2108.11819},
  year={2021}
}
Owner
student
PyTorch code of my WACV 2022 paper Improving Model Generalization by Agreement of Learned Representations from Data Augmentation

Improving Model Generalization by Agreement of Learned Representations from Data Augmentation (WACV 2022) Paper ArXiv Why it matters? When data augmen

Rowel Atienza 5 Mar 04, 2022
Trax — Deep Learning with Clear Code and Speed

Trax — Deep Learning with Clear Code and Speed Trax is an end-to-end library for deep learning that focuses on clear code and speed. It is actively us

Google 7.3k Dec 26, 2022
Аналитика доходности инвестиционного портфеля в Тинькофф брокере

Аналитика доходности инвестиционного портфеля Тиньков Видео на YouTube Для работы скрипта нужно установить три переменных окружения: export TINKOFF_TO

Alexey Goloburdin 64 Dec 17, 2022
The Power of Scale for Parameter-Efficient Prompt Tuning

The Power of Scale for Parameter-Efficient Prompt Tuning Implementation of soft embeddings from https://arxiv.org/abs/2104.08691v1 using Pytorch and H

Kip Parker 208 Dec 30, 2022
The project is an official implementation of our paper "3D Human Pose Estimation with Spatial and Temporal Transformers".

3D Human Pose Estimation with Spatial and Temporal Transformers This repo is the official implementation for 3D Human Pose Estimation with Spatial and

Ce Zheng 363 Dec 28, 2022
UDP++ (ECCVW 2020 Oral), (Winner of COCO 2020 Keypoint Challenge).

UDP-Pose This is the pytorch implementation for UDP++, which won the Fisrt place in COCO Keypoint Challenge at ECCV 2020 Workshop. Top-Down Results on

20 Jul 29, 2022
Official PyTorch implementation of Joint Object Detection and Multi-Object Tracking with Graph Neural Networks

This is the official PyTorch implementation of our paper: "Joint Object Detection and Multi-Object Tracking with Graph Neural Networks". Our project website and video demos are here.

Richard Wang 443 Dec 06, 2022
Crowd-Kit is a powerful Python library that implements commonly-used aggregation methods for crowdsourced annotation and offers the relevant metrics and datasets

Crowd-Kit: Computational Quality Control for Crowdsourcing Documentation Crowd-Kit is a powerful Python library that implements commonly-used aggregat

Toloka 125 Dec 30, 2022
AlphaNet Improved Training of Supernet with Alpha-Divergence

AlphaNet: Improved Training of Supernet with Alpha-Divergence This repository contains our PyTorch training code, evaluation code and pretrained model

Facebook Research 87 Oct 10, 2022
Official repository of ICCV21 paper "Viewpoint Invariant Dense Matching for Visual Geolocalization"

Viewpoint Invariant Dense Matching for Visual Geolocalization: PyTorch implementation This is the implementation of the ICCV21 paper: G Berton, C. Mas

Gabriele Berton 44 Jan 03, 2023
Code for our TKDE paper "Understanding WeChat User Preferences and “Wow” Diffusion"

wechat-wow-analysis Understanding WeChat User Preferences and “Wow” Diffusion. Fanjin Zhang, Jie Tang, Xueyi Liu, Zhenyu Hou, Yuxiao Dong, Jing Zhang,

18 Sep 16, 2022
Example of a Quantum LSTM

Example of a Quantum LSTM

Riccardo Di Sipio 36 Oct 31, 2022
Code implementation from my Medium blog post: [Transformers from Scratch in PyTorch]

transformer-from-scratch Code for my Medium blog post: Transformers from Scratch in PyTorch Note: This Transformer code does not include masked attent

Frank Odom 27 Dec 21, 2022
Adaptive FNO transformer - official Pytorch implementation

Adaptive Fourier Neural Operators: Efficient Token Mixers for Transformers This repository contains PyTorch implementation of the Adaptive Fourier Neu

NVIDIA Research Projects 77 Dec 29, 2022
Solutions and questions for AoC2021. Merry christmas!

Advent of Code 2021 Merry christmas! 🎄 🎅 To get solutions and approximate execution times for implementations, please execute the run.py script in t

Wilhelm Ågren 5 Dec 29, 2022
Strongly local p-norm-cut algorithms for semi-supervised learning and local graph clustering

Strongly local p-norm-cut algorithms for semi-supervised learning and local graph clustering

Meng Liu 2 Jul 19, 2022
[ECE NTUA] 👁 Computer Vision - Lab Projects & Theoretical Problem Sets (2020-2021)

Computer Vision - NTUA (2020-2021) This repository hosts the lab projects and theoretical problem sets of the Computer Vision course held by ECE NTUA

Dimitris Dimos 6 Jul 21, 2022
LocUNet is a deep learning method to localize a UE based solely on the reported signal strengths from a set of BSs.

LocUNet LocUNet is a deep learning method to localize a UE based solely on the reported signal strengths from a set of BSs. The method utilizes accura

4 Oct 05, 2022
Image Captioning on google cloud platform based on iot

Image-Captioning-on-google-cloud-platform-based-on-iot - Image Captioning on google cloud platform based on iot

Shweta_kumawat 1 Jan 20, 2022
Libraries, tools and tasks created and used at DeepMind Robotics.

dm_robotics: Libraries, tools, and tasks created and used for Robotics research at DeepMind. Package overview Package Summary Transformations Rigid bo

DeepMind 273 Jan 06, 2023