FastFCN: Rethinking Dilated Convolution in the Backbone for Semantic Segmentation.

Related tags

Deep LearningFastFCN
Overview

FastFCN: Rethinking Dilated Convolution in the Backbone for Semantic Segmentation

[Project] [Paper] [arXiv] [Home]

PWC

Official implementation of FastFCN: Rethinking Dilated Convolution in the Backbone for Semantic Segmentation.
A Faster, Stronger and Lighter framework for semantic segmentation, achieving the state-of-the-art performance and more than 3x acceleration.

@inproceedings{wu2019fastfcn,
  title     = {FastFCN: Rethinking Dilated Convolution in the Backbone for Semantic Segmentation},
  author    = {Wu, Huikai and Zhang, Junge and Huang, Kaiqi and Liang, Kongming and Yu Yizhou},
  booktitle = {arXiv preprint arXiv:1903.11816},
  year = {2019}
}

Contact: Hui-Kai Wu ([email protected])

Update

2020-04-15: Now support inference on a single image !!!

CUDA_VISIBLE_DEVICES=0,1,2,3 python -m experiments.segmentation.test_single_image --dataset [pcontext|ade20k] \
    --model [encnet|deeplab|psp] --jpu [JPU|JPU_X] \
    --backbone [resnet50|resnet101] [--ms] --resume {MODEL} --input-path {INPUT} --save-path {OUTPUT}

2020-04-15: New joint upsampling module is now available !!!

  • --jpu [JPU|JPU_X]: JPU is the original module in the arXiv paper; JPU_X is a pyramid version of JPU.

2020-02-20: FastFCN can now run on every OS with PyTorch>=1.1.0 and Python==3.*.*

  • Replace all C/C++ extensions with pure python extensions.

Version

  1. Original code, producing the results reported in the arXiv paper. [branch:v1.0.0]
  2. Pure PyTorch code, with torch.nn.DistributedDataParallel and torch.nn.SyncBatchNorm. [branch:latest]
  3. Pure Python code. [branch:master]

Overview

Framework

Joint Pyramid Upsampling (JPU)

Install

  1. PyTorch >= 1.1.0 (Note: The code is test in the environment with python=3.6, cuda=9.0)
  2. Download FastFCN
    git clone https://github.com/wuhuikai/FastFCN.git
    cd FastFCN
    
  3. Install Requirements
    nose
    tqdm
    scipy
    cython
    requests
    

Train and Test

PContext

python -m scripts.prepare_pcontext
Method Backbone mIoU FPS Model Scripts
EncNet ResNet-50 49.91 18.77
EncNet+JPU (ours) ResNet-50 51.05 37.56 GoogleDrive bash
PSP ResNet-50 50.58 18.08
PSP+JPU (ours) ResNet-50 50.89 28.48 GoogleDrive bash
DeepLabV3 ResNet-50 49.19 15.99
DeepLabV3+JPU (ours) ResNet-50 50.07 20.67 GoogleDrive bash
EncNet ResNet-101 52.60 (MS) 10.51
EncNet+JPU (ours) ResNet-101 54.03 (MS) 32.02 GoogleDrive bash

ADE20K

python -m scripts.prepare_ade20k

Training Set

Method Backbone mIoU (MS) Model Scripts
EncNet ResNet-50 41.11
EncNet+JPU (ours) ResNet-50 42.75 GoogleDrive bash
EncNet ResNet-101 44.65
EncNet+JPU (ours) ResNet-101 44.34 GoogleDrive bash

Training Set + Val Set

Method Backbone FinalScore (MS) Model Scripts
EncNet+JPU (ours) ResNet-50 GoogleDrive bash
EncNet ResNet-101 55.67
EncNet+JPU (ours) ResNet-101 55.84 GoogleDrive bash

Note: EncNet (ResNet-101) is trained with crop_size=576, while EncNet+JPU (ResNet-101) is trained with crop_size=480 for fitting 4 images into a 12G GPU.

Visual Results

Dataset Input GT EncNet Ours
PContext
ADE20K

More Visual Results

Acknowledgement

Code borrows heavily from PyTorch-Encoding.

Comments
  • Some problem when running test.py and train.py

    Some problem when running test.py and train.py

    Hi, I am a beginner in deep learning. Some problem occurred when I was running the code. First, I use the command 「 tar -xvf encnet_jpu_res50_pcontext.pth.tar 」 to extract the tar file, but it fails. Second, if i successfully extract the file and get checkpoint, which file should I put my checkpoint in ? Where should I extract my checkpoint file to? Thank You!

    opened by pp00704831 18
  • why i remove JPU,I also can  train model?

    why i remove JPU,I also can train model?

    Why does the code still execute without error when I delete the JPU module?(/FastFCN/encoding/nn/customize.py),I also can train model? These are my commands :(I did load the JPU module) CUDA_VISIBLE_DEVICES=4,5,6,7 python train.py --dataset pcontext --model encnet --jpu --aux --se-loss --backbone resnet101 --checkname encnet_res101_pcontext

    opened by E18301194 17
  • Segmentation fault

    Segmentation fault

    I think this problem is caused by my previous pytorch problem,so maybe i have to solve pytorch first.Could you give me some help? gcc:4.8 pytorch:1.1.0 python:3.5 and how could i change the pytorch version to 1.0.0?pip install torch==1.0?

    opened by Anikily 12
  • Performance Issue

    Performance Issue

    Thanks for your work. I have tried this script: https://github.com/wuhuikai/FastFCN/blob/master/experiments/segmentation/scripts/encnet_res50_pcontext.sh with the hardware and software: 4xTitanXp, Ubuntu16.04, CUDA9.0, PyToch1.0

    But I can't reproduce the performance reported in your paper. I got pixAcc: 0.7747, mIoU: 0.4785 for single-scale, and pixAcc: 0.7833, mIoU: 0.4898 for multi-scale.

    I would appreciate your help. Thanks for your consideration.

    bug 
    opened by tonysy 12
  • FastFCN has been supported by MMSegmentation.

    FastFCN has been supported by MMSegmentation.

    Hi, right now FastFCN has been supported by MMSegmentation. We do find using JPU with smaller feature maps from backbone could get similar or higher performance than original models with larger feature maps.

    There is still something to do for us, for example, we do not find obviously improvement about FPS in our implementation, thus we would try to figure it out in the future.

    Anyway, thanks for your work and hope more people from community could use FastFCN.

    Best,

    opened by MengzhangLI 9
  • RuntimeError: Failed downloading

    RuntimeError: Failed downloading

    Hi, thanks for your work. I try to run your code to train a model on the pascalContext dataset.But I got the following error: RuntimeError: Failed downloading url https://hangzh.s3.amazonaws.com/encoding/models/resnet50-ebb6acbb.zip I find the problem is I can not download the pretrained model. I find the author no longer provide the pretrained resnet model. https://github.com/zhanghang1989/PyTorch-Encoding/issues/273

    So, How can I solve this problem. Thanks for your consideration.

    opened by bufferXia 9
  • How could I set

    How could I set "resume" while running test_single_image?

    Hello!

    When I run test_single_image.py, I tried to set resume as path of resnet101-2a57e44d.pth and encountered an error.

    File "G:/gitfolder/FastFCN/experiments/segmentation/test_single_image.py", line 43, in test model.load_state_dict(checkpoint['state_dict'], strict=False) KeyError: 'state_dict

    I doubted that there existed a problem with "resume". Waiting for your reply.

    Thank you!

    opened by CN-HaoJiang 8
  • Questions about the SE-loss and  Aux-loss

    Questions about the SE-loss and Aux-loss

    Hi, first thank you for the great work. I just checked the codes and also had run some scripts. I am confused with the final loss which is composited with three individual losses. could you tell what is the se-loss and the aux-loss used for.

    opened by meanmee 7
  • Backbone weights download links not working anymore

    Backbone weights download links not working anymore

    Download links for the backbone do not seem to work anymore.

    I've tested with Resnet50 (https://hangzh.s3.amazonaws.com/encoding/models/resnet50-ebb6acbb.zip) and Resnet 101 (https://hangzh.s3.amazonaws.com/encoding/models/resnet101-2a57e44d.zip) too.

    I also tried to use torchivision weights instead, but I got matching errors when trying to load them.

    Could you consider reuploading the weights? That would be very helpful!

    opened by Khroto 6
  • Segmentation Fault

    Segmentation Fault

    我執行以下 command 準備 train model 但是發生 segmentation fault 有人有這個問題嗎 ? 謝謝幫忙 !

    run : CUDA_VISIBLE_DEVICES=0,1,2,3 python train.py --dataset pcontext --model encnet --jpu --aux --se-loss --backbone resnet101 --checkname encnet_res101_pcontext

    crashed : Using poly LR Scheduler! Starting Epoch: 0 Total Epoches: 80 0%| | 0/312 [00:00<?, ?it/s] =>Epoches 0, learning rate = 0.0010, previous best = 0.0000 Segmentation fault

    //------------ Nvidia GPU : Tesla P100-PCIE 16G x 4 CPU : GenuineIntel x 18 , Memory 140G totally

    opened by SimonTsungHanKuo 6
  • Need your suggestions

    Need your suggestions

    Hi, i have designed this SPP module for my network. But i am also interested in your work to replace my his module with JPU. Would you like to give me any suggestions? here is my implementation

    class SPP(nn.Module): def init(self, pool_sizes): super(SPP, self).init() self.pool_sizes = pool_sizes

    def forward(self, x):
        h, w = x.shape[2:]
        k_sizes = []
        strides = []
        for pool_size in self.pool_sizes:
            k_sizes.append((int(h / pool_size), int(w / pool_size)))
            strides.append((int(h / pool_size), int(w / pool_size)))
    
        spp_sum = x
    
        for i in range(len(self.pool_sizes)):
            out = F.avg_pool2d(x, k_sizes[i], stride=strides[i], padding=0)
            out = F.upsample(out, size=(h, w), mode="bilinear")
            spp_sum = spp_sum + out
    
        return spp_sum  
    
    opened by haideralimughal 5
  • add resnest and xception65

    add resnest and xception65

    Copy Resnest and xception65 from Pytorch-Encoding, and xception65 only can be used without pretrained models.

    Pls be careful as there are many changes!!

    I test it on my own server, and everything seems ok. As a caution, maybe you could test it by yourself first.My FastFCN

    I don't change the Readme.md and *.sh. Maybe you can rectify it if you agree this request.

    If the server resources are not tight, I will run the encnet+jpu+resnest101+pcontext and encnet+jpu_x+resnest101+pcontext, I will share you the results at issues or pull another request about Readme.md with my pth.tar.

    Thanks for your work again.

    opened by tjj1998 1
Releases(v1.0.0)
Code for Discriminative Sounding Objects Localization (NeurIPS 2020)

Discriminative Sounding Objects Localization Code for our NeurIPS 2020 paper Discriminative Sounding Objects Localization via Self-supervised Audiovis

51 Dec 11, 2022
Fine-Tune EleutherAI GPT-Neo to Generate Netflix Movie Descriptions in Only 47 Lines of Code Using Hugginface And DeepSpeed

GPT-Neo-2.7B Fine-Tuning Example Using HuggingFace & DeepSpeed Installation cd venv/bin ./pip install -r ../../requirements.txt ./pip install deepspe

Nikita 180 Jan 05, 2023
A python bot to move your mouse every few seconds to appear active on Skype, Teams or Zoom as you go AFK. 🐭 🤖

PyMouseBot If you're from GT and annoyed with SGVPN idle timeouts while working on development laptop, You might find this useful. A python cli bot to

Oaker Min 6 Oct 24, 2022
[SIGGRAPH Asia 2021] Pose with Style: Detail-Preserving Pose-Guided Image Synthesis with Conditional StyleGAN

Pose with Style: Detail-Preserving Pose-Guided Image Synthesis with Conditional StyleGAN [Paper] [Project Website] [Output resutls] Official Pytorch i

Badour AlBahar 215 Dec 17, 2022
PyTorch implementation of SQN based on CloserLook3D's encoder

SQN_pytorch This repo is an implementation of Semantic Query Network (SQN) using CloserLook3D's encoder in Pytorch. For TensorFlow implementation, che

PointCloudYC 1 Oct 21, 2021
Multimodal Temporal Context Network (MTCN)

Multimodal Temporal Context Network (MTCN) This repository implements the model proposed in the paper: Evangelos Kazakos, Jaesung Huh, Arsha Nagrani,

Evangelos Kazakos 13 Nov 24, 2022
This is project is the implementation of the DeepShift: Towards Multiplication-Less Neural Networks paper

DeepShift This is project is the implementation of the DeepShift: Towards Multiplication-Less Neural Networks paper, that aims to replace multiplicati

Mostafa Elhoushi 88 Dec 23, 2022
RETRO-pytorch - Implementation of RETRO, Deepmind's Retrieval based Attention net, in Pytorch

RETRO - Pytorch (wip) Implementation of RETRO, Deepmind's Retrieval based Attent

Phil Wang 556 Jan 04, 2023
PyTorch implementation of Neural Dual Contouring.

NDC PyTorch implementation of Neural Dual Contouring. Citation We are still writing the paper while adding more improvements and applications. If you

Zhiqin Chen 140 Dec 26, 2022
Official repo for SemanticGAN https://nv-tlabs.github.io/semanticGAN/

SemanticGAN This is the official code for: Semantic Segmentation with Generative Models: Semi-Supervised Learning and Strong Out-of-Domain Generalizat

151 Dec 28, 2022
Rotated Box Is Back : Accurate Box Proposal Network for Scene Text Detection

Rotated Box Is Back : Accurate Box Proposal Network for Scene Text Detection This material is supplementray code for paper accepted in ICDAR 2021 We h

NCSOFT 30 Dec 21, 2022
StackNet is a computational, scalable and analytical Meta modelling framework

StackNet This repository contains StackNet Meta modelling methodology (and software) which is part of my work as a PhD Student in the computer science

Marios Michailidis 1.3k Dec 15, 2022
Benchmarking the robustness of Spatial-Temporal Models

Benchmarking the robustness of Spatial-Temporal Models This repositery contains the code for the paper Benchmarking the Robustness of Spatial-Temporal

Yi Chenyu Ian 15 Dec 16, 2022
Node Editor Plug for Blender

NodeEditor Blender的程序化建模插件 Show Current 基本框架:自定义的tree-node-socket、tree中的node与socket采用字典查询、基于socket入度的拓扑排序 数据传递和处理依靠Tree中的字典,socket传递字典key TODO 增加更多的节点

Cuimi 11 Dec 03, 2022
A tutorial on training a DarkNet YOLOv4 model for the CrowdHuman dataset

YOLOv4 CrowdHuman Tutorial This is a tutorial demonstrating how to train a YOLOv4 people detector using Darknet and the CrowdHuman dataset. Table of c

JK Jung 118 Nov 10, 2022
Neural machine translation between the writings of Shakespeare and modern English using TensorFlow

Shakespeare translations using TensorFlow This is an example of using the new Google's TensorFlow library on monolingual translation going from modern

Motoki Wu 245 Dec 28, 2022
Pytorch implementation of Decoupled Spatial-Temporal Transformer for Video Inpainting

Decoupled Spatial-Temporal Transformer for Video Inpainting By Rui Liu, Hanming Deng, Yangyi Huang, Xiaoyu Shi, Lewei Lu, Wenxiu Sun, Xiaogang Wang, J

51 Dec 13, 2022
Code for the paper "PortraitNet: Real-time portrait segmentation network for mobile device" @ CAD&Graphics2019

PortraitNet Code for the paper "PortraitNet: Real-time portrait segmentation network for mobile device". @ CAD&Graphics 2019 Introduction We propose a

265 Dec 01, 2022
RTSeg: Real-time Semantic Segmentation Comparative Study

Real-time Semantic Segmentation Comparative Study The repository contains the official TensorFlow code used in our papers: RTSEG: REAL-TIME SEMANTIC S

Mennatullah Siam 592 Nov 18, 2022
The official implementation of Autoregressive Image Generation using Residual Quantization (CVPR '22)

Autoregressive Image Generation using Residual Quantization (CVPR 2022) The official implementation of "Autoregressive Image Generation using Residual

Kakao Brain 529 Dec 30, 2022