DilatedNet in Keras for image segmentation

Overview

Keras implementation of DilatedNet for semantic segmentation

A native Keras implementation of semantic segmentation according to Multi-Scale Context Aggregation by Dilated Convolutions (2016). Optionally uses the pretrained weights by the authors'.

The code has been tested on Tensorflow 1.3, Keras 1.2, and Python 3.6.

Using the pretrained model

Download and extract the pretrained model:

curl -L https://github.com/nicolov/segmentation_keras/releases/download/model/nicolov_segmentation_model.tar.gz | tar xvf -

Install dependencies and run:

pip install -r requirements.txt
# For GPU support
pip install tensorflow-gpu==1.3.0

python predict.py --weights_path conversion/converted/dilation8_pascal_voc.npy

The output image will be under images/cat_seg.png.

Converting the original Caffe model

Follow the instructions in the conversion folder to convert the weights to the TensorFlow format that can be used by Keras.

Training

Download the Augmented Pascal VOC dataset here:

curl -L http://www.eecs.berkeley.edu/Research/Projects/CS/vision/grouping/semantic_contours/benchmark.tgz | tar -xvf -

This will create a benchmark_RELEASE directory in the root of the repo. Use the convert_masks.py script to convert the provided masks in .mat format to RGB pngs:

python convert_masks.py \
    --in-dir benchmark_RELEASE/dataset/cls \
    --out-dir benchmark_RELEASE/dataset/pngs

Start training:

python train.py --batch-size 2

Model checkpoints are saved under trained/, and can be used with the predict.py script for testing.

The training code is currently limited to the frontend module, and thus only outputs 16x16 segmentation maps. The augmentation pipeline does mirroring but not cropping or rotation.


Fisher Yu and Vladlen Koltun, Multi-Scale Context Aggregation by Dilated Convolutions, 2016

Comments
  • training and validation loss nan

    training and validation loss nan

    First of all I just want to thank you for the great work. I am having an issue during training, my loss and val_loss is nan, however I am still getting values for accuracy and val_acc. I am training on the PASCAL_VOC 2012 dataset with the segmentation class pngs. rsz_screenshot_from_2017-08-09_17-24-13

    keras 1.2.1 & 2.0.6 tensorflow-gpu 1.2.1 python 3.6.1

    opened by Barfknecht 9
  • Fine tuning ...

    Fine tuning ...

    Hello,

    You have provided the pre-trained model of VOC. I have a small dataset with 2 classes, which I annotated based on VOC and I want to fine-tune it. Would you please guide me through the process?

    opened by MyVanitar 8
  • Modifying number of class

    Modifying number of class

    Hi Nicolov,

    Thanks for the great work! I tried to train new dataset by generating my own set of jpg and png masks. However I realized it only works for pre-defined 20 classes. For example I wanted to re-train this network to segment screws from background, I wasn't able to find way to add new classes but to use a existed color 0x181818 which was originally trained for cats. After training it did segmented the screw. However I'm still wondering is there any way to change the number of classes and specify which color value are associated with certain class?

    opened by francisbitontistudio 7
  • Black image after segmentation

    Black image after segmentation

    Hi! I have val accuracy = 1, but when i am trying to predict mask on the image from train set it displays me black image. Does anybody know what is the reason of this behaviour?

    opened by dimaxano 7
  • docker running error

    docker running error

    Hi, @nicolov ,

    For the caffe weight conversion, I got the following error:

    (tf_1.0) [email protected]:/data/code/segmentation_keras/conversion# docker run -v $(pwd):/workspace -ti `docker build -q .`
    Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
    "docker run" requires at least 1 argument(s).
    See 'docker run --help'.
    
    Usage:  docker run [OPTIONS] IMAGE [COMMAND] [ARG...]
    
    Run a command in a new container
    (tf_1.0) ro[email protected]:/data/code/segmentation_keras/conversion#
    
    

    It shows that docker daemon is not running. Any other command should I input before it?

    Thanks

    opened by amiltonwong 7
  • the way of loading the weight

    the way of loading the weight

    Hi nicolov,

    In the post, you explained how to do the weight conversion. Due to the development environment constraints, it is a little bit hard for me to follow exactly your step.

    In keras blog, author also show a way to load VGG16 weight from Keras directly. Do you think this weight can be used for your implementation? Do we have to use the converted caffe model weight for pascal_voc. The data set I will be using is of different domain with the data set published in the paper. Thanks for your advice.

    capture

    opened by wenouyang 5
  • Problems with CuDNN library

    Problems with CuDNN library

    While running train.py, this is the error message: Epoch 1/20 E tensorflow/stream_executor/cuda/cuda_dnn.cc:378] Loaded runtime CuDNN library: 6021 (compatibility version 6000) but source was compiled with 5105 (compatibility version 5100). If using a binary install, upgrade your CuDNN library to match. If building from sources, make sure the library loaded at runtime matches a compatible version specified during compile configuration.

    Since I don't have the root account, I can't install CuDNN v5. Do you know how I can fix this? Thanks!

    opened by Yuren-Zhong 4
  • IoU results

    IoU results

    Have you by any chance compared this to the original implementation with regards to the mean IoU? If so, what implementation of IoU did you use and what were your results?

    opened by Barfknecht 4
  • about the required pre-trained vgg model

    about the required pre-trained vgg model

    Hi, @nicolov ,

    According to this line, vgg_conv.npy is needed as pre-trained vgg model in training. Could you list the download location for corresponding caffemodel and prototxt file? And, is the conversion step the same as here?

    Thanks!

    opened by amiltonwong 4
  • regarding loading_weights

    regarding loading_weights

    Hi nicolov,

    In the train.py, you have included the function of load_weights(model, weights_path):. My understanding is that you are trying to load a pre-training vcg model. If I do not want to use this pretrained model because the problem I am working one may belong to a totally different domain, should I just skip calling this load_weights function? Or using a pre-trained model is always preferable, I am kind of confusing about this.

    In the notes, you mentioned that The training code is currently limited to the frontend module, and thus only outputs 16x16 segmentation maps. If I would like to leverage this code for my own data set, what are the modifications that I have to make? Do I still have to load the weights?

    Thank you very much!

    opened by wenouyang 4
  • Cannot locate Dockerfile: Dockerfile

    Cannot locate Dockerfile: Dockerfile

    Probably a rookie error but when I am trying to run the conversion step in conversion by running the docker I get the following error:

    $sudo docker run -v $(pwd):/workspace -ti `docker build -q .`
    time="2017-02-09T09:15:11-08:00" level=fatal msg="Cannot locate Dockerfile: Dockerfile" 
    docker: "run" requires a minimum of 1 argument. See 'docker run --help'.
    
    opened by mongoose54 4
  • Training freezes

    Training freezes

    On executing command: python train.py --batch-size 2 ,training freezes at last step of first epoch.

    All the libraries are according to the requirement.txt file

    opened by ghost 1
  • AtrousConvolution2D vs.Conv2DTranspose

    AtrousConvolution2D vs.Conv2DTranspose

    Hi @nicolov I was wondering whether in your model, you wouldn't need to have a Conv2DTranspose or Upsample layer to compensate for the maxpool and obtain predictions with the same size as your input image?

    opened by tinalegre 0
  • How to handle high resolution  images

    How to handle high resolution images

    Hello @nicolov ,

    let me first express my appreciation to your work in image segmentation its great (Y)

    small suggestion , i just want to notify you that there is a missing -- in input parsing . very minor change

    parser.add_argument('--input_path', nargs='?', default='images/cat.jpg',
                            help='Required path to input image') 
    

    I'm hoping you can help me in understanding how to handle high res images as 1028 and 4k ,

    also in the code i found you set input_width, input_height = 900, 900 and label_margin = 186 can you please illustrate what is the reason for this static number and how they effect on the output high and width

    output_height = input_height - 2 * label_margin
    output_width = input_width - 2 * label_margin
    
    opened by engahmed1190 2
  • Context module training implementation plans

    Context module training implementation plans

    Thanks for creating this implementation. Do you have any plans to implement training of the context module (to allow producing full resolution segmentation maps)?

    opened by OliverColeman 3
  • palette conversion not needed

    palette conversion not needed

    https://github.com/nicolov/segmentation_keras/blob/master/convert_masks.py isn't necessary.

    Just use Pillow and you can load the classes separately from the color palette, which means it will already be in the format you want!

    from https://github.com/aurora95/Keras-FCN/blob/master/utils/SegDataGenerator.py#L203

                    from PIL import Image
                    label = Image.open(label_filepath)
                    if self.save_to_dir and self.palette is None:
                        self.palette = label.palette
    

    cool right?

    opened by ahundt 6
Releases(caffemodel)
PyTorch Lightning + Hydra. A feature-rich template for rapid, scalable and reproducible ML experimentation with best practices. ⚡🔥⚡

Lightning-Hydra-Template A clean and scalable template to kickstart your deep learning project 🚀 ⚡ 🔥 Click on Use this template to initialize new re

Łukasz Zalewski 2.1k Jan 09, 2023
You are AllSet: A Multiset Function Framework for Hypergraph Neural Networks.

AllSet This is the repo for our paper: You are AllSet: A Multiset Function Framework for Hypergraph Neural Networks. We prepared all codes and a subse

Jianhao 51 Dec 24, 2022
This is an official implementation of CvT: Introducing Convolutions to Vision Transformers.

Introduction This is an official implementation of CvT: Introducing Convolutions to Vision Transformers. We present a new architecture, named Convolut

Bin Xiao 175 Jan 08, 2023
Semi-Supervised Semantic Segmentation via Adaptive Equalization Learning, NeurIPS 2021 (Spotlight)

Semi-Supervised Semantic Segmentation via Adaptive Equalization Learning, NeurIPS 2021 (Spotlight) Abstract Due to the limited and even imbalanced dat

Hanzhe Hu 99 Dec 12, 2022
This is a Python Module For Encryption, Hashing And Other stuff

EnroCrypt This is a Python Module For Encryption, Hashing And Other Basic Stuff You Need, With Secure Encryption And Strong Salted Hashing You Can Do

5 Sep 15, 2022
Official Code Release for "CLIP-Adapter: Better Vision-Language Models with Feature Adapters"

Official Code Release for "CLIP-Adapter: Better Vision-Language Models with Feature Adapters" Pipeline of CLIP-Adapter CLIP-Adapter is a drop-in modul

peng gao 157 Dec 26, 2022
Multi-objective gym environments for reinforcement learning.

MO-Gym: Multi-Objective Reinforcement Learning Environments Gym environments for multi-objective reinforcement learning (MORL). The environments follo

Lucas Alegre 74 Jan 03, 2023
Res2Net for Instance segmentation and Object detection using MaskRCNN

Res2Net for Instance segmentation and Object detection using MaskRCNN Since the MaskRCNN-benchmark of facebook is deprecated, we suggest to use our mm

Res2Net Applications 55 Oct 30, 2022
Learning Super-Features for Image Retrieval

Learning Super-Features for Image Retrieval This repository contains the code for running our FIRe model presented in our ICLR'22 paper: @inproceeding

NAVER 101 Dec 28, 2022
Flexible Networks for Learning Physical Dynamics of Deformable Objects (2021)

Flexible Networks for Learning Physical Dynamics of Deformable Objects (2021) By Jinhyung Park, Dohae Lee, In-Kwon Lee from Yonsei University (Seoul,

Jinhyung Park 0 Jan 09, 2022
Official repository of DeMFI (arXiv.)

DeMFI This is the official repository of DeMFI (Deep Joint Deblurring and Multi-Frame Interpolation). [ArXiv_ver.] Coming Soon. Reference Jihyong Oh a

Jihyong Oh 56 Dec 14, 2022
Supervised Contrastive Learning for Product Matching

Contrastive Product Matching This repository contains the code and data download links to reproduce the experiments of the paper "Supervised Contrasti

Web-based Systems Group @ University of Mannheim 18 Dec 10, 2022
Hands-On Machine Learning for Algorithmic Trading, published by Packt

Hands-On Machine Learning for Algorithmic Trading Hands-On Machine Learning for Algorithmic Trading, published by Packt This is the code repository fo

Packt 981 Dec 29, 2022
Code for generating a single image pretraining dataset

Single Image Pretraining of Visual Representations As shown in the paper A critical analysis of self-supervision, or what we can learn from a single i

Yuki M. Asano 12 Dec 19, 2022
Code for "Finding Regions of Heterogeneity in Decision-Making via Expected Conditional Covariance" at NeurIPS 2021

Finding Regions of Heterogeneity in Decision-Making via Expected Conditional Covariance Justin Lim, Christina X Ji, Michael Oberst, Saul Blecker, Leor

Sontag Lab 3 Feb 03, 2022
Keras attention models including botnet,CoaT,CoAtNet,CMT,cotnet,halonet,resnest,resnext,resnetd,volo,mlp-mixer,resmlp,gmlp,levit

Keras_cv_attention_models Keras_cv_attention_models Usage Basic Usage Layers Model surgery AotNet ResNetD ResNeXt ResNetQ BotNet VOLO ResNeSt HaloNet

319 Dec 28, 2022
Self-Supervised Document-to-Document Similarity Ranking via Contextualized Language Models and Hierarchical Inference

Self-Supervised Document Similarity Ranking (SDR) via Contextualized Language Models and Hierarchical Inference This repo is the implementation for SD

Microsoft 36 Nov 28, 2022
Using Streamlit to host a multi-page tool with model specs and classification metrics, while also accepting user input values for prediction.

Predicitng_viability Using Streamlit to host a multi-page tool with model specs and classification metrics, while also accepting user input values for

Gopalika Sharma 1 Nov 08, 2021
QAT(quantize aware training) for classification with MQBench

MQBench Quantization Aware Training with PyTorch I am using MQBench(Model Quantization Benchmark)(http://mqbench.tech/) to quantize the model for depl

Ling Zhang 29 Nov 18, 2022
Official code of our work, Unified Pre-training for Program Understanding and Generation [NAACL 2021].

PLBART Code pre-release of our work, Unified Pre-training for Program Understanding and Generation accepted at NAACL 2021. Note. A detailed documentat

Wasi Ahmad 138 Dec 30, 2022