Hi. First of all thanks for your repo! Have you considered moving to tf.keras instead of using separate keras? It would make your repo compatible with new TF 2.0 out of the box while still being compatible with older versions of TF as well.

Hope you can show an example on how to assign class_weights for each class in loss function. I tried with class_weights = [1,10,50,10,50] its return error due to tensor shape. My data is one-hot-encoded for all 5 classes. I am trying to use the PSPNet with class_weights.

Hi! I was working FPN with 'resnext101' backbone on Google Colab. I've trained the model and have done lots of experiments and the results were very good. Today, after I updated the segmentation models (actually, every time I use Google Colab, I have to reinstall it) I got the following error shown below. By the way, I tried to use Unet with 'vgg16' backbone and everything went well. I wonder why FPN with resnext101 backbone does not fit GPU memory as it fit two days ago.

Thank you very much @qubvel .

Edit1: FPN with vgg16 backbone is OK. FPN with vgg19 backbone is OK. FPN with resnet34 backbone is OK. FPN with resnet50 backbone is NOT OK (The same error is shown below). FPN with resnet101 backbone is NOT OK (The same error is shown below). FPN with resnext50 backbone is NOT OK (The same error is shown below).

Edit2: The related StackOverflow question.

Epoch 1/100
---------------------------------------------------------------------------
ResourceExhaustedError                    Traceback (most recent call last)
<ipython-input-22-1b2892f8cab2> in <module>()
----> 1 get_ipython().run_cell_magic('time', '', 'history = model.fit_generator(\n    generator = zipped_train_generator,\n  validation_data=(X_validation, y_validation),\n    steps_per_epoch=len(X_train) // NUM_BATCH,\n    callbacks= callbacks_list,\n    verbose = 1,\n    epochs = NUM_EPOCH)')

9 frames
</usr/local/lib/python3.6/dist-packages/decorator.py:decorator-gen-60> in time(self, line, cell, local_ns)

<timed exec> in <module>()

/usr/local/lib/python3.6/dist-packages/tensorflow/python/client/session.py in __call__(self, *args, **kwargs)
   1456         ret = tf_session.TF_SessionRunCallable(self._session._session,
   1457                                                self._handle, args,
-> 1458                                                run_metadata_ptr)
   1459         if run_metadata:
   1460           proto_data = tf_session.TF_GetBuffer(run_metadata_ptr)

ResourceExhaustedError: 2 root error(s) found.
  (0) Resource exhausted: OOM when allocating tensor with shape[32,128,112,112] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc
	 [[{{node training/RMSprop/gradients/zeros_21}}]]
Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.

	 [[loss/mul/_11081]]
Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.

  (1) Resource exhausted: OOM when allocating tensor with shape[32,128,112,112] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc
	 [[{{node training/RMSprop/gradients/zeros_21}}]]
Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.

0 successful operations.
0 derived errors ignored.

I tried to train a PSPNet and a VPN for a multi-class segmentation task. When I try to predict on new data (I tried also to predict on the training data), the predictions seem to follow some of the patterns, but the problem is that the predicted image ends up to be of lower resolution, meaning that the same class is predicted for 4X4 or 8x8 blocks. For the task that I am working with, pixel prediction is quite important. Has anyone faced similar problem or have any advice? Thanks

I have already cloned the repository using: !git clone https://github.com/qubvel/segmentation_models

Now when I try to load it (import it), it shows following error:

from segmentation_models import Unet

--------------------------------------------------------------------------
ModuleNotFoundError                       Traceback (most recent call last)
<ipython-input-21-95926c7db055> in <module>()
----> 1 from segmentation_models import Unet

/content/segmentation_models/__init__.py in <module>()
----> 1 from .segmentation_models import *

/content/segmentation_models/segmentation_models/__init__.py in <module>()
      3 from .__version__ import __version__
      4 
----> 5 from .unet import Unet
      6 from .fpn import FPN
      7 from .linknet import Linknet

/content/segmentation_models/segmentation_models/unet/__init__.py in <module>()
----> 1 from .model import Unet

/content/segmentation_models/segmentation_models/unet/model.py in <module>()
      2 from ..utils import freeze_model
      3 from ..utils import legacy_support
----> 4 from ..backbones import get_backbone, get_feature_layers
      5 
      6 old_args_map = {

/content/segmentation_models/segmentation_models/backbones/__init__.py in <module>()
----> 1 from classification_models import Classifiers
      2 from classification_models import resnext
      3 
      4 from . import inception_resnet_v2 as irv2
      5 from . import inception_v3 as iv3

/content/classification_models/__init__.py in <module>()
----> 1 from .classification_models import *

/content/classification_models/classification_models/__init__.py in <module>()
      3 from . import resnet as rn
      4 from . import senet as sn
----> 5 from . import keras_applications as ka
      6 
      7 

/content/classification_models/classification_models/keras_applications/__init__.py in <module>()
      1 import keras
----> 2 from .keras_applications.keras_applications import *
      3 
      4 set_keras_submodules(
      5     backend=keras.backend,

ModuleNotFoundError: No module named 'classification_models.classification_models.keras_applications.keras_applications.keras_applications'

Can anyone help regarding this? Thanks.

I get the listed error below when I try to run your demo code:

from segmentation_models import Unet
# prepare model
model = Unet(backbone_name='resnet34', encoder_weigths='imagenet')
model.compile('Adam', 'binary_crossentropy', ['binary_accuracy'])

TypeError                                 Traceback (most recent call last)
<ipython-input-12-b1cbdbcf0c01> in <module>()
      1 from segmentation_models import Unet
      2 # prepare model
----> 3 model = Unet(backbone_name='resnet34', encoder_weigths='imagenet')
      4 model.compile('Adam', 'binary_crossentropy', ['binary_accuracy'])

TypeError: Unet() got an unexpected keyword argument 'encoder_weigths'

HI @qubvel, Hi all,

I am training a UNet classifier for the OpenCitiesAI challenge (1 building class and ResNext50 backbone). @hasan-nn

At the end of the first epoch, I got the following metrics where the validation metrics are much better than the test metrics:

23366/23366 [==============================] - 15937s 682ms/step - loss: 0.2726 - iou_score: 0.5777 - f1-score: 0.7036 - val_loss: 0.0748 - val_iou_score: 0.7403 - val_f1-score: 0.7849 -

This issue is kind of related to the issue below: https://github.com/qubvel/segmentation_models/issues/285

Please advise.

Ali

I have trained a segmentation model, the backbone is densenet201 with pretrained model, the decoder is FPN But, when I load saved weight for predicting,raise error as below:

Traceback (most recent call last):
  File "test.py", line 62, in <module>
    main()
  File "test.py", line 57, in main
    apply_predict()
  File "test.py", line 27, in apply_predict
    model.load_weights("models/densenet201_fpn_w.h5")
  File "/home/jiangd/.conda/envs/tf110/lib/python3.5/site-packages/keras/engine/network.py", line 1166, in load_weights
    f, self.layers, reshape=reshape)
  File "/home/jiangd/.conda/envs/tf110/lib/python3.5/site-packages/keras/engine/saving.py", line 1030, in load_weights_from_hdf5_group
    str(len(filtered_layers)) + ' layers.')
ValueError: You are trying to load a weight file containing 428 layers into a model with 114 layers.

Hello. Images and masks in PNG format. 8 Classes. How to preprocess masks for categorical_crossentropy loss right?

I tried to use (128, 128, 1) mask, setting each pixel of the mask with a number from 0 to 7, but got error: Error when checking target: expected softmax to have shape (128, 128, 8) but got array with shape (128, 128, 1)

Code for model creation: `BACKBONE = 'resnet50' preprocess_input = get_preprocessing(BACKBONE)

model = Unet(BACKBONE, input_shape=(128,128,3), encoder_weights='imagenet', encoder_freeze=True, classes=8, activation='softmax')

model.compile('Adam', loss='categorical_crossentropy', metrics=[iou_score])`

Thanks. Sorry for bad English.

Hello! I have some problems with using unet tamplate If I try to use unet with x, y = X_train,y_train (X_train contains np.array of images as well as y_train and their shapes are (10,512, 512, 3) - 10 is count of samples 512x512 is resolution and 3 is rgb) But I get this error :Error when checking target: expected sigmoid to have shape (None, None, 1) but got array with shape (512, 512, 3)

Hi, when I run model.fit with the following code I get the error: NotImplementedError: Cannot convert a symbolic Tensor (dice_loss_plus_1focal_loss/truediv:0) to a numpy array. This error may indicate that you're trying to pass a Tensor to a NumPy call, which is not supported

the code: activation='softmax'

LR = 0.0001 optim = keras.optimizers.Adam(LR)

Segmentation models losses can be combined together by '+' and scaled by integer or float factor

set class weights for dice_loss (car: 1.; pedestrian: 2.; background: 0.5;)

dice_loss = sm.losses.DiceLoss(class_weights=np.array([0.25, 0.25, 0.25, 0.25])) focal_loss = sm.losses.CategoricalFocalLoss() total_loss = dice_loss + (1 * focal_loss)

####actulally total_loss can be imported directly from library, above example just show you how to manipulate with losses

total_loss = sm.losses.binary_focal_dice_loss # or sm.losses.categorical_focal_dice_loss

metrics = [sm.metrics.IOUScore(threshold=0.5), sm.metrics.FScore(threshold=0.5)]

checkpointer = tf.keras.callbacks.ModelCheckpoint('UNET_128_PaperModel/Model_Dice1{epoch:03d}.h5', verbose = 1, save_best_only = True)

BACKBONE1 = 'mobilenetv2' preprocess_input1 = sm.get_preprocessing(BACKBONE1)

preprocess input

X_train1 = preprocess_input1(X_train) X_test1 = preprocess_input1(X_test) ##################################################################### ###Model 1 #Using same backbone for both models

define model (Change to unet or Linknet based on the need )

model1 = sm.Unet(BACKBONE1, encoder_weights=None, classes=n_classes, activation=activation)

compile keras model with defined optimozer, loss and metrics

model1.compile(optim, total_loss, metrics=metrics)

#model1.compile(optimizer='adam', loss='categorical_crossentropy', metrics=metrics)

print(model1.summary())

My data is structured as a numpy array with X_train of shape (6632, 224, 224, 3) and y_train is one-hot encoded using to_categorical with shape (6632, 224, 224, 3)

When I google this error alot of it seems to do with the numpy version but I don't believe this to be the case....

Any help is much appreciated, Thanks!

I have used the example notebook to train a Linknet model. It is working with tensorflow 2.9.2. However I am having issues when I repeat the same steps with tensorflow 2.4.0. I checked the model.summary(). There is an issue in the softmax layer: Linknet(BACKBONE, encoder_weights='imagenet', classes=12, activation=activation)

TF 2.9.2:

conv2d (Conv2D)                (None, None, None,   1740        ['decoder_stage4c_relu[0][0]']   
                                12)                                                               
                                                                                                  
 softmax (Activation)           (None, None, None,   0           ['conv2d[0][0]']                 
                                12)                                                               
                                                                                                  
==================================================================================================
Total params: 11,523,285
Trainable params: 11,513,263
Non-trainable params: 10,022
__________________________________________________________________________

TF 2.4.0:

conv2d (Conv2D)                 (None, None, None, 1 1740        decoder_stage4c_relu[0][0]       
__________________________________________________________________________________________________
softmax (Activation)            (None, None, None, 1 0           conv2d[0][0]                     
==================================================================================================
Total params: 11,523,285
Trainable params: 11,513,263
Non-trainable params: 10,022
_________________________________________________________________

Hi!

I have a question about background in the calculation of the mIoU.

When we calculate the mIoU of the segmentaion, usually IoU of the background is included.

In that case, mIoU is at least 0.45 even the model didn't catch anything because of the IoU(background).

I wondering about Is the calculation(mIoU with background) correct or reasonable? If the calculation is not reasonable, then how can we measure the performance of the segmentation model?

Thank you.

Hi!

I am wondering about mIoU method. When I calculate mIoU of an image with incorrect prediction, if there are wrong label region, the IoU of the label is nan, not 0.

Here is the example.

Class in Ground Truth : Background, L1, L2 Class in Prediction : Background, L1, L2, L3

(1) mIoU = {IoU(BG) + IoU(L1) + IoU(L2)} / 3 Is it right?

I think, the formular should be (2) mIoU = {IoU(BG) + IoU(L1) + IoU(L2) + IoU(L3)} / 4

If (1) is right, could you explain about why the IoU of L3 is not included?

Thank you.

I am using the Qubvel segmentation models https://github.com/qubvel/segmentation_models repository to train an Inception-V3-encoder based model for a binary segmentation task. I am using (256 width x 256 height) images to train the models and they are working good. If I double one of the dimensions, say for example, (256 width x 512 height), it works fine as well. However, when I make adjustments for the aspect ratio and resize the images to a custom dimension, say (272 width x 256 height), the model throws an error as follows: ValueError: AConcatenatelayer requires inputs with matching shapes except for the concatenation axis. Received: input_shape=[(None, 16, 18, 2048), (None, 16, 17, 768)] Is there a way to use such custom dimensions to train these models? I am using RGB images and grayscale masks to train the models. Thanks.

I have a confusion regarding using CNN (VGG, ResNet) as a backbone of U-net using the segmentation library. My input shape is 512x512x3. As far I’ve understood in U-net the skip connection is used in before every layer where a downsampling happes (example: maxpool for VGG or conv with 2x2 stride for ResNet).But, in the model summary for both VGG and ResNet-based backbone I see the skip connection is there from the second downsampling (256x256x64), but there is no skip connection from the 512 resolution. Can someone explain the reasons? please check the detailed diagram of VGG19 for reference here: https://i.stack.imgur.com/FBqUy.png