Hi! I have a question about the definition of "labels" in the script "simclr.py".

On line 54 of "simclr.py", the authors defined:

labels = torch.zeros(logits.shape[0], dtype=torch.long).to(self.args.device)

So all the entries of "labels" are all zeros. But I think according to the paper, there should be an entry as 1 for the positive pair?

Thanks in advance for your reply!

Hi , I'm trying to understand the code in : loss/nt_xent.py

we are sending "representations" on both arguments

    def forward(self, zis, zjs):
        representations = torch.cat([zjs, zis], dim=0)
        similarity_matrix = self.similarity_function(representations, representations)

But when receiving it in cosine_similarity func somehow the sizes are: (N, 1, C) and y shape: (1, 2N, C), how can it be double if you sent the same argument

    def _cosine_simililarity(self, x, y):
        # x shape: (N, 1, C)
        # y shape: (1, 2N, C)
        # v shape: (N, 2N)
        v = self._cosine_similarity(x.unsqueeze(1), y.unsqueeze(0))
        return v

Thanks for your help.

Dear researcher, Thank you for the open-source code you provided, it is of great help to me for understanding contrastive learning. But I still have some confusion when training the SimCLR model with my local dataset, could you give me some guidance or tips? I would appreciate it if you could reply to this issue.

Hello,

Thanks for sharing the code, nice implementation.

The way you calculate the loss by using a mask is quite brilliant. But I have a question.

logits = torch.cat((positives, negatives), dim=1) So if I'm not wrong, the first column of logits is positive and the rest are negatives.

labels = torch.zeros(2 * self.batch_size).to(self.device).long() But your labels are all zeros, which means no matter positive or negative, the similarity should low.

So I wonder is the first column of labels supposed to be 1 instead of 0.

Thanks for your help.

In function info_nce_loss, the line 28, creates labels based on batch_size and on other side we have STL10 dataset which has 100,000 images which is divisible by batch_size of 32 and having batch_size like 128 or 64 gives a remainder of 32.

Having batch_size != 32, causes error in line 42, because the similarity matrix will based on features and labels will be based on batch size.

For instance, if the batch size = 128, the remaining images in the dataset in the last iter of data_loader is 32. Since we create two variant of each image we'll have 64 images. Now we have 128 x 2 = 256 labels from line 28, and we'll have similarity matrix of (64 x 128, 128 x 64) => (64 x 64) but with mask (256 x 256) causing "dimension mismatch"

Solution: Change Line 28 as below

labels = torch.cat([torch.arange(features.shape[0]//2) for i in range(self.args.n_views)], dim=0)

Considering the setting in 'scheduler = torch.optim.lr_scheduler.CosineAnnealingLR(optimizer, T_max=len(train_loader), eta_min=0, last_epoch=-1)'，I think 'scheduler.step()' should be called every step in 'for (xis,xjs),_ in train_loader'. Otherwise，lr will nerver change until 'len(train_loader)' epochs but not steps

Hi,

I find that the ResNet20 model for CIFAR-10 experiments is not fully correct. The head conv structure should be modified (stride=1 and no pooling,) because the image size of CIFAR-10 is very small.

Hi, thanks for the code!

When I tried to run it on single GPU (v-100), the utilizaiton rate is very low (~0-10%) even if I increase num_worker. Would you know why this happens and how to solve it? Thanks!

Hi, This code is really useful for me. Thanks! But I got a question about the NT-Xent loss. I noticed that you use L2 norm on z and then use cos_similarity after that. But cos_similarity already contain the function of l2 norm. Why use L2 norm first?

Hi @sthalles , Thank you for sharing your code!

Pl correct me if I am wrong: I see that in line loss/nt_xent.py line 57 (below) you are not computing contrastive loss for all negative pairs as you are reshaping total negatives in 2D array i.e. only a part of negative pairs are being used for a single positive pair, right? :

_negatives = similarity_matrix[self.mask_samples_from_same_repr].view(2 * self.batch_size, -1)_
_logits = torch.cat((positives, negatives), dim=1)_

Hope to hear from you soon.

-Ishan

First of all, thank you for your great work!

Method _validate in simclr.py will raise ZeroDivisionError at line 148 if the validation data loader performs only one iteration (since counter starts from 0).

I'm really appreciated about your good work :) I left a question because I got confused while studying through your great code.

First, I wonder why you used "batch_size=batch_size*2" differently from train_loader in the test_loader part of the file "mini_batch_logistic_regression_valuator.ipynb". Is it related to creating 2 views when doing data augmentation?

Also, in the last cell of this file, I'm confused whether the second "for" (of the two "for") in the large epoch "for" statement corresponds to the test process or the validation process. I thought it was a test process, because loss update, backpropagation, optimization, etc. were done only in the first "for", and the second yield only accuracy, but is that right? Or I'm confused if the second "for" is a validating process because the first "for" and the second "for" are going together in the entire epoch processing.

Hi, I just want to check all the experiments details and make sure I didn't miss any part(?

1. Training Phase : use SimCLR (two encoder branches) to train on ImageNet for 1000 epochs to get a init pretrained weights.
2. Fine-Tuned : load the init pretrained weights on the resnet18(50/101/...) with freezed parameters and concate with a linear classifier, and train the classifier with CIFAR10/STL10 training dataset for 100 epochs.
3. Test Phase : freeze all the encoder, classifier parameters, and test on the CIFAR10/STL10 testing dataset.

Is this the way how you get the top1 acc in the README?

Does anyone know how to add the confusion matrix in this code? After I added it according to the online one, something went wrong. I don't know what went wrong in my code.I can't solve it. please help help me! Thanks. def confusion_matrix(output, labels, conf_matrix):

preds = torch.argmax(output, dim=-1)
for p, t in zip(preds, labels):
    conf_matrix[p, t] += 1
return conf_matrix

Hi, I'm trying to experiment with CIFAR-10 with the default hyper-params, and it seems to yield a better score when using smaller batch size (e.g. 72% with batch size 256 yet 78% with batch size 128). Anyone in the same situation, here?

I am using pythion 3.7 on Win10, Anaconda Jupyter. I have successfully installed torch-1.10.0+cu113 torchaudio-0.10.0+cu113 torchvision-0.11.1+cu113. When trying to import torch , I get ModuleNotFoundError: No module named 'torch.cuda' Detailed error:

ModuleNotFoundError                       Traceback (most recent call last)
<ipython-input-1-bfd2c657fa76> in <module>
      1 import numpy as np
      2 import pandas as pd
----> 3 import torch
      4 import torch.nn as nn
      5 from sklearn.model_selection import train_test_split

~\AppData\Roaming\Python\Python38\site-packages\torch\__init__.py in <module>
    603 
    604 # Shared memory manager needs to know the exact location of manager executable
--> 605 _C._initExtension(manager_path())
    606 del manager_path
    607 

ModuleNotFoundError: No module named 'torch.cuda'

I found posts for similar error No module named 'torch.cuda.amp'. However, any of the suggested solutions worked. Please advise.