Hi, would you be interested in adding MultiMAE to Hugging Face? The Hub offers free hosting, and it would make your work more accessible and visible to the rest of the ML community. Models/datasets/spaces(web demos) can be added to a user account or organization similar to github.

Example from other organizations: Keras: https://huggingface.co/keras-io Microsoft: https://huggingface.co/microsoft Facebook: https://huggingface.co/facebook

Example spaces with repos: github: https://github.com/salesforce/BLIP Spaces: https://huggingface.co/spaces/salesforce/BLIP

github: https://github.com/facebookresearch/omnivore Spaces: https://huggingface.co/spaces/akhaliq/omnivore

and here are guides for adding spaces/models/datasets to your org

How to add a Space: https://huggingface.co/blog/gradio-spaces how to add models: https://huggingface.co/docs/hub/adding-a-model uploading a dataset: https://huggingface.co/docs/datasets/upload_dataset.html

Please let us know if you would be interested and if you have any questions, we can also help with the technical implementation.

Hey, Thank you for providing the code for the paper. The paper is really interesting and the project page is very well done!

I was wondering whether you've tested the performance of linear probing on the RGB image when trained with all 3 modalities. The results of the original MAE paper were not very good, it is interesting to understand if the additional supervision creates better representations that translate into better linear probing scores.

Thanks, Eliahu

Hi, I have successful made the pesudo labels and trained ‘rgb’ in/out-domain multimae model.

But when I trained model with 'rgb-semseg' in/out-domain, I met an error in multimae/input_adapters.py line 232

# Create patches [B, C, H, W] -> [B, (H*W), C]
x_patch = rearrange(self.proj(x), 'b d nh nw -> b (nh nw) d')

The full log is log.txt. x.size() is [batchsize, 64, 56, 56] before line 232. I can't find out what's wrong.

What's more, I don't know why the pseudo semeg label image resize into 1/4 (that is 224*224->56*56)in utils/datasets.py line 105

# Convert to Tensor
for task in task_dict:
    if task in ['depth']:
        img = torch.Tensor(np.array(task_dict[task]) / 2 ** 16)
        img = img.unsqueeze(0)  # 1 x H x W
    elif task in ['rgb']:
        img = TF.to_tensor(task_dict[task])
        img = TF.normalize(img, mean=self.rgb_mean, std=self.rgb_std)
    elif task in ['semseg', 'semseg_coco']:
        # TODO: add this to a config instead
        # Rescale to 0.25x size (stride 4)
        scale_factor = 0.25
        img = task_dict[task].resize((int(self.input_size * scale_factor), int(self.input_size * scale_factor)))
        # Using pil_to_tensor keeps it in uint8, to_tensor converts it to float (rescaled to [0, 1])
        img = TF.pil_to_tensor(img).to(torch.long).squeeze(0)

and then use nn.Conv2d in multimae/input_adapters.py line 198

if self.interpolate_class_emb:
    self.proj = nn.Sequential(
        nn.Upsample(scale_factor=(1 / self.P_H, 1 / self.P_W),
                    mode='bilinear'),  # Actually a downsample operation
        nn.Conv2d(in_channels=self.dim_class_emb, out_channels=self.dim_tokens,
                    kernel_size=1, stride=1),
    )
else:
    self.proj = nn.Conv2d(
        in_channels=self.dim_class_emb, out_channels=self.dim_tokens,
        kernel_size=(self.P_H, self.P_W), stride=(self.P_H, self.P_W)
    )
)

Thank you for any help.

Hey awesome work! I am trying to figure out how to modify the demo notebook to use the regular MAE instead of multiMAE. In particular i comment out all depth and semseg info but the resulting image infilling looks corrupted. Could you by chance share an example of proper usage of the regular MAE weights? Thanks so much for the help!

Hi, I am pseudo-tagging the imagenet-1k, and encountering some difficulties.

Firstly, I wonder what would happen if the classes of semeg are more than 255? How to use one channel depth png image to represent them? (Although COCO datasets is only 80 classes, the imagenet is more than 255 classes when fine-tuning)

Secondly, on the example of Colab notebook, the rgb2depth model of DPT could not input any size of imagenet pictures. How could we save all the pseudo labels down before the data augmentation cutting it into 224*224? We need to align the original images with the pseudo labeled image should we?

Thank you for any help.

Hi, I think the following correction holds: Line 357 in https://github.com/EPFL-VILAB/MultiMAE/blob/main/run_finetuning_depth.py should be dataset_train = build_regression_dataset(args, data_path=args.train_data_path, transform=train_transform) instead of dataset_train = build_regression_dataset(args, data_path=args.data_path, transform=train_transform) or the argument train_data_path should be changed to data_path.

Apart from that, I am trying to recreate your results on NYUv2 for depth. but the dataset preparation instructions are not clear from the instructions in SETUP. As explained about the folder structure , where should the GT be when finetuning for depth and evaluating. Apart from that, mask_valid for fine-tuning? RuntimeError: Found 0 logs in subfolders of: /tmp-network/user/varora/multimae/multimae_data/train/mask_valid

Hi, Thank you for the interesting work and the extensive experiments. Your depth results are based on the DPT head in the paper. In the colab, you use the spatial adapter head for inference. I was wondering if your fine-tuning results with the spatial adapter head were better/worse than the DPT head? Was the intention to implement this spatial head more to test a pure transformer based head (compared to DPT's convolution based refineNet like approach?)?

Thank you.

HI, I find that:

in MultiMae/run_finetuning_semseg.py line735

seg_pred_argmax = seg_pred[:num_classes].argmax(dim=1) 

I think it should be

seg_pred_argmax = seg_pred[:,:num_classes,:,:].argmax(dim=1)