Official implementation of "Watermarking Images in Self-Supervised Latent-Spaces"

Overview

🔍 Watermarking Images in Self-Supervised Latent-Spaces

PyTorch implementation and pretrained models for the paper. For details, see Watermarking Images in Self-Supervised Latent-Spaces.

If you find this repository useful, please consider giving a star and please cite as:

@inproceedings{fernandez2022sslwatermarking,
  title={Watermarking Images in Self-Supervised Latent Spaces},
  author={Fernandez, Pierre and Sablayrolles, Alexandre and Furon, Teddy and Jégou, Hervé and Douze, Matthijs},
  booktitle={IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)},
  year={2022},
  organization={IEEE},
}

[Webpage] [arXiv] [Spaces] [Colab]

Introduction

Illustration

The method uses:

  • a pretrained neural network and a normalization layer to extract features from images
  • an embedding stage that invisibly changes the image to push the feature in certain directions of the latent space
  • a decoding stage that detects or decode the mark that was added in the image

Usage

First, clone the repository locally and move inside the folder:

git clone https://github.com/facebookresearch/ssl_watermarking.git
cd ssl_watermarking

Then, install the dependencies:

pip install -r requirements.txt

This codebase has been developed with python version 3.8, PyTorch version 1.10.2, CUDA 10.2 and torchvision 0.11.3. The following considers ssl_watermarking/ as the root folder, all paths are relative to it.

PS: Trouble shooting for Augly

Images

You are free to use your own images.
Images to be watermarked must be put in a folder of the form <name/of/folder>/0/ for the dataloader to work. The image folder can be put on the ssl_watermarking folder, or in any other place, you will later need to specify the path to the folder by the argument --data_dir <name/of/folder>.
We already provide 8 high-resolution-images in the input folder, from the INRIA Holidays dataset.

⚠️ If images are too high resolution, an out-of-memory error might appear, you can try to resize your images beforehand.

Batching

At the moment, batching only works if all images of the folder have the same size. For images with different sizes, please --batch_size 1 argument.
If your images have the same dimensions, batching greatly speeds up the process.

Pretrained models & normalization layers

To watermark, you need:

  • a neural network model that extracts features from images.
  • a normalization layer that transforms the features so that they are more evenly distributed in the latent space.

We provide the weights used in all of our experiments:

To use these weights, create the folders models/ and normlayers/ into the ssl_watermarking directory and put:

  • dino_r50_plus.pth (weights of the backbone) in models/
  • out2048_yfcc_orig.pth (weights of the normalization layer) in normlayers/.

The ResNet model was trained using https://github.com/pierrefdz/dino/ (same as original dino with an additional rotation augmentation). The arguments used to train the model are available here.

The normalization layers that perform PCA whitening (see wikipedia) are obtained over 100k images of YFCC for "whitening" (resp. COCO for "whitening v1") and of their version resized to 128x128. If the input images have low resolution, we recommend using the normalization layer created from YFCC resized, otherwise, we recommend using the one created from the original sizes.

Watermarking

0️⃣ 0-bit

To perform 0-bit watermarking:

python main_0bit.py --data_dir <path/to/imgs> \
  --model_path <path/to/model> --normlayer_path <path/to/normlayer> \
  --target_psnr <PSNR> --target_fpr <FPR>

For instance, running:

python main_0bit.py --data_dir <path/to/yfcc1k> --model_path models/dino_r50_plus.pth --normlayer_path normlayers/out2048_yfcc_orig.pth --batch_size 1 --target_psnr 40 --target_fpr 1e-6

gives the following logs and output/df_agg.csv (see evaluation for details on the csv files).

To run detection only:

python main_0bit.py --decode_only True --data_dir <path/to/imgs> \
  --model_path <path/to/model> --normlayer_path <path/to/normlayer> \
  --target_fpr <FPR>

You should obtain a file in the output folder, such as decodings.csv:

index Marked filename
0 True 0_out.png
1 True 1_out.png
2 True 2_out.png

🔢 Multi-bit watermarking

To perform multi-bit watermarking (hide K bits in the image):

python main_multibit.py --data_dir <path/to/imgs> \
  --model_path <path/to/model> --normlayer_path <path/to/normlayer> \
  --target_psnr <PSNR> --num_bits <K>

For instance, running:

python main_multibit.py --data_dir <path/to/coco1k_resized> --model_path models/dino_r50_plus.pth --normlayer_path normlayers/out2048_coco_resized.pth --batch_size 128 --target_psnr 33 --num_bits 30

gives the following logs and output/df_agg.csv (see evaluation for details on the csv files).

To run decoding only:

python main_multibit.py --decode_only True --data_dir <path/to/imgs> \
  --model_path <path/to/model> --normlayer_path <path/to/normlayer> \
  --num_bits <K>

You should obtain a file in the output folder, such as decodings.csv:

index msg :filename
0 000010010101100101101111110101 0_out.png
1 011000101001001010000100111000 1_out.png
2 100001100010111010111100011000 2_out.png

📝 With your own messages

You can alternatively decide to use your own messages. Create a folder messages/ in ssl_watermarking and put a file called msgs.txt in it. The kth line of the message should be the message you want to hide in the kth image (if there are more images than messages, messages are repeated cyclically). It can be:

  • a text (e.g. "Hello world!"): put --msg_type text argument for main_multibit.py.
    The text messages are encoded using 8-bits characters, i.e. the first 256 characters, from Unicode UTF-8 (be careful when using special characters such as 🔍 ).
    If messages don't have same length, they are padded with white space.
  • a list of bits (e.g. "0010110011"): put --msg_type bit argument for main_multibit.py

Examples: text, bits. Then, append the argument --msg_path <path/to/msgs.txt> --msg_type <type> to the previous command line.

⚠️ If the --num_bit <K> argument (Default: 30) doesn't match the length of the messages computed from msgs.txt, say 56, the num_bits argument will be automatically set to 56. A warning will appear. To get rid of it, you just need to append --num_bit 56 to the previous command line.

📈 Evaluation

The previous commands should return the score of the detection on several attacks and store them in output/agg_df.csv and output/df.csv.

  • output/agg_df.csv gives general metrics for the decoding on several attacks. Ex: R, p-value, Bit accuracy, etc.

    Reduced example for 0-bit watermarking:

    log10_pvalue R marked
    mean min max std mean min max std mean min max std
    attack param0
    blur 11.0 -40.7 -81.3 -0.13 14.23 1.53e6 -3.19e5 7.00e6 1.10e6 0.99 False True 0.09
    center_crop 0.5 -21.46 -64.65 -0.14 9.17 6.89e5 -3.73e5 3.89e6 6.42e5 0.96 False True 0.18
    rotation 25.0 -26.29 -66.08 -0.02 10.95 5.87e5 -3.77e5 3.80e6 5.47e5 0.97 False True 0.14

    (where, R is the acceptance function that determines "how much" the feature lies in the cone, and the p-value is such that if we were drawing O(1/pvalue) random carriers, on expectation, one of them would give an R bigger or equal to the one that is observed.)

  • output/df.csv gives metrics for each image. Ex: detected or not, message, etc.

    Reduced example for 0-bit watermarking:

    img attack log10_pvalue R marked param0
    0 none -67.87 5.80e6 True -1.0
    0 meme_format -29.60 4.49e5 True -1.0
    0 rotation -28.56 4.70e5 True 35.0
    1 none -76.12 2.68e6 True -1.0
    1 meme_format -21.17 1.73e5 True -1.0
    1 rotation -12.29 1.23e5 True 35.0

You can deactivate the evaluation by setting --evaluate False.

💾 Saving images

The previous scripts store the attacked versions of the first image (of the folder of images to be watermarked) when evaluating on different attacks. They also save all watermarked images. The images are stored in output/imgs/.

You can choose not to save the images by setting --save False.

Data Augmentation

By default, the optimization is done for the watermark to be robust on different augmentations (crop, resize, blur, rotation, etc.). If you are not interested in robustness, you choose to set --data_augmentation None. You can then drastically reduce the number of epochs in the optimization: typically --epochs 10 should already give good results.

If you are interested in robustness to specific transformations, you can either:

  • change the default parameters used in the class All() of data_augmentation.py.
  • create a new data augmentation that inherits the DifferentiableDataAugmentation class. The main restriction is that the augmentation should be differentiable.

Using other architectures

Although we highly recommend using the resnet50 architecture with the given weights, other models from the torchvision of timm library can also be used. In this case, you can either not put any normalization layer (not recommended - gives far worse performance) or create a new normalization layer. To do so, please run:

python build_normalization_layer.py --model_name <model_name> --model_path <path/to/model/> --large_data_dir <path/to/big/dataset/for/PCA/whitening> 

You can improve the whitening step by using a dataset that has similar distribution to the images you want to watermark and a number of images in the order of 10K. You can also change the parameters of the resize crop transform (with the img_size and crop_size arguments) that is used before feature extraction to have images resized as little as possible.

Reproduce paper results

The paper uses images from CLIC, Multimedia Commons YFCC100M and COCO datasets. You will need to download them and extract 1k images from them (except CLIC that has less images) to reproduce results from the paper.

You also need to download the model and normalization layer weights (see Pretrained models & normalization layers).

Remark: The overlay onto screenshot transform (from Augly) that is used in the paper is the mobile version (Augly's default: web). To change it, you need to locate the file augly/utils/base_paths.py (run pip show augly to locate the Augly library). Then change the line "TEMPLATE_PATH = os.path.join(SCREENSHOT_TEMPLATES_DIR, "web.png")" to "TEMPLATE_PATH = os.path.join(SCREENSHOT_TEMPLATES_DIR, "mobile.png")".

Table 1: TPR for 0-bit watermarking

You will need to run:

python main_0bit.py --data_dir <path/to/yfcc1k> --model_path models/dino_r50_plus.pth --normlayer_path normlayers/out2048_yfcc_orig.pth --batch_size 1 --target_psnr 40 --target_fpr 1e-6 --output_dir output_ssl/

To compare with the supervised model, you need to download the supervised model weights (trained with a fork of the torchvision code, with additional rotation augmentation), and whitening layer weights and put them in the models (resp. the normlayers) folder. Then run:

python main_0bit.py --data_dir <path/to/yfcc1k> --model_path models/r50_90.pth.tar --normlayer_path normlayers/out2048_yfcc_orig_sup.pth --batch_size 1 --target_psnr 40 --target_fpr 1e-6 --output_dir output_sup/

Table 2: BER & WER for multi-bit watermarking

You will need to run:

python main_multibit.py --data_dir <path/to/yfcc1k> --model_path models/dino_r50_plus.pth --normlayer_path normlayers/out2048_yfcc_orig.pth --batch_size 1 --target_psnr 40 --num_bits 30

Table 3: BER for multi-bit watermarking on COCO resized to (128x128)

You will need to run:

python main_multibit.py --data_dir <path/to/coco1k_resized> --model_path models/dino_r50_plus.pth --normlayer_path normlayers/out2048_coco_resized.pth --batch_size 128 --target_psnr 33 --num_bits 30

License

ssl_watermarking is CC-BY-NC licensed, as found in the LICENSE file.

Owner
Meta Research
Meta Research
Unity Propagation in Bayesian Networks Handling Inconsistency via Unity Smoothing

This repository contains the scripts needed to generate the results from the paper Unity Propagation in Bayesian Networks Handling Inconsistency via U

0 Jan 19, 2022
It is modified Tensorflow 2.x version of Mask R-CNN

[TF 2.X] Mask R-CNN for Object Detection and Segmentation [Notice] : The original mask-rcnn uses the tensorflow 1.X version. I modified it for tensorf

Milner 34 Nov 09, 2022
Code for Referring Image Segmentation via Cross-Modal Progressive Comprehension, CVPR2020.

CMPC-Refseg Code of our CVPR 2020 paper Referring Image Segmentation via Cross-Modal Progressive Comprehension. Shaofei Huang*, Tianrui Hui*, Si Liu,

spyflying 55 Dec 01, 2022
This repository contains the source code of our work on designing efficient CNNs for computer vision

Efficient networks for Computer Vision This repo contains source code of our work on designing efficient networks for different computer vision tasks:

Sachin Mehta 386 Nov 26, 2022
OpenMatch: Open-set Consistency Regularization for Semi-supervised Learning with Outliers (NeurIPS 2021)

OpenMatch: Open-set Consistency Regularization for Semi-supervised Learning with Outliers (NeurIPS 2021) This is an PyTorch implementation of OpenMatc

Vision and Learning Group 38 Dec 26, 2022
The FIRST GANs-based omics-to-omics translation framework

OmiTrans Please also have a look at our multi-omics multi-task DL freamwork 👀 : OmiEmbed The FIRST GANs-based omics-to-omics translation framework Xi

Xiaoyu Zhang 6 Dec 14, 2022
A PyTorch implementation of the continual learning experiments with deep neural networks

Brain-Inspired Replay A PyTorch implementation of the continual learning experiments with deep neural networks described in the following paper: Brain

182 Dec 27, 2022
Pytorch implementation for Patient Knowledge Distillation for BERT Model Compression

Patient Knowledge Distillation for BERT Model Compression Knowledge distillation for BERT model Installation Run command below to install the environm

Siqi 180 Dec 19, 2022
Dyalog-apl-docset - Dyalog APL Dash Docset Generator

Dyalog APL Dash Docset Generator o alasa e kili sona kepeken tenpo lili a A Dash

Maciej Goszczycki 1 Jan 10, 2022
Pre-trained models for a Cascaded-FCN in caffe and tensorflow that segments

Cascaded-FCN This repository contains the pre-trained models for a Cascaded-FCN in caffe and tensorflow that segments the liver and its lesions out of

300 Nov 22, 2022
Text Extraction Formulation + Feedback Loop for state-of-the-art WSD (EMNLP 2021)

ConSeC is a novel approach to Word Sense Disambiguation (WSD), accepted at EMNLP 2021. It frames WSD as a text extraction task and features a feedback loop strategy that allows the disambiguation of

Sapienza NLP group 36 Dec 13, 2022
Easy to use Audio Tagging in PyTorch

Audio Classification, Tagging & Sound Event Detection in PyTorch Progress: Fine-tune on audio classification Fine-tune on audio tagging Fine-tune on s

sithu3 15 Dec 22, 2022
This is a beginner-friendly repo to make a collection of some unique and awesome projects. Everyone in the community can benefit & get inspired by the amazing projects present over here.

Awesome-Projects-Collection Quality over Quantity :) What to do? Add some unique and amazing projects as per your favourite tech stack for the communi

Rohan Sharma 178 Jan 01, 2023
VLG-Net: Video-Language Graph Matching Networks for Video Grounding

VLG-Net: Video-Language Graph Matching Networks for Video Grounding Introduction Official repository for VLG-Net: Video-Language Graph Matching Networ

Mattia Soldan 25 Dec 04, 2022
Official source code to CVPR'20 paper, "When2com: Multi-Agent Perception via Communication Graph Grouping"

When2com: Multi-Agent Perception via Communication Graph Grouping This is the PyTorch implementation of our paper: When2com: Multi-Agent Perception vi

34 Nov 09, 2022
This program uses trial auth token of Azure Cognitive Services to do speech synthesis for you.

🗣️ aspeak A simple text-to-speech client using azure TTS API(trial). 😆 TL;DR: This program uses trial auth token of Azure Cognitive Services to do s

Levi Zim 359 Jan 05, 2023
Intrusion Test Tool with Python

P3ntsT00L Uma ferramenta escrita em Python, feita para Teste de intrusão. Requisitos ter o python 3.9.8 instalado em sua máquina. ter a git instalada

josh washington 2 Dec 27, 2021
The description of FMFCC-A (audio track of FMFCC) dataset and Challenge resluts.

FMFCC-A This project is the description of FMFCC-A (audio track of FMFCC) dataset and Challenge resluts. The FMFCC-A dataset is shared through BaiduCl

18 Dec 24, 2022
Keras-tensorflow implementation of Fully Convolutional Networks for Semantic Segmentation(Unfinished)

Keras-FCN Fully convolutional networks and semantic segmentation with Keras. Models Models are found in models.py, and include ResNet and DenseNet bas

645 Dec 29, 2022
A Dynamic Residual Self-Attention Network for Lightweight Single Image Super-Resolution

DRSAN A Dynamic Residual Self-Attention Network for Lightweight Single Image Super-Resolution Karam Park, Jae Woong Soh, and Nam Ik Cho Environments U

4 May 10, 2022