[CVPR 2021] Pytorch implementation of Hijack-GAN: Unintended-Use of Pretrained, Black-Box GANs

Last update: Sep 05, 2022

Overview

Hijack-GAN: Unintended-Use of Pretrained, Black-Box GANs

In this work, we propose a framework HijackGAN, which enables non-linear latent space traversal and gain high-level controls, e.g., attributes, head poses, and landmarks, over unconditional image generation GANs in a fully black-box setting. It opens up the possibility of reusing GANs while raising concerns about unintended usage.

[Paper (CVPR 2021)][Project Page]

Prerequisites

Install required packages

pip install -r requirements.txt

Download pretrained GANs

Download the CelebAHQ pretrained weights of ProgressiveGAN [paper][code] and StyleGAN [paper][code], and then put those weights in ./models/pretrain. For example,

pretrain/
├── Pretrained_Models_Should_Be_Placed_Here
├── karras2018iclr-celebahq-1024x1024.pkl
├── karras2019stylegan-celebahq-1024x1024.pkl
├── pggan_celebahq_z.pt
├── stylegan_celebahq_z.pt
├── stylegan_headpose_z_dp.pt
└── stylegan_landmark_z.pt

Quick Start

Specify number of images to edit, a model to generate images, some parameters for editting.

LATENT_CODE_NUM=1
python edit.py \
    -m pggan_celebahq \
    -b boundaries/ \
    -n "$LATENT_CODE_NUM" \
    -o results/stylegan_celebahq_eyeglasses \
    --step_size 0.2 \
    --steps 40 \
    --attr_index 0 \
    --task attribute \
    --method ours

Usage

Important: For different given images (initial points), different step size and steps may be considered. In the following examples, we provide the parameters used in our paper. One could adjust them for better performance.

Specify Number of Samples

LATENT_CODE_NUM=1

Unconditional Modification

python edit.py \
    -m pggan_celebahq \
    -b boundaries/ \
    -n "$LATENT_CODE_NUM" \
    -o results/stylegan_celebahq_smile_editing \
    --step_size 0.2 \
    --steps 40 \
    --attr_index 0\
    --task attribute

Conditional Modification

python edit.py \
    -m pggan_celebahq \
    -b boundaries/ \
    -n "$LATENT_CODE_NUM" \
    -o results/stylegan_celebahq_smile_editing \
    --step_size 0.2 \
    --steps 40 \
    --attr_index 0\
    --condition\
    -i codes/pggan_cond/age.npy
    --task attribute

Head pose

Pitch

python edit.py \
    -m stylegan_celebahq \
    -b boundaries/ \
    -n "$LATENT_CODE_NUM" \
    -o results/ \
    --task head_pose \
    --method ours \
    --step_size 0.01 \
    --steps 2000 \
    --attr_index 1\
    --condition\
    --direction -1 \
    --demo

Yaw

python edit.py \
    -m stylegan_celebahq \
    -b boundaries/ \
    -n "$LATENT_CODE_NUM" \
    -o results/ \
    --task head_pose \
    --method ours \
    --step_size 0.1 \
    --steps 200 \
    --attr_index 0\
    --condition\
    --direction 1\
    --demo

Landmarks

Parameters for reference: (attr_index, step_size, steps) (4: 0.005 400) (5: 0.01 100), (6: 0.1 200), (8 0.1 200)

CUDA_VISIBLE_DEVICES=0 python edit.py \
    -m stylegan_celebahq \
    -b boundaries/ \
    -n "$LATENT_CODE_NUM" \
    -o results/ \
    --task landmark \
    --method ours \
    --step_size 0.1 \
    --steps 200 \
    --attr_index 6\
    --condition\
    --direction 1 \
    --demo

Generate Balanced Data

This a templeate showing how we generated balanced data for attribute manipulation (16 attributes in our internal experiments). You can modify it to fit your task better. Please first refer to here and replace YOUR_TASK_MODEL with your own classification model, and then run:

NUM=500000
CUDA_VISIBLE_DEVICES=0 python generate_balanced_data.py -m stylegan_celebahq \
    -o ./generated_data -K ./generated_data/indices.pkl -n "$NUM" -SI 0 --no_generated_imgs

[CVPR 2021] Pytorch implementation of Hijack-GAN: Unintended-Use of Pretrained, Black-Box GANs

Related tags

Overview

Hijack-GAN: Unintended-Use of Pretrained, Black-Box GANs

Prerequisites

Install required packages

Download pretrained GANs

Quick Start

Usage

Specify Number of Samples

Unconditional Modification

Conditional Modification

Head pose

Pitch

Yaw

Landmarks

Generate Balanced Data

Evaluations

TO-DO

Acknowledgment

Owner

Hui-Po Wang

ConvMixer unofficial implementation

Implementation of Auto-Conditioned Recurrent Networks for Extended Complex Human Motion Synthesis

Text to Image Generation with Semantic-Spatial Aware GAN

OcclusionFusion: realtime dynamic 3D reconstruction based on single-view RGB-D

Aerial Single-View Depth Completion with Image-Guided Uncertainty Estimation (RA-L/ICRA 2020)

LSTC: Boosting Atomic Action Detection with Long-Short-Term Context

Where-Got-Time - An NUS timetable generator which uses a genetic algorithm to optimise timetables to suit the needs of NUS students

Self-training with Weak Supervision (NAACL 2021)

The official implementation of ICCV paper "Box-Aware Feature Enhancement for Single Object Tracking on Point Clouds".

Exe-to-xlsm - Simple script to create VBscript of exe and inject to xlsm

Tools for computational pathology

A neuroanatomy-based augmented reality experience powered by computer vision. Features 3D visuals of the Atlas Brain Map slices.

Speedy Implementation of Instance-based Learning (IBL) agents in Python

Classification of EEG data using Deep Learning

Official repo for QHack—the quantum machine learning hackathon

Code for "Continuous-Time Meta-Learning with Forward Mode Differentiation" (ICLR 2022)

PyTorch code for the NAACL 2021 paper "Improving Generation and Evaluation of Visual Stories via Semantic Consistency"

Posterior temperature optimized Bayesian models for inverse problems in medical imaging

Python package for multiple object tracking research with focus on laboratory animals tracking.

Satellite labelling tool for manual labelling of storm top features such as overshooting tops, above-anvil plumes, cold U/Vs, rings etc.