StyleCLIP: Text-Driven Manipulation of StyleGAN Imagery

Overview

StyleCLIP: Text-Driven Manipulation of StyleGAN Imagery

Optimization: Open In Colab Global directions: Open In Colab Mapper: Open In Colab

Check our full demo video here:

StyleCLIP: Text-Driven Manipulation of StyleGAN Imagery
Or Patashnik*, Zongze Wu*, Eli Shechtman, Daniel Cohen-Or, Dani Lischinski
*Equal contribution, ordered alphabetically
https://arxiv.org/abs/2103.17249

Abstract: Inspired by the ability of StyleGAN to generate highly realistic images in a variety of domains, much recent work has focused on understanding how to use the latent spaces of StyleGAN to manipulate generated and real images. However, discovering semantically meaningful latent manipulations typically involves painstaking human examination of the many degrees of freedom, or an annotated collection of images for each desired manipulation. In this work, we explore leveraging the power of recently introduced Contrastive Language-Image Pre-training (CLIP) models in order to develop a text-based interface for StyleGAN image manipulation that does not require such manual effort. We first introduce an optimization scheme that utilizes a CLIP-based loss to modify an input latent vector in response to a user-provided text prompt. Next, we describe a latent mapper that infers a text-guided latent manipulation step for a given input image, allowing faster and more stable textbased manipulation. Finally, we present a method for mapping a text prompts to input-agnostic directions in StyleGAN’s style space, enabling interactive text-driven image manipulation. Extensive results and comparisons demonstrate the effectiveness of our approaches.

Description

Official Implementation of StyleCLIP, a method to manipulate images using a driving text. Our method uses the generative power of a pretrained StyleGAN generator, and the visual-language power of CLIP. In the paper we present three methods:

  • Latent vector optimization.
  • Latent mapper, trained to manipulate latent vectors according to a specific text description.
  • Global directions in the StyleSpace.

Updates

6/4/2021 Add mapper training and inference (including a jupyter notebook) code

6/4/2021 Add support for custom StyleGAN2 and StyleGAN2-ada models, and also custom images

2/4/2021 Add the global directions code (a local GUI and a colab notebook)

31/3/2021 Upload paper to arxiv, and video to YouTube

14/2/2021 Initial version

Setup (for all three methods)

For all the methods described in the paper, is it required to have:

Specific requirements for each method are described in its section. To install CLIP please run the following commands:

conda install --yes -c pytorch pytorch=1.7.1 torchvision cudatoolkit=<CUDA_VERSION>
pip install ftfy regex tqdm gdown
pip install git+https://github.com/openai/CLIP.git

Editing via Latent Vector Optimization

Setup

Here, the code relies on the Rosinality pytorch implementation of StyleGAN2. Some parts of the StyleGAN implementation were modified, so that the whole implementation is native pytorch.

In addition to the requirements mentioned before, a pretrained StyleGAN2 generator will attempt to be downloaded, (or manually download from here).

Usage

Given a textual description, one can both edit a given image, or generate a random image that best fits to the description. Both operations can be done through the main.py script, or the optimization_playground.ipynb notebook (Open In Colab).

Editing

To edit an image set --mode=edit. Editing can be done on both provided latent vector, and on a random latent vector from StyleGAN's latent space. It is recommended to adjust the --l2_lambda according to the desired edit.

Generating Free-style Images

To generate a free-style image set --mode=free_generation.

Editing via Latent Mapper

Here, we provide the code for the latent mapper. The mapper is trained to learn residuals from a given latent vector, according to the driving text. The code for the mapper is in mapper/.

Setup

As in the optimization, the code relies on Rosinality pytorch implementation of StyleGAN2. In addition the the StyleGAN weights, it is neccessary to have weights for the facial recognition network used in the ID loss. The weights can be downloaded from here.

The mapper is trained on latent vectors. It is recommended to train on inverted real images. To this end, we provide the CelebA-HQ that was inverted by e4e: train set, test set.

Usage

Training

  • The main training script is placed in mapper/scripts/train.py.
  • Training arguments can be found at mapper/options/train_options.py.
  • Intermediate training results are saved to opts.exp_dir. This includes checkpoints, train outputs, and test outputs. Additionally, if you have tensorboard installed, you can visualize tensorboard logs in opts.exp_dir/logs. Note that
  • To resume a training, please provide --checkpoint_path.
  • --description is where you provide the driving text.
  • If you perform an edit that is not supposed to change "colors" in the image, it is recommended to use the flag --no_fine_mapper.

Example for training a mapper for the moahwk hairstyle:

cd mapper
python train.py --exp_dir ../results/mohawk_hairstyle --no_fine_mapper --description "mohawk hairstyle"

All configurations for the examples shown in the paper are provided there.

Inference

  • The main inferece script is placed in mapper/scripts/inference.py.
  • Inference arguments can be found at mapper/options/test_options.py.
  • Adding the flag --couple_outputs will save image containing the input and output images side-by-side.

Pretrained models for variuos edits are provided. Please refer to utils.py for the complete links list.

We also provide a notebook for performing inference with the mapper Mapper notebook: Open In Colab

Editing via Global Direction

Here we provide GUI for editing images with the global directions. We provide both a jupyter notebook Open In Colab, and the GUI used in the video. For both, the linear direction are computed in real time. The code is located at global/.

Setup

Here, we rely on the official TensorFlow implementation of StyleGAN2.

It is required to have TensorFlow, version 1.14 or 1.15 (conda install -c anaconda tensorflow-gpu==1.14).

Usage

Local GUI

To start the local GUI please run the following commands:

cd global

# input dataset name 
dataset_name='ffhq' 

# pretrained StyleGAN2 model from standard [NVlabs implementation](https://github.com/NVlabs/stylegan2) will be download automatically.
# pretrained StyleGAN2-ada model could be download from https://nvlabs-fi-cdn.nvidia.com/stylegan2-ada/pretrained/ .
# for custom StyleGAN2 or StyleGAN2-ada model, please place the model under ./StyleCLIP/global/model/ folder.


# input prepare data 
python GetCode.py --dataset_name $dataset_name --code_type 'w'
python GetCode.py --dataset_name $dataset_name --code_type 's'
python GetCode.py --dataset_name $dataset_name --code_type 's_mean_std'

# preprocess (this may take a few hours). 
# we precompute the results for StyleGAN2 on ffhq, StyleGAN2-ada on afhqdog, afhqcat. For these model, we can skip the preprocess step.
python SingleChannel.py --dataset_name $dataset_name

# generated image to be manipulated 
# this operation will generate and replace the w_plu.npy and .jpg images in './data/dataset_name/' folder. 
# if you you want to keep the original data, please rename the original folder.
# to use custom images, please use e4e encoder to generate latents.pt, and place it in './data/dataset_name/' folder, and add --real flag while running this function.
# you may skip this step if you want to manipulate the real human faces we prepare in ./data/ffhq/ folder.   
python GetGUIData.py --dataset_name $dataset_name

# interactively manipulation 
python PlayInteractively.py --dataset_name $dataset_name

As shown in the video, to edit an image it is requires to write a neutral text and a target text. To operate the GUI, please do the following:

  • Maximize the window size
  • Double click on the left square to choose an image. The images are taken from global/data/ffhq, and the corresponding latent vectors are in global/data/ffhq/w_plus.npy.
  • Type a neutral text, then press enter
  • Modify the target text so that it will contain the target edit, then press enter.

You can now play with:

  • Manipulation strength - positive values correspond to moving along the target direction.
  • Disentanglement threshold - large value means more disentangled edit, just a few channels will be manipulated so only the target attribute will change (for example, grey hair). Small value means less disentangled edit, a large number of channels will be manipulated, related attributes will also change (such as wrinkle, skin color, glasses).
Examples:
Edit Neutral Text Target Text
Smile face smiling face
Gender female face male face
Blonde hair face with hair face with blonde hair
Hi-top fade face with hair face with Hi-top fade hair
Blue eyes face with eyes face with blue eyes

More examples could be found in the video and in the paper.

Pratice Tips:

In the terminal, for every manipulation, the number of channels being manipulated is printed (the number is controlled by the attribute (neutral, target) and the disentanglement threshold).

  1. For color transformation, usually 10-20 channels is enough. For large structure change (for example, Hi-top fade), usually 100-200 channels are required.
  2. For an attribute (neutral, target), if you give a low disentanglement threshold, there are just few channels (<20) being manipulated, and usually it is not enough for performing the desired edit.

Notebook

Open the notebook in colab and run all the cells. In the last cell you can play with the image.

beta corresponds to the disentanglement threshold, and alpha to the manipulation strength.

After you set the desired set of parameters, please run again the last cell to generate the image.

Editing Examples

In the following, we show some results obtained with our methods. All images are real, and were inverted into the StyleGAN's latent space using e4e. The driving text that was used for each edit appears below or above each image.

Latent Optimization

Latent Mapper

Global Directions

Related Works

The global directions we find for editing are direction in the S Space, which was introduced and analyzed in StyleSpace (Wu et al).

To edit real images, we inverted them to the StyleGAN's latent space using e4e (Tov et al.).

The code strcuture of the mapper is heavily based on pSp.

Citation

If you use this code for your research, please cite our paper:

@misc{patashnik2021styleclip,
      title={StyleCLIP: Text-Driven Manipulation of StyleGAN Imagery}, 
      author={Or Patashnik and Zongze Wu and Eli Shechtman and Daniel Cohen-Or and Dani Lischinski},
      year={2021},
      eprint={2103.17249},
      archivePrefix={arXiv},
      primaryClass={cs.CV}
}
Issues
  • [Errno 2] No such file or directory: 'stylegan2-ffhq-config-f.pt'

    [Errno 2] No such file or directory: 'stylegan2-ffhq-config-f.pt'

    error in colab

    [Errno 2] No such file or directory: 'stylegan2-ffhq-config-f.pt'

    from main import main
    from argparse import Namespace
    result = main(Namespace(**args))
    
    
    ---------------------------------------------------------------------------
    FileNotFoundError                         Traceback (most recent call last)
    <ipython-input-4-fbc1a59be2c9> in <module>()
          1 from main import main
          2 from argparse import Namespace
    ----> 3 result = main(Namespace(**args))
    
    3 frames
    /usr/local/lib/python3.6/dist-packages/torch/serialization.py in __init__(self, name, mode)
        209 class _open_file(_opener):
        210     def __init__(self, name, mode):
    --> 211         super(_open_file, self).__init__(open(name, mode))
        212 
        213     def __exit__(self, *args):
    
    FileNotFoundError: [Errno 2] No such file or directory: 'stylegan2-ffhq-config-f.pt'
    
    opened by molo32 8
  • Difficulty producing convincing results.

    Difficulty producing convincing results.

    I got this result with:

    python3 main.py --description "Blue Hair" --mode=edit
    

    image

    Do I need to change something in the invocation to get better results?

    opened by stolk 8
  • How to invert and edit an image

    How to invert and edit an image

    Hello,

    I looked at the codes but couldn't locate the code that inverts the image so it can be edited. You said you used e4e, but I couldn't find the relevant codes.

    Thank you.

    opened by ramtiin 6
  • CUDA 11 environment installation config?

    CUDA 11 environment installation config?

    Hey there, been having some trouble installing on RTX 30 series (which require CUDA 11), and was hoping I could get some tips on how to get it working (either from the developers or other users who've encountered the same problem?)

    Not sure if this is being run on a CUDA 11 environment in your group at present (if so please let me know what I've missed!), I've not been able to run the global/ subdirectory code, beginning with GetCode.py on either Python 3.6 or 3.7 (while 3.8 and above are incompatible, they require TensorFlow 2.2).

    I installed via conda as the pip installed TensorFlow was built against CUDA 10, and raised errors about missing *.so.10 libraries as a result, which disappeared when using the conda-forge package.

    After getting an error that "Setting up TensorFlow plugin "fused_bias_act.cu": Failed!" I tried some advice on an NVIDIA forum post for StyleGAN2, to change line 135 of global/dnnlib/tflib/custom_ops.py to

    compile_opts += f' --compiler-options \'-fPIC -D_GLIBCXX_USE_CXX11_ABI=1\''
    

    However this had no effect: there still seems to be a failure to register the GPU.

    To check whether I can use the environment I'm running

    StyleCLIP/global $ python GetCode.py --code_type "w"
    

    My environment setups (3.6 and 3.7 respectively) are as follows (after each is the error output for that environment)

    Click to show setup for Python 3.6, CUDA 11.0.221, PyTorch 1.7.1, TensorFlow 1.14.0

    conda create -n styleclip4
    conda activate styleclip4
    conda install -y "python<3.7" -c conda-forge # Python 3.6.13 (restricted by TensorFlow 1.x dependency)
    conda install -y pytorch==1.7.1 torchvision "cudatoolkit<11.2" -c pytorch
    # PyPi tensorflow-gpu package is built for CUDA 10, incompatible with 11, use conda-forge community package
    conda install "tensorflow-gpu<2" -c conda-forge # 1.14.0
    pip install git+https://github.com/openai/CLIP.git # forces pytorch 1.7.1 install
    pip install pandas requests opencv-python matplotlib scikit-learn gdown
    gdown https://drive.google.com/u/0/uc?id=1EM87UquaoQmk17Q8d5kYIAHqu0dkYqdT&export=download
    git clone https://github.com/omertov/encoder4editing.git
    

    Gives:

    /home/louis/miniconda3/envs/styleclip4/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:516: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
      _np_qint8 = np.dtype([("qint8", np.int8, 1)])
    /home/louis/miniconda3/envs/styleclip4/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:517: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
      _np_quint8 = np.dtype([("quint8", np.uint8, 1)])
    /home/louis/miniconda3/envs/styleclip4/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:518: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
      _np_qint16 = np.dtype([("qint16", np.int16, 1)])
    /home/louis/miniconda3/envs/styleclip4/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:519: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
      _np_quint16 = np.dtype([("quint16", np.uint16, 1)])
    /home/louis/miniconda3/envs/styleclip4/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:520: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
      _np_qint32 = np.dtype([("qint32", np.int32, 1)])
    /home/louis/miniconda3/envs/styleclip4/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:525: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
      np_resource = np.dtype([("resource", np.ubyte, 1)])
    /home/louis/miniconda3/envs/styleclip4/lib/python3.6/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:541: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
      _np_qint8 = np.dtype([("qint8", np.int8, 1)])
    /home/louis/miniconda3/envs/styleclip4/lib/python3.6/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:542: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
      _np_quint8 = np.dtype([("quint8", np.uint8, 1)])
    /home/louis/miniconda3/envs/styleclip4/lib/python3.6/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:543: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
      _np_qint16 = np.dtype([("qint16", np.int16, 1)])
    /home/louis/miniconda3/envs/styleclip4/lib/python3.6/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:544: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
      _np_quint16 = np.dtype([("quint16", np.uint16, 1)])
    /home/louis/miniconda3/envs/styleclip4/lib/python3.6/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:545: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
      _np_qint32 = np.dtype([("qint32", np.int32, 1)])
    /home/louis/miniconda3/envs/styleclip4/lib/python3.6/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:550: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
      np_resource = np.dtype([("resource", np.ubyte, 1)])
    Setting up TensorFlow plugin "fused_bias_act.cu": Failed!
    Traceback (most recent call last):
      File "GetCode.py", line 284, in <module>
        GetCode(Gs,random_state,num_img,num_once,dataset_name)
      File "GetCode.py", line 109, in GetCode
        dlatent_avg=Gs.get_var('dlatent_avg')
      File "/home/louis/dev/cv/StyleCLIP/global/dnnlib/tflib/network.py", line 396, in get_var
        return self.find_var(var_or_local_name).eval()
      File "/home/louis/dev/cv/StyleCLIP/global/dnnlib/tflib/network.py", line 391, in find_var
        return self._get_vars()[var_or_local_name] if isinstance(var_or_local_name, str) else var_or_local_name
      File "/home/louis/dev/cv/StyleCLIP/global/dnnlib/tflib/network.py", line 297, in _get_vars
        self._vars = OrderedDict(self._get_own_vars())
      File "/home/louis/dev/cv/StyleCLIP/global/dnnlib/tflib/network.py", line 286, in _get_own_vars
        self._init_graph()
      File "/home/louis/dev/cv/StyleCLIP/global/dnnlib/tflib/network.py", line 151, in _init_graph
        out_expr = self._build_func(*self._input_templates, **build_kwargs)
      File "<string>", line 187, in G_main
      File "/home/louis/dev/cv/StyleCLIP/global/dnnlib/tflib/network.py", line 232, in input_shape
        return self.input_shapes[0]
      File "/home/louis/dev/cv/StyleCLIP/global/dnnlib/tflib/network.py", line 219, in input_shapes
        self._input_shapes = [t.shape.as_list() for t in self.input_templates]
      File "/home/louis/dev/cv/StyleCLIP/global/dnnlib/tflib/network.py", line 267, in input_templates
        self._init_graph()
      File "/home/louis/dev/cv/StyleCLIP/global/dnnlib/tflib/network.py", line 151, in _init_graph
        out_expr = self._build_func(*self._input_templates, **build_kwargs)
      File "<string>", line 491, in G_synthesis_stylegan2
      File "<string>", line 455, in layer
      File "<string>", line 99, in modulated_conv2d_layer
      File "<string>", line 68, in apply_bias_act
      File "/home/louis/dev/cv/StyleCLIP/global/dnnlib/tflib/ops/fused_bias_act.py", line 72, in fused_bias_act
        return impl_dict[impl](x=x, b=b, axis=axis, act=act, alpha=alpha, gain=gain, clamp=clamp)
      File "/home/louis/dev/cv/StyleCLIP/global/dnnlib/tflib/ops/fused_bias_act.py", line 132, in _fused_bias_act_cuda
        cuda_op = _get_plugin().fused_bias_act
      File "/home/louis/dev/cv/StyleCLIP/global/dnnlib/tflib/ops/fused_bias_act.py", line 18, in _get_plugin
        return custom_ops.get_plugin(os.path.splitext(__file__)[0] + '.cu')
      File "/home/louis/dev/cv/StyleCLIP/global/dnnlib/tflib/custom_ops.py", line 139, in get_plugin
        compile_opts += f' --gpu-architecture={_get_cuda_gpu_arch_string()}'
      File "/home/louis/dev/cv/StyleCLIP/global/dnnlib/tflib/custom_ops.py", line 60, in _get_cuda_gpu_arch_string
        raise RuntimeError('No GPU devices found')
    RuntimeError: No GPU devices found
    

    Click to show setup for Python 3.7, CUDA 11.0.221, PyTorch 1.7.1, TensorFlow 1.14.0

    conda create -n styleclip3
    conda activate styleclip3
    conda install -y "python<3.8" -c conda-forge # Python 3.7.10 (restricted by TensorFlow 1.x dependency)
    conda install -y pytorch==1.7.1 torchvision "cudatoolkit<11.2" -c pytorch
    # PyPi tensorflow-gpu package is built for CUDA 10, incompatible with 11, use conda-forge community package
    conda install "tensorflow-gpu<2" -c conda-forge # 1.14.0
    pip install git+https://github.com/openai/CLIP.git # forces pytorch 1.7.1 install
    pip install pandas requests opencv-python matplotlib scikit-learn gdown
    gdown https://drive.google.com/u/0/uc?id=1EM87UquaoQmk17Q8d5kYIAHqu0dkYqdT&export=download
    git clone https://github.com/omertov/encoder4editing.git
    
    /home/louis/miniconda3/envs/styleclip3/lib/python3.7/site-packages/tensorflow/python/framework/dtypes.py:516: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
      _np_qint8 = np.dtype([("qint8", np.int8, 1)])
    /home/louis/miniconda3/envs/styleclip3/lib/python3.7/site-packages/tensorflow/python/framework/dtypes.py:517: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
      _np_quint8 = np.dtype([("quint8", np.uint8, 1)])
    /home/louis/miniconda3/envs/styleclip3/lib/python3.7/site-packages/tensorflow/python/framework/dtypes.py:518: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
      _np_qint16 = np.dtype([("qint16", np.int16, 1)])
    /home/louis/miniconda3/envs/styleclip3/lib/python3.7/site-packages/tensorflow/python/framework/dtypes.py:519: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
      _np_quint16 = np.dtype([("quint16", np.uint16, 1)])
    /home/louis/miniconda3/envs/styleclip3/lib/python3.7/site-packages/tensorflow/python/framework/dtypes.py:520: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
      _np_qint32 = np.dtype([("qint32", np.int32, 1)])
    /home/louis/miniconda3/envs/styleclip3/lib/python3.7/site-packages/tensorflow/python/framework/dtypes.py:525: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
      np_resource = np.dtype([("resource", np.ubyte, 1)])
    /home/louis/miniconda3/envs/styleclip3/lib/python3.7/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:541: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
      _np_qint8 = np.dtype([("qint8", np.int8, 1)])
    /home/louis/miniconda3/envs/styleclip3/lib/python3.7/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:542: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
      _np_quint8 = np.dtype([("quint8", np.uint8, 1)])
    /home/louis/miniconda3/envs/styleclip3/lib/python3.7/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:543: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
      _np_qint16 = np.dtype([("qint16", np.int16, 1)])
    /home/louis/miniconda3/envs/styleclip3/lib/python3.7/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:544: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
      _np_quint16 = np.dtype([("quint16", np.uint16, 1)])
    /home/louis/miniconda3/envs/styleclip3/lib/python3.7/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:545: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
      _np_qint32 = np.dtype([("qint32", np.int32, 1)])
    /home/louis/miniconda3/envs/styleclip3/lib/python3.7/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:550: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
      np_resource = np.dtype([("resource", np.ubyte, 1)])
    Setting up TensorFlow plugin "fused_bias_act.cu": Failed!
    Traceback (most recent call last):
      File "GetCode.py", line 284, in <module>
        GetCode(Gs,random_state,num_img,num_once,dataset_name)
      File "GetCode.py", line 109, in GetCode
        dlatent_avg=Gs.get_var('dlatent_avg')
      File "/home/louis/dev/cv/StyleCLIP/global/dnnlib/tflib/network.py", line 396, in get_var
        return self.find_var(var_or_local_name).eval()
      File "/home/louis/dev/cv/StyleCLIP/global/dnnlib/tflib/network.py", line 391, in find_var
        return self._get_vars()[var_or_local_name] if isinstance(var_or_local_name, str) else var_or_local_name
      File "/home/louis/dev/cv/StyleCLIP/global/dnnlib/tflib/network.py", line 297, in _get_vars
        self._vars = OrderedDict(self._get_own_vars())
      File "/home/louis/dev/cv/StyleCLIP/global/dnnlib/tflib/network.py", line 286, in _get_own_vars
        self._init_graph()
      File "/home/louis/dev/cv/StyleCLIP/global/dnnlib/tflib/network.py", line 151, in _init_graph
        out_expr = self._build_func(*self._input_templates, **build_kwargs)
      File "<string>", line 187, in G_main
      File "/home/louis/dev/cv/StyleCLIP/global/dnnlib/tflib/network.py", line 232, in input_shape
        return self.input_shapes[0]
      File "/home/louis/dev/cv/StyleCLIP/global/dnnlib/tflib/network.py", line 219, in input_shapes
        self._input_shapes = [t.shape.as_list() for t in self.input_templates]
      File "/home/louis/dev/cv/StyleCLIP/global/dnnlib/tflib/network.py", line 267, in input_templates
        self._init_graph()
      File "/home/louis/dev/cv/StyleCLIP/global/dnnlib/tflib/network.py", line 151, in _init_graph
        out_expr = self._build_func(*self._input_templates, **build_kwargs)
      File "<string>", line 491, in G_synthesis_stylegan2
      File "<string>", line 455, in layer
      File "<string>", line 99, in modulated_conv2d_layer
      File "<string>", line 68, in apply_bias_act
      File "/home/louis/dev/cv/StyleCLIP/global/dnnlib/tflib/ops/fused_bias_act.py", line 72, in fused_bias_act
        return impl_dict[impl](x=x, b=b, axis=axis, act=act, alpha=alpha, gain=gain, clamp=clamp)
      File "/home/louis/dev/cv/StyleCLIP/global/dnnlib/tflib/ops/fused_bias_act.py", line 132, in _fused_bias_act_cuda
        cuda_op = _get_plugin().fused_bias_act
      File "/home/louis/dev/cv/StyleCLIP/global/dnnlib/tflib/ops/fused_bias_act.py", line 18, in _get_plugin
        return custom_ops.get_plugin(os.path.splitext(__file__)[0] + '.cu')
      File "/home/louis/dev/cv/StyleCLIP/global/dnnlib/tflib/custom_ops.py", line 139, in get_plugin
        compile_opts += f' --gpu-architecture={_get_cuda_gpu_arch_string()}'
      File "/home/louis/dev/cv/StyleCLIP/global/dnnlib/tflib/custom_ops.py", line 60, in _get_cuda_gpu_arch_string
        raise RuntimeError('No GPU devices found')
    RuntimeError: No GPU devices found
    

    Requiring packages come from the anaconda channel instead for some reason is enforcing TensorFlow 1.10.0, which (once you get past some interface changes) still leads to the same GPU registration problem.

    Click to show setup for Python 3.6, CUDA 11.0.221, PyTorch 1.7.1, TensorFlow 1.10.0 (anaconda channel)

    conda create -n styleclip6
    conda activate styleclip6
    conda install -y "python<3.7" -c anaconda # Python 3.6.13 (restricted by TensorFlow 1.x dependency)
    conda install -y pytorch==1.7.1 torchvision "cudatoolkit>=11,<11.2" -c pytorch
    # PyPi tensorflow-gpu package is built for CUDA 10, incompatible with 11, use conda-forge community package
    conda install "tensorflow-gpu<2" -c anaconda # 1.10.0
    pip install git+https://github.com/openai/CLIP.git # forces pytorch 1.7.1 install
    pip install pandas requests opencv-python matplotlib scikit-learn gdown
    gdown https://drive.google.com/u/0/uc?id=1EM87UquaoQmk17Q8d5kYIAHqu0dkYqdT&export=download
    git clone https://github.com/omertov/encoder4editing.git
    

    CUDA can also be downloaded from conda-forge but upon installing pytorch, the package is superseded by the higher-priority cudatoolkit package in that channel (making it equivalent to the attempt above which failed)

    I'm out of ideas so giving up at this point, please let me know if there's a solution!

    opened by lmmx 5
  • Load other images, but return ValueError

    Load other images, but return ValueError

    I tried it with my pictures but failed all, and it returns: ValueError: operands could not be broadcast together with shapes (1,1,0) (1,1,512) (1,1,0), is my picture different from demo images? My 2 color pictures include a cat and a human face, both same size 1024*1024.

    opened by zhangjingcode 4
  • Using a user-specified image for the Global Directions Colab Notebook

    Using a user-specified image for the Global Directions Colab Notebook

    The Colab notebook for e2e creates a (1, 18, 512) latent vector which works with the Global Optimization network. However the Global Directions Notebook wants a 26-element list with 15 512-vectors, 3 256-vectors, 3 128-vectors, 3 64-vectors, and 2 32-vectors.

    How do you get a compatible vector from e2e for custom images to global directions?

    opened by minimaxir 4
  • What is the meaning of fs3?

    What is the meaning of fs3?

    Hi,I am following your work and thank you for your excellent paper.

    However, I have some confusion, please help me.

    1. What is the meaning of fs3 in the code, I don't know what that means or what it does.
    2. What do you estimate with 200 images, which I don't really understand in the paper.
    3. How to get delta_i or delta_ic in the paper?

    Looking forward to your responses, thank you very much!!!

    opened by sunpeng1996 3
  • Training on my own dataset?

    Training on my own dataset?

    Hello team, Thank your for this amazing project. I'd like to train StyleCLIP on own my dataset. I have the images already. What other data do I have to prepare? Could you please let me know the detail steps to train it?

    opened by tamnguyenvan 3
  • Utility method to download weights #17

    Utility method to download weights #17

    Automatically downloads weights from Google Drive using 'gdown' if not found. Updates readme. The dictionary in utils.py can be updated with new filenames and matching URLs if weights are added. @orpatashnik @Skylion007

    opened by lukestanley 3
  • Speeding up trainer - avenue for exploring

    Speeding up trainer - avenue for exploring

    I watched some youtube videos and came across pytorch lightning which supposedly simplifies distributed trainining. https://pytorch-lightning.readthedocs.io/en/latest/common/trainer.html (I don't want to wait 10 hours for training to finish / mainly because of the power usage on my gpu)

    opened by johndpope 3
  • How to encode existing image?

    How to encode existing image?

    The README suggests using e2e for encoding existing face image, but this repo seems to be empty. I tried using stylegan-encoder instead, but it is based on Tensorflow and I got stuck with downloading the model with "Google Drive quota exceeded" error. Any suggestions for what to use for StyleGAN encoding?

    opened by tambetm 3
  • Head cropping

    Head cropping

    Can we make the whole head visible? crop

    opened by qo4on 3
  • [Feature Request] Seeds for randomized initialization of free_generation

    [Feature Request] Seeds for randomized initialization of free_generation

    Simple: Currently free_generation starts from the average face of the model and diverges from there, but I think it would be far more interesting if it could start with randomized faces when given a seed, like the default generate.py in StyleGAN2-ADA / run_generator.py in StyleGAN2.

    Granular Control: Given that random faces may be a bit too random, perhaps a control for how similar / dissimilar the initial faces should be by lerping with the dlatent_avg like truncation_psi does by default, but applied to the latent of the face itself and not passed to StyleGAN2 so that it can still modify faces with a truncation_psi of 1.0 or whatever else is desired.

    Perhaps per layer or per layer range "faux truncation_psi" on the initial latents to allow for wild colors and features but a plain forward-facing head or any other combination of similar / dissimilar elements per layer. If simplified layer ranges are preferred, I'd recommend trying 0-5, 6-11, 12-17 as a starting point. These seem to be where facial shape switches to facial details switches to global coloration.

    opened by torridgristle 3
  • wrong result when editing other image with GUI

    wrong result when editing other image with GUI

    Hi, Thanks for your awesome work! I tried to edit with GUI by python PlayInteractively.py --dataset_name $dataset_name. On the official demo image, I get a good result by editing, that's really cool! Unfortunately, when I tried to edit with the other images which I put into the test dataset after e4e inverted (python GetGUIData.py --dataset_name $dataset_name), it seems to change to another image after editing? Images are in 1024x1024 resolution as same as the demo given. What should I do editing with other images? Thanks!

    official image editing result: src other image editing result: ours

    opened by juneleung 2
  • how to get dlatent_tmp if I have a real image?

    how to get dlatent_tmp if I have a real image?

    thank your great project,I read example in the StyleCLIP_global.ipynb,I find dlatent_tmp is pregenerated.however,I want to edit a real image ,What shall I do?

    opened by TomatoBoy90 2
  • Wrong result when “Editing via Latent Mapper”

    Wrong result when “Editing via Latent Mapper”

    Thank you for your amazing work.

    But when I finish configuring the environment and use the following example:

    python train.py --exp_dir ../results/mohawk_hairstyle --no_fine_mapper --description "mohawk hairstyle"

    I found clip_ Loss will not decrease,Do you have any advice or help.

    clip_loss

    opened by MeowChiu 2
  • Command errored out with exit status 128: git clone -q https://github.com/openai/CLIP.git

    Command errored out with exit status 128: git clone -q https://github.com/openai/CLIP.git

    (base) [email protected] Downloads % pip install git+https://github.com/openai/CLIP.git Collecting git+https://github.com/openai/CLIP.git Cloning https://github.com/openai/CLIP.git to /private/var/folders/kr/v9ktnnhn5f11s9pp5977pwqr0000gn/T/pip-req-build-jq1hz_wh ERROR: Command errored out with exit status 128: git clone -q https://github.com/openai/CLIP.git /private/var/folders/kr/v9ktnnhn5f11s9pp5977pwqr0000gn/T/pip-req-build-jq1hz_wh Check the logs for full command output.

    Anyone can help, thanks.

    opened by kennethliukai 2
  • About colab of Editing via Global Direction

    About colab of Editing via Global Direction

    About colab of Editing via Global Direction. How can I use the pt created by e4e instead of selecting the source image by image index?

    opened by cedro3 2
  • Style space of latent optimization and mapper

    Style space of latent optimization and mapper

    Hello I have read your paper and tested some examples on provided Colab. It was very interesting, but it left some questions about the style space S and W+.

    According to the mentioned paper, it seems that style space S is more disentangled than style space W+. Is there any reason why you use w+ space instead of S in the latent optimization and mapper method?

    opened by ssumin6 2
  • Colab OOM error when trying to use `latent_path` in the

    Colab OOM error when trying to use `latent_path` in the "Optimization" notebook

    As discussed in a previous issue (https://github.com/orpatashnik/StyleCLIP/issues/1), and suggested by @woctezuma I used this notebook to project a face image for editing.

    I saved the latent with torch.save(latent, 'latent.pt') and then uploaded it to the Optmization notebook environment. It loads fine, and the shape seems correct (torch.Size([18, 512]), I added some prints in the run_optimization.py) but I keep getting the following error:

    RuntimeError: CUDA out of memory. Tried to allocate 1.12 GiB (GPU 0; 15.90 GiB total capacity; 14.74 GiB already allocated; 221.75 MiB free; 14.88 GiB reserved in total by PyTorch)

    which I don't understand, since it should be pretty much the same whether I use a random latent or the one I have saved, or at least that's what I understand by looking at the code. Any ideas?

    I'm attaching the latent, just in case: latent.zip

    Thank you :)

    opened by bmachin 2
  • Latent Mapper Training

    Latent Mapper Training

    Incredible work done here - congratulations on the fantastic results.

    Is there a timeline for code for training the latent mapper technique by chance? Happy to pitch in on development work as well on that front if necessary.

    opened by ghunkins 2
  • How to use stylegan2-ada with StyleCLIP

    How to use stylegan2-ada with StyleCLIP

    Hello and thanks for sharing your wonderful work!

    Was curious, is there a way to utilize ada models with your work?

    opened by TheodoreGalanos 2
  • [Help] Unbound Local Error

    [Help] Unbound Local Error

    How do you fix "UnboundLocalError: local variable 'shape' referenced before assignment". I have no coding experience so hopefully it's not too hard.

    opened by maparicio23 2
  • [Question] How to generate images without optimization

    [Question] How to generate images without optimization

    Awesome paper! It is an excellent result. I watched the youtube video that you uploaded yesterday. In the demo, it generates images in real-time without optimization. When I experimented with your colab, I had to optimize it for each text. How do you generate images like the youtube demo? If possible, it would be great if you could publish the code of the youtube demo.

    opened by tommy19970714 2
  • How to improve e4e inversion

    How to improve e4e inversion

    Hi, first of all thanks your your amazing work :)

    Is there a way to improve e4e image inversion (e.g. increasing some kind of iteration parameters?). I've tried with some faces and sometimes the inverted result has little resemblance with original one... many thanks in advance

    opened by smithee77 2
  • CUDA out of memory when using own latent

    CUDA out of memory when using own latent

    Hi, great repo, love the idea.

    I have an issue with CUDA memory in the Google Colab playground notebook: CUDA out of memory. Tried to allocate 1.12 GiB (GPU 0; 14.76 GiB total capacity; 13.61 GiB already allocated; 75.75 MiB free; 13.76 GiB reserved in total by PyTorch)

    It happens at the start of the optimization process. I have used the recommended e4e to invert my own image. If relevant: the size of the latent is torch.Size([18, 512]).

    Is there any easy way to reduce the memory footprint of the notebook?

    Thanks in advance.

    opened by gdewael 2
  • [Notebook Form]

    [Notebook Form] "description" should be specified as a String

    Currently the form with the description input is description = 'A person with purple hair' #@param but this results in the user having to add quotes to the description themselves. Instead, it should be description = 'A person with purple hair' #@param {type:"string"} to have Google Colab apply the quotes automatically on its own.

    I think latent_path = None #@param should also have {type:"string"} specified in case of spaces, and to avoid needing the user to add quotes on their own.

    opened by torridgristle 2
  • [Feature Request] Generate Video of Rendered Frames, Included FFMPEG Command

    [Feature Request] Generate Video of Rendered Frames, Included FFMPEG Command

    I'm sure there's a more elegant way to implement this but my personal solution for rendering a nice video of the frames is to use ffmpeg as installed on Colab and its motion interpolation to generate a smooth video. It may be faster to just save more frames during the process, it may also end up jerkier if it goes back and forth at any point, but I haven't tested these things. I just set it and left it as is on my personal copy.

    !ffmpeg -pattern_type glob -r 5 -i './results/*.png' -filter:v "minterpolate='fps=15:mb_size=8:search_param=8:vsbmc=0:scd=none:mc_mode=obmc:me_mode=bidir'" -vcodec libx264 -crf 28 -pix_fmt yuv420p out.mp4

    It takes the frames as 5fps input and interpolates it 3x to 15fps and saves it in a reasonable quality MP4 with a codec and pixel format that is supported by most devices and websites, as opposed to the output it gives without specifying vcodec and pix_fmt and only tell it to output an mp4. Doing it that way causes issues on Twitter, Tumblr, and some devices just won't play it, so I'd recommend leaving those parts in.

    Perhaps the prompt could be the output name like "$description.mp4" but with some measure taken to truncate long descriptions.

    opened by torridgristle 2
  • The choice of parameters

    The choice of parameters

    Thank you for sharing.

    I played on the colab and obtained unpleasing results at most time. It is hard to decide what vaule to use if different edits require different parameters of the CLIP loss.

    Then I tried to use the pretrained CLIP model in my own work TediGAN.

    clip_results

    The obtained results are not sensitive to the weights.

    clip_results_cw

    Adding additional perceptual loss and image reconstruction loss may help stabilize the optimization process.

    opened by weihaox 2
  • Typos in

    Typos in "Optimization" notebook arguments

    Great work! Thank you for sharing it.

    I found the following typos in the "Optimization" notebook:

    • in the "Additional arguments" cell, the parameter size should be stylegan_size.
    • and, in the next cell, instead of from main import main it should be from optimization.run_optimization import main, I think.

    Other than that, it worked perfectly :)

    opened by bmachin 1
  • inference.py: ModuleNotFoundError: No module named 'mapper'

    inference.py: ModuleNotFoundError: No module named 'mapper'

    Run:

    python mapper/scripts/inference.py
    

    Result:

    Traceback (most recent call last):
      File "./mapper/scripts/inference.py", line 13, in <module>
        from mapper.training.train_utils import convert_s_tensor_to_list
    ModuleNotFoundError: No module named 'mapper'
    
    opened by yar3333 1
  • CPU or AMD GPU

    CPU or AMD GPU

    I would like to know is it possible to do all the procedures using only the device's CPU, or if it is possible with an AMD gpu

    opened by HibiVixx 0
  • About Mapper in Latent Mapper, does PixelNorm normalize the right dim?

    About Mapper in Latent Mapper, does PixelNorm normalize the right dim?

    Thanks for sharing this wonderful work! Here I am a little confused with the normalization layer: https://github.com/orpatashnik/StyleCLIP/blob/adc836d9077b241d92e9ce88da8cb6c01ada2999/mapper/latent_mappers.py#L16 In the StyleGAN2 mapper, as the input is always (Batch, latent_dim), the normalization layer is right here. https://github.com/orpatashnik/StyleCLIP/blob/adc836d9077b241d92e9ce88da8cb6c01ada2999/models/stylegan2/model.py#L16 But in the latent mapper, I think the input is always (Batch, n_latent, latent_dim)(Maybe I am wrong here), thus the normalization layer seems doesn't do the right normalization. I think for stylespace mapper, this reshape is right: https://github.com/orpatashnik/StyleCLIP/blob/adc836d9077b241d92e9ce88da8cb6c01ada2999/mapper/latent_mappers.py#L98 But for the others, some operations that are not intended may occur as PixelNorm normalize the wrong dim. https://github.com/orpatashnik/StyleCLIP/blob/adc836d9077b241d92e9ce88da8cb6c01ada2999/mapper/latent_mappers.py#L62-L67 Am I right or do I misunderstand something? Look forward to you reply, thanks.

    opened by Big-Brother-Pikachu 0
  • Problem in generating w, s, s_mean_std, and fs3 for non SG2 Ada (non 512x512)

    Problem in generating w, s, s_mean_std, and fs3 for non SG2 Ada (non 512x512)

    I am currently generating the w, s, s_mean_std, and fs3 for stylegan-car and cat from the official SG2 (non-ada) implementation. https://github.com/NVlabs/stylegan2

    These models are not 512x512 but are 256x256.

    opened by sarmientoj24 0
  • What is the fs3 for the original SG2 models like stylegan cat (not AFHQ)?

    What is the fs3 for the original SG2 models like stylegan cat (not AFHQ)?

    Or how do I calculate it?

    opened by sarmientoj24 3
  • nvlabs-sytlegan2-ada-pytorch version of StyleCLIP implementation (global directions)

    nvlabs-sytlegan2-ada-pytorch version of StyleCLIP implementation (global directions)

    Thank you for your great work @orpatashnik
    Personally, I hate tf (lot of version issue, etc)..... so I implemented the global methods StyleCLIP as a pytorch version.

    https://github.com/soushirou/StyleCLIP-pytorch

    Image inversion will be added next week :)

    opened by soushirou 0
  • How to use StyleCLIP on a StyleGAN2 that was trained on a custom, non-FFHQ dataset?

    How to use StyleCLIP on a StyleGAN2 that was trained on a custom, non-FFHQ dataset?

    Questions:

    1. I cannot seem to find any explicit instructions on how to use/train StyleCLIP on a trained StyleGAN2 (custom dataset, non-human face). Could you elaborate on this process? How do I incorporate the trained SG2?
    2. Do I need to still parts of the StyleCLIP for this process or can I use it outright?
    opened by sarmientoj24 5
  • Licensing?

    Licensing?

    This YouTube channel is selling use of this software via their Patreon:

    https://www.youtube.com/watch?v=sboiitHStWQ&t=251s

    opened by taylus 0
  • could the project with updated with the latest StyleGAN3?

    could the project with updated with the latest StyleGAN3?

    Newer StyleGAN3 would be a great update considering the improvements in generation.

    opened by GeorvityLabs 0
  • using .pkl file in global directions

    using .pkl file in global directions

    i have custom dataset and i used stylegan for image generation and now i have pretrained .pkl file. how can i use this file to generate images from text using global direction? thanks in advance

    opened by aksajanjua 0
Implementation of Analyzing and Improving the Image Quality of StyleGAN (StyleGAN 2) in PyTorch

Implementation of Analyzing and Improving the Image Quality of StyleGAN (StyleGAN 2) in PyTorch

Kim Seonghyeon 1.7k Mar 4, 2022
FactSeg: Foreground Activation Driven Small Object Semantic Segmentation in Large-Scale Remote Sensing Imagery (TGRS)

FactSeg: Foreground Activation Driven Small Object Semantic Segmentation in Large-Scale Remote Sensing Imagery by Ailong Ma, Junjue Wang*, Yanfei Zhon

Kingdrone 29 Feb 7, 2022
ManiSkill-Learn is a framework for training agents on SAPIEN Open-Source Manipulation Skill Challenge (ManiSkill Challenge), a large-scale learning-from-demonstrations benchmark for object manipulation.

ManiSkill-Learn ManiSkill-Learn is a framework for training agents on SAPIEN Open-Source Manipulation Skill Challenge, a large-scale learning-from-dem

Hao Su's Lab, UCSD 36 Feb 22, 2022
Attention-driven Robot Manipulation (ARM) which includes Q-attention

Attention-driven Robotic Manipulation (ARM) This codebase is home to: Q-attention: Enabling Efficient Learning for Vision-based Robotic Manipulation I

Stephen James 26 Feb 8, 2022
StyleSpace Analysis: Disentangled Controls for StyleGAN Image Generation

StyleSpace Analysis: Disentangled Controls for StyleGAN Image Generation Demo video: CVPR 2021 Oral: Single Channel Manipulation: Localized or attribu

Zongze Wu 203 Feb 26, 2022
Official Implementation for "ReStyle: A Residual-Based StyleGAN Encoder via Iterative Refinement" https://arxiv.org/abs/2104.02699

ReStyle: A Residual-Based StyleGAN Encoder via Iterative Refinement Recently, the power of unconditional image synthesis has significantly advanced th

null 813 Feb 27, 2022
Non-Official Pytorch implementation of "Face Identity Disentanglement via Latent Space Mapping" https://arxiv.org/abs/2005.07728 Using StyleGAN2 instead of StyleGAN

Face Identity Disentanglement via Latent Space Mapping - Implement in pytorch with StyleGAN 2 Description Pytorch implementation of the paper Face Ide

Daniel Roich 41 Feb 10, 2022
VOGUE: Try-On by StyleGAN Interpolation Optimization

VOGUE is a StyleGAN interpolation optimization algorithm for photo-realistic try-on. Top: shirt try-on automatically synthesized by our method in two different examples.

Wei ZHANG 51 Feb 22, 2022
Implementation of StyleSpace Analysis: Disentangled Controls for StyleGAN Image Generation in PyTorch

StyleSpace Analysis: Disentangled Controls for StyleGAN Image Generation Implementation of StyleSpace Analysis: Disentangled Controls for StyleGAN Ima

Xuanchi Ren 66 Feb 18, 2022
Segmentation in Style: Unsupervised Semantic Image Segmentation with Stylegan and CLIP

Segmentation in Style: Unsupervised Semantic Image Segmentation with Stylegan and CLIP Abstract: We introduce a method that allows to automatically se

Daniil Pakhomov 105 Feb 6, 2022
Official implementation of "StyleCariGAN: Caricature Generation via StyleGAN Feature Map Modulation" (SIGGRAPH 2021)

StyleCariGAN in PyTorch Official implementation of StyleCariGAN:Caricature Generation via StyleGAN Feature Map Modulation in PyTorch Requirements PyTo

PeterZhouSZ 49 Jan 5, 2022
GAN encoders in PyTorch that could match PGGAN, StyleGAN v1/v2, and BigGAN. Code also integrates the implementation of these GANs.

MTV-TSA: Adaptable GAN Encoders for Image Reconstruction via Multi-type Latent Vectors with Two-scale Attentions. This is the official code release fo

owl 30 Dec 22, 2021
Official implementation of "StyleCariGAN: Caricature Generation via StyleGAN Feature Map Modulation" (SIGGRAPH 2021)

StyleCariGAN: Caricature Generation via StyleGAN Feature Map Modulation This repository contains the official PyTorch implementation of the following

Wonjong Jang 214 Feb 23, 2022
(CVPR 2021) Lifting 2D StyleGAN for 3D-Aware Face Generation

Lifting 2D StyleGAN for 3D-Aware Face Generation Official implementation of paper "Lifting 2D StyleGAN for 3D-Aware Face Generation". Requirements You

Yichun Shi 45 Feb 24, 2022
StyleGAN - Official TensorFlow Implementation

StyleGAN — Official TensorFlow Implementation Picture: These people are not real – they were produced by our generator that allows control over differ

NVIDIA Research Projects 12.3k Mar 5, 2022
A tensorflow/keras implementation of StyleGAN to generate images of new Pokemon.

PokeGAN A tensorflow/keras implementation of StyleGAN to generate images of new Pokemon. Dataset The model has been trained on dataset that includes 8

null 16 Feb 27, 2022
Jittor 64*64 implementation of StyleGAN

StyleGanJittor (Tsinghua university computer graphics course) Overview Jittor 64

Song Shengyu 3 Jan 20, 2022
A web porting for NVlabs' StyleGAN2, to facilitate exploring all kinds characteristic of StyleGAN networks

This project is a web porting for NVlabs' StyleGAN2, to facilitate exploring all kinds characteristic of StyleGAN networks. Thanks for NVlabs' excelle

K.L. 146 Mar 1, 2022
Code and model benchmarks for "SEVIR : A Storm Event Imagery Dataset for Deep Learning Applications in Radar and Satellite Meteorology"

NeurIPS 2020 SEVIR Code for paper: SEVIR : A Storm Event Imagery Dataset for Deep Learning Applications in Radar and Satellite Meteorology Requirement

USAF - MIT Artificial Intelligence Accelerator 37 Feb 23, 2022
A utility to search, download and process Landsat 8 satellite imagery

Landsat-util Landsat-util is a command line utility that makes it easy to search, download, and process Landsat imagery. Docs For full documentation v

Development Seed 667 Feb 5, 2022
Satellite imagery for dummies.

felicette Satellite imagery for dummies. What can you do with this tool? TL;DR: Generate JPEG earth imagery from coordinates/location name with public

Shivashis Padhi 1.7k Mar 2, 2022
Download and process satellite imagery in Python using Sentinel Hub services.

Description The sentinelhub Python package allows users to make OGC (WMS and WCS) web requests to download and process satellite images within your Py

Sentinel Hub 582 Feb 24, 2022
Aerial Imagery dataset for fire detection: classification and segmentation (Unmanned Aerial Vehicle (UAV))

Aerial Imagery dataset for fire detection: classification and segmentation using Unmanned Aerial Vehicle (UAV) Title FLAME (Fire Luminosity Airborne-b

null 61 Feb 2, 2022
A collection of modules I have created to programmatically search for/download imagery from live cam feeds across the state of California.

A collection of modules that I have created to programmatically search for/download imagery from all publicly available live cam feeds across the state of California. In no way am I affiliated with any of these organizations and these modules/methods of gathering imagery are completely unofficial.

Chad Groom 4 Feb 17, 2022
Google Maps keeps old satellite imagery around for a while – this tool collects what's available for a user-specified region in the form of a GIF.

google-maps-at-88-mph The folks maintaining Google Maps regularly update the satellite imagery it serves its users, but outdated versions of the image

Noah Doersing 93 Feb 13, 2022
Experiments on Flood Segmentation on Sentinel-1 SAR Imagery with Cyclical Pseudo Labeling and Noisy Student Training

Flood Detection Challenge This repository contains code for our submission to the ETCI 2021 Competition on Flood Detection (Winning Solution #2). Acco

Siddha Ganju 69 Mar 1, 2022
Change is Everywhere: Single-Temporal Supervised Object Change Detection in Remote Sensing Imagery (ICCV 2021)

Change is Everywhere Single-Temporal Supervised Object Change Detection in Remote Sensing Imagery by Zhuo Zheng, Ailong Ma, Liangpei Zhang and Yanfei

Zhuo Zheng 77 Feb 23, 2022
A PyTorch implementation of Multi-digit Number Recognition from Street View Imagery using Deep Convolutional Neural Networks

SVHNClassifier-PyTorch A PyTorch implementation of Multi-digit Number Recognition from Street View Imagery using Deep Convolutional Neural Networks If

Potter Hsu 170 Feb 22, 2022