Instant Real-Time Example-Based Style Transfer to Facial Videos

Related tags

Deep LearningFaceBlit
Overview

FaceBlit: Instant Real-Time Example-Based Style Transfer to Facial Videos

The official implementation of

FaceBlit: Instant Real-Time Example-Based Style Transfer to Facial Videos
A. Texler, O. Texler, M. Kučera, M. Chai, and D. Sýkora
🌐 Project Page, 📄 Paper, 📚 BibTeX

FaceBlit is a system for real-time example-based face video stylization that retains textural details of the style in a semantically meaningful manner, i.e., strokes used to depict specific features in the style are present at the appropriate locations in the target image. As compared to previous techniques, our system preserves the identity of the target subject and runs in real-time without the need for large datasets nor lengthy training phase. To achieve this, we modify the existing face stylization pipeline of Fišer et al. [2017] so that it can quickly generate a set of guiding channels that handle identity preservation of the target subject while are still compatible with a faster variant of patch-based synthesis algorithm of Sýkora et al. [2019]. Thanks to these improvements we demonstrate a first face stylization pipeline that can instantly transfer artistic style from a single portrait to the target video at interactive rates even on mobile devices.

Teaser

Introduction

⚠️ DISCLAIMER: This is a research project, not a production-ready application, it may contain bugs!

This implementation is designed for two platforms - Windows and Android.

  • All C++ sources are located in FaceBlit/app/src/main/cpp, except for main.cpp and main_extension.cpp which can be found in FaceBlit/VS
  • All Java sources are stored in FaceBlit/app/src/main/java/texler/faceblit
  • Style exemplars (.png) are located in FaceBlit/app/src/main/res/drawable
  • Files holding detected landmarks (.txt) and lookup tables (.bytes) for each style are located in FaceBlit/app/src/main/res/raw
  • The algorithm assumes the style image and input video/image have the same resolution

Build and Run

  • Clone the repository git clone https://github.com/AnetaTexler/FaceBlit.git
  • The repository contains all necessary LIB files and includes for both platforms, except for the OpenCV DLL files for Windows
  • The project uses Dlib 19.21 which is added as one source file (FaceBlit/app/src/main/cpp/source.cpp) and will be compiled with other sources; so you don't have to worry about that

Windows

  • The OpenCV 4.5.0 is required, you can download the pre-built version directly from here and add opencv_world450d.dll and opencv_world450.dll files from opencv-4.5.0-vc14_vc15/build/x64/vc15/bin into your PATH
  • Open the solution FaceBlit/VS/FaceBlit.sln in Visual Studio (tested with VS 2019)
  • Provide a facial video/image or use existing sample videos and images in FaceBlit/VS/TESTS.
    • The input video/image has to be in resolution 768x1024 pixels (width x height)
  • In main() function in FaceBlit/VS/main.cpp, you can change parameters:
    • targetPath - path to input images and videos (there are some sample inputs in FaceBlit/VS/TESTS)
    • targetName - name of a target PNG image or MP4 video with extension (e.g. "target2.mp4")
    • styleName - name of a style with extension from the FaceBlit/app/src/main/res/drawable path (e.g. "style_het.png")
    • stylizeBG - true/false (true - stylize the whole image/video, does not always deliver pleasing results; false - stylize only face)
    • NNF_patchsize - voting patch size (odd number, ideal is 3 or 5); 0 for no voting
  • Finally, run the code and see results in FaceBlit/VS/TESTS

Android

  • OpenCV binaries (.so) are already included in the repository (FaceBlit/app/src/main/jniLibs)
  • Open the FaceBlit project in Android Studio (tested with Android Studio 4.1.3 and gradle 6.5), install NDK 21.0.6 via File > Settings > Appearance & Behavior > System Settings > Android SDK > SDK Tools and build the project.
  • Install the application on your mobile and face to the camera (works with both front and back). Press the right bottom button to display styles (scroll right to show more) and choose one. Wait a few seconds until the face detector loads, and enjoy the style transfer!

License

The algorithm is not patented. The code is released under the public domain - feel free to use it for research or commercial purposes.

Citing

If you find FaceBlit useful for your research or work, please use the following BibTeX entry.

@Article{Texler21-I3D,
    author    = "Aneta Texler and Ond\v{r}ej Texler and Michal Ku\v{c}era and Menglei Chai and Daniel S\'{y}kora",
    title     = "FaceBlit: Instant Real-time Example-based Style Transfer to Facial Videos",
    journal   = "Proceedings of the ACM in Computer Graphics and Interactive Techniques",
    volume    = "4",
    number    = "1",
    year      = "2021",
}
Owner
Aneta Texler
Aneta Texler
buildseg is a building extraction plugin of QGIS based on PaddlePaddle.

buildseg buildseg is a building extraction plugin of QGIS based on PaddlePaddle. TODO Extract building on 512x512 remote sensing images. Extract build

Yizhou Chen 11 Sep 26, 2022
Jittor implementation of PCT:Point Cloud Transformer

PCT: Point Cloud Transformer This is a Jittor implementation of PCT: Point Cloud Transformer.

MenghaoGuo 547 Jan 03, 2023
High-Resolution 3D Human Digitization from A Single Image.

PIFuHD: Multi-Level Pixel-Aligned Implicit Function for High-Resolution 3D Human Digitization (CVPR 2020) News: [2020/06/15] Demo with Google Colab (i

Meta Research 8.4k Dec 29, 2022
Code for "Continuous-Time Meta-Learning with Forward Mode Differentiation" (ICLR 2022)

Continuous-Time Meta-Learning with Forward Mode Differentiation ICLR 2022 (Spotlight) - Installation - Example - Citation This repository contains the

Tristan Deleu 25 Oct 20, 2022
This program automatically runs Python code copied in clipboard

CopyRun This program runs Python code which is copied in clipboard WARNING!! USE AT YOUR OWN RISK! NO GUARANTIES IF ANYTHING GETS BROKEN. DO NOT COPY

vertinski 4 Sep 10, 2021
Image-to-Image Translation in PyTorch

CycleGAN and pix2pix in PyTorch New: Please check out contrastive-unpaired-translation (CUT), our new unpaired image-to-image translation model that e

Jun-Yan Zhu 19k Jan 07, 2023
TransMIL: Transformer based Correlated Multiple Instance Learning for Whole Slide Image Classification

TransMIL: Transformer based Correlated Multiple Instance Learning for Whole Slide Image Classification [NeurIPS 2021] Abstract Multiple instance learn

132 Dec 30, 2022
PyTorch for Semantic Segmentation

PyTorch for Semantic Segmentation This repository contains some models for semantic segmentation and the pipeline of training and testing models, impl

Zijun Deng 1.7k Jan 06, 2023
Deal or No Deal? End-to-End Learning for Negotiation Dialogues

Introduction This is a PyTorch implementation of the following research papers: (1) Hierarchical Text Generation and Planning for Strategic Dialogue (

Facebook Research 1.4k Dec 29, 2022
A PyTorch implementation of "Multi-Scale Contrastive Siamese Networks for Self-Supervised Graph Representation Learning", IJCAI-21

MERIT A PyTorch implementation of our IJCAI-21 paper Multi-Scale Contrastive Siamese Networks for Self-Supervised Graph Representation Learning. Depen

Graph Analysis & Deep Learning Laboratory, GRAND 32 Jan 02, 2023
Unofficial implement with paper SpeakerGAN: Speaker identification with conditional generative adversarial network

Introduction This repository is about paper SpeakerGAN , and is unofficially implemented by Mingming Huang ( 7 Jan 03, 2023

Equivariant CNNs for the sphere and SO(3) implemented in PyTorch

Equivariant CNNs for the sphere and SO(3) implemented in PyTorch

Jonas Köhler 893 Dec 28, 2022
Pytorch implementation of the AAAI 2022 paper "Cross-Domain Empirical Risk Minimization for Unbiased Long-tailed Classification"

[AAAI22] Cross-Domain Empirical Risk Minimization for Unbiased Long-tailed Classification We point out the overlooked unbiasedness in long-tailed clas

PatatiPatata 28 Oct 18, 2022
Music source separation is a task to separate audio recordings into individual sources

Music Source Separation Music source separation is a task to separate audio recordings into individual sources. This repository is an PyTorch implmeme

Bytedance Inc. 958 Jan 03, 2023
Stochastic Downsampling for Cost-Adjustable Inference and Improved Regularization in Convolutional Networks

Stochastic Downsampling for Cost-Adjustable Inference and Improved Regularization in Convolutional Networks (SDPoint) This repository contains the cod

Jason Kuen 17 Jul 04, 2022
MHFormer: Multi-Hypothesis Transformer for 3D Human Pose Estimation

MHFormer: Multi-Hypothesis Transformer for 3D Human Pose Estimation This repo is the official implementation of "MHFormer: Multi-Hypothesis Transforme

Vegetabird 281 Jan 07, 2023
Acoustic mosquito detection code with Bayesian Neural Networks

HumBugDB Acoustic mosquito detection with Bayesian Neural Networks. Extract audio or features from our large-scale dataset on Zenodo. This repository

31 Nov 28, 2022
Models, datasets and tools for Facial keypoints detection

Template for Data Science Project This repo aims to give a robust starting point to any Data Science related project. It contains readymade tools setu

girafe.ai 1 Feb 11, 2022
PyTorch implementation of the WarpedGANSpace: Finding non-linear RBF paths in GAN latent space (ICCV 2021)

Authors official PyTorch implementation of the "WarpedGANSpace: Finding non-linear RBF paths in GAN latent space" [ICCV 2021].

Christos Tzelepis 100 Dec 06, 2022
Using fully convolutional networks for semantic segmentation with caffe for the cityscapes dataset

Using fully convolutional networks for semantic segmentation (Shelhamer et al.) with caffe for the cityscapes dataset How to get started Download the

Simon Guist 27 Jun 06, 2022