Instant Real-Time Example-Based Style Transfer to Facial Videos

Related tags

Deep LearningFaceBlit
Overview

FaceBlit: Instant Real-Time Example-Based Style Transfer to Facial Videos

The official implementation of

FaceBlit: Instant Real-Time Example-Based Style Transfer to Facial Videos
A. Texler, O. Texler, M. Kučera, M. Chai, and D. Sýkora
🌐 Project Page, 📄 Paper, 📚 BibTeX

FaceBlit is a system for real-time example-based face video stylization that retains textural details of the style in a semantically meaningful manner, i.e., strokes used to depict specific features in the style are present at the appropriate locations in the target image. As compared to previous techniques, our system preserves the identity of the target subject and runs in real-time without the need for large datasets nor lengthy training phase. To achieve this, we modify the existing face stylization pipeline of Fišer et al. [2017] so that it can quickly generate a set of guiding channels that handle identity preservation of the target subject while are still compatible with a faster variant of patch-based synthesis algorithm of Sýkora et al. [2019]. Thanks to these improvements we demonstrate a first face stylization pipeline that can instantly transfer artistic style from a single portrait to the target video at interactive rates even on mobile devices.

Teaser

Introduction

⚠️ DISCLAIMER: This is a research project, not a production-ready application, it may contain bugs!

This implementation is designed for two platforms - Windows and Android.

  • All C++ sources are located in FaceBlit/app/src/main/cpp, except for main.cpp and main_extension.cpp which can be found in FaceBlit/VS
  • All Java sources are stored in FaceBlit/app/src/main/java/texler/faceblit
  • Style exemplars (.png) are located in FaceBlit/app/src/main/res/drawable
  • Files holding detected landmarks (.txt) and lookup tables (.bytes) for each style are located in FaceBlit/app/src/main/res/raw
  • The algorithm assumes the style image and input video/image have the same resolution

Build and Run

  • Clone the repository git clone https://github.com/AnetaTexler/FaceBlit.git
  • The repository contains all necessary LIB files and includes for both platforms, except for the OpenCV DLL files for Windows
  • The project uses Dlib 19.21 which is added as one source file (FaceBlit/app/src/main/cpp/source.cpp) and will be compiled with other sources; so you don't have to worry about that

Windows

  • The OpenCV 4.5.0 is required, you can download the pre-built version directly from here and add opencv_world450d.dll and opencv_world450.dll files from opencv-4.5.0-vc14_vc15/build/x64/vc15/bin into your PATH
  • Open the solution FaceBlit/VS/FaceBlit.sln in Visual Studio (tested with VS 2019)
  • Provide a facial video/image or use existing sample videos and images in FaceBlit/VS/TESTS.
    • The input video/image has to be in resolution 768x1024 pixels (width x height)
  • In main() function in FaceBlit/VS/main.cpp, you can change parameters:
    • targetPath - path to input images and videos (there are some sample inputs in FaceBlit/VS/TESTS)
    • targetName - name of a target PNG image or MP4 video with extension (e.g. "target2.mp4")
    • styleName - name of a style with extension from the FaceBlit/app/src/main/res/drawable path (e.g. "style_het.png")
    • stylizeBG - true/false (true - stylize the whole image/video, does not always deliver pleasing results; false - stylize only face)
    • NNF_patchsize - voting patch size (odd number, ideal is 3 or 5); 0 for no voting
  • Finally, run the code and see results in FaceBlit/VS/TESTS

Android

  • OpenCV binaries (.so) are already included in the repository (FaceBlit/app/src/main/jniLibs)
  • Open the FaceBlit project in Android Studio (tested with Android Studio 4.1.3 and gradle 6.5), install NDK 21.0.6 via File > Settings > Appearance & Behavior > System Settings > Android SDK > SDK Tools and build the project.
  • Install the application on your mobile and face to the camera (works with both front and back). Press the right bottom button to display styles (scroll right to show more) and choose one. Wait a few seconds until the face detector loads, and enjoy the style transfer!

License

The algorithm is not patented. The code is released under the public domain - feel free to use it for research or commercial purposes.

Citing

If you find FaceBlit useful for your research or work, please use the following BibTeX entry.

@Article{Texler21-I3D,
    author    = "Aneta Texler and Ond\v{r}ej Texler and Michal Ku\v{c}era and Menglei Chai and Daniel S\'{y}kora",
    title     = "FaceBlit: Instant Real-time Example-based Style Transfer to Facial Videos",
    journal   = "Proceedings of the ACM in Computer Graphics and Interactive Techniques",
    volume    = "4",
    number    = "1",
    year      = "2021",
}
Owner
Aneta Texler
Aneta Texler
Subgraph Based Learning of Contextual Embedding

SLiCE Self-Supervised Learning of Contextual Embeddings for Link Prediction in Heterogeneous Networks Dataset details: We use four public benchmark da

Pacific Northwest National Laboratory 27 Dec 01, 2022
This repository contains the code for EMNLP-2021 paper "Word-Level Coreference Resolution"

Word-Level Coreference Resolution This is a repository with the code to reproduce the experiments described in the paper of the same name, which was a

79 Dec 27, 2022
Lightweight Python library for adding real-time object tracking to any detector.

Norfair is a customizable lightweight Python library for real-time 2D object tracking. Using Norfair, you can add tracking capabilities to any detecto

Tryolabs 1.7k Jan 05, 2023
SegNet-like Autoencoders in TensorFlow

SegNet SegNet is a TensorFlow implementation of the segmentation network proposed by Kendall et al., with cool features like strided deconvolution, a

Andrea Azzini 66 Nov 05, 2021
Research code of ICCV 2021 paper "Mesh Graphormer"

MeshGraphormer ✨ ✨ This is our research code of Mesh Graphormer. Mesh Graphormer is a new transformer-based method for human pose and mesh reconsructi

Microsoft 251 Jan 08, 2023
[CVPR2021] De-rendering the World's Revolutionary Artefacts

De-rendering the World's Revolutionary Artefacts Project Page | Video | Paper In CVPR 2021 Shangzhe Wu1,4, Ameesh Makadia4, Jiajun Wu2, Noah Snavely4,

49 Nov 06, 2022
Implementation for ACProp ( Momentum centering and asynchronous update for adaptive gradient methdos, NeurIPS 2021)

This repository contains code to reproduce results for submission NeurIPS 2021, "Momentum Centering and Asynchronous Update for Adaptive Gradient Meth

Juntang Zhuang 15 Jun 11, 2022
a baseline to practice

ccks2021_track3_baseline a baseline to practice 路径可能会有问题,自己改改 torch==1.7.1 pyhton==3.7.1 transformers==4.7.0 cuda==11.0 this is a baseline, you can fi

45 Nov 23, 2022
Core ML tools contain supporting tools for Core ML model conversion, editing, and validation.

Core ML Tools Use coremltools to convert machine learning models from third-party libraries to the Core ML format. The Python package contains the sup

Apple 3k Jan 08, 2023
HyperSeg: Patch-wise Hypernetwork for Real-time Semantic Segmentation Official PyTorch Implementation

: We present a novel, real-time, semantic segmentation network in which the encoder both encodes and generates the parameters (weights) of the decoder. Furthermore, to allow maximal adaptivity, the w

Yuval Nirkin 182 Dec 14, 2022
COPA-SSE contains crowdsourced explanations for the Balanced COPA dataset

COPA-SSE Repository for COPA-SSE: Semi-Structured Explanations for Commonsense Reasoning. COPA-SSE contains crowdsourced explanations for the Balanced

Ana Brassard 5 Jul 31, 2022
Code for EMNLP2021 paper "Allocating Large Vocabulary Capacity for Cross-lingual Language Model Pre-training"

VoCapXLM Code for EMNLP2021 paper Allocating Large Vocabulary Capacity for Cross-lingual Language Model Pre-training Environment DockerFile: dancingso

Bo Zheng 15 Jul 28, 2022
ChatBot-Pytorch - A GPT-2 ChatBot implemented using Pytorch and Huggingface-transformers

ChatBot-Pytorch A GPT-2 ChatBot implemented using Pytorch and Huggingface-transf

ParZival 42 Dec 09, 2022
(ICCV 2021) ProHMR - Probabilistic Modeling for Human Mesh Recovery

ProHMR - Probabilistic Modeling for Human Mesh Recovery Code repository for the paper: Probabilistic Modeling for Human Mesh Recovery Nikos Kolotouros

Nikos Kolotouros 209 Dec 13, 2022
Code and description for my BSc Project, September 2021

BSc-Project Disclaimer: This repo consists of only the additional python scripts necessary to run the agent. To run the project on your own personal d

Matin Tavakoli 20 Jul 19, 2022
Dilated RNNs in pytorch

PyTorch Dilated Recurrent Neural Networks PyTorch implementation of Dilated Recurrent Neural Networks (DilatedRNN). Getting Started Installation: $ pi

Zalando Research 200 Nov 17, 2022
Repository For Programmers Seeking a platform to show their skills

Programming-Nerds Repository For Programmers Seeking Pull Requests In hacktoberfest ❓ What's Hacktoberfest 2021? Hacktoberfest is the easiest way to g

42 Oct 29, 2022
Deep Federated Learning for Autonomous Driving

FADNet: Deep Federated Learning for Autonomous Driving Abstract Autonomous driving is an active research topic in both academia and industry. However,

AIOZ AI 12 Dec 01, 2022
Complete U-net Implementation with keras

U Net Lowered with Keras Complete U-net Implementation with keras Original Paper Link : https://arxiv.org/abs/1505.04597 Special Implementations : The

Sagnik Roy 14 Oct 10, 2022
This is an implementation for the CVPR2020 paper "Learning Invariant Representation for Unsupervised Image Restoration"

Learning Invariant Representation for Unsupervised Image Restoration (CVPR 2020) Introduction This is an implementation for the paper "Learning Invari

GarField 88 Nov 07, 2022