FaceBlit: Instant Real-Time Example-Based Style Transfer to Facial Videos

The official implementation of

FaceBlit: Instant Real-Time Example-Based Style Transfer to Facial Videos
A. Texler, O. Texler, M. Kučera, M. Chai, and D. Sýkora
🌐 Project Page, 📄 Paper, 📚 BibTeX

FaceBlit is a system for real-time example-based face video stylization that retains textural details of the style in a semantically meaningful manner, i.e., strokes used to depict specific features in the style are present at the appropriate locations in the target image. As compared to previous techniques, our system preserves the identity of the target subject and runs in real-time without the need for large datasets nor lengthy training phase. To achieve this, we modify the existing face stylization pipeline of Fišer et al. [2017] so that it can quickly generate a set of guiding channels that handle identity preservation of the target subject while are still compatible with a faster variant of patch-based synthesis algorithm of Sýkora et al. [2019]. Thanks to these improvements we demonstrate a first face stylization pipeline that can instantly transfer artistic style from a single portrait to the target video at interactive rates even on mobile devices.

Introduction

⚠️ DISCLAIMER: This is a research project, not a production-ready application, it may contain bugs!

This implementation is designed for two platforms - Windows and Android.

• All C++ sources are located in FaceBlit/app/src/main/cpp, except for main.cpp and main_extension.cpp which can be found in FaceBlit/VS
• All Java sources are stored in FaceBlit/app/src/main/java/texler/faceblit
• Style exemplars (.png) are located in FaceBlit/app/src/main/res/drawable
• Files holding detected landmarks (.txt) and lookup tables (.bytes) for each style are located in FaceBlit/app/src/main/res/raw
• The algorithm assumes the style image and input video/image have the same resolution

Build and Run

• Clone the repository git clone https://github.com/AnetaTexler/FaceBlit.git
• The repository contains all necessary LIB files and includes for both platforms, except for the OpenCV DLL files for Windows
• The project uses Dlib 19.21 which is added as one source file (FaceBlit/app/src/main/cpp/source.cpp) and will be compiled with other sources; so you don't have to worry about that

Windows

• The OpenCV 4.5.0 is required, you can download the pre-built version directly from here and add opencv_world450d.dll and opencv_world450.dll files from opencv-4.5.0-vc14_vc15/build/x64/vc15/bin into your PATH
• Open the solution FaceBlit/VS/FaceBlit.sln in Visual Studio (tested with VS 2019)
• Provide a facial video/image or use existing sample videos and images in FaceBlit/VS/TESTS.
  • The input video/image has to be in resolution 768x1024 pixels (width x height)
• In main() function in FaceBlit/VS/main.cpp, you can change parameters:
  • targetPath - path to input images and videos (there are some sample inputs in FaceBlit/VS/TESTS)
  • targetName - name of a target PNG image or MP4 video with extension (e.g. "target2.mp4")
  • styleName - name of a style with extension from the FaceBlit/app/src/main/res/drawable path (e.g. "style_het.png")
  • stylizeBG - true/false (true - stylize the whole image/video, does not always deliver pleasing results; false - stylize only face)
  • NNF_patchsize - voting patch size (odd number, ideal is 3 or 5); 0 for no voting
• Finally, run the code and see results in FaceBlit/VS/TESTS

Android

• OpenCV binaries (.so) are already included in the repository (FaceBlit/app/src/main/jniLibs)
• Open the FaceBlit project in Android Studio (tested with Android Studio 4.1.3 and gradle 6.5), install NDK 21.0.6 via File > Settings > Appearance & Behavior > System Settings > Android SDK > SDK Tools and build the project.
• Install the application on your mobile and face to the camera (works with both front and back). Press the right bottom button to display styles (scroll right to show more) and choose one. Wait a few seconds until the face detector loads, and enjoy the style transfer!

License

The algorithm is not patented. The code is released under the public domain - feel free to use it for research or commercial purposes.

Citing

If you find FaceBlit useful for your research or work, please use the following BibTeX entry.

@Article{Texler21-I3D,
    author    = "Aneta Texler and Ond\v{r}ej Texler and Michal Ku\v{c}era and Menglei Chai and Daniel S\'{y}kora",
    title     = "FaceBlit: Instant Real-time Example-based Style Transfer to Facial Videos",
    journal   = "Proceedings of the ACM in Computer Graphics and Interactive Techniques",
    volume    = "4",
    number    = "1",
    year      = "2021",
}