Zoom , GoogleMeets에서 Vtuber 데뷔하기

Overview

EasyVtuber

  • Facial landmark와 GAN을 이용한 Character Face Generation
  • Google Meets, Zoom 등에서 자신만의 웹툰, 만화 캐릭터로 대화해보세요!
  • 악세사리는 어느정도 추가해도 잘 작동해요!
  • 안타깝게도 RTX 2070 미만에서는 실시간으로 잘 작동하지 않을 수도 있어요 ㅠㅠ



Demo



Requirements

  • Python >= 3.8
  • Pytorch >= 1.7
  • pyvirtualcam
  • mediapipe
  • opencv-python



Quick Start

  • ※ 이 프로젝트는 사용 전 OBS 설치가 필수입니다
  • 아래 설치 순서를 지켜주세요!
  1. OBS studio 설치

    • OBS virtualcam을 사용하기 위해서 먼저 OBS Studio를 설치해야합니다
  2. pip install -r requirements.txt

    • OBS virtualcam을 설치되어있어야 requirements에 포함된 pyvirtualcam이 정상적으로 설치되어 사용할 수 있습니다
  3. pretrianed model download

    • 아래 파일들을 pretrained folder에 넣어주세요
      • combiner.pt
      • eyebrow_decomposer.pt
      • eyebrow_morphing_combiner.pt
      • face_morpher.pt
      • two_algo_face_rotator.pt
  4. character image를 character folder에 넣어주세요

    • character image 파일은 다음의 조건을 충족해야합니다
      • alpha 채널을 포함할 것(png 확장자)
      • 1명의 인간형 캐릭터일 것
      • 캐릭터가 정면을 볼 것
      • 캐릭터의 머리가 128 x 128 pixel 내에 들어올 것 (기본적으로 256 x 256으로 resize되기 때문에 256 x 256 기준 128x128 안에 들어와야함)

    Example image is refenced by TalkingHeadAnime2

5.python main.py --webcam_output

  • 실제 facial feature가 어떻게 잡히는지 보고 싶다면 --debug 옵션을 추가하여 실행해주세요



How to make Custom Character

  1. 네이버, 구글 등에서 본인이 원하는 캐릭터를 찾으세요!
    • 되도록이면 위의 4가지 조건을 맞춰주세요! google search

  2. 찾은 이미지에서 캐릭터 얼굴이 중앙으로 가도록 가로세로 1:1 비율로 이미지를 잘라주세요!
  3. 이미지 배경을 제거해서 alpha 채널을 만들어 주세요!
  4. 완성!
    • character folder에 이미지를 넣고 python main.py --output_webcam --character (.png_제외한_캐릭터파일_이름) 실행!



Folder Structure

      │
      ├── character/ - character images 
      ├── pretrained/ - save pretrained models 
      ├── tha2/ - Talking Head Anime2 Library source files 
      ├── facial_points.py - facial feature point constants
      ├── main.py - main script to excute
      ├── models.py - GAN models defined
      ├── pose.py - process facial landmark to pose vector
      └── utils.py - util fuctions for pre/postprocessing image



Usage

webcam으로 송출 시

  • python main.py --output_webcam

캐릭터 지정

  • python main.py --character (character folder에 있는 .png를 제외한 캐릭터 파일 이름)

facial feature 확인 시

  • python main.py --debug

동영상 파일 inference

  • python main.py --input video파일_경로 --output_dir frame_저장할_디렉토리



TODOs

  • Add eyebrow feature
  • Parameter Controller GUI
  • Automation of Making Drivable Character



Thanks to



Acknowledgements

  • EasyVtuber는 TalkingHeadAnime2를 기반으로 제작되었습니다.
  • tha2 folder의 source와 pretrained model file은 원저작자 repo의 Liscense를 확인하고 사용하시기 바랍니다.
Owner
Gunwoo Han
EE student / Deep Learning / Computer Vision / Manager of 파이썬 처음처럼
Gunwoo Han
The papers published in top-tier AI conferences in recent years.

AI-conference-papers The papers published in top-tier AI conferences in recent years. Paper table AAAI ICLR CVPR ICML ICCV ECCV NIPS 2019 ✔️ ✔️ ✔️ ✔️

Jinbae Park 6 Dec 09, 2022
A Python wrapper for the tesseract-ocr API

tesserocr A simple, Pillow-friendly, wrapper around the tesseract-ocr API for Optical Character Recognition (OCR). tesserocr integrates directly with

Fayez 1.7k Dec 31, 2022
Character Segmentation using TensorFlow

Character Segmentation Segment characters and spaces in one text line,from this paper Chinese English mixed Character Segmentation as Semantic Segment

26 Aug 25, 2022
Convolutional Recurrent Neural Network (CRNN) for image-based sequence recognition.

Convolutional Recurrent Neural Network This software implements the Convolutional Recurrent Neural Network (CRNN), a combination of CNN, RNN and CTC l

Baoguang Shi 2k Dec 31, 2022
Provides OCR (Optical Character Recognition) services through web applications

OCR4all As suggested by the name one of the main goals of OCR4all is to allow basically any given user to independently perform OCR on a wide variety

174 Dec 31, 2022
Reference Code for AAAI-20 paper "Multi-Stage Self-Supervised Learning for Graph Convolutional Networks on Graphs with Few Labels"

Reference Code for AAAI-20 paper "Multi-Stage Self-Supervised Learning for Graph Convolutional Networks on Graphs with Few Labels" Please refer to htt

Ke Sun 1 Feb 14, 2022
Course material for the Multi-agents and computer graphics course

TC2008B Course material for the Multi-agents and computer graphics course. Setup instructions Strongly recommend using a custom conda environment. Ins

16 Dec 13, 2022
A tensorflow implementation of EAST text detector

EAST: An Efficient and Accurate Scene Text Detector Introduction This is a tensorflow re-implementation of EAST: An Efficient and Accurate Scene Text

2.9k Jan 02, 2023
BD-ALL-DIGIT - This Is Bangladeshi All Sim Cloner Tools

BANGLADESHI ALL SIM CLONER TOOLS INSTALL TOOL ON TERMUX $ apt update $ apt upgra

MAHADI HASAN AFRIDI 2 Jan 19, 2022
👄 The most accurate natural language detection library for Java and the JVM, suitable for long and short text alike

Quick Info this library tries to solve language detection of very short words and phrases, even shorter than tweets makes use of both statistical and

Peter M. Stahl 532 Dec 28, 2022
Fusion 360 Add-in that creates a pair of toothed curves that can be used to split a body and create two pieces that slide and lock together.

Fusion-360-Add-In-PuzzleSpline Fusion 360 Add-in that creates a pair of toothed curves that can be used to split a body and create two pieces that sli

Michiel van Wessem 1 Nov 15, 2021
Detect text blocks and OCR poorly scanned PDFs in bulk. Python module available via pip.

doc2text doc2text extracts higher quality text by fixing common scan errors Developing text corpora can be a massive pain in the butt. Much of the tex

Joe Sutherland 1.3k Jan 04, 2023
color detection using python

colordetection color detection using python In this color detection Python project, we are going to build an application through which you can automat

Ruchith Kumar 1 Nov 04, 2021
Code for CVPR 2022 paper "SoftGroup for Instance Segmentation on 3D Point Clouds"

SoftGroup We provide code for reproducing results of the paper SoftGroup for 3D Instance Segmentation on Point Clouds (CVPR 2022) Author: Thang Vu, Ko

Thang Vu 231 Dec 27, 2022
Detect and fix skew in images containing text

Alyn Skew detection and correction in images containing text Image with skew Image after deskew Install and use via pip! Recommended way(using virtual

Kakul 230 Dec 21, 2022
An unofficial implementation of the paper "AutoVC: Zero-Shot Voice Style Transfer with Only Autoencoder Loss".

AutoVC: Zero-Shot Voice Style Transfer with Only Autoencoder Loss This is an unofficial implementation of AutoVC based on the official one. The reposi

Chien-yu Huang 27 Jun 16, 2022
Learn computer graphics by writing GPU shaders!

This repo contains a selection of projects designed to help you learn the basics of computer graphics. We'll be writing shaders to render interactive two-dimensional and three-dimensional scenes.

Eric Zhang 1.9k Jan 02, 2023
YOLOv5 in DOTA with CSL_label.(Oriented Object Detection)(Rotation Detection)(Rotated BBox)

YOLOv5_DOTA_OBB YOLOv5 in DOTA_OBB dataset with CSL_label.(Oriented Object Detection) Datasets and pretrained checkpoint Datasets : DOTA Pretrained Ch

1.1k Dec 30, 2022
Use Convolutional Recurrent Neural Network to recognize the Handwritten line text image without pre segmentation into words or characters. Use CTC loss Function to train.

Handwritten Line Text Recognition using Deep Learning with Tensorflow Description Use Convolutional Recurrent Neural Network to recognize the Handwrit

sushant097 224 Jan 07, 2023