๐
Depth-Aware Generative Adversarial Network for Talking Head Video Generation (CVPR 2022)
[Paper] โ [Project Page] โ [Demo] โ [Poster Video]
Fa-Ting Hong, Longhao Zhang, Li Shen, Dan Xu
The Hong Kong University of Science and Technology
Cartoon Sample
cartoon.mp4
Human Sample
celeb.mp4
Voxceleb1 Dataset
-
๐ฅ ๐ฅ โ May 19, 2022: The depth face model trained on Voxceleb2 is released! (The corresponding checkpoint of DaGAN will release soon). Click the LINK -
๐ฅ ๐ฅ โ April 25, 2022: Integrated into Huggingface Spaces๐ค using Gradio. Try out the web demo:(GPU version will come soon!)
-
๐ฅ ๐ฅ โ Add SPADE model, which produces more natural results.
๐ง
Dependencies and Installation
- Python >= 3.7 (Recommend to use Anaconda or Miniconda)
- PyTorch >= 1.7
- Option: NVIDIA GPU + CUDA
- Option: Linux
Installation
We now provide a clean version of DaGAN, which does not require customized CUDA extensions.
-
Clone repo
git clone https://github.com/harlanhong/CVPR2022-DaGAN.git cd CVPR2022-DaGAN -
Install dependent packages
pip install -r requirements.txt ## Install the Face Alignment lib cd face-alignment pip install -r requirements.txt python setup.py install
โก
Quick Inference
We take the paper version for an example. More models can be found here.
YAML configs
See config/vox-adv-256.yaml to get description of each parameter.
Pre-trained checkpoint
The pre-trained checkpoint of face depth network and our DaGAN checkpoints can be found under following link: OneDrive.
Inference! To run a demo, download checkpoint and run the following command:
CUDA_VISIBLE_DEVICES=0 python demo.py --config config/vox-adv-256.yaml --driving_video path/to/driving --source_image path/to/source --checkpoint path/to/checkpoint --relative --adapt_scale --kp_num 15 --generator DepthAwareGenerator
The result will be stored in result.mp4. The driving videos and source images should be cropped before it can be used in our method. To obtain some semi-automatic crop suggestions you can use python crop-video.py --inp some_youtube_video.mp4. It will generate commands for crops using ffmpeg.
๐ป
Training
Datasets
- VoxCeleb. Please follow the instruction from https://github.com/AliaksandrSiarohin/video-preprocessing.
Train on VoxCeleb
To train a model on specific dataset run:
CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7 python -m torch.distributed.launch --master_addr="0.0.0.0" --master_port=12348 run.py --config config/vox-adv-256.yaml --name DaGAN --rgbd --batchsize 12 --kp_num 15 --generator DepthAwareGenerator
The code will create a folder in the log directory (each run will create a new name-specific directory). Checkpoints will be saved to this folder. To check the loss values during training see log.txt. By default the batch size is tunned to run on 8 GeForce RTX 3090 gpu (You can obtain the best performance after about 150 epochs). You can change the batch size in the train_params in .yaml file.
Also, you can watch the training loss by running the following command:
tensorboard --logdir log/DaGAN/log
When you kill your process for some reasons in the middle of training, a zombie process may occur, you can kill it using our provided tool:
python kill_port.py PORT
Training on your own dataset
-
Resize all the videos to the same size e.g 256x256, the videos can be in '.gif', '.mp4' or folder with images. We recommend the later, for each video make a separate folder with all the frames in '.png' format. This format is loss-less, and it has better i/o performance.
-
Create a folder
data/dataset_namewith 2 subfolderstrainandtest, put training videos in thetrainand testing in thetest. -
Create a config
config/dataset_name.yaml, in dataset_params specify the root dir theroot_dir: data/dataset_name. Also adjust the number of epoch in train_params.
๐
Acknowledgement
Our DaGAN implementation is inspired by FOMM. We appreciate the authors of FOMM for making their codes available to public.
๐
BibTeX
@inproceedings{hong2022depth,
title={Depth-Aware Generative Adversarial Network for Talking Head Video Generation},
author={Hong, Fa-Ting and Zhang, Longhao and Shen, Li and Xu, Dan},
journal={IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},
year={2022}
}
๐ง
Contact
If you have any question, please email [email protected].
Can you please correct me where I made mistakes while running the demo one?
75 Nov 1, 2022
16 Oct 5, 2022
510 Nov 3, 2022
59 Oct 31, 2022
18 Nov 2, 2022
492 Nov 3, 2022
32 Jun 14, 2022
71 Oct 19, 2022
140 Oct 28, 2022
595 Nov 1, 2022
69 Nov 7, 2022
12 May 24, 2022
50 Nov 3, 2022
32 Oct 11, 2022
52 Oct 26, 2022
364 Oct 25, 2022
102 Oct 17, 2022
18 Oct 6, 2022
6 Sep 28, 2022