ForecastingNonverbalSignals

This is the implementation for the paper Forecasting Nonverbal Social Signals during Dyadic Interactions with Generative Adversarial Neural Networks.

Dependencies

• python 3.6
• tensorFlow 1.15
• numpy
• pickle5
• sklearn
• pandas
• h5py

Usage

1. Install virtual environment named SocialActionGAN with dependencies:

conda create -n SocialActionGAN python=3.6 tensorflow=1.15 pickle5 scikit-learn pandas h5py

2. Download UDIVA_2d.pickle, and put it in the folder dataset. Training model with the default parameters:

(SocialActionGAN): python train.py

3. Alternatively, download the pre-trained model and put it in the folder model. Forecast the motions, generate the ouput file based on the format of the challenge:

(SocialActionGAN): python generate.py --annotations_dir "/path_to/talk_annotations_test_masked/" --segments_path "/path_to/test_segments_topredict.csv"

Optional

1. Extract the training data, package it as UDIVA_2d.pickle:

(SocialActionGAN): python preprocessing.py --annotations_dir "/path_to/talk_annotations_train"

License

Apache License 2.0

Citation

If you use this repository for your research, please cite:

@misc{tuyen2021forecasting,
      title={Forecasting Nonverbal Social Signals during Dyadic Interactions with Generative Adversarial Neural Networks}, 
      author={Nguyen Tan Viet Tuyen and Oya Celiktutan},
      year={2021},
      eprint={2110.09378},
      archivePrefix={arXiv},
      primaryClass={cs.AI}
}