MUGEN Dataset
Setup
conda create --name MUGEN python=3.6
conda activate MUGEN
pip install --ignore-installed https://storage.googleapis.com/tensorflow/linux/gpu/tensorflow_gpu-1.12.0-cp36-cp36m-linux_x86_64.whl
module load cuda/9.0
module load cudnn/v7.4-cuda.10.0
git clone coinrun_MUGEN
cd coinrun_MUGEN
pip install -r requirements.txt
conda install -c conda-forge mpi4py
pip install -e .
Training Agents
Basic training commands:
python -m coinrun.train_agent --run-id myrun --save-interval 1
After each parameter update, this will save a copy of the agent to ./saved_models/. Results are logged to /tmp/tensorflow by default.
Run parallel training using MPI:
mpiexec -np 8 python -m coinrun.train_agent --run-id myrun
Train an agent on a fixed set of N levels. With N = 0, the training set is unbounded.
python -m coinrun.train_agent --run-id myrun --num-levels N
Continue training an agent from a checkpoint:
python -m coinrun.train_agent --run-id newrun --restore-id myrun
View training options:
python -m coinrun.train_agent --help
Example commands for MUGEN agents:
Base model
python -m coinrun.train_agent --run-id name_your_agent \
--architecture impala --paint-vel-info 1 --dropout 0.0 --l2-weight 0.0001 \
--num-levels 0 --use-lstm 1 --num-envs 96 --set-seed 80 \
--bump-head-penalty 0.25 -kill-monster-reward 10.0
Add squat penalty to reduce excessive squating
python -m coinrun.train_agent --run-id gamev2_fine_tune_m4_squat_penalty \
--architecture impala --paint-vel-info 1 --dropout 0.0 --l2-weight 0.0001 \
--num-levels 0 --use-lstm 1 --num-envs 96 --set-seed 811 \
--bump-head-penalty 0.1 --kill-monster-reward 5.0 --squat-penalty 0.1 \
--restore-id gamev2_fine_tune_m4_0
Larger model
python -m coinrun.train_agent --run-id gamev2_largearch_bump_head_penalty_0.05_0 \
--architecture impalalarge --paint-vel-info 1 --dropout 0.0 --l2-weight 0.0001 \
--num-levels 0 --use-lstm 1 --num-envs 96 --set-seed 51 \
--bump-head-penalty 0.05 -kill-monster-reward 10.0
Add reward for dying
python -m coinrun.train_agent --run-id gamev2_fine_tune_squat_penalty_die_reward_3.0 \
--architecture impala --paint-vel-info 1 --dropout 0.0 --l2-weight 0.0001 \
--num-levels 0 --use-lstm 1 --num-envs 96 --set-seed 857 \
--bump-head-penalty 0.1 --kill-monster-reward 5.0 --squat-penalty 0.1 \
--restore-id gamev2_fine_tune_m4_squat_penalty --die-penalty -3.0
Add jump penalty
python -m coinrun.train_agent --run-id gamev2_fine_tune_m4_jump_penalty \
--architecture impala --paint-vel-info 1 --dropout 0.0 --l2-weight 0.0001 \
--num-levels 0 --use-lstm 1 --num-envs 96 --set-seed 811 \
--bump-head-penalty 0.1 --kill-monster-reward 10.0 --jump-penalty 0.1 \
--restore-id gamev2_fine_tune_m4_0
Data Collection
Collect video data with trained agent. The following command will create a folder {save_dir}/{model_name}_seed_{seed}, which contains the audio semantic maps to reconstruct game audio, as well as the csv containing all game metadata. We use the csv for reconstructing video data in the next step.
python -m coinrun.collect_data --collect_data --paint-vel-info 1 \
--set-seed 406 --restore-id gamev2_fine_tune_squat_penalty_timeout_300 \
--save-dir \
--level-timeout 600 --num-levels-to-collect 2000
The next step is to create 3.2 second videos with audio by running the script gen_videos.sh. This script first parses the csv metadata of agent gameplay into a json format. Then, we sample 3 second clips, render to RGB, generate audio, and save .mp4s. Note that we apply some sampling logic in gen_videos.py to only generate videos for levels of sufficient length and with interesting game events. You can adjust the sampling logic to your liking here.
There are three outputs from this script:
- ./json_metadata - where full level jsons are saved for longer video rendering
- ./video_metadata - where 3.2 second video jsons are saved
- ./videos - where 3.2s .mp4 videos with audio are saved. We use these videos for human annotation.
bash gen_videos.sh
For example:
bash gen_videos.sh video_data model_gamev2_fine_tune_squat_penalty_timeout_300_seed_406
License Info
The majority of MUGEN is licensed under CC-BY-NC, however portions of the project are available under separate license terms: CoinRun, VideoGPT, VideoCLIP, and S3D are licensed under the MIT license; Tokenizer is licensed under the Apache 2.0 Pycocoevalcap is licensed under the BSD license; VGGSound is licensed under the CC-BY-4.0 license.
0 Sep 26, 2021
9 Mar 30, 2022
149 Dec 14, 2022
4 Feb 11, 2022
5 Sep 15, 2022
37 Dec 15, 2022
35 Sep 23, 2022
96 Dec 30, 2022
733 Dec 29, 2022
2.5k Dec 31, 2022
66 Oct 07, 2022
58 Oct 08, 2022
614 Dec 31, 2022
1 Jan 16, 2022
2 Apr 09, 2022
2 Jun 27, 2022
1 Feb 14, 2022
5 Oct 07, 2022
1.2k Dec 27, 2022
108 Dec 28, 2022