Skip to content

Rongjiehuang/Multi-Singer

Repository files navigation

Multi-Singer: Fast Multi-Singer Singing Voice Vocoder With A Large-Scale Corpus

PyTorch Implementation of (ACM MM'21)Multi-Singer: Fast Multi-Singer Singing Voice Vocoder With A Large-Scale Corpus.

arXiv GitHub Stars MIT License

Requirements

See requirements in requirement.txt:

  • linux
  • python 3.6
  • pytorch 1.0+
  • librosa
  • json, tqdm, logging

Getting started

Apply recipe to your own dataset

  • Put any wav files in data directory
  • Edit configuration in config/config.yaml

1. Pretrain

Use our checkpoint, or
you can also train the encoder on your own here, and set the enc_model_fpath in config/config.yaml. Please set params as those in encoder/params_data and encoder/params_model.

2. Preprocess

Extract mel-spectrogram

python preprocess.py -i data/wavs -o data/feature -c config/config.yaml

-i your audio folder

-o output acoustic feature folder

-c config file

3. Train

Training conditioned on mel-spectrogram

python train.py -i data/feature -o checkpoints/ --config config/config.yaml

-i acoustic feature folder

-o directory to save checkpoints

-c config file

4. Inference

python inference.py -i data/feature -o outputs/  -c checkpoints/*.pkl -g config/config.yaml

-i acoustic feature folder

-o directory to save generated speech

-c checkpoints file

-c config file

5. Singing Voice Synthesis

For Singing Voice Synthesis:

  • Take modified FastSpeech 2 for mel-spectrogram synthesis
  • Use synthesized mel-spectrogram in Multi-Singer for waveform synthesis.

Checkpoint

Trained on OpenSinger

Acknowledgements

GE2E
FastSpeech 2
Parallel WaveGAN

Citation

@inproceedings{huang2021multi,
  title={Multi-Singer: Fast Multi-Singer Singing Voice Vocoder With A Large-Scale Corpus},
  author={Huang, Rongjie and Chen, Feiyang and Ren, Yi and Liu, Jinglin and Cui, Chenye and Zhao, Zhou},
  booktitle={Proceedings of the 29th ACM International Conference on Multimedia},
  pages={3945--3954},
  year={2021}
}

Question

Feel free to contact me at rongjiehuang@zju.edu.cn

Releases

No releases published

Packages

No packages published

Languages