📰 Paper [link]

📄Code [link]

📄Dataset [link]

PyTorch implementation of our SLT2022 paper LIMUSE: LIGHTWEIGHT MULTI-MODAL SPEAKER EXTRACTION.

• In this paper, we propose a lightweight multi-modal speaker extraction framework, which incorporates multi-channel information, target speaker's visual feature and voiceprint as reference information and further apply Group Communication, Context Codec and an ultra-low bit quantization technology to reduce the model size and complexity while maintaining relatively high performance.

• What's more, we release source code and the dataset including extracted features used in our experiments to help you get started on our project quickly. Feel free to contact us if you have any questions or suggestions.

Our proposed model is a multi-stream architecture that takes multi-channel audio mixture, target speaker’s enrolled utterance and visual sequences of detected faces as inputs, and outputs the target speaker’s mask in the time domain. The encoded audio representations of the mixture are then multiplied by the generated mask to obtain the target audio. Please see the figure below for detailed model structure.

We evaluate our system on two-speaker speech separation and speaker extraction problems using GRID dataset. The pretrained face embedding extraction network is trained on LRW dataset and MS-Celeb-1M dataset. And we use SMS-WSJ toolkit to obtain simulated anechoic dual-channel audio mixture. We place 2 microphones at the center of the room. The distance between microphones is 7 cm.

• PyTorch version >= 1.6.0
• Python version >= 3.6

We have uploaded the dataset and extracted visual feature and speaker embeddings from video sequences and reference audios ahead of time so that you can directly download the dataset we released here and go on to the next step.

The directories are arranged like this:

data
├── lip_fea
|	├── test
|	├── train
|	├── valid
├── mixture
|	├── test
|	├── train
|	├── valid
├── ref
|	├── test
|	├── train
|	├── valid
├── target
|	├── test
|	├── train
|	├── valid
├── grid_vp.pkl

If you want to adjust configurations of the framework and the path of dataset, please modify the configuration file in option/train/train.yml.

Specify the path to train.yml file and run the training command:

python train.py -opt ./option/train/train.yml

This project supports full-precision and quantization training at the same time. Note that you need to modify two values of QA_flag in train.yml file if you would like to switch between full-precision and quantization stage. QA_flag in training settings stands for weight quantization while the one in net_conf stands for activation quantization.

tensorboard --logdir ./tensorboard

• Hyperparameters of LiMuSE

  Symbol	Description	Value
  N	Number of channels in audio encoder	128
  L	Length of the filters (in audio samples)	32
  P	Kernel size in convolutional blocks	3
  Ra	Number of repeats in audio block	2
  Rf	Number of repeats in fusion block	1
  S	Context size (in frames)	32
  K	Number of groups	-
  Wq	Weight Quantization bit	3
  Aq	Activation Quantization bit	8
  T0	Temperature increment per epoch	5

• Performance of LiMuSE under various configurations and comparison with baselines. K stands for number of groups. Q stands for quantization, CC stands for Context Codec, VIS stands for visual cue and VP stands for voiceprint cue. 1ch and 2ch represents single-channel and dual-channel mix speech input stream.

If you find this repo helpful, please consider citing:

@article{liu2022limuse,
  title={LIMUSE: LIGHTWEIGHT MULTI-MODAL SPEAKER EXTRACTION},
  author={Liu, Qinghua and Huang, Yating and Hao, Yunzhe and Xu, Jiaming and Xu, Bo},
  journal={IEEE SLT},
  year={2022},
  publisher={IEEE}
}