Binaural Speech Synthesis

Last update: Dec 18, 2022

Related tags

Overview

Binaural Speech Synthesis

This repository contains code to train a mono-to-binaural neural sound renderer. If you use this code or the provided dataset, please cite our paper "Neural Synthesis of Binaural Speech from Mono Audio",

@inproceedings{richard2021binaural,
  title={Neural Synthesis of Binaural Speech from Mono Audio},
  author={Richard, Alexander and Markovic, Dejan and Gebru, Israel D and Krenn, Steven and Butler, Gladstone and de la Torre, Fernando and Sheikh, Yaser},
  booktitle={International Conference on Learning Representations},
  year={2021}
}

Code

Detailed instructions how to use the code will be release prior to ICLR 2021.

Dataset

The dataset will be released prior to ICLR 2021.

License

The code and dataset are release under CC-NC 4.0 International license.

You might also like...

Silero Models: pre-trained speech-to-text, text-to-speech models and benchmarks made embarrassingly simple

3.2k Dec 31, 2022

PyTorch implementation of Microsoft's text-to-speech system FastSpeech 2: Fast and High-Quality End-to-End Text to Speech.

An implementation of Microsoft's "FastSpeech 2: Fast and High-Quality End-to-End Text to Speech"

1k Dec 30, 2022

Simple Speech to Text, Text to Speech

Simple Speech to Text, Text to Speech 1. Download Repository Opsi 1 Download repository ini, extract di lokasi yang diinginkan Opsi 2 Jika sudah famil

5 Dec 28, 2021

A Python module made to simplify the usage of Text To Speech and Speech Recognition.

Nav Module The solution for voice related stuff in Python Nav is a Python module which simplifies voice related stuff in Python. Just import the Modul

1 Dec 20, 2021

Code for ACL 2022 main conference paper "STEMM: Self-learning with Speech-text Manifold Mixup for Speech Translation".

STEMM: Self-learning with Speech-Text Manifold Mixup for Speech Translation This is a PyTorch implementation for the ACL 2022 main conference paper ST

29 Oct 16, 2022

Code release for NeX: Real-time View Synthesis with Neural Basis Expansion

NeX: Real-time View Synthesis with Neural Basis Expansion Project Page | Video | Paper | COLAB | Shiny Dataset We present NeX, a new approach to novel

537 Jan 5, 2023

Official implementation of MLP Singer: Towards Rapid Parallel Korean Singing Voice Synthesis

MLP Singer Official implementation of MLP Singer: Towards Rapid Parallel Korean Singing Voice Synthesis. Audio samples are available on our demo page.

103 Dec 23, 2022

DiffSinger: Singing Voice Synthesis via Shallow Diffusion Mechanism (SVS & TTS); AAAI 2022

DiffSinger: Singing Voice Synthesis via Shallow Diffusion Mechanism This repository is the official PyTorch implementation of our AAAI-2022 paper, in

829 Jan 7, 2023

PhoNLP: A BERT-based multi-task learning toolkit for part-of-speech tagging, named entity recognition and dependency parsing

PhoNLP is a multi-task learning model for joint part-of-speech (POS) tagging, named entity recognition (NER) and dependency parsing. Experiments on Vietnamese benchmark datasets show that PhoNLP produces state-of-the-art results, outperforming a single-task learning approach that fine-tunes the pre-trained Vietnamese language model PhoBERT for each task independently.

109 Dec 2, 2022

Comments

UserWarning: stft will soon require the return_complex parameter be given for real inputs

Hello,when I run the train.py, there is always a warning:

UserWarning: stft will soon require the return_complex parameter be given for real inputs, and will further require that return_complex=True in a future PyTorch release. (Triggered internally at /pytorch/aten/src/ATen/native/SpectralOps.cpp:639.) return _VF.stft(input, n_fft, hop_length, win_length, window, # type: ignore

then nothing else.

Could you help me solve it?

opened by yijingshihenxiule 7
about the coordinate system and the pretrained network

Hi, thanks for you great work!

Since I am a beginner in and Audio and 3D, if you don't mind, I have some questions (that might be evident for you):

You said that

Receiver positions are therefore the same at all times. The tranmitter is the in the origin of the coordinate system and, from the receiver's perspective, x points forward, y points right, and z points up. <

I took a look at the dataset, I guess that rx_positions is the positions of the receiver and tx_positions is the positions of the sound transmitter. If the origin of the coordinate system is in the transmitter, then why rx_positions are all zeros in (x,y,z) ?

My another question is about the network, will you release the pretrained model? If not, can the provided training code produce similar outstanding results?

And how the network generalizes, like for example, what if I change the mono-audio and the positions during inference? I have monoaudio and 3d positions of my own but I cannot finetune the model because I dont have ground-truth binaural audio.

Thanks for your reply and again great work!

opened by yihongXU 0
Adding Code of Conduct file

This is pull request was created automatically because we noticed your project was missing a Code of Conduct file.

Code of Conduct files facilitate respectful and constructive communities by establishing expected behaviors for project contributors.

This PR was crafted with love by Facebook's Open Source Team.
CLA Signed

opened by facebook-github-bot 0
Adding Contributing file

This is pull request was created automatically because we noticed your project was missing a Contributing file.

CONTRIBUTING files explain how a developer can contribute to the project - which you should actively encourage.

This PR was crafted with love by Facebook's Open Source Team.
CLA Signed

opened by facebook-github-bot 0

Releases(video_v1.0)

video_v1.0(Jun 21, 2021)

This release contains silent top-view videos of each test sequence. You can overlay these videos with binaural audio generated by your model to generate top-view visualizations similar to those used in our supplemental video.
Source code(tar.gz)
Source code(zip)
subject1.mp4(1.93 MB)
subject2.mp4(1.81 MB)
subject3.mp4(1.96 MB)
subject4.mp4(1.93 MB)
subject5.mp4(2.01 MB)
subject6.mp4(1.87 MB)
subject7.mp4(1.90 MB)
subject8.mp4(1.72 MB)
validation.mp4(1.45 MB)
v1.1(Jun 21, 2021)
This release contains two pre-trained binaural networks:

a small model with a single WaveNet block for faster experiments;

a large model with three WaveNet blocks as in the ICLR paper.

Source code(tar.gz)
Source code(zip)
binaural_network_1block.net(11.05 MB)
binaural_network_3blocks.net(32.96 MB)
v1.0(Apr 30, 2021)

Download the binaural dataset here.
Source code(tar.gz)
Source code(zip)
binaural_dataset.zip(1235.99 MB)

Owner

Facebook Research

GitHub Repository

A Facebook Messenger Chatbot using NLP

A Facebook Messenger Chatbot using NLP This project is about creating a messenger chatbot using basic NLP techniques and models like Logistic Regressi

6 Nov 20, 2022

Mkdocs + material + cool stuff

Modern-Python-Doc-Example mkdocs + material + cool stuff Doc is live here Features out of the box amazing good looking website thanks to mkdocs.org an

61 Oct 26, 2022

ZUNIT - Toward Zero-Shot Unsupervised Image-to-Image Translation

ZUNIT Dependencies you can install all the dependencies by pip install -r requirements.txt Datasets Download CUB dataset. Unzip the birds.zip at ./da

9 Jun 24, 2022

A modular framework for vision & language multimodal research from Facebook AI Research (FAIR)

MMF is a modular framework for vision and language multimodal research from Facebook AI Research. MMF contains reference implementations of state-of-t

5.1k Dec 26, 2022

Python package to easily retrain OpenAI's GPT-2 text-generating model on new texts

gpt-2-simple A simple Python package that wraps existing model fine-tuning and generation scripts for OpenAI's GPT-2 text generation model (specifical

3.1k Jan 07, 2023

Connectionist Temporal Classification (CTC) decoding algorithms: best path, beam search, lexicon search, prefix search, and token passing. Implemented in Python.

CTC Decoding Algorithms Update 2021: installable Python package Python implementation of some common Connectionist Temporal Classification (CTC) decod

736 Jan 03, 2023

Binaural Speech Synthesis

Related tags

Overview

Binaural Speech Synthesis

Code

Dataset

License

You might also like...

Silero Models: pre-trained speech-to-text, text-to-speech models and benchmarks made embarrassingly simple

PyTorch implementation of Microsoft's text-to-speech system FastSpeech 2: Fast and High-Quality End-to-End Text to Speech.

Simple Speech to Text, Text to Speech

A Python module made to simplify the usage of Text To Speech and Speech Recognition.

Code for ACL 2022 main conference paper "STEMM: Self-learning with Speech-text Manifold Mixup for Speech Translation".

Code release for NeX: Real-time View Synthesis with Neural Basis Expansion

Official implementation of MLP Singer: Towards Rapid Parallel Korean Singing Voice Synthesis

DiffSinger: Singing Voice Synthesis via Shallow Diffusion Mechanism (SVS & TTS); AAAI 2022

PhoNLP: A BERT-based multi-task learning toolkit for part-of-speech tagging, named entity recognition and dependency parsing

Comments

UserWarning: stft will soon require the return_complex parameter be given for real inputs

about the coordinate system and the pretrained network

Adding Code of Conduct file

Adding Contributing file

Releases(video_v1.0)

video_v1.0(Jun 21, 2021)

v1.1(Jun 21, 2021)

v1.0(Apr 30, 2021)

Owner

Facebook Research

A Facebook Messenger Chatbot using NLP

Mkdocs + material + cool stuff

ZUNIT - Toward Zero-Shot Unsupervised Image-to-Image Translation

A modular framework for vision & language multimodal research from Facebook AI Research (FAIR)

Python package to easily retrain OpenAI's GPT-2 text-generating model on new texts

Connectionist Temporal Classification (CTC) decoding algorithms: best path, beam search, lexicon search, prefix search, and token passing. Implemented in Python.

Py65 65816 - Add support for the 65C816 to py65

Code for "Parallel Instance Query Network for Named Entity Recognition", accepted at ACL 2022.

PyTorch Implementation of the paper Single Image Texture Translation for Data Augmentation

A Multi-modal Model Chinese Spell Checker Released on ACL2021.

Signature remover is a NLP based solution which removes email signatures from the rest of the text.

Persian-lexicon - A lexicon of 70K unique Persian (Farsi) words

This is the source code of RPG (Reward-Randomized Policy Gradient)

LSTM based Sentiment Classification using Tensorflow - Amazon Reviews Rating

Speech to text streamlit app

Voice Assistant inspired by Google Assistant, Cortana, Alexa, Siri, ...

Nystromformer: A Nystrom-based Algorithm for Approximating Self-Attention

nlp基础任务

Script to download some free japanese lessons in portuguse from NHK

Modified GPT using average pooling to reduce the softmax attention memory constraints.