🎡 A repository for manually annotating files to create labeled acoustic datasets for machine learning.

Overview

sound_event_detection

A repository for manually annotating audio files to create labeled datasets for machine learning.

How to get started

I'm assuming you are running this on a Mac computer (this is the only operating system tested).

First, make sure you have installed Python3, FFmpeg, and SoX via Homebrew:

brew install python3 sox ffmpeg

Now, clone the repository and install all require dependencies:

cd ~
git clone [email protected]:jim-schwoebel/sound_event_detection.git
cd sound_event_detection
pip3 install -r requirements.txt

How to label

Just put audio files in the ./data folder, run label_files.py, and then you're ready to get started labeling! See the video below for a quick view on how this can occur (with as many files in the ./data directory that are there).

organizing data

First, put all the audio files in the ./data folder. This will allow for the script to go through all these files and set a window (usually 20 milliseconds) to label these audio files. Note that all the audio files in this folder must be uniquely named (e.g. 1.wav, 2.wav, etc.).

labeling data

Run the script with

cd ~
cd sound_event_detection
python3 label_files.py

This will then ask you for a few things - like the number of classes. Then, all the files are segmented into windows and you can annotate each file. In the example below, 19 files are created (@ 0.50 second windows for a 10 second speech file). See an example terminal session below.

how many classes do you want? (leave blank for 2) 
2
what is class 1? 
silence
what is class 2? 
speech
making fast_0.wav
making fast_1.wav
making fast_2.wav
making fast_3.wav
making fast_4.wav
making fast_5.wav
making fast_6.wav
making fast_7.wav
making fast_8.wav
making fast_9.wav
making fast_10.wav
making fast_11.wav
making fast_12.wav
making fast_13.wav
making fast_14.wav
making fast_15.wav
making fast_16.wav
making fast_17.wav
making fast_18.wav

fast_0.wav:

 File Size: 16.0k     Bit Rate: 257k
  Encoding: Signed PCM    
  Channels: 1 @ 16-bit   
Samplerate: 16000Hz      
Replaygain: off         
  Duration: 00:00:00.50  

In:100%  00:00:00.50 [00:00:00.00] Out:22.0k [      |      ]        Clip:0    
Done.
silence (0) or speech (1)?  0

After you finish annotating the file, the windowed events are then automatically sorted into the right folders (in the ./data/ directory). In this case, the 0.50 second serial snippets are in the 'speech' and 'silence' directory - all from 1 file (fast.wav). If you had multiple audio files, all the audio file windows would be sorted into these folders to easily prepare these files for machine learning.

What results is a .CSV annotation file for the entire length of the session in the ./processed/ folder along with the base audio file (e.g. 'fast.wav'). See below for the example annotation. This annotation is necessary for visualizing the file later (the 0.80 probability here can be changed in the settings.json to other values).

filename	onset	offset	event_label	probability
fast.wav	0	0.5	silence	0.8
fast.wav	0.5	1	speech	0.8
fast.wav	1	1.5	speech	0.8
fast.wav	1.5	2	speech	0.8
fast.wav	2	2.5	speech	0.8
fast.wav	2.5	3	speech	0.8
fast.wav	3	3.5	speech	0.8
fast.wav	3.5	4	speech	0.8
fast.wav	4	4.5	speech	0.8
fast.wav	4.5	5	speech	0.8
fast.wav	5	5.5	speech	0.8
fast.wav	5.5	6	speech	0.8
fast.wav	6	6.5	speech	0.8
fast.wav	6.5	7	speech	0.8
fast.wav	7	7.5	speech	0.8
fast.wav	7.5	8	speech	0.8
fast.wav	8	8.5	speech	0.8
fast.wav	8.5	9	speech	0.8
fast.wav	9	9.5	speech	0.8

changing default settings

You can change a few settings with the SETTINGS.JSON file. Note that for most speech recognition problems, a good window for humans to hear and annotate is 0.20 seconds (or 200 milliseconds), which is the default window used in this repository.

Setting (Variable) Description Possible values Default value
overlapping Determines whether or not to use overlapping windows for splicing. True or False False
model_feature models data in the timesplit variable + plots onto .CSV file output (for the visualize_feature visualization) True or False True
plot_feature Allows for the ability to plot spectrograms while labeling (8 visuals). True or False False
probability_default Sets the default probability amount (only useful if probability_labeltype == True) for each labeled session. 0.0-1.0 0.80
probability_labeltype Allows for you to automatically or manually label files with probability of events occuring. If True, the probability event metric is automatically computed with the probability_default value; if False, the probability event metric is manually annotated by the user. True or False True
timesplit The window to splice audio by for object detection. If random splicing, the audio will randomly select an interval between 0.20 and 1 seconds (allows for data augmentation). 0.20-60 or "random" 0.20
visualize_feature Allows for the ability to plot events after labeling each audio file. True or False False

Using machine learning models

training machine learning models from labels

You can train a machine learning model easily by running the train_audioTPOT.py script.

cd ~
cd sound_event_detection
python3 train_audioTPOT.py

You will then be prompted for a few things:

Is this a classification (c) or regression (r) problem? --> c
How many classes do you want to train? --> 2 
What is the name of class 1? --> silence
What is the name of class 2? --> speech

After this, all the audio files will be featurized with the librosa_featurizing embedding and modeled using TPOT, an AutoML package. Note that much of this code base is from the Voicebook repository: chapter_4_modeling. In this scenario, 25% of the data is left out for cross-validation.

A machine learning model is then trained on all the data provided in each folder in the ./data directory. Note that if you properly named the classes with label_files.py, then the classes should align (e.g. if you labeled two classes, speech and silence, you can train two classes, silence and speech).

making predictions on new files

You can then easily deploy this machine learning model on new audio files using the load_audioTPOT script.

applying pre-trained models

If instead you'd like to use some pre-trained models, you can use the ones included in the ./models directory. Here is an overview of all the current models and their accuracies.

Note many of these are overfitted on small datasets, so use these models at your own risk!! :)

Visualizing labels and predictions

We can use a third-party library called sed_vis (MIT licensed) to visualize annotated files. I've created a modification script that uses argv[] to pass through the .CSV file label and the audio file so that it works in this interface.

To visualize the files, all you need to do is place the audio file in the ./data folder (and assuming you already have a labeled file known as test.csv with an audio file test.wav - these will be generate with label.py), you can run

cd ~
cd sound_event_detection
python3 ./sed_vis/visualize.py ./processed/test.wav ./processed/test.csv

What will result will be a visualization like this with all the annotated sound events.

You can just change the command slightly to visualize all the machine learning models in the ./models directory as well. All you need to do is change the .CSV reference here (e.g. usually it's filename_2.csv):

cd ~
cd sound_event_detection
python3 ./sed_vis/visualize.py ./processed/test.wav ./processed/test_2.csv"

With this machine learning visualization, you can better hear how machine learning models are under- or over-fitted and augment datasets, as necessary, for machine learning training.

Datasets generated with script

Datasets used: [AudioSet], the [Common Voice Project], [YouTube], and [train-emotions].

Future things to do

  1. debug why accuracy is coming out as 1.2 across all models instead of some (make better experience).
  2. make sure all files are mono for the visualization library.
  3. add regression capabilities {train_audioTPOT should allow for regression modeling and outputs}.
  4. add YouTube integration for data (e.g. download YouTube video or playlist via link + auto label).
  5. clean up readme and transfer most of this info to wiki. Use landing page to generate interest to star/clone.

Other resources

If you're interested to learn more about voice computing, I highly encoursge you to check out thie Voicebook repository. This repo contains 200+ open source scripts to get started with voice computing.

Here are some other libraries that may be of interest to learn more about sound event detection:

Owner
Jim Schwoebel
Making voice computing accessible to everyone!
Jim Schwoebel
[Singing Log] Let your program learn to sing!

[Singing Log] Let your program learn to sing! You must have thought this was changelog when you saw the English title, but it's not, it's chΓ nggΔ“log. What it does is allow your program to print logs

黄巍 22 Sep 03, 2022
Basically Play Pauses the song when it is safe to do so. when you die in a round

Basically Play Pauses the song when it is safe to do so. when you die in a round

AG_1436 1 Feb 13, 2022
This is a realtime voice translator program which gets input from user at any language and converts it to the desired language that the user asks

This is a realtime voice translator program which gets input from user at any language and converts it to the desired language that the user asks ...

Mohan Ram S 1 Dec 30, 2021
Read music meta data and length of MP3, OGG, OPUS, MP4, M4A, FLAC, WMA and Wave files with python 2 or 3

tinytag tinytag is a library for reading music meta data of MP3, OGG, OPUS, MP4, M4A, FLAC, WMA and Wave files with python Install pip install tinytag

Tom Wallroth 577 Dec 26, 2022
Nayeli: cool telegram groups vc music project

Nayeli-music Nayeli πŸ₯€ is cool telegram 🍎 groups vc music project πŸŽ‹ . Nayeli-music Nayeli Deployment πŸŽ‹ πŸ“² Esy deploy 🐾️ Source Owner β™₯️ ❄️ He is s

Kasun bandara 2 Dec 20, 2021
Omniscient Mozart, being able to transcribe everything in the music, including vocal, drum, chord, beat, instruments, and more.

OMNIZART Omnizart is a Python library that aims for democratizing automatic music transcription. Given polyphonic music, it is able to transcribe pitc

MCTLab 1.3k Jan 08, 2023
Code for paper 'Audio-Driven Emotional Video Portraits'.

Audio-Driven Emotional Video Portraits [CVPR2021] Xinya Ji, Zhou Hang, Kaisiyuan Wang, Wayne Wu, Chen Change Loy, Xun Cao, Feng Xu [Project] [Paper] G

197 Dec 31, 2022
Manipulate audio with a simple and easy high level interface

Pydub Pydub lets you do stuff to audio in a way that isn't stupid. Stuff you might be looking for: Installing Pydub API Documentation Dependencies Pla

James Robert 6.6k Jan 01, 2023
Sequencer: Deep LSTM for Image Classification

Sequencer: Deep LSTM for Image Classification Created by Yuki Tatsunami Masato Taki This repository contains implementation for Sequencer. Abstract In

Yuki Tatsunami 111 Dec 16, 2022
An 8D music player made to enjoy Halloween this year!🀘

HAPPY HALLOWEEN buddy! Split Player Hello There! Welcome to SplitPlayer... Supposed To Be A 8DPlayer.... You Decide.... It can play the ordinary audio

Akshat Kumar Singh 1 Nov 04, 2021
GNOME powered sound conversion

SoundConverter A simple sound converter application for the GNOME environment. It reads anything the GStreamer library can read, and writes Ogg Vorbis

Gautier Portet 188 Dec 17, 2022
Terminal-based music player written in Python for the best music in the world 🎡 🎧 πŸ’»

audius-terminal-player Terminal-based music player written in Python for the best music in the world 🎡 🎧 πŸ’» Browse and listen to Audius from the com

Audius 21 Jul 23, 2022
Speech recognition module for Python, supporting several engines and APIs, online and offline.

SpeechRecognition Library for performing speech recognition, with support for several engines and APIs, online and offline. Speech recognition engine/

Anthony Zhang 6.7k Jan 08, 2023
Audio processor to map oracle notes in the VoG raid in Destiny 2 to call outs.

vog_oracles Audio processor to map oracle notes in the VoG raid in Destiny 2 to call outs. Huge thanks to mzucker on GitHub for the note detection cod

19 Sep 29, 2022
A telegram bot for which is help to play songs in vc πŸ₯° give 🌟 and fork this repo before use 😏

TamilVcMusic 🌟 TamilVCMusicBot 🌟 Give your πŸ’™ Before clicking on deploy to heroku just click on fork and star just below How to deploy Click the bel

TamilBots 150 Dec 13, 2022
MIDI-DDSP: Detailed Control of Musical Performance via Hierarchical Modeling

MIDI-DDSP: Detailed Control of Musical Performance via Hierarchical Modeling Demos | Blog Post | Colab Notebook | Paper | MIDI-DDSP is a hierarchical

Magenta 239 Jan 03, 2023
A voice assistant which can be used to interact with your computer and controls your pc operations

Introduction πŸ‘¨β€πŸ’» It is a voice assistant which can be used to interact with your computer and also you have been seeing it in Iron man movies, but t

Sujith 84 Dec 22, 2022
A tool for retrieving audio in the past

Rewinder A tool for retrieving audio in the past. Ever felt like, I need to remember that discussion which happened 10 min back. Now you can! Rewind a

Bharat 1 Jan 24, 2022
An audio digital processing toolbox based on a workflow/pipeline principle

AudioTK Audio ToolKit is a set of audio filters. It helps assembling workflows for specific audio processing workloads. The audio workflow is split in

Matthieu Brucher 238 Oct 18, 2022
Audio augmentations library for PyTorch for audio in the time-domain

Audio augmentations library for PyTorch for audio in the time-domain, with support for stochastic data augmentations as used often in self-supervised / contrastive learning.

Janne 166 Jan 08, 2023