Reading list for research topics in sound event detection

Overview

Reading List for topics in Sound Event Detection

Introduction

Sound event detection aims at processing the continuous acoustic signal and converting it into symbolic descriptions of the corresponding sound events present at the auditory scene. Sound event detection can be utilized in a variety of applications, including context-based indexing and retrieval in multimedia databases, unobtrusive monitoring in health care, and surveillance. Recently (since 2017), to utilise large multimedia data available, learning acoustic information from weak annotations was formulated. This reading list consists of papers which use weak annotation for learning symbolic descriptions of the corresponding sound events in the audio.

Papers covering multiple sub-areas are listed in both the sections. If there are any areas, papers, and datasets I missed, please let me know or feel free to make a pull request.

Maintained by Soham Deshmukh

Recent Content

INTERSPEECH 2021 papers added
ICASSP 2021 papers added

Table of Contents

Research papers

Survey papers

Sound event detection and time–frequency segmentation from weakly labelled data, TASLP 2019

Areas

Learning formulation

Weakly supervised scalable audio content analysis, ICME 2016

Audio Event Detection using Weakly Labeled Data, 24th ACM Multimedia Conference 2016

An approach for self-training audio event detectors using web data, 25th EUSIPCO 2017

A joint detection-classification model for audio tagging of weakly labelled data, ICASSP 2017

Connectionist Temporal Localization for Sound Event Detection with Sequential Labeling, ICASSP 2019

Multi-Task Learning for Interpretable Weakly Labelled Sound Event Detection, ArXiv 2020

A Sequential Self Teaching Approach for Improving Generalization in Sound Event Recognition, ICML 2020

Non-Negative Matrix Factorization-Convolutional Neural Network (NMF-CNN) For Sound Event Detection, ArXiv 2020

Duration robust weakly supervised sound event detection, ICASSP 2020

SeCoST:: Sequential Co-Supervision for Large Scale Weakly Labeled Audio Event Detection, ICASSP 2020

Guided Learning for Weakly-Labeled Semi-Supervised Sound Event Detection, ICASSP 2020

Unsupervised Contrastive Learning of Sound Event Representations, ICASSP 2021

Sound Event Detection Based on Curriculum Learning Considering Learning Difficulty of Events, ICASSP 2021

Comparison of Deep Co-Training and Mean-Teacher Approaches for Semi-Supervised Audio Tagging, ICASSP 2021

Enhancing Audio Augmentation Methods with Consistency Learning, ICASSP 2021

Network Architecture

Weakly-supervised audio event detection using event-specific Gaussian filters and fully convolutional networks, ICASSP 2017

Deep CNN Framework for Audio Event Recognition using Weakly Labeled Web Data, NIPS Workshop on Machine Learning for Audio 2017

Large-Scale Weakly Supervised Audio Classification Using Gated Convolutional Neural Network, ICASSP 2018

Orthogonality-Regularized Masked NMF for Learning on Weakly Labeled Audio Data, ICASSP 2018

Sound event detection and time–frequency segmentation from weakly labelled data, TASLP 2019

Attention-based Atrous Convolutional Neural Networks: Visualisation and Understanding Perspectives of Acoustic Scenes, ICASSP 2019

Sound Event Detection of Weakly Labelled Data With CNN-Transformer and Automatic Threshold Optimization, TASLP 2020

DD-CNN: Depthwise Disout Convolutional Neural Network for Low-complexity Acoustic Scene Classification, ArXiv 2020

Effective Perturbation based Semi-Supervised Learning Method for Sound Event Detection, INTERSPEECH 2020

Weakly-Supervised Sound Event Detection with Self-Attention, ICASSP 2020

Improving Deep Learning Sound Events Classifiers using Gram Matrix Feature-wise Correlations, ICASSP 2021

An Improved Event-Independent Network for Polyphonic Sound Event Localization and Detection, ICASSP 2021

AST: Audio Spectrogram Transformer, INTERSPEECH 2021

Event Specific Attention for Polyphonic Sound Event Detection, INTERSPEECH 2021

Pooling functions

Adaptive Pooling Operators for Weakly Labeled Sound Event Detection, TASLP 2018

Comparing the Max and Noisy-Or Pooling Functions in Multiple Instance Learning for Weakly Supervised Sequence Learning Tasks, Interspeech 2018

A Comparison of Five Multiple Instance Learning Pooling Functions for Sound Event Detection with Weak Labeling, ICASSP 2019

Hierarchical Pooling Structure for Weakly Labeled Sound Event Detection, INTERSPEECH 2019

Weakly labelled audioset tagging with attention neural networks, TASLP 2019

Sound event detection and time–frequency segmentation from weakly labelled data, TASLP 2019

Multi-Task Learning for Interpretable Weakly Labelled Sound Event Detection, ArXiv 2019

A Global-Local Attention Framework for Weakly Labelled Audio Tagging, ICASSP 2021

Missing or noisy audio:

Sound event detection and time–frequency segmentation from weakly labelled data, TASLP 2019

Multi-Task Learning for Interpretable Weakly Labelled Sound Event Detection, ArXiv 2019

Improving weakly supervised sound event detection with self-supervised auxiliary tasks, INTERSPEECH 2021

Data Augmentation:

SpecAugment++: A Hidden Space Data Augmentation Method for Acoustic Scene Classification, INTERSPEECH 2021

Generative Learning

Acoustic Scene Generation with Conditional Samplernn, ICASSP 2019

Representation Learning

Towards Learning a Universal Non-Semantic Representation of Speech, INTERSPEECH 2021

Contrastive Predictive Coding of Audio with an Adversary, INTERSPEECH 2020

ACCDOA: Activity-Coupled Cartesian Direction of Arrival Representation for Sound Event Localization and Detection, ICASSP 2021

FRILL: A Non-Semantic Speech Embedding for Mobile Devices, INTERSPEECH 2021

Multi-Task Learning

Multi-Task Learning for Interpretable Weakly Labelled Sound Event Detection, ArXiv 2019

Multi-Task Learning and post processing optimisation for sound event detection, DCASE 2019

Label-efficient audio classification through multitask learning and self-supervision, ICLR 2019

Few-Shot Learning

Few-Shot Audio Classification with Attentional Graph Neural Networks, INTERSPEECH 2019

Continual Learning of New Sound Classes Using Generative Replay, WASSPA 2019

Few-Shot Sound Event Detection, ICASSP 2020

Few-Shot Continual Learning for Audio Classification, ICASSP 2021

Unsupervised and Semi-Supervised Few-Shot Acoustic Event Classification, ICASSP 2021

Knowledge Transfer

Do sound event representations generalize to other audio tasks? A case study in audio transfer learning, INTERSPEECH 2021

Transfer learning of weakly labelled audio, WASPAA 2017

Knowledge Transfer from Weakly Labeled Audio Using Convolutional Neural Network for Sound Events and Scenes, ICASSP 2018

PANNs: Large-Scale Pretrained Audio Neural Networks for Audio Pattern Recognition, TASLP 2020

Polyphonic SED

A first attempt at polyphonic sound event detection using connectionist temporal classification, ICASSP 2017

Polyphonic Sound Event Detection with Weak Labeling, Thesis 2018

Polyphonic Sound Event Detection and Localization using a Two-Stage Strategy, DCASE 2019

Evaluation of Post-Processing Algorithms for Polyphonic Sound Event Detection, WASPAA 2019

Specialized Decision Surface and Disentangled Feature for Weakly-Supervised Polyphonic Sound Event Detection, TASLP 2020

Joint task

A Joint Separation-Classification Model for Sound Event Detection of Weakly Labelled Data, ICASSP 2018

A Joint Framework for Audio Tagging and Weakly Supervised Acoustic Event Detection Using DenseNet with Global Average Pooling, INTERSPEECH 2020

Loss function

Impact of Sound Duration and Inactive Frames on Sound Event Detection Performance, ICASSP 2021

Audio and Visual

A Light-Weight Multimodal Framework for Improved Environmental Audio Tagging, ICASSP 2018

Large Scale Audiovisual Learning of Sounds with Weakly Labeled Data, IJCAI 2020

Labelling unlabelled videos from scratch with multi-modal self-supervision, NeurIPS 2020

Audio-Visual Event Recognition Through the Lens of Adversary, ICASSP 2021

Audio and Text [Audio Captioning]

Automated audio captioning with recurrent neural networks, WASPAA 2017

Audio caption: Listen and tell, ICASSP 2018

AudioCaps: Generating captions for audios in the wild, NAACL 2019

Audio Captioning Based on Combined Audio and Semantic Embeddings, ISM 2020

Clotho: An Audio Captioning Dataset, ICASSP 2020

A Transformer-based Audio Captioning Model with Keyword Estimation, INTERSPEECH 2020

Text-to-Audio Grounding: Building Correspondence Between Captions and Sound Events, ICASSP 2021

Learning Contextual Tag Embeddings for Cross-Modal Alignment of Audio and Tags, ICASSP 2021

Strongly and Weakly labelled data

Audio event and scene recognition: A unified approach using strongly and weakly labeled data, IJCNN 2017

Others

Sound Event Detection Using Point-Labeled Data, WASPAA 2019

Dataset

Task Dataset Source Num. Files
Sound Event Classification ESC-50 freesound.org 2k files
Sound Event Classification DCASE17 Task 4 YT videos 2k files
Sound Event Classification US8K freesound.org 8k files
Sound Event Classification FSD50K freesound.org 50k files
Sound Event Classification AudioSet YT videos 2M files
COVID-19 Detection using Coughs DiCOVA Volunteers recording audio via a website 1k files
Few-shot Bioacoustic Event Detection DCASE21 Task 5 audio 4k+ files
Acoustic Scene Classification DCASE18 Task 1 Recorded by TUT 1.5k
Various VGG-Sound Web videos 200k files
Audio Captioning Clotho freesound.org 5k files
Audio Captioning AudioCaps YT videos 51k files
Action Recognition UCF101 Web videos 13k files
Unlabeled YFCC100M Yahoo videos 1M files

Other audio-based datasets to consider
DCASE dataset list

Workshops/Conferences/Journals

List of old workshops (archived) and on-going workshops/conferences/journals:

Venues link
Machine Learning for Audio Signal Processing, NIPS 2017 workshop https://nips.cc/Conferences/2017/Schedule?showEvent=8790
MLSP: Machine Learning for Signal Processing https://ieeemlsp.cc/
WASPAA: IEEE Workshop on Applications of Signal Processing to Audio and Acoustics https://www.waspaa.com
ICASSP: IEEE International Conference on Acoustics Speech and Signal Processing https://2021.ieeeicassp.org/
INTERSPEECH https://www.interspeech2021.org/
IEEE/ACM Transactions on Audio, Speech and Language Processing https://dl.acm.org/journal/taslp
DCASE http://dcase.community/

Tutorials

Sound Event Detection: A Tutorial

Resources

Computational Analysis of Sound Scenes and Events

More

If you are interested in audio-captioning, K. Drossos maintains a detailed reading list here

Owner
Soham
Applied Scientist at Microsoft
Soham
🎵 A repository for manually annotating files to create labeled acoustic datasets for machine learning.

🎵 A repository for manually annotating files to create labeled acoustic datasets for machine learning.

Jim Schwoebel 28 Dec 22, 2022
convert-to-opus-cli is a Python CLI program for converting audio files to opus audio format.

convert-to-opus-cli convert-to-opus-cli is a Python CLI program for converting audio files to opus audio format. Installation Must have installed ffmp

4 Dec 21, 2022
MusicBrainz Picard

MusicBrainz Picard MusicBrainz Picard is a cross-platform (Linux/Mac OS X/Windows) application written in Python and is the official MusicBrainz tagge

MetaBrainz Foundation 3k Dec 31, 2022
Python game programming in Jupyter notebooks.

Jupylet Jupylet is a Python library for programming 2D and 3D games, graphics, music and sound synthesizers, interactively in a Jupyter notebook. It i

Nir Aides 178 Dec 09, 2022
Pyroomacoustics is a package for audio signal processing for indoor applications. It was developed as a fast prototyping platform for beamforming algorithms in indoor scenarios.

Summary Pyroomacoustics is a software package aimed at the rapid development and testing of audio array processing algorithms. The content of the pack

Audiovisual Communications Laboratory 1k Jan 09, 2023
DaisyXmusic ❤ A bot that can play music on Telegram Group and Channel Voice Chats

DaisyXmusic ❤ is the best and only Telegram VC player with playlists, Multi Playback, Channel play and more

TeamOfDaisyX 34 Oct 22, 2022
TONet: Tone-Octave Network for Singing Melody Extraction from Polyphonic Music

TONet Introduction The official implementation of "TONet: Tone-Octave Network for Singing Melody Extraction from Polyphonic Music", in ICASSP 2022 We

Knut(Ke) Chen 29 Dec 01, 2022
A lightweight yet powerful audio-to-MIDI converter with pitch bend detection

Basic Pitch is a Python library for Automatic Music Transcription (AMT), using lightweight neural network developed by Spotify's Audio Intelligence La

Spotify 1.4k Jan 01, 2023
Pyrogram bot to automate streaming music in voice chats

Pyrogram bot to automate streaming music in voice chats Help If you face an error, want to discuss this project or get support for it, join it's group

Roj 124 Oct 21, 2022
Simple, hackable offline speech to text - using the VOSK-API.

Nerd Dictation Offline Speech to Text for Desktop Linux. This is a utility that provides simple access speech to text for using in Linux without being

Campbell Barton 844 Jan 07, 2023
Implementation of "Slow-Fast Auditory Streams for Audio Recognition, ICASSP, 2021" in PyTorch

Auditory Slow-Fast This repository implements the model proposed in the paper: Evangelos Kazakos, Arsha Nagrani, Andrew Zisserman, Dima Damen, Slow-Fa

Evangelos Kazakos 57 Dec 07, 2022
Datamoshing with FFmpeg

ffmosher Datamoshing with FFmpeg Drag and drop video onto mosh.bat to create a datamoshed video. To datamosh an image, please ensure the file is in a

18 Sep 11, 2022
Converting UGG files from Rode Wireless Go II transmitters (unsompressed recordings) to WAV format

Rode_WirelessGoII_UGG2wav Converting UGG files from Rode Wireless Go II transmitters (uncompressed recordings) to WAV format Story I backuped the .ugg

Ján Mazanec 31 Dec 22, 2022
Omniscient Mozart, being able to transcribe everything in the music, including vocal, drum, chord, beat, instruments, and more.

OMNIZART Omnizart is a Python library that aims for democratizing automatic music transcription. Given polyphonic music, it is able to transcribe pitc

MCTLab 1.3k Jan 08, 2023
GNU Radio – the Free and Open Software Radio Ecosystem

GNU Radio is a free & open-source software development toolkit that provides signal processing blocks to implement software radios. It can be used wit

GNU Radio 4.1k Jan 06, 2023
Dataset and baseline code for the VocalSound dataset (ICASSP2022).

VocalSound: A Dataset for Improving Human Vocal Sounds Recognition Introduction Citing Download VocalSound Dataset Details Baseline Experiment Contact

Yuan Gong 58 Jan 03, 2023
commonfate 📦commonfate 📦 - Common Fate Model and Transform.

Common Fate Transform and Model for Python This package is a python implementation of the Common Fate Transform and Model to be used for audio source

Fabian-Robert Stöter 18 Jan 08, 2022
Audio Retrieval with Natural Language Queries: A Benchmark Study

Audio Retrieval with Natural Language Queries: A Benchmark Study Paper | Project page | Text-to-audio search demo This repository is the implementatio

21 Oct 31, 2022
Audio library for modelling loudness

Loudness Loudness is a C++ library with Python bindings for modelling perceived loudness. The library consists of processing modules which can be casc

Dominic Ward 33 Oct 02, 2022
Nayeli: cool telegram groups vc music project

Nayeli-music Nayeli 🥀 is cool telegram 🍎 groups vc music project 🎋 . Nayeli-music Nayeli Deployment 🎋 📲 Esy deploy 🐾️ Source Owner ♥️ ❄️ He is s

Kasun bandara 2 Dec 20, 2021