commonfate 📦commonfate 📦 - Common Fate Model and Transform.

Related tags

Audiocommonfate
Overview

Common Fate Transform and Model for Python

This package is a python implementation of the Common Fate Transform and Model to be used for audio source separation as described in an ICASSP 2016 paper "Common Fate Model for Unison source Separation".

Common Fate Transform

cft

The Common Fate Transform is based on a signal representation that divides a complex spectrogram into a grid of patches of arbitrary size. These complex patches are then processed by a two-dimensional discrete Fourier transform, forming a tensor representation which reveals spectral and temporal modulation textures.

Common Fate Model

cfm

An adapted factorization model similar to the PARAFAC/CANDECOMP factorisation allows to decompose the common fate transform tesnor into different time-varying harmonic sources based on their particular common modulation profile: hence the name Common Fate Model.

Usage

See the full API documentation at http://aliutkus.github.io/commonfate.

Applying the Common Fate Transform

import commonfate

# # forward transform

# STFT Parameters

framelength = 1024
hopsize = 256
X = commonfate.transform.forward(signal, framelength, hopsize)

# Patch Parameters
W = (32, 48)
mhop = (16, 24)

Z = commonfate.transform.forward(X, W, mhop, real=False)

# inverse transform of cft
Y = commonfate.transform.inverse(
    Z, fdim=2, hop=mhop, shape=X.shape, real=False
)
# back to time domain
y = commonfate.transform.inverse(
    Y, fdim=1, hop=hopsize, shape=x.shape
)

Fitting the Common Fate Model

import commonfate

# initialiase and fit the common fate model
cfm = commonfate.model.CFM(z, nb_components=10, nb_iter=100).fit()

# get the fitted factors
(A, H, C) = cfm.factors

# returns the of z approximation using the fitted factors
z_hat = cfm.approx()

Decompose an audio signal using CFT and CFM

commonfate has a built-in wrapper which computes the Common Fate Transform, fits the model according to the Common Fate Model and return the synthesised time domain signal components obtained through wiener / soft mask filtering.

The following example requires to install pysoundfile.

import commonfate
import soundfile as sf

# loading signal
(audio, fs) = sf.read(filename, always_2d=True)

# decomposes the audio signal into
# (nb_components, nb_samples, nb_channels)
components = decompose.process(
    audio,
    nb_iter=100,
    nb_components=10,
    n_fft=1024,
    n_hop=256,
    cft_patch=(32, 48),
    cft_hop=(16, 24)
)

# write out the third component to wave file
sf.write(
    "comp_3.wav",
    components[2, ...],
    fs
)

Optimisations

The current common fate model implementation makes heavily use of the Einstein Notation. We use the numpy einsum module which can be slow on large tensors. To speed up the computation time we recommend to install Daniel Smith's opt_einsum package.

Installation via pip
pip install -e 'git+https://github.com/dgasmith/opt_einsum.git#egg=opt_einsum'

commonfate automatically detects if the package is installed.

References

You can download and read the paper here. If you use this package, please reference to the following publication:

@inproceedings{stoeter2016cfm,
  TITLE = {{Common Fate Model for Unison source Separation}},
  AUTHOR = {St{\"o}ter, Fabian-Robert and Liutkus, Antoine and Badeau, Roland and Edler, Bernd and Magron, Paul},
  BOOKTITLE = {{41st International Conference on Acoustics, Speech and Signal Processing (ICASSP)}},
  ADDRESS = {Shanghai, China},
  PUBLISHER = {{IEEE}},
  SERIES = {Proceedings of the 41st International Conference on Acoustics, Speech and Signal Processing (ICASSP)},
  YEAR = {2016},
  KEYWORDS = {Non-Negative tensor factorization ; Sound source separation ; Common Fate Model},
}
You might also like...
C++ library for audio and music analysis, description and synthesis, including Python bindings

Essentia Essentia is an open-source C++ library for audio analysis and audio-based music information retrieval released under the Affero GPL license.

An app made in Python using the PyTube and Tkinter libraries to download videos and MP3 audio.

yt-dl (GUI Edition) An app made in Python using the PyTube and Tkinter libraries to download videos and MP3 audio. How do I download this? Windows: Fi

Small Python application that links a Digico console and Reaper, handling automatic marker insertion and tracking.
Small Python application that links a Digico console and Reaper, handling automatic marker insertion and tracking.

Digico-Reaper-Link This is a small GUI based helper application designed to help with using Digico's Copy Audio function with a Reaper DAW used for re

Anki vector Music ❤ is the best and only Telegram VC player with playlists, Multi Playback, Channel play and more
Anki vector Music ❤ is the best and only Telegram VC player with playlists, Multi Playback, Channel play and more

Anki Vector Music 🎵 A bot that can play music on Telegram Group and Channel Voice Chats Available on telegram as @Anki Vector Music Features 🔥 Thumb

Just-Music - Spotify API Driven Music Web app, that allows to listen and control and share songs

Just Music... Just Music Is A Web APP That Allows Users To Play Song Using Spoti

Audio fingerprinting and recognition in Python
Audio fingerprinting and recognition in Python

dejavu Audio fingerprinting and recognition algorithm implemented in Python, see the explanation here: How it works Dejavu can memorize audio by liste

Python library for audio and music analysis

librosa A python package for music and audio analysis. Documentation See https://librosa.org/doc/ for a complete reference manual and introductory tut

?️ Open Source Audio Matching and Mastering
?️ Open Source Audio Matching and Mastering

Matching + Mastering = ❤️ Matchering 2.0 is a novel Containerized Web Application and Python Library for audio matching and mastering. It follows a si

Python Audio Analysis Library: Feature Extraction, Classification, Segmentation and Applications
Python Audio Analysis Library: Feature Extraction, Classification, Segmentation and Applications

A Python library for audio feature extraction, classification, segmentation and applications This doc contains general info. Click here for the comple

Comments
  • Change factorisation notation so that it matches the paper

    Change factorisation notation so that it matches the paper

    https://github.com/aliutkus/commonfate/blob/master/commonfate/model.py#L121

    is different to paper where we state:

    notation

    the output should therefore be changed to (A, H, C)

    enhancement 
    opened by faroit 1
  • Raised a MemoryError after called decompose.process

    Raised a MemoryError after called decompose.process

    code: (audio, fs) = sf.read('1.wav', always_2d=True) components = commonfate.decompose.process( audio, nb_components=10, ) sf.write( "comp_3.wav", components[2, ...], fs ) raise errors: Traceback (most recent call last): File "F:/py_project/independent-project.git/music_ctl_lettin/music_feature_engineering/process_music.py", line 190, in nb_components=10, File "C:\Users\Besitzer\AppData\Local\Programs\Python\Python36-32\lib\site-packages\commonfate\decompose.py", line 63, in process n_hop, File "C:\Users\Besitzer\AppData\Local\Programs\Python\Python36-32\lib\site-packages\commonfate\transform.py", line 325, in forward stft = fftFunction(stft, frameShape, axes=range(len(frameShape))) File "C:\Users\Besitzer\AppData\Local\Programs\Python\Python36-32\lib\site-packages\numpy\fft\fftpack.py", line 1099, in rfftn a = rfft(a, s[-1], axes[-1], norm) File "C:\Users\Besitzer\AppData\Local\Programs\Python\Python36-32\lib\site-packages\numpy\fft\fftpack.py", line 372, in rfft _real_fft_cache) File "C:\Users\Besitzer\AppData\Local\Programs\Python\Python36-32\lib\site-packages\numpy\fft\fftpack.py", line 83, in _raw_fft r = work_function(a, wsave) MemoryError

    Any ideas?

    opened by okideal 1
Releases(0.1.3)
Owner
Fabian-Robert Stöter
Audio-ML researcher
Fabian-Robert Stöter
This Bot can extract audios and subtitles from video files

Send any valid video file and the bot shows you available streams in it that can be extracted!!

TroJanzHEX 56 Nov 22, 2022
A useful tool to generate chord progressions according to melody MIDIs

Auto chord generator, pure python package that generate chord progressions according to given melodies

Billy Yi 53 Dec 30, 2022
Use python MIDI to write some simple music

Use Python MIDI to write songs

小宝 1 Nov 19, 2021
Desktop music recognition application for windows

MusicRecognizer Music recognition application for windows You can choose from which of the devices the recording will be made. If you choose speakers,

Nikita Merzlyakov 28 Dec 13, 2022
Speech recognition module for Python, supporting several engines and APIs, online and offline.

SpeechRecognition Library for performing speech recognition, with support for several engines and APIs, online and offline. Speech recognition engine/

Anthony Zhang 6.7k Jan 08, 2023
nicfit 425 Jan 01, 2023
Pianote - An application that helps musicians practice piano ear training

Pianote Pianote is an application that helps musicians practice piano ear traini

3 Aug 17, 2022
An app made in Python using the PyTube and Tkinter libraries to download videos and MP3 audio.

yt-dl (GUI Edition) An app made in Python using the PyTube and Tkinter libraries to download videos and MP3 audio. How do I download this? Windows: Fi

1 Oct 23, 2021
[Singing Log] Let your program learn to sing!

[Singing Log] Let your program learn to sing! You must have thought this was changelog when you saw the English title, but it's not, it's chànggēlog. What it does is allow your program to print logs

黄巍 22 Sep 03, 2022
?️ Open Source Audio Matching and Mastering

Matching + Mastering = ❤️ Matchering 2.0 is a novel Containerized Web Application and Python Library for audio matching and mastering. It follows a si

Sergey Grishakov 781 Jan 05, 2023
Any-to-any voice conversion using synthetic specific-speaker speeches as intermedium features

MediumVC MediumVC is an utterance-level method towards any-to-any VC. Before that, we propose SingleVC to perform A2O tasks(Xi → Ŷi) , Xi means utter

谷下雨 47 Dec 25, 2022
LibXtract is a simple, portable, lightweight library of audio feature extraction functions.

LibXtract LibXtract is a simple, portable, lightweight library of audio feature extraction functions. The purpose of the library is to provide a relat

Jamie Bullock 215 Nov 16, 2022
Neural building blocks for speaker diarization: speech activity detection, speaker change detection, overlapped speech detection, speaker embedding

⚠️ Checkout develop branch to see what is coming in pyannote.audio 2.0: a much smaller and cleaner codebase Python-first API (the good old pyannote-au

pyannote 2.1k Dec 31, 2022
Delta TTA(Text To Audio) SoftWare

Text-To-Audio-Windows Delta TTA(Text To Audio) SoftWare Info You Can Use It For Convert Your Text To Audio File You Just Write Your Text And Your End

Delta Inc. 2 Dec 14, 2021
This library provides common speech features for ASR including MFCCs and filterbank energies.

python_speech_features This library provides common speech features for ASR including MFCCs and filterbank energies. If you are not sure what MFCCs ar

James Lyons 2.2k Jan 04, 2023
Just-Music - Spotify API Driven Music Web app, that allows to listen and control and share songs

Just Music... Just Music Is A Web APP That Allows Users To Play Song Using Spoti

Ayush Mishra 3 May 01, 2022
PianoPlayer - Automatic fingering generator for piano scores

PianoPlayer - Automatic fingering generator for piano scores

Marco Musy 571 Jan 02, 2023
A Python library and tools AUCTUS A6 based radios.

A Python library and tools AUCTUS A6 based radios.

Jonathan Hart 6 Nov 23, 2022
Jarvis From Basic to Advance - make a voice assistant similar to JARVIS (in iron man movie)

JARVIS (Basic to Advance) This was my attempt to make a voice assistant similar to JARVIS (in iron man movie) Let's be honest, it's not as intelligent

codesempai 17 Dec 25, 2022
Telegram Bot to play music in VoiceChat with Channel Support and autostarts Radio.

VCPlayerBot Telegram bot to stream videos in telegram voicechat for both groups and channels. Supports live streams, YouTube videos and telegram media

Abdisamad Omar Mohamed 1 Oct 15, 2021