Machine listening

Statistical models for audio

October 10, 2014 — November 12, 2019

computers are awful
java
machine learning
machine listening
making things
music
python
signal processing
Figure 1

Machine listening! My preferred term for a diverse field that is also called Music Information Retrieval, speech processing, and probably other names.

I’m not going to talk about speech recognition; That boat is full.

Machine listening: machine learning, from audio. Everything from that Shazam app doohickey, to teaching computers to recognise speech, to doing artsy things with sound. I’m mostly concerned with the third one. Statistics, features, descriptors, metrics, kernels, and affinities and the spaces and topologies they induce. for musical audio e.g. your MP3 pop song library. This has considerable overlap with musical metrics but there I start from scores and transcriptions.

Polyphony and its problems.

Approximate logarithmic perception and its problems.

Should I create a separate psychoacoustics notebook? Yes.

Should I create a separate features notebook? Yes.

See also musical corpora, musical metrics, synchronisation, sparse basis dictionaries, speech recognition, learning gamelan, analysis/resynthesis, etc.

1 Features

See auditory features.

2 Implementations

Here are some options for doing machine listening.

2.1 Musicbricks

musicbricks is an umbrella project to unify (post hoc) many of the efforts mentioned individually below, plus a few other new ones.

  • Fraunhofer ML software (C++) is part of this project, including such things as

    • Real-Time Pitch Detection
    • MusicBricksTranscriber
    • Goatify Pdf
    • Time Stretch Pitch Shift Library

2.2 Librosa

See LibROSA ( lsource). I have been using it a lot recently, and I highly recommend it, especially if your pipeline already includes Python. Sleek minimal design, with a curated set of algorithms (compare and contrast with the chaos of the vamp plugins ecosystem). Python-based, but fast enough because it uses the numpy numerical libraries well. The API design meshes well with Scikit-learn, the de facto Python machine learning standard, and it’s flexible and hackable.

  • see also talkbox for a nice-looking but abandoned (?) alternative, which is nonetheless worth it for Alexander Schindler’s lovely MIR lecture based around it.

  • amen is a remix program built on librosa

2.3 Sonicannotator

SonicAnnotator seems to be about cobbling together vamp plugins for batch analysis. That is more steps than I want in an already clunky workflow in the current projects. It’s also more about RDF ontologies where I want matrices of floats.

2.4 Essentia

For C++ and Python there is Essentia, as seen in Freesound, which is a high recommendation IMO. (Watch out, the source download is enormous; just shy of half a gigabyte.) Features Python and vamp integration, and a great many algorithms. I haven’t given it a fair chance because LibROSA has been such a joy to use. However, the intriguing Dunya project is based off it.

2.5 echonest

echonest is (was?) a proprietary system that was used to generate the Million Songs Database. Seems to be gradually decaying, and was bought up by Spotify. Has great demos, such as autocanonisation.

2.6 aubio

aubio

is a tool designed for the extraction of annotations from audio signals. Its features include segmenting a sound file before each of its attacks, performing pitch detection, tapping the beat and producing midi streams from live audio.…

aubio currently provides the following features:

  • digital filters
  • phase vocoder
  • onset detection (several methods)
  • pitch tracking (several methods)
  • beat and tempo tracking
  • mel frequency cepstrum coefficients (MFCC)
  • transient / steady-state separation

…aubio is written in C and is known to run on most modern architectures and platforms.

The Python interface has been written in C so that aubio arrays can be viewed directly in Python as NumPy arrays. This makes the aubio module quite efficient, not to say fast.

2.7 MARSYAS

🏗

2.8 RP extract

🏗

2.9 phonological corpus tools

Speech-focused, phonological corpus tools is another research library for largeish corpus analysis, similarity-classification etc. As presaged, I’m not really here for speech stuff.

2.10 Metamorph, smstools

John Glover, SoundCloud staffer, has several analysis libraries culminating in Metamorph,

a new open source library for performing high-level sound transformations based on a sinusoids plus noise plus transients model. It is written in C++, can be built as both a Python extension module and a Csound opcode, and currently runs on Mac OS X and Linux.

It is designed to work primarily on monophonic, quasi-harmonic sound sources and can be used in a non-real-time context to process pre-recorded sound files or can operate in a real-time (streaming) mode.

See also the related spectral modeling and synthesis package, smstools.

2.11 Sinusoidal modelling with simplsound

“sinusoidal modelling”: Simplsound (Glover, Lazzarini, and Timoney 2009) is a Python implementation of that technique.

2.12 SCMIR

If you use a lot of Supercollider, you might like SCMIR, a native Supercollider thingy. It has the virtues that

  • it can run in real-time, which is lovely.

It has the vices that

  • It runs in Supercollider, which is a backwater language unserviced by modern development infrastructure, or decent machine learning libraries, and

  • a fraught development process; I can’t even link directly to it because the author doesn’t provide it its own anchor tag, let alone a whole web page or source code repository. Release schedule is opaque and sporadic. Consequently, it is effectively a lone guy’s pet project, rather than an active community endeavour.

    That is to say this is the Etsy sweater of MIR. If on balance this sounds like a snug deal to you, you can download SCMIR from somewhere or other on Nick Collins’ homepage.

4 References

Abe, Kobayashi, and Imai. 1995. “Harmonics Tracking and Pitch Extraction Based on Instantaneous Frequency.” In International Conference on Acoustics, Speech, and Signal Processing, 1995. ICASSP-95.
Alías, Socoró, and Sevillano. 2016. “A Review of Physical and Perceptual Feature Extraction Techniques for Speech, Music and Environmental Sounds.” Applied Sciences.
Anglade, Benetos, Mauch, et al. 2010. “Improving Music Genre Classification Using Automatically Induced Harmony Rules.” Journal of New Music Research.
Benetos, Cherla, and Weyde. 2013. “An Efficient Shift-Invariant Model for Polyphonic Music Transcription.” In 6th International Workshop on Machine Learning and Music.
Bertin-Mahieux, Ellis, Whitman, et al. 2011. “The Million Song Dataset.” In 12th International Society for Music Information Retrieval Conference (ISMIR 2011).
Bigand, Parncutt, and Lerdahl. 1996. “Perception of Musical Tension in Short Chord Sequences: The Influence of Harmonic Function, Sensory Dissonance, Horizontal Motion, and Musical Training.” Perception & Psychophysics.
Blackman, and Tukey. 1959. The measurement of power spectra from the point of view of communications engineering.
Bogert, Healy, and Tukey. 1963. “The Quefrency Alanysis of Time Series for Echoes: Cepstrum, Pseudo-Autocovariance, Cross-Cepstrum and Saphe Cracking.” In.
Boulanger-Lewandowski, Bengio, and Vincent. 2012. “Modeling Temporal Dependencies in High-Dimensional Sequences: Application to Polyphonic Music Generation and Transcription.” In 29th International Conference on Machine Learning.
Box, Jenkins, Reinsel, et al. 2016. Time Series Analysis: Forecasting and Control. Wiley Series in Probability and Statistics.
Burred, Ponsot, Goupil, et al. 2018. “CLEESE: An Open-Source Audio-Transformation Toolbox for Data-Driven Experiments in Speech and Music Cognition.” Preprint.
Carabias-Orti, Virtanen, Vera-Candeas, et al. 2011. “Musical Instrument Sound Multi-Excitation Model for Non-Negative Spectrogram Factorization.” IEEE Journal of Selected Topics in Signal Processing.
Carmi. 2013. “Compressive System Identification: Sequential Methods and Entropy Bounds.” Digital Signal Processing.
Carter. 1987. “Coherence and Time Delay Estimation.” Proceedings of the IEEE.
Chen, and Wang. n.d. “High-Level Music Descriptor Extraction Algorithm Based on Combination of Multi-Channel Cnns and Lstm.” In Proceedings of the 18th International Society for Music Information Retrieval Conference (ISMIR’2017), Suzhou, China.
Childers, Skinner, and Kemerait. 1977. “The Cepstrum: A Guide to Processing.” Proceedings of the IEEE.
Choi, Keunwoo, and Cho. 2019. “Deep Unsupervised Drum Transcription.” In.
Choi, Keunwoo, Fazekas, Sandler, et al. 2017. “Transfer Learning for Music Classification and Regression Tasks.” In Proceeding of The 18th International Society of Music Information Retrieval (ISMIR) Conference 2017.
Choi, Jeong, Lee, Park, et al. 2019. “Zero-Shot Learning for Audio-Based Music Classification and Tagging.” In.
Cochran, Cooley, Favin, et al. 1967. “What Is the Fast Fourier Transform?” Proceedings of the IEEE.
Cooley, Lewis, and Welch. 1970. “The Application of the Fast Fourier Transform Algorithm to the Estimation of Spectra and Cross-Spectra.” Journal of Sound and Vibration.
Defferrard, Benzi, Vandergheynst, et al. 2017. “FMA: A Dataset For Music Analysis.” In Proceedings of the 18th International Society for Music Information Retrieval Conference (ISMIR’2017), Suzhou, China.
Demopoulos, and Katchabaw. 2007. “Music Information Retrieval: A Survey of Issues and Approaches.”
Dieleman, and Schrauwen. 2014. “End to End Learning for Music Audio.” In 2014 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).
Dörfler, Velasco, Flexer, et al. 2010. “Sparse Regression in Time-Frequency Representations of Complex Audio.” In.
Du, Kibbe, and Lin. 2006. “Improved Peak Detection in Mass Spectrum by Incorporating Continuous Wavelet Transform-Based Pattern Matching.” Bioinformatics.
Elbaz, and Zibulevsky. 2017. “Perceptual Audio Loss Function for Deep Learning.” In Proceedings of the 18th International Society for Music Information Retrieval Conference (ISMIR’2017), Suzhou, China.
Fitzgerald. 2010. “Harmonic/Percussive Separation Using Median Filtering.”
Flamary, Févotte, Courty, et al. 2016. “Optimal Spectral Transportation with Application to Music Transcription.” In arXiv:1609.09799 [Cs, Stat].
Fonseca, Pons, Favory, et al. 2017. “Freesound Datasets: A Platform for the Creation of Open Audio Datasets.” In Proceedings of the 18th International Society for Music Information Retrieval Conference (ISMIR’2017), Suzhou, China.
Fuentes, Maia, Rocamora, et al. 2019. “Tracking Beats and Microtiming in Afro-Latin American Music Using Conditional Random Fields and Deep Learning.” In.
Fu, Lu, Ting, et al. 2011. “A Survey of Audio-Based Music Classification and Annotation.” IEEE Transactions on Multimedia.
Glover, Lazzarini, and Timoney. 2009. “Simpl: A Python Library for Sinusoidal Modelling.” In DAFx 09 Proceedings of the 12th International Conference on Digital Audio Effects, Politecnico Di Milano, Como Campus, Sept. 1-4, Como, Italy.
Godsill, and Davy. 2005. “Bayesian Computational Models for Inharmonicity in Musical Instruments.” In IEEE Workshop on Applications of Signal Processing to Audio and Acoustics, 2005.
Gómez, and Herrera. 2004. “Estimating The Tonality Of Polyphonic Audio Files: Cognitive Versus Machine Learning Modelling Strategies.” In ISMIR.
Grosche, P., Muller, and Kurth. 2010. “Cyclic Tempogram - a Mid-Level Tempo Representation for Music Signals.” In 2010 IEEE International Conference on Acoustics Speech and Signal Processing (ICASSP).
Grosche, Peter, Müller, and Sapp. 2010. “What Makes Beat Tracking Difficult? A Case Study on Chopin Mazurkas.” In Proceedings of the International Conference on Music Information Retrieval (ISMIR 2010).
Harris. 1978. “On the Use of Windows for Harmonic Analysis with the Discrete Fourier Transform.” Proceedings of the IEEE.
Helmholtz. 1863. Die Lehre von Den Tonempfindungen Als Physiologische Grundlage Für Die Theorie Der Musik.
Hermes. 1988. “Measurement of Pitch by Subharmonic Summation.” The Journal of the Acoustical Society of America.
Hershey, Chaudhuri, Ellis, et al. 2017. “CNN Architectures for Large-Scale Audio Classification.” In Proc. IEEE ICASSP 2017.
Hinton, Deng, Yu, et al. 2012. “Deep Neural Networks for Acoustic Modeling in Speech Recognition: The Shared Views of Four Research Groups.” IEEE Signal Processing Magazine.
Hoffman, Matthew, Bach, and Blei. 2010. “Online Learning for Latent Dirichlet Allocation.” In Advances in Neural Information Processing Systems.
Hoffman, Matthew D, Blei, and Cook. 2010. “Bayesian Nonparametric Matrix Factorization for Recorded Music.” In International Conference on Machine Learning.
Irizarry. 2001. “Local Harmonic Estimation in Musical Sound Signals.” Journal of the American Statistical Association.
Joël Bensoam, and David Roze. 2013. “Solving Interactions Between Nonlinear Resonators.” In Proceedings of the Sound and Music Computing Conference.
Kailath, Sayed, and Hassibi. 2000. Linear Estimation. Prentice Hall Information and System Sciences Series.
Kalouptsidis, Mileounis, Babadi, et al. 2011. “Adaptive Algorithms for Sparse System Identification.” Signal Processing.
Kereliuk, Depalle, and Pasquier. 2013. “Audio Interpolation and Morphing via Structured-Sparse Linear Regression.” In.
Lahat, Niederjohn, and Krubsack. 1987. “A Spectral Autocorrelation Method for Measurement of the Fundamental Frequency of Noise-Corrupted Speech.” IEEE Transactions on Acoustics, Speech and Signal Processing.
Lattner, Dorfler, and Arzt. 2019. “Learning Complex Basis Functions for Invariant Representations of Audio.” In Proceedings of the 20th Conference of the International Society for Music Information Retrieval.
Lattner, and Grachten. 2019. “High-Level Control of Drum Track Generation Using Learned Patterns of Rhythmic Interaction.” In IEEE Workshop on Applications of Signal Processing to Audio and Acoustics (WASPAA 2019).
Ljung. 1999. System Identification: Theory for the User. Prentice Hall Information and System Sciences Series.
Luo, Yin-Jyun, Agres, and Herremans. 2019. “Learning Disentangled Representations of Timbre and Pitch for Musical Instrument Sounds Using Gaussian Mixture Variational Autoencoders.” In Proceedings of the 20th Conference of the International Society for Music Information Retrieval.
Luo, Yi, Chen, Hershey, et al. 2016. “Deep Clustering and Conventional Networks for Music Separation: Stronger Together.” arXiv:1611.06265 [Cs, Stat].
MacKinlay, and Botev. 2019. “Mosaic Style Transfer Using Sparse Autocorrelograms.” In Proceedings of the 20th Conference of the International Society for Music Information Retrieval.
Makhoul, Kubala, Schwartz, et al. 1999. “Performance Measures For Information Extraction.” In In Proceedings of DARPA Broadcast News Workshop.
Marchand, and Peeters. 2014. “The Modulation Scale Spectrum And Its Application To Rhythm-Content Analysis.” In DAFX (Digital Audio Effects).
Maxwell, Pasquier, and Whitman. 2009. “Hierarchical Sequential Memory for Music: A Cognitive Model.” In Proceedings of the Tenth International Society for Music Information Retrieval Conference (ISMIR 2009).
McFee, and Ellis. 2011. “Analyzing Song Structure with Spectral Clustering.” In IEEE Conference on Computer Vision and Pattern Recognition (CVPR).
Mesaros, Heittola, and Virtanen. 2016. “Metrics for Polyphonic Sound Event Detection.” Applied Sciences.
Moorer. 1974. “The Optimum Comb Method of Pitch Period Analysis of Continuous Digitized Speech.” IEEE Transactions on Acoustics, Speech and Signal Processing.
Müller, Meinard, and Driedger. 2012. “Data-Driven Sound Track Generation.” In Multimodal Music Processing.
Müller, M., Ellis, Klapuri, et al. 2011. “Signal Processing for Music Analysis.” IEEE Journal of Selected Topics in Signal Processing.
Noll. 1967. “Cepstrum Pitch Determination.” The Journal of the Acoustical Society of America.
Nussbaum-Thom, Cui, Ramabhadran, et al. 2016. “Acoustic Modeling Using Bidirectional Gated Recurrent Convolutional Units.” In.
Oppenheim, and Schafer. 2004. “From Frequency to Quefrency: A History of the Cepstrum.” IEEE Signal Processing Magazine.
Parncutt. 1997. “A Model of the Perceptual Root(s) of a Chord Accounting for Voicing and Prevailing Tonality.” In Music, Gestalt, and Computing. Lecture Notes in Computer Science 1317.
Pati, Lerch, and Hadjeres. 2019. “Learning to Traverse Latent Spaces for Musical Score Inpainting.” In Proceedings of the 20th Conference of the International Society for Music Information Retrieval.
Paulus, Müller, and Klapuri. 2010. “Audio-Based Music Structure Analysis.” In ISMIR.
Phan, Hertel, Maass, et al. 2016. “Robust Audio Event Recognition with 1-Max Pooling Convolutional Neural Networks.” In Interspeech 2016.
Pickens, and Iliopoulos. 2005. “Markov Random Fields and Maximum Entropy Modeling for Music Information Retrieval.” In ISMIR.
Plomp, and Levelt. 1965. “Tonal Consonance and Critical Bandwidth.” The Journal of the Acoustical Society of America.
Pons, Lidy, and Serra. 2016. “Experimenting with Musically Motivated Convolutional Neural Networks.” In 2016 14th International Workshop on Content-Based Multimedia Indexing (CBMI).
Pons, and Serra. 2017. “Designing Efficient Architectures for Modeling Temporal Features with Convolutional Neural Networks.” In.
Robertson, Andrew N. 2011. “A Bayesian Approach to Drum Tracking.” In.
Robertson, A. N., and Plumbley. 2006. “Real-Time Interactive Musical Systems: An Overview.” Proc. Of the Digital Music Research Network, Goldsmiths University, London.
Robertson, Andrew, and Plumbley. 2007. “B-Keeper: A Beat-Tracker for Live Performance.” In Proceedings of the 7th International Conference on New Interfaces for Musical Expression. NIME ’07.
Robertson, Andrew, and Plumbley. 2013. “Synchronizing Sequencing Software to a Live Drummer.” Computer Music Journal.
Robertson, Andrew, Stark, and Davies. 2013. “Percussive Beat Tracking Using Real-Time Median Filtering.” In Proceedings of European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases.
Robertson, Andrew, Stark, and Plumbley. 2011. “Real-Time Visual Beat Tracking Using a Comb Filter Matrix.” In Proceedings of the International Computer Music Conference 2011.
Rochebois, and Charbonneau. 1997. “Cross-Synthesis Using Interverted Principal Harmonic Sub-Spaces.” In Music, Gestalt, and Computing. Lecture Notes in Computer Science 1317.
Sainath, and Li. 2016. “Modeling Time-Frequency Patterns with LSTM Vs. Convolutional Architectures for LVCSR Tasks.” Submitted to Proc. Interspeech.
Salamon, Gomez, Ellis, et al. 2014. “Melody Extraction from Polyphonic Music Signals: Approaches, Applications, and Challenges.” IEEE Signal Processing Magazine.
Salamon, Serrà, and Gómez. 2013. “Tonal Representations for Music Retrieval: From Version Identification to Query-by-Humming.” International Journal of Multimedia Information Retrieval.
Schlüter, and Böck. 2014. “Improved Musical Onset Detection with Convolutional Neural Networks.” In 2014 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).
Schmidt, and Kim. 2011. “Learning Emotion-Based Acoustic Features with Deep Belief Networks.” In 2011 IEEE Workshop on Applications of Signal Processing to Audio and Acoustics (WASPAA).
Scholler, and Purwins. 2011. “Sparse Approximations for Drum Sound Classification.” IEEE Journal of Selected Topics in Signal Processing.
Sergé, Bertaux, Rigneault, et al. 2008. “Dynamic Multiple-Target Tracing to Probe Spatiotemporal Cartography of Cell Membranes.” Nature Methods.
Serrà, Corral, Boguñá, et al. 2012. “Measuring the Evolution of Contemporary Western Popular Music.” Scientific Reports.
Smith, Evan C., and Lewicki. 2004. “Learning Efficient Auditory Codes Using Spikes Predicts Cochlear Filters.” In Advances in Neural Information Processing Systems.
Smith, Evan, and Lewicki. 2005. “Efficient Coding of Time-Relative Structure Using Spikes.” Neural Computation.
Smith, Evan C., and Lewicki. 2006. “Efficient Auditory Coding.” Nature.
Smyth, and Elmore. 2009. “Explorations in Convolutional Synthesis.” In Proceedings of the 6th Sound and Music Computing Conference, Porto, Portugal.
Southall, Wu, Lerch, et al. 2017. “MDB Drums — An Annotated Subset of MedleyDB for Automatic Drum Transcription.” In Late Breaking Demo (Extended Abstract), Proceedings of the International Society for Music Information Retrieval Conference (ISMIR).
Terhardt, Stoll, and Seewann. 1982. “Algorithm for Extraction of Pitch and Pitch Salience from Complex Tonal Signals.” The Journal of the Acoustical Society of America.
Thickstun, Harchaoui, Foster, et al. 2017. “MIREX 2017: Frequency Domain Convolutions for Multiple F0 Estimation.”
Thickstun, Harchaoui, Foster, et al. 2018. “Invariances and Data Augmentation for Supervised Music Transcription.” In 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).
Thickstun, Harchaoui, and Kakade. 2017. “Learning Features of Music from Scratch.” In Proceedings of International Conference on Learning Representations (ICLR) 2017.
Venkataramani, and Smaragdis. 2017. “End to End Source Separation with Adaptive Front-Ends.” arXiv:1705.02514 [Cs].
Welch. 1967. “The Use of Fast Fourier Transform for the Estimation of Power Spectra: A Method Based on Time Averaging over Short, Modified Periodograms.” IEEE Transactions on Audio and Electroacoustics.
Wu, and Lerch. 2017. “Automatic Drum Transcription Using the Student-Teacher Learning Paradigm with Unlabeled Music Data.” In Proceedings of the International Society for Music Information Retrieval Conference (ISMIR).
Yang, Chou, and Yang. 2017. “MidiNet: A Convolutional Generative Adversarial Network for Symbolic-Domain Music Generation.” In Proceedings of the 18th International Society for Music Information Retrieval Conference (ISMIR’2017), Suzhou, China.
Yoshii, and Goto. 2012. “Infinite Composite Autoregressive Models for Music Signal Analysis.” In.