Machine learning and statistics in Julia

1 Machine learning

Let’s put the automatic differentiation, the optimizers, and the samplers together to do differentiable learning!

The deep learning toolkits have shorter feature lists than the lengthy ones of those fancy Python/C++ libraries (e.g. mobile app building, cuDNN-backed optimisations are all less present in Julia libraries). But maybe the elegance/performance of Julia makes some of those features irrelevant? I, for one, don’t care about most of those because I’m a researcher, not a deployer.

Having said that, Tensorflow.jl gets all the features because it invokes C++ tensorflow. Surely one misses the benefit of Julia this way, since there are two different array-processing infrastructures to data between, and a different approach to JIT versus pre-compiled execution. Or no?

Flux.jl sounds like a reimplementation of Tensorflow-style differentiable programming inside Julia, which strikes me as the right way to do this to benefit from the end-to-end-optimised design philosophy of Julia.

Flux is a library for machine learning. It comes “batteries-included” with many useful tools built in, but also lets you use the full power of the Julia language where you need it. The whole stack is implemented in clean Julia code (right down to the GPU kernels) and any part can be tweaked to your liking.

It’s missing some features of e.g. Tensorflow, but includes compensatory surprising/unique feature combinations. GPU support seems to suggest it will support common CUDA optimizations (even some CuDNN ops), although I have suspicions that not every CUDA op is supported and also CUDA itself can be scanty (does CUDA do a GPU Discrete Cosine Transform yet?).

Its end-to-end Julia philosophy supports neat tricks. Favourite: DiffEqFlux — see below — which makes Neural ODEs sort-of simple to create.

I have not used it enough to know yet, but I suspect that the generic nature of Flux works against it in one sense, which is that I imagine there are not convenient distributed multi-GPU trainers at the moment. I should confirm this.

Knet.jl is another deep learning library that claims to show the ease of implementing deep learning frameworks in Julia.

Alternatively, Mocha.jl is a belt-and-braces deep learning thing, with a library of pre-defined layers deprecated and unmaintained.

If one were aiming to do that, why not do something left-field like use the dynamical systems approach to deep learning? This neat trick was popularised by Haber and Ruthotto et al, who have released some of their models as Meganet.jl. I’m curious to see how they work. (seems to have paused).

There are various Gaussian Process options.

MLJ is a scikit-learn-like pipeline for data analysis in Julia which standardises model composition and automates some of the training etc. It has various adaptors for other ML systems via MLJModels.

See also * FluxTraining.jl * FluxML/FastAI.jl: Port of FastAI V2 API to Julia

2 Statistics, probability and data analysis

Hayden Klok and Yoni Nazarathy are writing a free Julia Statistics textbook (preprint) (Nazarathy and Klok 2021) which seems a thorough introduction to statistics as well as Julia, albeit statistics in a classical frame that won’t be fashionable with either your learning theory or Bayesian types.

A good starting point for doing stuff is JuliaStats which organisation produces many statistics megapackages, for kernel density estimates, generalised linear models, loess etc. Install them all using Statskit:

using StatsKit

Less well known but handy is F. Bagge Carlson’s TotalLeastSquares which does neat errors-in-variables models Bagge Carlson, F., Machine Learning and System Identification for Estimation in Physical Systems (PhD Thesis 2018).

using RDatasets
iris = dataset("datasets", "iris")
neuro = dataset("boot", "neuro")

3 Differentiating, optimisation

4 ODEs, PDEs, SDEs

Chris Rauckackas is a veritable wizard with this stuff; read his blog.

Here is a tour of fun tricks with stochastic PDEs. There is a lot of tooling for this; DiffEqOperators … does something. DiffEqFlux (EZ neural ODEs works with Flux and claims to make Neural ODEs simple. The implementation of these things in Python, for the award-winning NeurIPS paper that made them famous was a nightmare. +1 for Julia here. The neural SDE section is mostly Julia; Go check that out.

There are many PDE options. Gridap seems fresh.

5 Configuration

See experiment tracking in ML for what I mean here.

• [ANN] Configurations.jl - Options/Configurations made easy - Package Announcements / Package announcements - JuliaLang
• Home · Configurations.jl

6 Matrix Factorisation and completion

NMF.jl contains reference implementations of non-negative matrix factorisation.

Laplacians.jl by Dan Spielman et al. is a matrix factorisation toolkit especially for Laplacian (graph adjacency) matrices.

Once again, F. Bagge Carlson’s TotalLeastSquares solves certain matrix factorisation and completion problems.

7 Signal processing

DSP.jl has been split off from core and now needs to be installed separately. Also, DirectConvolutions has sensible convolution code.

FFTs are provided by AbstractFFTs, which in principle wraps many FFT implementations. I don’t know if there is a GPU implementation yet, but there for sure is the classic CPU implementation provided by FFTW.jl which uses FFTW internally.

As for how to use these things, Numerical tours of data sciences has a Julia edition with lots of signal processing content.

JuliaAudio processes audio. They recommend PortAudio.jl as a real-time soundcard interface, which looks sorta simple. See rkat’s example of how this works. There are useful abstractions like SampledSignals to load audio and keep the data and signal rate bundled together. Although, as SampledSignal maintainer Spencer Russell points out, AxisArrays might be the right data structure for sample signals, and you could use SampledSignals purely for IO, and ignore its data structures thereafter.

Images.jl processes images.

8 QMC

Low discrepancy and other QMC stuff. Mostly I want low discrepancy sequences. There are two options with near identical interfaces; I’m not sure of the differences.

Sobol.jl claims to have been performance profiled:

] add Sobol
using Sobol
s = SobolSeq(2)
# Then
x = next!(s)

QMC.jl:

] add https://github.com/PieterjanRobbe/QMC.jl
using QMC
lat = LatSeq(2)
#then
next(lat)

9 References