Generic dependency managers

May 3, 2021 — June 17, 2022

computers are awful

python

How do you install the right versions of everything for some Python code you are developing? How do you deploy that sustainably? How do you share it with others?

1 Containerization

An obvious frontrunner on popularity alone. See containerization.

2 Homebrew

See homebrew.

3 Nix

A package manager that also manages dependencies.

4 Vagrant

vagrant: Virtual machines as dependency managers.

If you are a developer, Vagrant will isolate dependencies and their configuration within a single disposable, consistent environment, without sacrificing any of the tools you are used to working with (editors, browsers, debuggers, etc). Once you or someone else creates a single Vagrantfile, you just need to vagrant up and everything is installed and configured for you to work. Other members of your team create their development environments from the same configuration, so whether you are working on Linux, Mac OS X, or Windows, all your team members are running code in the same environment, against the same dependencies, all configured the same way

I didn’t find that clear, but this example was illustrative:

$ vagrant init hashicorp/bionic64
$ vagrant up
  Bringing machine 'default' up with 'virtualbox' provider...
  ==> default: Importing base box 'hashicorp/bionic64'...
  ==> default: Forwarding ports...
  default: 22 (guest)
  => 2222 (host) (adapter 1)
  ==> default: Waiting for machine to boot...

$ vagrant ssh
  vagrant@bionic64:~$ _

5 Spack

For HPC specifically, there is spack.

Spack is a package manager for supercomputers, Linux, and macOS. It makes installing scientific software easy. With Spack, you can build a package with multiple versions, configurations, platforms, and compilers, and all of these builds can coexist on the same machine.

Spack isn’t tied to a particular language; you can build a software stack in Python or R, link to libraries written in C, C++, or Fortran, and easily swap compilers. Use Spack to install in your home directory, to manage shared installations and modules on a cluster, or to build combinatorial versions of software for testing.

Bonus feature: also lets us prototype on macOS. I do not know what particular HPC affordances it offers.

6 asdf

Have not yet used. asdf-vm/asdf: Extendable version manager with support for Ruby, Node.js, Elixir, Erlang & more.

7 direnv

direnv is an extension for your shell. It augments existing shells with a new feature that can load and unload environment variables depending on the current directory.

Use cases

Load 12factor apps environment variables

Create per-project isolated development environments

Load secrets for deployment

This does indeed create a toolkit for developing isolated environments with specific packages.

8 Build tools

Modern build tools will also manage dependencies. e.g. Google’s Bazel spends so much energy doing this when you build tensorflow that I become suspicious that it has mirrored the entire internet to my local hard drive.

9 Cheating

intoli/exodus: Painless relocation of Linux binaries–and all of their dependencies–without containers.

Exodus is a tool that makes it easy to successfully relocate Linux ELF binaries from one system to another. This is useful in situations where you don’t have root access on a machine or where a package simply isn’t available for a given Linux distribution. For example, CentOS 6.X and Amazon Linux don’t have packages for Google Chrome or aria2. Server-oriented distributions tend to have more limited and outdated packages than desktop distributions, so it’s fairly common that one might have a piece of software installed on their laptop that they can’t easily install on a remote machine.