Conda-like Python environments

A parallel python package system to pip

April 18, 2011 — March 2, 2025

computers are awful

python

standards

Suspiciously similar content

A parallel python package system to pip which is a generalisation and kinda-sorta compatible. It replaces many build difficulties with portability and intellectual-property difficulties.

Designed to handle the heavy lifting of installing Python software with hefty compiled dependencies.

There are two parts here with two separate licences:

the Anaconda Python distribution
the conda Python package manager

I am slightly confused about how these two relate. The distinction is important since licensing Anaconda proper can be expensive. See, e.g.

Anaconda is not free for commercial use (any more) so what are the alternatives?
Conda/Anaconda no longer free to use?
See also mamba or Pixi below, which aim to reduce licensing risk by reimplementing the more licensing-vulnerable parts of the Anaconda ecosystem and improve in other ways.
Can I install an entirely non-Anaconda Python distribution through the conda package manager?

Some things that are (or were?) painful to install by pip are painless via conda. Conversely, some things that are painful to install by conda are easy by pip.

I recommend figuring out which pain points are worse in this system through trial and error. If conda does not bring substantial value, choose pip. Sometimes it would be worth understanding conda’s current licensing and future licensing risks, but it would have to be a strong win.

This is an updated recommendation; previously I preferred conda — pip used to be much worse, and Anaconda’s licensing used to be less restrictive.

1 Setup

Download, e.g. Linux x64 Miniconda from the download page.

bash Miniconda3-latest-Linux-x86_64.sh
# login/logout here
# or do something like `exec bash -` if you are fancy
# Less aggressive conda
conda config --set auto_activate_base false
# conda for fish users
conda init fish

Alternatively, try miniforge: A conda-forge distribution or fastchan, fast.ai’s conda mini-distribution.

curl -L -O https://github.com/conda-forge/miniforge/releases/latest/download/Mambaforge-$(uname)-$(uname -m).sh
bash Mambaforge-$(uname)-$(uname -m).sh

It is worth installing one of these minimalist distros rather than the default Anaconda distro since Anaconda default is gigantic but does not have what I need, so it simply wastes space. Some of these might have less onerous licensing than the mainline? I am not sure.

To install something with tricky dependencies like ViTables, I do this:

conda install pytables=3.2
conda install pyqt=4

Aside: I use fish shell, so I need to do some extra setup. Specifically, I add the line

source (conda info --root)/etc/fish/conf.d/conda.fish

into ~/.config/fish/config.fish. This is now automated by

conda init fish

For Jupyter compatibility, I need:

conda install nb_conda_kernels

2 Dependencies

The main selling point of conda is that specifying dependencies for ad hoc Python scripts or packages is easy.

Conda has a slightly different dependency management and packaging workflow from the pip ecosystem. See, e.g. Tim Hopper’s explanation of this environment.yml malarkey, or the creators’ rationale and manual.

One exports the current conda environment config, by convention, into environment.yml.

conda env export > environment.yml

conda env create --file environment.yml

Which to use out of conda env create and conda create? If it involves .yaml environment configs, then conda env create. Confusing these two is a quagmire of opaque errors, poor documentation and sadness.

One point of friction I quickly encountered is that the automatically created environments are not generic. I might specify from the command line a package that I know will install sanely on any platform (e.g., matplotlib), but the version stored in the environment file is specific to where I installed it (macOS, Linux, Windows…) and architecture (x64, ARM…). For GPU software, there are even more incompatibilities because there are more choices of architecture. So to share environments with collaborators on different platforms, I need to… be them, I guess? Buy them new laptops that match my laptop? I don’t know, this seems weird. Maybe I’m missing something.

Conda will fill up my hard disk if not regularly disciplined via conda clean.

conda clean --all --yes

If I have limited space in my home directory, I might need to move the cache:

configure PKGS_DIR in ~/.condarc:

conda config --add pkgs_dirs /SOME/OTHER/PATH/.conda

Possibly also required?

chmod a-rwx ~/.conda

I might also want to avoid installing the gigantic MKL library, not being a fan. It comes baked in by default for most Anaconda installs. I can usually disable it by request:

conda create -n pynomkl python nomkl

Clearly, the packagers do not test this configuration often because it fails sometimes even for packages that notionally do not need MKL. Worth attempting, however. Between the various versions and installed copies, MKL alone used about 10GB on my Mac when I last checked. I also try to reduce the number of copies of MKL by starting from miniconda as my base Anaconda distribution, cautiously adding things as I need them.

3 Local environment

A local environment folder is more isolated, keeping packages in a local folder rather than keeping all environments somewhere global, where I need to remember what I named them all.

conda config --set env_prompt '({name})'
conda env create --prefix ./env/myenv --file environment_linux.yml
conda activate ./env/myenv

Gotcha: in fish shell the first line needs to be

conda config --set env_prompt '${name}$'

I am not sure why. As far as I know, fish command substitution does not happen inside strings. Either way, this will add the line

env_prompt: ({name})

to .condarc.

4 Mamba

Mamba is a fully compatible drop-in replacement for conda. It was started in 2019 by Wolf Vollprecht.

The introductory blog post is an enlightening read, which also explains conda better than conda does. The mamba 1.0 release announcement is also very good. See mamba-org/mamba for more. The fact that the authors of this system can articulate their ideas is a major selling point in my opinion.

It explicitly targets package installation for less mainstream configurations such as R and vscode development environments. In fact, it is not even Python-specific.

Provide a convenient way to install developer tools in VSCode workspaces from conda-forge with micromamba. Get NodeJS, Go, Rust, Python or JupyterLab installed by running a single command.

It also inherits some of the debilities of conda, e.g. that dependencies are platform- and architecture-specific.

5 Pixi

7 Reasons to Switch from Conda to Pixi | prefix.dev:

…we’re solving conda users’ pain points with pixi – a package manager that’s 10x faster than conda, integrates with PyPI package world much more deeply, eliminates unnecessary steps, allows you to use tasks for collaboration, and more.

The goal of mamba was always to be a drop-in replacement for conda. With pixi, we are going a step further with a more opinionated workflow. Pixi is written in Rust, is up to 4x faster than micromamba, and natively supports lockfiles and cross-platform tasks.

6 Magic

Magic is a Pixi derivative that targets Modular’s python derivative Mojo.

blog post: Why Magic?

7 Robocorp

Robocorp tools claim to make conda install more generic.

RCC is a command-line tool that allows you to create, manage, and distribute Python-based self-contained automation packages - or robots 🤖 as we call them.

Together with the robot.yaml configuration file, rcc provides the foundation to build and share automation with ease.

In short, the RCC toolchain helps you to get rid of the phrase: “Works on my machine” so that you can actually build and run your robots more freely.