Poetry, the python package manager

A python package manager that I briefly used

April 18, 2023 — March 2, 2025

computers are awful
python
standards

Suspiciously similar content

poetry is one of many tools for dependency management and packaging in Python that claims improvements over the baseline default pip. I was briefly using it, but I have since moved on to other tools, because it does not deliver quality of life improvements for my particular use case.

1 Initial thoughts

No! Wait! The new new new hipness is poetry. All the other previous hipnesses were not the real eternal ultimate hipness that transcends time. I know we said this every previous time a new Python packaging system came out, but this time it’s real and our love will last forever ONO.

⛔️⛔️UPDATE⛔️⛔️ I broke up with poetry

Surprise twist: it turns out this love was not actually eternal and my ardour for poetry has cooled. Poetry no longer has an edge over other similar projects in terms of function and has a problematic history of getting logjammed; see Issue #4595: Governance—or, “What do we do with all these pull requests?”.

It might be usable if your needs are modest or you are prepared to jump into the project discord, which seems to be where the poetry hobbyists organise, but since I want to use this project merely incidentally, as a tool to develop something else, hobbyist levels of engagement are not something I can participate in. poetry is not ready for prime-time, at least for my use-case.

Note also that poetry is having difficulty staying current with the (admittedly annoying) local versions, as made famous by CUDA-supporting packages. There is an example of the kind of antics that make it work below.

Poetry is a tool for dependency management and packaging in Python. It allows you to declare the libraries your project depends on, and it will manage (install/update) them for you.

From the introduction:

Packaging systems and dependency management in Python are rather convoluted and hard to understand for newcomers. Even for seasoned developers, it might be cumbersome at times to create all files needed in a Python project: setup.py, requirements.txt, setup.cfg, MANIFEST.in and the newly added Pipfile.

So I wanted a tool that would limit everything to a single configuration file to do: dependency management, packaging and publishing.

It takes inspiration from tools that exist in other languages, like composer (PHP) or cargo (Rust).

And, finally, I started poetry to bring another exhaustive dependency resolver to the Python community apart from Conda’s.

What about Pipenv?

In short: I do not like the CLI it provides, or some of the decisions made, and I think we can make a better and more intuitive one.

Low-key dissing on similarly dysfunctional competitors is an important part of Python packaging.

Lazy install is via this terrifying command line (do not run if you do not know what this does):

curl -sSL https://install.python-poetry.org | python -

Poetry is similar to pipenv, in that it (by default, but not necessarily) manages dependencies in a local venv. It has a more full-service approach than systems built on pip. For example, it has its own dependency resolver, which uses modern dependency metadata but also works with previous dependency specifications by brute force if needed It separates specified dependencies from the ones it resolves in practice, which means dependencies seem to transport much better than conda, which generally requires you to hand-maintain a special dependency file full of just the stuff you actually wanted. In practice, its many small conveniences and thoughtful workflow are helpful. For example, it sets up the current package for development by default so that imports work as similarly as possible across this local environment and when it is distributed to users.

Recommended config:

poetry config virtualenvs.create true
poetry config virtualenvs.in-project true  # local venvs are easier for my brain.

The cache gets corrupted, giving errors about hashes:

rm -r ~/Library/Caches/pypoetry/cache
rm -r ~/Library/Caches/pypoetry/artifacts
poetry shell finds the wrong venv

Yes, it does this for me sometimes too. It is not consistent, though, and seems to be a particular shell environment that causes this glitch.

Force it to use the correct venv with:

poetry shell -C $PWD/.venv

In fact, they have removed poetry shell as of 2.0.0 because it is no good.

The new way is something like:

$ eval (poetry env activate)
How do I specify Python versions with pyenv and poetry? 
poetry env use $(pyenv which python)

2 CUDA and other local versions in poetry

Figure 1

As mentioned in the main python packaging page, poetry does not support installing build variants/profiles, which means I cannot install GPU software, and thus in practice it is burdensome to use for machine learning applications. There are workarounds: Instructions for installing PyTorch show a representative installation specification for PyTorch.

[tool.poetry.dependencies]
python = "^3.10"
numpy = "^1.23.2"
torch = { version = "1.12.1", source="torch"}
torchaudio = { version = "0.12.1", source="torch"}
torchvision = { version = "0.13.1", source="torch"}

[[tool.poetry.source]]
name = "torch"
url = "https://download.pytorch.org/whl/cu116"
secondary = true

Note that this produces various errors and downloads gigabytes of supporting files unnecessarily, but it eventually works. It was too burdensome for my workflow, so I switched back to pip.

There is a new way for torch 2.0 and later:

poetry source add --priority=supplemental torch https://download.pytorch.org/whl/cu118
poetry add torch==2.0.1+cu118 --source torch

or:

poetry add "https://download.pytorch.org/whl/cu118/torch-2.0.0%2Bcu118-cp310-cp310-linux_x86_64.whl"

I have not tried it.

poetry and PyTorch notionally play nice in PyTorch 2.1, in the sense that PyTorch 2.1 is supposed to be installable with poetry, with CUDA. It is not yet clear to me how we would set up PyTorch so it works either with or without CUDA.

3 Jupyter kernels from my poetry env

Easy:

poetry run python -m ipykernel install --user --name pymedphys

4 Dev dependencies

Poetry does not specifically support dev dependencies as such. What they do support are generic dependency groups which might happen to be dev dependencies but, like, that’s just a label man.

[tool.poetry.group.dev]  # This part can be left out
optional = true

[tool.poetry.group.dev.dependencies]
ipdb = "~0.13.13"
ipykernel = "~6.29.4"
scalene = "~1.5.41"

Now, we install:

poetry install --with dev