Modern Python Setup

Python setup is very painful since it has millions of ways to do the same thing.

You can install Python via,

• install python from their official site

• Install Python via homebrew or another package manager

• install via third-party tools like pyenv

• install via conda

You can install python packages via,

• pip

• conda

• poetry

You can manage your Python environment via,

• venv

• conda

• virtualenv

It's a mess! As someone I forgot the name said on Twitter,

The greatest mind of our generation is figuring out how to set up Python environment properly.

I stumbled across Anne Lena's post and talk Python to Me podcast An Unbiased Evaluation of Python Environment and Packaging Tools, it is a very good post and I highly recommend you to read it. I have tried most of the tools she mentioned in the post for fun.

In this post, I will try to summarize what I have learned from playing with these tools and her post, and give you a lazy-man guide on what I end up using to save you some time.

5 Pillars of Python Environment Management

As said in Anne Lena's post, there are 5 pillars of Python environment management and her image is illustrated here,

The 5 pillars are,

• python version management

• python package management

• python environment management

• build a package

• publish a package

What I end up using and why

My goal is to figure out tool stacks to manage Python environment for docker-would-be-a-overkill kinda scenarios.

• use as few tools as possible

• use the tools with good support and adhere to PEP standard

I ended up using the following tool stack,

pyenv for managing python version is a no brainer, and i don't wish to use apt-get or brew to install system python and you will be confused when you have more than one python version installed on your system. Like this

$ brew list | grep python

python@3.11
python@3.12

Instead you can have,

$ pyenv versions

  system
* 3.7.10 (set by /Users/adamzhang/.pyenv/version)
  3.8.16
  3.9.1
  3.9.17
  3.10.12
  3.11.4

You could easily switch between different python version with pyenv and it is very easy to install a new python version with pyenv.

As for the rest of 4 pillars, we have some candidates to do both 4 tasks,

• poetry

• pdm

• rye

The conclusion is,

pdm is a winner for checking all the boxes and conforming to PEP standard. It is also able to detect all python versions installed by pyenv, brew and system install like this,

1. /Users/adamzhang/.pyenv/shims/python3 (3.7)
2. /Users/adamzhang/.pyenv/shims/python (3.7)
3. /opt/homebrew/bin/python3.12 (3.12)
4. /Users/adamzhang/.pyenv/shims/python3.11 (3.11)
5. /opt/homebrew/bin/python3.11 (3.11)
6. /Users/adamzhang/.pyenv/versions/3.11.4/bin/python3.11 (3.11)
7. /opt/homebrew/Caskroom/miniconda/base/bin/python3.10 (3.10)
8. /Users/adamzhang/.pyenv/shims/python3.10 (3.10)
9. /Users/adamzhang/.pyenv/versions/3.10.12/bin/python3.10 (3.10)
10. /Users/adamzhang/.pyenv/versions/3.9.17/bin/python3.9 (3.9)
11. /usr/bin/python3 (3.9)
12. /Users/adamzhang/.pyenv/versions/3.9.1/bin/python3.9 (3.9)
13. /Users/adamzhang/.pyenv/versions/3.8.16/bin/python3.8 (3.8)
14. /Users/adamzhang/.pyenv/versions/3.7.10/bin/python3.7m (3.7)
15. /Users/adamzhang/.pyenv/versions/3.7.10/bin/python3.7 (3.7)
16. /Users/adamzhang/.pyenv/shims/python3.7m (3.7)
17. /Users/adamzhang/.pyenv/shims/python3.7 (3.7)
18. /opt/homebrew/Cellar/python@3.12/3.12.0/Frameworks/Python.framework/Versions/3.12/bin/python3.12 (3.12)

Some concerns for pdm is that it is not as popular as poetry or as shiny as rye, but it is still a very good tool.

My personal take for others are that,

• poetry: good tool but their dev break back compatibility too often without any warning and break people's CI, interesting to know moreportal here to Anthony, my fav python developer.

• rye: it's very new. It even does Python version management. I will let it marinate for a while before I use it.

Summary

In this blog, I talked about my minimal-ish Python set-up tool and the reason behind it.

But remember, there is no best tool, only the tool that fits your needs and fits your team's needs. (team > you)

Reference

• anna-lena's post on python environment and management tools