Python Can’t Get Package Management Right

The struggle is real. When a developer uses multiple package managers, there’s a risk of modules being overwritten or conflicting. Nearly 4.71% of packages on PyPI have module conflicts in their dependency graphs, leading to broken environments or even security risks.

Often, different package managers use different lock file formats, which can cause issues when switching between tools or collaborating with others using different package managers. In Python, things can get much worse when you consider dependency management.  

In the Python world there are multiple package managers including pip, Conda, Poetry, pipenv, pyenv which seem to have their own flaws. 

Why it matters? This makes it confusing for both new as well experienced developers, and eventually things feel unreasonably slow. Most users try to solve this by replicating other’s environment without giving it a second thought, and that does not work either.

But, Python is deaf for dependency resolution 

One of the primary issues in Python dependency management is handling conflicting dependencies. 

For instance, pip, the default package manager, cannot handle two different versions of the same package. This situation, often touted as “dependency hell”, causes entire installations to fail, leading to unexpected behaviour in projects.

A few months ago, one of the Reddit users mentioned that Python really feels like a programming environment from the early 80s, where a developer had a single project on their pc, and that was all they worked on for years. 

“Python wants to do everything on the global system level, including runtime versioning and packages. That means that any two developers can think they have a working project on their system, even though they have radically different setups. This makes handing off and deploying Python applications a nightmare,” he added further, suggesting why dependency resolution is a nightmare on Python. 

However, the most important and weird part of the dependency resolution is that pip makes assumptions. The pip documentation on dependency resolution explains that pip makes assumptions about package versions during installation and later checks these assumptions, which can lead to conflicts if the assumptions are incorrect.

Managing dependencies can be resource-heavy. One user reported having about 100+ GB of their hard drive filled with Python virtual dependencies, highlighting the storage impact of multiple environments.

Ergo, Virtual Environments

“I’m afraid of having 2000 folders, each one with a different virtual environment,” said one Reddit user expressing confusion about virtual environments. Running a project solely or in isolation becomes cumbersome. 

While virtual environments are essential for project isolation and dependency management, there are instances where users find virtual environments problematic rather than solving the problem. 

Previously, users have reported that package versions and dependencies can still conflict within virtual environments, requiring manual resolution in some cases that directly question the isolation in Python. 

Some developers view virtual environments as wasteful, believing they unnecessarily duplicate libraries for each project. As one Reddit user stated, “It seems like you’re installing a new copy of every library every time you start a new project, which seems like a waste of resources.”

The complexity of virtual environments can be overwhelming for those new to Python. A Reddit user expressed extreme frustration, saying, “I spend way more time just trying my computer to get my virtual environment up, project dependencies installed, and IDE configured than I do actually coding.”

Several developers recommend using Docker to avoid virtual environment issues altogether. This approach encapsulates the entire environment, making it more reproducible across different systems.