Python Virtual Environments

Overview

Many of us work on multiple Python projects at the same time. Multiple projects may depend on different versions of the same library. This is a problem. Even if you work with a single project and you deploy it to production, you may run into trouble, because the system's Python on your production server might change due to OS upgrade or security patch, and your application might fail as a result. In general, you want full control over the Python environment of your projects. Enter virtual environments...

The basic idea of a virtual environment is to have a Python interpreter and its site-packages separate from the system one. Also, you can have many of them. That solves both problems. You can assign a separate virtual environment with its own dependencies for each project and forget about conflicts with other projects and the system's Python.

In this tutorial, you'll learn the concepts behind virtual environments and how to create and use them, and you'll discover various alternatives for special situations.

Virtualenv

The virtualenv package supports this concept. You can install virtualenv using pip install virtualenv.

Once virtualenv is installed, you can start creating virtual environments. Let's create two environments called "venv_1" and "venv_2".

Let's see what happened.

Inside the "bin" sub-directory, you'll find a few executables and symlinks. Those include the Python interpreter itself, pip, easy_install, and most importantly a few activate scripts.

The activate script is the key. In order to activate a specific virtual environment, you source the activate script, as in: source ~/venv_1/bin_activate. The effect is setting a bunch of environment variables and changing the prompt to the name of the activated virtual environment. It also adds a deactivate() shell function that will reset everything. Once a virtual environment is activated, typing python will launch its Python with its dependencies.

Let's deactivate:

If you have multiple Python interpreters installed on your systems, you can specify which one to use for your virtual environment using the -p option. Here is a Python 3 virtual environment:

Virtualenv works even on pypy.

You can switch directly from one environment to the other without deactivating first:

OK. Let's see how to use two different versions of the same package in two different virtual environments. This is as simple as activating each environment and installing the desired version. The environments are totally independent, so the fact that there is a different version in another environment is a non-issue.

Let's install the sh package version 1.0.0 to "venv_1".

Let's switch to "venv_2" and install version 1.11.

Now, let's switch back to "venv_1" and verify that its version of the sh package is still 1.0.

Virtualenvwrapper

All that activating, deactivating and switching can get old after a while. If you manage a lot of virtual environments, it can get out of control. That's where virtualenvwrapper comes in. Virtualenvwrapper lets you list, create, delete and copy virtual environments. It also lets you switch environments easily. 

Here are all the commands:

I use pretty much two commands: mkvirtualenv and workon. All the virtual environments are created under ~/.virtualenvironments.

Here is how to create a new virtual environment:

This is similar to virtualenv, but you don't specify a directory, just a name. Your new environment is here:

To switch between environments, you use the workon command, which without arguments just lists all the virtual environments. I have quite a few:

Virtualenv-Burrito

Virtualenv-Burrito is a project to install both virtualenv and virtualenvwrapper in one command.

Python 3 Venv

The venv module was added to Python 3.3 and provides built-in virtual environment creation and management just like virtualenv. The command to create virtual environments is pyenv. Otherwise it is pretty similar to virtualenv.

Conda

Virtual environments are very useful for isolating dependencies between different projects. But that doesn't extend to native libraries. Many C extensions depend on particular versions of native libraries, and those can't be virtual environment specific. 

Conda can address this problem. It is a package management system that handles all dependencies, not just Python dependencies. It can even be used for packaging non-Python software.

Alternatives to Virtual Environments

Do you have to use virtual environments? Not really. If for some reason you're not fond of the magic of virtual environments, there are other options.

Manually Vendorize

The functionality of a virtual environment is pretty simple. If you need total control, you can just do it yourself and copy exactly the subset of tools and packages you want into a target directory structure, set up some environment variables, and you're good to go.

VM or Dockerized System Python

If you bake your applications into a docker container or a cloud image then there will be just one project, and you may not need a virtual environment in the middle. You can just build on top of the system Python, being sure it will not be changed.

Tox

If all you care about is testing your code under different interpreters and environments then Tox can do all the heavy lifting for you. It will still use virtual environments under the covers, but you don't have to deal with them.

Best Practices

There are some packaging and virtual environment best practices that have emerged over time for robust Python systems.

Pin Versions in Your Requirements Files

Pinning means specifying precisely the versions of your dependencies. If a new version comes out and you install your application on a new server, you'll still use the version you tested against and that works, and not the latest and greatest. There is a downside here—you'll have to explicitly upgrade versions if you want to keep up with the progress made by your dependencies—but it is usually worth it.

Never Use the System Python

Relying on the system version is bad practice because there are other tools that rely on it, and if you start upgrading system packages, you may break them. You may also be affected by security updates that modify system packages, or in general if you want to upgrade your OS you may find that the system Python is now different.

Use a Private Package Index When Baking Images

PyPI may be down. If you need to bake a new image and can't access PyPI, you're in trouble. Devpi is a good option to avoid frustration.

Conclusion

There are many options for managing multiple Python projects on the same machine without conflicts. Figure out which option is best for you and use it. It is fast and easy to create virtual environments. Don't hesitate to take advantage of this useful tool. If you have special requirements, there are many solutions too.

Tags:

Comments

Related Articles