Overview
Many of us work on multiple Python projects at the same time. Multiple projects may depend on different versions of the same library. This is a problem. Even if you work with a single project and you deploy it to production, you may run into trouble, because the system's Python on your production server might change due to OS upgrade or security patch, and your application might fail as a result. In general, you want full control over the Python environment of your projects. Enter virtual environments...
The basic idea of a virtual environment is to have a Python interpreter and its site-packages separate from the system one. Also, you can have many of them. That solves both problems. You can assign a separate virtual environment with its own dependencies for each project and forget about conflicts with other projects and the system's Python.
In this tutorial, you'll learn the concepts behind virtual environments and how to create and use them, and you'll discover various alternatives for special situations.
Virtualenv
The virtualenv package supports this concept. You can install virtualenv using pip install virtualenv.
Once virtualenv is installed, you can start creating virtual environments. Let's create two environments called "venv_1" and "venv_2".
~ > virtualenv ~/venv_1 Using real prefix '/usr/local/Cellar/python/2.7.10/Frameworks/Python.framework/Versions/2.7' New python executable in /Users/gigi/venv_1/bin/python2.7 Also creating executable in /Users/gigi/venv_1/bin/python Installing setuptools, pip, wheel...done. ~ > virtualenv ~/venv_2 Using real prefix '/usr/local/Cellar/python/2.7.10/Frameworks/Python.framework/Versions/2.7' New python executable in /Users/gigi/venv_2/bin/python2.7 Also creating executable in /Users/gigi/venv_2/bin/python Installing setuptools, pip, wheel...done.
Let's see what happened.
~ > ls -la ~/venv_1 total 16 drwxr-xr-x 7 gigi staff 238 Mar 29 23:12 . drwxr-xr-x+ 254 gigi staff 8636 Mar 29 23:12 .. lrwxr-xr-x 1 gigi staff 79 Mar 29 23:12 .Python -> /usr/local/Cellar/python/2.7.10/Frameworks/Python.framework/Versions/2.7/Python drwxr-xr-x 16 gigi staff 544 Mar 29 23:12 bin drwxr-xr-x 3 gigi staff 102 Mar 29 23:12 include drwxr-xr-x 3 gigi staff 102 Mar 29 23:12 lib -rw-r--r-- 1 gigi staff 60 Mar 29 23:12 pip-selfcheck.json
Inside the "bin" sub-directory, you'll find a few executables and symlinks. Those include the Python interpreter itself, pip, easy_install, and most importantly a few activate scripts.
~ > ls -la ~/venv_1/bin total 136 drwxr-xr-x 16 gigi staff 544 Mar 29 23:12 . drwxr-xr-x 7 gigi staff 238 Mar 29 23:12 .. -rw-r--r-- 1 gigi staff 2077 Mar 29 23:12 activate -rw-r--r-- 1 gigi staff 1019 Mar 29 23:12 activate.csh -rw-r--r-- 1 gigi staff 2217 Mar 29 23:12 activate.fish -rw-r--r-- 1 gigi staff 1137 Mar 29 23:12 activate_this.py -rwxr-xr-x 1 gigi staff 249 Mar 29 23:12 easy_install -rwxr-xr-x 1 gigi staff 249 Mar 29 23:12 easy_install-2.7 -rwxr-xr-x 1 gigi staff 221 Mar 29 23:12 pip -rwxr-xr-x 1 gigi staff 221 Mar 29 23:12 pip2 -rwxr-xr-x 1 gigi staff 221 Mar 29 23:12 pip2.7 lrwxr-xr-x 1 gigi staff 9 Mar 29 23:12 python -> python2.7 -rwxr-xr-x 1 gigi staff 2336 Mar 29 23:12 python-config lrwxr-xr-x 1 gigi staff 9 Mar 29 23:12 python2 -> python2.7 -rwxr-xr-x 1 gigi staff 12744 Mar 29 23:12 python2.7 -rwxr-xr-x 1 gigi staff 228 Mar 29 23:12 wheel
The activate script is the key. In order to activate a specific virtual environment, you source the activate script, as in: source ~/venv_1/bin_activate. The effect is setting a bunch of environment variables and changing the prompt to the name of the activated virtual environment. It also adds a deactivate() shell function that will reset everything. Once a virtual environment is activated, typing python will launch its Python with its dependencies.
~ > source ~/venv_1/bin/activate (venv_1) ~ > which python /Users/gigi/venv_1/bin/python (venv_1) ~ >
Let's deactivate:
(venv_1) ~ > deactivate ~ > which python /usr/local/bin/python
If you have multiple Python interpreters installed on your systems, you can specify which one to use for your virtual environment using the -p option. Here is a Python 3 virtual environment:
~ > virtualenv ~/venv_3 -p /usr/local/bin/python3 Running virtualenv with interpreter /usr/local/bin/python3 Using base prefix '/usr/local/Cellar/python3/3.5.1/Frameworks/Python.framework/Versions/3.5' New python executable in /Users/gigi/venv_3/bin/python3.5 Also creating executable in /Users/gigi/venv_3/bin/python Installing setuptools, pip...done. ~ > source ~/venv_3/bin/activate (venv_3)~ > python Python 3.5.1 (default, Jan 9 2016, 19:28:52) [GCC 4.2.1 Compatible Apple LLVM 7.0.0 (clang-700.1.76)] on darwin Type "help", "copyright", "credits" or "license" for more information. >>>
Virtualenv works even on pypy.
~ > virtualenv ~/venv_pypy -p `which pypy` Running virtualenv with interpreter /usr/local/bin/pypy New pypy executable in /Users/gigi/venv_pypy/bin/pypy Installing setuptools, pip...done. ~ > source ~/venv_pypy/bin/activate (venv_pypy)~ > python Python 2.7.10 (5f8302b8bf9f53056e40426f10c72151564e5b19, Feb 11 2016, 20:39:39) [PyPy 4.0.1 with GCC 4.2.1 Compatible Apple LLVM 7.0.2 (clang-700.1.81)] on darwin Type "help", "copyright", "credits" or "license" for more information. >>>>
You can switch directly from one environment to the other without deactivating first:
(venv_3)~ > source ~/venv_2/bin/activate (venv_2) ~ > which python /Users/gigi/venv_2/bin/python
OK. Let's see how to use two different versions of the same package in two different virtual environments. This is as simple as activating each environment and installing the desired version. The environments are totally independent, so the fact that there is a different version in another environment is a non-issue.
Let's install the sh package version 1.0.0 to "venv_1".
(venv_1) ~ > pip install sh==1.0 Collecting sh==1.0.0 Downloading sh-1.0.tar.gz Building wheels for collected packages: sh Running setup.py bdist_wheel for sh ... done Stored in directory: /Users/gigi/Library/Caches/pip/wheels/f9/fb/a1/383f6dc2834b319a788a006d3ab7cc014ee852485f00b9e8c3 Successfully built sh Installing collected packages: sh Successfully installed sh-1.0 (venv_1) ~ > pip freeze | grep sh sh==1.0
Let's switch to "venv_2" and install version 1.11.
(venv_1) ~ > source ~/venv_2/bin/activate (venv_2) ~ > pip install sh==1.11 Collecting sh==1.11 Downloading sh-1.11.tar.gz Building wheels for collected packages: sh Running setup.py bdist_wheel for sh ... done Stored in directory: /Users/gigi/Library/Caches/pip/wheels/ba/4f/a5/ec77d662c3bf38f564b5ab16f1f3dbb9575922826fe810961c Successfully built sh Installing collected packages: sh Successfully installed sh-1.11 (venv_2) ~ > pip freeze | grep sh sh==1.11
Now, let's switch back to "venv_1" and verify that its version of the sh package is still 1.0.
(venv_2) ~ > source ~/venv_1/bin/activate (venv_1) ~ > pip freeze | grep sh sh==1.0 (venv_1) ~ >
Virtualenvwrapper
All that activating, deactivating and switching can get old after a while. If you manage a lot of virtual environments, it can get out of control. That's where virtualenvwrapper comes in. Virtualenvwrapper lets you list, create, delete and copy virtual environments. It also lets you switch environments easily.
Here are all the commands:
~ > virtualenvwrapper
virtualenvwrapper is a set of extensions to Ian Bicking's virtualenv
tool. The extensions include wrappers for creating and deleting
virtual environments and otherwise managing your development workflow,
making it easier to work on more than one project at a time without
introducing conflicts in their dependencies.
For more information please refer to the documentation:
http://virtualenvwrapper.readthedocs.org/en/latest/command_ref.html
Commands available:
add2virtualenv: add directory to the import path
allvirtualenv: run a command in all virtualenvs
cdproject: change directory to the active project
cdsitepackages: change to the site-packages directory
cdvirtualenv: change to the $VIRTUAL_ENV directory
cpvirtualenv: duplicate the named virtualenv to make a new one
lssitepackages: list contents of the site-packages directory
lsvirtualenv: list virtualenvs
mkproject: create a new project directory and its associated virtualenv
mktmpenv: create a temporary virtualenv
mkvirtualenv: Create a new virtualenv in $WORKON_HOME
rmvirtualenv: Remove a virtualenv
setvirtualenvproject: associate a project directory with a virtualenv
showvirtualenv: show details of a single virtualenv
toggleglobalsitepackages: turn access to global site-packages on/off
virtualenvwrapper: show this help message
wipeenv: remove all packages installed in the current virtualenv
workon: list or change working virtualenvs
I use pretty much two commands: mkvirtualenv and workon. All the virtual environments are created under ~/.virtualenvironments.
Here is how to create a new virtual environment:
~ > mkvirtualenv venv New python executable in venv/bin/python2.7 Also creating executable in venv/bin/python Installing setuptools, pip...done. (venv)~ >
This is similar to virtualenv, but you don't specify a directory, just a name. Your new environment is here:
(venv)~ > tree -L 2 ~/.virtualenvs/venv/
/Users/gigi/.virtualenvs/venv/
├── bin
│ ├── activate
│ ├── activate.csh
│ ├── activate.fish
│ ├── activate_this.py
│ ├── easy_install
│ ├── easy_install-2.7
│ ├── get_env_details
│ ├── pip
│ ├── pip2
│ ├── pip2.7
│ ├── postactivate
│ ├── postdeactivate
│ ├── preactivate
│ ├── predeactivate
│ ├── python -> python2.7
│ ├── python2 -> python2.7
│ └── python2.7
├── include
│ └── python2.7 -> /usr/local/Cellar/python/2.7.10/Frameworks/Python.framework/Versions/2.7/include/python2.7
└── lib
└── python2.7
To switch between environments, you use the workon command, which without arguments just lists all the virtual environments. I have quite a few:
(venv)~ > workon acme_server conman curr-gen nupic over-achiever pandas prime_hunter pypy quote-service venv work (venv)~ > workon conman (conman) ~ >
Virtualenv-Burrito
Virtualenv-Burrito is a project to install both virtualenv and virtualenvwrapper in one command.
Python 3 Venv
The venv module was added to Python 3.3 and provides built-in virtual environment creation and management just like virtualenv. The command to create virtual environments is pyenv. Otherwise it is pretty similar to virtualenv.
Conda
Virtual environments are very useful for isolating dependencies between different projects. But that doesn't extend to native libraries. Many C extensions depend on particular versions of native libraries, and those can't be virtual environment specific.
Conda can address this problem. It is a package management system that handles all dependencies, not just Python dependencies. It can even be used for packaging non-Python software.
Alternatives to Virtual Environments
Do you have to use virtual environments? Not really. If for some reason you're not fond of the magic of virtual environments, there are other options.
Manually Vendorize
The functionality of a virtual environment is pretty simple. If you need total control, you can just do it yourself and copy exactly the subset of tools and packages you want into a target directory structure, set up some environment variables, and you're good to go.
VM or Dockerized System Python
If you bake your applications into a docker container or a cloud image then there will be just one project, and you may not need a virtual environment in the middle. You can just build on top of the system Python, being sure it will not be changed.
Tox
If all you care about is testing your code under different interpreters and environments then Tox can do all the heavy lifting for you. It will still use virtual environments under the covers, but you don't have to deal with them.
Best Practices
There are some packaging and virtual environment best practices that have emerged over time for robust Python systems.
Pin Versions in Your Requirements Files
Pinning means specifying precisely the versions of your dependencies. If a new version comes out and you install your application on a new server, you'll still use the version you tested against and that works, and not the latest and greatest. There is a downside here—you'll have to explicitly upgrade versions if you want to keep up with the progress made by your dependencies—but it is usually worth it.
Never Use the System Python
Relying on the system version is bad practice because there are other tools that rely on it, and if you start upgrading system packages, you may break them. You may also be affected by security updates that modify system packages, or in general if you want to upgrade your OS you may find that the system Python is now different.
Use a Private Package Index When Baking Images
PyPI may be down. If you need to bake a new image and can't access PyPI, you're in trouble. Devpi is a good option to avoid frustration.
Conclusion
There are many options for managing multiple Python projects on the same machine without conflicts. Figure out which option is best for you and use it. It is fast and easy to create virtual environments. Don't hesitate to take advantage of this useful tool. If you have special requirements, there are many solutions too.
Comments