How to Use Python Packages

Python packages allow you to break down large systems and organize their modules in a consistent way that you and other people can use and reuse efficiently. Python's motto of "Batteries Included" means that it comes preloaded with lots of useful packages in the standard library. 

But there are also many amazing third-party packages you can take advantage of. In this tutorial you'll learn all you need to know about what packages are exactly, how to import modules from packages, exploring the built-in package in Python's standard library, and installing third-party packages.

What Are Packages?

Before we can talk about packages, let's talk about modules. Modules are the source files with *.py extension where you (and everyone else) put the functions and classes that comprise your program. Packages are the manifestation of Python's hierarchical namespaces concept. To quote from the Zen of Python: 

"Namespaces are one honking great idea -- let's do more of those!"

To view the whole Zen of Python, type import this in a Python interactive session:

Namespaces help with organizing code and preventing naming conflicts. This is critical when multiple people work together or when using packages developed by other people.

While packages represents a hierarchy of sub-packages and modules, which are files the hierarchy doesn't have to file-system based where packages and sub-packages are directories and sub-directories. It is much more flexible than that.

Example Package

Let's take a look at a package called "ansible". It is not a package from the standard library. You'll see later how to find and install third-party packages. Now, let's just check out the directory file structure. 

The packages will be installed typically into the Python interpreter's site-packages directory, which will be located somewhere (depending on version, OS, and distribution) under the "lib". On the Mac, for Python 2.7 it will be located in "<interpreter root>/lib/python2.7/site-packages". Here is how the "ansible" package is organized:

There are two modules and 15 directories. Each directory is a sub-package of the main ansible package. Looking inside the ansible/utils directory, we can see it contains additional modules and even one more sub-package:

The Search Path

When you import a module, Python will go through a search algorithm based on the search path, which is a list of directories to start the search. The search path is a list of directories available through sys.path, and you can manipulate it dynamically (add, remove or move around items in the search path). The site-packages directory is always there.

To import the path.py module from ansible/utils, you'll need to use the following command:

import ansible.utils.path

If you also want to use the standard os.oath module, you'll use the following command:

import os.path

Now you can use either or both path modules with no conflicts due to the difference namespace they belong to.

Exploring the Standard Library

The standard library has a lot of packages. It's worth exploring it whenever you need to accomplish some task and you're not sure how. There is a very high likelihood that for any general-purpose task like math, shell integration, OS integration, string manipulation, networking and common file formats, there is a well-designed, well-performing and well-tested package in the standard library. 

You can really trust standard library packages because it is a big deal to get into the standard library. Either the package was designed by Python's core developers or it was heavily reviewed and often heavily used in the field as a third-party library before making it into the standard library.

Here are all the packages in the standard library organized by topic.

PyPI

The standard library is awesome, but there'll often be some special functionality you need that is not standard. It doesn't mean you have to write it from scratch. Python has a vibrant and active community that develops and shares freely a lot of code. Enter PyPI - the Python Package Index. PyPI hosts all publicly available packages and provides a one-stop shop for browsing through them.

Browsing PyPI

PyPI organizes the packages in a browsable index. You can browse and search by topic, environment, framework, development, status, intended audience, license, natural language, programming language (yes, there are Python packages that support many programming languages), and operating system. 

There is also a distinction between Python 2 and Python 3 packages, and you can see how popular a package is by the number of recent downloads. For example, the ansible package is available on PyPI, and here is its metadata:

Installing Packages

There are two ways to install packages from PyPI. You can download the package and then run python setup.py install. But the modern way is to use pip, setuptools and wheel. Pip stands for Pip Installs Packages (yes, it's one of those acronyms) and is your front end for installation. If you have Python 2 >=2.7.9 or Python 3 >=3.4 installed from python.org, you will already have pip and setuptools, but will need to upgrade to the latest version: pip install -U pip setuptools or python -m pip install -U pip setuptools on Windows.

Use pip to install wheel: pip install wheel.

If you're using an earlier version of Python, consider upgrading. If you're unable to, you'll need to install pip, setuptools and wheel on your own. Check the instructions.

Python packages are always installed into an environment. A common practice I will not cover here is to use virtual environments to manage multiple independent installations of Python with different interpreters and/or different sets of installed packages.

Best Practices

The Python packaging authority provides a lot of guidance on the best practices around packaging. This is important because it is an area of active development and recommendations evolve quickly.

Also, if you want to do something special like installing packages from alternative repositories instead of PyPI or using pip in a more sophisticated way, you'll find great discussions and practical advice. 

Conclusion

When you're a Python beginner, you learn the core language and have fun playing with it. Pretty soon you discover the standard library, and as you gain more experience you benefit more and more from its richness. 

The next stage in your evolution as a Pythonista is to incorporate the vast awesomeness the Python community has put on PyPI into your systems. Packages as the deployment unit of reusable Python code enable this ecosystem.

Tags:

Comments

Related Articles