Python packages allow you to break down large systems and organize their modules in a consistent way that you and other people can use and reuse efficiently. Python's motto of "Batteries Included" means that it comes preloaded with lots of useful packages in the standard library.
But there are also many amazing third-party packages you can take advantage of. In this tutorial you'll learn all you need to know about what packages are exactly, how to import modules from packages, exploring the built-in package in Python's standard library, and installing third-party packages.
What Are Packages?
Before we can talk about packages, let's talk about modules. Modules are the source files with *.py extension where you (and everyone else) put the functions and classes that comprise your program. Packages are the manifestation of Python's hierarchical namespaces concept. To quote from the Zen of Python:
"Namespaces are one honking great idea -- let's do more of those!"
To view the whole Zen of Python, type import this in a Python interactive session:
import this The Zen of Python, by Tim Peters Beautiful is better than ugly. Explicit is better than implicit. Simple is better than complex. Complex is better than complicated. Flat is better than nested. Sparse is better than dense. Readability counts. Special cases aren't special enough to break the rules. Although practicality beats purity. Errors should never pass silently. Unless explicitly silenced. In the face of ambiguity, refuse the temptation to guess. There should be one-- and preferably only one --obvious way to do it. Although that way may not be obvious at first unless you're Dutch. Now is better than never. Although never is often better than *right* now. If the implementation is hard to explain, it's a bad idea. If the implementation is easy to explain, it may be a good idea. Namespaces are one honking great idea -- let's do more of those!
Namespaces help with organizing code and preventing naming conflicts. This is critical when multiple people work together or when using packages developed by other people.
While packages represents a hierarchy of sub-packages and modules, which are files the hierarchy doesn't have to file-system based where packages and sub-packages are directories and sub-directories. It is much more flexible than that.
Example Package
Let's take a look at a package called "ansible". It is not a package from the standard library. You'll see later how to find and install third-party packages. Now, let's just check out the directory file structure.
The packages will be installed typically into the Python interpreter's site-packages directory, which will be located somewhere (depending on version, OS, and distribution) under the "lib". On the Mac, for Python 2.7 it will be located in "<interpreter root>/lib/python2.7/site-packages". Here is how the "ansible" package is organized:
tree ansible -L 1 ansible ├── __init__.py ├── cli ├── compat ├── config ├── constants.py ├── errors ├── executor ├── galaxy ├── inventory ├── module_utils ├── modules ├── parsing ├── playbook ├── plugins ├── template ├── utils └── vars 15 directories, 2 files
There are two modules and 15 directories. Each directory is a sub-package of the main ansible package. Looking inside the ansible/utils directory, we can see it contains additional modules and even one more sub-package:
tree ansible/utils -L 1 ansible/utils ├── __init__.py ├── boolean.py ├── cmd_functions.py ├── color.py ├── debug.py ├── display.py ├── encrypt.py ├── hashing.py ├── listify.py ├── module_docs.py ├── module_docs_fragments ├── path.py ├── shlex.py ├── unicode.py └── vars.py 1 directory, 14 files
The Search Path
When you import a module, Python will go through a search algorithm based on the search path, which is a list of directories to start the search. The search path is a list of directories available through sys.path, and you can manipulate it dynamically (add, remove or move around items in the search path). The site-packages directory is always there.
To import the path.py module from ansible/utils, you'll need to use the following command:
import ansible.utils.path
If you also want to use the standard os.oath module, you'll use the following command:
import os.path
Now you can use either or both path modules with no conflicts due to the difference namespace they belong to.
Exploring the Standard Library
The standard library has a lot of packages. It's worth exploring it whenever you need to accomplish some task and you're not sure how. There is a very high likelihood that for any general-purpose task like math, shell integration, OS integration, string manipulation, networking and common file formats, there is a well-designed, well-performing and well-tested package in the standard library.
You can really trust standard library packages because it is a big deal to get into the standard library. Either the package was designed by Python's core developers or it was heavily reviewed and often heavily used in the field as a third-party library before making it into the standard library.
Here are all the packages in the standard library organized by topic.
PyPI
The standard library is awesome, but there'll often be some special functionality you need that is not standard. It doesn't mean you have to write it from scratch. Python has a vibrant and active community that develops and shares freely a lot of code. Enter PyPI - the Python Package Index. PyPI hosts all publicly available packages and provides a one-stop shop for browsing through them.
Browsing PyPI
PyPI organizes the packages in a browsable index. You can browse and search by topic, environment, framework, development, status, intended audience, license, natural language, programming language (yes, there are Python packages that support many programming languages), and operating system.
There is also a distinction between Python 2 and Python 3 packages, and you can see how popular a package is by the number of recent downloads. For example, the ansible package is available on PyPI, and here is its metadata:
Downloads (All Versions): 5528 downloads in the last day 72037 downloads in the last week 289967 downloads in the last month Author: Ansible, Inc. Home Page: http://ansible.com/ Bug Tracker: http://github.com/ansible/ansible/ License: GPLv3 Categories Development Status :: 5 - Production/Stable Environment :: Console Intended Audience :: Developers Intended Audience :: Information Technology Intended Audience :: System Administrators License :: OSI Approved :: GNU General Public License v3 or later (GPLv3+) Natural Language :: English Operating System :: POSIX Programming Language :: Python :: 2.6 Programming Language :: Python :: 2.7 Topic :: System :: Installation/Setup Topic :: System :: Systems Administration Topic :: Utilities Package Index Owner: James.Laska, jimi, ansible Package Index Maintainer: mpdehaan, badger DOAP record: ansible-2.0.0.2.xml
Installing Packages
There are two ways to install packages from PyPI. You can download the package and then run python setup.py install. But the modern way is to use pip, setuptools and wheel. Pip stands for Pip Installs Packages (yes, it's one of those acronyms) and is your front end for installation. If you have Python 2 >=2.7.9 or Python 3 >=3.4 installed from python.org, you will already have pip and setuptools, but will need to upgrade to the latest version: pip install -U pip setuptools or python -m pip install -U pip setuptools on Windows.
Use pip to install wheel: pip install wheel.
If you're using an earlier version of Python, consider upgrading. If you're unable to, you'll need to install pip, setuptools and wheel on your own. Check the instructions.
Python packages are always installed into an environment. A common practice I will not cover here is to use virtual environments to manage multiple independent installations of Python with different interpreters and/or different sets of installed packages.
Best Practices
The Python packaging authority provides a lot of guidance on the best practices around packaging. This is important because it is an area of active development and recommendations evolve quickly.
Also, if you want to do something special like installing packages from alternative repositories instead of PyPI or using pip in a more sophisticated way, you'll find great discussions and practical advice.
Conclusion
When you're a Python beginner, you learn the core language and have fun playing with it. Pretty soon you discover the standard library, and as you gain more experience you benefit more and more from its richness.
The next stage in your evolution as a Pythonista is to incorporate the vast awesomeness the Python community has put on PyPI into your systems. Packages as the deployment unit of reusable Python code enable this ecosystem.
Comments