Overview

Python packages are the building blocks of Python applications. They encapsulate some coherent functionality that can be imported and used by many applications and systems. But first, developers need to find your package and be able to install it. Python provides a free public repository for packages, which is the de facto standard for sharing Python packages. You can also use private package repositories for proprietary packages.

In this tutorial you'll learn how to share your own packages with the community. If you have proprietary packages you need to share just within your company, you will learn how to do that too.

For background, see How to Use Python Packages and How to Write Your Own Python Packages. 

What Is PyPI?

PyPI stands for the Python Package Index. It is a public repository for uploading your packages. Pip is aware of PyPI and can install and/or upgrade packages from PyPI. PyPI used to be called the "Cheese Shop" after Monty Python's famous sketch. If you hear people refer to the "Cheese Shop" in a Python packaging context, don't be alarmed. It's just PyPI.

Prepare a Package for Upload

Before uploading a package, you need to have a package. I'll use the conman package I introduced in the article How to Write Your Own Python Packages. Since PyPI contains thousands of packages, it is very important to be able to describe your package properly if you want people to find it. PyPI supports an impressive set of metadata tags to let people find the right package for the job.

The setup.py file contains a lot of important information used to install your package. But it can also include the metadata used to classify your package on PyPI. Packages are classified using multiple metadata tags. Some of them are textual and some of them have a list of possible values. The full list is available on PyPI's List Classifiers page.

Let's add a few classifiers to setup.py. There is no need to increment the version number as it is only metadata and the code remains the same:

from setuptools import setup, find_packages



setup(name='conman',

      version='0.3',

      url='https://github.com/the-gigi/conman',

      license='MIT',

      author='Gigi Sayfan',

      author_email='[email protected]',

      description='Manage configuration files',

      classifiers=[

          'Development Status :: 3 - Alpha',

          'Intended Audience :: Developers',

          'Topic :: Software Development :: Libraries',

          'License :: OSI Approved :: MIT License',

          'Programming Language :: Python :: 2',

          'Programming Language :: Python :: 2.6',

          'Programming Language :: Python :: 2.7',

      ],

      packages=find_packages(exclude=['tests']),

      long_description=open('README.md').read(),

      zip_safe=False,

      setup_requires=['nose>=1.0'],

      test_suite='nose.collector')

Create an Account

You need to create an account on PyPI to be able to upload packages. Fill in this form and verify your identity by clicking on the URL in the verification email. Now, you need to create a .pypyrc file in your home directory that will contain the information needed to upload packages.

[distutils]

index-servers=pypi



[pypi]

repository = https://pypi.python.org/pypi

username = the_gigi

You can add your password too, but it's safer if you don't in case some bad element gets hold of your laptop. This is especially important if you upload popular packages because if someone can upload or upgrade your packages, all the people that use these packages will be vulnerable.

Testing

If you want to test the package registration and upload process and not worry about publishing something incomplete, you can work with the alternative PyPI testing site. Extend your ~/.pypirc file to include a 'pypitest' section.

[distutils]

index-servers=

    pypi

    pypitest



[pypitest]

repository = https://testpypi.python.org/pypi

username = the_gigi



[pypi]

repository = https://pypi.python.org/pypi

username = the_gigi

Remember that the test site is cleaned up regularly, so don't rely on it. It is intended for testing purposes only.

Register Your Package

If this is the first release of your package, you need to register it with PyPI. Twine has a register command, but I can't figure out how to use it. Following the documentation produces an error, and checking the unit tests for twine there is no test for the register command. Oh, well. You can do it manually too using this form to upload the PKG-INFO file. If you use Python 2.7.9+ or Python 3.2+, you can also safely register using python setup.py register.

Let's register conman on the PyPI test site. Note the -r pypitest, which based on the section in ~/.pypirc will register with the test site.

python setup.py register -r pypitest

running register

running egg_info

writing conman.egg-info/PKG-INFO

writing top-level names to conman.egg-info/top_level.txt

writing dependency_links to conman.egg-info/dependency_links.txt

reading manifest file 'conman.egg-info/SOURCES.txt'

reading manifest template 'MANIFEST.in'

writing manifest file 'conman.egg-info/SOURCES.txt'

running check

Password:

Registering conman to https://testpypi.python.org/pypi

Server response (200): OK

Twine

You can upload a package using python setup.py upload, but it is not secure as it used to send your username and password over HTTP until Python 2.7.9 and Python 3.2. Twine always uses HTTPS and has additional benefits like uploading pre-created distributions, and it supports any packaging format, including wheels. I will use twine for the actual upload.

Twine is not part of the standard library so you need to install it: pip install twine.

Upload Your Package

Finally, it's time to actually upload the package. 

> twine upload -r pypitest -p ******* dist/*

Uploading distributions to https://testpypi.python.org/pypi

Uploading conman-0.3-py2-none-any.whl

Uploading conman-0.3-py2.py3-none-any.whl

Uploading conman-0.3.tar.gz

Twine uploaded all the distribution formats, both the source and the wheels. 

Test Your Package

Once your package is on PyPI, you should make sure you can install it and everything works. Here I create a one-time virtual environment, pip install conman from the PyPI testing site, and then import it. You may want to run more thorough tests for your package.

> mkvirtualenv test_conman_pypi

New python executable in test_conman_pypi/bin/python2.7

Also creating executable in test_conman_pypi/bin/python

Installing setuptools, pip...done.

Usage: source deactivate



removes the 'bin' directory of the environment activated with 'source

activate' from PATH.

(test_conman_pypi) > pip install -i https://testpypi.python.org/pypi conman

Downloading/unpacking conman

  Downloading conman-0.3-py2-none-any.whl

  Storing download in cache at /Users/gigi/.cache/pip/https%3A%2F%2Ftestpypi.python.org%2Fpackages%2Fpy2%2Fc%2Fconman%2Fconman-0.3-py2-none-any.whl

Installing collected packages: conman

Successfully installed conman

Cleaning up...

(test_conman_pypi) > python

Python 2.7.10 (default, Jun 10 2015, 19:43:32)

[GCC 4.2.1 Compatible Apple LLVM 6.0 (clang-600.0.57)] on darwin

Type "help", "copyright", "credits" or "license" for more information.

>>> import conman

>>>

Note that the wheel distribution was installed by default.

Versioning

When you evolve your packages and upload new versions, it is important to follow a sensible versioning scheme. People will get pretty upset if an unintentional upgrade breaks their code. Your versioning scheme must comply with PEP-440 -- Version identification and dependency specification. 

This specification allows multiple schemes to choose from. I recommend using the popular Semantic Versioning scheme. It is pretty much "<major>.<minor>.<patch>", which corresponds to PEP-440's "<major>.<minor>.<micro>". Just beware of versions continuing the hyphen or plus signs, which are not compatible with PEP-440.

Private Package Repositories

PyPI is great, but sometimes you don't want to share your packages. Many companies and organizations have engineering teams that use Python and need to share packages between them, but are not allowed to share them publicly on PyPI. This is not a problem. You can share packages on private package repositories under your control. 

Note that sometimes you may want to have a private package repository under your control just to manage your third-party dependencies. For example, a package author can decide to delete a package from PyPI. If your system relies on being able to install this package from PyPI, you're in trouble.

Devpi

Devpi (which stands for Development Package Index) is a drop-in replacement for the public PyPI server. It is open source and MIT licensed, so you can run it inside your firewall. Devpi is very powerful and has many features that allow it to function as your ultimate packaging server:

• Fast PyPI mirror
• Uploading, testing and staging with private indexes
• Index inheritance
• Web interface and search
• Replication
• Importing/Exporting
• Jenkins integration

Devpi has excellent documentation, a plugin system and is in active development with a vibrant community.

Conclusion

Python provides a complete solution for hosting your packages and making them available to your fellow Pythonistas. There is a streamlined process assisted by tools to package and upload packages and make them easy to find and install. 

If you need to keep things private, Devpi is here for you as a mature and robust private package repository.