Distributing your own package on PyPi

In Regular Expressions Demystified I developed a little python package and distributed it via PyPi.

I wanted to publish my second self-written package as well, but coming back after almost a year, some things have changed in the world of PyPi, i.e. the old tutorials aren’t working anymore.

So I wrote this article to bring some clarity into this topic.

Table of Contents

Distutils vs Setuptools

Distutils

Distutils is still the standard tool for packaging in Python. It comes pre-installed with Python2 and Python3. Although it is still ‘standard’ it has some disadvantages tbc

for the case you stumble upon Distutils2: It was an attempt to combine the best from distutils and setuptools.

tl;dr; development of distutils2 has stopped, you shouldn’t use it!

Setuptools

As far as I can see setuptools is becoming the new de facto standard for packaging Python projects.

Wheel

A wheel is a

setuptools extension for building wheels that provides the bdist_wheel setuptools command
A command line tool for working with wheel files

The Advantages of wheels

Faster installation for pure Python and native C extension packages.
Avoids arbitrary code execution for installation. (Avoids setup.py)
Installation of a C extension does not require a compiler on Linux, Windows or macOS.
Allows better caching for testing and continuous integration.
Creates .pyc files as part of installation to ensure they match the Python interpreter used.
More consistent installs across platforms and machines.

Twine

Twine is a library / tool for publishing Python packages on PyPI.

You should favor using twine over python setup.py upload for the following reasons:

Verified HTTPS connections
Uploading doesn’t require executing setup.py
Uploading files that have already been created, allowing testing of distributions before release
Supports uploading any packaging format (including wheels)

For the next steps we are using all three – setuptools, wheel and twine.

Setup a project for packaging

Depending if your using pip or pipenv you have to either

pip install setuptools wheel twine

pipenv install setuptools wheel twine

The most basic project setup for a distribution looks like this:

/example_pkg
  /example_pkg
    __init__.py
    main.py
  setup.py

The setup.py contains the following lines:

import setuptools

setuptools.setup(
    name='example_pkg',
    version='0.0.1',
    packages=['example_pkg'],
    author='Joern Boegeholz',
    author_email='boegeholz.joern@gmail.com',
    description='An example Package',
)

Source Distribution

Having created the above setup, you can now build a distribution with the following command:

python setup.py sdist

Your project folder will now look like this:

/example_pkg 
  /dist
    example_pkg-0.0.1.tar.gz
  /example_pkg 
    __init__.py 
    main.py 
  setup.py

This command will create a example_pkg-0.0.1.tar.gz in a dist folder file which contains the source code of your package hence source distribution or sdist.

You can now distribute your package as a zip file. A user can download and unpack it.

With

python setup.py install

from inside the package folder, it will be installed into your Python site-packages.

Built Distribution with Wheel

With

python setup.py bdist_wheel

you can create a wheel.

Your dist folder now contains a example_pkg-0.0.1-py3-none-any.whl

That’s a funny filename py3-none-any.whl, so let’s decompose it:

py3 -> meant for usage with Python 3
none -> not OS-specific
any -> suitable to run on any processor architecture

If your package doesn’t contain any C extension and is compatible with Python2 as well, you can add a setup.cfg file to your project folder and insert these lines:

[bdist_wheel]
universal = 1

After running python setup.py bdist_wheel again the file

example_pkg-0.0.1-py2.py3-none-any.whl is created.

It is recommended to always create both: sdist and bdist_wheel

Uploading to PyPi

twine upload dist/*

The complete guide can be found here.

Migrating from distutils to setuptools

If You are using distutils and want to migrate here is the workflow in a nutshell:

Step 1 – Install requirements

pip install setuptools wheel twine

Step 2 – Change setup.py

Instead of

from distutils.core import setup
setup(...)

write

import setuptools
setuptools.setup(...)

Step 3 – Adding / updating setup.cfg

[bdist_wheel]
universal = 1

Dependencies

If your package requires other packages to work, you have to add the requirements to the setup.py as well:

setup(
...
    install_requires=[
        'pip',
        'pipdeptree'
    ],
...)