Python Packaging without Complication

daveops.com/pycon2017

dave@forgac.com

@tylerdave

(1/68)

Before I get started: I just want to note that I'm going to cover a lot of topics briefly
I have these slides, notes, and links to further reading about everything I mention at the URL here
And I'll have the URL up again at the end so if you miss it now you can get it then

def beautiful_code():
   """This code is so beautiful."""
    
   print('Hello, PyCon!')

(2/68)

You write beautiful Python code
If you think it'll be useful to others, you want to share it
When you share it, you want to do it in a way that people to know how to download and use it
And you may even want people to collaborate with you to make the code even better

“I want to share my code!
What do I need to do?”

(3/68)

What do we need in order to do that?

Package(4/68)

First, we need to package the code
And then upload it to the package index
This allows people to install it the way they expect, via pip

Documentation(5/68)

I also said that we want people to know how to use the code
This means having documentation
What else?

Source Hosting(6/68)

If we want people to be able to collaboarate with us to make the code better
then we need to put the code somewhere people can access it
This typically means putting it on GitHub, Bitbucket, GitLab, or another provider

Tests(7/68)

And we want to make sure the when people make changes they don't break things
So we need tests

CI(8/68)

we want the tests to be run automaticlaly
so we can see if any incoming contributions breaks things
so we should set up a continuous integration service to run our tests

License(9/68)

we should have a license so people know under what terms they can use the code

Contributing(10/68)

we should probably have contributing instructions so people know how to go about helping
this should include things like
should they open an issue first?
submit a pull request?
what kind of things do you want help with?

we need

Code

Package

Documentation

Source Hosting

Tests

License

Contributing

(11/68)

Now we have a list of things we need to do to share our code

“But I just wanted to share my code!”

(12/68)

at this point you may be thinking that setting up all of this sounds like a lot of work
and you know it would be

TL;DR(13/68)

If you only take one thing away from this talk, I hope it's this:
Because someone has already done a lot of work to get these things set up for you

TL;DR

Cookiecutter

(13/68)

If you only take one thing away from this talk, I hope it's this:
Because someone has already done a lot of work to get these things set up for you

Use cookiecutter
cookiecutter is a tool that allows you to make new projects from templates
templates can be for any kind of project but there a number of good ones for python packages -- i have one linked with the notes
My recommendation is to start with a template that's close to what you want
fork it, and then updated it for your preferences
you can then use that when you make new projects

Terminology(14/68)

before we go any further I want to go over some terminology just to make sure we're using the same language.

ModuleSaved Python code(15/68)

Anything importable from a file

Import Packagenamespace / directory(16/68)

Just a directory structure for organizing python modules

Distribution PackageShareable / installable(17/68)

Then a distribution package is your code
bundled-up in a way that it can be shared and installed
This talk is about sharing your code so for the rest of the talk the term "package" refers to one of these
of course to make things more complex there are multiple types of distrubtion packages

Source Distributionsdist(18/68)

a source distribution contains bundled source code
this includes extension source if you have extensions in anohter language like C
when you install a source distribution any extensions are complied at install time

Built DistributionWheel(19/68)

built distributions are built or compiled beforehand so they just need to be unpacked at install time
eggs are an older format may see referenced
but wheels are the current built distribution format and you should be making them for your projects
for a while the package index only supported uploading weels for windows and mac
more recently the manylinux platform tag was developed
it defines a lowest-common denominator linux interface which allows wheels to be built that will work on most linux systems

Wheel(20/68)

and of course there are multiple types of wheels too

WheelUniversal(20/68)

and of course there are multiple types of wheels too
a universal wheel is one that contains only python code and is compatible with python 2 and 3

WheelUniversalPure Python(20/68)

and of course there are multiple types of wheels too
a universal wheel is one that contains only python code and is compatible with python 2 and 3
a pure python wheel is specific to either 2 or 3 but is still only python code

WheelUniversalPure PythonPlatform(20/68)

and of course there are multiple types of wheels too
a universal wheel is one that contains only python code and is compatible with python 2 and 3
a pure python wheel is specific to either 2 or 3 but is still only python code
a platform wheel has compiled code so it's specific to a target platform
one has to be created for every platform on which you want to

History(21/68)

this is the extremely short version of packaging history
the packaging ecosystem grew organically and for a long time it wasn't clear what was considered the current best practice
if you searched online you'd find lots of conflicting information about how to do things

HistoryCopy / Replace(21/68)

this is the extremely short version of packaging history
the packaging ecosystem grew organically and for a long time it wasn't clear what was considered the current best practice
if you searched online you'd find lots of conflicting information about how to do things
so the way i'd go about packaging was: i'd find a project that seemed to have it together and make a copy
and then replace their code with mine and then change the project name in the files

HistoryCopy / ReplacePyPA(21/68)

this is the extremely short version of packaging history
the packaging ecosystem grew organically and for a long time it wasn't clear what was considered the current best practice
if you searched online you'd find lots of conflicting information about how to do things
so the way i'd go about packaging was: i'd find a project that seemed to have it together and make a copy
and then replace their code with mine and then change the project name in the files
then in 2011 the packaging authority working group was created to take over maintenance of core packaging tools
and they've since worked to make some great improvements to the packaging ecosystem

History

Copy / Replace

PyPA

PyPUG

https://packaging.python.org/

(21/68)

this is the extremely short version of packaging history
the packaging ecosystem grew organically and for a long time it wasn't clear what was considered the current best practice
if you searched online you'd find lots of conflicting information about how to do things
so the way i'd go about packaging was: i'd find a project that seemed to have it together and make a copy
and then replace their code with mine and then change the project name in the files
then in 2011 the packaging authority working group was created to take over maintenance of core packaging tools
and they've since worked to make some great improvements to the packaging ecosystem
besides the specification and tooling improvements, another major contribution of the working group is maintenance of the python packaging user guide
so we finally have an authoritative resource for how we should be creating and installing packages

“Let's make a package!”

(22/68)

Let's make a package for our code

Using cookiecutter

$ cookiecutter cookiecutter-python-package
…
full_name (default is "Dave Forgac")?
email (default is "tylerdave@tylerdave.com")?
github_username (default is "tylerdave")?
project_name (default is "Example")? Example Package
repo_name (default is "boilerplate")? Example-Package
project_short_description (default is "A short description")?
 Example package for PyCon talk.
release_date (default is "2017-05-21")?
year (default is "2017")?
version (default is "0.1.0")?

(23/68)

when you use cookiecutter it prompts you for some information about your project
and then generates a directory structure for your project customized with your input

Layout

.
├── data
│   └── data_file
├── DESCRIPTION.rst
├── MANIFEST.in
├── README.rst
├── sample
│   ├── __init__.py
│   └── package_data.dat
├── setup.cfg
├── setup.py
└── tests
    ├── __init__.py
    └── test_sample.py

(24/68)

here is an overview of the files that make up a minimal package
most templates will have more but we'll take a look at these files first

Code

.
├── data
│   └── data_file
├── DESCRIPTION.rst
├── MANIFEST.in
├── README.rst
├── sample
│   ├── __init__.py
│   └── package_data.dat
├── setup.cfg
├── setup.py
└── tests
    ├── __init__.py
    └── test_sample.py

(25/68)

The first part is your code
the directory was created by cookiecutter and then the code was copied into it
a common pattern is to store your code in an import package with the same name as the project

setup.py

.
├── data
│   └── data_file
├── DESCRIPTION.rst
├── MANIFEST.in
├── README.rst
├── sample
│   ├── __init__.py
│   └── package_data.dat
├── setup.cfg
├── setup.py
└── tests
    ├── __init__.py
    └── test_sample.py

(26/68)

most important file for packaging and distribution
has info about the package that's used during installation and by the package index
note that it's just python code and that it's executed during installation
this would allow you to do some fancy things
but don't do fancy things

setup.cfg

.
├── data
│   └── data_file
├── DESCRIPTION.rst
├── MANIFEST.in
├── README.rst
├── sample
│   ├── __init__.py
│   └── package_data.dat
├── setup.cfg
├── setup.py
└── tests
    ├── __init__.py
    └── test_sample.py

(27/68)

configuration file to go along with setup.py
it's commonly used for configuring wheel building options and configuration for packaging related plugins and tooling

MANIFEST.in

.
├── data
│   └── data_file
├── DESCRIPTION.rst
├── MANIFEST.in
├── README.rst
├── sample
│   ├── __init__.py
│   └── package_data.dat
├── setup.cfg
├── setup.py
└── tests
    ├── __init__.py
    └── test_sample.py

(28/68)

by default your python code will be included when you build a distrubtion package
but if you need any non-python files to be included they need to be listed in MANIFEST.in
this is typically used for things like data files, configuration, and docs

README.rst

.
├── data
│   └── data_file
├── DESCRIPTION.rst
├── MANIFEST.in
├── README.rst
├── sample
│   ├── __init__.py
│   └── package_data.dat
├── setup.cfg
├── setup.py
└── tests
    ├── __init__.py
    └── test_sample.py

(29/68)

the README is what people see when they browse your project on source hosting like github
it's also commonly used as the long discription for the package index
it should contain general information about what your code does, basic usage instructions, and where to find more info like a link to your docs

setup.py (1/2)

from setuptools import setup, find_packages
      
setup(
    name='Example Package',
    version='0.1.0',     # PEP440-compliant version
    description='Example package for PyCon talk.',
    long_description='Displayed on PyPI project page.',
    url='https://github.com/tylerdave/Example-Package',
    author='Dave Forgac',
    author_email='tylerdave@tylerdave.com',
    packages=find_packages(exclude=['docs', 'tests'])
    install_requires=['requests'],
    package_data={
        'sample': ['package_data.dat']
        }
    …

(30/68)

now let's take a look at a setup.py
we import setup and optionally find_packages from setuptools
then the packaging settings are defined as arguments to the setup function

setup.py (1/2)

from setuptools import setup, find_packages
     
setup(
    name='Example Package',
    version='0.1.0',     # PEP440-compliant version
    description='Example package for PyCon talk.',
    long_description='Displayed on PyPI project page.',
    url='https://github.com/tylerdave/Example-Package',
    author='Dave Forgac',
    author_email='tylerdave@tylerdave.com',
    packages=find_packages(exclude=['docs', 'tests'])
    install_requires=['requests'],
    package_data={
        'sample': ['package_data.dat']
        }
    …

(31/68)

then we define basic package meta-information
like name of project
a description of the project
the version number
author information
long_description
often you'll see long_description loaded from README / DESCRIPTION

setup.py (1/2)

from setuptools import setup, find_packages
     
setup(
    name='Example Package',
    version='0.1.0',     # PEP440-compliant version
    description='Example package for PyCon talk.',
    long_description='Displayed on PyPI project page.',
    url='https://github.com/tylerdave/Example-Package',
    author='Dave Forgac',
    author_email='tylerdave@tylerdave.com',
    packages=find_packages(exclude=['docs', 'tests'])
    install_requires=['requests'],
    package_data={
        'sample': ['package_data.dat']
        }
    …

(32/68)

then we define what import packages should be included in the distribution
they can be listed manually but the find_package we imported above will do it for you
if you use it you exclude things that aren't meant to be installed

setup.py (1/2)

from setuptools import setup, find_packages
     
setup(
    name='Example Package',
    version='0.1.0',     # PEP440-compliant version
    description='Example package for PyCon talk.',
    long_description='Displayed on PyPI project page.',
    url='https://github.com/tylerdave/Example-Package',
    author='Dave Forgac',
    author_email='tylerdave@tylerdave.com',
    packages=find_packages(exclude=['docs', 'tests'])
    install_requires=['requests'],
    package_data={
        'sample': ['package_data.dat']
        }
    …

(33/68)

install_requires is what is used to determine the dependencies of your package at install time
so when you pip install the package, things listed here will be installed
note that this is different from requirements.txt we'll look at that in a moment but i've linked to a page with more details about the distinctions in the notes

setup.py (1/2)

from setuptools import setup, find_packages
     
setup(
    name='Example Package',
    version='0.1.0',     # PEP440-compliant version
    description='Example package for PyCon talk.',
    long_description='Displayed on PyPI project page.',
    url='https://github.com/tylerdave/Example-Package',
    author='Dave Forgac',
    author_email='tylerdave@tylerdave.com',
    packages=find_packages(exclude=['docs', 'tests'])
    install_requires=['requests'],
    package_data={
        'sample': ['package_data.dat']
        }
    …

(34/68)

then package data allows you to define extra files that will be included in your pacakge
they are exposed to your code via the name on the left so you don't have to worry about calculating paths

setup.py (2/2)

    …
    entry_points={
        'console_scripts': [
            'hello=example_package.cli:say_hello',
        ],
    }
    license='MIT',
    classifiers=[
        'Development Status :: 3 - Alpha',
        'License :: OSI Approved :: MIT License',
        'Programming Language :: Python :: 2',
        'Programming Language :: Python :: 2.7',
        'Programming Language :: Python :: 3',
        'Programming Language :: Python :: 3.4',
        'Programming Language :: Python :: 3.5',
        'Programming Language :: Python :: 3.6',
    ])

(35/68)

Entry points defines interfaces that your package provides
this is commonly used for defining plugins
console_scripts is a special type of entry point
it creates commands that map command names on the left to your python code on the right
it's good to use this because it does the right thing depending on the OS
on linux and mac it'll create a wrapper script and put it on the path
on windows it'll create a small exe file that knows how to find the right code

setup.py (2/2)

    …
    entry_points={
        'console_scripts': [
            'hello=example_package.cli:say_hello',
        ],
    }
    license='MIT',
    classifiers=[
        'Development Status :: 3 - Alpha',
        'License :: OSI Approved :: MIT License',
        'Programming Language :: Python :: 2',
        'Programming Language :: Python :: 2.7',
        'Programming Language :: Python :: 3',
        'Programming Language :: Python :: 3.4',
        'Programming Language :: Python :: 3.5',
        'Programming Language :: Python :: 3.6',
    ])

(36/68)

Indicate the license under which your code is released
Then the classifiers section lists categorization information about the package
this is typically used to indicate which versions of python you intend to support
but as of now nothing is enforced based on this information

setup.py (2/2)

    …
    entry_points={
        'console_scripts': [
            'hello=example_package.cli:say_hello',
        ],
    }
    license='MIT',
    classifiers=[
        'Development Status :: 3 - Alpha',
        'License :: OSI Approved :: MIT License',
        'Programming Language :: Python :: 2',
        'Programming Language :: Python :: 2.7',
        'Programming Language :: Python :: 3',
        'Programming Language :: Python :: 3.4',
        'Programming Language :: Python :: 3.5',
        'Programming Language :: Python :: 3.6',
    ])

(37/68)

install_requires ensures dependencies are met
package_data is mapping pkg name to relative path
data_files can specify data outside of your path
scripts can specify an individual script
entry_points is used for dynamic plugin discovery
most common entry_point is console scripts
console scripts makes an executable wrapper

setup.cfg

[bdist_wheel]
universal = 1                
    
[flake8]
exclude = docs

(38/68)

the setup.cfg file contains additional packaging settings
here we see settings for wheel building
and for the plugin flake8

MANIFEST.in

include LICENSE
include README.rst
    
recursive-include tests *
recursive-exclude * __pycache__
recursive-exclude * *.py[co]

(39/68)

manifest.in lists files that should be included or excluded on top of what setuptools finds

README.rst

===============================
Example Package
===============================
    
License: MIT
    
Documentation: https://example.com
    
Usage
-----
    
``sample --help``

(40/68)

the readme is in restructured text format and

LICENSE(41/68)

If you're publishing your code, you need a license
Coose from one of the common OS licenses unless your laywer tells you otherwise
include it in a file named license and reference it in your README and setup.py

Tests

./tests

tox.ini

(42/68)

even if this is just an experimental project, you need tests
but at minimum, test that the most basic use case works as expected
then if you're looking to improve coverage, note that you want help with tests in your contributing instructions
people are a lot more likely to expand on existing tests than they are to create them for you from scratch
your template should include configuration for running your tests
in my case using tox

Docs

./docs

(43/68)

like tests you should have at least basic usage documentation
again, people are a lot more likely to add to existing documentation than they are to add docs wholesale
the templates will give you good structure for your documentation and you should at least fill in the usage information

CI

.travis.yml

(44/68)

the template should include configuration for your CI service
i use travis ci by default but if you have a different service your template should have configuration for that

requirements.txt(45/68)

this serves a different purpose than the install_requires in setup.py
this is generally a list of dependencies used by developers
a great tool for managing this is called pip-tools
i have a link to it in my further reading

Also Common

(rst | md | txt)

HISTORY or CHANGES or CHANGELOG

CONTRIBUTING

AUTHORS

(46/68)

“Now let's share our code!”

(47/68)

ok, we have all of our files in place and we're ready to share the code

Git init

cd Example-Package
git init
git add .
git commit -m 'initial commit'

(48/68)

assuming we're using git we'll add our code to source control

Run tests

tox
…
OK
___________________________ summary __________________________
  py27: commands succeeded
  py34: commands succeeded
  py35: commands succeeded
  py36: commands succeeded

(49/68)

Commit

git add .
git commit -m "add hello pycon functionality"

(50/68)

Services(51/68)

GitHub

Create a github repo

(52/68)

we're going to create a new github repo

Travis CI

https://travis-ci.org/

sync

travis enable

(53/68)

add our repo to travis
sign in with github
click + next to my repos

Git push

To git@github.com:tylerdave/Example-Package.git
 * [new branch]      master -> master
Branch master set up to track remote branch master
from origin.

(54/68)

push our code to github

See builds succeeding!

build jobs

(55/68)

go back to travis and see our builds are succeeding

ReadTheDocs logo

(56/68)

next we're going to set up documentation hosting
sign up for account / login
click import from github
select project, verify settings
click build, wait -> view

PyPI

https://pypi.python.org/

(57/68)

create an account on pypi
you'll use this when uplading packages

PyPI

https://pypi.python.org/

Test PyPI

https://testpypi.python.org/

(57/68)

create an account on pypi
you'll use this when uplading packages

note that there's a test instance of the package index
so if you want to experiment with the upload process you can do it there without adding more junk to the real index

Save PyPI Settings

$HOME/.pypirc

[distutils]
index-servers=pypi
[pypi]
repository = https://pypi.python.org/pypi
username = <username>
password = <password>

(58/68)

to make things easier for you
you should save some settings in a .pypirc file in your home directory
Can omit password and the tools will prompt you for them instead
Can configure alternate indexes

Virtualenvpewvirtualenvwrapper(59/68)

should work in a virtualenv
check out pew or virtualenvwrapper to make things easier for you

Packaging Tools

pip install wheel

pip install twine

(60/68)

depending on your environment you may need to install wheel
and install twine, which is a tool for uploading your packages to the index

Build dist. files

./setup.py sdist

./setup.py bdist_wheel

make dist

(61/68)

run commands to build distrubiton files
if you use one of the default templates it'll give you a Makefile that includes make dist

Upload

twine upload dist/*

(62/68)

twine will register the package if it's not already registered
and then upload your distribution files
you should use twine because the upload command that's built into setup.py is insecure

Install

pip install example-package

(63/68)

now you can install your package via pip

Develop Mode

./setup.py develop

pip install -e .

(64/68)

going-forward you can install your package in editable development mode
this will make it so that you can edit your code and see the changes in your virtualenv without reinstalling
note that if you change the entry points at all you do need to re-run the install command

Rinse, Repeat(65/68)

then you can change your code
update the version
commit and push
verify tests are passing
build distribution files
upload them to pypi

Versionsbumpversionversioneer(66/68)

two tools that take different approaches to this are bumpversion and versioneer

Recap(67/68)

RecapDIY(67/68)

don't set up all this stuff by hand
use existing templates

RecapDIYUse the tools(67/68)

don't set up all this stuff by hand
use existing templates

take the time to set up tools and services that will make your life easier in the long run

RecapDIYUse the toolsShare your code!(67/68)

don't set up all this stuff by hand
use existing templates

take the time to set up tools and services that will make your life easier in the long run

share your code so other people can benefit from the cool stuff you do

Thank you!

Please leave feedback

daveops.com/pycon2017

dave@forgac.com

@tylerdave

(68/68)

Help

Keyboard shortcuts

↑, ←, Pg Up, k

Go to previous slide

↓, →, Pg Dn, Space, j

Go to next slide

Home

Go to first slide

End

Go to last slide

Number + Return

Go to specific slide

b / m / f

Toggle blackout / mirrored / fullscreen mode

Clone slideshow

Toggle presenter mode

Restart the presentation timer

?, h

Toggle this help

Share Your Code!

Python Packaging without Complication

Package

Documentation

Source Hosting

Tests

CI

License

Contributing

we need

TL;DR

TL;DR

Terminology

Module

Saved Python code

Import Package

namespace / directory

Distribution Package

Shareable / installable

Source Distribution

sdist

Built Distribution

Wheel

Wheel

Wheel

Universal

Wheel

Universal

Pure Python

Wheel

Universal

Pure Python

Platform

History

History

Copy / Replace

History

Copy / Replace

PyPA

History

Copy / Replace

PyPA

PyPUG

Using cookiecutter

Layout

Code

setup.py

setup.cfg

MANIFEST.in

README.rst

setup.py (1/2)

setup.py (1/2)

setup.py (1/2)

setup.py (1/2)

setup.py (1/2)

setup.py (2/2)

setup.py (2/2)

setup.py (2/2)

setup.cfg

MANIFEST.in

README.rst

LICENSE

Tests

Docs

CI

requirements.txt

Also Common

(rst | md | txt)

Git init

Run tests

Commit

Services

GitHub

Travis CI

Git push

See builds succeeding!

PyPI

PyPI

Test PyPI

Save PyPI Settings