FOSSCON 2015: Writing Better Command Line Applications with Python – Slides, Notes, & Examples

This page contains my slides and some additional notes for my FOSSCON talk.

Slides

Further Reading

As I was preparing this talk I realized that I wouldn’t have nearly enough time to cover everything that would be useful to someone writing command line applications. Here is some more information about topics I chose to skip.

Virtualenv

Virtualenv allows you to isolate the Python environment and dependencies for your application from the rest of the system. You should develop your application in a virtualenv and package it so that others can install it in a virtualenv as well. Following the packaging instructions below will help you do that.

I use and recommend a tool called virtualenvwrapper to manage the virtualenvs on development systems.

Cookiecutter

If you develop an application that follows best practices and the recommendations in my talk you will end up with a fair amount of boilerplate and repetition. Cookiecutter allows you to create the directory structure and files needed for a new application based on a template repository. I have created a repo for Python command line apps using Click. In order to start working on a new project, install cookiecutter and then run:

cookiecutter https://github.com/tylerdave/cookiecutter-python-package.git

It will prompt you for information about your project and create the needed files in a new directory. You can then install the package in editable mode and start developing.

cd projectdirectory
pip install -e .

Logging

The logging module in the Python standard library provides a lot of functionality that you’d look for in logging like log levels and the ability to log to different destinations like file, stderr, etc.

I’d also recommend using Structlog if you intend to do any sort of analysis or reporting on your logs (and you should!)

Python Packaging

You should package your application so it’s easy for others to install and manage and you should share it on PyPI if possible / practical. To do this you should follow the directions in the Python Packaging User Guide (PyPUG). If you want more background or details on packaging you can also view my recent talk: PyOhio 2015: Python Packaging from Init to Deploy.

Example Code

The examples used in the talk are available on GitHub in the FOSSCON 2015 Example repo. You can install it and experiment by cloning the repo and then running this within the repo dir:

pip install -e .

Credits

The content of my talk was based upon my own experiences with writing command line applications with Python but the following sources helped me formulate it and remind me of some points I didn’t want to miss:

Armin Ronacher’s introduction to Click: Building Command Line Applications with Click

Mark Smith’s EuroPython 2014 talk: Writing Awesome Command-Line Programs in Python

Eliot Eshelman’s blog post: Shell Script or Python Script/Program?

PyOhio 2015: Python Packaging from Init to Deploy – Video & Addendum

See also: Python Packaging from Init to Deploy – Talk Notes

The opening minute of my introduction was cut-off but I was basically just saying that the goal of the talk was that even if you haven’t distributed a Python package before, you should understand the basic process and know where to look for more detailed help. Contact me via Twitter or email with any questions or feedback!

Addendum

I realized that I missed a couple points during the talk. Here are a few that I remember:

  • The point of having at least minimal tests and docs from the very beginning is that other people are a whole lot more likely to submit a pull request adding to the docs/tests than they are to create them for you wholesale. Make it easy for people to help you!
  • PyPI is also known as “The Cheese Shop.”
  • I like versioneer but there are definitely other options for single-sourcing the project version. A tool that takes a slightly different approach but seems useful is bumpversion.

PyOhio 2015: Python Packaging from Init to Deploy – Talk Notes

Python Packaging from Init to DeployA few years ago (circa 2009) I was working in the support department of a web hosting company. I had a number of tools and scripts that I used for day-to-day troubleshooting. Some of these were third-party packages that I installed via pip. Others were Python scripts that I had written myself or that my coworkers had shared.

We noticed that the ones I installed via pip were way easier to use. Every time I shared a script I had to tell the person using it to make the file executable and place it in their $PATH. I also had to remember to tell them to install any dependencies. I usually forgot to do at least one of those things so I had to help people get the scripts working every single time. Then once we got the thing working, someone would come up with an update and we’d have to manage sharing that update.

Clearly this wasn’t working.

I needed to find out how to make my Python tools as easy to share and install as the third-party ones I installed. I thought it can’t be that hard because look at all of the stuff that’s been uploaded to PyPI? I did some searching to find how to make my Python scripts into installable packages and got a ton of answers. The problem was that a lot of them were contradictory. It was pretty clear that I needed a setup.py with some information in it but beyond that I didn’t know what to believe. Should I use distutils? setuptools? distribute? I didn’t know who to believe so I finally ended up copying a well-known project and just replacing their code and package details with mine. It worked for my purposes but was hardly the most obvious process.

Fast forward to 2014. I was at a new job and again had the need to make an in-house distributable Python package. This time when I searched I found an authoritative result! The Python community, realizing that special attention was needed for packaging, created the Python Packaging Authority and they created the Python Packaging User Guide. My PyOhio talk is about what I learned from the guide and additional tools and tips I found along the way.

Links

The following are links to more details about topics from the talk:

Python Packaging Authority (PyPA)
Python Packaging User Guide (PyPUG)
Packaging Tool Recommendations
Wheel vs. Egg
install_requires vs. requirements.txt
PEP440
Versioneer

Slides

APIStrat 2014 Wrap-up

Last week I attended API Strategy & Practice conference in Chicago. This is a quick recap of my overall experience without getting too much into the individual presentations and panels.

First though, this trip included one of the more bizarre travel fiascos that I’ve experienced. I’ve had flights delayed or cancelled plenty of times due to weather, equipment problems, or even missing flight staff but this was the first time that my travel was delayed 38 hours because some guy decided to self-harm and set the basement of a major air traffic control center on fire. The extra time in Chicago proved enjoyable for the most part though so I shouldn’t complain too much.

As for the conference itself, I’d say there was nothing particularly mind-blowing but that’s probably because I’ve been trying to read everything I can about the space for the last couple years. There were some very good subtle points brought up that gave me more to think about though. I think the most compelling talks for me were on the not-typically-exciting subjects of documentation and testing. Mostly, it was a reaffirmation of a lot of what I’ve learned and have been trying to follow recently.

The real value of this conference was the “hallway track.” I got to spend a good amount of time talking with people who are actually working on a lot of the same problems that I am and I found those discussions especially informative. I came away inspired to update my development workflow to encourage better documentation and testing. I’ll write about this in more detail in the coming days at some point.

Hello again, world!

I figured it’s time to start the whole personal website thing again. “But, Dave, you’re a Python developer; why would you use WordPress,” you ask?!

Trying to be one of the cool kids I experimented with using static site generators like Pelican (and even Octopress!) I just really couldn’t get them to do what I wanted and in the end I found composing posts in Markdown was actually eating up more of my time (with formatting) than anything else. In the end I decided that I want to spend my time on problems that are interesting to me; mucking around with a new blogging framework is not one of those problems. I also decided to abandon posts from my old blogs as most of the technical information there is out-of-date at this point and the personal posts are downright boring. I may eventually export things and give the static site generators another go but here I am for now.