PyOhio 2016: Fake It Before You Make It: Mocking Your Way to Better HTTP APIs

For this talk my co-presenter and I created a demo website. Most of the examples are hosted there:

http://betterapis.com/

Further Reading

Follow-up

I’ll post follow-up to any feedback or questions we get from the talk.

PyTennessee 2016: Writing Command Line Applications that Click

Sunday Feb 7, 2016, 1pm-3pm

Tutorial Repo and Installation Instructions:

https://github.com/tylerdave/PyTN2016-Click-Tutorial

Session Listing: https://www.pytennessee.org/schedule/presentation/117/

Follow-up:

I think the tutorial went well and I got some good feedback about the format (so I’m now splitting the tutorial runner into its own project!)

I really only had one issue come-up: The tutorial runner’s solve command overwrote unsaved work. This caused problems for a couple attendees who were keeping notes in the working file — something I hadn’t considered. I’ve since updated the command to prompt the user to confirm overwriting the file. In my new project I’ll do the same at minimum and may find better options for handling this.

 

FOSSCON 2015: Writing Better Command Line Applications with Python – Slides, Notes, & Examples

This page contains my slides and some additional notes for my FOSSCON talk.

Slides

Further Reading

As I was preparing this talk I realized that I wouldn’t have nearly enough time to cover everything that would be useful to someone writing command line applications. Here is some more information about topics I chose to skip.

Virtualenv

Virtualenv allows you to isolate the Python environment and dependencies for your application from the rest of the system. You should develop your application in a virtualenv and package it so that others can install it in a virtualenv as well. Following the packaging instructions below will help you do that.

I use and recommend a tool called virtualenvwrapper to manage the virtualenvs on development systems.

Cookiecutter

If you develop an application that follows best practices and the recommendations in my talk you will end up with a fair amount of boilerplate and repetition. Cookiecutter allows you to create the directory structure and files needed for a new application based on a template repository. I have created a repo for Python command line apps using Click. In order to start working on a new project, install cookiecutter and then run:

cookiecutter https://github.com/tylerdave/cookiecutter-python-package.git

It will prompt you for information about your project and create the needed files in a new directory. You can then install the package in editable mode and start developing.

cd projectdirectory
pip install -e .

Logging

The logging module in the Python standard library provides a lot of functionality that you’d look for in logging like log levels and the ability to log to different destinations like file, stderr, etc.

I’d also recommend using Structlog if you intend to do any sort of analysis or reporting on your logs (and you should!)

Python Packaging

You should package your application so it’s easy for others to install and manage and you should share it on PyPI if possible / practical. To do this you should follow the directions in the Python Packaging User Guide (PyPUG). If you want more background or details on packaging you can also view my recent talk: PyOhio 2015: Python Packaging from Init to Deploy.

Example Code

The examples used in the talk are available on GitHub in the FOSSCON 2015 Example repo. You can install it and experiment by cloning the repo and then running this within the repo dir:

pip install -e .

Credits

The content of my talk was based upon my own experiences with writing command line applications with Python but the following sources helped me formulate it and remind me of some points I didn’t want to miss:

Armin Ronacher’s introduction to Click: Building Command Line Applications with Click

Mark Smith’s EuroPython 2014 talk: Writing Awesome Command-Line Programs in Python

Eliot Eshelman’s blog post: Shell Script or Python Script/Program?

PyOhio 2015: Python Packaging from Init to Deploy – Video & Addendum

See also: Python Packaging from Init to Deploy – Talk Notes

The opening minute of my introduction was cut-off but I was basically just saying that the goal of the talk was that even if you haven’t distributed a Python package before, you should understand the basic process and know where to look for more detailed help. Contact me via Twitter or email with any questions or feedback!

Addendum

I realized that I missed a couple points during the talk. Here are a few that I remember:

  • The point of having at least minimal tests and docs from the very beginning is that other people are a whole lot more likely to submit a pull request adding to the docs/tests than they are to create them for you wholesale. Make it easy for people to help you!
  • PyPI is also known as “The Cheese Shop.”
  • I like versioneer but there are definitely other options for single-sourcing the project version. A tool that takes a slightly different approach but seems useful is bumpversion.

PyOhio 2015: Python Packaging from Init to Deploy – Talk Notes

Python Packaging from Init to DeployA few years ago (circa 2009) I was working in the support department of a web hosting company. I had a number of tools and scripts that I used for day-to-day troubleshooting. Some of these were third-party packages that I installed via pip. Others were Python scripts that I had written myself or that my coworkers had shared.

We noticed that the ones I installed via pip were way easier to use. Every time I shared a script I had to tell the person using it to make the file executable and place it in their $PATH. I also had to remember to tell them to install any dependencies. I usually forgot to do at least one of those things so I had to help people get the scripts working every single time. Then once we got the thing working, someone would come up with an update and we’d have to manage sharing that update.

Clearly this wasn’t working.

I needed to find out how to make my Python tools as easy to share and install as the third-party ones I installed. I thought it can’t be that hard because look at all of the stuff that’s been uploaded to PyPI? I did some searching to find how to make my Python scripts into installable packages and got a ton of answers. The problem was that a lot of them were contradictory. It was pretty clear that I needed a setup.py with some information in it but beyond that I didn’t know what to believe. Should I use distutils? setuptools? distribute? I didn’t know who to believe so I finally ended up copying a well-known project and just replacing their code and package details with mine. It worked for my purposes but was hardly the most obvious process.

Fast forward to 2014. I was at a new job and again had the need to make an in-house distributable Python package. This time when I searched I found an authoritative result! The Python community, realizing that special attention was needed for packaging, created the Python Packaging Authority and they created the Python Packaging User Guide. My PyOhio talk is about what I learned from the guide and additional tools and tips I found along the way.

Links

The following are links to more details about topics from the talk:

Python Packaging Authority (PyPA)
Python Packaging User Guide (PyPUG)
Packaging Tool Recommendations
Wheel vs. Egg
install_requires vs. requirements.txt
PEP440
Versioneer

Slides

APIStrat 2014 Wrap-up

Last week I attended API Strategy & Practice conference in Chicago. This is a quick recap of my overall experience without getting too much into the individual presentations and panels.

First though, this trip included one of the more bizarre travel fiascos that I’ve experienced. I’ve had flights delayed or cancelled plenty of times due to weather, equipment problems, or even missing flight staff but this was the first time that my travel was delayed 38 hours because some guy decided to self-harm and set the basement of a major air traffic control center on fire. The extra time in Chicago proved enjoyable for the most part though so I shouldn’t complain too much.

As for the conference itself, I’d say there was nothing particularly mind-blowing but that’s probably because I’ve been trying to read everything I can about the space for the last couple years. There were some very good subtle points brought up that gave me more to think about though. I think the most compelling talks for me were on the not-typically-exciting subjects of documentation and testing. Mostly, it was a reaffirmation of a lot of what I’ve learned and have been trying to follow recently.

The real value of this conference was the “hallway track.” I got to spend a good amount of time talking with people who are actually working on a lot of the same problems that I am and I found those discussions especially informative. I came away inspired to update my development workflow to encourage better documentation and testing. I’ll write about this in more detail in the coming days at some point.

Hello again, world!

I figured it’s time to start the whole personal website thing again. “But, Dave, you’re a Python developer; why would you use WordPress,” you ask?!

Trying to be one of the cool kids I experimented with using static site generators like Pelican (and even Octopress!) I just really couldn’t get them to do what I wanted and in the end I found composing posts in Markdown was actually eating up more of my time (with formatting) than anything else. In the end I decided that I want to spend my time on problems that are interesting to me; mucking around with a new blogging framework is not one of those problems. I also decided to abandon posts from my old blogs as most of the technical information there is out-of-date at this point and the personal posts are downright boring. I may eventually export things and give the static site generators another go but here I am for now.

Fixing the Honda Insight USB MP3 Sort Order

This surprisingly still-popular post was imported from my old blog because people seem to still be finding it useful.

Last week I purchased a 2010 Honda Insight and I absolutely love it. One of the features I was excited about was the USB port in the center console. Rather than waste an MP3 player / iPod by leaving it plugged in you can simply load your MP3s onto a cheap USB thumb drive and have access to a ton of music right from the stereo controls. Yesterday I loaded up an 8GB USB drive and quickly discovered my only complaint about the car: the seemingly random order in which MP3s are played.

At first I thought that they may be in alphabetical order, but that didn’t make sense as all of my file names are already prefixed with the track number. After a little research I found that the built-in MP3 player plays the files in the order in which they are actually stored on the drive (in the FAT)! I then thought there must be some option to change the sort order but I was sorely disappointed. This was the behavior of some of the first (cheap) MP3 players in the early ’00s — it is inexcusable in late 2009.

I found a forum thread discussing the problem where it was suggested that you could use a utility called FATSort to reorder the FAT to be in alphabetical order. This seemed like a viable work-around but the utility didn’t recognize my FAT32-formatted drive. I then tried another Linux utility also called FATSort but it reported errors on the drive and couldn’t continue.

Finally a solution:

I was finally able to find another utility that serves the same purpose: DriveSort. I verified that it works on Windows 7 so it should work with Vista and XP as well and it supports both FAT16 and FAT32 formatted drives.

Before following these instructions make sure to make a complete backup of any drives with which you’re going to be working.

Download and run the program and then do the following:

  • Click “Disk” -> “Open” and select the proper disk
  • Click the arrow on the “Sort current folder” button and check “Long name sort” and “Subdirectories”
  • Click the arrow on the “Save current folder” button and check “Subdirectories”
  • Click “Folder” -> “Sort”
  • Click “Folder” -> “Save”
  • Click “Disk” -> “Close”

Now the files should be stored in alphabetical order within the FAT so the Insight’s MP3 player should play them in the proper order.

I still think it’s ridiculous that you have to use this hack to get the player to work as expected but at least there’s this work-around.