Python Release Preparation

Posted on

I’m working on few projects mainly written in python and even if I work alone I try to adopt good practices all along and I’m trying to improve my flow related to the release part. In open source, it’s often said “release often, release early”, nothing can be perfect from the start, especially when you are are a junior working alone, many things can be improved but having feedback is important to make these improvements.

The release process should be more or less the same whatever the language used but here I’ll give the tools that I use with python. I recently read this article from OSS Watch about the release management best practice and I decided to use it as a base here. The checklist provided make a lot a sense even if a part of would be set up at the beginning of the project or from time to time such as the licenses used for the project, the README file, etc…

Now let’s have a look to the different steps:

  1. Update the documentation
  2. Check the documentation
  3. Check if all tests are passing
  4. Update the ChangeLog
  5. Bump the package version
  6. Create a distribution package
  7. Sign your package
  8. Create the checksum
  9. Test the installation of the package
  10. Upload your package for distribution

I’m not going to detail here how to start the project, I would recommend anyway to use a CookieCutter as it can be a quite cumbersome process. It’s possible to tweak/create your own template if you specific needs which make a perfect tool to start a project in no time.

Update and check the documentation

Once I wrote the code and have enough to bump the version of my code, it’s a good time to review the documentation. Anyway, a code without documentation won’t be used and even for me, when I go back to it after few weeks, I’m happy to have at least a good README file and at least up to date docstrings. Usually the documentation should be written at the same time of the code but let’s be realistic, we tend to be lazy and forget most of the time. If you are lazy, in order to not forget where you forgot a docstring or where you should add a dedicated page to explain something deeper you can:

  • create an issue
  • add a todo tag in the doc string that you’ll be searching using a grep command

In general I use a bit of both but I would recommend to always open an issue, it’s easier to track and if you have the chance of working with other people, it will be easier to comment on it as well.

Nevertheless, once it’s done, you can generate the documentation you’ve updated. When it comes to the API documentation, you have two options to generate it. At the moment, I recommend to use sphinx-apidoc command as it’s automatic and does a good job.

$ sphinx-apidoc -f -o <where/to/generate/the/api/documentation> <path/to/the/source/package>
$ make html

Now, check the log to see if you had any errors during the compilation and open the files locally to see if nothing is missing.

Check if tests are passing

If you are writing tests (which I encourage greatly), ensure that they are all passing. Personaly, I’ve still a lot to learn on writing good tests but to run my test suite I’m using the combination Py.test and Tox. You usually can’t run the tests on all the Python versions you aim to support but you can at least run the tests on your local version. Otherwise you can use some services such as Travis-CI or if you have good system admin skills, you can install your own Jenkins instance or now even using Gitlab.

$ tox
$ # or to run a specific version
$ tox -e py35

Alternatively, if you don’t use these tools, you can run the tests using:

$ python test

Update the ChangeLog

By default, the CookieCutter named the ChangeLog file HISTORY.rst. It’s an important file as it will tell the user what changed from what version to another and if possible using clear sentences such as described by Keep a ChangeLog. Take your time to review it, if possible name the issues and link them to your issue manager. Of course, don’t forget to commit your update.

Bump the version

It’s now time to bump the version according to the Semantic Version. The tool Bumpversion works very well for that purpose. You can configure it both through the setup.cfg or the bumversion.cfg file. In both cases I recommend the following configuration:

current_version = 0.0.1
commit = True
tag = True

search = version='{current_version}'
replace = version='{new_version}'

Don’t forget to update this file if you need to update other specific files. I recommend as well to sign your commit with your GPG key. This can be achieved through the ~/.gitconfig file

  signingkey = <key_id>

  gpgsign = true

  program = gpg2

Thanks to that, even the automatic commits and tags generated by Bumpversion will be signed.

Creating a bump is quite simple as long as you don’t have work in progress, if so stash them before bumping.

$ bumpversion patch

Package creation, test and upload

If you used the CookieCutter, you should have a working file. In order to create a package you just need to use the command:

$ python sdist

This will create tarball with the source ready to be installed. If you created a pure python program/library, you won’t need to compile anything on the host and in that case, it’s recommended to create a wheel package which will be faster to download and install:

$ python bdist_wheel --universal

We need now to generate the checksum in order for the users to check that what they are downloading hasn’t been tempered in any ways.

md5sum dist/<PACKAGENAME>-<VERSION>.tar.gz >> dist/checksum.txt
sha256sum dist/<PACKAGENAME>-<VERSION>.tar.gz >> dist/checksum.txt

All you packages will be stored in the /dist folder at the root of your package. Once this is done, we can upload this on the pypi test server. This will help to test the upload process and verify that there are no errors during the upload. Secondly, it will let you to test the download and installation process from a server as a normal user. In order to do that, you can configure the ~/.pypirc file:

index-servers = 
sign = true
identity = <gpg_id>

username = <username>
password = <password>

repository =
username = <username>
password = <password>

If you never registered your program, don’t forget to do it:

$ # On pypi
$ python register
$ # On the test server
$ python register -r test dist/<package-name>-<version>.tar.gz

We can upload on the test server the newly created package using Twine (force the use HTTPS).

$ twine upload -r test dist/*

We can now create a fresh install somewhere and create a virtual environment to work with.

$ # create a folder somewhere where to work
$ mkdir /tmp/my-installation-test && cd /tmp/my-installation-test
$ # Create the virtual environment with virtualenv or pyven depending on your
$ # installed version
$ pyvenv venv
$ # Activate the environment
$ source venv/bin/activate
$ # Install your program/library
$ pip install -i <package-name>

From there, you shouldn’t get any errors, just use the program/library to see if everything works as expected.

>>> # import the library to play with
>>> import <package-name>

If everything is ok, from that point you can perform the upload on the official (or unofficial if you have your own server) pypi server.

$ twine upload dist/*

You’re finally done.


Release more often. I tend to release only when I have a certain number of relevant patches (this excludes typo fixes) whereas I should release a new version with every relevant commited patches. I should follow more closely the Semantic Versioning in order to update the patch number or the minor versions.

I would like as well take advantage of Github API to manage the release as well there. At the moment the releases follow the tags that I push which is a good start, but it would be much better to have a proper release containing the source, the wheel, a checksum and a ChangeLog. I found an small article giving an idea on how to use the API and it exists at least one library automating this process.

My ChangeLog can be improved as well. I try to follow the Keep a ChangeLog rules that are very good but still, I think I can do a better job at that.

If the program/library starts to be used by other users/developers, then be able to generate packaging for the main distributions and submit them to be in the stable trees.