scripting the Open Build Service from scratch

(Well, not entirely.)

In my day job I do packaging for openSUSE. My responsibility is taking care of the Python package ecosystem. Right now we are in the middle of one transition (to a different way of packaging Python modules) and starting another one (converting the distro to Python3-by-default). For both of these things I need to modify lots of packages at once.

The bulk of what I do involves the openSUSE Build Service, an instance of OBS that runs our distributions. The Build Service is a thing that allows us to build dozens of variants of each package for each of the supported distributions and architectures. It consists of several independent parts:

• backend, which schedules and runs the actual build jobs on our server farm
• API server, which allows us, the users, to control the backend, makes sure that files go where the backend can find them, etc.
• web interface, which is a clickable client for the API. You can view packages, modify source files, configure build targets and so on
• osc, the command line tool which is another client for the API.

osc would be the natural starting point for scripting. Unfortunately, osc is also a horrible mess that grew organically alongside the Build Service. It works perfectly fine as a end-user tool, but it's unwieldy for shell-based scripting and difficult to use as a library because it doesn't have a consistent enough internal design.

The main reason for this is that UI code is interwoven with API calls and local non-OBS functionality. Also, osc tries to emulate a version control system and bases its OBS interaction on this model.

There is also osc2, a from-scratch rewrite with the intent to split the UI and logic into separate parts and impose some sort of order on the overall chaos. Unfortunately, it is a typical Generation 2 Project, deeply layered, overly complicated and overly generic. And also not nearly feature-complete, for the obvious reason that it was mostly abandoned before it got anywhere.

We are considering some serious refactoring of osc, and it seems possible to reuse the functionality while fixing the structure. We also want Something Usable Now(tm). Hence my work on a tiny library called "osclib". The idea is to make it a thin wrapper around the API and gradually modify osc the command line client to use osclib where appropriate.

Hard to say if this will ever go anywhere, but osclib is a nice exercise in understanding the OBS API. Also, when scripting things, you often don't need the rich functionality of handling every possible special case and command line switch.

osclib relies on osc for parsing the config file (and extracting login information from it), but does its own HTTP communication through Requests. At the moment, it has one class (to wrap the API server connection), about five functions in total, and can accomplish what I wanted to do in the first place: download a list of every spec file in the Tumbleweed distro, let me modify them offline, then create a branch project for each touched package and upload the modified spec file into it.

The hardest part was not actually writing the code, but reading osc sources and the very sparse OBS API documentation and figuring out what to do. For example, in order to upload a file, you need to create a "commit" through a separate API request.

So the fact that I can write a simple script that performs the mass update I mentioned is a big win :)

You can find osclib on my github. It is currently part of a forked osc repo, because osclib is written in Python 3 and the system install of osc is in Python 2 and at this stage it's too much effort to manage the dependencies properly. So instead it's the neighboring directory and you simply add it to your PYTHONPATH.