Legacy and Systemd

28 Feb 2016

Writing something about Systemd has been on my TODO list for a while, but given how easily discussions about it turn into flame wars, I wanted to tread carefully. People have lobbed a lot of criticism at Systemd. Some of it is built on either false beliefs about how Systemd works, or a superficial understanding of how something else works, but much of it is valid. Unfortunately, both varieties are too often bundled with personal attacks, which tend to kill off any productive discussion before it even begins. I want to be clear on one thing before I begin: this post isn’t about Lennart. It also isn’t about PulseAudio, or any of his other work, or Redhat, or Canonical, or any particular social dynamic in the free software community. It’s about a piece of software, its design, and its implications for the rest of the ecosystem it inhabits. Hopefully it contains some insights that are generalizable as well.

I’m highly critical of Systemd. I do think there’s some good stuff. Examples:

So, what’s the problem? Well, if it takes off, we’re stuck with it. There was a post that showed up on Hacker News recently, where the author argued:

Just as all software contains bugs, all software is legacy. Software development is about solving specific problems that exist now and not about solving none-specific or future problems. Every few years, once a decade perhaps, there is a paradigm shift. Given enough time parts of the technological bedrock will have changed enough to render previous patterns obsolete.

The important thing is that we learn from that and we make sure the future legacy is easy to understand. Write clean interfaces, good tests, descriptive commit messages.

I’d add something to this: Stay focused. Don’t do more than you have to. Do one thing and do it well. Keep your interfaces small. Maybe clean implies small, but let’s be explicit about this. People sometimes call parts of this “UNIX Philosophy,” I just call it common sense or, good design. UNIX doesn’t have a monopoly on simplicity.

Oh, and one more big thing: make it easy to delete.

Systemd is not going to be easy to delete, for a few different reasons. Rich Felker, the author of Musl, has written some criticism of Systemd, and he makes one point about this that is highly relevant:

For 30+ years, the choice of init system used has been completely irrelevant to everybody but system integrators and administrators. User applications have had no reason to know or care whether you use sysvinit with runlevels, upstart, my minimal init with a hard-coded rc script or a more elaborate process-supervision system, or even /bin/sh. Ironically, this sort of modularity and interchangibility is what made systemd possible; if we were starting from the kind of monolithic, API-lock-in-oriented product systemd aims to be, swapping out the init system for something new and innovative would not even be an option.

(Emphasis mine).

I do want to spend a bit of time justifying the claim that Systemd is in fact going to be hard to delete. There are a few reasons.

Firstly, it’s monolithic, and non-modular. Lennart has argued that that isn’t true, but I don’t buy it.

The post claims you can disable lots of functionality, “Not totally unlike … the Linux kernel.” I actually downloaded the tarball and played around with this. Even passing every single --disable- flag, it still built about 60% of the repo. That’s nothing like the Linux kernel. Besides, the first thing anybody learns about the design of the Linux kernel is that it is monolithic. Lennart draws a distinction between monolithic and non-modular, but I don’t see one.

In fairness, I did this a few months ago, while his post is from 2013. Maybe his claim was more accurate at the time, I don’t know. However, if that’s true it supports my next point:

Even if it is currently designed such that you can easily swap around components, nobody does this with Systemd, and nobody really suggests doing it. A pertinent quote that I unfortunately can’t find the original source for: “That which is untested is broken.” If folks aren’t exercising those code paths, they’ll stop working. If Systemd isn’t a monolith now, it will be.

Secondly, its interfaces are far from simple. Take sd_notify as an example. As Rich points out, this and dbus are basically the only ways of correctly handling notifying the init system that a service is up and running. They both require specific knowledge of Systemd. sd_notify requires making a call to a C library, which comes with a whole mess of problems. For anything that’s not actually written in C, you now have to deal with the FFI. For Go, this means giving upon some of the really nice deployability that it otherwise has. For shell scripts, this means having to write wrapper programs in C (So Lennart’s myth #4 is no myth). For anything, if you want to cross compile you now need a C cross compiler, regardless of what language you’re actually using for the rest of your program. This is for one function call, that really shouldn’t need an FFI. Rich suggests an alternative (just write to a file descriptor and close it) that solves all this.

The sd_notify family of functions also does a lot more than this simple “I’m ready” notification. More stuff to get stuck to.

In fairness, there really just isn’t a way to achieve this without the daemons knowing at least a little bit about the init system, but the interface could be dead simple, and it isn’t anywhere near that.

Lastly, Systemd encompasses way more than just the system start-up and process monitoring that people think of when they think of an init system. Stuff like Journald has nothing to do with what Systemd is supposed to be doing. I haven’t looked at how separable that is from the rest of the system, but I argued earlier that it isn’t likely to stay that way.

Systemd bugs me because I worry about what it (and software like it) will do to the future of our ecosystem. It’s a popular target, but by no means the only problematic program. I wish we in the free software community were more concious of this kind of thing. I also wish those of us who are could argue with more civility.