Systems Software is Only as Irrelevant as we Let it Be
18 years after Rob Pike's Systems Software Research is Irrelevant, much has changed, yet the end result is, for the most part, painfully familiar.
Unix was dead by 2000. Good riddance.
This talk was delivered at a time when the fall of all direct descendants of Unix was imminent. The writing was on the wall: The BSDs had stagnated. The rapidly improving performance of small Windows systems was pushing IRIX and HP-UX out of their respective markets. AIX, much like most things IBM has done over the last 25 years, existed only because of legacy systems. Linux's resemblance to Unix is only skin-deep; for better or worse, Linux and the GNU userland show not even an inkling of the Unix design philosophy.
There is little question about why Unix died (I'll get to Plan 9 in a moment). By all accounts, Unix was incredible at what it was designed for. However, The WWW came with a bundle of new challenges, and Unix didn't have the solutions. Lo and behold, an OS originally designed around character terminals strapped to mainframes didn't play nice with bitmapped displays and networked microcomputers.
Paintings are made with brushes.
I don't think Plan 9 was a failure. In fact, I think it was one of the greatest successes of systems software since Unix. It was a legitimate masterpiece of software architecture. It was original, it drew inspiration from the old but didn't get caught up in past pretentions. There's a reason it's used as an example piece in universities. However, Plan 9's beauty and its commercial failure are tied together at the hip.
Sure, the OSes with market dominance aren't works of art, but they were never meant to be. Windows and Linux are tools for doing work. DOS was cheap and easy to understand. Unix and Plan 9 were anything but affordable, and Unix certainly was not intuitive by the early 90s. Linux might be just as unintuitive as Unix, but at least it was free. There may be merit in a beautifully-designed system, but if the beauty comes at the expense of usability or performance, then that system might as well be a painting on a wall- beautiful, interesting, maybe even inspiring, but not something I would ever use as a general-purpose tool.
There is merit in the art of computer programming. I can appreciate a design that is elegant, original, and beautiful (I can even appreciate Pike's own works of art- I use Sam every day). However, Michelangelo didn't paint the Sistine Chapel using another painting. He painted the Sistine Chapel using a paintbrush- a tool. People create art using tools, and tools are a prerequisite for the creation of art. It's hypocritical to scorn the creation of the tools, then bemoan the loss of the art.
Pike's words still ring true, mostly.
Besides a JS interpreter, what really makes Chrome different from Netscape in 2000? TCP/IP has only truly evolved due to necessity, replacing SSL with TLS. Compare Windows XP, which was released in August 2001, to Windows 10. What has changed, outside of aesthetics? The introduction of PowerShell, a better search tool in the start menu, and... 2 gigabytes of overhead? What we see today is a variation of Pike's "industrial responses to interesting but technically flawed pieces of systems software." we see layers of features, pseudo-features and hacks growing on top of each other, without anyone ever removing or replacing the underlying infrastructure. User-facing pieces systems software (that would include operating systems, network and graphical engines, etc) has stagnated to the point where we can't even conceive of replacing them, even when they're 40 years old and do nothing but force us to kludge over them at every turn. Systems software isn't an easy money grab for a startup, nor does it make a big corporation's shareholders happy. Rebuilding from the ground up means long-term investment. It means putting faith in people over protocol.
Nearly two decades after Pike's talk, little has changed. In fact, it's probably gotten worse. My conclusion on this front is much the same as Pike's- good luck getting a corporation or university to give money to a groundbreaking user-facing systems software project. If they do, good luck getting the autonomy to write something that's actually unique or better than its predecessors. As a hobbyist, I (and the other dailyprog folks, and you too maybe) have the chance to try stuff without being handed a rule book describing every last detail of how things "have to be."
Go isn't helping.
Given his preaching of the "art" of systems software, I find it ironic that Pike is largely responsible for Go. According to Pike, one of the last realms where real innovation and creativity could be found was in programming languages. Go takes the two things that make C immensely useful (its granularity of control and stripped-down runtime), throws that to the wind, and replaces it with an inflexible concurrency model, while inexplicably keeping around the weak, inexpressive, and unhelpful type system. C didn't have an opportunity to benefit from the progress that has been made in type theory over the last 30 years. Go doesn't have an excuse. In Pike's own words, "narrowness of experience leads to narrowness of imagination." Go is the very antithesis of imaginitive, artistic software, yet it's also lackluster as a tool, with poor dependency management and lack of metaprogramming capabilities.
Systems software research is not irrelevant.
Something I've come to realize while working in industrial automation is that, in many ways, systems software may be more relevant than ever, although it might kinda smell funny. I develop an operating system with unique properties that make it highly suitable for our products. Embedded operating systems, and software for embedded systems in general, are flourishing and the amount of fascinating new and unique software being written in these realms is incredible. Protothread, an embedded coroutines library, was a game-changer in embedded systems and a contemporary of a huge uptick in the popularity of coroutines in various libraries and language syntaxes. FreeRTOS, Contiki, ChibiOS, RIOT, and others are real pieces of systems software, each with active developers, real world use cases, and new and unique approaches to the problems they each solve. They certainly are not irrelevant- odds are, you use at least one of them every single day without even knowing it. That's the thing about modern systems software research and development- if it does its job right, you never even know it exists. If systems software appears irrelevant to the people outside of the industry, that's a good thing. It means we're doing our job.
We often fail to stand back and assess the foundation of the things we create, and the result is that those foundations deteriorate without ever being repaired or replaced. Even if you aren't doing it for the betterment of the craft, be willing to push boundaries and reject the "standard way of doing things" for the sake of the end product. Systems software research is only as irrelevant as we let it be, and the art of computer programming was never gone in the first place.