The X11 SECURITY extension from the 1990s

Security isn’t exactly a strong point of X11, and improving it is one of the main reasons why Wayland is such a vast improvement over X11. Just one of the many examples of X11 being inherently insecure is that keyloggers are entirely trivial on X11, because keylogger functionality is effectively built into it. Of course, this isn’t exactly news, and as Peter Hofmann details, there is an old X11 extension that adds somewhat rudimentary security to X11: the X11 SECURITY extension.

This extension is part of every X.org installation, but it hasn’t seen any meaningful work in a long, long time. What it does is allow you to do is set X11 clients as “trusted” and “untrusted”, where untrusted clients cannot interact with tusted ones. This provides some basic security – it actually prevents keylogging! – but only very basic, as Hoffman notes:

The thing is that it’s immediately clear that this extension — in its current state — is not the answer to “X11 is insecure”: You only have two classes, trusted and untrusted. That’s not enough. For example: When you run your browser as untrusted, you can’t simultaneously run some sandboxed program (Snap, Flatpak, …) in a meaningful way, because those two clients can spy on each other again. You want a proper per-client isolation instead.

Sandboxing plays an important role here. If you run programs “the traditional way” (i.e., full access to the filesystem and network), then an attacker can do all kinds of things and X11 keylogging is just one of a million concerns.
↫ Peter Hofmann

but it also happens to break a lot of things, and many applications simply don’t work with it at all. Oddly enough, Firefox has no issues with it, and will happily run in untrusted mode.

The biggest problem, however, is that untrusted clients only have access to exactly two other X11 extensions, which leads to a whole host of problems, like no scaling, broken keyboard layouts, no 3D acceleration, and so on. On top of all of that, it breaks clipboard functionality, as anything copied in an untrusted client cannot be pasted anywhere else.

As such, Hoffman concludes:

In its current state, I’d say the SECURITY extension is “somewhat useful”, but more work would have to be done. Both in X.Org and in the clients. You would have to come up with a new clipboard protocol, for example. And the list goes on. (See where I’m going with this?) It’s not that simple.
↫ Peter Hofmann

Since pretty much nobody adopted it when this extension came out in the ’90s, and it hasn’t seen much work since, the amount of work that would be required to bring it up to modern standards would be astronomical, and trying to get clients to adopt it would probably prove fruitless considering Wayland already exists, and offers all of the potential security benefits and then some. People often claim it would be “easy” to modernise X11, but just this one particular issue – security, kind of important – shows just how quickly the X11 house of cards comes crashing down if you try to do anything to drag it out of its ’80s and ’90s mindset.