Show HN: Xlibe – A serverless Xlib (X11) compatibility layer for Haiku (github.com)
While there are some projects that implement partial versions of Xlib APIs on non-X11 platforms for compatibility (most notably, Tk), I don't know of any others sophisticated enough to run Cairo and GTK applications (albeit with a few hacks). So I figured this might be technically interesting.
As of now it just implements the base Xlib APIs, it doesn't implement XRender, GLX, XInput2, etc. (though it's more than possible, I just didn't see a need yet.) Cross-program interaction is also very limited, and there are plenty of X11 features that likely can't be implemented; but this works sufficiently well that GTK3 is now in Haiku's default package repositories, with GIMP and Inkscape atop it to boot, using this.
And for the fun of it, I did try to compile Xnest (the X.org server variant running on top of Xlib) on top of this; it crashed on startup in the keyboard handlers. However, that could potentially be because I didn't have the necessary data files and not because of anything missing in Xlibe; I didn't investigate too far (or it could be an actual incompatibility; Xlibe's keyboard subsystem is rather primitive and is most of the reason I had to patch GTK for full functionality.)
(A few more screenshots, including GTK and WINE running atop Xlibe: https://www.haiku-os.org/blog/waddlesplash/2022-01-10_haiku_...)
47 comments
[ 4.2 ms ] story [ 88.3 ms ] thread* Every X11 "Window" gets its own BView (which is itself vaguely analogous to a QWidget in Qt, etc.) and of course top-level "Window"s attached to the root get their own "BWindow" as well.
* All drawing goes through the Haiku drawing APIs. (This does create some artifacts in applications that expect X11 drawing isn't antialiased, like most Xt/Xaw applications, but it works tolerably well.)
* Since Haiku has no "display server" / "windowing system" divide, all operations related to both are directly handled in Xlibe. This creates some trickiness when applications request "window manager bypasses" in tandem with modality management, and also in decorations/borders handling, but it works well enough.
I think a better title might include mentioning what Haiku is, though 15 seconds of googling worked just as well. :)
As I noted to another commenter, this makes heavy and direct use of the Haiku APIs to do basically everything.
Haiku actually does have a full draw-call-forwarding remote desktop system, with an HTML5 client to boot, and you could run this through that, but it's kind of "experimental". For straight X11, using something like Xpra with a HTML5 client is probably what you want.
(I ask because, as a side project I've been working on something that lets me have a more responsive laptop by offloading resource intensive graphical programs to another machine, and streaming the UI back to me. I guess it could be made to run in a browser.)
https://www.phoenixframework.org/
What techniques are you using, is remote desktop or a new desktop application framework?
> That's because this isn't just any old stipple image, that stipple is actually the background of a full X server session running in your browser using HTML5 canvas.
Thank you!
However, that being said, if you wanted to do something similar or equivalent on some other platform, you could probably copy quite a lot of the structure, and then delete the actual implementation and replace it with your own. I restructured things a bunch in the early phases of working on this, and there's a lot of boilerplate code (e.g. macros.c, large parts of the event dispatcher) which you could reuse verbatim; though it's mixed in with the non-boilerplate code.
There were some other projects I ran into that looked like they were more generic, e.g. https://github.com/Abestanis/SDL2X11Emulation (also check the forks of that -- though I'm one of them, I experimented with it a bit before starting on this thing; but it's very incomplete and SDL2 really isn't designed for this), but none anywhere near as advanced as this is now.
I have nothing to do with Serenity and have never even used C++ but your code was easy to read and so the possibility just jumped out at me.
GTK will always owner-draw its buttons, so you don't get a real native Be widget. And it's clearly possible to do a good-enough X server integration as demonstrated by WSL, Quartz etc with minor complaints about clipboard syncing, decorations, having to run another daemon etc. After all that, what would you say is the main benefit to this approach?
> GTK will always owner-draw its buttons, so you don't get a real native Be widget.
Well, I think it might be potentially possible to write a theme that uses Haiku control-drawing APIs, just like for Qt (already there are some CSS files that make GTK look like Haiku, but Haiku itself has a control-theming engine, so it would be good to look into that. But not something that's on my radar.)
> And it's clearly possible to do a good-enough X server integration as demonstrated by WSL, Quartz etc with minor complaints about clipboard syncing, decorations, having to run another daemon etc.
Having used X11+WSL and X11+forwarding quite extensively (and XQuartz a bit), my complaints about that approach are ... far from minor. Everything both looks and feels like it has been awkwardly shoehorned in. Window modality, sizing, and positioning always runs into problems.
Now, some of that is just MS Windows being Windows, and X11 not having a good way to deal with some of the differences between X11 models and Windows, etc. But I think a large part of it, having looked at X11 internals (and as you can see, knowing more than a bit about them in order to write this thing and make it work) is that distinctions which are more minor at the base levels of the API become significantly bigger the further up you go -- the window manager vs. display server is probably the most obvious one here.
Xlibe can largely forget there even is a "display server / window manager" distinction and take calls that deal with one or the other and invoke the necessary Haiku APIs.
> After all that, what would you say is the main benefit to this approach?
Besides the ones outlined above, I would say it's much more possible to integrate with the rest of the system in broad ways. For instance, clipboard handling can pretend to do the strange set/get property dance in Xlibe for compatibility's sake, and then turn around and invoke the (much more sane) Haiku clipboard APIs internally. With a real X11 server, the clipboard handling is just the standard get/set property system, and you have to have something else (in the X11 server) that forwards clipboard updates in both directions. That can quickly get very complicated.
Another thing is keyboard handling. I don't bother creating X11 keymap structures at all in Xlibe; applications either get XK_* values (for e.g. Return, etc.) or UTF-8 text (which can be locale-converted if needed.) That means you can swap keymaps in Haiku and Xlibe will follow right along. I have no idea how much code that would wind up being in an X11 server port to deal with, but I know for sure it would not be fun at all, having dealt with X11 keyboard APIs from more than one vantage point.
Drag and drop would probably be another, but I didn't implement that in Xlibe (yet?)
I thought one time of bringing X apps to Windows in a source-compatible way by embedding an X server similar to Kdrive in a library that communicated with the application via a ring buffer or similar. It's cool to see similar work actually being done.
So this is a compatibility library? Not just a server that runs without a root window?
> but this works sufficiently well that GTK3 is now in Haiku's default package repositories, with GIMP and Inkscape atop it to boot, using this
That is absolutely fantastic:)
> And for the fun of it, I did try to compile Xnest (the X.org server variant running on top of Xlib) on top of this; it crashed on startup in the keyboard handlers
I don't have a Haiku box handy to test - is Xephyr any easier/better? It seems to be the "modern" option.
Correct!
> I don't have a Haiku box handy to test - is Xephyr any easier/better? It seems to be the "modern" option.
Both Xnest and Xephyr are both "backends" for the main X.org server, though at two different "levels" and with two different goals. The main technical difference is that Xnest runs on Xlib, and Xephyr runs on XCB -- which is a much lower-level library that more or less deals with the X11 protocol directly, as opposed to having a slightly abstracted view of things as Xlib provides. (Modern Xlib is itself implemented on top of XCB.) So, since Xlibe implements Xlib, Xephyr isn't an option.
It's possible to integrate with the rest of the system much more smoothly than a full server will allow. GTK apps, for instance, feel like second-class citizens on basically anything except maybe GNOME (but do they even feel "at home" there...?) so it just didn't seem worth the effort to write a full GTK port vs. writing this, instead.
I discussed more of the "why" on the forum posts introducing this last winter: https://discuss.haiku-os.org/t/xlibe-an-xlib-x11-compatibili...
> And if you don't have xcb, does that limit what applications can run with this library? I think I remember seeing programs that depended on XCB directly.
There are a few (I believe Qt on Linux now uses XCB instead of Xlib), but not enough that it really matters (we have a native Qt port on Haiku already, anyway, e.g.)
Too late, I did, and it's already being used "in production" to power GTK on Haiku!
> It might work for some clients
Well, GTK was the one client I really cared about, and it works for that, so...
It should work for Tk, too, but there's one really annoying bug somewhere in Xlibe which is causing Tk problems; and I didn't figure out what it was in the Tk codebase (this is doubly strange because Tk itself has Xlib compatibility layers to run on other OSes, you would think they would have excised any such quirky behavior.) The "AndroWish" developer said he'd try to take a look and figure out what was going on, at some point.
> you'll get stuck in the same place that every other re-implementation of X11 got stuck at
So far as I've been able to tell, this is now the single most advanced Xlib compatibility layer ever written (well, that's publicly available and locateable, anyway. I wouldn't be surprised if there's more advanced ones behind closed doors.)
At least, I don't see any others powerful enough to run GTK, do you? So I think I've not gotten stuck at that same place then!
This reply makes no sense, I would greatly advise against falling for the sunk cost fallacy. If all you care about is GTK, then you could have just done a native port of GTK. That would actually be a lot easier than trying to reimplement every X client library. With GTK4 it should be even easier because the backends were simplified and a lot of the weird X idiosyncrasies were removed, and you might actually have a chance to get hardware acceleration to work without having to go through the pain of dealing with GLX. The problem with Xlib is that's is not just the one library, it's actually about 30 libraries. So like I was saying earlier, it's cool that you have one, but the hard and annoying part that isn't worth it is doing the other 29. Nobody has gotten that far, and if you do get that far, the best way to do it would be to copy a huge amount of code. So the goal here can't really be reached in any practical terms, unless you just want to stop here.
Haiku should want some form of generic compatibility, and this is one way to do it (I would have been lazy and used the embedded X server approach for it if I wanted to do it for some OS, but waddlesplash apparently has different tastes). Perhaps one day someone will want to do the work for a 'native' GTK4 port, but that does not degrade the important of this work for Haiku.
I get the desire to take shortcuts to get things ported quickly but that will not actually help in making these applications properly usable.
I will say that I don't think it's the best path for an alternative OS, and other AltOSs might wish to pick a different path. IMHO, Reimplementing POSIX is a mug's game, a lot of work, while Linux will always be ahead and POSIX isn't much good anyway. Microsoft showed the way with WSL2. This would be a better path for alt OSs in general:
A. Implement the OS with almost no compromises. Ignore POSIX/X11/etc. if they don't match the vision.
B. Port/Create a few application to show and use the benefits of the OS and whatever it brings to the table.
C. Implement an equivalent of WSL2 which should give broad compatibility at the relatively low cost of keeping up with a Linux kernel, given the kernel's rather good backward compat record we wouldn't have to be very prompt with porting new versions.
At which point users would have a good reason to use the OS (B.) and no practical shortage of apps (C.) even if the experience with these apps would be somewhat lacking. One can then concentrate on improving the OS or on porting the most used applications. This acts like a Cuckoo - in the beginning, there'll a mostly Linux userland with a few AltOS apps. Eventually one ports enough so that the system is really AltOS with a few Linux apps on the side.
Maybe GTK4 would be easier to build a Haiku backend for (as was done for QT), but that doesn't help anybody run GTK3 applications on Haiku does it.
[1] xlib is kind of terrible IMHO; it adds mandatory abstractions that are superfluous at best and in many cases (again IMHO) harmfully force synchronicity on the asynchronous protocol that is X11. Lots of xlib based programs wait for a response when they have other things they could be doing, and the result is poor experiences blamed on X, when xlib's fault. sigh
Was fixing that the intent behind libxcb?
(honest question, my understanding of X stuff is limited, I just feel to recall Awesome depends on libxcb instead of xlib and read some vague blurb about it back then)
Yes.
Like I noted in another comment, XCB is basically just a loose wrapper around the X11 protocol. It is indeed a lot more verbose than Xlib, and modern Xlib is actually written on top of XCB, so you can use the two in the same application on the same X11 context if you really need to for some reason.
LOL! Do you really think I woke up one day and said "hmm, I need a GTK port. I know, I'll reimplement Xlib!" without even taking a look at the GTK codebase? Of course I looked at the GTK codebase, and so did a number of other Haiku developers, and we came to the conclusion that writing a native Haiku backend would be a ridiculously unpleasant task (as one other Haiku developer pointed out, GDK backends are typically > 1 MB of source code, and in a dialect of C that I am, uh, if I put it politely, not a fan of.)
It turns out that writing an Xlib compatibility layer was, in the final analysis, a far more expedient and efficient way to port GTK. If I went the long way around, I would have had to write probably over 100k lines of code across GTK, Cairo, and who knows how many other projects (I stopped counting how many dependent libraries I built in the first stages of this attempt.)
Meanwhile, Xlibe does the job with only ~7000 SLOC. That's compared to ~23000 SLOC in the "gdk/x11" folder alone, not to mention all the other X11-related files scattered around the GDK and GTK source trees, or all the ifdefs throughout the codebase, and all the other projects that GTK depends on which interact with the X11 layers in GDK. There is a reason it took literal years to get GTK on Wayland natively...
> With GTK4 it should be even easier because the backends were simplified and a lot of the weird X idiosyncrasies were removed
I find that hard to believe given how deeply some of those X idiosyncrasies are embedded.
But anyway, supposing you're correct: Why should I care about GTK4? Even GIMP of all projects, the (more or less) flagship GTK application and one of the primary reasons a GTK port is generally desirable, has decided that for now, a GTK4 port is "out of scope" (no, really [1]) at least for the immediate future. So even if a GTK4 port was more technically feasible, why exactly would I care if none of the software I want runs on GTK4?
Oh, and let's not forget: with an Xlib compatibility layer, I can also get GTK2 and then whatever applications are still stuck on that; plus plenty of other X11-only things which otherwise I wouldn't be able to get.
[1]: https://gitlab.gnome.org/GNOME/gimp/-/issues/6440
> you might actually have a chance to get hardware acceleration to work without having to go through the pain of dealing with GLX
I didn't yet try to implement GLX but it looks more than feasible. Sure, it's not the most pleasant of APIs to deal with, but implementing it doesn't look so bad.
> The problem with Xlib is that's is not just the one library, it's actually about 30 libraries.
If you mean libraries like Xt, Xaw, Xmu, Xpm, etc., those just run on top of Xlib itself, they don't interact with the X11 protocol directly, and so I can and did build and use those unmodified directly from X.org. For all they know they are running with a strange X11 server that provides an odd but completely valid featureset. Xeyes, which you can see a screenshot of in the README, uses Xt and Xaw, so clearly those work already in this environment.
> Nobody has gotten that far, and if you do get that far
I think I have gotten "that far." I can run Xt/Xaw applications (albeit with a few font rendering glitches, though I just didn't bother to patch that), I can run GTK, and sure, I didn't implement every single drawing mode or every quirky feature, but I don't really care about running Motif, so why does that matter so much?
> So the goal here can't really be reached in any practical terms, unless you just want to stop here.
How about this: You tell me some X...
No. I would also greatly suggest against making these wrong assumptions. But from this comment it actually does sound like you didn't look at the GTK4 codebase.
>It turns out that writing an Xlib compatibility layer was, in the final analysis, a far more expedient and efficient way to port GTK.
This analysis is extremely incorrect and wrong. I'll get more into the details.
>If I went the long way around, I would have had to write probably over 100k lines of code across GTK, Cairo, and who knows how many other projects (I stopped counting how many dependent libraries I built in the first stages of this attempt.
Implementing the full X11 protocol with all the extensions and all the quirks in the server is much, more more than 100k lines. Also this entire sentence is extremely incorrect and I'm very confused as to what you were trying to do, you don't have to do anything in Cairo or Pango because those support client side rendering and have supported that for a really long time now. I assume you were basing this analysis off an ancient Haiku port or something that was trying to do things the old way? That stuff is obsolete.
>Meanwhile, Xlibe does the job with only ~7000 SLOC. That's compared to ~23000 SLOC in the "gdk/x11" folder alone
This comparison is complete nonsense. Please don't sink yourself to the level of doing this. The GDK X11 backend includes support for a bunch of extensions that you didn't implement. So no, it's not "doing the job" at all. IIRC if you look at the GTK4 backends for the more modern windowing systems (Wayland, macos) those have both shrunk to around 13000 SLOC. A native Haiku backend would probably be even shorter if you dropped all the optional features and only implemented the bare minimum, I bet you could do it in... around 7000 SLOC. There is nothing magical about any particular approach to this, it's all the same code you have to write in either case. Draw a box here, render some text here, translate these input coordinates, etc. You can't avoid this by implementing Xlib or doing any other approach.
>and all the other projects that GTK depends on which interact with the X11 layers in GDK.
Well actually those will still break with your approach if they link against the other libraries or depend on some X server functionality.
>There is a reason it took literal years to get GTK on Wayland natively...
Actually it took years because the protocol was still being worked on while the backend was being developed.
>I find that hard to believe given how deeply some of those X idiosyncrasies are embedded.
Well it's true, they were removed in GTK4, the entire rendering pipeline was rewritten.
>I didn't yet try to implement GLX but it looks more than feasible.
You could get it to work but it's not going to be feasible without also hacking the mesa implementation or re-implementing several parts of mesa yourself along with the GLX extensions. IMO this is not a good way to spend your time, please don't waste your donors' money on this. If you have a working EGL backend already you can avoid all that and just use the EGL support that GTK has already.
>But anyway, supposing you're correct: Why should I care about GTK4?
I don't know, you tell me. When you say "GTK" to me that means all versions of GTK that are currently in use by applications. If all you care about is GIMP then I would still say a better approach is to do a native backend because then you can also just skip everything that isn't used in GIMP. It's the same approach really.
>Oh, and let's not forget: with an Xlib compatibility layer, I can also get plenty of other X11-only things which otherwise I wouldn't be able to get.
Well no, this is also wrong, to get the full...
I gave it a cursory glance, but indeed I didn't look at it much in depth, because all the big GTK-based applications don't yet run on it, so it was a moot point.
> Implementing the full X11 protocol with all the extensions and all the quirks in the server is much, more more than 100k lines.
Sure, but I have already stated, multiple times, that we neither need nor want to implement all the extensions and quirks.
> you don't have to do anything in Cairo or Pango because those support client side rendering and have supported that for a really long time now.
GDK's backends presume Cairo has support for native "surfaces"; e.g. the GDK X11 backend uses Cairo's Xlib interfacing layer to draw stuff to the screen. I didn't look too closely, but the other backends seem to make similar usages on Windows and macOS. So I would have probably had to either revive the BeOS interfacing code in Cairo, or would have had to do lots of software copies in a Haiku backend of GDK in lieu of a Cairo interface.
Pango indeed doesn't deal with native "surfaces", so that wouldn't be a problem.
> I assume you were basing this analysis off an ancient Haiku port or something that was trying to do things the old way?
Nope. There may have been some work on GTK 1 for BeOS or something like that, I can't recall, but I didn't go looking for it, much less analyze it.
> The GDK X11 backend includes support for a bunch of extensions that you didn't implement.
For now. Some of those extensions (like XSync and XInput2) are probably worth implementing. This also ignores all the other X11-related code scattered throughout the GTK/GDK trees, anyway.
> A native Haiku backend would probably be even shorter if you dropped all the optional features and only implemented the bare minimum
Okay, but that's again GTK4, and totally ignores GTK2 and GTK3 which quite a lot of applications still use, and the entire point of a GTK port is to run GTK applications. Not much use in porting a version of GTK that very little supports.
> You can't avoid this by implementing Xlib or doing any other approach.
No, but I can decide that the GTK codebase is so far outside the realm of my and the other Haiku developer's programming tastes that we'd quite literally rather write an X11 implementation than deal with it. Even the minimal time I had to spend with the GTK code in a debugger while fixing incompatibilities in Xlibe was more than enough, thanks.
Furthermore, GTK is entirely C while Haiku's APIs are C++. Is GTK ever going to accept a C++ backend upstream? Almost certainly not. So if I am going to have to maintain this totally outside GTK, forever, why not do it in a way that's much more resistant to GTK changes?
And, again, there's more than just GTK here. Someone took Xlibe and apparently got some audio plugins from Linux (LV2, I think) to work with minimal code changes.
> Well actually those will still break with your approach if they link against the other libraries or depend on some X server functionality.
Again: which other libraries? You mean like Xinerama or something? If something actually needs Xinerama, or any of those other libraries, then we can implement those, too. (So far, those have turned out to be entirely optional, I haven't yet tried to build something that couldn't just disable those features.)
> Well it's true, they were removed in GTK4, the entire rendering pipeline was rewritten.
Okay. What about input events? Does GTK4 still have keysym values that map 1:1 to X11 keysyms, or did they break with that?
> Actually it took years because the protocol was still being worked on while the backend was being developed.
It's my recollection that Qt had a working Wayland backend long before GTK did. Am I incorrect?
> You could get it to work but it's no...
Look, a little snark never hurt anybody.
The fact is that you are being extremely pessimistic about whether this will work at all, when in fact it is already being used "in production," and most of the other things you are saying are not possible or very difficult I have already investigated and made similar assessments to the ones I made before I started this.
So all evidence points to you being largely wrong, and your supreme confidence that you are correct despite all the evidence to the contrary makes it quite attractive to have a bit of fun while trying to demonstrate where you are incorrect.
That's nothing new in open source, or even software development in general, where much worse like actual insults and derogatory comments are much more an unfortunate norm, and toxicity is often the rule of the day. Plenty of stuff that goes on (or at least used to) on the Linux kernel mailing list would get you warned and then banned from Haiku's discussion channels. But at the same time, there is a place for criticism, even strong criticism, in any healthy discourse.
Does something like Gtk1.2[0] (which AFAIK does more heavy use of xlib than modern GTK) work? How about Motif?
[0] http://runtimeterror.com/pages/badsector/gtk12/
Hard, as I mentioned to other commenters, Xlibe makes heavy and direct use of Haiku APIs everywhere to do just about everything. However...
> It'd be neat if it was possible to use this not only to target Haiku but also other window systems like Win32 and Cocoa.
Using Xlibe as a starting point to write an equivalent library for Win32, Cocoa, or even Wayland would probably save you a lot of time, effort, and hassle in that a lot of Xlib general architecture, some behavioral oddities, etc. are already worked out in Xlibe as a matter of course. I'd be happy to guide others through what that process might look like if there was interest.
> Does something like Gtk1.2[0] (which AFAIK does more heavy use of xlib than modern GTK) work?
I didn't try it, but modern GTK actually makes pretty heavy use of Xlib; and Xt/Xaw do as well, which also work, so I imagine it would; or if it didn't, improving Xlibe so that it did would not be too hard.
> How about Motif?
Motif relies heavily on strange and implicit behaviors of X11. I was advised by one of the Emacs developers that it was probably inadvisable to try and get it to run under this. However, I did build it and it at least starts and does display things, but there are drawing artifacts and events do not work quite right.
So, it's probably theoretically possible, but I don't think it's really worth spending the time and effort in doing so. It is probably a better use of time to refactor/rewrite whatever software still uses Motif as its UI toolkit to use something else instead...
To my eye, it looked like it would be fairly straight-forward to port this over to any other graphical API that provided the same basic concepts ( windows, views, points, lines ).
You would have to modify code all over the place but the code looks easy to identify and easy to map to alternatives.
Am I looking at this too simplistically?