Note that the results will differ between compositors, different GPUs, and different configurations. This is somewhat less the case with X11 since there is only one X server implementation (that Linux desktop systems use, anyhow.)
I think there may still be the issue that many compositor/GPU combinations don't get hardware cursor planes, which would definitely cause a latency discrepancy like this.
Beautiful work. Could it be worth repeating the experiment with the monitor running at a very low refresh rate, i.e. 30hz? If Wayland is always a frame slower than X11, it should be much easier to observe.
The various wayland compositors all have huge problems with keyboards, mice, and other input devices because the wayland spec and reference implementation(s) chose not to support such things. So each wayland compositor picks a different solution. Popular ones are libei and libinput but often there's just no support for "advanced" mouse/keyboard inputs at all (looking at you, weston). And you can never tell if a particular piece of linux software will work on any given wayland. The fragmentation is terrible. On X with the strong X11 reference that actually implements everything you can be sure something that works on one will work on another.
Hell, it's been 12 years and not a single wayland compositor supports screen readers for the visually disabled yet. They are toys.
This is why I ultimately ended up on Windows after switching from macOS and focusing my free time development work on improving the power user/developer experience on Windows with a tiling window manager ecosystem.[1]
One of the really cool things that has come out of this is seeing how people are adding accessibility workflows on top of the ecosystem. There is one person who controls their entire desktop via the tiling window manager using voice control![2]
Windows has added several things with the newer WDDMs to help alleviate compositor problems: flip model, overlays, etc. They've definitely identified the problem areas and are trying to correct them. It's been quite the turnaround since Vista.
I'm told they left out color calibration support, too (as I read it, there's no way for calibration apps to control, or even know, what compensation is applied when they're trying to measure things). Oh, and client side window decorations still around, giving up on the chance to fix an ancient security hole.
So no good for games, no good for professional graphics, no good if you don't see well... basically no good if you're any different from the people who hacked it together.
But, hey, with any luck they cut down on the screen tearing that I've never noticed.
If you control what goes into a distribution, you can not drop support for stuff that's working and already implements those needs in favor of whatever you pull out of your butt.
If not now, soon. Gtk itself is entirely dropping X support. And it's not their fault of wayland is so incomplete but I do wish people like them, and distro managers, would realize that the waylands are not actually replacements for X yet. And they might not ever be if you're visually impaired.
Color management support, though taking a long time, is moving along[1], and compositor implementations have recently started appearing. Device calibration support is a different question, but a much more specialized one.
It's unfortunate that colour calibration took a while to be fully implemented (xx-color-management-v4 landed on most compositors half a year ago I think, though LTS distros may need to wait longer to get it).
That being said, thanks to Nvidia I never got colour calibration to work right in X11 either, so I have no horse in this race. Would be cool to finally get HDR working for the first time, but I don't know if that'll ever happen on Linux with Nvidia hardware. Guess I should keep dual booting for HDR content until either Windows 10 dies off or I get new hardware.
I do actually notice the lack of tearing in Wayland, especially under heavy load. Used to annoy me to no end to see my windows tear when just dragging them across a 1080p screen. I don't know if it was an Intel driver bug (I tried all the config settings) or something X11 specific, but Wayland did finally fix the tearing issues I had.
I haven't noticed any problems with colours in either Gnome or Gamescope (except for the lack of HDR, of course, but that's also true on X11) so whatever is causing issues for you seems to be DE specific. Looks like we both have issues other people never encountered, that's what makes graphics stacks so impossible to debug and code for.
Wayland is an extensible protocol. The core is literally just "here a buffer", but it has several extension in various phases that can greatly extend its capabilities. This way, a Wayland compositor can be used as a kiosk screen for a single program, or be a full-blown compositor+window manager, all relying on standard Linux interfaces for the display part.
Color calibration can absolutely be retrofitted into this (versioned) protocol, and there is work ongoing.
> there's just no support for "advanced" mouse/keyboard inputs at all
It feels like you're probably blaming the wrong people here. You should look at the companies that make this peripherals that don't also offer you linux drivers.
You say this, but have you seen the quality of the drivers they produce on platforms they ostensibly support? You do not want vendor mouse/keyboard software on any platform. Instead, improve the USB class drivers so that you don’t need third-party drivers.
I agree with you on general drivers, but what generic devices does Linux still lack? Controller HID support seems pretty universal, as do all manners of touch screens/touch pads, drawing tablets, and just about anything resembling a keyboard.
The only hardware I know for sure is lacking is stuff like Stream Decks, but that kind of hardware is difficult to generalise for.
Plus, most working hardware that doesn't fit a standard HID protocol already has drivers in the Linux kernel written by manufacturers.
Nope. Because I don't mean special hardware. I mean using normal hardware in advanced ways: like sharing mouse and keyboards between computers. Something that's worked perfectly in X for 20+ years but still doesn't really work with the waylands even if the wayland compositor in question supports libei (which most do not).
There are definitely a lot of missing protocols when it comes to input though - wl_seat has pointer, keyboard and touch input support, but where are gamepads and wheels?
Has implications for competitive FPS gaming on modern Linux — one more problem to fix. For example "The Finals" allows Linux players, but imagine having this much input delay and having to revert to classic x11 to play games, lol.
It's actually just cursor latency specific to the windowed environment you're running. From what I've experienced (with a 4060 Ti) it doesn't seem to impact FPS games at all.
I haven't tried any games that use a cursor with Wayland yet so I don't know if it would have an impact there.
I think it has to do with whether or not the game in question is reading the mouse device directly (e.g. through SSL) or via the compositor. If it's reading the device directly it stands to reason that there would be less latency.
I have trouble believing that 6.5ms in increased latency would be perceptible to any more than a fraction of a percent of the most elite gamers. Most the people claiming that this level of difference is impacting their gameplay are victims of confirmation bias.
You may think 6.5 ms of input latency is imperceptible. But combine it with the rest of the stack (monitor refresh rate, local network latency, RTT between client and server, time for server to register input from client and calculate “winner”), and it becomes the diff between an L and W. In the case of pros, the diff between a multimillion dollar cash prize and nil.
I would postulate that 100% of professional (i.e. elite) competitive gamers would be able to tell the difference. See this old touchscreen demonstration: https://www.youtube.com/watch?v=vOvQCPLkPt4
It doesn't need to be perceptible to cause a difference in a game.
Suppose two players notice each other at the same time (e.g. as would naturally happen when walking around a corner in a shooter), first to shoot wins, and their total latencies are identical Gaussians with a standard deviation of 100ms. Then a 6.5ms reduction in latency is worth an additional 2.5% chance of winning the trade. Maybe you won't notice this on a moment by moment basis, but take statistics and its impact should be measurable.
In ELO terms a 2.5% gain in win rate is around a 10 point increase (simplifying by assuming that single Gaussian is the entire game). That's small, but if you were a hardcore player and all it took to raise your ELO by 10 points was using a better monitor/mouse/OS... why not? Doing that is cheap compared to the time investment required to improve your ELO another 10 points with practice (unless you're just starting).
Also, I think you'd be surprised what people can perceive in a context where they are practiced. Speed runners hit frame perfect tricks in 60FPS games. That's not reaction time but it does intimately involve consistent control latency between practice and execution.
> Suppose two players notice each other at the same time (e.g. as would naturally happen when walking around a corner in a shooter)
This is not true for third person games. Depending on a left sided or right sided peek and your angle or approach, players see asymmetrically.
For example, Fortnite is a right side peek game. Peeking right is safer than peeking left as less of your body is exposed before your camera turns the corner.
I believe distance also plays a part in the angles.
Yeah, network latency and client side prediction and accuracy will also play huge roles. The actual distributions will be very complex, but in general reacting faster is going to be better.
Do people not play deathmatches on LAN parties anymore these days? 2.5 is huge if the game lasts long enough that someone would be leading with 200. ;)
There are noticability thresholds where this could push it over. For fighting games if you have the reactions to whiff punish N frame recovery moves this may push you to only being able to punish N+1 recovery moves and really impact your ranking. This is a little over 1/3rd of a 60hz frame.
About the difference between 60hz and 120hz monitor, instantly noticeable just by moving the mouse in windows (just by looking
at the distance cursor updates as it moves). Would you argue that all gaming monitors are placebo?
I actually would. Gaming monitors are the equivalent of fancy audiophile gear. It's a way to fleece people by making them think they can perceive a difference that isn't really there.
David Eagleman has done some work with drummers. Granted the audio system might be a bit more accurate than the visual, or maybe drummers are just weird. On the other hand, vim taking 30 milliseconds to start (ugh) and having sluggish cursor motions is why I'm on vi now. Haven't tried Wayland. Maybe in some number of years once it's more portable and more developed? (And how many years has it already been out?)
> “I was working with Larry Mullen, Jr., on one of the U2 albums,” Eno told me. “ ‘All That You Don’t Leave Behind,’ or whatever it’s called.” Mullen was playing drums over a recording of the band and a click track—a computer-generated beat that was meant to keep all the overdubbed parts in synch. In this case, however, Mullen thought that the click track was slightly off: it was a fraction of a beat behind the rest of the band. “I said, ‘No, that can’t be so, Larry,’ ” Eno recalled. “ ‘We’ve all worked to that track, so it must be right.’ But he said, ‘Sorry, I just can’t play to it.’ ”
> Eno eventually adjusted the click to Mullen’s satisfaction, but he was just humoring him. It was only later, after the drummer had left, that Eno checked the original track again and realized that Mullen was right: the click was off by six milliseconds. “The thing is,” Eno told me, “when we were adjusting it I once had it two milliseconds to the wrong side of the beat, and he said, ‘No, you’ve got to come back a bit.’ Which I think is absolutely staggering.”
Those sorts of latencies actually are noticeable! As an example, 6.5ms latency between a virtual instrument and its UI is definitely noticeable.
I didn’t think it was. But it is. I promise!
It’s not necessarily a reaction-time game-winning thing. It’s a feel.
With virtual instruments, my experience is that when you get down to ~3ms you don’t notice the latency anymore… but!, when you go below 3ms, it starts feeling more physically real.
As I type this, I'm playing "the finals" on Linux via Proton on Wayland. I won't pretend I'm any kind of competitive gamer type, but it's perfectly fine and I don't feel like I'm being held back by input lag. So this is very much a niche issue to have.
"Competitive FPS gaming" stops allowing anything besides fresh Windows installs after a certain point. It's a diminutive distinction, like pointing out that emulating GameCube games won't let Smash Melee players fight competitively; nobody playing on either side actually cares.
> having to revert to classic x11 to play games, lol.
It would be more of a problem the other way around, if we had to resort to Wayland to get low latency. I think most of us using Linux for gaming and casual stuff are happy to stick to X11 for now and the foreseeable future. It has good support in software, its failure modes are well-documented, and it doesn't add one more layer to the pile of complexity that desktop linux already is; at least, not one us users have to consciously keep in mind as a common source of issues.
From outside it's hard to tell if it's truly protocol differences or just the age of the implementations on X11, but when Wayland came out every project has claimed improvements over the old X11 stack. Also, from the early Wayland days presentations bashed the protocol as something that couldn't be fixed without a rework that was not going to happen due to the dead weight of backwards compatibility and awful older hardware.
As a user applications running on Wayland have consistently improved on how nice things feel if you don't miss your latency deadlines. It's easy to perceive on apps, and straight out obvious in games.
I got to diamond in Apex Legends on Linux using Wayland on KDE Plasma. Didn't feel any noticeable difference between wayland, x and windows in my Apex playing.
I get major lag spikes when the gpu is under heavy load (like doing Stable Diffusion inference or something). TBF I haven't A/B tested with X11, but I don't ever remember it being like that. An extra frame of latency isn't great on it's own, but the occasional spikes in lag are really irritating.
May still happen especially if it is thrashing vram in and out of system memory or something, but have you tried lowering priority of the stable diffusion process?
I can also attest to horrific lagspikes on an Optimus laptop even if Intel is driving the desktop. Memory pressure is definitely the problem here. Lagspikes actually lessened when I switched to Wayland Gnome. I think they lessened further with PREEMPT_RT on kernel 6.12. Nvidia requires an environment variable to build on real time kernels but it plays surprisingly nice as of driver 570. But if you have this config, you need at least 11th gen Intel iGPU or AMD APU, because i915 does not build for real-time kernels. Only the Xe driver works and only if you force_probe the ID if it's Tiger Lake.
...Which I don't get because the Xe driver is said to explicitly support, at minimum, Tiger Lake. I played Minecraft on the iGPU with Xe and it was perfectly fine. It... drew 3D graphics at expected framerates.
For anyone who uses ffmpeg for this type of per frame analysis, `ffmpeg -skip_frame nokey -i file -vsync 0 -frame_pts true out%d.png` will get the "presentation time" of each frame in the video. That's more precise than just dumping frames and calculating timestamps. You can also do something similar in a web browser by playing a <video> and using `requestVideoFrameCallback()`. Though, you might need to set `.playbackRate` to a low value if the computer can't decode all the frames fast enough.
> With my 144Hz screen,....Wayland, on average, has roughly 6.5ms more cursor latency than X11 on my system...Interestingly, the difference is very close to 1 full screen refresh. I don't know whether or not that's a coincidence.
The fact that the latency is almost 1/144th of a second means that it might become 1/60th of a second on standard 60Hz monitors. This is hard to notice consciously without training, but most people can "feel" the difference even if they can't explain it.
I was pretty shocked by the eyestrain difference I felt going from 30hz to 60hz with a 4K monitor while only doing coding tasks (i.e. text and mouse, no real graphics or animations).
I recently went from 60 to 144, also on a 4k (27") monitor, and to me too it has been a very noticeable upgrade. No tearing of any kind when dragging windows around, and scrolling feels buttery smooth.
It's almost certainly because of one extra frame of buffering between the mouse move and the screen. Vsync can cause this, but it should be possible to get vsync with just double buffering.
> The fact that the latency is almost 1/144th of a second means that it might become 1/60th of a second on standard 60Hz monitors.
My guess: the "true" numbers are close to 2.5 (half a frame of random phase of when the mouse is touched vs. refresh, plus 2 frames to move cursor) and 3.5. If you throw out the low outlier from each set you get pretty close to that.
(of course, the 125Hz mouse poll rate is another confound for many users, but this guy used a 1KHz mouse).
> This is hard to notice consciously without training, but most people can "feel" the difference even if they can't explain it.
Yah. 7ms difference is not bad vs 16.6ms is starting to be a lot.
IMO, we should be putting in effort on computers to reach 1.6 frames of latency -- half a frame of random phase, plus one frame, plus a little bit of processing time.
To have a compositor not introduce a frame of latency more or less requires it to race the beam, which has definitely been suggested[1], but you can see how it’d be difficult, and so far no operating systems have tried as far as I know. And as for good frame pacing support in UI toolkits (and not just game engines), well, one can dream. Until both of these are in place, 2.5±0.5 seems to be the hard limit, the question here is more where the Mutter is losing another frame (which even the greats tell us[2] is not hard to do by accident).
I’ve read these arguments quite a few times and always found them a bit questionable. Sure, if everything is driven by the vblank time (or any other clock that counts in frames), it makes sense. But that’s a silly approach! There is nothing whatsoever special about allocating one full frame interval to the compositor to composite a frame — if it takes 16ms to composite reliably, it will take 16ms to composite reliably at 30Hz or 60Hz or 144Hz. So shouldn’t the system clock itself on a time basis, not a frame basis?
Put another way, a system fast enough to composite at 144Hz should be able to composite at 60Hz while only allocating 1/144 seconds to the compositor, which would require offsetting the presentation times as seen by the compositor’s clients by some fraction of a frame time, which doesn’t actually seem that bad.
It gets even better if variable refresh rate / frame timing works well, because then frames don’t drop even if some fraction of compositing operations are a bit too slow.
I assume I’m missing some reason why this isn’t done.
We have multiple processes on a time-share system. This mouse is updating at 1kHz, it just doesn't make sense consistently rerendering everything when the screen aint updating.
We need to cut some deadline and doing it at vsync is the easiest way
What do you mean? Most games run on XWayland when you're in a Wayland session, wouldn't input latency caused by XWayland therefore affect gamers who use Wayland?
I’ll look into it, but it used to be a choice in the game settings that bypassed the drawing display server entirely, and caused that full screen flicker you might recall when tabbing back to your desktop (and the switchover of rendering control, and subsequent huge performance loss of the running game).
I wasn’t aware that this changed, but you could be right. Its definitely the same on Windows as it always was, which is the platform I most developed games for.
> When using Fullscreen Optimizations, your game believes that it is running in Fullscreen Exclusive, but behind the scenes, Windows has the game running in borderless windowed mode. When a game is run in borderless windowed mode, the game does not have full control of the display– that overarching control is given back to the Desktop Window Manager (DWM).
I think Windows now also supports hardware overlays, so games running in fullscreen Windows can skip the compositor, which reduces the latency in no longer allowing games to directly write to the FB.
The DWM can do this even for windowed applications if they use flip model -- it will dynamically transition individual windows between compositing and independent flip (overlay).
Most compositors can do direct scanout when they detect a full screen surface. (This is broken right now with Gnome on X11 because of the way window frames are drawn: https://gitlab.gnome.org/GNOME/mutter/-/issues/2794 ) If the game uses the platform cursor then that will reflect the additional latency, but the actual input signals won't be delayed.
If an application framebuffer is full screen and in a compatible pixel format the compositor can do "direct scan out" where the compositor sends the framebuffer directly to the crtc instead of compositing first. I know that wlroots supports that. I'm not sure how much performance it saves to be honest.
And I'll second that most commercial games do go through XWayland, though it depends on what games you like!
Wayland is just a protocol that is probably implemented over and over and each implementation might have issues. It's mostly likely the different implementations having issues.
Was Wayland not architected to be simpler to implement than the (antique) X11 protocol?
Not arguing though, you are right it is just issues: Drivers, the Wayland implementations, how some plethora of apps and libraries have been battle tested then optimized, years over years, for X11. Not as much for Wayland display yet.
Architecture contributes, but all it takes to blow input latency is some inefficient code almost anywhere in the input or rendering pipeline. The Apple IIe didn’t really have very many layers between the keyboard and the CRT. Wayland might have an optimal architecture in theory for latency — I don’t know — and latency could still be ruined by not fully optimizing the other components in the stack.
> Was Wayland not architected to be simpler to implement than the (antique) X11 protocol?
I think it's safe to assume that it is actually simpler, given that we already have multiple Wayland implementations, but still basically just the one X11 server implementation. Can one or more of those implementations shave off one or two milliseconds overs the next 40 years... Probably yes.
Sure, on linux all you really have is xorg, what used to be xfree86, but I have used xsgi, and there are X servers for windows, heck, I have even seen this amazing demo Xserver that ran on the web.
footnote: there is something wrong with the commonly used web search engines and I am unable to find that X11 web demo, I think they are prioritizing recent content over good content. you would think with how interesting that web demo was it was it would show up right away. but I got nothing, so no links. Anyway it was a deep dive into the intricacies of X11 window painting and the author had ported a limited subset of an X server to javascript in order to provide a live demonstration of the topic. I will keep looking.
Wayland is simpler because it does less. And among the things Wayland doesn't do and X does are things that people need. It means these things have to be implemented elsewhere.
It is a common problem with these "simple" things. The problem is complex, and if you simplify one part, you are just pushing complexity elsewhere, you are not making it disappear. It is sometimes a good thing, but in the case of Wayland, it didn't go well.
There is also the option to make complexity disappear by just convincing people that they should not want to do what they want to do. This was the kind of response to what I needed to do. Then I had to switch back to x11.
In any case wayland is not bad if you only have pretty basic needs I guess, some basic things look easier to me there from a user perspective, and troubleshooting x11-related issues for a non-technical person is no fun either.
Gnome was fixed a few years ago to remove significant latency it was adding. May have been a factor if you used that as well, but it wasn't wayland specific to my knowledge.
> Was Wayland not architected to fix latency issues?
It was designed to fix tearing issues, not latency issues.
But then the Wayland designers found out that players often prefer tearing to minimize latency, so the tearing protocol was implemented.
When a committee of perfectionists, rather than real people or companies, design something, you often get something that is completely unusable by anyone but that committee.
And that's exactly how it's played out so far. Wayland is still largely incomplete, not to mention it doesn't even have a reference implementation [1], and still doesn't include/describe several essential desktop APIs and features, which results in this protocol not having a ton of universal tools and applications that work across all of its implementations, including but not limited to:
* Keyboard layout switching and input management
* Display configuration
* Clipboard management
* Tools to manage windows/automate tasks
* Lots more
It's a damn ugly mess that has led to a lot of fragmentation with no obvious benefit. Currently, only KDE and Gnome have somewhat usable Wayland implementations. If you use anything else? You're SoL.
Of the things you list, IMO, maybe the display configuration should be part of the Wayland protocol.
It might be a better technical design to have the other stuff outside of the display protocol. Just because Xorg implemented something does not mean you have to put it in the Wayland protocol.
Routing of input events to windows has turned out to also be extremely important for a display subsystem in practice. It wasn’t just X11 that did this, it was also the NeXT window server, and Sun’s NeWS, and SGI’s pre-X window system, and the original VAXstation window system on VMS, as well as how both modern macOS and modern iOS and iOS-derived platforms work.
In any of these cases there may be one or more daemons behind the scenes handling the “raw” input—possibly even in cooperation with kernel-level pre-processing code, to ensure low latencey—but most event delivery to applications is associated with windows, with options to get lower-level access if needed.
One of the things that helps many of the systems described above with latency is kernel participation, whether by pushing much of the preprocessing of events down to the drivers so there’s little for userspace to do, or by implementing kernel-level zero-copy IPC (e.g. use of Mach messages by NeXT and Apple).
If human interface IPC happens entirely in userspace and requires multiple context switches to get an event from device to a display change, you’ll wind up with hitches and delays unless there’s some sort of scheduler hinting that ensures each stage in the pipeline runs immediately after the last.
This is, of course, why there was a lot of desire by Wayland advocates for kernel dbus support, but they went at the problem backwards: “Let’s take DBus, and make it fast by putting it in-kernel,” *without* first trying to make it as fast as possible without kernel support, *and* without trying to figure out the minimal feature set for kernel-level IPC that would be needed to support it (which may not look like DBus).
It's almost like you don't want tearing when browsing the web and your documents, but games can take over the precise control of rendering.. because they are games that want to do so?
The numbers presented in the article (~1 additional frame of latency on Wayland) are from someone using GNOME with the Mutter compositor. This means that neither the X11 or Wayland session has screen tearing. I imagine the X11 input latency numbers would go down by around half a frame on average if the author was using a different window manager without a compositor.
As I understand it, the tearing thing (no vsync) was only implemented after Valve pleaded, begged, and sacrificed a small animal to the Wayland developers.
I'm pretty sure this is incorrect? I half-remember having cursor rendering bugs on GNOME Wayland with NVidia ages ago which I had to disable something h hardware cursor related to fix. But I don't know if that's what you're talking about or not. Could you link to a source which says that Mutter doesn't make use of hardware cursor?
(Hi, author here by the way! ... Don't worry, that disabling hardware cursors thing was at least one OS re-install ago)
I wonder if the latency is due to the compositor receiving a damage event for the area of the cursor, thus being a frame late to render. But probably that wouldn't be an issue with hardware cursors.
Yes, it should use hardware cursor by default on most hardware. But I don't think that RPi's VideoCore (in the linked article) supports hardware cursor or multiple planes. The difference between X11 and Wayland might then be that the cursor isn't vsynced as part of the composite process on X11, where it is on Wayland.
Mutter definitely throttles cursor updates on Wayland, too, which will contribute slightly to latency even with a hardware cursor. In general, with Wayland, the compositor is responsible for cursor updates, and I'm not sure which other ones throttle. But that would be where the difference comes from when using hardware cursors.
I think the difference mort96 is seeing is the cursor update throttling. If it updates at the same rate of the refresh rate then it's a crapshoot where in that interval it hits relative to vsync, with a worst case additional latency of the frame rate. X11 updates the cursor whenever it moves, so on scanout it's always where you expect it to be, even if the motion was almost immediately before vsync.
I should mention that in the past there's been problematic interactions on amdgpu with atomic updates of the cursor and display planes. This resulted in either one being starved from changing, causing stuttering in one or the other depending how it was handled. IIRC, that's why the throttle is there. You could try MUTTER_DEBUG_ENABLE_ATOMIC_KMS=0 to see if they only added the throttle to the atomic code path.
That there's a probability of 4.20e-05 that the observed difference of that large would happen by chance due to observation noise if there would be no real difference (given your assumptions about the data generating process holds).
The normality assumption is a large stretch, especially since there an absolute lower limit near the observation and a somewhat discrete distribution, so a t-test isn't appropriate. But then again it looks significant, so there's no real need for a test
Ported to Linux? It runs on Linux. What exactly would need porting?
What would be required to make surfaceflinger a practical Linux desktop compositor is merely some translation layer, equivalent to XWayland, that supports hardware acceleration. Such things have been written, but not open source and they never got traction.
> Ported to Linux? It runs on Linux. What exactly would need porting?
It runs on Android/Linux, not GNU/Linux. I would be surprised if there wasn't at least a bit of work needed to handle things like bionic vs glibc, and you'd really want it to act "normal" by being installed to FHS paths instead of Android's weird filesystem layout. All doable, I expect, but it would be a port.
This is excellent. Too often people guess at things when they could be more empirical about them. Ever since I learned the scientific method (I think 3rd or 4th grade) I was all about the 'let's design an experiment' :-).
Let me state up front that I have no idea why Wayland would have this additional latency. That said, having been active in the computer community at the 'birth' of X11 (yes I'm that old) I can tell you that there was, especially early on, a constant whine about screen latency. Whether it was cursor response or xterm scrolling. When "workstations" became a thing, they sometimes had explicit display hardware for just the mouse because that would cut out the latency of rendering the mouse in the frame. (not to mention the infamous XOR patent[1])
As a result of all this whinging, the code paths that were between keyboard/mouse input and their effect on the screen, were constantly being evaluated for ways to "speed them up and reduce latency." Wayland, being something relatively "new" compared to X11, has not had this level of scrutiny for as long. I'm looking forward to folks fixing it though.
I'll be reading a dream of spring in my grave at this rate.
I understand I'm complaining about free things, but this is a forced change for the worse for so long. Wayland adoption should have been predicated on a near universal superiority in all input and display requirements.
Intel and AMD and Nvidia and Arm makers should be all in on a viable desktop Linux as a consortium. Governments should be doing the same because a secure Linux desktop is actually possible. It is the fastest path to showcasing their CPUs and 3d bling, advanced vector /computer hardware.
Wayland simply came at a time to further the delay of the Linux desktop, at a time when Windows was attempting to kill Windows with its horrid tiles and Apple staunchly refused a half billion in extra market cap by offering osx on general x86.
Wayland is a protocol. The problems people complain about are generally implementation details specific to GNOME or KDE or (in general) one particular implementation.
There's rarely any such thing as "universal superiority", usually you're making a tradeoff. In the case of X vs Wayland it's usually latency vs. tearing. Personally I'm happy with Wayland because there was a time when watching certain videos with certain media players on Linux was incredibly painful because of how blatant and obtrusive the tearing was. Watching the same video under Wayland worked fine.
Early automobiles didn't have "universal superiority" to horses, but that wasn't an inhibitor to adoption.
I mean most of the things are the fault of a badly designed or non existent protocols:
-Problems with non western input systems
-Accessibility
-Remote control(took around 2 years to be stable I think?)
-Bad color management
Then there's the things that did work in x11 but not in wayland:
-Bad support for keymapping(the input library says keymapping should be implemented by the compositor, gnome says not in scope, so we have a regression)
-bad nvidia support for the first two years? three years?
While these things are compositor/hw vendor faults, the rush to use wayland and nearly every distro making it as default, forced major regressions and wayland kinda promised to improve the x11 based experience.
Yes, and to the parents point it was a CHANGE in protocol.
I get there was cruft in x. But the moment selected was a barrier to Linux desktop adoption precisely when the greatest opportunity in decades was present.
And the desktop was reimplemented.
Now in this period kde and gnome both decided to do rewrites, Ubuntu did their own desktop, got it up to snuff, and abandoned it. The lunacy wasn't just Wayland.
If we are complaining the gnome compositor sucks... I mean , should that be the goddamn reference implementation? What percent of desktops are gnome, 80% at least? If the gnome composting ready for primetime, then Wayland isn't ready for primetime.
>If the gnome composting ready for primetime, then Wayland isn't ready for primetime.
I use Sway, which uses a different compos[i]tor than Gnome. I would like to see similar results for wlroots, Sway's compositor, though I'm not actually interested enough to do the experiment (I guess that would be comparing Sway with i3). Cursor lag in Sway is not enough to bother me. I have on occasion used Gnome on the same machine(s), and never been bothered by lag.
As others have pointed out, Wayland is a protocol, not a compositor.
I have no idea of the difference(s) in performance between wlroots and Gnome's compositor. Protocols do not have performance, implementations do. If someone can prove that the Wayland protocol, on a certain set of tasks, and in a certain environment, has a better or worse performance than the X11 protocol, then it might be possible to make abstract comparisons on protocol.
Given that Gnome runs on both X and Wayland, it might be interesting to hear from the Gnome authors on the performance differences.
I think that gnome has had built-in IME, but at least for a long time, it wasn't possible to use a third party system with gnome, or use gnome's with other compositors. And I'm pretty sure the situation was the same for sreen readers and on-screen keyboards. The wlroots project created their own protocols to support external applications to provide such features, since that is out of scope for a compositor like sway, but there are still missing pieces.
> the rush to use wayland and nearly every distro making it as default,
Which rush? It has been done by only a small fraction of distros like Fedora, after years of development of the first wayland compositors. Fedora main purpose has always been to implement bleeding edge tech stuff early so that bugs get found and fixed before people using more stable distros have to suffer from it.
Nobody has been forced in any regression and x11 has continued to be available until now and there is no sign that the most conservative distros will drop x11 support anytime soon.
Nvidia support is still poor (at least on latest cards), I'm forced to use X or I get tons of glitches.
I need the proprietary drivers for machine learning.
Not that I mind particularly, X is fine and everything works as expected.
I can even have the integrated (amd gpu) take care of the desktop while the Nvidia gpu does machine learning (I need all the vram I can get, the desktop alone would be 100-150mb of vram) - then I start a game on steam and the nvidia gpu gets used.
Funnily enough I had wayland enabled by default when I installed the system and I didn't understand why I was getting random freeze and artifacts for weeks. Then I realized I was not using X11
I've used Wayland on Fedora for 16 years. Had to disable Nvidia GPU on laptop And use Intel GPU for first 11 years. Now have been using AMD GPU smoothly for 5 years.
But I hear Nvidia is slowly getting back on the game. Let's hope they get stable support less than 20 years of wayland being in use on major desktop Linux distros.
> Wayland is a protocol. The problems people complain about are generally implementation details specific to GNOME or KDE or (in general) one particular implementation.
I feel like at some point this is a cop-out. Wayland is a protocol but its also a "system" involving many components. If the product as a whole doesn't work well then its still a failure regardless of which component's fault it is.
Its a little like responding to someone saying we haven't reached the year of linux on the desktop by saying: well actually linux is just the kernel and its been ready for the desktop for ages. Technically true but also missing the point.
X is fine, most of the problems people bring up are niche and minor. Meanwhile, Wayland induces problems of its own, like breaking all sorts of accessibility systems and multi-window X applications with no solution in sight.
I believe you. Can you share an example? To be clear, I'm pretty sure this can be done with Gtk and Qt, but maybe you are talking about older apps written directly in Xlib or Xt?
It's not the toolkit (as they work on X11 and other OSes with the existing toolkits), it's Wayland that's the issue (there are a series of Wayland protocols to implement this, but they are being blocked).
To be fair this wayland issue is also niche. Author of the linked article wrote that this is reported by really small percentage of users most of the people do not notice it.
What components? Wayland is literally an XML protocol that turns an XML file into a method of communication. libwayland-server and libwayland-client only handle communication and the internal event loop. Its completely up to the developer to write the implementations of these functions and register them in the server.
Then a client is going to query the server and ask request to do stuff via a unix socket. In fact, you don't even need libwayland, you can raw dog it over sockets manually. The idea is that there are standard protocols that can be queried and used,and you can implement this in any environment you want to. You could write the "frontend" in html and JS and run a wayland compositor on the web (which has been done [1]), you could do it with text or anything really, most people use some graphics stack.
Nit: Wayland isn't an XML protocol. The "calls" and their arguments are described in XML, but the data is transmitted in a fairly simple binary encoding.
- the compositor, of which there are multiple implementations (gnome, kde, all the wlroots compositors)
- the client, which often uses one of several toolkits (gtk, qt, several smaller frameworks, or even directly using the protocol)
- the wayland protocol (or rather protocols, because there are several extensions) itself
- other specifications and side channels for communication, in particular dbus.
Many issues (although I don't think the one in OP) are due to the protocol being underspecified, and so the client and compositor disagree about some semantics, or doesn't even have a standard way to accomplish something across all compositors.
The key detail in the “Wayland is a protocol” is that there are several other implementations, some of them extremely mature. The implementation being tested here isn’t exactly know to be a good one.
If there were a single Wayland implementation in existence, I’d agree with your sentiment.
The author was using Gnome, which is meant to be the most mature Wayland implementation given its developed by Redhat which is also the same company that is the biggest driver of Wayland protocol.
"Wayland is a protocol" is exactly the problem. Protocols suck; they just mean multiplying the possibility of bugs. A standard implementation is far more valuable any day.
With X11, it was simple: everybody used Xfree86 (or eventually the Xorg fork, but forks are not reimplementations) and libX11 (later libxcb was shimmed underneath with careful planning). The WM was bespoke, but it was small, nonintrusive, and out of the critical path, so bugs in it were neither numerous nor disastrous.
But today, with Wayland, there is no plan. And there is no limit to the bugs, which must get patched time and time again every time they are implemented.
X had garbage handling of multiple monitors and especially multiple monitors with different DPIs, and there was "no plan" to deal with that either. Nobody wanted to work on the X codebase anymore. The architecture bore no resemblance to the way any other part of the desktop stack (or hardware) works.
Most of the garbage aspect is because toolkits refuse to support multiple monitors on DPI with X11 with the argument that "Wayland is just around the corner", for decades now.
For example Qt does per-monitor DPI just fine on X11; it's just that the way to specify/override the DPI values just sucks (an environment variable).
This stupid decision is going to chase us until the end of times since Xwayland will have no standardized way to tell its clients about per-display DPI.
Yes. It just proves that all you needed is a better way to specify the per-monitor DPI, one that can be updated afterwards, or even set by the WM on windows.
X11 used to provide separate displays, but at some point due to hardware changes (and quite probably due to prominence of intel hardware, actually) it was changed to merged framebuffer with virtual cut out displays.
In a way, Wayland in this case developed a solution for issue its creators brought into this world first
It can still provide seperate displays. The problem is you couldn't do something like drag a window from display 1 to 2°. IIRC it's also annoying to launch two instances of a program on both displays. The hacky merged framebuffer thing is a workaround to these problems. But you can have independent DPIs on each display.
Yeah there were certainly tradeoffs. It's much harder to use separate displays now, though - last time I tried, I could address the two displays individually (":0.0" and ":0.1") if I launched X on its own, but something (maybe the display manager?) was merging them into a single virtual display (":0") as soon as I tried to use an actual desktop environment. (This was was Mint 20, MATE edition, a few years ago - I gave up and reverted to a single-monitor setup at that point.)
X could do seveal different screens I did have this working once. However then moving an application to a different display was impossible (an app could do it but it was a lot of work so nobody bothered). I few cad programs supported two streens but they were seperate and the two didn't meet.
Most people want to drag windown between screens and sometimes even split down the middle. One large display supports that much easier so that is what everyone switched to in the late 1990
I was using it that way until about 2020. (Mint 13 MATE supported, but it seems that capability was lost somewhere along the line. Shame, because I have a dual monitor setup where the second monitor is often displaying the picture from a different device, so in that situation I absolutely cannot have applications deciding to open on the busy-elsewhere screen. I miss being able to set a movie running on one monitor and have it not disappear if I flipped virtual desktops on the other!)
Yes, separate screens would be a much better model for me as well. Much better than KDE randomly deciding to show KRunner on the turned off TV for some reason unless I manually disable the output.
X11 does a lot of things that are outdated now, and multiple independent screens is one of them. Ideally, you'd be able to select either independent screens or one big virtual screen, and still have a window manager be able to move windows between independent screens. I don't know how that would be achieved though.
X's "mechanism, not policy" has proven to be a failure. Without a consistent policy you end up with a pile of things that don't work together. "One big virtual screen" is a policy and it's one that works well enough.
Deciding what needs to be in the protocol and what should be in the application is never easy. It's important to be able to iterate quickly and avoid locking in bad practices. I think that's what Wayland tried to do by making everything a fine-grained extension, but it doesn't really work like that as the extensions become mandatory in practice.
For the record: you can specify a different DPI _for each monitor_ for Qt X11. You just cannot change it after the program has started, which is exactly the limitation I was referring to.
But you can definitely move windows to another monitor and Qt will use the right DPI for it. It is the same behavior as Wayland. "One large wide screen display" is exactly how Wayland works...
That, i think, is the main issue. Nobody wants to work with GTK1, or GTK2, or GTK3 anymore. Nobody wants to work with QT1, or QT2, or QT3 or QT4 anymore. Everybody wants the new shiny toy. Over and over again.
It is CADT all over.
Earlier X was developed by an industry consortium. Now Wayland is a monopoly pushed by RedHat.
Exactly, same situation as with audio. Instead of improving the OSS API we got ALSA, PulseAudio, and now PipeWire (and a bunch of less mainstream alternatives) -- all with their own weird issues. And of course it's application (or at least toolkit) developers that bear the cost of this churn, not the developers that get to work on their fancy new NIH tech that they'll then abandon before being 100% stable for yet another reimplementation.
Software has bugs and water is wet. Wait til you hear about HTTP, TCP, UDP, IP torrents, etc... and "simple" is not really a term I would designate to X11. I mean, its fine, but even just the ecosystem surrounding X is convoluted, outdated and absurd. Things like xinit, startx, .Xauthority, xresources, xhost etc... are all a mess.
> there was a time when watching certain videos with certain media players on Linux was incredibly painful because of how blatant and obtrusive the tearing was
It was because of the crap LCD monitors (5 to 20 ms GtG) and how they are driven. The problem persists today. The (Wayland) solution was to render and display a complete frame at a time without taking into account the timings involved in hardware (you always have a good static image, but you have to wait).
I tried Tails (comes with some Wayland compositor) on a laptop. The GUI performance was terrible with only a Tor browser open and one tab.
If you do not care about hardware, you will, sooner or later, run into problems. Not everybody has your shiny 240 Hz monitor.
You're arguing against Wayland, but for a more secure Linux desktop? I recommend you spend more time getting to know the X11 protocol then, because it has plenty of design decisions that simply cannot be secured. The same people who used to develop XFree86 designed Wayland to fix things that could not be fixed in the scope of X11.
> the X11 protocol [...] has plenty of design decisions that simply cannot be secured.
I've been hearing this for over a decade now. I don't get it. Just because xorg currently makes different clients aware of each other and broadcasts keypresses and mouse movements to all clients and allows screen capturing doesn't mean it has to. You could essentially give every application the impression that they are the only thing running.
It might seem difficult to implement, but compare it to the effort that has gone into wayland across the whole ecosystem. Maybe that was the point - motivating people to work on X was too difficult, and the wayland approach manages to diffuse the work out to more people.
I was really bullish on Wayland 10 years ago. Not so much any more. In retrospect it seems like a failure in technical leadership.
It'll be challenging to even figure out which one of the things connecting to $DISPLAY is the real window manager. Good luck on your lonely[1] journey!
[1]: The people who actually developed Xorg are now working on various Wayland-related things.
> It'll be challenging to even figure out which one of the things connecting to $DISPLAY is the real window manager.
I suspect it would be less challenging than writing a whole new wayland server.
Off the top of my head, I'd use a separate abstract domain socket for the window manager including some UUID, and then pass that to the window manager when launching it.
You could create these sockets on demand - one for each security context. On linux typically a different security contexts will either have different UIDs - in which case filesystem permissions would be sufficient - or they have different mount namespaces - in which case you make different sockets visible in different namespaces.
For SSH forwarding you could have SSH ask the X server for a new socket for forwarding purposes - so remote clients can't snoop on local clients.
> Good luck on your lonely[1] journey!
>
> [1]: The people who actually developed Xorg are now working on various Wayland-related things.
This is what I mean by a failure of technical leadership.
"You could create these sockets on demand - one for each security context. On linux typically a different security contexts will either have different UIDs - in which case filesystem permissions would be sufficient - or they have different mount namespaces - in which case you make different sockets visible in different namespaces."
This is reminiscent of how Trusted Solaris[0] implements Mandatory Access Control (MAC) a la Orange Book[1].
> For SSH forwarding you could have SSH ask the X server for a new socket for forwarding purposes - so remote clients can't snoop on local clients.
SSH pretty much already does this. Per default (using -X) X11 forwarding is in untrusted mode, which makes certain unsafe X11 extensions unavailable.
So remote clients already cannot snoop the whole keyboard input.
> Off the top of my head, I'd use a separate abstract domain socket for the window manager including some UUID, and then pass that to the window manager when launching it.
And then the window manager spawns whatever programs the user wants, and they end up sharing that $DISPLAY.
I'm not trying to claim that X is secure as is. The claim I'm making is:
1. There is nothing fundamentally insecure about the design of X that couldn't be fixed with a bit of effort.
2. The effort required would be significantly less than what has gone into Wayland over the last 16 years.
Fixing the security of the system certainly would involve changes to more than just xorg. Anywhere there is a security boundary you'd need to make modifications - but that's OK, most of these security boundaries are younger than wayland anyway.
This discussion of $DISPLAY seems like distracting minutiae. Sure it's a medium size problem that would need solving, but that's all it is.
To address your specific point: it could be the responsibility of the window manager to set $DISPLAY to a less powerful socket when starting processes. Ultimately it doesn't matter though, because if the display server isn't doing some sort of sandboxing then the spawned process can just ptrace xorg and do whatever it wants. X being secure only matters in the presence of a security boundary and in that case it would be the responsibility of whatever is setting up that boundary to create a less privileged socket. Whether that be ssh or flatpak or whatever.
So now we've reached the point of "you could secure X11 if you changed everything". Sure, fine.
Once again, for emphasis: nobody has stepped up to do that, in the history of X (outside of proprietary products, of unknown quality).
Wayland is not some outsider project competing against X11. It's the people who were developing Xorg, saying they're fed up with X11 and need a clean slate to fix its problems.
Here's my unpriviledged user trying to ptrace my Wayland compositor running as me:
Nearby, I have a Chromebook where a potentially hostile virtual machine can display Wayland windows seamlessly on the host. The worst the VM can do is fill bitmaps with offensive imagery.
In a world where the people doing the work switched to developing the Wayland ecosystem, arguing about a "could" that requires changing every window manager seems like a guarantee X11 will not get improved. Feel free to put effort into it...
> Wayland adoption should have been predicated on a near universal superiority in all input and display requirements.
Totally agree. The people saying "Wayland is a protocol" miss the point. Wayland is a protocol, but Wayland adoption means implementing stuff that uses that protocol, and then pushing it onto users.
Measure twice, cut once. Look before you leap. All that kind of thing. Get it working FIRST, then release it.
You have to releaseethings like this in parts because it needs too many external people to do things to make it useful. Managing those parts is something nobody has figured out and so people live you end up using it before it is ready for your use and then complaining.
When I say "release" I mean "release to users". You can release stuff to other developers, no problem. But it should all come with large warnings saying "This is not for daily use". The failure is when distros like Fedora start not only shipping it, but then saying they're going to drop the working alternative before Wayland is actually ready to do everything that X does.
(Also, I don't use Wayland. I mean I tried it out but don't see any real benefit so I don't use it regularly.)
This would be best solved by maintaining support for old protocols instead of reinventing the wheel. There is no chicken/egg problem when you have proper backwards compatibility.
> I understand I'm complaining about free things, but this is a forced change for the worse for so long.
Then write code.
Asahi Lina has demonstrated that a single person can write the appropriate shims to make things work.
Vulkan being effectively universally available on all the Linux graphics cards means that you have the hardest layer of abstraction to the GPU taken care of.
A single or small number of people could write a layer that sits above Wayland and X11 and does it right. However, no one has.
This argument don't make sense because Wayland started as a hobby and not to replace x11, was after it got traction, other people/companies started contributing that it matter
I don’t really agree, because I think there’s something of a chicken-and-egg effect here.
With complex projects like these that have to work well on such a wide array of hardware and configurations, lots of real world usage is required to achieve any level of refinement. Without widespread adoption, Wayland likely would have been stuck in experimental/toy status for much longer than it will as things are going currently.
Display devices (usually part of your GPU) still have "explicit display hardware just for the mouse" in form of cursor planes. Later this has been generalized as overlay planes.
Planes can be updated and repositioned without redrawing the rest of the screen (the regular screen image is on the primary plane), so moving the cursor is just a case of committing the new plane position.
The input latency introduced by GNOME's Mutter (the Wayland server used here) is likely simply a matter of their input sampling and commit timing strategy. Different servers have different strategies and priorities there, which can be good and bad.
Wayland, which is a protocol, is not involved in the process of positioning regular cursors, so this is entirely display server internals and optimization. What happens on the protocol level is allowing clients to set the cursor image, and telling clients where the cursor is.
Protocols can bake in unfortunate performance implications simply by virtue of defining an interface that doesn't fit the shape needed for good performance. Furthermore, this tends to happen "by default" unless there is a strong voice for performance in the design process.
Hopefully this general concern doesn't apply to Wayland and the "shape" you have described doesn't sound bad, but the devil is in the details.
Yeah, Wayland isn't designed in such a way that would require any additional latency on cursor updates. The Wayland protocols almost entirely regard how applications talk to the compositor, and don't really specify how the compositor handles input or output directly. So the pipeline from mouse input coming from evdev devices and then eventually going to DRM planes doesn't actually involve Wayland.
I don't think the Wayland protocol is actually involved in this. Wayland describes how clients communicate with the compositor. Neither the cursor, nor the mouse are a client, so no where in the path between moving the mouse and the cursor moving on screen is Wayland actually involved.
The story is different for applications like games that hide the system cursor to display their own. In those cases, the client needs to receive mouse events from the compositor, then redraw the surface appropriately, all of which does go through Wayland.
According to Asahi Lina, X does async ioctl that can update the cursor even during the scanout of the current frame, while Wayland does atomic, synced updates on everything, cursor involved, which has the benefit of no tearing and the cursor's state is always in sync with the content, but it does add an average of 1 more frame latency (either updates just in time for the next frame), or it will go to the next frame.
This is not what Wayland does, it is what a particular display server with Wayland support decided to do.
Second, just to be clear, this only discusses mouse cursors on the desktop - not the content of windows, and in particular not games even if they have cursors. Just the white cursor you browse the Web with.
Anyway, what you refer to is the legacy drm interface that was replaced by the atomic one. The legacy interface is very broken and does not expose new hardware features, but it did indeed handle cursors as its own magical entity.
The atomic API does support tearing updates, but cursor updates are currently rejected in that path as drivers are not ready for that, and at the same time, current consensus is that tearing is toggled on when a particular fullscreen game demands it, and games composite any cursors in their own render pass so they're unaffected. Drivers will probably support this eventually, but it's not meant to be a general solution.
The legacy API could let some hardware swap the cursor position mid-scanout, possibly tearing the cursor, but just because the call is made mid-scanout does not mean that the driver or hardware would do it.
> but it does add an average of 1 more frame latency
If you commit just in time (display servers aim to commit as late as possible), then the delay between the commit and a tearing update made just before the pixels were pushed is dependent on the cursor position - if the cursor is at the first line shown, it makes no difference, if on the last shown, it'll be almost a frame newer.
Averaging cursor positions mean half a frame of extra latency, but with a steady sampling rate instead of rolling shutter.
Proper commit timing is usually the proper solution, and more importantly helps every other aspect of content delivery as well.
Sure, it's what Gnome Wayland does, but the Wayland protocol does sort of mandate that every frame should be perfect, and the cursor has to match the underlying content, e.g. if it moves over a text it has to change to denote that it is selectable.
> Anyway, what you refer to is the legacy drm interface that was replaced by the atomic one. The legacy interface is very broken and does not expose new hardware features, but it did indeed handle cursors as its own magical entity.
Isn't it what many people refer to as "hardware cursor"? Is it possible for Wayland to rely on such a feature?
Wayland display servers will already be using what is commonly referred to as hardware cursors.
They just use the atomic API to move a cursor or overlay plane, which reflect how the hardware handles things. That the legacy API exposed a specialized cursor API was just a quirk of the design.
Note that planes are a power optimization more than anything else, as it allows e.g. the cursor to move or for decoded video frames to be displayed while GPU's render-related units are powered down. Drawing the cursor move, even though the render task is a rounding error, would require the render-related units to be on.
Thank you. So, if I get this right, the cursor position, which is what the video card needs to position the mouse pointer picture on the screen as an overlay to the actual framebuffer, isn't updated asynchronously to the screen update (ie. whenever the mouse is moved), but instead each time a frame is being rendered, and thus the pointer is only moved at these times, which may avoid tearing (though I don't see why) and other nasty effects, yet introduces a small rendering lag.
I don't know however if the mouse pointer picture is still handled the VESA way, or if GPUs video cards nowadays have a more generic API, or what.
There really isn't such a thing as "the actual framebuffer". Instead the display hardware can do composition during scanout from a set of buffers at a set of positions with varying capabilities. These buffers then just being arbitrary dmabufs.
It doesn't give a damn if you give it 2 buffers and one contains a mouse cursor and the other everything else or if you give it 2 buffers and one is everything including the mouse and the other is a video, allowing complete power collapse of the GPU rendering units.
Often they support more than 2 of these as well, and with color conversions, 1D & 3D LUTs, and a handful of other useful properties. Mobile SoCs in particular, like your typical mid/high end snapdragon, actually have upwards of a dozen overlay planes. This is how Android manages to almost never hit GPU composition at all.
On desktop linux all of these go through the drm/kms APIs.
Well, GPUs from the big players do give a damn as they tend to have surprisingly limited plane count and capabilities. It is often just a single primary, cursor and overlay plane, sometimes the latter is shared across all outputs, and sometimes what the plane can do depends on what the plane it overlaps with is doing.
Mobile chips are as you mention far ahead in this space, with some having outright arbitrary plane counts.
Even though it's only 3 planes, they are relatively feature-rich still. In a typical desktop UI that would indeed be primary, cursor, and video planes. But if the system cursor is hidden, such as in a game, that frees up a plane that can be used for something else - such as the aforementioned game.
What you are showing is just a standard color pipeline, which is the bare minimum for color management.
On AMD in particular, the cursor plane must match several aspects of any plane it overlaps with, including transform and color pipeline IIRC.
The AMD SoC in my laptop (much newer than the steam deck) only exposes two overlay planes to share among all 4 display controllers. Intel used to have a single overlay plane per display.
The Raspberry Pi 5 on the other hand intentionally limited the exposed overlay planes to "just" 48, as it can be as many as you have memory for.
> which may avoid tearing (though I don't see why)
What I meant here is that I didn't see why asynchronous updates may introduce tearing; but my tired brain couldn't quite formulate it properly. And to answer that, it's clear to me know that an update of the pointer position while the pointer sprite is being drawn would introduce a shift somewhere within that sprite, which is, I suppose, the tearing discussed (and not a whole frame tearing).
> I don't know however if the mouse pointer picture is still handled the VESA way, or if GPUs video cards nowadays have a more generic API, or what.
Also, the VESA interface doesn't seem to handle mouse pointers, it's something that was available in the VGA BIOS, to provide a uniform support for this feature, as each vendor most likely did it their own way.
> This is not what Wayland does, it is what a particular display server with Wayland support decided to do.
To the user that's an irrelevant distinction.
I also don't think this matters that much - with X11 this was optimized in one place by people that care about such details while with Wayland now every compositor developer (who in general are much more interested in window management policty) needs to become a low leve performance expert.
> Second, just to be clear, this only discusses mouse cursors on the desktop - not the content of windows, and in particular not games even if they have cursors.
Games can and sometimes do use "hardware" cursors as well - after all, they also care about latency.
It seems like it should be possible to do the X async method without tearing.
When updating the cursor position, check if line being output overlaps with the cursor. If it isn't, it's safe to update the hardware cursor immediately, without tearing. Otherwise, defer updating the cursor until later (vblank would work) to avoid tearing.
Of course, this assumes it's possible to read what row of the frame buffer is being displayed. I think most hardware would support it, but I could see driver support being poorly tested, or possibly even missing entirely from Linux's video APIs.
This would have to be done by the kernel driver for you GPU. I kind of doubt that it's possible (you're not really scanning out lines anymore with things like Display Stream Compression, partial panel self refresh and weird buffer formats), and doubt even more that kernel devs would consider it worth the maintenance burden...
I mean at some point it's a fundamental choice though, right? You can either have sync problems or lag problems and there's a threshold past which improving one makes the other worse. (This is true in audio, at least, and while I don't know video that well I can't see why it would be different.)
Well there are opportunities to do the wrong thing though, like sending an event to the client every time it get an update. Which means that high poll rate mice would DDOS less efficient clients. This used to be a problem in Mutter, but that particular issue was fixed.
You're kind of getting tripped up on terminology. The OP didn't measure Wayland; they measured GNOME Shell which does take responsibility for its performance. Also, I'm not aware of any latency-related mistakes in Wayland/Weston (given its goal of tear-free compositing).
> You're kind of getting tripped up on terminology.
I'm not. My comment doesn't address the latency of gnome shell. I understand the boring technical distinctions between wayland and wayland client libraries and wayland display servers and gnome shell and mutter and sway, blah blah blah. Much like I understand that Linux is a kernel. That it is inspired by UNIX, but it is technically not a UNIX. I also understand that if someone describes themselves as a Linux user they probably don't just mean that they have a Android phone or that the display controller in their dishwasher or wireless access point happens to include Linux the kernel.
The "well acksually wayland is just the name of the protocol" that emerges whenever a problem is brought up is a symptom of the underlying problem with wayland the system. The confusion that gives rise to these deflections is also a symptom of that problem.
By Conway's law systems end up resembling the organisations that produce them. In this way the design of wayland the system seems to be designed by people who don't want to work together. I can see a parallel with microservice architecture.
The distinction between protocol and implementation IS significant here.
Imagine comparing HTTP1.1 vs HTTP3. These are protocols, but in practice one compares implementations. I can pick curl for http1.1, but Python for http3, and http3 would very likely measures as slower.
All this proves is that it's possible for a protocol to not be the determining factor; which says nothing about whether it's possible that it _is_ a determining factor.
You’re quite right. We’d need similar benchmarks done with other compositors.
I very much doubt that Wayland makes a difference for this test; Wayland is for IPC between the client and server. Moving the cursor around is done by the server, without needing to talk to the client.
Which brings up the other problem that Wayland introduced, that instead of one incredibly old inscrutable software stack doing these things there are now five (and counting!) new and insufficiently tested software stacks doing these things in slightly different ways.
It would have been nice if KDE and Valve could (would?) work together to reimplement KWin's features on top of wlroots. That would have basically made Gnome the sole holdout, and I imagine they'd eventually have switched to the extended wlroots as well, or at least forked it.
It most definitely can be because the design of the protocol can force a bound on how performant it can be. While its true that certain implementations of HTTP can be faster or slower, even the best implementations of HTTP 1.1 (for example) are quite slow when you compare it to the alternatives.
I know this because I worked at a place which was entirely based around microservices using Rest HTTP as the protocol/interface, some teams had to use alternate methods because HTTP protocol was the bottleneck.
> I understand the boring technical distinctions between wayland and wayland client libraries and wayland display servers and gnome shell and mutter and sway
I do not. Does anyone know a good but quick introduction into these concepts and the problems they cause or fix?
All I know is that Wayland was supposed to be faster by taking some slow parts out of the loop, but that doesn't seem to be working, according to these figures.
I would just rather pay someone 100 bucks and not care whose fault it is. (And yes, that's why I don't use Linux on the desktop anymore, as much as I loved i3 and tiling environments.)
It's probably in a hundred places that all add up, all of which are no particular person's responsibility. So, that probably means that 10 people from 10 different projects need to get on a call or mailing list together and find a plan of attack.
But it won't happen. And people will keep wondering why Linux can't ever get a foothold on the desktop.
The difference is that x11 was SK bag that you required a tons of plugins to have a working system(xorg) and no one could reliable maintain that mess of codes and plugins where every change could break another
The extension situation is already much worse in Wayland than in X11. In my X11 server there are about 25 extensions and only a handful are required for modern desktop functionality. Wayland already has well over a hundred extensions (protocols) and dozens of them have to be implemented by each compositor just to get basic stuff working.
I have a news flash for you, the situation with extensions in Wayland is ten times worse than X11. Wayland requires an extension for what used to be considered basic functionality in a desktop environment.
Could've fooled me! I've only ever seen Linux devices run Xorg. Maybe some embedded hardware runs an alternative, but Xorg seems to be the de facto standard for desktop Linux.
This isn't a big issue because Xorg splits a basic GUI into dozens of different daemons and services (compositors, window managers, input daemons) so it seems like there are different X11 implementations even when there's usually only Xorg.
Give Wayland a bit more time and it will also ossify to the point where it becomes unviable to create competing implementations. Most of the smaller compositors are already there.
Your biggest strength is also your biggest weakness.
The organisation of Wayland sounds great, but it is very hard to share optimised code between compositors since key parts that affect performance (in this case latency) are largely developed outside of any shared library code.
The "organisation" of Wayland reminds me of the UNIX wars; this is going to get worse before it gets better.
SVR4 Wayland anyone?
xref the time it has taken the Rust rewrite of the GNU coreutils and arguably coreutils is a much easier problem.
Is it a weakness of the web that HTTP is just a protocol specification?
The "problem" with this in Wayland is that before people 'ran Xorg with GNOME on top", now they just run GNOME the same way they run Chrome or Firefox to use HTTP - it will take time for people to get used to this.
I think the answer is also Wayland. It's very confusing because there is wayland the protocol and wayland the system, where the protocol is a part of the system - but there is no separate name for it. When people discuss wayland they are very rarely talking about wayland the protocol, and are much more likely talking about wayland the system.
It's not an "argument", it's just a description of what Wayland is. But no, the correct protocol has had copy-paste since day one, and I dont remember there being issues with screensavers.
In the metaphor of a web server and a web browser, Wayland would be the HTTP specification. What you're usually interested in is what server you're running, e.g. GNOME's Mutter, KDE's Kwin, sway, niri, or what client you're running, e.g. Gtk4, Qt6, etc.
Wayland don't support screensaver and clipboard is managed by portals no?, idk why the complain tho, no one uses screensaver and clipboard being managed by a portal dsounds more logical than creating a protocol for that
On sway, if you use the proprietary Nvidia drivers, it is often necessary to disable hardware cursors. I wonder if there is something similar happening here. Maybe wayland gnome doesn't use hardware cursors?
> Wayland, being something relatively "new" compared to X11, has not had this level of scrutiny for as long. I'm looking forward to folks fixing it though.
Part of the problem will doubtless be the USB (and Bluetooth) stacks including the device hardware and firmware. When keyboards and mice were serial devices with their own interrupt making the code path fast was achievable. I'm not so confident that modern peripheral stacks can be made to run with the same priority. It becomes even more challenging for devices sitting on a bus with multiple device classes or multiple protocols between the device and driver (USB -> Bluetooth -> Mouse).
I hope devices can be sped up but we're a long way from a keypress triggering an interrupt and being handled in tens of milliseconds[0].
As a systems I guy I completely agree with this. One of the things that was interesting about X11 is that it draws an API "wall" around the stuff involved in presentation. Many folks don't remember this but we had X "terminals" which was a computer system that did nothing else except render the screen and handle the input.
In that compartmentalization there was nothing else competing for attention. You didn't get USB bus contention because there was just the one mouse, ever, on the line between the user and the X11 server in the X terminal.
The article you're commenting on is about someone running X11 and Wayland on the same computer with the same mouse and experiencing higher input latency on Wayland. I don't think differences between serial and USB mice are relevant here.
We don't even know if it's actually a 1.5 frame latency for real because the test didn't try to establish that. The author said it looked like it was 1.5 frames, but that could be a coincidence. It just looks like a 1.5 frame overhead. It could, in reality, be a constant time overhead rather than a function of frames and so would be fine with a lower frame rate.
Tearing would affect everyone that uses a computer with X11 but your proposed example of a TV with 30i refresh rate would only affect the tiny subset of users that use a CRT television as a monitor, right?
I don't think there was ever a 30i screen. GP probably meant 60 interlaced, which needs two full scans for a change, so an effective 30 full frames per second, but enough for 60 half images. For a Wayland compositor and the entire hardware stack would treat it as 30hz only if it really wanted to avoid interlace artifacts, but I don't recall that being a configuration that existed in my lifetime.
I'm not sure where this absolute "tearing is bad" and "correctness" mentality comes from. To me, the "correct" thing specifically for the mouse cursor is to minimize input lag. Avoiding tearing is for content. A moving cursor is in various places along the screen anyway, which will include biological "afterglow" in one's eyes and therefore there's going to be some ghosted perception of the cursor anyway. Tearing the cursor just adds another ghost. And at the same time, this ghosting is the reason keeping cursor input lag to a minimum is so important, to keep the brain's hand-eye coordination in sync with the laggy display.
I guess you're asking what happens if the UI element can't be updated in the same cycle?
I'd say: it really doesn't matter at all, in fact it's even more argument to make the pointer as low latency as possible. The mouse pointer is your control element as the controller of the interface. The UI element (and everything else onscreen) is the computer's element. It is an interaction between two entities and there is no need to introduce an expectation that the computer be "zero frame delay" on UI elements. It certainly sucks if it's more than a few frames for it to react, but so be it. However for the mouse cursor itself, it should absolutely be immediate because it is the user's representation on this virtual surface, and as such should follow inputs without being unnecessarily impeded. (Case in point: some UIs intentionally add delay or effects [like a fade-in] to UI element response. It's a… "choice"… but sometimes it works.)
Ultimately, the mouse pointer is an extension of hand-eye coordination; the only thing worse than making it lag (which you can learn to adapt to, albeit it will be very annoying) is to make it lag inconsistently (which you can't adapt to and will grow frustrated with extremely quickly.)
And, unfortunately, from personal experience, the mouse pointer on Wayland gets choppy if system load is high. X11 was significantly better with this; with Wayland the compositor [kwin_wayland to be specific, might be a KDE issue] doesn't even seem to be "unnice" or have some kind of scheduling priority over other tasks — I'm not sure if I'm missing something there but that seems to be a glaring oversight on its own.
Once upon a time XFree86 and Xorg updated the pointer directly in a SIGIO handler. But that's ancient history at this point, and nowadays I wouldn't expect Wayland and Xorg to have a hugely different situation in this area.
IIRC it all started going downhill in Xorg when glamour appeared. After the cursor rendering path wasn't async-safe for execution from the signal handler (which something opengl-backed certainly wouldn't be), the latency was worse.
I remember when even if your Linux box was thrashing your mouse pointer would stay responsive, and that was a reliable indicator of if the kernel was hung or not. If the pointer prematurely became unresponsive, it was because you were on an IDE/PATA host and needed to enable unmask irq w/hdparm. An unresponsive pointer in XFree86 was that useful of a signal that something was wrong or misconfigured... ah, the good old days.
> they sometimes had explicit display hardware for just the mouse because that would cut out the latency
That's still the case. The mouse cursor is rendered with a sprite/overlay independent from the window composer swapchain, otherwise the lag would be very noticeable (at 60Hz at least). The trickery starts when dragging windows or icons around, because that makes the swapchain lag visible. Some window composers don't care any longer because a high refresh rate makes the lag not as visible (eg macOS) others (eg Windows I think) switch to a software mouse cursor while dragging.
This article only tests one compositor for Wayland and one for X11. It also doesn't specify if vsync is enabled for either one. Some X11 compositors default to vsync off, whereas every Wayland compositor I've seen defaults to on. This alone would be enough to explain the latency differences measured here.
I like what's being done here but we need a more thorough experiment before we can start drawing conclusions.
> Let me state up front that I have no idea why Wayland would have this additional latency
It's not really hard to guess at - it's probably caused by abstraction layers resulting in multiple layers of buffering. Of course, it's all just speculation until proven by experiment, so good on TFA for doing some of that.
Retro hardware from the C64 or NES era was single-tasking with such predictable timing you could change the video resolution partway through the screen refresh and have two resolutions on screen at once. If you want, you can check the player input and update Mario's position right before he gets scanned out - the minimum possible latency by design is zero frames. (Of course, in practice the whole screen data is buffered up during vblank, which is the same latency as using a framebuffer. Also the NES didn't allow changing video registers outside of blanking periods. But the C64 did.)
X11 isn't that close to the metal, but it's designed with a single level of buffering (assuming a compositing window manager isn't used). There is a single framebuffer covering the whole screen. Framebuffer updates are completely asynchronous to scanout, so there is around 0.5 frames of latency on average between when a pixel is changed and when it's scanned out. Apps borrow pixels from this big global framebuffer to draw into. An app sends a "draw rectangle" command relative to its own borrowed space, and the server calculates where the rectangle should go on the screen, and draws it into the framebuffer, and then with an average latency of half a frame, that part of the framebuffer is scanned out to the screen.
On Wayland, there are more layers. The app has to draw into its own pixel buffer (noting that it is probably double- or triple-buffered and has to redraw the entire window rather than relying on non-changing stuff to still be there) and then once the compositor receives the buffer from the app, it has to do the same thing again, copying all the per-app buffers into a big screen-size buffer, which is probably also double- or triple-buffered, and once the vblank hits, the most recent full frame is drawn to the screen. It's just more work with more layers, and there's no surprise that latency is higher. It's hardly the first or 100th time that an elegant software architecture resulted in the computer having to do more work for the sake of keeping the architecture pure.
Something important to note is that tearing reduces latency. If new data is available while the scanout is partway through a frame, you have the choice to draw it immediately, causing tearing, or wait until the next frame, increasing latency. Always doing the high-latency thing is an explicit design goal of Wayland. (A third theoretical choice is to always time things precisely so that the new frame is calculated right before the end of vblank, but... in a modular multitasking architecture with apps written by different organizations and unpredictable CPU load, good luck with that.)
Now the really big caveat: I'd be surprised if GNOME's X11 WM wasn't compositing. If it's compositing, then X11 behaves similarly to Wayland but with even more IPC and what I just said is no reason it should have one frame less latency on average. Still something to think about though.
My response assumes that you are talking about the "frame warping" feature of Reflex 2 (saying DLSS 4 was misquoting Nvidia's PR jumble) not simply frame-gen/MFG or upscaling. MFG, while producing even more interpolated frames, does not improve latency and will probably make it worse. I suppose an argument could be made for upscaling in a roundabout may (lower internal resolution = more frames = faster response time) but that would be more true for DLSS 3 as it has a lower GPU overhead than the transformer-based DLSS 4
Anyways, even if it were the case that graphic stack produces a response to the input by warping the frame during interpolation / before rendering, the input that triggers said response would (certius paribus) still go through libinput (and most likely the Wayland compositor) so timing would remain the same. For any features to improve on response time, it would likely have to circumvent one (if not both) libraries.
I also have no ideas how it works, but my guess would be that is actually translates exactly like that into games. Why would the mouse on the desktop have a latency that games do not have. Linux does not have a real fullscreen mode like Windows has (that I do not use anymore anyway for faster alt-tabbing). So my guess is, the cursor or mouse input gets into games very much the same way as it gets in the desktop.
Wayland has support for games completely taking over the GPU; notoriously critical for VR support. This is basically the same as Windows exclusive full screen.
Wine/proton would need to support it, XWayland would need to support it (Wine/Proton are one major step away from native Wayland support: Vulkan), and finally the compositor would need to support it. Gnome is about the worst compositor that you could be testing any of this stuff on, they are comically hostile towards ideas not their own. The chances of this ever working on Gnome are near zero. KDE is pretty good for support, Hyprland seems to be trying to support every under the sun.
> notoriously critical for VR support. This is basically the same as Windows exclusive full screen.
D3D12 dropped support for exclusive fullscreen, and I don't think headsets even go through DXGI but their own swap chain APIs. Why do VR games on Linux need the equivalent from the Wayland compositor?
Can I have the source for this? DX12 dropped exclujsive FS? Why does every game still has this as a option then? Or are you talking about VR specifically?
Games may show it as an option if the engine itself can support other GPU APIs and the devs working on the options menu don't know or care that this particular settings combination is misleading.
> Gnome is about the worst compositor that you could be testing any of this stuff on, they are comically hostile towards ideas not their own. The chances of this ever working on Gnome are near zero.
GNOME has supported direct scanout for fullscreen apps for a while and drm-lease was implemented not too long ago either.
I doubt that Wine would actually request it. Some compositors also have "tearing mode," which is basically as good: the game gets full control of the presentation timing, so vsync/tearing would be determined by what the game wants to do.
Looked up the mouse to see how fast it updates and read that it is wireless. Kind of disappointing and might explain some of the variance. Although if X11 is consistently faster the test should still have some validity.
Would like to see a test with gnome and the wireless mouse removed from the equation.
It depends on the model. Wireless mice don't have to be slower than wired. But even then, for this test it's irrelevant. It's the same mouse in both scenarios, and we only care about the difference. Even if a wired mouse stripped X ms on both sides, the result would be the same.
That's only a problem if the noise of wireless is higher than the signal (measured delay difference). And we can see from the data it isn't. Unless you're claiming it can be consistently different by ~5ms between those two tests... but that's very unlikely. Like, "you should demonstrate it" unlikely.
Why this does not surprise me? every attempt to rewrite a working solution to make it more "modern, easy to maintain and future proof" rarely do so. It always end up slower, with more latency and lifted by faster hardware, not by faster software.
Every 20 years a new generation comes weaker, pampered with the abtsractions of the previous generations who did the grunt work for them. After 3 generations of software developers all we have is library/framework callers and nobody really knows about performance and optimization.
This is not my experience. When I upgraded several of my computers to KDE 6 switching to Wayland, overall responsiveness and snappiness of the system increased very visibly. There are still features that Wayland lacks compared to X11, but I am willing to compromise in favor of its other benefits.
On the other hand, I just did a fresh install of Artix Linux. Installed KDE just to have something functional while I get my tiling setup working. Boot into Plasma(Wayland) and it utterly shits itself, turning my main monitor on and off repeatedly until it finally crashes. So I pick Plasma(X11) instead and that just works.
In fact, in almost 2 decades of using Linux every day, I can't remember X doing anything similar. It's always just worked.
> In fact, in almost 2 decades of using Linux every day, I can't remember X doing anything similar. It's always just worked.
Well, we have very different memories then. Sure, X worked reliably once configured. But configuring it was a marathon in hell, as per by my memory, and all the litany of forum posts crying out for help all across the internet. Like, I have at one point had to rescue an install by changing back the config file without a screen at all!
Yes. Grandparent talked about 20 years ago. 15 years ago Ubuntu required no configuration at all if you had a graphics card that worked (most newer ones did, Nvidia was a nightmare).
I mean, I've certainly had issues with X before, back when you had to configure it. But they were usually initial issues while figuring out the right configuration. And if you had a GPU where the choice was between shitty proprietary blob drivers and woefully incomplete open ones, you might have a bad experience regardless of what you put in the config. But that's not really Xorg's fault. It can't magically make shitty drivers good. With decent drivers, once configured, it was rock solid, and has been for as long as I've been using Linux. And for a long time the configuration has been addressed; it now figures that stuff out on its own the vast majority of the time, unless you have really specific needs, in which case I bet Wayland compositors don't fare any better.
I also fundamentally disagree with the idea that X "does too much", which is often cited in favour of Wayland. The fact that X encompasses everything about how I interact with my computer is one of the things I love about it. I might switch WMs and DEs, but my xmodmap, xresources etc remain the same.
I used X11 in the early 90s (HP-UX) and since 2009 (Ubuntu until 22.04 made me switch to Debian.) I never had to configure anything. Maybe some xmodmap in the 90s. I did everything with dialog windows in this century, like I would do it in Windows.
Same here, switching from i3 to sway resulted in a noticeably more responsive experience on my aging hardware. Of course, this is just an anecdote, and I could probably get the same results on X with some fiddling, but I do think the value of simpler and more modern systems is demonstrated here.
Old computers had less latency, but otoh on many OSes a single app crashing meant the whole OS was irresponsive and you had to reboot the whole system.
Yes, I was expecting some software that listens directly to the mouse input, and watches a pixel on the screen. Messing around with cameras and counting frames introduces a whole bunch of other variables like the quality of the monitor and mouse and phone used for recording.
The camera, mouse, and monitor all stayed the same for the tests, but there was a significant difference in latency. Out of the 16 times the experiment was run, only once did Wayland have lower latency. It would be an amazing coincidence if the monitor, mouse, and/or camera were the reason for this.
Figuring out why there's increased latency is a job for software tooling, but I think this guy's experiment is one of the best ways to measure what users care about: The time it takes for an input (such as mouse movement) to result in a change (such as cursor movement).
Note that this doesn't mean that the Wayland protocol itself is the reason for the higher latency. It may be Gnome's implementation (testing with a wlroots compositor might shed light on this). It may be differences in default configuration options. It may be that Wayland and X11 start up different services, and the Wayland helper processes increase load on the machine. But I seriously doubt the reason for the difference in latency was because the same hardware was used throughout the experiment.
The mouse and monitor don't matter here. Unless their delay completely dominates the timing (it doesn't here) they can be ignored because the setup is constant been the tests. We're interested in the difference, not the absolute numbers.
I got results with a P-value of under 0.001. That should be enough to demonstrate that there's a real difference.
Using a camera allows the methodology to he identical between Wayland and X, while I don't know how to listen for mouse movements from software in a way that wouldn't introduce its own problems. What if the photons are emitted from the screen after the same number of milliseconds across X and Wayland, but Mutter is a few milliseconds slower about notifying my measurement application? Conversely, what if Mutter has more real latency than X, but the latency is introduced after the stage where my measurement application sits?
The variables you mention are identical between the Wayland test and the X test. It is admittedly a challenge for reproducibility, but doesn't affect these results.
500 comments
[ 2.6 ms ] story [ 342 ms ] threadI think there may still be the issue that many compositor/GPU combinations don't get hardware cursor planes, which would definitely cause a latency discrepancy like this.
Hell, it's been 12 years and not a single wayland compositor supports screen readers for the visually disabled yet. They are toys.
One of the really cool things that has come out of this is seeing how people are adding accessibility workflows on top of the ecosystem. There is one person who controls their entire desktop via the tiling window manager using voice control![2]
[1]: https://github.com/LGUG2Z/awesome-komorebi
[2]: https://youtu.be/fiPJLmhnnXM
So no good for games, no good for professional graphics, no good if you don't see well... basically no good if you're any different from the people who hacked it together.
But, hey, with any luck they cut down on the screen tearing that I've never noticed.
Why would you expect it any different? How can one implement things that they have no need or no hardware for? The entitlement is a bit jarring.
Also I think they merged something last year: https://github.com/swaywm/sway/pull/7681
[1] https://gitlab.freedesktop.org/wayland/wayland-protocols/-/m...
That being said, thanks to Nvidia I never got colour calibration to work right in X11 either, so I have no horse in this race. Would be cool to finally get HDR working for the first time, but I don't know if that'll ever happen on Linux with Nvidia hardware. Guess I should keep dual booting for HDR content until either Windows 10 dies off or I get new hardware.
I do actually notice the lack of tearing in Wayland, especially under heavy load. Used to annoy me to no end to see my windows tear when just dragging them across a 1080p screen. I don't know if it was an Intel driver bug (I tried all the config settings) or something X11 specific, but Wayland did finally fix the tearing issues I had.
I haven't noticed any problems with colours in either Gnome or Gamescope (except for the lack of HDR, of course, but that's also true on X11) so whatever is causing issues for you seems to be DE specific. Looks like we both have issues other people never encountered, that's what makes graphics stacks so impossible to debug and code for.
Color calibration can absolutely be retrofitted into this (versioned) protocol, and there is work ongoing.
It feels like you're probably blaming the wrong people here. You should look at the companies that make this peripherals that don't also offer you linux drivers.
The only hardware I know for sure is lacking is stuff like Stream Decks, but that kind of hardware is difficult to generalise for.
Plus, most working hardware that doesn't fit a standard HID protocol already has drivers in the Linux kernel written by manufacturers.
I know that Georges Stavracas has been working on a controller for the Elgato devices for a while (https://flathub.org/apps/com.feaneron.Boatswain) though it needs direct access to the devices instead of through some generic driver (https://gitlab.gnome.org/World/boatswain#udev-rules).
I haven't tried any games that use a cursor with Wayland yet so I don't know if it would have an impact there.
I think it has to do with whether or not the game in question is reading the mouse device directly (e.g. through SSL) or via the compositor. If it's reading the device directly it stands to reason that there would be less latency.
Suppose two players notice each other at the same time (e.g. as would naturally happen when walking around a corner in a shooter), first to shoot wins, and their total latencies are identical Gaussians with a standard deviation of 100ms. Then a 6.5ms reduction in latency is worth an additional 2.5% chance of winning the trade. Maybe you won't notice this on a moment by moment basis, but take statistics and its impact should be measurable.
In ELO terms a 2.5% gain in win rate is around a 10 point increase (simplifying by assuming that single Gaussian is the entire game). That's small, but if you were a hardcore player and all it took to raise your ELO by 10 points was using a better monitor/mouse/OS... why not? Doing that is cheap compared to the time investment required to improve your ELO another 10 points with practice (unless you're just starting).
Also, I think you'd be surprised what people can perceive in a context where they are practiced. Speed runners hit frame perfect tricks in 60FPS games. That's not reaction time but it does intimately involve consistent control latency between practice and execution.
> Suppose two players notice each other at the same time (e.g. as would naturally happen when walking around a corner in a shooter)
This is not true for third person games. Depending on a left sided or right sided peek and your angle or approach, players see asymmetrically.
For example, Fortnite is a right side peek game. Peeking right is safer than peeking left as less of your body is exposed before your camera turns the corner.
I believe distance also plays a part in the angles.
https://www.youtube.com/watch?v=OX31kZbAXsA
> “I was working with Larry Mullen, Jr., on one of the U2 albums,” Eno told me. “ ‘All That You Don’t Leave Behind,’ or whatever it’s called.” Mullen was playing drums over a recording of the band and a click track—a computer-generated beat that was meant to keep all the overdubbed parts in synch. In this case, however, Mullen thought that the click track was slightly off: it was a fraction of a beat behind the rest of the band. “I said, ‘No, that can’t be so, Larry,’ ” Eno recalled. “ ‘We’ve all worked to that track, so it must be right.’ But he said, ‘Sorry, I just can’t play to it.’ ”
> Eno eventually adjusted the click to Mullen’s satisfaction, but he was just humoring him. It was only later, after the drummer had left, that Eno checked the original track again and realized that Mullen was right: the click was off by six milliseconds. “The thing is,” Eno told me, “when we were adjusting it I once had it two milliseconds to the wrong side of the beat, and he said, ‘No, you’ve got to come back a bit.’ Which I think is absolutely staggering.”
I didn’t think it was. But it is. I promise!
It’s not necessarily a reaction-time game-winning thing. It’s a feel.
With virtual instruments, my experience is that when you get down to ~3ms you don’t notice the latency anymore… but!, when you go below 3ms, it starts feeling more physically real.
That has not been proven in the article. Input handling follows different paths for full screen games.
It would be more of a problem the other way around, if we had to resort to Wayland to get low latency. I think most of us using Linux for gaming and casual stuff are happy to stick to X11 for now and the foreseeable future. It has good support in software, its failure modes are well-documented, and it doesn't add one more layer to the pile of complexity that desktop linux already is; at least, not one us users have to consciously keep in mind as a common source of issues.
https://github.com/ValveSoftware/csgo-osx-linux/issues/3856
From outside it's hard to tell if it's truly protocol differences or just the age of the implementations on X11, but when Wayland came out every project has claimed improvements over the old X11 stack. Also, from the early Wayland days presentations bashed the protocol as something that couldn't be fixed without a rework that was not going to happen due to the dead weight of backwards compatibility and awful older hardware.
As a user applications running on Wayland have consistently improved on how nice things feel if you don't miss your latency deadlines. It's easy to perceive on apps, and straight out obvious in games.
...Which I don't get because the Xe driver is said to explicitly support, at minimum, Tiger Lake. I played Minecraft on the iGPU with Xe and it was perfectly fine. It... drew 3D graphics at expected framerates.
> With my 144Hz screen,....Wayland, on average, has roughly 6.5ms more cursor latency than X11 on my system...Interestingly, the difference is very close to 1 full screen refresh. I don't know whether or not that's a coincidence.
The fact that the latency is almost 1/144th of a second means that it might become 1/60th of a second on standard 60Hz monitors. This is hard to notice consciously without training, but most people can "feel" the difference even if they can't explain it.
My guess: the "true" numbers are close to 2.5 (half a frame of random phase of when the mouse is touched vs. refresh, plus 2 frames to move cursor) and 3.5. If you throw out the low outlier from each set you get pretty close to that.
(of course, the 125Hz mouse poll rate is another confound for many users, but this guy used a 1KHz mouse).
> This is hard to notice consciously without training, but most people can "feel" the difference even if they can't explain it.
Yah. 7ms difference is not bad vs 16.6ms is starting to be a lot.
IMO, we should be putting in effort on computers to reach 1.6 frames of latency -- half a frame of random phase, plus one frame, plus a little bit of processing time.
[1] https://raphlinus.github.io/ui/graphics/2020/09/13/composito...
[2] http://number-none.com/blow/john_carmack_on_inlined_code.htm...
Put another way, a system fast enough to composite at 144Hz should be able to composite at 60Hz while only allocating 1/144 seconds to the compositor, which would require offsetting the presentation times as seen by the compositor’s clients by some fraction of a frame time, which doesn’t actually seem that bad.
It gets even better if variable refresh rate / frame timing works well, because then frames don’t drop even if some fraction of compositing operations are a bit too slow.
I assume I’m missing some reason why this isn’t done.
We need to cut some deadline and doing it at vsync is the easiest way
Anything related to the compositor won’t affect the game, unless it’s in windowed mode, then there can be some strange interactions.
In windowed mode they can be captured, then things get tricky, but in full screen they draw direct to the gpu, vulkan itself is completely headless.
Gamescope's direct scanout bypasses this.
I wasn’t aware that this changed, but you could be right. Its definitely the same on Windows as it always was, which is the platform I most developed games for.
Not really. Most games use "borderless windowed" mode instead of fullscreen exclusive, and even FSE is not true exclusive mode anymore in most cases.
https://devblogs.microsoft.com/directx/demystifying-full-scr...
> When using Fullscreen Optimizations, your game believes that it is running in Fullscreen Exclusive, but behind the scenes, Windows has the game running in borderless windowed mode. When a game is run in borderless windowed mode, the game does not have full control of the display– that overarching control is given back to the Desktop Window Manager (DWM).
see also https://learn.microsoft.com/en-us/windows/win32/direct3ddxgi...
If an application framebuffer is full screen and in a compatible pixel format the compositor can do "direct scan out" where the compositor sends the framebuffer directly to the crtc instead of compositing first. I know that wlroots supports that. I'm not sure how much performance it saves to be honest.
And I'll second that most commercial games do go through XWayland, though it depends on what games you like!
https://github.com/ValveSoftware/gamescope
I also find Wayland more laggy and more buggy without concrete evidence of course.
It has significantly improved lately though, far more stable than it used to be.
Not arguing though, you are right it is just issues: Drivers, the Wayland implementations, how some plethora of apps and libraries have been battle tested then optimized, years over years, for X11. Not as much for Wayland display yet.
I think it's safe to assume that it is actually simpler, given that we already have multiple Wayland implementations, but still basically just the one X11 server implementation. Can one or more of those implementations shave off one or two milliseconds overs the next 40 years... Probably yes.
footnote: there is something wrong with the commonly used web search engines and I am unable to find that X11 web demo, I think they are prioritizing recent content over good content. you would think with how interesting that web demo was it was it would show up right away. but I got nothing, so no links. Anyway it was a deep dive into the intricacies of X11 window painting and the author had ported a limited subset of an X server to javascript in order to provide a live demonstration of the topic. I will keep looking.
Found it. https://magcius.github.io/xplain/article/x-basics.html
It is a common problem with these "simple" things. The problem is complex, and if you simplify one part, you are just pushing complexity elsewhere, you are not making it disappear. It is sometimes a good thing, but in the case of Wayland, it didn't go well.
In any case wayland is not bad if you only have pretty basic needs I guess, some basic things look easier to me there from a user perspective, and troubleshooting x11-related issues for a non-technical person is no fun either.
It was designed to fix tearing issues, not latency issues.
But then the Wayland designers found out that players often prefer tearing to minimize latency, so the tearing protocol was implemented.
When a committee of perfectionists, rather than real people or companies, design something, you often get something that is completely unusable by anyone but that committee.
And that's exactly how it's played out so far. Wayland is still largely incomplete, not to mention it doesn't even have a reference implementation [1], and still doesn't include/describe several essential desktop APIs and features, which results in this protocol not having a ton of universal tools and applications that work across all of its implementations, including but not limited to:
It's a damn ugly mess that has led to a lot of fragmentation with no obvious benefit. Currently, only KDE and Gnome have somewhat usable Wayland implementations. If you use anything else? You're SoL.1. https://gitlab.freedesktop.org/wayland/wayland/-/issues/233
It might be a better technical design to have the other stuff outside of the display protocol. Just because Xorg implemented something does not mean you have to put it in the Wayland protocol.
In any of these cases there may be one or more daemons behind the scenes handling the “raw” input—possibly even in cooperation with kernel-level pre-processing code, to ensure low latencey—but most event delivery to applications is associated with windows, with options to get lower-level access if needed.
One of the things that helps many of the systems described above with latency is kernel participation, whether by pushing much of the preprocessing of events down to the drivers so there’s little for userspace to do, or by implementing kernel-level zero-copy IPC (e.g. use of Mach messages by NeXT and Apple).
If human interface IPC happens entirely in userspace and requires multiple context switches to get an event from device to a display change, you’ll wind up with hitches and delays unless there’s some sort of scheduler hinting that ensures each stage in the pipeline runs immediately after the last.
This is, of course, why there was a lot of desire by Wayland advocates for kernel dbus support, but they went at the problem backwards: “Let’s take DBus, and make it fast by putting it in-kernel,” *without* first trying to make it as fast as possible without kernel support, *and* without trying to figure out the minimal feature set for kernel-level IPC that would be needed to support it (which may not look like DBus).
How is that bad design?
Actually is was designed because the X11 codebase was bad and nobody wanted to work on it.
This sounds like "Wayland codebase is good and everybody wants to work on it".
Wlroots exists.
Good luck using it as your graphics subsystem.
(Hi, author here by the way! ... Don't worry, that disabling hardware cursors thing was at least one OS re-install ago)
Mutter definitely throttles cursor updates on Wayland, too, which will contribute slightly to latency even with a hardware cursor. In general, with Wayland, the compositor is responsible for cursor updates, and I'm not sure which other ones throttle. But that would be where the difference comes from when using hardware cursors.
I think the difference mort96 is seeing is the cursor update throttling. If it updates at the same rate of the refresh rate then it's a crapshoot where in that interval it hits relative to vsync, with a worst case additional latency of the frame rate. X11 updates the cursor whenever it moves, so on scanout it's always where you expect it to be, even if the motion was almost immediately before vsync.
I should mention that in the past there's been problematic interactions on amdgpu with atomic updates of the cursor and display planes. This resulted in either one being starved from changing, causing stuttering in one or the other depending how it was handled. IIRC, that's why the throttle is there. You could try MUTTER_DEBUG_ENABLE_ATOMIC_KMS=0 to see if they only added the throttle to the atomic code path.
It's all software.
The two sample t-test statistic is -4.74 with a p-value of 4.20e-05.
Possibly because the best contributors don’t care about GUI anyway.
Otherwise, it's actually far more advanced than Wayland, and it had essential modern features implemented years ago, including HDR, VRR, and others.
Why hasn't it been ported to Linux? Probably NiH, probably it's Google's child.
What would be required to make surfaceflinger a practical Linux desktop compositor is merely some translation layer, equivalent to XWayland, that supports hardware acceleration. Such things have been written, but not open source and they never got traction.
The idea has been toyed with for over a decade: https://news.ycombinator.com/item?id=5317638
It runs on Android/Linux, not GNU/Linux. I would be surprised if there wasn't at least a bit of work needed to handle things like bionic vs glibc, and you'd really want it to act "normal" by being installed to FHS paths instead of Android's weird filesystem layout. All doable, I expect, but it would be a port.
Let me state up front that I have no idea why Wayland would have this additional latency. That said, having been active in the computer community at the 'birth' of X11 (yes I'm that old) I can tell you that there was, especially early on, a constant whine about screen latency. Whether it was cursor response or xterm scrolling. When "workstations" became a thing, they sometimes had explicit display hardware for just the mouse because that would cut out the latency of rendering the mouse in the frame. (not to mention the infamous XOR patent[1])
As a result of all this whinging, the code paths that were between keyboard/mouse input and their effect on the screen, were constantly being evaluated for ways to "speed them up and reduce latency." Wayland, being something relatively "new" compared to X11, has not had this level of scrutiny for as long. I'm looking forward to folks fixing it though.
[1] https://patents.google.com/patent/US4197590
A fullscreen app ought to be taking a fast path through / around the compositor.
I'll be reading a dream of spring in my grave at this rate.
I understand I'm complaining about free things, but this is a forced change for the worse for so long. Wayland adoption should have been predicated on a near universal superiority in all input and display requirements.
Intel and AMD and Nvidia and Arm makers should be all in on a viable desktop Linux as a consortium. Governments should be doing the same because a secure Linux desktop is actually possible. It is the fastest path to showcasing their CPUs and 3d bling, advanced vector /computer hardware.
Wayland simply came at a time to further the delay of the Linux desktop, at a time when Windows was attempting to kill Windows with its horrid tiles and Apple staunchly refused a half billion in extra market cap by offering osx on general x86.
There's rarely any such thing as "universal superiority", usually you're making a tradeoff. In the case of X vs Wayland it's usually latency vs. tearing. Personally I'm happy with Wayland because there was a time when watching certain videos with certain media players on Linux was incredibly painful because of how blatant and obtrusive the tearing was. Watching the same video under Wayland worked fine.
Early automobiles didn't have "universal superiority" to horses, but that wasn't an inhibitor to adoption.
-Problems with non western input systems
-Accessibility
-Remote control(took around 2 years to be stable I think?)
-Bad color management
Then there's the things that did work in x11 but not in wayland:
-Bad support for keymapping(the input library says keymapping should be implemented by the compositor, gnome says not in scope, so we have a regression)
-bad nvidia support for the first two years? three years?
While these things are compositor/hw vendor faults, the rush to use wayland and nearly every distro making it as default, forced major regressions and wayland kinda promised to improve the x11 based experience.
I get there was cruft in x. But the moment selected was a barrier to Linux desktop adoption precisely when the greatest opportunity in decades was present.
And the desktop was reimplemented.
Now in this period kde and gnome both decided to do rewrites, Ubuntu did their own desktop, got it up to snuff, and abandoned it. The lunacy wasn't just Wayland.
If we are complaining the gnome compositor sucks... I mean , should that be the goddamn reference implementation? What percent of desktops are gnome, 80% at least? If the gnome composting ready for primetime, then Wayland isn't ready for primetime.
I use Sway, which uses a different compos[i]tor than Gnome. I would like to see similar results for wlroots, Sway's compositor, though I'm not actually interested enough to do the experiment (I guess that would be comparing Sway with i3). Cursor lag in Sway is not enough to bother me. I have on occasion used Gnome on the same machine(s), and never been bothered by lag.
As others have pointed out, Wayland is a protocol, not a compositor.
But the Wayland protocol requires a compositor, so here we are.
Given that Gnome runs on both X and Wayland, it might be interesting to hear from the Gnome authors on the performance differences.
With regards to accessibility, what problems have you had exactly?
Fedora shipped a broken screen reader for 8 years.
Edit: Found it https://ar.al/2024/06/23/fedora-has-been-shipping-with-a-bro...
Not a bug, apparently. https://gitlab.freedesktop.org/libinput/libinput/-/issues/10...
That may be fine.
Neo2 does not work. Neo has 3 modifier keys, Gnome/Mutter/Wayland/Whatever does only support two. Neo2 has a compose key, Wayland does not honor it.
https://neo-layout.org/
I use mod4 for navigation (arrow keys, site up) and compose for Slavish (read Polish) input (źżąę)
Which rush? It has been done by only a small fraction of distros like Fedora, after years of development of the first wayland compositors. Fedora main purpose has always been to implement bleeding edge tech stuff early so that bugs get found and fixed before people using more stable distros have to suffer from it.
Nobody has been forced in any regression and x11 has continued to be available until now and there is no sign that the most conservative distros will drop x11 support anytime soon.
Not that I mind particularly, X is fine and everything works as expected. I can even have the integrated (amd gpu) take care of the desktop while the Nvidia gpu does machine learning (I need all the vram I can get, the desktop alone would be 100-150mb of vram) - then I start a game on steam and the nvidia gpu gets used.
Funnily enough I had wayland enabled by default when I installed the system and I didn't understand why I was getting random freeze and artifacts for weeks. Then I realized I was not using X11
I feel like at some point this is a cop-out. Wayland is a protocol but its also a "system" involving many components. If the product as a whole doesn't work well then its still a failure regardless of which component's fault it is.
Its a little like responding to someone saying we haven't reached the year of linux on the desktop by saying: well actually linux is just the kernel and its been ready for the desktop for ages. Technically true but also missing the point.
Then a client is going to query the server and ask request to do stuff via a unix socket. In fact, you don't even need libwayland, you can raw dog it over sockets manually. The idea is that there are standard protocols that can be queried and used,and you can implement this in any environment you want to. You could write the "frontend" in html and JS and run a wayland compositor on the web (which has been done [1]), you could do it with text or anything really, most people use some graphics stack.
[1] https://github.com/udevbe/greenfield
- the compositor, of which there are multiple implementations (gnome, kde, all the wlroots compositors)
- the client, which often uses one of several toolkits (gtk, qt, several smaller frameworks, or even directly using the protocol)
- the wayland protocol (or rather protocols, because there are several extensions) itself
- other specifications and side channels for communication, in particular dbus.
Many issues (although I don't think the one in OP) are due to the protocol being underspecified, and so the client and compositor disagree about some semantics, or doesn't even have a standard way to accomplish something across all compositors.
If there were a single Wayland implementation in existence, I’d agree with your sentiment.
It's also the implementation most unsuspecting users will end up with.
With X11, it was simple: everybody used Xfree86 (or eventually the Xorg fork, but forks are not reimplementations) and libX11 (later libxcb was shimmed underneath with careful planning). The WM was bespoke, but it was small, nonintrusive, and out of the critical path, so bugs in it were neither numerous nor disastrous.
But today, with Wayland, there is no plan. And there is no limit to the bugs, which must get patched time and time again every time they are implemented.
For example Qt does per-monitor DPI just fine on X11; it's just that the way to specify/override the DPI values just sucks (an environment variable).
This stupid decision is going to chase us until the end of times since Xwayland will have no standardized way to tell its clients about per-display DPI.
This is something feasible on Wayland, X draws one large wide screen display.
In a way, Wayland in this case developed a solution for issue its creators brought into this world first
° For most programs.
Most people want to drag windown between screens and sometimes even split down the middle. One large display supports that much easier so that is what everyone switched to in the late 1990
X's "mechanism, not policy" has proven to be a failure. Without a consistent policy you end up with a pile of things that don't work together. "One big virtual screen" is a policy and it's one that works well enough.
Deciding what needs to be in the protocol and what should be in the application is never easy. It's important to be able to iterate quickly and avoid locking in bad practices. I think that's what Wayland tried to do by making everything a fine-grained extension, but it doesn't really work like that as the extensions become mandatory in practice.
Windows does this. Try to use in Windows 2 monitors with 2 different scalling factors. It is hit or miss. 100 and 150 works. 100 and 125 doesn't.
But you can definitely move windows to another monitor and Qt will use the right DPI for it. It is the same behavior as Wayland. "One large wide screen display" is exactly how Wayland works...
That, i think, is the main issue. Nobody wants to work with GTK1, or GTK2, or GTK3 anymore. Nobody wants to work with QT1, or QT2, or QT3 or QT4 anymore. Everybody wants the new shiny toy. Over and over again.
It is CADT all over. Earlier X was developed by an industry consortium. Now Wayland is a monopoly pushed by RedHat.
Pushed but still it seems it flails and takes too long to do the basic stuff
Can we trust them to "care" about desktops?
Oh wait. There is no more RedHat. There is IBM.
It was because of the crap LCD monitors (5 to 20 ms GtG) and how they are driven. The problem persists today. The (Wayland) solution was to render and display a complete frame at a time without taking into account the timings involved in hardware (you always have a good static image, but you have to wait).
I tried Tails (comes with some Wayland compositor) on a laptop. The GUI performance was terrible with only a Tor browser open and one tab.
If you do not care about hardware, you will, sooner or later, run into problems. Not everybody has your shiny 240 Hz monitor.
I can't tell if you are serious or not.
About 16 years old, for comparison, X is 40.
I've been hearing this for over a decade now. I don't get it. Just because xorg currently makes different clients aware of each other and broadcasts keypresses and mouse movements to all clients and allows screen capturing doesn't mean it has to. You could essentially give every application the impression that they are the only thing running.
It might seem difficult to implement, but compare it to the effort that has gone into wayland across the whole ecosystem. Maybe that was the point - motivating people to work on X was too difficult, and the wayland approach manages to diffuse the work out to more people.
I was really bullish on Wayland 10 years ago. Not so much any more. In retrospect it seems like a failure in technical leadership.
[1]: The people who actually developed Xorg are now working on various Wayland-related things.
I suspect it would be less challenging than writing a whole new wayland server.
Off the top of my head, I'd use a separate abstract domain socket for the window manager including some UUID, and then pass that to the window manager when launching it.
You could create these sockets on demand - one for each security context. On linux typically a different security contexts will either have different UIDs - in which case filesystem permissions would be sufficient - or they have different mount namespaces - in which case you make different sockets visible in different namespaces.
For SSH forwarding you could have SSH ask the X server for a new socket for forwarding purposes - so remote clients can't snoop on local clients.
> Good luck on your lonely[1] journey! > > [1]: The people who actually developed Xorg are now working on various Wayland-related things.
This is what I mean by a failure of technical leadership.
This is reminiscent of how Trusted Solaris[0] implements Mandatory Access Control (MAC) a la Orange Book[1].
[0] https://www.oracle.com/technetwork/server-storage/solaris10/... [1] https://public.milcyber.org/activities/magazine/articles/202...
SSH pretty much already does this. Per default (using -X) X11 forwarding is in untrusted mode, which makes certain unsafe X11 extensions unavailable. So remote clients already cannot snoop the whole keyboard input.
And then the window manager spawns whatever programs the user wants, and they end up sharing that $DISPLAY.
1. There is nothing fundamentally insecure about the design of X that couldn't be fixed with a bit of effort.
2. The effort required would be significantly less than what has gone into Wayland over the last 16 years.
Fixing the security of the system certainly would involve changes to more than just xorg. Anywhere there is a security boundary you'd need to make modifications - but that's OK, most of these security boundaries are younger than wayland anyway.
This discussion of $DISPLAY seems like distracting minutiae. Sure it's a medium size problem that would need solving, but that's all it is.
To address your specific point: it could be the responsibility of the window manager to set $DISPLAY to a less powerful socket when starting processes. Ultimately it doesn't matter though, because if the display server isn't doing some sort of sandboxing then the spawned process can just ptrace xorg and do whatever it wants. X being secure only matters in the presence of a security boundary and in that case it would be the responsibility of whatever is setting up that boundary to create a less privileged socket. Whether that be ssh or flatpak or whatever.
Once again, for emphasis: nobody has stepped up to do that, in the history of X (outside of proprietary products, of unknown quality).
Wayland is not some outsider project competing against X11. It's the people who were developing Xorg, saying they're fed up with X11 and need a clean slate to fix its problems.
Here's my unpriviledged user trying to ptrace my Wayland compositor running as me:
Nearby, I have a Chromebook where a potentially hostile virtual machine can display Wayland windows seamlessly on the host. The worst the VM can do is fill bitmaps with offensive imagery.In a world where the people doing the work switched to developing the Wayland ecosystem, arguing about a "could" that requires changing every window manager seems like a guarantee X11 will not get improved. Feel free to put effort into it...
> It might seem difficult to implement
This is exactly what OS did (my daily driver): https://forum.qubes-os.org/t/inter-vm-keyboard-isolation/315...
Can you explain who is forced to do what in that context?
Coincidentally I have got a graphics driver that likes crashing on OpenGL (AMD Ryzen 7 7840U w/ Radeon 780M Graphics)
Totally agree. The people saying "Wayland is a protocol" miss the point. Wayland is a protocol, but Wayland adoption means implementing stuff that uses that protocol, and then pushing it onto users.
Measure twice, cut once. Look before you leap. All that kind of thing. Get it working FIRST, then release it.
(Also, I don't use Wayland. I mean I tried it out but don't see any real benefit so I don't use it regularly.)
30 years ago X came with a server and some clients. Why it is so hard to do this, for wayland, today ?
Then write code.
Asahi Lina has demonstrated that a single person can write the appropriate shims to make things work.
Vulkan being effectively universally available on all the Linux graphics cards means that you have the hardest layer of abstraction to the GPU taken care of.
A single or small number of people could write a layer that sits above Wayland and X11 and does it right. However, no one has.
This argument don't make sense because Wayland started as a hobby and not to replace x11, was after it got traction, other people/companies started contributing that it matter
With complex projects like these that have to work well on such a wide array of hardware and configurations, lots of real world usage is required to achieve any level of refinement. Without widespread adoption, Wayland likely would have been stuck in experimental/toy status for much longer than it will as things are going currently.
Planes can be updated and repositioned without redrawing the rest of the screen (the regular screen image is on the primary plane), so moving the cursor is just a case of committing the new plane position.
The input latency introduced by GNOME's Mutter (the Wayland server used here) is likely simply a matter of their input sampling and commit timing strategy. Different servers have different strategies and priorities there, which can be good and bad.
Wayland, which is a protocol, is not involved in the process of positioning regular cursors, so this is entirely display server internals and optimization. What happens on the protocol level is allowing clients to set the cursor image, and telling clients where the cursor is.
Hopefully this general concern doesn't apply to Wayland and the "shape" you have described doesn't sound bad, but the devil is in the details.
The story is different for applications like games that hide the system cursor to display their own. In those cases, the client needs to receive mouse events from the compositor, then redraw the surface appropriately, all of which does go through Wayland.
Second, just to be clear, this only discusses mouse cursors on the desktop - not the content of windows, and in particular not games even if they have cursors. Just the white cursor you browse the Web with.
Anyway, what you refer to is the legacy drm interface that was replaced by the atomic one. The legacy interface is very broken and does not expose new hardware features, but it did indeed handle cursors as its own magical entity.
The atomic API does support tearing updates, but cursor updates are currently rejected in that path as drivers are not ready for that, and at the same time, current consensus is that tearing is toggled on when a particular fullscreen game demands it, and games composite any cursors in their own render pass so they're unaffected. Drivers will probably support this eventually, but it's not meant to be a general solution.
The legacy API could let some hardware swap the cursor position mid-scanout, possibly tearing the cursor, but just because the call is made mid-scanout does not mean that the driver or hardware would do it.
> but it does add an average of 1 more frame latency
If you commit just in time (display servers aim to commit as late as possible), then the delay between the commit and a tearing update made just before the pixels were pushed is dependent on the cursor position - if the cursor is at the first line shown, it makes no difference, if on the last shown, it'll be almost a frame newer.
Averaging cursor positions mean half a frame of extra latency, but with a steady sampling rate instead of rolling shutter.
Proper commit timing is usually the proper solution, and more importantly helps every other aspect of content delivery as well.
I do believe it is a useful tradeoff, though.
Isn't it what many people refer to as "hardware cursor"? Is it possible for Wayland to rely on such a feature?
They just use the atomic API to move a cursor or overlay plane, which reflect how the hardware handles things. That the legacy API exposed a specialized cursor API was just a quirk of the design.
Note that planes are a power optimization more than anything else, as it allows e.g. the cursor to move or for decoded video frames to be displayed while GPU's render-related units are powered down. Drawing the cursor move, even though the render task is a rounding error, would require the render-related units to be on.
I don't know however if the mouse pointer picture is still handled the VESA way, or if GPUs video cards nowadays have a more generic API, or what.
It doesn't give a damn if you give it 2 buffers and one contains a mouse cursor and the other everything else or if you give it 2 buffers and one is everything including the mouse and the other is a video, allowing complete power collapse of the GPU rendering units.
Often they support more than 2 of these as well, and with color conversions, 1D & 3D LUTs, and a handful of other useful properties. Mobile SoCs in particular, like your typical mid/high end snapdragon, actually have upwards of a dozen overlay planes. This is how Android manages to almost never hit GPU composition at all.
On desktop linux all of these go through the drm/kms APIs.
Mobile chips are as you mention far ahead in this space, with some having outright arbitrary plane counts.
Even though it's only 3 planes, they are relatively feature-rich still. In a typical desktop UI that would indeed be primary, cursor, and video planes. But if the system cursor is hidden, such as in a game, that frees up a plane that can be used for something else - such as the aforementioned game.
On AMD in particular, the cursor plane must match several aspects of any plane it overlaps with, including transform and color pipeline IIRC.
The AMD SoC in my laptop (much newer than the steam deck) only exposes two overlay planes to share among all 4 display controllers. Intel used to have a single overlay plane per display.
The Raspberry Pi 5 on the other hand intentionally limited the exposed overlay planes to "just" 48, as it can be as many as you have memory for.
You can peek at the reported capabilities of various devices here: https://drmdb.emersion.fr/
What I meant here is that I didn't see why asynchronous updates may introduce tearing; but my tired brain couldn't quite formulate it properly. And to answer that, it's clear to me know that an update of the pointer position while the pointer sprite is being drawn would introduce a shift somewhere within that sprite, which is, I suppose, the tearing discussed (and not a whole frame tearing).
> I don't know however if the mouse pointer picture is still handled the VESA way, or if GPUs video cards nowadays have a more generic API, or what.
Also, the VESA interface doesn't seem to handle mouse pointers, it's something that was available in the VGA BIOS, to provide a uniform support for this feature, as each vendor most likely did it their own way.
To the user that's an irrelevant distinction.
I also don't think this matters that much - with X11 this was optimized in one place by people that care about such details while with Wayland now every compositor developer (who in general are much more interested in window management policty) needs to become a low leve performance expert.
> Second, just to be clear, this only discusses mouse cursors on the desktop - not the content of windows, and in particular not games even if they have cursors.
Games can and sometimes do use "hardware" cursors as well - after all, they also care about latency.
When updating the cursor position, check if line being output overlaps with the cursor. If it isn't, it's safe to update the hardware cursor immediately, without tearing. Otherwise, defer updating the cursor until later (vblank would work) to avoid tearing.
Of course, this assumes it's possible to read what row of the frame buffer is being displayed. I think most hardware would support it, but I could see driver support being poorly tested, or possibly even missing entirely from Linux's video APIs.
But given that different displays work differently, I'm not sure it would worth the hassle.
> Wayland, which is a protocol
This is wayland's biggest weakness. The effect is diffusion of responsibility.
I'm not. My comment doesn't address the latency of gnome shell. I understand the boring technical distinctions between wayland and wayland client libraries and wayland display servers and gnome shell and mutter and sway, blah blah blah. Much like I understand that Linux is a kernel. That it is inspired by UNIX, but it is technically not a UNIX. I also understand that if someone describes themselves as a Linux user they probably don't just mean that they have a Android phone or that the display controller in their dishwasher or wireless access point happens to include Linux the kernel.
The "well acksually wayland is just the name of the protocol" that emerges whenever a problem is brought up is a symptom of the underlying problem with wayland the system. The confusion that gives rise to these deflections is also a symptom of that problem.
By Conway's law systems end up resembling the organisations that produce them. In this way the design of wayland the system seems to be designed by people who don't want to work together. I can see a parallel with microservice architecture.
Imagine comparing HTTP1.1 vs HTTP3. These are protocols, but in practice one compares implementations. I can pick curl for http1.1, but Python for http3, and http3 would very likely measures as slower.
Is that the protocols fault?
I very much doubt that Wayland makes a difference for this test; Wayland is for IPC between the client and server. Moving the cursor around is done by the server, without needing to talk to the client.
> well acksually
It’s not the protocol’s fault, but the system and organisation that brought it.
I know this because I worked at a place which was entirely based around microservices using Rest HTTP as the protocol/interface, some teams had to use alternate methods because HTTP protocol was the bottleneck.
I do not. Does anyone know a good but quick introduction into these concepts and the problems they cause or fix?
All I know is that Wayland was supposed to be faster by taking some slow parts out of the loop, but that doesn't seem to be working, according to these figures.
I would just rather pay someone 100 bucks and not care whose fault it is. (And yes, that's why I don't use Linux on the desktop anymore, as much as I loved i3 and tiling environments.)
It's probably in a hundred places that all add up, all of which are no particular person's responsibility. So, that probably means that 10 people from 10 different projects need to get on a call or mailing list together and find a plan of attack.
But it won't happen. And people will keep wondering why Linux can't ever get a foothold on the desktop.
Maintained: Only for bugfixes, except by metux, who is working on starting a fork.
https://wayland.app/protocols/
This isn't a big issue because Xorg splits a basic GUI into dozens of different daemons and services (compositors, window managers, input daemons) so it seems like there are different X11 implementations even when there's usually only Xorg.
Every time someone complains about wayland there's someone informing you how it achtually it isn't.
The organisation of Wayland sounds great, but it is very hard to share optimised code between compositors since key parts that affect performance (in this case latency) are largely developed outside of any shared library code.
The "organisation" of Wayland reminds me of the UNIX wars; this is going to get worse before it gets better.
SVR4 Wayland anyone?
xref the time it has taken the Rust rewrite of the GNU coreutils and arguably coreutils is a much easier problem.
The "problem" with this in Wayland is that before people 'ran Xorg with GNOME on top", now they just run GNOME the same way they run Chrome or Firefox to use HTTP - it will take time for people to get used to this.
I think the answer is also Wayland. It's very confusing because there is wayland the protocol and wayland the system, where the protocol is a part of the system - but there is no separate name for it. When people discuss wayland they are very rarely talking about wayland the protocol, and are much more likely talking about wayland the system.
In the metaphor of a web server and a web browser, Wayland would be the HTTP specification. What you're usually interested in is what server you're running, e.g. GNOME's Mutter, KDE's Kwin, sway, niri, or what client you're running, e.g. Gtk4, Qt6, etc.
Part of the problem will doubtless be the USB (and Bluetooth) stacks including the device hardware and firmware. When keyboards and mice were serial devices with their own interrupt making the code path fast was achievable. I'm not so confident that modern peripheral stacks can be made to run with the same priority. It becomes even more challenging for devices sitting on a bus with multiple device classes or multiple protocols between the device and driver (USB -> Bluetooth -> Mouse).
I hope devices can be sped up but we're a long way from a keypress triggering an interrupt and being handled in tens of milliseconds[0].
[0] https://danluu.com/input-lag/
In that compartmentalization there was nothing else competing for attention. You didn't get USB bus contention because there was just the one mouse, ever, on the line between the user and the X11 server in the X terminal.
Alternatively you move straight up to USB 4, where we get pci-e interrupts again! :)
In theory, yes. Not so, in practice.
> USB 3.1 signals at 10 GHz.
Yes but a mouse works at 12 MHz.
Tearing would affect everyone that uses a computer with X11 but your proposed example of a TV with 30i refresh rate would only affect the tiny subset of users that use a CRT television as a monitor, right?
I'd say: it really doesn't matter at all, in fact it's even more argument to make the pointer as low latency as possible. The mouse pointer is your control element as the controller of the interface. The UI element (and everything else onscreen) is the computer's element. It is an interaction between two entities and there is no need to introduce an expectation that the computer be "zero frame delay" on UI elements. It certainly sucks if it's more than a few frames for it to react, but so be it. However for the mouse cursor itself, it should absolutely be immediate because it is the user's representation on this virtual surface, and as such should follow inputs without being unnecessarily impeded. (Case in point: some UIs intentionally add delay or effects [like a fade-in] to UI element response. It's a… "choice"… but sometimes it works.)
Ultimately, the mouse pointer is an extension of hand-eye coordination; the only thing worse than making it lag (which you can learn to adapt to, albeit it will be very annoying) is to make it lag inconsistently (which you can't adapt to and will grow frustrated with extremely quickly.)
And, unfortunately, from personal experience, the mouse pointer on Wayland gets choppy if system load is high. X11 was significantly better with this; with Wayland the compositor [kwin_wayland to be specific, might be a KDE issue] doesn't even seem to be "unnice" or have some kind of scheduling priority over other tasks — I'm not sure if I'm missing something there but that seems to be a glaring oversight on its own.
IIRC it all started going downhill in Xorg when glamour appeared. After the cursor rendering path wasn't async-safe for execution from the signal handler (which something opengl-backed certainly wouldn't be), the latency was worse.
I remember when even if your Linux box was thrashing your mouse pointer would stay responsive, and that was a reliable indicator of if the kernel was hung or not. If the pointer prematurely became unresponsive, it was because you were on an IDE/PATA host and needed to enable unmask irq w/hdparm. An unresponsive pointer in XFree86 was that useful of a signal that something was wrong or misconfigured... ah, the good old days.
That's still the case. The mouse cursor is rendered with a sprite/overlay independent from the window composer swapchain, otherwise the lag would be very noticeable (at 60Hz at least). The trickery starts when dragging windows or icons around, because that makes the swapchain lag visible. Some window composers don't care any longer because a high refresh rate makes the lag not as visible (eg macOS) others (eg Windows I think) switch to a software mouse cursor while dragging.
Cool. Never really thought about why they don't exactly keep up with the cursor.
I like what's being done here but we need a more thorough experiment before we can start drawing conclusions.
It's not really hard to guess at - it's probably caused by abstraction layers resulting in multiple layers of buffering. Of course, it's all just speculation until proven by experiment, so good on TFA for doing some of that.
Retro hardware from the C64 or NES era was single-tasking with such predictable timing you could change the video resolution partway through the screen refresh and have two resolutions on screen at once. If you want, you can check the player input and update Mario's position right before he gets scanned out - the minimum possible latency by design is zero frames. (Of course, in practice the whole screen data is buffered up during vblank, which is the same latency as using a framebuffer. Also the NES didn't allow changing video registers outside of blanking periods. But the C64 did.)
X11 isn't that close to the metal, but it's designed with a single level of buffering (assuming a compositing window manager isn't used). There is a single framebuffer covering the whole screen. Framebuffer updates are completely asynchronous to scanout, so there is around 0.5 frames of latency on average between when a pixel is changed and when it's scanned out. Apps borrow pixels from this big global framebuffer to draw into. An app sends a "draw rectangle" command relative to its own borrowed space, and the server calculates where the rectangle should go on the screen, and draws it into the framebuffer, and then with an average latency of half a frame, that part of the framebuffer is scanned out to the screen.
On Wayland, there are more layers. The app has to draw into its own pixel buffer (noting that it is probably double- or triple-buffered and has to redraw the entire window rather than relying on non-changing stuff to still be there) and then once the compositor receives the buffer from the app, it has to do the same thing again, copying all the per-app buffers into a big screen-size buffer, which is probably also double- or triple-buffered, and once the vblank hits, the most recent full frame is drawn to the screen. It's just more work with more layers, and there's no surprise that latency is higher. It's hardly the first or 100th time that an elegant software architecture resulted in the computer having to do more work for the sake of keeping the architecture pure.
Something important to note is that tearing reduces latency. If new data is available while the scanout is partway through a frame, you have the choice to draw it immediately, causing tearing, or wait until the next frame, increasing latency. Always doing the high-latency thing is an explicit design goal of Wayland. (A third theoretical choice is to always time things precisely so that the new frame is calculated right before the end of vblank, but... in a modular multitasking architecture with apps written by different organizations and unpredictable CPU load, good luck with that.)
Now the really big caveat: I'd be surprised if GNOME's X11 WM wasn't compositing. If it's compositing, then X11 behaves similarly to Wayland but with even more IPC and what I just said is no reason it should have one frame less latency on average. Still something to think about though.
DLSS was designed to solve performance issues, not architecture problems and poor design.
And you absolutely wouldn't want your fonts or UI to be upscaled by it. You will hate every second of your experience.
Anyways, even if it were the case that graphic stack produces a response to the input by warping the frame during interpolation / before rendering, the input that triggers said response would (certius paribus) still go through libinput (and most likely the Wayland compositor) so timing would remain the same. For any features to improve on response time, it would likely have to circumvent one (if not both) libraries.
Wine/proton would need to support it, XWayland would need to support it (Wine/Proton are one major step away from native Wayland support: Vulkan), and finally the compositor would need to support it. Gnome is about the worst compositor that you could be testing any of this stuff on, they are comically hostile towards ideas not their own. The chances of this ever working on Gnome are near zero. KDE is pretty good for support, Hyprland seems to be trying to support every under the sun.
D3D12 dropped support for exclusive fullscreen, and I don't think headsets even go through DXGI but their own swap chain APIs. Why do VR games on Linux need the equivalent from the Wayland compositor?
Games may show it as an option if the engine itself can support other GPU APIs and the devs working on the options menu don't know or care that this particular settings combination is misleading.
[1]: https://learn.microsoft.com/en-us/windows/win32/direct3d12/s...
GNOME has supported direct scanout for fullscreen apps for a while and drm-lease was implemented not too long ago either.
Would like to see a test with gnome and the wireless mouse removed from the equation.
USB mice update at 1 kHz.
In fact, in almost 2 decades of using Linux every day, I can't remember X doing anything similar. It's always just worked.
Well, we have very different memories then. Sure, X worked reliably once configured. But configuring it was a marathon in hell, as per by my memory, and all the litany of forum posts crying out for help all across the internet. Like, I have at one point had to rescue an install by changing back the config file without a screen at all!
I also fundamentally disagree with the idea that X "does too much", which is often cited in favour of Wayland. The fact that X encompasses everything about how I interact with my computer is one of the things I love about it. I might switch WMs and DEs, but my xmodmap, xresources etc remain the same.
It hasn't needed any manual configuration for a normal setup in ages.
There’s nothing Wayland-specific that would introduce this latency, so I wonder if wlroots/sway have any additions lag.
After upgrading to plasma6 from 5, all the desktop animations have started stuttering. Probably your hardware is too new.
Figuring out why there's increased latency is a job for software tooling, but I think this guy's experiment is one of the best ways to measure what users care about: The time it takes for an input (such as mouse movement) to result in a change (such as cursor movement).
Note that this doesn't mean that the Wayland protocol itself is the reason for the higher latency. It may be Gnome's implementation (testing with a wlroots compositor might shed light on this). It may be differences in default configuration options. It may be that Wayland and X11 start up different services, and the Wayland helper processes increase load on the machine. But I seriously doubt the reason for the difference in latency was because the same hardware was used throughout the experiment.
Using a camera allows the methodology to he identical between Wayland and X, while I don't know how to listen for mouse movements from software in a way that wouldn't introduce its own problems. What if the photons are emitted from the screen after the same number of milliseconds across X and Wayland, but Mutter is a few milliseconds slower about notifying my measurement application? Conversely, what if Mutter has more real latency than X, but the latency is introduced after the stage where my measurement application sits?
The variables you mention are identical between the Wayland test and the X test. It is admittedly a challenge for reproducibility, but doesn't affect these results.