I can't pretend to know enough about the actual implications of this, but I have to wonder - if the goal is to encourage contributors, why not grease the wheels further on fedpkg/mock?
Why must we all have even slower performance? This is more of an aside - I love Fedora, but it already stands out of the pack as a bit slower than others.
It's not particularly difficult to rebuild things as-is, is my point - I suppose.
It all feels so self-fulfilling. People will want to profile (and benefit from this more)... because things that are context-switch heavy may have slowed down considerably.
I'm considering how I may batch my own rebuilds to remove this option with these utilities, mainly to see apples-to-apples what the difference is
I know they've published hand-wavy numbers/benchmarks, but I don't trust them for a multitude of reasons
I'll be happy to share! I'll probably be waiting for the next release so I can get their pointer-enabled builds.
However they do that may give me helpful hints RE: rebuilding the same sources without it. I know the tools to do it, but what I'm imagining right now is a bit convoluted.
I'm open to suggestions on things to test. I want to avoid too much reliance on synthetics -- and my 'real world' differs like all of ours
My original post has (currently) been down-struck a bit but you're the only reply. I appreciate you taking the time to engage
I understand it seems dismissive, perhaps even aggressive, but the Fedora/SuSE people tend to set trends. I just want to measure them.
The snark is me finding it ironic that I can (relatively easily) measure this with the same methods they want to avoid... rebuilds with the right options under mock / fedpkg
subtext: not that many people find profiling interesting or important... but they will if you start messing with their performance.
I find it interesting though so I'm not mad, I just fairly called it self-fulfilling -- I'm going to be profiling because the minority who regularly do, don't want to rebuild (like I will be)
https://docs.kernel.org/x86/orc-unwinder.html (which I found when looking up ORC, as mentioned in this thread) has some stats for the Linux kernel on the cost of frame pointers: "5-10%" and "3.49% more cache misses with
framepointer enabled and 6.59% more cycles".
Between this, the various sorts of C language safety checks compilers have been slowly adding, and all the spectre-related defenses, it really does feel like performance is getting slowly consumed by a series of single-digit percentage tweaks.
I agree -- it's easy to think that a few percent here/there don't hurt, but there's no global context on this. Team A takes away ~4% here, B goes on to do ~3%, and so on and so forth.
In only five of these, and consider the chains of Pythonic imports you've seen for example, you've lost roughly a quarter of your performance. (grant me an additive/compounding? nature)
This is assuming that the measurements taken in a vacuum even hold up to reality. I've seen reports for misc. other things in the 20% range, so my confidence is remarkably low.
I'm not against making it easier, but I don't think everyone always need to be ready for profiling. Making opting in easier lets us ignore the hand-wavy numbers.
edit: To add, the simplification of this performance cost down to percentages completely ignores scale. It's a deceptive statistician's trick.
Your favorite browser accidentally releasing a debug/verbose-logging version that consumed in the range of 1-10% more power globally would be significant.
I don't think it's completely untenable, though I do understand why it wouldn't be preferred.
The people served by this change to the defaults are in the minority. I don't buy that a developer that would gift the community with sufficient performance fixes derived from these... wouldn't manage with what's already available.
I think either making the processes to rebuild easier/more well known, less costly (time), or transparent (hello, package groups) is more appropriate, personally.
I'm not saying it's right, but that's how I sit with this.
I want to do better to avoid prescribing the solution. We can do better than paying the cost of profiling constantly. This information is useful to some users sometimes, but may have completely varying impact on performance forever if made the default.
I understand that angle, and I think edited this point in above before you saw it here...
> I don't buy that a developer that would gift the community with sufficient performance fixes derived from these... wouldn't manage with what's already available.
I don't think there's much productive to be had at this point, simply a difference in beliefs and theory-crafting
I'm particularly 'jaded' about this (negatively hopeful), because they can likely start with a penalty
edit: 'We' should do better to provide profiling, but I don't think it's a worthwhile default given the open-ended expense.
I predicted this would happen when ORC was introduced to the kernel. I'm tired of this eternal wheel of history in computing where everyone feels the need to rewrite stuff that works. If it wasn't for this we'd already have a DWARF unwinder in the kernel and on every other profiling-tool-of-the-week and this would be a non-issue.
But no! Let's come up with a new debuginfo format, that should simplify things right? And a couple years later the main distros have to downgrade to the least common denominator (in this case, frame pointers) just because profilers and toolchains can't implement one. single. common. debuginfo. format for once.
> And a couple years later the main distros have to downgrade to the least common denominator (in this case, frame pointers) just because profilers and toolchains can't implement one. single. common. debuginfo. format for once.
You don't even need a common debug information format. All you need is a mechanism through which the kernel can ask userspace to collect its own stack. But because the Linux world has elevated shipping the org chart to the highest level of virtue, it's hard to coordinate reasonable solutions.
Introducing a context switch, performing unwinding in process, with a relatively expensive method, would distort profiles sufficiently to make them not that useful anymore.
Sure, but that's not really counter to my point. An interrupt processed in the kernel isn't comparable to sending a signal to a userspace process, scheduling it (might not even be possible immediately!), walking the stack there, returning to the kernel and continuing. For stuff like bpf maps the kernel is even going to need the stacktrace in some form. Aside from the overhead, the delay between sample interrupt and the collected stack will be substantial.
And then there's the whole fun of doing the stack walk in userspace in an async signal safe way.
You're overly exaggerating the impact slow stack unwinding can have on the reliability of profiles. Most people are literally just fine with the poor man's profiler (aka "watch gdb"), which would be significantly slower than anything being suggested here, and doesn't even have kernel help in freezing the traced process (the kernel can freeze it the moment the profiling event happens). The proprietary OS I'm working with right now actually does this, and the only visible limitation is that there are no stack traces on kernel crash. It does CoW the stack frames while the unwinder is going on.
I still wouldn't do this, but I also see no reason why a monolithic kernel like Linux that already embeds multiple NTFS implementations can't embed a single decent DWARF unwinder.
IME profiling using the gdb style manual sampling method work kinda OK if you have a very skewed profile, but basically not at all after that. Sure, lots of people don't know how to use a profiler with call graphs and everything. But a) partially that's because it's so hard to get a useful call-graph profile b) lots of people don't work on performance sensitive code, but plenty others do.
The cost of walking stacks is pretty easy to see if you use perf report on a dwarf profile. A small part of that is just the larger data file, but if you profile perf report you can see that overwhelming the cost is unwinding. In my quick test case at least 70%. I'm sure that libunwind (as used by perf) could be improved by 2-3x - but you'd still be vastly slower.
> IME profiling using the gdb style manual sampling method work kinda OK if you have a very skewed profile, but basically not at all after that.
I really don't see what you mean here. How can you have a skewed profile with "manual sampling"? Do you really care about skew between your samples? Isn't it actually _better_ if you space your samples at random intervals, rather than strictly 0-skew regular?
Also this is completely orthogonal with whether you end up with a call graph or not. I would think most people end up with a call graph even if you did your samples with poor man's. There are multiple scripts floating around that do it even if you start from one-symbol-per-line text dumps. Even Google had one of this?
Note I can totally understand the point if you are tracing e.g. in response to a performance counter. But in these cases the kernel will help you freeze anyway, and I don't think this is what most users are doing, nor what should be doing.
"Skewed" was perhaps not a good descriptor. What I mean is workloads where most of the time is spent in a small part of the program. Many programs have more flat profiles though. Profiling those with gdb doesn't work well.
It also doesn't work very well when you have many processes/threads.
There is no "context switch". Read my post, please. The kernel already has to return from kernel mode to user mode at some point. Until it does, the user stack is stable. We collect the user stack when we return to user space. There are no extra privilege level transitions, which is what I think you meant by "context switch".
As for the speed of that collection: as I mention in my post, there's nothing the kernel can do that gives it magical performance superpowers that userspace lacks. User space can walk frame pointers too.
Based on the approach there I do think you'd end up with several times more context switches. For one, signal delivery alone causes additional syscalls. But more importantly, for things like bpf maps keyed by the stack id you need the stack at that point, not later. So you can't just defer the walk.
> As for the speed of that collection: as I mention in my post, there's nothing the kernel can do that gives it magical performance superpowers that userspace lacks. User space can walk frame pointers too.
It's not that the kernel is magically more efficient, it's that fp based stack walking is more efficient than the alternatives. And once you have fp based stack walks, there's no need for the additional dispatch for AOT languages.
I think it could be useful to have something like you describe for more dynamic languages. But even then I think you'd want to make that a profile time opt-in thing.
> For one, signal delivery alone causes additional syscalls.
No, it doesn't. The kernel is going to return to user space eventually anyway. The signal just makes the kernel "return" to a place in userspace other than the point it entered the kernel: the signal handler. When the signal handler returns (and no, you don't have to use the sigreturn system call), userspace resumes execution where it left off.
> But more importantly, for things like bpf maps keyed by the stack id you need the stack at that point, not later. So you can't just defer the walk.
Insertion into those maps can be deferred until userspace gets around to collecting its stack.
Furthermore, kernel side immediate stack collection still doesn't solve the page fault problem.
Re bpf: I don't think you can just defer the insertion into maps, that'd block execution, as code can depend an the contents of the bpf maps before continuing on with the bpf program. Iirc for bpf based tracing, the evaluation happens synchronously at the insertion points.
Not necessarily. sigreturn doesn't do anything that userspace can't do on its own. It's kind of an ugly vestige of the days when system calls were "no big deal" instead of the performance-killers they are today.
While it's true that user threads can't reset their signal masks without a system call, 1) SA_NODEFER means we don't have to, and 2) in theory, we could make an rseq field that would allow us to implement sigprocmask(2) without an entry into the kernel!
> Re bpf: I don't think you can just defer the insertion into maps, that'd block execution, as code can depend an the contents of the bpf maps before continuing on with the bpf program. Iirc for bpf based tracing, the evaluation happens synchronously at the insertion points.
I think you're right, and that's unfortunate. I'm still not sure, however, whether the need to support BPF stack-map insertion is sufficient to kill the userspace-driven unwinding scheme I've described, because (at least in my experience) this BPF-map-key scenario is a pretty fringe case.
Let's look at it this way: is it possible in principle to reconcile a solution to the pagefault problem with the needs of BPF stack-hash map keys? Even if I were to concede that userspace stack unwinding belonged in the kernel, and even if I were to additionally concede that we should do userspace stack unwinding via frame pointer walking, we'd still need to make the kernel-mode unwinder capable of taking page faults (and thus sleeping) in order to make unwinding fully robust. Let's suppose we did userspace unwinding via sleepable eBPF instead of a signal: would it be possible to accommodate, in this model, the needs of eBPF-based stack-keyed maps?
In other words: it's fundamentally impossible for an eBPF program attached to a tracepoint to query a map the key of which is a complete (user+kernel) stack if, in that context, eBPF cannot sleep. This impossibility is independent of my specific proposal and is inherent to the need for any robust stack-walker, user- or kernel-mode, to be able to take page faults and thus sleep. If I'm right about this impossibility, then we have to give up on one of 1) fully robust full (user+kernel) stack walking, and 2) stack-keyed eBPF maps. Which would you prefer?
> While it's true that user threads can't reset their signal masks without a system call, 1) SA_NODEFER means we don't have to, and 2) in theory, we could make an rseq field that would allow us to implement sigprocmask(2) without an entry into the kernel!
Writing a signal handler without SA_NODEFER is ... not fun.
> I think you're right, and that's unfortunate. I'm still not sure, however, whether the need to support BPF stack-map insertion is sufficient to kill the userspace-driven unwinding scheme I've described, because (at least in my experience) this BPF-map-key scenario is a pretty fringe case.
I don't think it's all that fringe, but yes, it's not as common as plain profiling.
The thing that makes it the most niche is the difficulty around lack of frame pointers in libc - other libraries matter far less.
> If I'm right about this impossibility, then we have to give up on one of 1) fully robust full (user+kernel) stack walking, and 2) stack-keyed eBPF maps. Which would you prefer?
I'd give up 1). The stack page fault issue isn't all that big a deal in practice, although I do agree that it's quite ugly. Whereas being able to build things like "offwake" profiles (showing the stack of the waiter and the wakee + duration of sleeps) allows to debug/profile issues that are extremely hard to debug otherwise. And rely quite fundamentally on stack-keyed BPF maps.
You don't need eBPF maps for wake-up profiles. If you just log wakeups the usual way, postprocessing tools get all the information they need to figure out what woke what. That people use eBPF maps for this purpose is a consequence of the inadequacy of Linux profiling tools.
IME that doesn't work, because the un-aggregated data volumes are far too high. The whole point of using something like eBPF is that you don't need to log all that data.
Is data volume too high? I don't see it. Windows and Android both collect useful traces and do post-hoc analysis on them instead of using things like eBPF stack maps to do online analysis.
Android doesn't run highly concurrent workloads, so it's not surprising that you'd not fall over. But I haven't used it, so I really can't say anything well qualified.
I've only used ETW a little, but I've found it to very quickly be overwhelmed with the number of events, even on quite small machines (vm with 4 cores).
I think part of our different experience comes from using profiling and tracing in different workloads. I typically profile/trace while hacking on postgres, or occasionally when debugging production issues. Commonly very busy, many core workloads.
If you're interested I can collect some numbers to show the difference between collecting all the trace events for an offwake trace, and using a map. From memory, using maps reduces throughout 20-30%, full collection way way more.
Making code less efficient, mainly for the sake of some stupid tool (and I use that term in both meanings...), and one that's supposed to help improve efficiency? That's incredibly ironic.
The thing is that DWARF unwinding (the only userspace option that doesn’t use frame pointers) is really unreliable. In fact it has such serious problems that it’s not that usable at all.
The real solution to things like this is to improve the debugging information so that return addresses can be tracked more accurately, but no one seems willing to do that work. It reminds me of encountering "value optimised out" when I can clearly see where the value is if I inspect the Asm.
Explicit constant frame pointers are almost an anachronism from an era when compilers were far worse at code generation. It seemed like a good idea at the time because it made compilers simpler, and of course on 16-bit x86 bp was the only other register that defaulted to the stack segment and [sp+nn] didn't exist. Those points are not really relevant anymore with 32-bit and 64-bit x86 using a flat memory model. Sometimes the extra byte needed for esp/rsp-relative access is worse, sometimes the extra register consumed by ebp/rbp-relative access is worse. It's really something that needs to be decided case-by-case like a intelligent Asm programmer would.
People who build distros and people who fix those low-level tools are different sets of people, I guess. So they have to build distro using tools available today. If tomorrow situation changes, they'll probably revisit this decision.
Just keeping track of the frame pointer is probably not the perfect.
In fact it might be the exact opposite. We have very complicated mechanisms that try to keep track of every possible optimization. Maybe that effort is getting in the way of a solution that slightly limits frame pointer optimization but easily figures out the value in all situations.
So because the "perfect" mechanism is having implementation problems, and nobody has tried to make the "good" mechanism, we're stuck with the very mediocre "reserve a register".
> The real solution to things like this is to improve the debugging information so that return addresses can be tracked more accurately, but no one seems willing to do that work. It reminds me of encountering "value optimised out" when I can clearly see where the value is if I inspect the Asm.
As someone who’s excited about Fedora’s change, I agree. But while waiting for that to happen, I want to have a good debug/tracing experience, which currently only frame pointers can provide.
Any that involve looking at the whole system with perf, especially where you don't want to or cannot ask the user to recompile the entire Linux distro before collecting the data. So pretty much anything that isn't a microbenchmark.
> In fact [DWARF] has such serious problems that it’s not that usable at all.
This is likely wrong. The kernel does not have a DWARF unwinder. He's probably just desymbolizing a poor stack trace taken by the kernel chasing ebps, not unwinding it with DWARF.
Part of the earlier discussions on this topic, the pro-FP side makes the argument that if FP are disabled by default, you need to recompile a ton of the distro to enable them. If they're enabled by default, and there's a particular library you want to disable them for performance reasons, that's relatively trivial.
-fomit-frame-pointer is pretty high on the list of most destructive micro-optimizations in the history of open source. When you repurpose the backtrace pointer (RBP) for other things, it totally savages our ability to have native backtraces. All for a pointless ~1% performance boost. I've always thought it was some kind of grand conspiracy to obfuscate the behaviors of closed source software but for some reason every single Linux distro that claims to love freedom has to do this too. I don't want to trust binaries built by strangers on the Internet if they hide their behaviors. Back when I worked at Google I used to read a lot of the company-wide core build configs, and the first thing they always did was make sure -fno-omit-frame-pointer is always passed, no matter what build flags a specific team uses. Don't tell me DWARF is an acceptable alternative. While it can be nice sugar on top with Linux perf report when it actually works, frame pointers let me produce a backtrace in just ten lines of C without any dependencies. If I wanted to do the same with DWARF I'd need to link some GPL library or write 10,000 lines of byzantine code to figure out which function called the one that's crashing. Stuff like this is why everyone uses scripting languages these days, since the main value prop of high-level languages is their ability to reliably print backtraces.
> I don't want to trust binaries built by strangers on the Internet if they hide their behaviors.
If a binary really wanted to hide its behavior (at least to the same extent that it could with -fomit-frame-pointer), it could just make fake stack frames beyond the current PC.
DWARF is not just the better alternative, it’s the only correct solution.
Not only do you get more complete and correct backtraces with DWARF, you also get preservation of stack-saved registers and the ability to reconstitute a subset of local state in a given frame.
libunwind is MIT licensed, not GPL.
I’ve also written a full DWARF stack unwinder myself, including a full DWARF expression interpreter — it’s nowhere near 10k lines of code.
It's not so much DWARF is the problem as it is that the information is just lost in omit-framepointer mode. You can't reconstruct it from nothing.
There's a related problem of compilers generating imperfect debuginfo that fails to reconstruct local variables from registers/memory due to an incomplete DWARF representation, but I don't think that plays a roll in unwinding. (But maybe something similar does.)
It's lost in the translation to DWARF. Of course the program knows how to unwind its own stack correctly, but explaining how to do this for arbitrary programs is untenable without DWARF to help, and it's annoying to parse and compilers don't emit it correctly.
If the information is lost by DWARF then I'd say that DWARF is clearly the problem.
Unless DWARF can handle it fine and compilers are thoroughly broken, in which case it's a tossup between "DWARF is fundamentally flawed" and "compiler designers are lazy for some unknown reason"?
The problem is that with perf the stack trace is collected by the kernel (when the interrupt from the performance counters fires), and the kernel doesn’t implement dwarf unwinding of user space code.
It _could_, but the kernel folks have made it pretty clear they don’t want to put any code related to dwarf in the kernel.
They give an example of where perf was unable to decipher the stack using DWARF; it only begs the question of whether the issue is perf or DWARF. It does show that frame pointers make things easier if all you care about are named call traces, but nobody disputes that. (It could also be the case that enabling frame pointers had the sideeffect of preventing certain optimizations. I wouldn't be surprised if flame graph generators use frame pointers (e.g. libunwind) preferentially, and using DWARF on that same binary would have resulted in the same graph.)
DWARF is useful for so much more than flame graphs. Flame graphs are cool and useful, but they're over hyped. Anybody who has ever spent any significant portion of their career debugging compiled binaries knows that proper debug information is infinitely more useful than flame graphs could ever be. Flame graphs are also useful... very useful. But that's my point: if they advocated even half as much for ensuring everybody made debug info more readily accessible bydefault as they did crying for frame pointers, the world would be a much better place.
Is DWARF too complex? There are many layers to DWARF, and there are compromises everybody could make so we can get at an optimal balance for which DWARF info we should ensure is always there. And we could also spend more time fixing tooling so it works properly with that minimal debug data.
> it only begs the question of whether the issue is perf or DWARF
AFAIK there are two issues at play here.
1. There's a bug in the Linux kernel where it sometimes return an invalid RBP register through `perf_event_open`.
2. The compiler doesn't generate the necessary DWARF info to support unwinding asynchronously, so depending on where exactly you land in a function you might not be able to unwind the stack. (This is why unwinding from *within* the program always works correctly but unwinding from *outside* the program with perf doesn't.)
I would be interested in taking / motivating a deeper dive into the failure modes of -fasychronous-unwind-tables, would you happen to recall a scenario where the undwinding problems you discuss reproduce ~consistentlyish?
It was quite a long time ago, so unfortunately at this point I don't remember what exact problems I encountered with `-fasychronous-unwind-tables`. (I think it might have been completely broken and did nothing on 32-bit ARM? But I might be misremembering.)
And there's of course the issue that you'd need everything to be compiled with it (because the program you're profiling is going to call into external dynamically linked libraries), which on an average system you most likely won't have.
I suspect the dwarf issue from the post is just missing debug info for some library, or too deep stacks.
But leaving that aside, for profiling, dwarf is just very expensive (perf, where the data volume explodes, due to copying stacks) or not available (bpf stacks), because the unwinding has to happen in the kernel. Even if the kernel had a dwarf unwinder, it's vastly more expensive to do that, compared to unwinding via frame pointers.
Try running a system wide profile on larger and busy box. Dwarf based call graphs are basically not usable. The profile quickly is ginormous, and viewing profiles is extremely slow.
> DWARF is useful for so much more than flame graphs. Flame graphs are cool and useful, but they're over hyped. Anybody who has ever spent any significant portion of their career debugging compiled binaries knows that proper debug information is infinitely more useful than flame graphs could ever be. Flame graphs are also useful... very useful. But that's my point: if they advocated even half as much for ensuring everybody made debug info more readily accessible by default as they did crying for frame pointers, the world would be a much better place.
I want both really badly but the fact is that flamegraphs drive performance optimizations which save companies millions of dollars, while symbols sometimes fix bugs and sometimes get blamed for making their software easy to reverse engineer. You can understand why one gets more focus than the other.
Seems like there's a strong difference of opinion on this, the article's author also says
> The thing is that DWARF unwinding (the only userspace option that doesn’t use frame pointers) is really unreliable. In fact it has such serious problems that it’s not that usable at all.
I'm not that familiar with the problem space - can you elaborate on why DWARF is better? or speculate on why the author is experiencing (apparent) reliability issues with it?
It's "correct" in the sense that if it works perfectly it gives you all the information and is strictly better than frame pointers in the amount of information it gives. In practice, the thing a profiler really wants is basically just a backtrace, and fast. DWARF isn't particularly great at producing these reliably or quickly, because compilers will mess up and the format is complicated.
Isn't correct unwind information a precondition for handling C++ exceptions? It seems like a compiler that produces incorrect unwind information would display incorrect behavior when a C++ exception is being caught.
My assumption is, having not looked very closely at it, is that in cases where correct unwinding matters for program correctness compilers will make sure things work, and everything outside of that rots into not being reliable because it’s not tested.
I used gimli-rs to do backtraces and it did a fantastic job at great speed (faster even than LLVM’s symbolizer). It’s MIT-licensed and definitely not 10k lines. It might be a bit Byzantine in the implementation, but it’s all hidden behind a mostly easy-to-use API.
For a minor performance decrease you in return get profiling data which allows you to identify areas where you can improve performance. For the most expensive lines of code it's important that you can optimize on production traffic.
If 1% saves you millions you want to recompile the code anyway to get use of optimizing for your current hardware, not generic distro target. Then you can turn on all flags you want.
> since the main value prop of high-level languages is their ability to reliably print backtraces.
Don't tell the golang folks!
Besides the inbuilt panic behavior which prints a very helpful backtrace, non-fatal error handling is spinkled all over with fmt.Sprintf "I am here at 134738djfjrsj so you find me in the logs"
Reclaiming %ebp for application use was worth a lot more than 1% back in the day. Now that pretty much everything is on architectures with much less register pressure than 32 bit x86 the tradeoff is more questionable.
From memory, gcc's switch to omitting frame pointers by default on x86 was in the late 2000s - when amd64 was at least clearly on the winning path, if it hadn't already. I understand changing it on 32bit x86, but...
I've always thought it was some kind of grand conspiracy to obfuscate the behaviors of closed source software but for some reason every single Linux distro that claims to love freedom has to do this too.
That's quite a stretch, and I'm saying this as someone who probably believes in things others would consider conspiracy theories. Freedom is irrelevant when you already have the source and the binaries and can do whatever you want with them... and the latter applies even if you're analysing closed-source software.
All for a pointless ~1% performance boost.
I can rebut with "All for a pointless waste of a register for something that will only be useful the <1% of the time the code is actually run."
If I wanted to do the same with DWARF I'd need to link some GPL library or write 10,000 lines of byzantine code to figure out which function called the one that's crashing.
Or just learn how to read a memory dump. If your code is somehow crashing so often enough that it becomes such a burden, then I'd say the solution is elsewhere. It's not hard to walk the stack manually.
> I can rebut with "All for a pointless waste of a register for something that will only be useful the <1% of the time the code is actually run."
When I worked at Google, profiling was done so often that it certainly exceeded 1%. There's something called Google-Wide Profiling[1] but teams often opt into much more frequent profiling of important micro services they deploy. It's not just for identifying crashes, but more for performance optimization. It's liberating to debug performance bottlenecks when you can just pull up a flame graph from half an hour ago and compare that against a flame graph from one week ago without worrying about re-deploying the old version to get profiling data.
Btw I think this is somewhat of a cultural bias depending on your previous experience. Some people simply value performance at all costs. Some people value more observability and debugability.
> When I worked at Google, profiling was done so often that it certainly exceeded 1%.
So a 10% use case for 0.001% of engineers out there. And I'm pretty sure you didn't run stock Fedora distro on servers either ?
If you're that high end that profiling code for few % gains is worth millions you can just compile the stuff with whatever flags and optimization you need instead of relying on distro to reduce performance for everyone to appease to few engineers at big corp
I have a similar, but different story. When I worked at Google, one rule for C++ code was to always write a namespace prefix (so std::vector<T> instead of vector<T>, to avoid dark corners of the language). One exception was `string` - for some reason it was not `std::string` but just string.
I've read more about this inconsistency, and it turned out that at some point libc++ changed something in the string class and it caused a 1% performance regression in "an important google-wide benchmark". That 1% of performance was so important, that since that up to at least few years ago every C++ programmer had to put up with that. I guess it makes some sense - 1% of CPU cost across the whole Google fleet is a lot of saved CPU hours (and that equals money).
I'm not sure I understand how "string" would improve runtime performance by 1%? Did Google replace std::string across their codebase with their own custom version?
It's a question of software consumers versus software developers. Disabling frame pointers basically shivs developers to hand out a 1% gain to the consumer class. Making that tradeoff might make sense for something like Windows, but I would have thought open source would be different. But nope. Tablet first GUI design, opaque binaries, and binary protocols with some misguided dream that normies will convert to the year of the Linux desktop and there shall be one laptop per child.
This is a skill that is not only hard to learn, but also takes time to apply properly. Yes, I can read a memory dump. But I'd sacrifice more than 1% of performance to not do it manually. And the infra cost to account for that is also lower than the value of my time.
Telling basically all devs without such experience to "just" learn an advanced skill is not a good solution.
All for a pointless waste of a register for something that will only be useful the <1% of the time the code is actually run.
That may be true, but having proper stack traces will probably cumulate to optimizations that gain much more than the 1% loss of enabling frame pointers [1].
Ideally, it would be much easier to rebuild whole Linux systems. Then people who need it could enable frame pointers. Sadly, rebuilding a mainstream Linux distribution is much harder than rebuilding e.g. Gentoo or NixOS. Though, in practice even rebuilding a NixOS system is fraught with issues. I have done large rebuilds, but it would often fail because e.g. unit tests have issues that only manifest themselves with enough build parallelism.
[1] Note that on x86_32, the performance improvement was larger due to register pressure.
With 32 bit code this was absolutely not the case. I have not looked so closely at 64 bit code, but in 32 bit land -fomit-frame-pointer was often closer to a 20-30% performance boost. Of all of gcc's optimization options it was the only one that I ever found to make a significant difference in real code.
I love these "rules of thumb", "black magic sorcery", and "best practices" that grey-haired old wizards pick up.
More often than not, they're invalidated by some advancement in either computer hardware or software. Not just "no longer relevant", but often counterproductive.
Worse still, these internalised tricks and rules are often related to Moore's law scaling, making them exponentially wrong over time. Not in the figurative sense, but a literal one.
You see this with people claiming that I'm exaggerating when I say all code should be parallel. Meanwhile AMD is about to sell a 128 core / 256 thread processor. You can have two per host for a staggering 512 hardware threads. Single threaded code is capable of utilising just 0.2% of that machine!
A similar "tuning best practice" that I still see applied in the field is database servers with dedicated data, log, temp, and templog drives... on cloud VMs where this is guaranteed to have worse performance than simply pooling the same amount of disk into a single logical drive.
People pick these things up and hold on to them when they're wrong by orders of magnitude.
I can't think of many (any?) other industries where more experienced senior staff can be this wrong about as many things...
I dont know man, 10 years ago I saw an alarmist talk from Martin Odersky about how 128 core computers were around the corner and we should all switch to Scala for safe&easy parallellism and now when I go to the laptop shop it’s all 8, maybe 10 cores.
We’ve been promised these magical ultra parallel machines for a decade now and either they’re much harder to build than the alarmists say, or there's simply no market.
One reason is that they are already here. It's just called the GPU which happens to be way more parallel than puny 128 cores. That's a major reason why just bunch of low power CPUs are not really a thing, as they get trashed by GPUs.
What we are seeing is a hierarchy of processing cores. It's already visible even in the desktop market. Intel has their efficiency cores. Apple has the power and efficiency cores. And the GPU on top. So we already have 3 different stages here, at the top the least amount of cores but the best single threaded performance going all the way down to, let me check how many threads, almost 10000 threads on a flagship consumer GPU running simultaneously right now.
Except GPU cores are nothing like CPU cores, GPGPU was around 10 years ago, and whatever scala language/framework features he was shilling are probably useless for that kind of programming - so that's a failed prediction.
On that I fully agree. And the reason why scala approach doesn't work is because it's horrendously inefficient. What actually does work at massive scale is the GPU style workload.
Intel tried with their Larrabee on what happens if they just toss in ton of traditional low performance CPU cores. It failed to perform. It's really hard to beat the modern SIMT style GPUs when it comes to massively parallel computation.
You can go to Micro Center and get a 128 thread processor. Everything you can rent on AWS has a similar number of threads ("vCPU" in their lingo).
Now whether or not all 128 threads need to operate on the same chunk of memory or not is somewhat up for debate. It is very good for some applications (Let's Encrypt issues all of the Internet's TLS certificates with one Postgres instance), and irrelevant for others (just run 128 copies of your node app and send requests to them at random).
You do know that developers deploy things to the cloud.
And on AWS there are instance types with 448 VCPUs [1] and in fact 128 VCPUs is a very common instance type. In fact with the growth of Kubernetes these larger configurations are increasingly popular because it's more cost efficient to pack the containers into fewer/larger nodes.
Also would add that Scala is in my opinion by far the best language for writing safe, highly concurrent code when you pair it with frameworks like ZIO. [2]
> In fact with the growth of Kubernetes these larger configurations are increasingly popular because it's more cost efficient to pack the containers into fewer/larger nodes.
That kind of is defeating your argument though: If people pack many containers on large boxes, the individual container probably is not making use of the possible parallelization across all cores.
I don't know, the only thing I ever noticed whne using swap was :
> Disabling swap doesn't prevent pathological behaviour at near-OOM, although it's true that having swap may prolong it.
Basically, the memory manager will just bulldoze through moving all the other applications to the swap to let the leaking program pursue its leak.
> you are left with a system in an unpredictable state. Having no swap doesn't avoid this.
Yes, it does: the program that leaked my 16GB is killed at the very beginning of the situation, and I don't have to either wait dozens of minutes for it to fill the swap on top of that to finally be killed, or just hard reboot the machine and lose the whole state.
> Yes, it does: the program that leaked my 16GB is killed at the very beginning of the situation
How, and by whom? In a situation where you are running out of memory, it is programs that allocate the most often that have the highest chance of seeing an OOM, not those that allocate the most total amount.
Also, on Linux, just because you don't have a swap partition doesn't mean things won't get swapped out to disk. Any memory mapped files will be swapped back to disk before issuing OOM problems - and the most common case of memory mapped files is executables and libraries. Which is of course the very thing that leads to a kind of livelock of swapless Linux systems that live close to 100% memory utilization: every time a thread is woken up, the instruction pipeline stalls until the next CPU instruction can be read from disk, slowing everything to a crawl.
By the kernel OOM killer? Is that supposed to be a trick question?
> In a situation where you are running out of memory, it is programs that allocate the most often that have the highest chance of seeing an OOM, not those that allocate the most total amount.
I don't really care; it still beats the machine being unresponsive for hours, especially if it's a remote one I dont have access to.
> Also, on Linux, just because you don't have a swap partition doesn't mean things won't get swapped out to disk. Any memory mapped files will be swapped back to disk before issuing OOM problems
That's not what is commonly understood by swapping though; that's a cache purge.
> every time a thread is woken up, the instruction pipeline stalls until the next CPU instruction can be read from disk, slowing everything to a crawl.
When you have swap enabled, yes; that the whole problem. However, IME, OoM just leads to the problematic program being quickly killed, then the user being able to go back to work.
> However, IME, OoM just leads to the problematic program being quickly killed,
I think this is the divide. In my experience, OOM killer rarely kills the culprit. It doesn't actually know which programs are behaving irregularly (that's the 'trick' in his 'trick question') and the OOM score heuristic isn't reliable in desktop usecases. If you know ahead of time which process will likely cause you trouble, you can adjust oom_score_adj for that process to make it the first target, but this is not great for common desktop use.
The OOMKiller is a pretty blunt and simple tool, that starts killing things until memory pressure is gone. Also, depending on how large the memory mapped areas are and specific OOM settings, your executables might start getting swapped out before the OOMKiller is actually called.
> That's not what is commonly understood by swapping though; that's a cache purge.
No, it's exactly a form of swapping. I'm not talking about purging Linux's IO caches.
I am talking about taking a memory resident page of your process and unmapping it, such that the next it is accessed it causes a page fault and the VMM loads it back from disk. When you have swap space, this can happen to any page. When you don't, this can still happen to read-only pages that already have a corresponding file on disk - which is usually your executable and any shared libraries you loaded.
Now, whether you'll really hit this situation or not really depends on how much of your memory is being taken up by the code of running processes. In many common server workloads, this should be very rare. However, in desktop workloads, where you might have lots of programs running at the same time, the total amount of memory being taken up by all their code may end up high enough to enter this state before calling the OOMKiller.
The trick is not disabling swap, but having a reasonable size swap. The old rule of thumb of 2x RAM was reasonable in the 90s, but after that, it should be 2x RAM or 512MB, whichever is smaller. Nobody needs a 2TB swap volume on their 1TB ram server.
If you've got something leaking a lot and fast, 512MB swap is going to fill up in seconds, not minutes. If you have a slower leak, swap usage provides a clear signal something is wrong and may give you a chance to intervene before the OOM killer. Even in fast death, swap usage may be useful forensics, if you record it at a high enough time resolution.
> The old rule of thumb of 2x RAM was reasonable in the 90s, but after that, it should be 2x RAM or 512MB, whichever is smaller.
This is exactly why I don't have swap for over a decade on my machines (since my machines have 16+ GB RAM). The mere 1/32 improvement is not worth the headaches of swap.
Maybe now but back when we used HDDs having no swap was much better than having swap on desktop Linux. No swap, and the OOM kicks in and (sometimes) kills the offending program. Enable swap and when you run out of memory your computer just grinds to a complete halt requiring a hard reset.
I actually have been forced to go back to desktop Linux again recently after almost 20 years of avoiding it. It's pretty disappointing to see that Linux's OOM handling behaviour is still completely broken (at least on desktop). Windows and Mac behave much more sanely when you run out of memory, gradually slowing down rather than suddenly freezing for like a minute while it figures out what to kill.
I did eventually find a Reddit thread that suggested enabling zram (compressed ram swap disk) which does appear to have helped but why isn't that the default? Also I now get a kernel panic about once a week but that could be unrelated.
> Single threaded code is capable of utilising just 0.2% of that machine!
You can still server 512 different clients at once and still utilize 100% of the CPU.
The parallelism is only a requirement if you want to compute one thing faster, not a bunch of things at once. Very much the desirable property for databases (although even that can be worked around with sharding if you're stubborn enough) but many developers code could be painfully serial yet still be perfectly fine to scale.
> I can't think of many (any?) other industries where more experienced senior staff can be this wrong about as many things...
Lawyer enters the thread.
It's more common than you think. When complex work is done within power structures, it's just not enough to be "right" about something. You also have to be able to sell your solution to people who are wrong and powerful, which is sometimes not possible.
The knowledge a lawyer acquires about the law doesn't go stale very fast. Many laws have been on the books for a century or more, and precedent is also established slowly and incrementally.
A lawyer from 10 years ago that memorised some good "rules of thumb" about generic matters such as liability or court procedure won't have to throw out everything they know to be relevant today or another decade into the future.
Meanwhile in IT, I regularly see network engineers upgrading links from 500 megabits to 1 gibabit when they should be looking at upgrading to 100 or even 200 gigabits.
Their mental model of what's fast, relevant, or desirable is "off" by two orders of magnitude. Their knowledge was accurate just a decade ago, but is hilariously wrong now.
I can't think of any other mainstream profession where institutional knowledge is deprecated as quickly.
> A similar "tuning best practice" that I still see applied in the field is database servers with dedicated data, log, temp, and templog drives... on cloud VMs where this is guaranteed to have worse performance than simply pooling the same amount of disk into a single logical drive.
It's not guaranteed to be worse. For latency sensitive OLTP workloads dedicated, smaller, disks for the journal can result in substantially lower & predictable latency. Random reads for the data exhausting iops, causing stalls of journal writes isn't fun. There's also per-disk queuing on a bunch of levels, and avoiding intermingling data and journal writes can be quite beneficial.
That's not to say it's always better, at all. But making it out to be a stupid thing one should never do isn't wise either.
Some of the things you claim are very context-specific and not true in general, which I find jars a bit with the (perceived by me to be) authoritative tone of your post.
> I love these "rules of thumb", "black magic sorcery", and "best practices" that grey-haired old wizards pick up.
Just quoting this to point out what I mean by addressing your tone. It's fine to disagree, but this feels like you're setting up that everyone who disagrees with you must be condescending and stuck in the past. Yes, performance advice is often tied to the specific situation, but that's also true for the things you state.
> Worse still, these internalised tricks and rules are often related to Moore's law scaling, making them exponentially wrong over time. Not in the figurative sense, but a literal one.
Implying that Moore's Law makes anything exponentially faster is an anachronism (talk about old knowledge!), it only makes things exponentially more parallel nowadays. See next point why that's relevant.
As others have already clarified, -fomit-frame-pointer was sometimes 20% on x86. For some applications, people would kill for a 20% boost in single-threaded performance, even today!
And importantly, that this specific optimization has become useless on x64 says nothing about the value of optimizations in general. Different ones are still useful, to give an example let's say SIMD (where it applies).
> You see this with people claiming that I'm exaggerating when I say all code should be parallel. Meanwhile AMD is about to sell a 128 core / 256 thread processor. You can have two per host for a staggering 512 hardware threads. Single threaded code is capable of utilising just 0.2% of that machine!
I'm not sure which strawman "grey-haired" developer you have in mind that buys a 512-thread server to run single-threaded code on it and then needs you to explain to them that they should parallelize it. Maybe you've met that person. But in general: Doesn't most code live in web servers nowadays anyway? If so, it's either already running in parallel, or can be made to do so with very little effort.
BTW, parallelizing code in a web server can be harmful (!) to total throughput due to communication overheads. This is something that I find people don't always realize when they naively parallelize something, and is also sometimes missed by benchmarks that don't put the server under heavy load.
> A similar "tuning best practice" that I still see applied in the field is database servers with dedicated data, log, temp, and templog drives... on cloud VMs where this is guaranteed to have worse performance than simply pooling the same amount of disk into a single logical drive.
A statement like this depends on many concrete details about the specific cloud infrastructure (you didn't say AWS, you said "cloud"), and cloud VMs are usually not the best performing way to run a database anyway. So again, this is very context-specific.
Also, if database performance is critical enough that you consider tuning these things, then renting bare metal should at least be a consideration as well. People who have never measured it are sometimes surprised by how large the virtualization overhead is (and how cheap it is to rent physical servers). On actual hardware, this advice might still be relevant.
> I can't think of many (any?) other industries where more experienced senior staff can be this wrong about as many things...
The actual point is that you should measure things in concrete cases, not make general claims. In that sense, I unfortunately don't find your "updated" way to state what supposedly is and isn't relevant not much of an improvement to the "outdated" claims you argue against so vehemently.
> Some of the things you claim are very context-specific and not true in general, which I find jars a bit with the (perceived by me to be) authoritative tone of your post.
So what? Who cares? I find the whiny tone of your post to be offensive.
> You see this with people claiming that I'm exaggerating when I say all code should be parallel. Meanwhile AMD is about to sell a 128 core / 256 thread processor. You can have two per host for a staggering 512 hardware threads. Single threaded code is capable of utilising just 0.2% of that machine!
Whether or not this argument holds water depends entirely on the context of the situation; what is the relationship between the program being considered and the computer? If the program is the reason for that hardware to exist, then obviously you want to use that hardware to it's fullest potential. If you've bought a 256 core machine specifically to run this program, then obviously you want to use all those cores. Some sort of threaded or multiprocess architecture should be an easy sell in this sort of circumstance. It shouldn't even be a discussion.
But on the other hand, what if the program under consideration is ancillary? In that case, the program should use the bare minimum resources necessary, so as to not get in the way of whatever the primary program may be. If you have say.. a music playing daemon, then odds are your program should not be trying to use the host machine to it's fullest because that hardware actually exists to get some other job done. Ancillary programs should be designed to survive on leftover scraps, not with the assumption that they'll have first dibs on the use of the hardware.
These are kinds of generic recommendations you listed are precisely the type that go out-of-date.
Even on a laptop, I now have 16 threads and almost never see more than 1 utilised even when I'm waiting.
Any time a human waits for a computer that is just 6% utilised, a developer screwed up.
Just this minute, I downloaded a program installer using a 10 Gbps Internet link.
Yes. Ten gigabits, dedicated to me, personally! Your idea of "what is a fast Internet link" is wrong.
I downloaded a 600 MB software package in less than a second, and then had to wait for 30 seconds while the anti-malware tools scanned it... with a single thread.
Extracting the file was 10 seconds because 'zip' is by default single threaded.
Installing it took a solid minute during which my CPU usage never peaked above 2 cores because the installer is also single threaded. The only reason it uses more than 1 core is because the anti-malware scans it... again... on a second thread.
Similarly, any time I see anyone manipulating ~1 GB of data, invariably they do it with GUI or CLI tools that are single threaded. This was desirable in an era when everyone had a single mechanical HHD. Multiple read or write threads would have caused disk head seeks, slowing down the process. These days when everyone has NVMe SSDs, the only way to fully utilise the 500K IOPS available from a $150 device is by throwing dozens of parallel threads at it!
There's no practical way to avail yourself to all the latest advancements that find their way into your work in computers.
Even as someone who tries to actively question assumptions that may no longer hold I still find myself constantly flat-footed.
I can't pretend like I know all the various advancements in processor architecture of the past 5 years (since 2018) or how modern compilers exploit them and also the latest techniques in GPU computing.
Heck even old rules of thumb like the speed of platter drives are being challenged - mechanical drives with 500M/s throughput are hitting the shelves this year. I just upgraded a platter RAID where I'm getting ~1000M/s. The SSD delta has changed (that's of course assuming SSD hasn't moved which I'm sure is also false). I'm sure there's new advancement in material science and manufacturing that I'm also not privy too which has made this possible.
Storage alone is another vast world. I don't know all the features of ext4, xfs, zfs, hammer, and btrfs or how to competently compare and tune them for various workloads or how that analysis has changed in say the past 24 months. This is something I became intimately aware of when upgrading the array. I didn't know any of the modern tools or approaches.
All knowledge in this space eventually becomes vestigial and outdated and it's physically impossible to keep up. I haven't even kept up with the latest C features. Then there's Go, Rust, Typescript and countless other languages that each require at minimum, a hundred hours a year each to competently follow.
The best we can do is have humility and continually approach things as a student because we never know what table leg has been knocked out underneath us without our noticing.
Not just X86, the older X86. Before Pentium Pro there was not even register renaming. So one was both register starved and the registers you saw where actually the registers you got. Even after that it was still somewhat starved but it alleviated the problem greatly.
Nowadays X64 has a decent amounts of registers + the HW internally has several times more the registers, so saving one is almost meaningless.
It was. The amount of improvement varied by application, but this was testing with real world applications like JPEG libraries, MP3 libraries, compression, etc... Of course this was back in the 90s and early 2000s so back in the dinosaur ages by computer standards. Almost certainly using gcc 2.95 on a Pentium 75. It was the only optimization flag that made more than a couple percent difference.
> Stuff like this is why everyone uses scripting languages these days
Do they? It seems like everyone is moving away from scripting languages for new projects these days, at least where web browser-related requirements doesn't push them into using Javascript.
Indeed, before this current pendulum swing, scripting languages had a good run. I'm not convinced it was because of stack traces, though. They won hearts and minds on promises of no longer having to do "XML sit-ups" and whatnot.
Don't look at hackernews or reddit for what most people actually use. "Scripting languges" are doing just fine for majority of CRUD and CRM stuff most software houses produce.
The article mentioned shows the worst case being closer yo 10%
> Compiling the kernel turned out to be 2.4% slower, and a Blender test case regressed by 2%. The worst case appears to be Python programs, which can see as much as a 10% performance hit. To many, these costs were seen as unacceptable.
Sacrificing that much for the absolute minority of people that need to dig that much into performance doesn't seem to be worth it for distro. Would be interesting to see why some cases still get that much regression tho.
That's a very specific case (the Python interpreter main loop), and for that reason the Python interpreter is still using -fomit-frame-pointer. It is specifically called out in the Fedora Change: https://fedoraproject.org/wiki/Changes/fno-omit-frame-pointe...
While it may be true that only the absolute minority actually push computing forward and optimize things, sidelining that minority will have disastrous consequences for everyone.
-fomit-frame-pointer is pretty high on the list of most destructive micro-optimizations in the history of open source. When you repurpose the backtrace pointer (RBP) for other things, it totally savages our ability to have native backtraces
Funny. I always learned that the *BP registers stood for Base Pointer and not Backtrace. And then I would use BP to reference function temp variables from that part of the call frame.
The Intel “ASM86 Language Reference Manual” calls it that as well.
Yes, and that base pointer gets pushed on the stack when a function call is made. And that allows to accurately restore stack frames to get the backtrace.
So, not sure what's funny.
> but for some reason every single Linux distro that claims to love freedom has to do this too ... If I wanted to do the same with DWARF I'd need to link some GPL library or write 10,000 lines of byzantine code to figure out which function called the one that's crashing.
I mean, presumably you just answered your own question: while corporate software might not be able to link GPL code, Linux software — that mostly already is GPLed — certainly can. In fact, that's presumably why the library you're referencing (libdwarf?) is under the GPL in the first place — it was created by a Linux developer, for use in their own and others' FOSS Linux software, where it easily solves this problem for them. And because they have this easy solution to this problem on hand, they don't really see the need to turn on frame pointers in their projects' Makefiles.
Seems like a reasonable solution to a fraught problem.
Man hours (spent recompiling the kernel with the right flags in this case) are often undervalued when comparing different solutions.
I have found this is especially true when doing cost analysis. It is so easy to look at something that costs $10,000 a month and think it's expensive, despite the fact that it saves twice that much in developer time (salary).
How many people are using Fedora? Why must all of them suffer because you want to save a very small amount of your time? Why can't the developers--which include me!!--who want to profile things take the time to custom compile their boxes instead of demanding that all of the poor end users who don't even know what "profile" means have to take a reduction in performance? How many end users would have to exist to make you decide it isn't worth their pain and extra cost for you to have to spend a minutes of your time to help them? (Maybe then further answer that question under the scenario where the users in question by and large never actually install your software.)
What prevents anyone of us to maintain a fork with same SRPMS as fedora but keeping the omition of frame pointers? I guess someone will do that if there really is an observable performance advantage. There are already fedora forks such as Nobara providing some kernel patches and some specific compilation options or patches for some apps.
While I'm sure many users would, I'd be shocked if, after being asked the honest scenario of "hello: we'd like to make your device permanently 1-2% less efficient--which means everything you do on it will take 1-2% longer and your battery will die 1-2% sooner--so that, just in case it ever comes up, it will be easier for us (as it isn't like we need this, but it will save us some time and effort) we can use either remote telemetry from your device or work with you directly to make our product a bit faster" anywhere near most people agreed.
Because that's not the hand you're dealt with. Performance problems don't just crop up on my machine with custom build flags, but on everyone's machines (yes, including end user machines!) If I get a bug report for something hanging or being slow, I can guide people through collecting a profile and sharing it with me. I can't get them to recompile their kernel and all of userspace.
I recently built SPECint2k6 for RISC-V (on JH7110) with and without -fomit-frame-pointer and to my surprise the default (with frame pointers) was a tiny bit faster (though so close to the noise floor that I'll settle for "the same"). I haven't dug into why frame pointers seem to cost nothing on RISC-V but it's likely that it's just that there's plenty of registers.
Unfortunately, it increases the code size by 10%. I was looking into this just last week, and can confirm that it's still a problem on the latest version of Rust nightly: https://github.com/tock/tock/pull/1660
I wish we could have frame pointers, because they would make working in embedded land so much easier and more reliable, but a 10% increase in code size just isn't worth it.
In the embedded world, you are limited by the amount of storage space available for your code. In a lot of cases the code is in flash or one time programmable rom memory on the microcontroller. Even if you have off-chip storage, you usually have to load the code into ram in order to run, and ram size is also quite limited. For low end microcontrollers it's common for rom / ram sizes to be on the order of 32kb (although they go much lower than that). In this world, the most common optimization level is -Os (or -Oz if available) because running out of storage space means you need a more expensive microcontroller, which increases the cost of each device you are selling, and might require a redesign of the PCB.
There isn’t a simple relationship.
Ignoring the code space issue (and I$ pressure), the added instructions in this case are for function entry/exit so if that not on your hot path, it might not matter at all.
As others have noted, the issue isn't execution speed. Rather it's the fact that the memory and disk requirements increase by 10% which has a measurable effect on cost.
We currently load the entire program into RAM to make things easier, so anything to lower storage requirements is a win. Someday we'll be able to mmap the disk to do XIP, but we don't have that implemented yet.
Essentially nothing in a world where almost everyone is running 7 electron apps at once and Windows has lost the ability to suspend properly.
Especially when all those apps were built on these platforms exactly because of how annoying it is to debug C programs. The total energy usage could even be negative due to this.
It's a fair point that other things waste unimaginably more, but in terms of debugging -- omitting the frame pointer hasn't been a debugging headache for a long time and even if it were that still wouldn't itself justify running it on every host under the sun including the vast majority that will never have any debugging run on it.
I doubt C debugging has had a meaningful role in the use of electron in any case.
You have to trade that against the large optimizations that were never made because performance analysis of the whole system was essentially unavailable. This is a very new change in Fedora, but hopefully we'll now see lots of big performance improvements in common workloads.
The analysis I was doing when writing this blog post was about understanding where 50% of the time of my program was going. The fix literally doubled the speed of the program: https://bugzilla.redhat.com/show_bug.cgi?id=2166686
The biggest problem with frame pointer unwinding isn't even the performance hit, but the inability to support unwinding of anything that isn't AOT compiled native code.
If you unwind using frame pointers through a Python program, you'll get a bunch of meaningless interpreter frames instead of information about the operation of the program being interpreted. That, and since the FP unwinder can't sleep and can't take page faults, it'll fail to traverse stack frames that happen to be paged out.
The right way to solve the kernel unwind of userspace problem is to get the kernel out of the business of unwinding userspace.
Unwinding only paged-in, AOT-compiled native code is a step in the wrong direction, especially because it's precisely the people for whom unwinding kinda-sorta works now (kernel, libc, C language diehards, etc.) who would need to approve and coordinate a more robust and comprehensive solution that also, unlike frame pointers, has zero cost when not in use.
Where are the extra "context switches"? Can you count them for me? Under my proposal, we defer stack collection until we're going to return to user space anyway. The user stack of a thread cannot change while that thread is running in user mode.
So after looking at this, I do now agree [can't edit my original comment]. Are you planning to implement anything like this? I think similar to the suggestion to fix DWARF, enabling frame pointers may not be the best solution, it may be reversed in future when we get something better, but it's the one that actually works right now.
The Linux perf profiler has support for symbolization of dynamic code generated by a JIT. This is supported by at least the JVM, Node, and Python (3.12).
This is an incredibly hacky mechanism that has to be individually enabled in each context in which it's used. What if I can't write to tmp? What if PIDs wrap around?
I don't see why the "individually enabled" part is inherent to the design. The JITs could emit these files automatically, if they wanted to.
But the main point is that "inability to support unwinding of anything that isn't AOT compiled native code" is factually incorrect: as far as I know, frame pointers plus this "hack" is currently the only widely available way to do just that.
No, they can't, because doing so would be a constant drag on performance for processes not being profiled. Plus, there's no provision for VA reuse for the trampolines, so as the set of described functions grows, so does the magic text file in /tmp.
It would be nice to have something like this, but there are generally implementation challenges with getting userspace to do anything on behalf of the kernel.
I wonder how significantly frame pointers damages the stack engine optimization in x86 processors.
There's still a bunch of pushes and pops, I guess because that's shorter code than storing into an explicit offset off the stack pointer, but since you've read the stack pointer anyway and synchronized it, the uop sequence and instruction dependencies wouldn't end up being a lot different I would have thought.
It does damage it actually. GCC currently generates slightly pessimised code when frame pointers are enabled, for example always using the frame pointer to access stack variables even if it knows the offset from the stack pointer. I guess this is simply because distributions were all using -fomit-frame-pointer so the frame pointer case was neglected. Hopefully this will improve now that most distros have switched to enabling frame pointers.
I’m looking forward to this change. Ensuring fast, reliable stack traces when debugging under pressure or tracing complex systems is invaluable. Once you try it it’s hard to go back.
It’ll take some years until frame pointers become more mainstream. Still, some people might decide to never enable them in libraries they use or their own code.
That’s why I’ve been working on a BPF DWARF-based unwinder to fill the gap. It’s fully OSS (Apache 2.0 for user space and GPL for BPF) and should get out of beta in ~1month.
Previous HN discussion [0]. Happy to help anybody that wants to integrate it in their tools / systems.
I have moved my Linux Desktop to Fedora (Recommended by my 26yo son).
Been in the Linux space since the beginning so have seen distros come and go.
Who remembers Slackware?
What a journey Linux has been.
Hope the framepointers work out (I think they will squeeze out some more performance)
Thanks to all the amazing people who developed the Linux miracle.
What didn't work or wasn't well put together on slackware?
The only thing that was annoying at the time was having to manually download and compile third party software as well as some of its depencency. In reality it wasn't that much of a big deal. You just had to know how to read a readme and install text file really. But this is kind of a moot point in 2023 with:
1. slackbuilds and its clients, which is pretty much like the AUR for archlinux users or pkgsrc and makes you a lot of additionnal software available very easily.
2. All the software nowadays made installable with flatpak, appimage or via some bash script.
3. podman and docker
If anything modern software deployment methods made slackware experience closer to rpm or deb based distros than 20 years ago.
I had the exact opposite experience. Debian always had to add their own variations and special sauce to the configurations and layout. Slackware was as stock as you could get, effectively identical to what you'd get with a configure; make; make install.
I also moved from a decade of Ubuntu to Fedora over a year ago
Was a fantastic decision, I think the project is run much better and the pace of development and innovation for software devs specifically is much better.
Heh. Looking for discussions in debian around this, I found a bug requesting a glibc build with frame pointers enabled. Reported by me. 8 years ago ...
199 comments
[ 3.6 ms ] story [ 254 ms ] threadWhy must we all have even slower performance? This is more of an aside - I love Fedora, but it already stands out of the pack as a bit slower than others.
It's not particularly difficult to rebuild things as-is, is my point - I suppose.
It all feels so self-fulfilling. People will want to profile (and benefit from this more)... because things that are context-switch heavy may have slowed down considerably.
I'm considering how I may batch my own rebuilds to remove this option with these utilities, mainly to see apples-to-apples what the difference is
I know they've published hand-wavy numbers/benchmarks, but I don't trust them for a multitude of reasons
However they do that may give me helpful hints RE: rebuilding the same sources without it. I know the tools to do it, but what I'm imagining right now is a bit convoluted.
I'm open to suggestions on things to test. I want to avoid too much reliance on synthetics -- and my 'real world' differs like all of ours
My original post has (currently) been down-struck a bit but you're the only reply. I appreciate you taking the time to engage
I understand it seems dismissive, perhaps even aggressive, but the Fedora/SuSE people tend to set trends. I just want to measure them.
The snark is me finding it ironic that I can (relatively easily) measure this with the same methods they want to avoid... rebuilds with the right options under mock / fedpkg
subtext: not that many people find profiling interesting or important... but they will if you start messing with their performance.
I find it interesting though so I'm not mad, I just fairly called it self-fulfilling -- I'm going to be profiling because the minority who regularly do, don't want to rebuild (like I will be)
Between this, the various sorts of C language safety checks compilers have been slowly adding, and all the spectre-related defenses, it really does feel like performance is getting slowly consumed by a series of single-digit percentage tweaks.
Doesn't including an explicit, predictable, frame pointer make life moderately easier for shell code?
In only five of these, and consider the chains of Pythonic imports you've seen for example, you've lost roughly a quarter of your performance. (grant me an additive/compounding? nature)
This is assuming that the measurements taken in a vacuum even hold up to reality. I've seen reports for misc. other things in the 20% range, so my confidence is remarkably low.
I'm not against making it easier, but I don't think everyone always need to be ready for profiling. Making opting in easier lets us ignore the hand-wavy numbers.
edit: To add, the simplification of this performance cost down to percentages completely ignores scale. It's a deceptive statistician's trick.
Your favorite browser accidentally releasing a debug/verbose-logging version that consumed in the range of 1-10% more power globally would be significant.
The people served by this change to the defaults are in the minority. I don't buy that a developer that would gift the community with sufficient performance fixes derived from these... wouldn't manage with what's already available.
I think either making the processes to rebuild easier/more well known, less costly (time), or transparent (hello, package groups) is more appropriate, personally.
I'm not saying it's right, but that's how I sit with this.
I want to do better to avoid prescribing the solution. We can do better than paying the cost of profiling constantly. This information is useful to some users sometimes, but may have completely varying impact on performance forever if made the default.
Requiring rebuilding no matter how easy would completely remove all non-dev testers and demotivate the majority of developers.
> I don't buy that a developer that would gift the community with sufficient performance fixes derived from these... wouldn't manage with what's already available.
I don't think there's much productive to be had at this point, simply a difference in beliefs and theory-crafting
I'm particularly 'jaded' about this (negatively hopeful), because they can likely start with a penalty
edit: 'We' should do better to provide profiling, but I don't think it's a worthwhile default given the open-ended expense.
But no! Let's come up with a new debuginfo format, that should simplify things right? And a couple years later the main distros have to downgrade to the least common denominator (in this case, frame pointers) just because profilers and toolchains can't implement one. single. common. debuginfo. format for once.
You don't even need a common debug information format. All you need is a mechanism through which the kernel can ask userspace to collect its own stack. But because the Linux world has elevated shipping the org chart to the highest level of virtue, it's hard to coordinate reasonable solutions.
And then there's the whole fun of doing the stack walk in userspace in an async signal safe way.
I still wouldn't do this, but I also see no reason why a monolithic kernel like Linux that already embeds multiple NTFS implementations can't embed a single decent DWARF unwinder.
IME profiling using the gdb style manual sampling method work kinda OK if you have a very skewed profile, but basically not at all after that. Sure, lots of people don't know how to use a profiler with call graphs and everything. But a) partially that's because it's so hard to get a useful call-graph profile b) lots of people don't work on performance sensitive code, but plenty others do.
The cost of walking stacks is pretty easy to see if you use perf report on a dwarf profile. A small part of that is just the larger data file, but if you profile perf report you can see that overwhelming the cost is unwinding. In my quick test case at least 70%. I'm sure that libunwind (as used by perf) could be improved by 2-3x - but you'd still be vastly slower.
I really don't see what you mean here. How can you have a skewed profile with "manual sampling"? Do you really care about skew between your samples? Isn't it actually _better_ if you space your samples at random intervals, rather than strictly 0-skew regular?
Also this is completely orthogonal with whether you end up with a call graph or not. I would think most people end up with a call graph even if you did your samples with poor man's. There are multiple scripts floating around that do it even if you start from one-symbol-per-line text dumps. Even Google had one of this?
Note I can totally understand the point if you are tracing e.g. in response to a performance counter. But in these cases the kernel will help you freeze anyway, and I don't think this is what most users are doing, nor what should be doing.
It also doesn't work very well when you have many processes/threads.
As for the speed of that collection: as I mention in my post, there's nothing the kernel can do that gives it magical performance superpowers that userspace lacks. User space can walk frame pointers too.
Based on the approach there I do think you'd end up with several times more context switches. For one, signal delivery alone causes additional syscalls. But more importantly, for things like bpf maps keyed by the stack id you need the stack at that point, not later. So you can't just defer the walk.
> As for the speed of that collection: as I mention in my post, there's nothing the kernel can do that gives it magical performance superpowers that userspace lacks. User space can walk frame pointers too.
It's not that the kernel is magically more efficient, it's that fp based stack walking is more efficient than the alternatives. And once you have fp based stack walks, there's no need for the additional dispatch for AOT languages.
I think it could be useful to have something like you describe for more dynamic languages. But even then I think you'd want to make that a profile time opt-in thing.
No, it doesn't. The kernel is going to return to user space eventually anyway. The signal just makes the kernel "return" to a place in userspace other than the point it entered the kernel: the signal handler. When the signal handler returns (and no, you don't have to use the sigreturn system call), userspace resumes execution where it left off.
> But more importantly, for things like bpf maps keyed by the stack id you need the stack at that point, not later. So you can't just defer the walk.
Insertion into those maps can be deferred until userspace gets around to collecting its stack.
Furthermore, kernel side immediate stack collection still doesn't solve the page fault problem.
Re bpf: I don't think you can just defer the insertion into maps, that'd block execution, as code can depend an the contents of the bpf maps before continuing on with the bpf program. Iirc for bpf based tracing, the evaluation happens synchronously at the insertion points.
Not necessarily. sigreturn doesn't do anything that userspace can't do on its own. It's kind of an ugly vestige of the days when system calls were "no big deal" instead of the performance-killers they are today.
While it's true that user threads can't reset their signal masks without a system call, 1) SA_NODEFER means we don't have to, and 2) in theory, we could make an rseq field that would allow us to implement sigprocmask(2) without an entry into the kernel!
> Re bpf: I don't think you can just defer the insertion into maps, that'd block execution, as code can depend an the contents of the bpf maps before continuing on with the bpf program. Iirc for bpf based tracing, the evaluation happens synchronously at the insertion points.
I think you're right, and that's unfortunate. I'm still not sure, however, whether the need to support BPF stack-map insertion is sufficient to kill the userspace-driven unwinding scheme I've described, because (at least in my experience) this BPF-map-key scenario is a pretty fringe case.
Let's look at it this way: is it possible in principle to reconcile a solution to the pagefault problem with the needs of BPF stack-hash map keys? Even if I were to concede that userspace stack unwinding belonged in the kernel, and even if I were to additionally concede that we should do userspace stack unwinding via frame pointer walking, we'd still need to make the kernel-mode unwinder capable of taking page faults (and thus sleeping) in order to make unwinding fully robust. Let's suppose we did userspace unwinding via sleepable eBPF instead of a signal: would it be possible to accommodate, in this model, the needs of eBPF-based stack-keyed maps?
In other words: it's fundamentally impossible for an eBPF program attached to a tracepoint to query a map the key of which is a complete (user+kernel) stack if, in that context, eBPF cannot sleep. This impossibility is independent of my specific proposal and is inherent to the need for any robust stack-walker, user- or kernel-mode, to be able to take page faults and thus sleep. If I'm right about this impossibility, then we have to give up on one of 1) fully robust full (user+kernel) stack walking, and 2) stack-keyed eBPF maps. Which would you prefer?
Writing a signal handler without SA_NODEFER is ... not fun.
> I think you're right, and that's unfortunate. I'm still not sure, however, whether the need to support BPF stack-map insertion is sufficient to kill the userspace-driven unwinding scheme I've described, because (at least in my experience) this BPF-map-key scenario is a pretty fringe case.
I don't think it's all that fringe, but yes, it's not as common as plain profiling.
The thing that makes it the most niche is the difficulty around lack of frame pointers in libc - other libraries matter far less.
> If I'm right about this impossibility, then we have to give up on one of 1) fully robust full (user+kernel) stack walking, and 2) stack-keyed eBPF maps. Which would you prefer?
I'd give up 1). The stack page fault issue isn't all that big a deal in practice, although I do agree that it's quite ugly. Whereas being able to build things like "offwake" profiles (showing the stack of the waiter and the wakee + duration of sleeps) allows to debug/profile issues that are extremely hard to debug otherwise. And rely quite fundamentally on stack-keyed BPF maps.
I've only used ETW a little, but I've found it to very quickly be overwhelmed with the number of events, even on quite small machines (vm with 4 cores).
I think part of our different experience comes from using profiling and tracing in different workloads. I typically profile/trace while hacking on postgres, or occasionally when debugging production issues. Commonly very busy, many core workloads.
If you're interested I can collect some numbers to show the difference between collecting all the trace events for an offwake trace, and using a map. From memory, using maps reduces throughout 20-30%, full collection way way more.
The thing is that DWARF unwinding (the only userspace option that doesn’t use frame pointers) is really unreliable. In fact it has such serious problems that it’s not that usable at all.
The real solution to things like this is to improve the debugging information so that return addresses can be tracked more accurately, but no one seems willing to do that work. It reminds me of encountering "value optimised out" when I can clearly see where the value is if I inspect the Asm.
Explicit constant frame pointers are almost an anachronism from an era when compilers were far worse at code generation. It seemed like a good idea at the time because it made compilers simpler, and of course on 16-bit x86 bp was the only other register that defaulted to the stack segment and [sp+nn] didn't exist. Those points are not really relevant anymore with 32-bit and 64-bit x86 using a flat memory model. Sometimes the extra byte needed for esp/rsp-relative access is worse, sometimes the extra register consumed by ebp/rbp-relative access is worse. It's really something that needs to be decided case-by-case like a intelligent Asm programmer would.
Is the "real solution" a case of letting "the perfect" get in the way of "good enough" (sic)?
In fact it might be the exact opposite. We have very complicated mechanisms that try to keep track of every possible optimization. Maybe that effort is getting in the way of a solution that slightly limits frame pointer optimization but easily figures out the value in all situations.
So because the "perfect" mechanism is having implementation problems, and nobody has tried to make the "good" mechanism, we're stuck with the very mediocre "reserve a register".
As someone who’s excited about Fedora’s change, I agree. But while waiting for that to happen, I want to have a good debug/tracing experience, which currently only frame pointers can provide.
What's the rush? We've lived without adequate unwinding this long. We can afford to take a little bit more time and do it right.
Can you share an example of when that happens?
This is likely wrong. The kernel does not have a DWARF unwinder. He's probably just desymbolizing a poor stack trace taken by the kernel chasing ebps, not unwinding it with DWARF.
If a binary really wanted to hide its behavior (at least to the same extent that it could with -fomit-frame-pointer), it could just make fake stack frames beyond the current PC.
Not only do you get more complete and correct backtraces with DWARF, you also get preservation of stack-saved registers and the ability to reconstitute a subset of local state in a given frame.
libunwind is MIT licensed, not GPL.
I’ve also written a full DWARF stack unwinder myself, including a full DWARF expression interpreter — it’s nowhere near 10k lines of code.
There's a related problem of compilers generating imperfect debuginfo that fails to reconstruct local variables from registers/memory due to an incomplete DWARF representation, but I don't think that plays a roll in unwinding. (But maybe something similar does.)
Unless DWARF can handle it fine and compilers are thoroughly broken, in which case it's a tossup between "DWARF is fundamentally flawed" and "compiler designers are lazy for some unknown reason"?
It _could_, but the kernel folks have made it pretty clear they don’t want to put any code related to dwarf in the kernel.
DWARF is useful for so much more than flame graphs. Flame graphs are cool and useful, but they're over hyped. Anybody who has ever spent any significant portion of their career debugging compiled binaries knows that proper debug information is infinitely more useful than flame graphs could ever be. Flame graphs are also useful... very useful. But that's my point: if they advocated even half as much for ensuring everybody made debug info more readily accessible by default as they did crying for frame pointers, the world would be a much better place.
Is DWARF too complex? There are many layers to DWARF, and there are compromises everybody could make so we can get at an optimal balance for which DWARF info we should ensure is always there. And we could also spend more time fixing tooling so it works properly with that minimal debug data.
AFAIK there are two issues at play here.
1. There's a bug in the Linux kernel where it sometimes return an invalid RBP register through `perf_event_open`.
2. The compiler doesn't generate the necessary DWARF info to support unwinding asynchronously, so depending on where exactly you land in a function you might not be able to unwind the stack. (This is why unwinding from *within* the program always works correctly but unwinding from *outside* the program with perf doesn't.)
Source: I wrote my own sampling profiler.
And there's of course the issue that you'd need everything to be compiled with it (because the program you're profiling is going to call into external dynamically linked libraries), which on an average system you most likely won't have.
But leaving that aside, for profiling, dwarf is just very expensive (perf, where the data volume explodes, due to copying stacks) or not available (bpf stacks), because the unwinding has to happen in the kernel. Even if the kernel had a dwarf unwinder, it's vastly more expensive to do that, compared to unwinding via frame pointers.
Try running a system wide profile on larger and busy box. Dwarf based call graphs are basically not usable. The profile quickly is ginormous, and viewing profiles is extremely slow.
I want both really badly but the fact is that flamegraphs drive performance optimizations which save companies millions of dollars, while symbols sometimes fix bugs and sometimes get blamed for making their software easy to reverse engineer. You can understand why one gets more focus than the other.
> The thing is that DWARF unwinding (the only userspace option that doesn’t use frame pointers) is really unreliable. In fact it has such serious problems that it’s not that usable at all.
I'm not that familiar with the problem space - can you elaborate on why DWARF is better? or speculate on why the author is experiencing (apparent) reliability issues with it?
This results in a net saving of money over time.
Don't tell the golang folks!
Besides the inbuilt panic behavior which prints a very helpful backtrace, non-fatal error handling is spinkled all over with fmt.Sprintf "I am here at 134738djfjrsj so you find me in the logs"
Don't confuse problems in Go language with your own ineptitude to use it.
That's quite a stretch, and I'm saying this as someone who probably believes in things others would consider conspiracy theories. Freedom is irrelevant when you already have the source and the binaries and can do whatever you want with them... and the latter applies even if you're analysing closed-source software.
All for a pointless ~1% performance boost.
I can rebut with "All for a pointless waste of a register for something that will only be useful the <1% of the time the code is actually run."
If I wanted to do the same with DWARF I'd need to link some GPL library or write 10,000 lines of byzantine code to figure out which function called the one that's crashing.
Or just learn how to read a memory dump. If your code is somehow crashing so often enough that it becomes such a burden, then I'd say the solution is elsewhere. It's not hard to walk the stack manually.
When I worked at Google, profiling was done so often that it certainly exceeded 1%. There's something called Google-Wide Profiling[1] but teams often opt into much more frequent profiling of important micro services they deploy. It's not just for identifying crashes, but more for performance optimization. It's liberating to debug performance bottlenecks when you can just pull up a flame graph from half an hour ago and compare that against a flame graph from one week ago without worrying about re-deploying the old version to get profiling data.
Btw I think this is somewhat of a cultural bias depending on your previous experience. Some people simply value performance at all costs. Some people value more observability and debugability.
[1]: https://research.google/pubs/pub36575/
So a 10% use case for 0.001% of engineers out there. And I'm pretty sure you didn't run stock Fedora distro on servers either ?
If you're that high end that profiling code for few % gains is worth millions you can just compile the stuff with whatever flags and optimization you need instead of relying on distro to reduce performance for everyone to appease to few engineers at big corp
I've read more about this inconsistency, and it turned out that at some point libc++ changed something in the string class and it caused a 1% performance regression in "an important google-wide benchmark". That 1% of performance was so important, that since that up to at least few years ago every C++ programmer had to put up with that. I guess it makes some sense - 1% of CPU cost across the whole Google fleet is a lot of saved CPU hours (and that equals money).
This is a skill that is not only hard to learn, but also takes time to apply properly. Yes, I can read a memory dump. But I'd sacrifice more than 1% of performance to not do it manually. And the infra cost to account for that is also lower than the value of my time.
Telling basically all devs without such experience to "just" learn an advanced skill is not a good solution.
That may be true, but having proper stack traces will probably cumulate to optimizations that gain much more than the 1% loss of enabling frame pointers [1].
Ideally, it would be much easier to rebuild whole Linux systems. Then people who need it could enable frame pointers. Sadly, rebuilding a mainstream Linux distribution is much harder than rebuilding e.g. Gentoo or NixOS. Though, in practice even rebuilding a NixOS system is fraught with issues. I have done large rebuilds, but it would often fail because e.g. unit tests have issues that only manifest themselves with enough build parallelism.
[1] Note that on x86_32, the performance improvement was larger due to register pressure.
With 32 bit code this was absolutely not the case. I have not looked so closely at 64 bit code, but in 32 bit land -fomit-frame-pointer was often closer to a 20-30% performance boost. Of all of gcc's optimization options it was the only one that I ever found to make a significant difference in real code.
More often than not, they're invalidated by some advancement in either computer hardware or software. Not just "no longer relevant", but often counterproductive.
Worse still, these internalised tricks and rules are often related to Moore's law scaling, making them exponentially wrong over time. Not in the figurative sense, but a literal one.
You see this with people claiming that I'm exaggerating when I say all code should be parallel. Meanwhile AMD is about to sell a 128 core / 256 thread processor. You can have two per host for a staggering 512 hardware threads. Single threaded code is capable of utilising just 0.2% of that machine!
A similar "tuning best practice" that I still see applied in the field is database servers with dedicated data, log, temp, and templog drives... on cloud VMs where this is guaranteed to have worse performance than simply pooling the same amount of disk into a single logical drive.
People pick these things up and hold on to them when they're wrong by orders of magnitude.
I can't think of many (any?) other industries where more experienced senior staff can be this wrong about as many things...
-https://manpages.ubuntu.com/manpages/xenial/man1/pxz.1.html
-https://lzip.nongnu.org/plzip.html
And so on, however the Compression-ratio is slightly worse
We’ve been promised these magical ultra parallel machines for a decade now and either they’re much harder to build than the alarmists say, or there's simply no market.
What we are seeing is a hierarchy of processing cores. It's already visible even in the desktop market. Intel has their efficiency cores. Apple has the power and efficiency cores. And the GPU on top. So we already have 3 different stages here, at the top the least amount of cores but the best single threaded performance going all the way down to, let me check how many threads, almost 10000 threads on a flagship consumer GPU running simultaneously right now.
Intel tried with their Larrabee on what happens if they just toss in ton of traditional low performance CPU cores. It failed to perform. It's really hard to beat the modern SIMT style GPUs when it comes to massively parallel computation.
Now whether or not all 128 threads need to operate on the same chunk of memory or not is somewhat up for debate. It is very good for some applications (Let's Encrypt issues all of the Internet's TLS certificates with one Postgres instance), and irrelevant for others (just run 128 copies of your node app and send requests to them at random).
And on AWS there are instance types with 448 VCPUs [1] and in fact 128 VCPUs is a very common instance type. In fact with the growth of Kubernetes these larger configurations are increasingly popular because it's more cost efficient to pack the containers into fewer/larger nodes.
Also would add that Scala is in my opinion by far the best language for writing safe, highly concurrent code when you pair it with frameworks like ZIO. [2]
[1] https://instances.vantage.sh
[2] https://zio.dev
That kind of is defeating your argument though: If people pack many containers on large boxes, the individual container probably is not making use of the possible parallelization across all cores.
If you have an app that runs 24/7 at 100% bursting across all cores then Kubernetes will simply schedule a single container on that instance.
Edit to clarify: SwapPING bad, having swap space good.
> Disabling swap doesn't prevent pathological behaviour at near-OOM, although it's true that having swap may prolong it.
Basically, the memory manager will just bulldoze through moving all the other applications to the swap to let the leaking program pursue its leak.
> you are left with a system in an unpredictable state. Having no swap doesn't avoid this.
Yes, it does: the program that leaked my 16GB is killed at the very beginning of the situation, and I don't have to either wait dozens of minutes for it to fill the swap on top of that to finally be killed, or just hard reboot the machine and lose the whole state.
How, and by whom? In a situation where you are running out of memory, it is programs that allocate the most often that have the highest chance of seeing an OOM, not those that allocate the most total amount.
Also, on Linux, just because you don't have a swap partition doesn't mean things won't get swapped out to disk. Any memory mapped files will be swapped back to disk before issuing OOM problems - and the most common case of memory mapped files is executables and libraries. Which is of course the very thing that leads to a kind of livelock of swapless Linux systems that live close to 100% memory utilization: every time a thread is woken up, the instruction pipeline stalls until the next CPU instruction can be read from disk, slowing everything to a crawl.
By the kernel OOM killer? Is that supposed to be a trick question?
> In a situation where you are running out of memory, it is programs that allocate the most often that have the highest chance of seeing an OOM, not those that allocate the most total amount.
I don't really care; it still beats the machine being unresponsive for hours, especially if it's a remote one I dont have access to.
> Also, on Linux, just because you don't have a swap partition doesn't mean things won't get swapped out to disk. Any memory mapped files will be swapped back to disk before issuing OOM problems
That's not what is commonly understood by swapping though; that's a cache purge.
> every time a thread is woken up, the instruction pipeline stalls until the next CPU instruction can be read from disk, slowing everything to a crawl.
When you have swap enabled, yes; that the whole problem. However, IME, OoM just leads to the problematic program being quickly killed, then the user being able to go back to work.
I think this is the divide. In my experience, OOM killer rarely kills the culprit. It doesn't actually know which programs are behaving irregularly (that's the 'trick' in his 'trick question') and the OOM score heuristic isn't reliable in desktop usecases. If you know ahead of time which process will likely cause you trouble, you can adjust oom_score_adj for that process to make it the first target, but this is not great for common desktop use.
> That's not what is commonly understood by swapping though; that's a cache purge.
No, it's exactly a form of swapping. I'm not talking about purging Linux's IO caches.
I am talking about taking a memory resident page of your process and unmapping it, such that the next it is accessed it causes a page fault and the VMM loads it back from disk. When you have swap space, this can happen to any page. When you don't, this can still happen to read-only pages that already have a corresponding file on disk - which is usually your executable and any shared libraries you loaded.
Now, whether you'll really hit this situation or not really depends on how much of your memory is being taken up by the code of running processes. In many common server workloads, this should be very rare. However, in desktop workloads, where you might have lots of programs running at the same time, the total amount of memory being taken up by all their code may end up high enough to enter this state before calling the OOMKiller.
If you've got something leaking a lot and fast, 512MB swap is going to fill up in seconds, not minutes. If you have a slower leak, swap usage provides a clear signal something is wrong and may give you a chance to intervene before the OOM killer. Even in fast death, swap usage may be useful forensics, if you record it at a high enough time resolution.
This is exactly why I don't have swap for over a decade on my machines (since my machines have 16+ GB RAM). The mere 1/32 improvement is not worth the headaches of swap.
I actually have been forced to go back to desktop Linux again recently after almost 20 years of avoiding it. It's pretty disappointing to see that Linux's OOM handling behaviour is still completely broken (at least on desktop). Windows and Mac behave much more sanely when you run out of memory, gradually slowing down rather than suddenly freezing for like a minute while it figures out what to kill.
I did eventually find a Reddit thread that suggested enabling zram (compressed ram swap disk) which does appear to have helped but why isn't that the default? Also I now get a kernel panic about once a week but that could be unrelated.
You can still server 512 different clients at once and still utilize 100% of the CPU.
The parallelism is only a requirement if you want to compute one thing faster, not a bunch of things at once. Very much the desirable property for databases (although even that can be worked around with sharding if you're stubborn enough) but many developers code could be painfully serial yet still be perfectly fine to scale.
Lawyer enters the thread.
It's more common than you think. When complex work is done within power structures, it's just not enough to be "right" about something. You also have to be able to sell your solution to people who are wrong and powerful, which is sometimes not possible.
A lawyer from 10 years ago that memorised some good "rules of thumb" about generic matters such as liability or court procedure won't have to throw out everything they know to be relevant today or another decade into the future.
Meanwhile in IT, I regularly see network engineers upgrading links from 500 megabits to 1 gibabit when they should be looking at upgrading to 100 or even 200 gigabits.
Their mental model of what's fast, relevant, or desirable is "off" by two orders of magnitude. Their knowledge was accurate just a decade ago, but is hilariously wrong now.
I can't think of any other mainstream profession where institutional knowledge is deprecated as quickly.
It's not guaranteed to be worse. For latency sensitive OLTP workloads dedicated, smaller, disks for the journal can result in substantially lower & predictable latency. Random reads for the data exhausting iops, causing stalls of journal writes isn't fun. There's also per-disk queuing on a bunch of levels, and avoiding intermingling data and journal writes can be quite beneficial.
That's not to say it's always better, at all. But making it out to be a stupid thing one should never do isn't wise either.
I mean, you're clearly exaggerating. My ad-hoc Python scripts don't have to be parallel.
Promise? That's fantastic!
Because few things deserve to use more than 0.2% percent of that machine.
Sure, you could parallel it, and now it takes 0.3% to do what could easily be done quicker and saner by 0.1%.
I too like to anthropomorphize scripts and punish them for their transgressions by constraining their resources.
> I love these "rules of thumb", "black magic sorcery", and "best practices" that grey-haired old wizards pick up.
Just quoting this to point out what I mean by addressing your tone. It's fine to disagree, but this feels like you're setting up that everyone who disagrees with you must be condescending and stuck in the past. Yes, performance advice is often tied to the specific situation, but that's also true for the things you state.
> Worse still, these internalised tricks and rules are often related to Moore's law scaling, making them exponentially wrong over time. Not in the figurative sense, but a literal one.
Implying that Moore's Law makes anything exponentially faster is an anachronism (talk about old knowledge!), it only makes things exponentially more parallel nowadays. See next point why that's relevant.
As others have already clarified, -fomit-frame-pointer was sometimes 20% on x86. For some applications, people would kill for a 20% boost in single-threaded performance, even today!
And importantly, that this specific optimization has become useless on x64 says nothing about the value of optimizations in general. Different ones are still useful, to give an example let's say SIMD (where it applies).
> You see this with people claiming that I'm exaggerating when I say all code should be parallel. Meanwhile AMD is about to sell a 128 core / 256 thread processor. You can have two per host for a staggering 512 hardware threads. Single threaded code is capable of utilising just 0.2% of that machine!
I'm not sure which strawman "grey-haired" developer you have in mind that buys a 512-thread server to run single-threaded code on it and then needs you to explain to them that they should parallelize it. Maybe you've met that person. But in general: Doesn't most code live in web servers nowadays anyway? If so, it's either already running in parallel, or can be made to do so with very little effort.
BTW, parallelizing code in a web server can be harmful (!) to total throughput due to communication overheads. This is something that I find people don't always realize when they naively parallelize something, and is also sometimes missed by benchmarks that don't put the server under heavy load.
> A similar "tuning best practice" that I still see applied in the field is database servers with dedicated data, log, temp, and templog drives... on cloud VMs where this is guaranteed to have worse performance than simply pooling the same amount of disk into a single logical drive.
A statement like this depends on many concrete details about the specific cloud infrastructure (you didn't say AWS, you said "cloud"), and cloud VMs are usually not the best performing way to run a database anyway. So again, this is very context-specific.
Also, if database performance is critical enough that you consider tuning these things, then renting bare metal should at least be a consideration as well. People who have never measured it are sometimes surprised by how large the virtualization overhead is (and how cheap it is to rent physical servers). On actual hardware, this advice might still be relevant.
> I can't think of many (any?) other industries where more experienced senior staff can be this wrong about as many things...
The actual point is that you should measure things in concrete cases, not make general claims. In that sense, I unfortunately don't find your "updated" way to state what supposedly is and isn't relevant not much of an improvement to the "outdated" claims you argue against so vehemently.
So what? Who cares? I find the whiny tone of your post to be offensive.
Whether or not this argument holds water depends entirely on the context of the situation; what is the relationship between the program being considered and the computer? If the program is the reason for that hardware to exist, then obviously you want to use that hardware to it's fullest potential. If you've bought a 256 core machine specifically to run this program, then obviously you want to use all those cores. Some sort of threaded or multiprocess architecture should be an easy sell in this sort of circumstance. It shouldn't even be a discussion.
But on the other hand, what if the program under consideration is ancillary? In that case, the program should use the bare minimum resources necessary, so as to not get in the way of whatever the primary program may be. If you have say.. a music playing daemon, then odds are your program should not be trying to use the host machine to it's fullest because that hardware actually exists to get some other job done. Ancillary programs should be designed to survive on leftover scraps, not with the assumption that they'll have first dibs on the use of the hardware.
Even on a laptop, I now have 16 threads and almost never see more than 1 utilised even when I'm waiting.
Any time a human waits for a computer that is just 6% utilised, a developer screwed up.
Just this minute, I downloaded a program installer using a 10 Gbps Internet link.
Yes. Ten gigabits, dedicated to me, personally! Your idea of "what is a fast Internet link" is wrong.
I downloaded a 600 MB software package in less than a second, and then had to wait for 30 seconds while the anti-malware tools scanned it... with a single thread.
Extracting the file was 10 seconds because 'zip' is by default single threaded.
Installing it took a solid minute during which my CPU usage never peaked above 2 cores because the installer is also single threaded. The only reason it uses more than 1 core is because the anti-malware scans it... again... on a second thread.
Similarly, any time I see anyone manipulating ~1 GB of data, invariably they do it with GUI or CLI tools that are single threaded. This was desirable in an era when everyone had a single mechanical HHD. Multiple read or write threads would have caused disk head seeks, slowing down the process. These days when everyone has NVMe SSDs, the only way to fully utilise the 500K IOPS available from a $150 device is by throwing dozens of parallel threads at it!
Even as someone who tries to actively question assumptions that may no longer hold I still find myself constantly flat-footed.
I can't pretend like I know all the various advancements in processor architecture of the past 5 years (since 2018) or how modern compilers exploit them and also the latest techniques in GPU computing.
Heck even old rules of thumb like the speed of platter drives are being challenged - mechanical drives with 500M/s throughput are hitting the shelves this year. I just upgraded a platter RAID where I'm getting ~1000M/s. The SSD delta has changed (that's of course assuming SSD hasn't moved which I'm sure is also false). I'm sure there's new advancement in material science and manufacturing that I'm also not privy too which has made this possible.
Storage alone is another vast world. I don't know all the features of ext4, xfs, zfs, hammer, and btrfs or how to competently compare and tune them for various workloads or how that analysis has changed in say the past 24 months. This is something I became intimately aware of when upgrading the array. I didn't know any of the modern tools or approaches.
All knowledge in this space eventually becomes vestigial and outdated and it's physically impossible to keep up. I haven't even kept up with the latest C features. Then there's Go, Rust, Typescript and countless other languages that each require at minimum, a hundred hours a year each to competently follow.
The best we can do is have humility and continually approach things as a student because we never know what table leg has been knocked out underneath us without our noticing.
X86. No other 32b architecture is that register-starved even if the larger number of registers is somewhat mitigated by being load-store.
Nowadays X64 has a decent amounts of registers + the HW internally has several times more the registers, so saving one is almost meaningless.
Do they? It seems like everyone is moving away from scripting languages for new projects these days, at least where web browser-related requirements doesn't push them into using Javascript.
Indeed, before this current pendulum swing, scripting languages had a good run. I'm not convinced it was because of stack traces, though. They won hearts and minds on promises of no longer having to do "XML sit-ups" and whatnot.
> Compiling the kernel turned out to be 2.4% slower, and a Blender test case regressed by 2%. The worst case appears to be Python programs, which can see as much as a 10% performance hit. To many, these costs were seen as unacceptable.
Sacrificing that much for the absolute minority of people that need to dig that much into performance doesn't seem to be worth it for distro. Would be interesting to see why some cases still get that much regression tho.
The Intel “ASM86 Language Reference Manual” calls it that as well.
https://community.intel.com/cipcp26785/attachments/cipcp2678...
I mean, presumably you just answered your own question: while corporate software might not be able to link GPL code, Linux software — that mostly already is GPLed — certainly can. In fact, that's presumably why the library you're referencing (libdwarf?) is under the GPL in the first place — it was created by a Linux developer, for use in their own and others' FOSS Linux software, where it easily solves this problem for them. And because they have this easy solution to this problem on hand, they don't really see the need to turn on frame pointers in their projects' Makefiles.
Man hours (spent recompiling the kernel with the right flags in this case) are often undervalued when comparing different solutions.
I have found this is especially true when doing cost analysis. It is so easy to look at something that costs $10,000 a month and think it's expensive, despite the fact that it saves twice that much in developer time (salary).
What prevents anyone of us to maintain a fork with same SRPMS as fedora but keeping the omition of frame pointers? I guess someone will do that if there really is an observable performance advantage. There are already fedora forks such as Nobara providing some kernel patches and some specific compilation options or patches for some apps.
Also 1-2% performance hit is peanuts, given the benefit.
It isn't a religious battle of ideology, it's a concrete tradeoff of something small for something large.
I wish we could have frame pointers, because they would make working in embedded land so much easier and more reliable, but a 10% increase in code size just isn't worth it.
We currently load the entire program into RAM to make things easier, so anything to lower storage requirements is a win. Someday we'll be able to mmap the disk to do XIP, but we don't have that implemented yet.
I imagine it's pretty substantial compared to the cost of putting a DWARF unwinder in the kernel.
Especially when all those apps were built on these platforms exactly because of how annoying it is to debug C programs. The total energy usage could even be negative due to this.
I doubt C debugging has had a meaningful role in the use of electron in any case.
The analysis I was doing when writing this blog post was about understanding where 50% of the time of my program was going. The fix literally doubled the speed of the program: https://bugzilla.redhat.com/show_bug.cgi?id=2166686
If you unwind using frame pointers through a Python program, you'll get a bunch of meaningless interpreter frames instead of information about the operation of the program being interpreted. That, and since the FP unwinder can't sleep and can't take page faults, it'll fail to traverse stack frames that happen to be paged out.
The right way to solve the kernel unwind of userspace problem is to get the kernel out of the business of unwinding userspace.
Concrete proposal here: https://lists.fedoraproject.org/archives/list/devel@lists.fe...
Unwinding only paged-in, AOT-compiled native code is a step in the wrong direction, especially because it's precisely the people for whom unwinding kinda-sorta works now (kernel, libc, C language diehards, etc.) who would need to approve and coordinate a more robust and comprehensive solution that also, unlike frame pointers, has zero cost when not in use.
https://github.com/torvalds/linux/blob/0513e464f9007b70b9674...
https://www.brendangregg.com/Slides/KernelRecipes_Perf_Event... (pp. 53-58)
https://docs.python.org/3.12/howto/perf_profiling.html#perf-...
But the main point is that "inability to support unwinding of anything that isn't AOT compiled native code" is factually incorrect: as far as I know, frame pointers plus this "hack" is currently the only widely available way to do just that.
There's still a bunch of pushes and pops, I guess because that's shorter code than storing into an explicit offset off the stack pointer, but since you've read the stack pointer anyway and synchronized it, the uop sequence and instruction dependencies wouldn't end up being a lot different I would have thought.
It’ll take some years until frame pointers become more mainstream. Still, some people might decide to never enable them in libraries they use or their own code.
That’s why I’ve been working on a BPF DWARF-based unwinder to fill the gap. It’s fully OSS (Apache 2.0 for user space and GPL for BPF) and should get out of beta in ~1month.
Previous HN discussion [0]. Happy to help anybody that wants to integrate it in their tools / systems.
- [0]: https://news.ycombinator.com/item?id=33788794
I still have nightmares about it. What a terribly put together thing. Moving to Debian was such a breath of fresh air, packages that works!
The only thing that was annoying at the time was having to manually download and compile third party software as well as some of its depencency. In reality it wasn't that much of a big deal. You just had to know how to read a readme and install text file really. But this is kind of a moot point in 2023 with: 1. slackbuilds and its clients, which is pretty much like the AUR for archlinux users or pkgsrc and makes you a lot of additionnal software available very easily. 2. All the software nowadays made installable with flatpak, appimage or via some bash script. 3. podman and docker
If anything modern software deployment methods made slackware experience closer to rpm or deb based distros than 20 years ago.
I much preferred stock.
Was a fantastic decision, I think the project is run much better and the pace of development and innovation for software devs specifically is much better.
https://news.ycombinator.com/item?id=34426745 ("Fedora's tempest in a stack frame" from lwn.net, 17 days ago, 31 comments)