I was at Google in 2005 on the other side of the argument. My view back then was simple:
Even if $BIG_COMPANY makes a decision to compile everything with frame pointers, the rest of the community is not. So we'll be stuck fighting an unwinnable argument with a much larger community. Turns out that it was a ~20 year argument.
I ended up writing some patches to make libunwind work for gperftools and maintained libunwind for some number of years as a consequence of that work.
Having moved on to other areas of computing, I'm now a passive observer. But it's fascinating to read history from the other perspective.
It “wastes” a register when you’re not actively using them. On x86 that can make a big difference, though with the added registers of x86_64 it much less significant.
It's not just the loss of an architectural register, it's also the added cost to the prologue/epilogue.
Even on x86_64, it can make a difference, in particular for small functions, which might not be inlined for a variety of reasons.
A typical case would be C++ virtual member functions. (They can sometimes be devirtualized, or speculatively partially devirtualized, using LTO+PGO, but there are lots of legitimate cases where they cannot.)
CPUs spend an enormous amount of time waiting for IO and memory, and push/pop and similar are just insanely well optimized. As the article also claims, I would be very surprised to see any effect, unless that more instructions themselves would spill the I-cache.
I've seen around 1-3% on non micro benchmarks, real applications.
Aee also this benchmark from Phoronix [0]:
Of the 100 tests carried out for this article, when taking the geometric mean of all these benchmarks it equated to about a 14% performance penalty of the software with -O2 compared to when adding -fno-omit-frame-pointer.
I'm not saying these benchmarks or the workloads I've seen are representative of the "real world", but people keep repeating that frame pointers are basically free, which is just not the case.
Wasting a register on comparatively more modern ISA's (PA-RISC 2.0, MIPS64, POWER, aarch64 etc – they are all more modern and have an abundance of general purpose registers) is not a concern.
The actual «wastage» is in having to generate a prologue and an epilogue for each function – 2x instructions to preserve the old frame pointer and set a new one up, and 2x instruction at the point of return – to restore the previous frame pointer.
Generally, it is not a big deal with an exception of a pathological case of a very large number of very small functions calling each other frequently where the extra 4x instructions per each such a function will be filling up the L1 instruction cache «unnessarily».
Yes, inlining (and LTO can take it a notch or two higher) does away with the problem altogether, however the number of projects that default to «-Os» (or even to «-O2») to build a release product is substantial to large.
There is also a significant number of projects that go to great lengths to force-override CFLAGS/CXXFLAGS (usually with «-O2 -g» or even with «-O») or make it extraordinary difficult to change the project's default CFLAGS, for no apparent reasons which eliminates a number of advanced optimisations in builds with default build settings.
I wasn't talking about functional problems. It was a simple observation that big companies were not going to convince Linux distributors to add frame pointers anytime soon and that what those distributors do is relevant.
All of the companies involved believed that they were special and decided to build their own (poorly managed) distribution called "third party code" and having to deal with it was not my best experience working at these companies.
Oh, I just assumed you were talking about Google's Linux distribution and applications it runs on its fleet. I must have mis-assumed. Re-reading... maybe you weren't talking about any builds but just whether or not to oppose kernel and toolchain defaulting to omit frame pointers?
Google didn't have a Linux distribution for a long time (the one everyone used on the desktop was an outdated rpm based distro, we mostly ignored it for development purposes).
What existed was a x86 to x86 cross compilation environment and the libraries involved were manually imported by developers who needed that particular library.
My argument was about the cost of ensuring that those libraries were compiled with frame pointers when much of the open source community was defaulting to omit-fp.
That was done in 2005. But the task of auditing the supply chain to ensure that every single shared library you ever linked with was compiled a certain way was still hard. Nothing prevented an intern or a new employee from checking in a library without frame pointers into the third-party repo.
In 2024, you'd probably create a "build container" that all developers are required to use to build binaries or pay a linux distributor to build that container.
But cross compilation was the preferred approach back then. So all binaries had a rpath (run time search path to look for shared library) that ignored the distributor supplied libraries.
Having come from a open source background, I found this system hard to digest. But there was a lot of social pressure to work as a bee in a system that thousands of other very competent engineers are using (quite successfully).
I remember briefly talking to a chrome OS related group who were using the "build your own custom distro" approach, before deciding to move to another faang.
Since Google would rather have the best brains in the industry build the next search indexing algorithm or the browser, they didn't have the time to invest human capital into building a better open source friendly dev environment.
A natural alternative is to contract out the work. Linux distributors were good candidates for such contract work.
But the vibe back then was Google could build better alternatives to some of these libraries and therefore bridging the gap between dev experience as an open source dev vs in house software engineer wasn't important.
You could see the same argument play out in git vs monorepo etc, where people take up strong positions on a particular narrow tech topic, whereas the larger issue gets ignored as a result of these belief systems.
You do get occasional regressions. eg. We found an extremely obscure bug involving enabling frame pointers, valgrind, glibc ifuncs and inlining (all at the same time):
The clear and obvious win would have been adoption of a universal userspace generic unwind facility, like Windows has --- one that works with multiple languages. Turning on frame pointers is throwing in the towel on the performance tooling ecosystem coordination problem: we can't get people to fix unwind information, so we do this instead? Ugh.
Yes, although the universal mechanisms that have been proposed so far have been quite ridiculous - for example having every program handle a "frame pointer signal" in userspace, which doesn't account for the reality that we need to do frame unwinding thousands of times a second with the least possible overhead. Frame pointers work for most things, and where they don't work (interpreted code) you're often not that interested in performance.
> every program handle a "frame pointer signal" in userspace
Yep. That's my proposal.
> which doesn't account for the reality that we need to do frame unwinding thousands of times a second with the least possible overhead
Yes, it does. The kernel has to return to userspace anyway at some point, and pushing a signal frame during that return is cheap. The cost of signal delivery is the entry into the kernel, and after a perf counter overflow, you've already paid that cost. Why would the actual unwinding be any faster in the kernel than in userspace?
Also, so what if a thread enters the kernel and samples the stack multiple times before returning to userspace? While in the kernel, the userspace stack cannot change --- therefore, it's sufficient to delay userspace stack collection until the kernel returns to userspace anyway.
You might ask "Don't we have to restore the signal mask after handling the profiling signal?"
Not if you don't define the signal to change the signal mask. sigreturn(2) is optional.
This sounds vastly more complex already than following a linked list. You've also ignored the other cost which is getting the stack trace data out of the program. Anyway I'm keen to see your implementation and test how it works in reality.
I propose that a frame pointer daemon be introduced too, for managing the frame pointer signals. We shall modify _start() to open up an io_uring connection to SystemD so that a program may share its .eh_frame data. That way the kernel can still unwind its stack in case apt upgrade changes the elf inode.
Neither of you has identified anything technically wrong with unwinding via signal and neither of you has proposed a mechanism through which we might support semantically informative unwinding through paged-out code or interpreted languages.
I don't need to. Fedora and Ubuntu have already changed their policies to restore frame pointers. As far as I can tell, your proposal is no longer on the table. If you aren't willing to accept the decision, then you should at least understand that the onus is on you now to justify why things need to change.
We have to deal with reality if we want to measure and improve software performance today. The current reality is that frame pointers are the best choice. Brendan's article outlines a couple of possible future scenarios where we turn frame pointers off again, but they require work that is not done yet (in one case, advances in CPUs).
I think this came off somewhat aggressive. I vouched for the comment because flagging it is an absurd overreaction, but I also don't think pointing out isolated individuals would be of much help.
Barriers to progress here are best identified on a community level, wouldn't you say?
But people, please calm down. Filing an issue or posting to the mailing list to make a case isn't sending a SWAT team to people's home. It's a technical issue, one well within the envelope of topics which can be resolved politely and on the merits.
The biggest objection seems to be the Java/JIT case. eh_frame supports a "personality function" which is AIUI basically a callback for performing custom unwinding. If the personality function could also support custom logic for producing backtraces, then the profiling sampler could effectively read the JVM's own metadata about the JIT'ted code, which I assume it must have in order to produce backtraces for the JVM itself.
Not a problem in practice. The way you solve it is to just translate DWARF into a simpler representation that doesn't require you to walk anything. (But I understand why people don't want to do it. DWARF is insanely complex and annoying to deal with.)
In this thread[1] we're discussing problems with using DWARF directly for unwinding, not possible translations of the metadata into other formats (like ORC or whatever).
I wasn't talking about other formats. I was talking about preloading the information contained in DWARF into a more efficient in-memory representation once when your profiler starts, and then the problem of "the overhead is too high for realtime use" disappears.
For a busy 64-CPU production JVM, I tested Google's Java symbol logging agent that just logged timestamp, symbol, address, size. The c2 compiler was so busy, constantly, that the overhead of this was too high to be practical (beyond startup analysis). And all this was generating was a timestamp log to do symbol lookup. For DWARF to walk stacks there's a lot more steps, so while I could see it work for light workloads I doubt it's practical for the heavy production workloads I typically analyze. What do you think? Have you tested on a large production server where c2 is a measurable portion of CPU constantly, the code cache is >1Gbyte and under heavy load?
I regularly profile heavy time-sensitive (as in: if the code takes too long to run it breaks) workloads, and I even do non-sampling memory profiling (meaning: on every memory allocation and deallocation I grab a full backtrace, which is orders of magnitude more data than normal sampling profiling) and it works just fine with minimal slowdown even though I get the unwinding info from vanilla DWARF.
Granted, this is using optimized tooling which uses a bunch of tricks to side-step the problem of DWARF being slow, I only profile native code (and some VMs which do ahead-of-time codegen) and I've never worked with JVM, but in principle I don't see why it wouldn't be practical on JVM too, although it certainly would be harder and might require better tooling (which might not exist currently). If you have the luxury of enabling frame pointers then that certainly would be easier and simpler.
(Somewhat related, but I really wish we would standardize on something better than DWARF for unwinding tables and basic debug info. Having done a lot of work with DWARF and its complexity I wouldn't wish it upon my worst enemy.)
DWARF-based unwinding can be a bottleneck for time-sensitive program analysis tools. For instance the perf profiler is forced to copy the whole stack on taking each sample and to build the backtraces offline: this solution has a memory and time overhead but also serious confidentiality and security flaws.
So if I get this correctly, the problem with DWARF is that building the backtrace online (on each sample) in comparison to frame pointers is an expensive operation which, however, can be mitigated by building the backtrace offline at the expense of copying the stack.
However, paper also mentions
Similarly, the Linux kernel by default relies on a frame pointer to provide reliable backtraces. This incurs in a space and time overhead; for instance it has been reported (https://lwn.net/Articles/727553/) that the kernel’s .text size increases by about 3.2%, resulting in a broad kernel-wide slowdown.
and
Measurements have shown a slowdown of 5-10% for some workloads (https://lore.kernel.org/lkml/20170602104048.jkkzssljsompjdwy@suse.de/T/#u).
There's always room for improvement, for example, Samply [0] is a wonderful profiler that uses the same APIs that `perf` uses, but unwinds the stacks as they come rather than dumping them all to disk and then having to process them in bulk.
Samply unwinds significantly faster than `perf` because it caches unwind information.
That being said, this approach still has some limitations, such as that very deep stacks won't be unwound, as the size of the process stack the kernel sends is quite limited.
> Profiling has been broken for 20 years and we've only now just fixed it.
It was a shame when they went away. Lots of people, certainly on other systems and probably Linux too, have found the absence of frame pointers painful this whole time and tried to keep them available in as many environments as possible. It’s validating (if also kind of frustrating) to see mainstream Linux bring them back.
I’m sincerely curious. While I realize that using dwarf for unwinding is annoying, why is it so bad that it’s worth pessimizing all code on the system? It’s slow on Debian derivatives because they package only the slow unwinding path for perf for example, for license reasons, but with decent tooling I barely notice the difference. What am I missing?
Using DWARF is annoying with perf because the kernel doesn't support stack unwinding with DWARF, and never will. The kernel has to be the one which unwinds the user space stacks, because it is the one managing the perf counters and handling the interrupts.
Since it can't unwind the user space stacks, the kernel has to copy the entire stack (8192 bytes by default) into the output file (perf.data) and then the perf user space program will unwind it later. It does this for each sample, which is usually hundreds of times per second, per CPU. Though it depends how you configured the collection.
That does have a significant overhead: first, runtime overhead: copying 8k bytes, hundreds of times per second, and writing it to disk, all don't come for free. You spend quite a bit of CPU time doing the memcpy operation which consumes memory bandwidth too. You also frequently need to increase the size of the perf memory buffer to accommodate all this data while it waits for user space to write it to disk. Second, disk space overhead, since the 8k stack bytes per sample are far larger than the stack trace would be. And third, it does require that you install debuginfo packages to get the DWARF info, which is usually a pain to do on production machines, and they consume a lot of disk space on their own.
Many of these overheads aren't too bad in simple cases (lower sample rates, fewer CPUs, or targeting a single task). But with larger machines with hundreds of CPUs, full system collections, and higher frequencies, the overhead can increase exponentially.
I'm not certain I know what you mean by the "slow unwinding path for perf", as there is no faster path for user space when frame pointers are disabled (except Intel LBR as outlined in the blog).
I assume you're talking about the “nondistro build”?
The difference between -g (--call-graph fp) and --call-graph dwarf is large even with perf linked directly against binutils, at least in my experience (and it was much, much worse until I got some patches into binutils to make it faster). This is both on record and report.
There are also weird bugs with --call-graph dwarf that perf upstream isn't doing anything about, around inlining. It's not perfect by any means.
I’m not sure what use case you’re coming from but it sounds like you’re saying something like: most end users don’t use a profiler or debugger so why should they pay the cost of debuggability? That’s fine I guess if you’re throwing software over a wall to users and taking no responsibility for their experience. But a lot of people who build software do take some responsibility for bugs and performance problems that their users experience. This stuff is invaluable for them. End users benefit (tremendously) from software being debuggable even if the users themselves never run a profiler or debugger that uses the frame pointers (because the developers are able to find and fix problems reported by other users or the developers themselves).
While I somewhat support the idea of avoiding debug-mode applications shipped to end customers, it seems in this particular case Brendan argues about the need to debug binaries in runtime on the server side. If there is a binary component running in production, the choice of running it with debug enabled (and/or with an ability to activate debug in runtime) is purely choice of the system owners (who nowadays are both the developers and the operators of components).
“Observability” primarily refers to ability to view system state beyond of what blackbox monitoring does. Again, the term primarily refers to server side operations and not to software shipped to end users.
As much as spying on users is ugly, it’s not related to debugging server side.
> Let's make software more inefficient (even if it's tiny, it all adds up!)
I'm not sure if you know who the author of that blog is, but if there's anyone in the world who cares about (and is knowledgeable about) improving performance, it's him. You can be pretty darn sure he wouldn't do this if he believed it would make software more inefficient.
You're confusing things with the analogy. The invasiveness of telemetry is collateral damage, not failure to meet its primary objective (gathering useful data for debugging, spying on people, whatever you think it is). In this case his primary objective literally is to improve performance... which aligns with your own goal, and which he has successfully demonstrated in the past.
We have an embarrassment of riches in terms of compute power, slowing down everything by a negligible amount is worth it if it makes profiling and observability even 20% easier. In almost all cases you cannot just walk up to a production Linux system and do meaningful performance analysis without serious work, unlike say a Solaris production system from 15 years ago.
We have an embarrassment of riches in terms of compute power yet all software is still incredibly slow because we and up slowing down everything by a "negligible" amount here and another "negligible" amount there and so on.
> In almost all cases you cannot just walk up to a production Linux system and do meaningful performance analysis without serious work
So? That's one very very very specific use case that others should not have to pay for. Not even with a relatively small 1% perf hit.
The issue is we can't see what running software is doing, because frame pointers are omitted, symbols are stripped, etc. in the name of performance, so we can't see what the software is even doing without changing it. And in many cases changing it will "fix" the symptom, before we can determine what the issue actually was.
When computers are slow, the primary way out is finding out WHY they are slow.
Finding this out requires... meaningful performance analysis. That's right, this 1% perf hit will make it extremely easy to find the reasons for 5,10,20,50% slowdowns, and enable developers (and you!) to fix them.
By making it easier to profile software, YOU can notice performance issues in the software you're running and deliver a nice patch that will improve performance by more than was lost, making this 1% slowdown a very effective, high-interest investment.
Nice sales pitch but in reality no one is going to spend time optimizing bottlenecks that aren't repdoducible. And if you can reproduce them you can do that on systems compiled for profiling to find the cause which isn't even the hard part anyway.
You seem to be stuck in the 90-ties. Computing is 64-bit nowadays, not 32-bit, and modern architectures/ABIs integrate frame pointers and frame records in a way that’s both natural and performant.
Of course, if you cede RBP to be a frame pointer, you may as well have two stacks, one which is pointed into by RBP and stores the activation frames, and the other one which is pointed into by RSP and stores the return addresses only. At this point, you don't even need to "walk the stack" because the call stack is literally just a flat array of return addresses.
Why do we normally store the return addresses near to the local variables in the first place, again? There are so many downsides.
It simplifies storage management. A stack frame is a simple bump pointer which is always in cache and only one guard page for overflow, in your proposal you need two guard pages and double the stack manipulations and doubling the chance of a cache miss.
Yes, two guard pages are needed. No, the stack management stays the same: it's just "CALL func" at the call site, "SUB RBP, <frame_size>" at the prologue and "ADD RBP, <frame_size>; RET" at the epilogue. As for chances of a cache miss... probably, but I guess you also double them up when you enable CFET/Shadow Stack so eh.
In exchange, it becomes very difficult for the stack smashing to corrupt the return address.
Note the ‘shadow stacks’ CPU feature mentioned briefly in the article, though it’s more for security reasons. It’s pretty similar to what you describe.
Shadow stacks have been proposed as an alternative, although it's my understanding that in current CPUs they hold only a limited number of frames, like 16 or 32?
You may be ready for Forth [1] ;-). Strangely, the Wikipedia article apparently doesn't put forward that Forth allows access both to the parameter and the return stack, which is a major feature of the model.
I'm not sure what you're referring to with "vocabulary stack" here, perhaps the dictionary? More of a linked list, really a distinctive data structure of its own.
Maybe OP refers to vocabulary search order manipulation [1]. It's sort of like namespaces, but "stacked". There's also the old MARKER and FORGET pair [2].
The dictionary pointer can also be manipulated in some dialects. That can be used directly as the stack variant of the arena allocator idea. It is particularly useful for text concatenation.
While here, why do we grow the stack the wrong way so misbehaved programs cause security issues? I know the reason of course, like so many things it last made sense 30 years ago, but the effects have been interesting.
>Why do we normally store the return addresses near to the local variables in the first place, again? There are so many downsides.
The advantage of storing them elsewhere is not quite clear (unless you have hardware support for things like shadow stacks).
You'd have to argue that the cost of moving things to this other page and managing two pointers (where one is less powerful in the ISA) is meaningfully cheaper than the other equally effective mitigation of stack cookies/protectors which are already able to provide protection only where needed. There is no real security benefit to doing this over what we currently have with stack protectors since an arbitrary read/write will still lead to a CFI bypass.
> The advantage of storing them elsewhere is not quite clear (unless you have hardware support for things like shadow stacks).
The classic buffer overflow issue should spring immediately to mind. By having a separate return address stack it's far less vulnerable to corruption through overflowing your data structures. This stops a bunch of attacks which purposely put crafted return addresses into position that will jump the program to malicious code.
It's not a panacea, but generally keeping code pointers away from data structures is a good idea.
That's very interesting to me - I had seen the `[unknown]` mountain in my profiles but never knew why. I think it's a tough thing to justify: 2% performance is actually a pretty big difference.
It would be really nice to have fine-grained control over frame pointer inclusion: provided fine-grained profiling, we could determine whether we needed the frame pointers for a given function or compilation unit. I wouldn't be surprised if we see that only a handful of operations are dramatically slowed by frame pointer inclusion while the rest don't really care.
Sure, but how many of the people running distro compiled code do perf analysis? And how many of the people who need to do perf analysis are unable to use a with-frame-pointers version when they need to? And how many of those 10% perf improvements are in common distro code that get upstreamed to improve general user experience, as opposed to being in private application code?
If you're netflix then "enable frame pointers" is a no-brainer. But if you're a distro who's building code for millions of users, many of whom will likely never need to fire up a profiler, I think the question is at least a little trickier. The overall best tradeoff might end up being still to enable frame pointers, but I can see the other side too.
It's not a technical tradeoff, it's a refusal to compromise. Lack of frame pointers prevents many groups from using software built by distros altogether. If a distro decides that they'd rather make things go 1% faster for grandma, at the cost of alienating thousands of engineers at places like Netflix and Google who simply want to volunteer millions of dollars of their employers resources helping distros to find 10x performance improvements, then the distros are doing a great disservice to both grandma and themselves.
I mean if you need to do performance analysis on a software just recompile it. Why it's such a big deal?
In the end a 2% of performance of every application it's a big deal. On a single computer may not be that significant, think about all the computers, servers, clusters, that run a Linux distro. And yes, I would ask a Google engineer that if scaled on the I don't know how many servers and computers that Google has a 2% increase in CPU usage is not a big deal: we are probably talking about hundreds of kW more of energy consumption!
We talk a lot of energy efficiency these days, to me wasting a 2% only to make the performance analysis of some software easier (that is that you can analyze directly the package shipped by the distro and you don't have to recompile it) it's something stupid.
The average European home consumes 400 watts at any given moment. Modern digital smart meters can consume 4 watts on average, which is 1% of a household's power consumption. On the grand scheme of society, these 1% losses in each home add up. If we consider all the grid monitoring equipment that's typically employed by electrical companies outside the home, the problem becomes much greater. In order to maximize energy efficiency and improve our environmental footprint, we must remove these metering and monitoring devices, which don't actually contribute to the delivery and consumption of power.
> I mean if you need to do performance analysis on a software just recompile it. Why it's such a big deal?
Not having them enabled on all dependencies makes them significantly less useful if your application interacts with the system in a meaningful way. "Just recompile" (glibc + xorg + qt + whatever else you use) is a very hefty thing.
Having them enabled by default also enables devs to ask their users for traces. I can reasonably tell someone to install sysprof or something, click a few buttons and send me a trace. I cannot reasonably tell them to compile the project (+ dependencies). And tbh, even devs cba if they have to jump through too many hoops.
> We talk a lot of energy efficiency these days, to me wasting a 2% only to make the performance analysis of some software easier (that is that you can analyze directly the package shipped by the distro and you don't have to recompile it) it's something stupid.
We're trading all kinds of efficiencies for all kinds of niche benefits all the time. This enables analysis of what is shipped in real use across the whole stack with the alternative being much less useful or a lot more (human) work. To me it's worth 2% and I'm shocked (well... not really) it's not for more people.
I would say the question here is what should be the default, and that the answer is clearly "frame pointers", from my point of view.
Code eking out every possible cycle of performance can enable a no-frame-pointer optimization and see if it helps. But it's a bad default for libc, and for the kernel.
The performance cost in your case may be much smaller than 2 per cent.
Don't completely trust the benchmarks on this; they are a bit synthetic and real-world applications tend to produce very different results.
Plus, profiling is important. I was able to speed up various segments of my code by up to 20 per cent by profiling them carefully.
And, at the end of the day, if your application is so sensitive about any loss of performance, you can simply profile your code in your lab using frame pointers, then omit them in the version released to your customers.
> And, at the end of the day, if your application is so sensitive about any loss of performance, you can simply profile your code in your lab using frame pointers, then omit them in the version released to your customers.
That is what should be done but TFA is about distros shipping code with frame pointers to end uses because some developers are too lazy to recompile libc when profiling. Somehow shipping different copies of libc, one indended for end users on low-powered devices and one indended for developers is not even considered.
If you can't introspect the release version of your software, you have no way of determining what the issue is. You're doing psuedo-science and guesswork to try and replicate the issue on a development version of the software. And if you put in a few new logging statements into the release version, there's a pretty good chance that simply restarting the software will cause the symptom to go away.
The measured overhead is slightly less than 1%. There have been some rare historical cases where frame pointers have caused performance to blow up but those are fixed.
As much as the return of frame pointers is a good thing, it's largely unnecessary -- it arrives at a point where multiple eBPF-based profilers are available that do fine using .eh_frame and also manually unwinding high level language runtime stacks: Both Parca from PolarSignals as well the artist formerly known as Prodfiler (now Elastic Universal Profiling) do fine.
So this is a solution for a problem, and it arrives just at the moment that people have solved the problem more generically ;)
(Prodfiler coauthor here, we had solved all of this by the time we launched in Summer 2021)
Parca is open-source, Prodfiler's eBPF code is GPL, and the rest of Prodfiler is currently going through OTel donation, so my point is: There's now multiple FOSS implementations of a more generic and powerful technique.
So not a perf issue there, but they don't think the workflow is suitable for whole-system profiling. Perf issues were in the context of `perf` using DWARF:
Also I've heard that the whole .eh_frame unwinding is more fragile than a simple frame pointer. I've seen enough broken stack traces myself, but honestly I never tried if -fno-omit-frame-pointer would have helped.
Yes and no. A simple frame pointer needs to be present in all libraries, and depending on build settings, this might not be the case. .eh_frame tends to be emitted almost everywhere...
So it's both similarly fragile, but one is almost never disabled.
The broader point is: For HLL runtimes you need to be able to switch between native and interpreted unwinds anyhow, so you'll always do some amount of lifting in eBPF land.
And yes, having frame pointers removes a lot of complexity, so it's net a very good thing. It's just that the situation wasnt nearly as dire as described, because people that care about profiling had built solutions.
Forget eBPF even -- why do the job of userspace in the kernel? Instead of unwinding via eBPF, we should ask userspace to unwind itself using a synchronous signal delivered to userspace whenever we've requested a stack sample.
Context switches are incredibly expensive. Given the sampling rate of eBPF profilers all the useful information would get lost in the context switch noise.
Things get even more complicated because context switches can mean CPU migrations, making many of your data useless.
What makes you think doing unwinding in userspace would do any more context switches (by which I think you mean privilege level transitions) than we do today? See my other comment on the subject.
> Things get even more complicated because context switches can mean CPU migrations, making many of your data useless.
No it doesn't. If a user space thread is blocked on doing kernel work, its stack isn't going to change, not even if that thread ends up resuming on a different thread.
I'm under the impression that eh_frame stack traces are much slower than frame pointer stack traces, which makes always-on profiling, such as seen in tcmalloc, impractical.
First of all, I think the .eh_frame unwinding y'all pioneered is great.
But I think you're only thinking about CPU profiling at <= 100 Hz / core. However, Brendan's article is also talking about Off-CPU profiling, and as far as I can tell, all known techniques (scheduler tracing, wall clock sampling) require stack unwinding to occur 1-3 orders of magnitude more often than for CPU profiling.
For those use cases, I don't think .eh_frame unwinding will be good enough, at least not for continuous profiling. E.g. see [1][2] for an example of how frame pointer unwinding allowed the Go runtime to lower execution tracing overhead from 10-20% to 1-2%, even so it was already using a relatively fast lookup table approach.
If you're sufficiently in control of your deployment details to ensure that BPF is available at all. CAP_SYS_PTRACE is available ~everywhere for everyone.
so what is the downside to using e.g. dwarf-based stack walking (supported by perf) for libc, which was the original stated problem?
in the discussion the issue gets conflated with jit-ted languages, but that has nothing to do with the crusade to enable frame pointer for system libraries.
and if you care that much for dwarf overhead... just cache the unwind information in your system-level profiler? no need to rebuild everything.
Extremely light on the details, and also conflates it with the JIT which makes it harder to understand the point, so I was wondering about the same thing as well.
The way perf does it is slow, as the entire stack is copied into user-space and is then asynchronously unwound.
This is solvable as Brendan calls out, we’ve created an eBPF-based profiler at Polar Signals, that essentially does what you said, it optimized the unwind tables, caches them in bpf maps, and then synchronously unwinds as opposed to copying the whole stack into user-space.
This conveniently sidesteps the whole issue of getting DWARF data in the first place, which is also still a broken disjointed mess on Linux. Hell, Windows solved this many many years ago.
You'd need a pretty special distro to have enabled -fno-asynchronous-unwind-tables by default in its toolchain.
By default on most Linux distros the frame tables are built into all the binaries, and end up in the GNU_EH_FRAME segment, which is always available in any running process. Doesn't sound a broken and disjointed mess to me. Sounds more like a smoothly running solved problem.
It should also be said that you need some sort of DWARF-like information to understand inlining. If I have a function A that inlines B that in turn inlines C, I'd often like to understand that C takes a bunch of time, and with frame pointers only, that information gets lost.
Inlined functions can be symbolized using DWARF line information[0] while unwinding requires DWARF unwind information (CFI), which the x86_64 ABI mandates in every single ELF in the `.eh_frame` section
> The frame pointer register (x29) must always address a valid frame record. Some functions — such as leaf functions or tail calls — may opt not to create an entry in this list. As a result, stack traces are always meaningful, even without debug information.
On Apple platforms, there is often an interpretability problem of another kind: Because of the prevalence of deeply nested blocks / closures, backtraces for Objective C / Swift apps are often spread across numerous threads. I don't know of a good solution for that yet.
I'm not very familiar with Objective C and Swift, so this might not make sense. But JS used to have a similar problem with async/await. The v8 engine solved it by walking the chain of JS promises to recover the "logical stack" developers are interested in [1].
This doesn't detract from the content at all but the register counts are off; SI and DI count as GPRs on i686 bringing it to 6+BP (not 4+BP) meanwhile x86_64 has 14+BP (not 16+BP).
"To emphasize this point: on 64-bit x86 with frame pointers, you have twice as many registers as on 32-bit x86 without frame pointers, and these registers are twice as wide. A 64-bit value (more common than you'd expect even when pointers are 32 bits) takes two registers on 32-bit x86, but only a single register on 64-bit x86."
Brendan is such a treasure to the community (buy his book it’s great).
I wasn’t doing extreme performance stuff when -fomit-frame-pointer became the norm, so maybe it was a big win for enough people to be a sane default, but even that seems dubious: “just works” profiling is how you figure out when you’re in an extreme performance scenario (if you’re an SG14 WG type, you know it and are used to all the defaults being wrong for you).
I’m deeply grateful for all the legends who have worked on libunwind, gperf stuff, perftool, DTrace, eBPF: these are the too-often-unsung heroes of software that is still fast after decades of Moore’s law free-riding.
But they’ve been fighting an uphill battle against a weird alliance of people trying to game compiler benchmarks and the really irresponsible posture that “developer time is more expensive” which is only sometimes true and never true if you care about people on low-spec gear, which is the community of users who that is already the least-resourced part of the global community.
I’m fortunate enough to have a fairly modern desktop, laptop, and phone: for me it’s merely annoying that chat applications and music players and windowing systems offer nothing new except enshittification in terms of features while needing 10-100x the resources they did a decade ago.
But for half of my career and 2/3rds of my time coding, I was on low-spec gear most of the time, and I would have been largely excluded if people didn’t care a lot about old computers back then.
I’m trying to help a couple of aspiring hackers get started right now it’s a real struggle to get their environments set up with limitations like Intel Macs and WSL2 as the Linux option (WSL2 is very cool but it’s not loved up enough by e.g. yarn projects).
If you want new hackers, you need to make things work well on older computers.
At the very least Chrome needs to link to the system libGL.so and friends for gpu acceleration, libva.so for video acceleration, and so on. And these are linked against glibc of course.
I remember when the omission of stack frame pointers started spreading at the beginning of the 2000s. I was in college at the time, studying computer sciences in a very poor third-world country. Our computers were old and far from powerful. So, for most course projects, we would eschew interprets and use compilers. Mind you, what my college lacked in money it compensated by having interesting course work. We studied and implemented low level data-structures, compilers, assembly-code numerical routines and even a device driver for Minix.
During my first two years in college, if one of our programs did something funny, I would attach gdb and see what was happening at assembly level. I got used to "walking the stack" manually, though the debugger often helped a lot. Happy times, until all of the sudden, "-fomit-frame-pointer" was all the rage, and stack traces stopped making sense. Just like that, debugging that segfault or illegal instruction became exponentially harder. A short time later, I started using Python for almost everything to avoid broken debugging sessions. So, I lost an order of magnitude or two with "-fomit-frame-pointer". But learning Python served me well for other adventures.
You unfortunately do not control everything on your machine and how it's compiled. I think source vased distros like gentoo are the only exception but have their own warts.
I'm glad he mentioned Fedora because it's been a tiresome battle to keep frame pointers enabled in the whole distribution (eg https://pagure.io/fesco/issue/3084).
There's a persistent myth that frame pointers have a huge overhead, because there was a single Python case that had a +10% slow down (now fixed). The actual measured overhead is under 1%, which is far outweighed by the benefits we've been able to make in certain applications.
Initially we just turned off frame pointers for the Python 3.9 interpreter in Fedora. They are back on in Python 3.12 where it seems the upstream bug has been fixed, although I can't find the actual fix right now.
I believe it's a misrepresentation to say that "actual measured overhead is under 1%". I don't think such a claim can be universally applied because this depends on the very workload you're measuring the overhead with.
I didn't preserve the data involved but in a variety of workloads including netperf, page allocator microbenchmark, pgbench and sqlite, enabling framepointer introduced overhead of around the 5-10% mark.
Significance in their results IMO is in the fact that they measured the impact by using PostgreSQL and SQLite. If anything, DBMS are one of the best ways to really stress out the system.
From the docs: "pgbench is a simple program for running benchmark tests on PostgreSQL. It runs the same sequence of SQL commands over and over"
While it might call itself a benchmark, it behaves very microbenchmark-y.
The other numbers I and others have shared have been from actual production workloads. Not a simple program that tests same sequence of commands over and over.
While pgbench might be "simple" program, as in a test-runner, workloads that are run by it are far from it. It runs TPC-B by default but can also run your own arbitrary script that defines whatever the workload is? It also allows to run queries concurrently so I fail to understand the reasoning of it "being simple" or "microbenchmarkey". It's far from the truth I think.
If I call the same "get statistics" command over and over in a loop (with zero queries), or 100% the same invalid query (to test the error path performance), I believe we'd call that a micro-benchmark, despite involving a full database. It's a completely unrealistic artificial workload to test a particular type of operation.
The pgbench docs make it sound microbenchmark-y by describing making the same call over and over. If people find that this simulates actual production workloads, then yes, it can be considered a macro-benchmark.
In August 1990, the TPC approved its second benchmark, TPC-B. In contrast to TPC-A, TPC-B is not an OLTP benchmark. Rather, TPC-B can be looked at as a database stress test, characterized by:
Significant disk input/output
Moderate system and application execution time
Transaction integrity
TPC-B measures throughput in terms of how many transactions per second a system can perform. Because there are substantial differences between the two benchmarks (OLTP vs. database stress test), TPC-B results cannot be compared to TPC-A.
...
Transactions are submitted by programs all executing concurrently.
I don't think I have. I was only responding to the factually incorrect statement of yours that pgbench is a microbenchmark.
> which was about whether databases could even have micro-benchmarks.
No, this was an argument of yours that you pulled out out of nowhere. The topic very specifically was about the pgbench and not whether or not databases can have micro-benchmarks. Obvious answer is, yes, they can as any other software out there.
I think that you kinda tried to imply that pgbench was one of such micro-benchmarks in disguise and which is why I c/p the description which proves that it is not.
> You also missed the word "Obsolete"
I did not since that was not the topic being discussed at all. And in a technical sense, it doesn't matter at all. pgbench still runs so it is very much "not obsolete".
I didn't pull this argument out of nowhere, please read the direct comment I was replying to. Your position is also completely untenable: this benchmark was obsoleted by its creators 29 years ago, who very clearly say it is obsolete, and you're arguing that it isn't because it "still runs."
I'm guessing that this discussion would be more productive if you would please say who you are and the company you work for. I'm Brendan Gregg, I work for Intel, and I'm well known in the performance space. Who are you?
> I'm guessing that this discussion would be more productive if you would please say who you are and the company you work for. I'm Brendan Gregg, I work for Intel, and I'm well known in the performance space. Who are you?
Wow, just wow. How ridiculous this is?
> Your position is also completely untenable: this benchmark was obsoleted by its creators 29 years ago, who very clearly say it is obsolete, and you're arguing that it isn't because it "still runs."
My only position in the whole thread was "1% of overhead cannot be universally claimed" and I still stand by it 100%. pgbench experiment from Linux kernel folks was just one of the counter-examples that can be found in the wild that goes against your claim. And which you first disputed by saying that it is a micro-benchmark (meaning that you have no idea what it is) and now you're disputing it by saying it's obsolete (yes, but still very relevant in database development, used in the Linux kernel and not the slightest technical reasoning after all).
Personally, I couldn't care less about this but if names is what you're after, you're not disagreeing with me but with the methodology and results presented by Mel Gorman, Linux kernel developer.
You are backing away from your other positions, for example:
> I fail to understand the reasoning of it "being simple" or "microbenchmarkey". It's far from the truth I think.
Do you now agree that TPC-B is too simple and microbenchmarky? And if not, please tell me (as I'm working on the problem of industry benchmarking in general) what would it take to convince someone like you to stop elevating obsoleted benchmarks like TPC-B? Is there anything?
Yes, indeed, it is very ridiculous to pull out arguments such as "who are you". I mean, wtf?
Your replies demonstrate lack of technical depth in certain areas and which makes me personally doubt in your experiment results. And you know what, that is totally fine.
But your attitude? A total disbelief.
> Do you now agree that TPC-B is too simple and microbenchmarky?
No, why would I, you did not present any evidence that would support that claim of yours?
And contrary to you, and to your own embarrassment, I do not use personal attacks when I go out of technical depth.
> what would it take to convince someone like you to stop elevating obsoleted benchmarks like TPC-B? Is there anything?
You don't have to convince me anything. Remember that this is just a random internet page where people are exchanging their opinions.
A major postgres contributor (Andres Freund) disagreed with you about pgbench but, yes, feel free to dismiss them just because you found some words on a web page.
I am just a minor PostgreSQL contributor and consultant of no import, but do you seriously think you are superior to PostgreSQL core devs and know more than them about PostgreSQL just because you are a Linux kernel dev? I really do not like your attitude here and your appeal to your own authority when you are not even an authority here.
Pgbench is used heavily both in the development of PostgreSQL and by people who tune PostgreSQL. So it is not obsolete. Maybe it is a bad benchmark and that the PostgreSQL community should stop using it but then you need a stronger argument than just some claim about obsoleteness from 1995. If a PostgreSQL core developer says in 2024 that it is still relevant I think that weighs a bit higher than a claim from 1995.
Those are numbers from 7 years ago, so they're beginning to get a bit stale as people start to put more weight behind having frame pointers and make upstream contributions to their compilers to improve their output. People put it at <1% from much more recent testing by the very R.W.M. Jones you're replying to [0] and separate testing by others like Brendan Gregg [1b], whose post this is commenting on (and included [1b] in the Appendix as well), with similar accounts by others in the last couple years. Oh, and if you use flamegraph, you might want to check the repo for a familiar name.
Some programs, like Python, have reported worse, 2-7% [2], but there is traction on tackling that [1a] (see both rwmj's and brendangregg's replies to sibling comments, they've both done a lot of upstreamed work wrt. frame pointers, performance, and profiling).
As has been frequently pointed out, the benefits from improved profiling cannot be understated, even a 10% cost to having frame pointers can be well worth it when you leverage that information to target the actual bottlenecks that are eating up your cycles. Plus, you can always disable it in specific hotspots later when needed, which is much easier than the reverse.
Something, something, premature optimisation -- though in seriousness, this information benefits actual optimisation, exactly because we don't have the information and understanding that would allow truly universal claims, precisely because things like this haven't been available, and so haven't been widely used. We know frame pointers, from additional register pressure and extended function prologue/epilogue, can be a detriment in certain hotspots; that's why we have granular control. But without them, we often don't know which hotspots are actually affected, so I'm sure even the databases would benefit... though the "my database is the fastest database" problem has always been the result of endless micro-benchmarking, rather than actual end-to-end program performance and latency, so even a claimed "10%" drop there probably doesn't impact actual real-world usage, but that's a reason why some of the most interesting profiling work lately has been from ideas like causal profilers and continuous profilers, which answer exactly that.
While improved profiling is useful, achieving it by wasting a register is annoying, because it is just a very dumb solution.
The choice made by Intel when they have designed 8086 to use 2 separate registers for the stack pointer and for the frame pointer was a big mistake.
It is very easy to use a single register as both the stack pointer and the frame pointer, as it is standard for instance in IBM POWER.
Unfortunately in the Intel/AMD CPUs using a single register is difficult, because the simplest implementation is unreliable since interrupts may occur between 2 instructions that must form an atomic sequence (and they may clobber the stack before new space is allocated after writing the old frame pointer value in the stack).
It would have been very easy to correct this in new CPUs by detecting that instruction sequence and blocking the interrupts between them.
Intel had already done this once early in the history of the x86 CPUs, when they have discovered a mistake in the design of the ISA, that interrupts could occur between updating the stack segment and the stack pointer. Then they had corrected this by detecting such an instruction sequence and blocking the interrupts at the boundary between those instructions.
The same could have been done now, to enable the use of the stack pointer as also the frame pointer. (This would be done by always saving the stack pointer in the top of the stack whenever stack space is allocated, so that the stack pointer always points to the previous frame pointer, i.e. to the start of the linked list containing all stack frames.)
> As has been frequently pointed out, the benefits from improved profiling cannot be understated, even a 10% cost to having frame pointers can be well worth it when you leverage that information to target the actual bottlenecks that are eating up your cycles.
Few can leverage that information because the open source software you are talking about lacks telemetry in the self hosted case.
The profiling issue really comes down to the cultural opposition in these communities to collecting telemetry and opening it for anyone to see and use. The average user struggles to ally with a trustworthy actor who will share the information like profiling freely and anonymize it at a per-user level, the level that is actually useful. Such things exist, like the Linux hardware site, but only because they have not attracted the attention of agitators.
Basically users are okay with profiling, so long as it is quietly done by Amazon or Microsoft or Google, and not by the guy actually writing the code and giving it out for everyone to use for free. It’s one of the most moronic cultural trends, and blame can be put squarely on product growth grifters who equivocate telemetry with privacy violations; open source maintainers, who have enough responsibilities as is, besides educating their users; and Apple, who have made their essentially vaporous claims about privacy a central part of their brand.
Of course people know the answer to your question. Why doesn’t Google publish every profile of every piece of open source software? What exactly is sensitive about their workloads? Meta publishes a whole library about every single one of its customers, for anyone to freely read. I don’t buy into the holiness of the backend developer’s “cleverness” or whatever is deemed sensitive, and it’s so hypocritical.
> Basically users are okay with profiling, so long as it is quietly done by Amazon or Microsoft or Google, and not by the guy actually writing the code and giving it out for everyone to use for free.
No; the groups are approximately "cares whether software respects the user, including privacy", or "doesn't know or doesn't care". I seriously doubt that any meaningful number of people are okay with companies invading their privacy but not smaller projects.
I think the kind of profiling information you're imagining is a little different from what I am.
Continuous profiling of your system that gets relayed to someone else by telemetry is very different from continuous profiling of your own system, handled only by yourself (or, generalising, your community/group/company). You seem to be imagining we're operating more in the former, whereas I am imagining more in the latter.
When it's our own system, better instrumented for our own uses, and we're the only ones getting the information, then there's nothing to worry about, and we can get much more meaningful and informative profiling done when more information about the system is available. I don't even need telemetry. When it's "someone else's" system, in other words, when we have no say in telemetry (or have to exercise a right to opt-out, rather than a more self-executing contract around opt-in), then we start to have exactly the kinds of issues you're envisaging.
When it's not completely out of our hands, then we need to recognise different users, different demands, different contexts. Catering to the user matters, and when it comes to sensitive information, well, people have different priorities and threat models.
If I'm opening a calendar on my phone, I don't expect it to be heavily instrumented and relaying all of that, I just want to see my calendar. When I open a calendar on my phone, and it is unreasonably slow, then I might want to submit relevant telemetry back in some capacity. Meanwhile, if I'm running the calendar server, I'm absolutely wanting to have all my instrumentation available and recording every morsel I reasonably can about that server, otherwise improving it or fixing it becomes much harder.
From the other side, if I'm running the server, I may want telemetry from users, but if it's not essential, then I can "make do" with only the occasional opt-in telemetry. I also have other means of profiling real usage, not just scooping it all up from unknowing users (or begrudging users). Those often have some other "cost", but in turn, they don't have the "cost" of demanding it from users. For people to freely choose requires acknowledging the asymmetries present, and that means we can't just take the path of least resistance, as we may have to pay for it later.
In short, it's a consent issue. Many violate that, knowingly, because they care not for the consequences. Many others don't even seem to think about it, and just go ahead regardless. And it's so much easier behind closed doors. Open source in comparison, even if not everything is public, must contend with the fact that the actions and consequences are (PRs, telemetry traffic, etc), so it inhabits a space in which violating consent is much more easily held accountable (though no guarantee).
Of course, this does not mean it's always done properly in open source. It's often an uphill battle to get telemetry that's off-by-default, where users explicitly consent via opt-in, as people see how that could easily be undermined, or later invalidated. Many opt-in mechanisms (e.g. a toggle in the settings menu) often do not have expiration built in, so fail to check at a later point that someone still consents. Not to say that's the way you must do it, just giving an example of a way that people seem to be more in favour of, as with the generally favourable response to such features making their way into "permissions" on mobile.
We can see how the suspicion creeps in, informed by experience... but that's also known by another word: vigilance.
So, users are not "okay" with it. There's a power imbalance where these companies are afforded the impunity because many are left to conclude they have no choice but to let them get away with it. That hasn't formed in a vacuum, and it's not so simple that we just pull back the curtain and...
Telemetry is exceedingly useful, and it's basically a guaranteed boon when you operate your own systems. But telemetry isn't essential, and it's not the heart of the matter I was addressing. Again, the crux of this is consent, as an imbalance of power easily distorts the nature of consent.
Suppose Chrome added new telemetry, for example, like it did when WebRTC was added in Chrome 28, so we really can just track this against something we're all familiar (enough with). When a user clicks "Update", or it auto-updated and "seamlessly" switched version in the background / between launches, well, did the user consent to the newly added telemetry?
Perhaps most importantly: did they even know? After all, the headline feature of Chrome 28 was Blink, not some feature that had only really been shown off in a few demos, and was still a little while away from mass adoption. No reporting on Chrome 28 that I could find from the time even mentions WebRTC, despite entire separate articles going out just based on seeing WebRTC demos! Notifications got more
So, capabilities to alter software like this are, knowingly or unknowingly, undermine the nature of consent that many find implicit in downloading a browser, since what you download and what you end up using may be two very different things.
Now, let's consider a second imbalance. Did you even download Chrome? Most Android devices often have it preinstalled, or some similar "open-core" browser (often a Chromium-derivative). Some are even protected from being uninstalled, so you can't opt out that way, and Apple only just had to open up iOS to non-Safari backed browsers.
So the notion of consent via the choice to install is easily undermined.
Lastly, because we really could go on all day with examples, what about when you do use it? Didn't you consent then?
Well, they may try to onboard you, and have you pretend to read some EULA, or just have it linked and give up the charade. If you don't tick the box for "I read and agree to this EULA", you don't progress. Of course, this is hardly a robust system. Enforceability aside, the moment you had it over to someone else to look at a webpage, did they consent to the same EULA you did?
... Basically, all the "default" ways to consider consent are nebulous, potentially non-binding, and may be self-defeating. After all, you generally don't consent to every single line of code, every single feature, and so on, you are usually assumed to consent to the entire thing or nothing. Granularity with permissions has improved that somewhat, but there is usually still a bulk core you must accept before everything else; otherwise the software is usually kept in a non-functional state.
I'm not focused too specifically on Chrome here, but rather the broad patterns of how user consent typically assumed in software don't quite pan out as is often claimed. Was that telemetry the specific reason why libwebrtc was adopted by others? I'm not privy to the conversations that occurred with these decisions, but I imagine it's more one factor among many (not to mention, Pion is in/for Go, which was only 4 years old then, and the pion git repo only goes back to 2018). People were excited out of the gate, and libwebrtc being available (and C++) would have kept them in-step (all had support within 2013). But, again, really this is nothing to do with the actual topic at hand, so let's not get distracted.
The user has no opportunity to meaningfully consent to this. Ask most people about these things, and they wouldn't even recognise the features by now (as WebRTC or whatever is ubiquitous), let alone any mechanisms they may have to control how it engages with them.
Yet, the onus is put on the user. Why do we not ask about anything/anyone else in the equation, or consider what influences the user?
A recent example I think illustrates the imbalance and how it affects and warps consent is the recent snafu wit...
> Rethink how we engage, respect them, respect their consent.
One way to characterize this is leadership. Most open source software authors are terrible leaders!
You’re way too polite. Brother, who is making a mistake and deserves blame? Users? Open source non corporate software maintainers? Google employees? Someone does. It can’t be “we.” I don’t make any of these mistakes, leave me out of it! I tell every non corporate open source maintainer to add basic anonymized telemetry, PR a specific opt-out solution with my preferred Plausible, and argue relentlessly with users to probe the vapors they base their telemetry fears on. We’re both trying to engage on the issue, but the average HN reader is downvoting me. Because “vibes.” Vibes are dumb! Just don’t be afraid to say it.
"Agitators". We don't trust telemetry precisely because of comments like that. World is full of people like you who apparently see absolutely nothing wrong with exfiltrating identifying information from other people's computers. We have to actively resist such attempts, they are constant, never ending and it only seems to get worse over time but you dismiss it all as "cultural opposition" to telemetry.
For the record I'm NOT OK with being profiled, measured or otherwise studied in any way without my explicit consent. That even extends to the unethical human experiments that corporations run on people and which they euphemistically call A/B tests. I don't care if it's Google or a hobbyist developer, I will block it if I can and I will not lose a second of sleep over it.
> World is full of people like you who apparently see absolutely nothing wrong with exfiltrating identifying information from other people's computers.
True. But such people are like cockroaches. They know what they are doing will be unpopular with their targets, so they keep it hidden. This is easy enough to do in closed designs, car manufacturers selling your driving habits to insurance companies and health monitoring app selling menstrual cycle data to retailers selling to women.
Compare that do say Debian and RedHat. They too collect performance data. But the code is open source, Debian has repeatable builds so you can be 100% sure that is the code in use, and every so often someone takes a look it. Guess what, the data they send back is so unidentifiable it satisfies even the most paranoid of their 1000's of members.
All it takes is a little bit of sunlight to keep the cockroaches at bay, and then we can safely let the devs collect the data they need to improve code. And everyone benefits.
I fully support the schemes for telemetry that already happens. It’s just obnoxious that there’s no apparent reason behind the support of one kind of anonymous telemetry versus another. For every 1 person who digests the package repo’s explanation of the telemetry strategy, there are 19 who feel okay with GitHub having all the telemetry and none of the open source repos they host because of vibes.
I'd prefer discussing technical merits of given approach rather than who is who and who did what since that leads to appeal to authority fallacy.
You're correct, results might be stale, although I wouldn't hold my breath for it since there has been no fundamental change in a way how frame pointers are handled as far as my understanding goes. Perhaps smaller improvements in compiler technology but CPUs did not undergo any significant change w.r.t. that context.
That said, nowhere in this thread have we seen a dispute of those Linux kernel results other than categorically rejecting them as being "microbenchmarks", which they are not.
> though the "my database is the fastest database" problem has always been the result of endless micro-benchmarking, rather than actual end-to-end program performance and latency
Quite the opposite. All database benchmarks are end-to-end program performance and latency analysis. "Cheating" in database benchmarks is done elsewhere.
You probably already know, but with OCaml 5 the only way to get flamegraphs working is to either:
* use framepointers [1]
* use LBR (but LBR has a limited depth, and may not work on on all CPUs, I'm assuming due to bugs in perf)
* implement some deep changes in how perf works to handle the 2 stacks in OCaml (I don't even know if this would be possible), or write/adapt some eBPF code to do it
If you need more evidence to keep it enabled in future releases, you can use OCaml 5 as an example (unfortunately there aren't many OCaml applications, so that may not carry too much weight on its own).
[1]: I haven't actually realised that Fedora39 has already enabled FP by default, nice! (I still do most of my day-to-day profiling on an ~CentOS 7 system with 'perf record --call-graph dwarf -F 47 -a', I was aware that there was a discussion to enable FP by default, but haven't noticed it has actually been done already)
Also, if you want LBR on Zen3 or earlier, you're SOL. I noticed that one the hard way.
Stuff like this is making me lean more towards getting an Intel for my next computer, at least on desktop where their worse power efficiency is less of an issue. But then they keep gating AVX-512 due to their E-cores not supporting it... You really can't win these days.
It would be, yes. x86 had very few registers, so anything you could do to free them up was vital. Arm 32bit has 32 general purpose registers I think, and RISC V certainly does. In fact there's no difference between 32 and 64 bit in that respect. If anything, 64-bit frame pointers make it marginally worse.
Sadly, no. 32-bit ARM only has 16 GPR’s (two of which are zero and link), mostly because of the stupid predication bits in the instruction encoding.
That said, I don’t know how valuable getting rid of FP on ARM is - I once benchmarked ffmpeg on 32-bit x86 before and after enabling FP and PIC (basically removing 2 GPRs) and the difference was huge (>10%) but that’s an extreme example.
Arm32 doesn’t have a zero-value register. Its non-general-purpose registers are PC, LR, SP, FP – tho the link register can be used for temporary values.
I'm still baffled by this attitude. That "under 1%" overhead is why computers are measurably slower than 30 years ago to use. All those "under 1%" overhead add up
Are his books (the one about Systems Performance and eBPF) relevant for normal software engineers who want to improve performance in normal services? I don’t work for faang, and our usual performance issues are solved by adding indexes here and there, caching, and simple code analysis. Tools like Datadog help a lot already.
Diving into flame graphs being worthwhile for optimization, assumes that your workload is CPU-bound. Most business software does not have such workloads, and rather (as you yourself have noted) spend most of their time waiting for I/O (database, network, filesystem, etc).
And so, (as you again have noted), your best bet is to just use plain old logging and tracing (like what datadog provides) to find out where the waiting is happening.
Profiling is a pretty basic technique that is applicable to all software engineering. I'm not sure what a "normal" service is here, but I think we all have an obligation to understand what's happening in the systems we own.
Some people may believe that 100ms latency is acceptable for a CLI tool, but what if it could be 3ms? On some aesthetic level, it also feels good to be able to eliminate excess. Finally, you should learn it because you won't necessarily have that job forever.
I think so and I vaguely seem to recall -fno-schedule-insns2 being the only thing that fixes it. To get the full power of frame pointers and hackable binary, what I use is:
The only flag that's potentially problematic is -fno-optimize-sibling-calls since it breaks the optimal approach to writing interpreters and slows down code that's written in a more mathematical style.
"I could say that times have changed and now the original 2004 reasons for omitting frame pointers are no longer valid in 2024."
The original 2004 reason for omitting frame pointers is still valid in 2024: it's still a big performance win on the register-starved 32-bit x86 architecture. What has changed is that the 32-bit x86 architecture is much less relevant nowadays (other than legacy software, for most people it's only used for a small instant while starting up the firmware), and other common 32-bit architectures (like embedded 32-bit ARM) are not as register-starved as the 32-bit x86.
I thought we'd been using /Oy (Frame-Pointer Omission) for years on Windows and that there was a pdata section on x64 that was used for stack-walking however to my great surprise I just read on MSDN that "In x64 compilers, /Oy and /Oy- are not available."
Does this mean Microsoft decided they weren't going to support breaking profilers and debuggers OR is there some magic in the pdata section that makes it work even if you omit the frame-pointer?
“Recovering a broken stack on x64 machines on Windows is trickier because the x64 uses unwind codes for stack walking rather than a frame pointer chain.”
Microsoft has had excellent universal unwinding support for decades now. I'm disappointed to see someone as prominent as this article's author present as infeasible what Microsoft has had working for so long.
Overall, I am for frame pointers, but after some years working in this space, I thought I would share some thoughts:
* Many frame pointer unwinders don't account for a problem they have that DWARF unwind info doesn't have: the fact that the frame set-up is not atomic, it's done in two instructions, `push $rbp` and `mov $rsp $rbp`, and if when a snapshot is taken we are in the `push`, we'll miss the parent frame. I think this might be able to be fired by inspecting the code, but I think this might only be as good as a heuristic as there could be other `push %rbp` unrelated to the stack frame. I would love to hear if there's a better approach!
* I developed the solution Brendan mentions which allows faster, in-kernel unwinding without frame pointers using BPF [0]. This doesn't use DWARF CFI (the unwind info) as-is but converts it into a random-access format that we can use in BPF. He mentions not supporting JVM languages, and while it's true that right now it only supports JIT sections that have frame pointers, I planned to implement a full JVM interpreter unwinder. I have left Polar Signals since and shifted priorities but it's feasible to get a JVM unwinder to work in lockstep with the native unwinder.
* In an ideal world, enabling frame pointers should be done on a case-by-case. Benchmarking is key, and the tradeoffs that you make might change a lot depending on the industry you are in, and what your software is doing. In the past I have seen large projects enabling/disabling frame pointers not doing an in-depth assessment of losses/gains of performance, observability, and how they connect to business metrics. The Fedora folks have done a superb and rigorous job here.
* Related to the previous point, having a build system that enables you to change this system-wide, including libraries your software depends on can be awesome to not only test these changes but also put them in production.
* Lastly, I am quite excited about SFrame that Indu is working on. It's going to solve a lot of the problems we are facing right now while letting users decide whether they use frame pointers. I can't wait for it, but I am afraid it might take several years until all the infrastructure is in place and everybody upgrades to it.
On the third point, you have to do frame pointers across the whole Linux distro in order to be able to get good flamegraphs. You have to do whole system analysis to really understand what's going on. The way that current binary Linux distros (like Fedora and Debian) works makes any alternative impossible.
Great comments, thanks for sharing. The non-atomic frame setup is indeed problematic for CPU profilers, but it's not an issue for allocation profiling, Off-CPU profiling or other types off non-interrupt driven profiling. But as you mentioned, there might be ways to solve that problem.
It could be one instruction: ENTER N,0 (where N is the amount of stack space to reserve for locals)---this is the same as:
PUSH EBP
MOV ESP,ESP
SUB SP,N
(I don't recall if ENTER is x86-64 or not). But even with this, the frame setup isn't atomic with respect to CALL, and if the snapshot is taken after the CALL but before the ENTER, we still don't get the fame setup.
As for the reason why ENTER isn't used, it was deemed too slow. LEAVE (MOV SP,BP; POP BP) is used as it's just as fast as, if not faster, than the sequence it replaces. If ENTER were just the PUSH/MOV/SUB sequence, it probably would be used, but it's that other operand (which is 0 above in my example) that kills it performance wise (it's for nested functions to gain access to upper stack frames and is every expensive to use).
269 comments
[ 3.0 ms ] story [ 296 ms ] threadEven if $BIG_COMPANY makes a decision to compile everything with frame pointers, the rest of the community is not. So we'll be stuck fighting an unwinnable argument with a much larger community. Turns out that it was a ~20 year argument.
I ended up writing some patches to make libunwind work for gperftools and maintained libunwind for some number of years as a consequence of that work.
Having moved on to other areas of computing, I'm now a passive observer. But it's fascinating to read history from the other perspective.
In what way would you be stuck? What functional problems does adding frame pointers introduce?
Aee also this benchmark from Phoronix [0]:
I'm not saying these benchmarks or the workloads I've seen are representative of the "real world", but people keep repeating that frame pointers are basically free, which is just not the case.[0] https://www.phoronix.com/review/fedora-frame-pointer
The actual «wastage» is in having to generate a prologue and an epilogue for each function – 2x instructions to preserve the old frame pointer and set a new one up, and 2x instruction at the point of return – to restore the previous frame pointer.
Generally, it is not a big deal with an exception of a pathological case of a very large number of very small functions calling each other frequently where the extra 4x instructions per each such a function will be filling up the L1 instruction cache «unnessarily».
There is also a significant number of projects that go to great lengths to force-override CFLAGS/CXXFLAGS (usually with «-O2 -g» or even with «-O») or make it extraordinary difficult to change the project's default CFLAGS, for no apparent reasons which eliminates a number of advanced optimisations in builds with default build settings.
All of the companies involved believed that they were special and decided to build their own (poorly managed) distribution called "third party code" and having to deal with it was not my best experience working at these companies.
What existed was a x86 to x86 cross compilation environment and the libraries involved were manually imported by developers who needed that particular library.
My argument was about the cost of ensuring that those libraries were compiled with frame pointers when much of the open source community was defaulting to omit-fp.
In 2024, you'd probably create a "build container" that all developers are required to use to build binaries or pay a linux distributor to build that container.
But cross compilation was the preferred approach back then. So all binaries had a rpath (run time search path to look for shared library) that ignored the distributor supplied libraries.
Having come from a open source background, I found this system hard to digest. But there was a lot of social pressure to work as a bee in a system that thousands of other very competent engineers are using (quite successfully).
I remember briefly talking to a chrome OS related group who were using the "build your own custom distro" approach, before deciding to move to another faang.
What does this mean?
Since Google would rather have the best brains in the industry build the next search indexing algorithm or the browser, they didn't have the time to invest human capital into building a better open source friendly dev environment.
A natural alternative is to contract out the work. Linux distributors were good candidates for such contract work.
But the vibe back then was Google could build better alternatives to some of these libraries and therefore bridging the gap between dev experience as an open source dev vs in house software engineer wasn't important.
You could see the same argument play out in git vs monorepo etc, where people take up strong positions on a particular narrow tech topic, whereas the larger issue gets ignored as a result of these belief systems.
https://bugzilla.redhat.com/show_bug.cgi?id=2267598 https://github.com/tukaani-project/xz/commit/82ecc538193b380...
[1]: https://news.ycombinator.com/item?id=39865810
[2]: https://news.ycombinator.com/item?id=39867301
Yep. That's my proposal.
> which doesn't account for the reality that we need to do frame unwinding thousands of times a second with the least possible overhead
Yes, it does. The kernel has to return to userspace anyway at some point, and pushing a signal frame during that return is cheap. The cost of signal delivery is the entry into the kernel, and after a perf counter overflow, you've already paid that cost. Why would the actual unwinding be any faster in the kernel than in userspace?
Also, so what if a thread enters the kernel and samples the stack multiple times before returning to userspace? While in the kernel, the userspace stack cannot change --- therefore, it's sufficient to delay userspace stack collection until the kernel returns to userspace anyway.
You might ask "Don't we have to restore the signal mask after handling the profiling signal?"
Not if you don't define the signal to change the signal mask. sigreturn(2) is optional.
Sarcasm is not a technical argument.
> I don't have to
Pick one, jart.
Barriers to progress here are best identified on a community level, wouldn't you say?
But people, please calm down. Filing an issue or posting to the mailing list to make a case isn't sending a SWAT team to people's home. It's a technical issue, one well within the envelope of topics which can be resolved politely and on the merits.
What entitles you to this opportunity?
> The overhead to walk DWARF is also too high, as it was designed for non-realtime use.
Source: I wrote multiple profilers.
[1]: https://news.ycombinator.com/item?id=39732010
Granted, this is using optimized tooling which uses a bunch of tricks to side-step the problem of DWARF being slow, I only profile native code (and some VMs which do ahead-of-time codegen) and I've never worked with JVM, but in principle I don't see why it wouldn't be practical on JVM too, although it certainly would be harder and might require better tooling (which might not exist currently). If you have the luxury of enabling frame pointers then that certainly would be easier and simpler.
(Somewhat related, but I really wish we would standardize on something better than DWARF for unwinding tables and basic debug info. Having done a lot of work with DWARF and its complexity I wouldn't wish it upon my worst enemy.)
However, paper also mentions
and> Javier Honduvilla Coto (Polar Signals) did some interesting work using an eBPF walker to reduce the overhead, but...Java.
I'm waiting for this to happen: https://github.com/open-telemetry/community/issues/1918
Samply unwinds significantly faster than `perf` because it caches unwind information.
That being said, this approach still has some limitations, such as that very deep stacks won't be unwound, as the size of the process stack the kernel sends is quite limited.
- [0]: https://github.com/mstange/samply
> Profiling has been broken for 20 years and we've only now just fixed it.
It was a shame when they went away. Lots of people, certainly on other systems and probably Linux too, have found the absence of frame pointers painful this whole time and tried to keep them available in as many environments as possible. It’s validating (if also kind of frustrating) to see mainstream Linux bring them back.
Since it can't unwind the user space stacks, the kernel has to copy the entire stack (8192 bytes by default) into the output file (perf.data) and then the perf user space program will unwind it later. It does this for each sample, which is usually hundreds of times per second, per CPU. Though it depends how you configured the collection.
That does have a significant overhead: first, runtime overhead: copying 8k bytes, hundreds of times per second, and writing it to disk, all don't come for free. You spend quite a bit of CPU time doing the memcpy operation which consumes memory bandwidth too. You also frequently need to increase the size of the perf memory buffer to accommodate all this data while it waits for user space to write it to disk. Second, disk space overhead, since the 8k stack bytes per sample are far larger than the stack trace would be. And third, it does require that you install debuginfo packages to get the DWARF info, which is usually a pain to do on production machines, and they consume a lot of disk space on their own.
Many of these overheads aren't too bad in simple cases (lower sample rates, fewer CPUs, or targeting a single task). But with larger machines with hundreds of CPUs, full system collections, and higher frequencies, the overhead can increase exponentially.
I'm not certain I know what you mean by the "slow unwinding path for perf", as there is no faster path for user space when frame pointers are disabled (except Intel LBR as outlined in the blog).
The difference between -g (--call-graph fp) and --call-graph dwarf is large even with perf linked directly against binutils, at least in my experience (and it was much, much worse until I got some patches into binutils to make it faster). This is both on record and report.
There are also weird bugs with --call-graph dwarf that perf upstream isn't doing anything about, around inlining. It's not perfect by any means.
“Observability” primarily refers to ability to view system state beyond of what blackbox monitoring does. Again, the term primarily refers to server side operations and not to software shipped to end users.
As much as spying on users is ugly, it’s not related to debugging server side.
> Be kind. Don't be snarky. Converse curiously; don't cross-examine. Edit out swipes.
> Please don't fulminate. Please don't sneer, including at the rest of the community.
I'm not sure if you know who the author of that blog is, but if there's anyone in the world who cares about (and is knowledgeable about) improving performance, it's him. You can be pretty darn sure he wouldn't do this if he believed it would make software more inefficient.
> In almost all cases you cannot just walk up to a production Linux system and do meaningful performance analysis without serious work
So? That's one very very very specific use case that others should not have to pay for. Not even with a relatively small 1% perf hit.
Finding this out requires... meaningful performance analysis. That's right, this 1% perf hit will make it extremely easy to find the reasons for 5,10,20,50% slowdowns, and enable developers (and you!) to fix them.
By making it easier to profile software, YOU can notice performance issues in the software you're running and deliver a nice patch that will improve performance by more than was lost, making this 1% slowdown a very effective, high-interest investment.
Isn't the whole point of enabling frame pointers everywhere to make it easier to profile software so that it can be made more efficient?
Why do we normally store the return addresses near to the local variables in the first place, again? There are so many downsides.
In exchange, it becomes very difficult for the stack smashing to corrupt the return address.
[1] https://en.wikipedia.org/wiki/Forth_(programming_language)
STOIC, a variant of Forth, includes a file stack when loading words
The dictionary pointer can also be manipulated in some dialects. That can be used directly as the stack variant of the arena allocator idea. It is particularly useful for text concatenation.
[1] https://forth-standard.org/standard/search [2] https://forth-standard.org/standard/core/MARKER
The advantage of storing them elsewhere is not quite clear (unless you have hardware support for things like shadow stacks).
You'd have to argue that the cost of moving things to this other page and managing two pointers (where one is less powerful in the ISA) is meaningfully cheaper than the other equally effective mitigation of stack cookies/protectors which are already able to provide protection only where needed. There is no real security benefit to doing this over what we currently have with stack protectors since an arbitrary read/write will still lead to a CFI bypass.
The classic buffer overflow issue should spring immediately to mind. By having a separate return address stack it's far less vulnerable to corruption through overflowing your data structures. This stops a bunch of attacks which purposely put crafted return addresses into position that will jump the program to malicious code.
It's not a panacea, but generally keeping code pointers away from data structures is a good idea.
It would be really nice to have fine-grained control over frame pointer inclusion: provided fine-grained profiling, we could determine whether we needed the frame pointers for a given function or compilation unit. I wouldn't be surprised if we see that only a handful of operations are dramatically slowed by frame pointer inclusion while the rest don't really care.
No it's not, particularly when it can help you identify hotspots via profiling that can net you improvements of 10% or more.
If you're netflix then "enable frame pointers" is a no-brainer. But if you're a distro who's building code for millions of users, many of whom will likely never need to fire up a profiler, I think the question is at least a little trickier. The overall best tradeoff might end up being still to enable frame pointers, but I can see the other side too.
In the end a 2% of performance of every application it's a big deal. On a single computer may not be that significant, think about all the computers, servers, clusters, that run a Linux distro. And yes, I would ask a Google engineer that if scaled on the I don't know how many servers and computers that Google has a 2% increase in CPU usage is not a big deal: we are probably talking about hundreds of kW more of energy consumption!
We talk a lot of energy efficiency these days, to me wasting a 2% only to make the performance analysis of some software easier (that is that you can analyze directly the package shipped by the distro and you don't have to recompile it) it's something stupid.
Not having them enabled on all dependencies makes them significantly less useful if your application interacts with the system in a meaningful way. "Just recompile" (glibc + xorg + qt + whatever else you use) is a very hefty thing.
Having them enabled by default also enables devs to ask their users for traces. I can reasonably tell someone to install sysprof or something, click a few buttons and send me a trace. I cannot reasonably tell them to compile the project (+ dependencies). And tbh, even devs cba if they have to jump through too many hoops.
> We talk a lot of energy efficiency these days, to me wasting a 2% only to make the performance analysis of some software easier (that is that you can analyze directly the package shipped by the distro and you don't have to recompile it) it's something stupid.
We're trading all kinds of efficiencies for all kinds of niche benefits all the time. This enables analysis of what is shipped in real use across the whole stack with the alternative being much less useful or a lot more (human) work. To me it's worth 2% and I'm shocked (well... not really) it's not for more people.
Code eking out every possible cycle of performance can enable a no-frame-pointer optimization and see if it helps. But it's a bad default for libc, and for the kernel.
Don't completely trust the benchmarks on this; they are a bit synthetic and real-world applications tend to produce very different results.
Plus, profiling is important. I was able to speed up various segments of my code by up to 20 per cent by profiling them carefully.
And, at the end of the day, if your application is so sensitive about any loss of performance, you can simply profile your code in your lab using frame pointers, then omit them in the version released to your customers.
That is what should be done but TFA is about distros shipping code with frame pointers to end uses because some developers are too lazy to recompile libc when profiling. Somehow shipping different copies of libc, one indended for end users on low-powered devices and one indended for developers is not even considered.
So this is a solution for a problem, and it arrives just at the moment that people have solved the problem more generically ;)
(Prodfiler coauthor here, we had solved all of this by the time we launched in Summer 2021)
https://gitlab.com/freedesktop-sdk/freedesktop-sdk/-/issues/...
So not a perf issue there, but they don't think the workflow is suitable for whole-system profiling. Perf issues were in the context of `perf` using DWARF:
https://gitlab.com/freedesktop-sdk/freedesktop-sdk/-/issues/...
So it's both similarly fragile, but one is almost never disabled.
The broader point is: For HLL runtimes you need to be able to switch between native and interpreted unwinds anyhow, so you'll always do some amount of lifting in eBPF land.
And yes, having frame pointers removes a lot of complexity, so it's net a very good thing. It's just that the situation wasnt nearly as dire as described, because people that care about profiling had built solutions.
Things get even more complicated because context switches can mean CPU migrations, making many of your data useless.
> Things get even more complicated because context switches can mean CPU migrations, making many of your data useless.
No it doesn't. If a user space thread is blocked on doing kernel work, its stack isn't going to change, not even if that thread ends up resuming on a different thread.
But I think you're only thinking about CPU profiling at <= 100 Hz / core. However, Brendan's article is also talking about Off-CPU profiling, and as far as I can tell, all known techniques (scheduler tracing, wall clock sampling) require stack unwinding to occur 1-3 orders of magnitude more often than for CPU profiling.
For those use cases, I don't think .eh_frame unwinding will be good enough, at least not for continuous profiling. E.g. see [1][2] for an example of how frame pointer unwinding allowed the Go runtime to lower execution tracing overhead from 10-20% to 1-2%, even so it was already using a relatively fast lookup table approach.
[1] https://go.dev/blog/execution-traces-2024
[2] https://blog.felixge.de/reducing-gos-execution-tracer-overhe...
in the discussion the issue gets conflated with jit-ted languages, but that has nothing to do with the crusade to enable frame pointer for system libraries.
and if you care that much for dwarf overhead... just cache the unwind information in your system-level profiler? no need to rebuild everything.
This is solvable as Brendan calls out, we’ve created an eBPF-based profiler at Polar Signals, that essentially does what you said, it optimized the unwind tables, caches them in bpf maps, and then synchronously unwinds as opposed to copying the whole stack into user-space.
By default on most Linux distros the frame tables are built into all the binaries, and end up in the GNU_EH_FRAME segment, which is always available in any running process. Doesn't sound a broken and disjointed mess to me. Sounds more like a smoothly running solved problem.
- [0] This line information might or might not be present in an executable but luckily there's debuginfod (https://sourceware.org/elfutils/Debuginfod.html)
> The frame pointer register (x29) must always address a valid frame record. Some functions — such as leaf functions or tail calls — may opt not to create an entry in this list. As a result, stack traces are always meaningful, even without debug information.
https://developer.apple.com/documentation/xcode/writing-arm6...
[1] https://v8.dev/blog/fast-async
That is, on i686 you have 7 GPRs without frame pointers, while on x86_64 you have 14 GPRs even with frame pointers.
Copying a comment of mine from an older related discussion (https://news.ycombinator.com/item?id=38632848):
"To emphasize this point: on 64-bit x86 with frame pointers, you have twice as many registers as on 32-bit x86 without frame pointers, and these registers are twice as wide. A 64-bit value (more common than you'd expect even when pointers are 32 bits) takes two registers on 32-bit x86, but only a single register on 64-bit x86."
I wasn’t doing extreme performance stuff when -fomit-frame-pointer became the norm, so maybe it was a big win for enough people to be a sane default, but even that seems dubious: “just works” profiling is how you figure out when you’re in an extreme performance scenario (if you’re an SG14 WG type, you know it and are used to all the defaults being wrong for you).
I’m deeply grateful for all the legends who have worked on libunwind, gperf stuff, perftool, DTrace, eBPF: these are the too-often-unsung heroes of software that is still fast after decades of Moore’s law free-riding.
But they’ve been fighting an uphill battle against a weird alliance of people trying to game compiler benchmarks and the really irresponsible posture that “developer time is more expensive” which is only sometimes true and never true if you care about people on low-spec gear, which is the community of users who that is already the least-resourced part of the global community.
I’m fortunate enough to have a fairly modern desktop, laptop, and phone: for me it’s merely annoying that chat applications and music players and windowing systems offer nothing new except enshittification in terms of features while needing 10-100x the resources they did a decade ago.
But for half of my career and 2/3rds of my time coding, I was on low-spec gear most of the time, and I would have been largely excluded if people didn’t care a lot about old computers back then.
I’m trying to help a couple of aspiring hackers get started right now it’s a real struggle to get their environments set up with limitations like Intel Macs and WSL2 as the Linux option (WSL2 is very cool but it’s not loved up enough by e.g. yarn projects).
If you want new hackers, you need to make things work well on older computers.
Thanks again Brendan et al!
I guess the similar situation with msvcrt’s.
During my first two years in college, if one of our programs did something funny, I would attach gdb and see what was happening at assembly level. I got used to "walking the stack" manually, though the debugger often helped a lot. Happy times, until all of the sudden, "-fomit-frame-pointer" was all the rage, and stack traces stopped making sense. Just like that, debugging that segfault or illegal instruction became exponentially harder. A short time later, I started using Python for almost everything to avoid broken debugging sessions. So, I lost an order of magnitude or two with "-fomit-frame-pointer". But learning Python served me well for other adventures.
There's a persistent myth that frame pointers have a huge overhead, because there was a single Python case that had a +10% slow down (now fixed). The actual measured overhead is under 1%, which is far outweighed by the benefits we've been able to make in certain applications.
Initially we just turned off frame pointers for the Python 3.9 interpreter in Fedora. They are back on in Python 3.12 where it seems the upstream bug has been fixed, although I can't find the actual fix right now.
Fedora tracking bug: https://bugzilla.redhat.com/2158729
Fedora change in Python 3.9 to disable frame pointers: https://src.fedoraproject.org/rpms/python3.9/c/9b71f8369141c...
FWIW your results don't quite match the measurements from Linux kernel folks who claim that the overhead is anywhere between 5-10%. Source: https://lore.kernel.org/lkml/20170602104048.jkkzssljsompjdwy...
Significance in their results IMO is in the fact that they measured the impact by using PostgreSQL and SQLite. If anything, DBMS are one of the best ways to really stress out the system.While it might call itself a benchmark, it behaves very microbenchmark-y.
The other numbers I and others have shared have been from actual production workloads. Not a simple program that tests same sequence of commands over and over.
The pgbench docs make it sound microbenchmark-y by describing making the same call over and over. If people find that this simulates actual production workloads, then yes, it can be considered a macro-benchmark.
From https://www.tpc.org/tpcb/, a TPC-B workload that pgbench runs by default:
You also missed the word "Obsolete" splattered all over the website you sent me, and the text that TPC-B was "Obsolete as of 6/6/95".
> which was about whether databases could even have micro-benchmarks.
No, this was an argument of yours that you pulled out out of nowhere. The topic very specifically was about the pgbench and not whether or not databases can have micro-benchmarks. Obvious answer is, yes, they can as any other software out there.
I think that you kinda tried to imply that pgbench was one of such micro-benchmarks in disguise and which is why I c/p the description which proves that it is not.
> You also missed the word "Obsolete"
I did not since that was not the topic being discussed at all. And in a technical sense, it doesn't matter at all. pgbench still runs so it is very much "not obsolete".
I'm guessing that this discussion would be more productive if you would please say who you are and the company you work for. I'm Brendan Gregg, I work for Intel, and I'm well known in the performance space. Who are you?
Wow, just wow. How ridiculous this is?
> Your position is also completely untenable: this benchmark was obsoleted by its creators 29 years ago, who very clearly say it is obsolete, and you're arguing that it isn't because it "still runs."
My only position in the whole thread was "1% of overhead cannot be universally claimed" and I still stand by it 100%. pgbench experiment from Linux kernel folks was just one of the counter-examples that can be found in the wild that goes against your claim. And which you first disputed by saying that it is a micro-benchmark (meaning that you have no idea what it is) and now you're disputing it by saying it's obsolete (yes, but still very relevant in database development, used in the Linux kernel and not the slightest technical reasoning after all).
Personally, I couldn't care less about this but if names is what you're after, you're not disagreeing with me but with the methodology and results presented by Mel Gorman, Linux kernel developer.
You are backing away from your other positions, for example:
> I fail to understand the reasoning of it "being simple" or "microbenchmarkey". It's far from the truth I think.
Do you now agree that TPC-B is too simple and microbenchmarky? And if not, please tell me (as I'm working on the problem of industry benchmarking in general) what would it take to convince someone like you to stop elevating obsoleted benchmarks like TPC-B? Is there anything?
Your replies demonstrate lack of technical depth in certain areas and which makes me personally doubt in your experiment results. And you know what, that is totally fine.
But your attitude? A total disbelief.
> Do you now agree that TPC-B is too simple and microbenchmarky?
No, why would I, you did not present any evidence that would support that claim of yours?
And contrary to you, and to your own embarrassment, I do not use personal attacks when I go out of technical depth.
> what would it take to convince someone like you to stop elevating obsoleted benchmarks like TPC-B? Is there anything?
You don't have to convince me anything. Remember that this is just a random internet page where people are exchanging their opinions.
Wish you a pleasant day.
I am just a minor PostgreSQL contributor and consultant of no import, but do you seriously think you are superior to PostgreSQL core devs and know more than them about PostgreSQL just because you are a Linux kernel dev? I really do not like your attitude here and your appeal to your own authority when you are not even an authority here.
Pgbench is used heavily both in the development of PostgreSQL and by people who tune PostgreSQL. So it is not obsolete. Maybe it is a bad benchmark and that the PostgreSQL community should stop using it but then you need a stronger argument than just some claim about obsoleteness from 1995. If a PostgreSQL core developer says in 2024 that it is still relevant I think that weighs a bit higher than a claim from 1995.
Some programs, like Python, have reported worse, 2-7% [2], but there is traction on tackling that [1a] (see both rwmj's and brendangregg's replies to sibling comments, they've both done a lot of upstreamed work wrt. frame pointers, performance, and profiling).
As has been frequently pointed out, the benefits from improved profiling cannot be understated, even a 10% cost to having frame pointers can be well worth it when you leverage that information to target the actual bottlenecks that are eating up your cycles. Plus, you can always disable it in specific hotspots later when needed, which is much easier than the reverse.
Something, something, premature optimisation -- though in seriousness, this information benefits actual optimisation, exactly because we don't have the information and understanding that would allow truly universal claims, precisely because things like this haven't been available, and so haven't been widely used. We know frame pointers, from additional register pressure and extended function prologue/epilogue, can be a detriment in certain hotspots; that's why we have granular control. But without them, we often don't know which hotspots are actually affected, so I'm sure even the databases would benefit... though the "my database is the fastest database" problem has always been the result of endless micro-benchmarking, rather than actual end-to-end program performance and latency, so even a claimed "10%" drop there probably doesn't impact actual real-world usage, but that's a reason why some of the most interesting profiling work lately has been from ideas like causal profilers and continuous profilers, which answer exactly that.
[0]: https://rwmj.wordpress.com/2023/02/14/frame-pointers-vs-dwar... [1a]: https://pagure.io/fesco/issue/2817#comment-826636 [1b]: https://pagure.io/fesco/issue/2817#comment-826805 [2]: https://discuss.python.org/t/the-performance-of-python-with-...
The choice made by Intel when they have designed 8086 to use 2 separate registers for the stack pointer and for the frame pointer was a big mistake.
It is very easy to use a single register as both the stack pointer and the frame pointer, as it is standard for instance in IBM POWER.
Unfortunately in the Intel/AMD CPUs using a single register is difficult, because the simplest implementation is unreliable since interrupts may occur between 2 instructions that must form an atomic sequence (and they may clobber the stack before new space is allocated after writing the old frame pointer value in the stack).
It would have been very easy to correct this in new CPUs by detecting that instruction sequence and blocking the interrupts between them.
Intel had already done this once early in the history of the x86 CPUs, when they have discovered a mistake in the design of the ISA, that interrupts could occur between updating the stack segment and the stack pointer. Then they had corrected this by detecting such an instruction sequence and blocking the interrupts at the boundary between those instructions.
The same could have been done now, to enable the use of the stack pointer as also the frame pointer. (This would be done by always saving the stack pointer in the top of the stack whenever stack space is allocated, so that the stack pointer always points to the previous frame pointer, i.e. to the start of the linked list containing all stack frames.)
Few can leverage that information because the open source software you are talking about lacks telemetry in the self hosted case.
The profiling issue really comes down to the cultural opposition in these communities to collecting telemetry and opening it for anyone to see and use. The average user struggles to ally with a trustworthy actor who will share the information like profiling freely and anonymize it at a per-user level, the level that is actually useful. Such things exist, like the Linux hardware site, but only because they have not attracted the attention of agitators.
Basically users are okay with profiling, so long as it is quietly done by Amazon or Microsoft or Google, and not by the guy actually writing the code and giving it out for everyone to use for free. It’s one of the most moronic cultural trends, and blame can be put squarely on product growth grifters who equivocate telemetry with privacy violations; open source maintainers, who have enough responsibilities as is, besides educating their users; and Apple, who have made their essentially vaporous claims about privacy a central part of their brand.
Of course people know the answer to your question. Why doesn’t Google publish every profile of every piece of open source software? What exactly is sensitive about their workloads? Meta publishes a whole library about every single one of its customers, for anyone to freely read. I don’t buy into the holiness of the backend developer’s “cleverness” or whatever is deemed sensitive, and it’s so hypocritical.
No; the groups are approximately "cares whether software respects the user, including privacy", or "doesn't know or doesn't care". I seriously doubt that any meaningful number of people are okay with companies invading their privacy but not smaller projects.
Continuous profiling of your system that gets relayed to someone else by telemetry is very different from continuous profiling of your own system, handled only by yourself (or, generalising, your community/group/company). You seem to be imagining we're operating more in the former, whereas I am imagining more in the latter.
When it's our own system, better instrumented for our own uses, and we're the only ones getting the information, then there's nothing to worry about, and we can get much more meaningful and informative profiling done when more information about the system is available. I don't even need telemetry. When it's "someone else's" system, in other words, when we have no say in telemetry (or have to exercise a right to opt-out, rather than a more self-executing contract around opt-in), then we start to have exactly the kinds of issues you're envisaging.
When it's not completely out of our hands, then we need to recognise different users, different demands, different contexts. Catering to the user matters, and when it comes to sensitive information, well, people have different priorities and threat models.
If I'm opening a calendar on my phone, I don't expect it to be heavily instrumented and relaying all of that, I just want to see my calendar. When I open a calendar on my phone, and it is unreasonably slow, then I might want to submit relevant telemetry back in some capacity. Meanwhile, if I'm running the calendar server, I'm absolutely wanting to have all my instrumentation available and recording every morsel I reasonably can about that server, otherwise improving it or fixing it becomes much harder.
From the other side, if I'm running the server, I may want telemetry from users, but if it's not essential, then I can "make do" with only the occasional opt-in telemetry. I also have other means of profiling real usage, not just scooping it all up from unknowing users (or begrudging users). Those often have some other "cost", but in turn, they don't have the "cost" of demanding it from users. For people to freely choose requires acknowledging the asymmetries present, and that means we can't just take the path of least resistance, as we may have to pay for it later.
In short, it's a consent issue. Many violate that, knowingly, because they care not for the consequences. Many others don't even seem to think about it, and just go ahead regardless. And it's so much easier behind closed doors. Open source in comparison, even if not everything is public, must contend with the fact that the actions and consequences are (PRs, telemetry traffic, etc), so it inhabits a space in which violating consent is much more easily held accountable (though no guarantee).
Of course, this does not mean it's always done properly in open source. It's often an uphill battle to get telemetry that's off-by-default, where users explicitly consent via opt-in, as people see how that could easily be undermined, or later invalidated. Many opt-in mechanisms (e.g. a toggle in the settings menu) often do not have expiration built in, so fail to check at a later point that someone still consents. Not to say that's the way you must do it, just giving an example of a way that people seem to be more in favour of, as with the generally favourable response to such features making their way into "permissions" on mobile.
We can see how the suspicion creeps in, informed by experience... but that's also known by another word: vigilance.
So, users are not "okay" with it. There's a power imbalance where these companies are afforded the impunity because many are left to conclude they have no choice but to let them get away with it. That hasn't formed in a vacuum, and it's not so simple that we just pull back the curtain and...
Suppose Chrome added new telemetry, for example, like it did when WebRTC was added in Chrome 28, so we really can just track this against something we're all familiar (enough with). When a user clicks "Update", or it auto-updated and "seamlessly" switched version in the background / between launches, well, did the user consent to the newly added telemetry?
Perhaps most importantly: did they even know? After all, the headline feature of Chrome 28 was Blink, not some feature that had only really been shown off in a few demos, and was still a little while away from mass adoption. No reporting on Chrome 28 that I could find from the time even mentions WebRTC, despite entire separate articles going out just based on seeing WebRTC demos! Notifications got more
So, capabilities to alter software like this are, knowingly or unknowingly, undermine the nature of consent that many find implicit in downloading a browser, since what you download and what you end up using may be two very different things.
Now, let's consider a second imbalance. Did you even download Chrome? Most Android devices often have it preinstalled, or some similar "open-core" browser (often a Chromium-derivative). Some are even protected from being uninstalled, so you can't opt out that way, and Apple only just had to open up iOS to non-Safari backed browsers.
So the notion of consent via the choice to install is easily undermined.
Lastly, because we really could go on all day with examples, what about when you do use it? Didn't you consent then?
Well, they may try to onboard you, and have you pretend to read some EULA, or just have it linked and give up the charade. If you don't tick the box for "I read and agree to this EULA", you don't progress. Of course, this is hardly a robust system. Enforceability aside, the moment you had it over to someone else to look at a webpage, did they consent to the same EULA you did?
... Basically, all the "default" ways to consider consent are nebulous, potentially non-binding, and may be self-defeating. After all, you generally don't consent to every single line of code, every single feature, and so on, you are usually assumed to consent to the entire thing or nothing. Granularity with permissions has improved that somewhat, but there is usually still a bulk core you must accept before everything else; otherwise the software is usually kept in a non-functional state.
I'm not focused too specifically on Chrome here, but rather the broad patterns of how user consent typically assumed in software don't quite pan out as is often claimed. Was that telemetry the specific reason why libwebrtc was adopted by others? I'm not privy to the conversations that occurred with these decisions, but I imagine it's more one factor among many (not to mention, Pion is in/for Go, which was only 4 years old then, and the pion git repo only goes back to 2018). People were excited out of the gate, and libwebrtc being available (and C++) would have kept them in-step (all had support within 2013). But, again, really this is nothing to do with the actual topic at hand, so let's not get distracted.
The user has no opportunity to meaningfully consent to this. Ask most people about these things, and they wouldn't even recognise the features by now (as WebRTC or whatever is ubiquitous), let alone any mechanisms they may have to control how it engages with them.
Yet, the onus is put on the user. Why do we not ask about anything/anyone else in the equation, or consider what influences the user?
A recent example I think illustrates the imbalance and how it affects and warps consent is the recent snafu wit...
> Rethink how we engage, respect them, respect their consent.
One way to characterize this is leadership. Most open source software authors are terrible leaders!
You’re way too polite. Brother, who is making a mistake and deserves blame? Users? Open source non corporate software maintainers? Google employees? Someone does. It can’t be “we.” I don’t make any of these mistakes, leave me out of it! I tell every non corporate open source maintainer to add basic anonymized telemetry, PR a specific opt-out solution with my preferred Plausible, and argue relentlessly with users to probe the vapors they base their telemetry fears on. We’re both trying to engage on the issue, but the average HN reader is downvoting me. Because “vibes.” Vibes are dumb! Just don’t be afraid to say it.
For the record I'm NOT OK with being profiled, measured or otherwise studied in any way without my explicit consent. That even extends to the unethical human experiments that corporations run on people and which they euphemistically call A/B tests. I don't care if it's Google or a hobbyist developer, I will block it if I can and I will not lose a second of sleep over it.
True. But such people are like cockroaches. They know what they are doing will be unpopular with their targets, so they keep it hidden. This is easy enough to do in closed designs, car manufacturers selling your driving habits to insurance companies and health monitoring app selling menstrual cycle data to retailers selling to women.
Compare that do say Debian and RedHat. They too collect performance data. But the code is open source, Debian has repeatable builds so you can be 100% sure that is the code in use, and every so often someone takes a look it. Guess what, the data they send back is so unidentifiable it satisfies even the most paranoid of their 1000's of members.
All it takes is a little bit of sunlight to keep the cockroaches at bay, and then we can safely let the devs collect the data they need to improve code. And everyone benefits.
You're correct, results might be stale, although I wouldn't hold my breath for it since there has been no fundamental change in a way how frame pointers are handled as far as my understanding goes. Perhaps smaller improvements in compiler technology but CPUs did not undergo any significant change w.r.t. that context.
That said, nowhere in this thread have we seen a dispute of those Linux kernel results other than categorically rejecting them as being "microbenchmarks", which they are not.
> though the "my database is the fastest database" problem has always been the result of endless micro-benchmarking, rather than actual end-to-end program performance and latency
Quite the opposite. All database benchmarks are end-to-end program performance and latency analysis. "Cheating" in database benchmarks is done elsewhere.
* use framepointers [1]
* use LBR (but LBR has a limited depth, and may not work on on all CPUs, I'm assuming due to bugs in perf)
* implement some deep changes in how perf works to handle the 2 stacks in OCaml (I don't even know if this would be possible), or write/adapt some eBPF code to do it
OCaml 5 has a separate stack for OCaml code and C code, and although GDB can link them based on DWARF info, perf DWARF call-graphs cannot (https://github.com/ocaml/ocaml/issues/12563#issuecomment-193...)
If you need more evidence to keep it enabled in future releases, you can use OCaml 5 as an example (unfortunately there aren't many OCaml applications, so that may not carry too much weight on its own).
[1]: I haven't actually realised that Fedora39 has already enabled FP by default, nice! (I still do most of my day-to-day profiling on an ~CentOS 7 system with 'perf record --call-graph dwarf -F 47 -a', I was aware that there was a discussion to enable FP by default, but haven't noticed it has actually been done already)
Stuff like this is making me lean more towards getting an Intel for my next computer, at least on desktop where their worse power efficiency is less of an issue. But then they keep gating AVX-512 due to their E-cores not supporting it... You really can't win these days.
The reason we removed them was not a myth but comes from the pre-64 bit days. Not that long ago actually.
Even today if you want to repurpose older 64 bit systems with a new life then this of optimization still makes sense.
Ideally it should be the default also for security critical systems because not everything needs to be optimized for "observability"
Isn't that just 32-bit x86, which isn't used in IoT? The other 32-bit ISAs aren't register-starved like x86.
That said, I don’t know how valuable getting rid of FP on ARM is - I once benchmarked ffmpeg on 32-bit x86 before and after enabling FP and PIC (basically removing 2 GPRs) and the difference was huge (>10%) but that’s an extreme example.
And so, (as you again have noted), your best bet is to just use plain old logging and tracing (like what datadog provides) to find out where the waiting is happening.
Some people may believe that 100ms latency is acceptable for a CLI tool, but what if it could be 3ms? On some aesthetic level, it also feels good to be able to eliminate excess. Finally, you should learn it because you won't necessarily have that job forever.
https://news.ycombinator.com/item?id=38896343
There definitely have been regressions with frame pointers being enabled, although we've fixed all the ones we've found in current (2024) Fedora.
"I could say that times have changed and now the original 2004 reasons for omitting frame pointers are no longer valid in 2024."
The original 2004 reason for omitting frame pointers is still valid in 2024: it's still a big performance win on the register-starved 32-bit x86 architecture. What has changed is that the 32-bit x86 architecture is much less relevant nowadays (other than legacy software, for most people it's only used for a small instant while starting up the firmware), and other common 32-bit architectures (like embedded 32-bit ARM) are not as register-starved as the 32-bit x86.
Does this mean Microsoft decided they weren't going to support breaking profilers and debuggers OR is there some magic in the pdata section that makes it work even if you omit the frame-pointer?
“Recovering a broken stack on x64 machines on Windows is trickier because the x64 uses unwind codes for stack walking rather than a frame pointer chain.”
More details are here: https://learn.microsoft.com/en-us/cpp/build/exception-handli...
The default is omission. If you have a Windows machine, in all likelihood almost no 64 bit code running on it has frame pointers.
> OR is there some magic in the pdata section that makes it work even if you omit the frame-pointer
You haven't ever needed frame pointers to unwind using ... unwind information. The same thing exists for linux as `.eh_frame` section.
* Many frame pointer unwinders don't account for a problem they have that DWARF unwind info doesn't have: the fact that the frame set-up is not atomic, it's done in two instructions, `push $rbp` and `mov $rsp $rbp`, and if when a snapshot is taken we are in the `push`, we'll miss the parent frame. I think this might be able to be fired by inspecting the code, but I think this might only be as good as a heuristic as there could be other `push %rbp` unrelated to the stack frame. I would love to hear if there's a better approach!
* I developed the solution Brendan mentions which allows faster, in-kernel unwinding without frame pointers using BPF [0]. This doesn't use DWARF CFI (the unwind info) as-is but converts it into a random-access format that we can use in BPF. He mentions not supporting JVM languages, and while it's true that right now it only supports JIT sections that have frame pointers, I planned to implement a full JVM interpreter unwinder. I have left Polar Signals since and shifted priorities but it's feasible to get a JVM unwinder to work in lockstep with the native unwinder.
* In an ideal world, enabling frame pointers should be done on a case-by-case. Benchmarking is key, and the tradeoffs that you make might change a lot depending on the industry you are in, and what your software is doing. In the past I have seen large projects enabling/disabling frame pointers not doing an in-depth assessment of losses/gains of performance, observability, and how they connect to business metrics. The Fedora folks have done a superb and rigorous job here.
* Related to the previous point, having a build system that enables you to change this system-wide, including libraries your software depends on can be awesome to not only test these changes but also put them in production.
* Lastly, I am quite excited about SFrame that Indu is working on. It's going to solve a lot of the problems we are facing right now while letting users decide whether they use frame pointers. I can't wait for it, but I am afraid it might take several years until all the infrastructure is in place and everybody upgrades to it.
- [0]: https://web.archive.org/web/20231222054207/https://www.polar...
As for the reason why ENTER isn't used, it was deemed too slow. LEAVE (MOV SP,BP; POP BP) is used as it's just as fast as, if not faster, than the sequence it replaces. If ENTER were just the PUSH/MOV/SUB sequence, it probably would be used, but it's that other operand (which is 0 above in my example) that kills it performance wise (it's for nested functions to gain access to upper stack frames and is every expensive to use).