> One day we were chatting in a group meeting and he said he was working on a TPC-C benchmark that was spending too much time in the SCSI disk driver, so he wrote a 10-line awk script running against kernel memory to look at the SPARC machine code execution path and find which branches were being predicted wrong, then wrote another awk script to flip the branch prediction bits in memory and it showed a significant speedup.
SPARC exposed read and write access to branch prediction!? That's... fascinating. Here I thought branch prediction was always a completely opaque thing that the CPU did internally and maybe if you were lucky it might let the program be able to know that it was happening after the fact. Hm... I wonder if it's easier to perform Spectre type attacks if you can just ask the CPU about that kind of behavior? No need to do timing attacks if the CPU will just tell you
I would assume you have to be root to do this branch predictor stuff. If you're root, why do you need Spectre? If you're attacking some trusted execution thing, then I assume you wouldn't be able to do the branch predictor stuff.
IIRC you specified the likely path in assembly code, but not all SPARCs had those opcodes. They were some sort of extention. By default they had simple fixed prediction (branch would not be taken, backwards branch would be taken etc).
Some of the pentium series also had settable branch prediction bias flags.
SPARC had instructions (BP, Branch with Prediction) which hinted the branch predictor whether the branch was likely or unlikely. Presumably this script just edited the opcodes in memory to avoid needing to figure out how to change the generated code from the compiler, recompile, and replace the kernel.
It would seem more likely to be a scsi/kex timing issue and there are direct hacks into that when you have a loose dev/kmem philsophy.
If you are/can directly hack the pipeline, Adrian better have included that script in the bmark publish. Probably not, because then directors would be calling him at 2am because XYZ corps workload was slow and “could we have your nop script?”
These were simpler RISC implementations in those days, the compiler optimizer stage was in charge of deciding whether to set the branch bit or not, and the prefetch would do what the bit said, and stall if it got it wrong.
We used to have a piece of Java software called Scout - it did a lot of small memory allocations... Led to runtime of multiple days for some datasets. I was reading the book Solaris Internals. The edition I had was updated for the ultrasparc 3's, and I was reading that section that talked about memory pages and the new support for huge pages, eg 4GB pages. We switched on 4GB pages and the week long runs of Scout went to only a few hours. That performance gain was real money and time.
i didn't know what i was expecting when i started to read this but it wasn't that. i didn't know the author before the article but i thought it was really an amazing read. i think it speaks very well to the power of collaboration and a congenial and healthy work environment.
as another commenter mentioned the thing about the branch prediction bit flipping in awk was quite something. the SE Toolkit to monitor kernel perf, which could run for years without issue - was something cool too. the guy who found and fixed a kernel bug offering the biggest performance win in Solaris 8 in the course of writing the first book in a series called Solaris Internals - wow. the description of "the can of worms project" in which they run workloads that look like what customers might use, things that that go beyond typical eng team benchmarks, and the meltdown prevention and heading off of customer issues, it's a great concept from my personal point of view. i appreciated the touching on other more human subjects like the capacity planning DB pricing calculator for customers and the bizdev, relationship, and product design points.
i started off being jealous of what the guy was saying but i was so blown away by everything that i had forgotten that feeling by the end of the article. it was replaced with awe.
Thanks! That’s cool. I’m glad I found time to write down what I remembered and the names of many of the people I met along the way. I have notes but haven’t had time to tell the story of the Netflix years yet…
A fun reminder that Sun charged what they damn pleased for hardware and wasted the money on a party culture and a lot of make-work jobs for their pampered staff. This is why Linux won.
That's b*ll, really. Sun's "party culture", if that's what you'd like to call it, is and was "of its time". As were 1000$-a-head New York night club bashes for a "successful" team at Wall St. pre-Lehman.
All great to have had if you happened to have been in the right place at the right time. It was as-common then as free snacks and ping-pong tables are today. We lucky old bastards.
I'd rather call the mid-to-late 1990s the "Golden Age" of systems. All on-premises (hence a glut of money going to hardware), more and more business needs for software (and most of it proprietary, with a nice chunk of "professional services" costs on-top to configure & customize). IBM, Oracle, SAP or Microsoft just played that latter game, software/services, with higher stakes and higher wins than Sun, HP/Compaq, Dell etc did for hardware.
What killed Sun wasn't their "party culture". It was their arrogance to believe SPARC the top of the world years after cheap&cheerful x86 had proven itself, and their arrogance to give customers the message "we know best". Sun had more than one chance to pivot to-Linux, or off-hardware, or both, and botched it every time. If you fail to read the market, and refuse to sell customers what they ask for, then no matter your past record, your future looks dull.
PCs are still garbage hardware, compared to the minicomputers and workstations that were available until the mid '90s.
No spec, no datasheets, no reliable supply chain, poor ESD, poor cooling. No cooperation between the hardware vendor and the OS vendor. Macs, were they still just making shinier NeXT hw as opposed to lifestyle props, would've been a welcome exception.
Don't even get me started on the senescence of what passes for "commercial Linux". People actually pay RH/IBM for this.
To young engineers reading this, note that his early sales experience taught him how to deliver value to customers, not just checkins to git. If someone tries to promote you into sales engineer and you shy away because of wearing a tie or whatever, you are losing out on a huge opportunity to improve your engineering career.
Conversely, the article also shows a path out of being a sales engineer. It can be difficult for SEs: they are typically technical, but not coders, and sales-y, but not account managers. Good career paths for SEs can be hard to find. Kudos to Adrian and to Sun for finding a way.
25 comments
[ 2.8 ms ] story [ 64.0 ms ] threadSPARC exposed read and write access to branch prediction!? That's... fascinating. Here I thought branch prediction was always a completely opaque thing that the CPU did internally and maybe if you were lucky it might let the program be able to know that it was happening after the fact. Hm... I wonder if it's easier to perform Spectre type attacks if you can just ask the CPU about that kind of behavior? No need to do timing attacks if the CPU will just tell you
See for example
https://developer.arm.com/documentation/ddi0595/2020-12/AArc...
but good luck 1. having a cpu with it (can't wait for arm laptops and needing caniuse.org for my assembly code), and 2. figuring out how to use it
Some of the pentium series also had settable branch prediction bias flags.
If you are/can directly hack the pipeline, Adrian better have included that script in the bmark publish. Probably not, because then directors would be calling him at 2am because XYZ corps workload was slow and “could we have your nop script?”
That was an interesting trip down memory lane.
as another commenter mentioned the thing about the branch prediction bit flipping in awk was quite something. the SE Toolkit to monitor kernel perf, which could run for years without issue - was something cool too. the guy who found and fixed a kernel bug offering the biggest performance win in Solaris 8 in the course of writing the first book in a series called Solaris Internals - wow. the description of "the can of worms project" in which they run workloads that look like what customers might use, things that that go beyond typical eng team benchmarks, and the meltdown prevention and heading off of customer issues, it's a great concept from my personal point of view. i appreciated the touching on other more human subjects like the capacity planning DB pricing calculator for customers and the bizdev, relationship, and product design points.
i started off being jealous of what the guy was saying but i was so blown away by everything that i had forgotten that feeling by the end of the article. it was replaced with awe.
I'd rather call the mid-to-late 1990s the "Golden Age" of systems. All on-premises (hence a glut of money going to hardware), more and more business needs for software (and most of it proprietary, with a nice chunk of "professional services" costs on-top to configure & customize). IBM, Oracle, SAP or Microsoft just played that latter game, software/services, with higher stakes and higher wins than Sun, HP/Compaq, Dell etc did for hardware.
What killed Sun wasn't their "party culture". It was their arrogance to believe SPARC the top of the world years after cheap&cheerful x86 had proven itself, and their arrogance to give customers the message "we know best". Sun had more than one chance to pivot to-Linux, or off-hardware, or both, and botched it every time. If you fail to read the market, and refuse to sell customers what they ask for, then no matter your past record, your future looks dull.
No spec, no datasheets, no reliable supply chain, poor ESD, poor cooling. No cooperation between the hardware vendor and the OS vendor. Macs, were they still just making shinier NeXT hw as opposed to lifestyle props, would've been a welcome exception.
Don't even get me started on the senescence of what passes for "commercial Linux". People actually pay RH/IBM for this.
tl;dr: Get off my lawn!