> Axboe shared about his latest interesting Linux I/O performance optimization, "Something I've had in the back of my mind for years, and finally did it today. Which is kind of sad, since it was literally a 5 min job, yielding a more than 6% improvement. Would likely be even larger on a full scale distro style kernel config."
Seems irrelevant. From the sound of it this was something that would always have been easy for him to do. He just never got to do it until now.
Maybe, maybe not. Lots of stuff gets put in "the back of my mind" because I'm not sure how to do it immediately. Maybe I'll think about it occasionally, or note a consequence like "oh this would be affected if I get around to X". Then when I actually sit down to do it weeks or months later it seams easy - even though when I first thought it I wasn't sure how to even do it.
Point being- it's pretty unlikely that something bouncing around the back of someone's head for years would have taken 5 minutes to do right away, just as it's unlikely that it would have taken 6 years if they'd started when they first had the thought. I suspect there's been some amount of thinking, and potentially that other work was subtly influenced by this thought so that finally doing the task was "easy".
In my life and career experience, it wouldn't take 5 minutes if you got to it the first time you had the thought.
That being said, on average, it would take 5 minutes if you started doing it the 5th - 10th time you thought about it. Definitely doesn't need 6 years indeed.
So it seems we very often drag our feet when it comes to thoughts that are already well-baked / perfectly fermented.
I am reminded of the story of the plumber who was called to fix a water leak. He replaced a washer and charged $100. The client was outraged: “That washer was only a dollar’s worth of parts!”
The plumber replied: “Sure. I’m charging you $1 for the washer and $99 for knowing where to fit it.”
This famous "granddaddy of them all" story about knowing where to make a chalk mark was validated in a 1965 letter to the editor of Life magazine. Although most Internet sources act as though this story is apocryphal, it has some real (if not conclusive) provenance, and definitely fits the known personality quirks of both Henry Ford and Charles Steinmetz...
Title should be "Six years of thinking allow programmer to solve problem in 5 minutes".
I remember contracting for IBM in the 90s and we had a huge performance boost due to caching and, like the anecdote in this post, there was a slight chance of something going out of date. The IBM guys said "no thanks, IBM prefers bullet proof over fast".
Yep, especially as IT moved downmarket from mainframes run by government, military and large corporations to the average Joe, who can reboot, reinstall or just buy a new one.
(c.1986) Telephone company accounting system with System/38 (pre AS/400) and System R (early SQL), CL+RPGIII. The call rating process required half a dozen lookups to other tables to complete each record in a batch. Something that would be well handled by a modern join. But at the time the vendor was like, "Gee Ma it works! Ship it." Compiled RPG III was lightning fast and a lot like working in assembly with fixed/prealloc string handling. I made a template and adapted it for each table that cached recent lookups for each. 6x improvement. Then realized that since records were autonomous and batches run and posted separately, it was never necessary to process calls in chrono order. Pre-indexing on destination number resulted in cache hits almost all the time and 10x speed improvement over the 6x.
When I asked my Dad how long something took, he'd say something like "2 years and 20 minutes." 2 years to get around to it. Mom took his habit into the real world and published an ad for the company with a cut out "Round Tuit" and copy, "How many times have you heard someone say, 'I'll do that when I get a round tuit.' Now that you have your very own Round Tuit, there's no limit to what you can accomplish!"
Round Tuits were cut out and appeared on walls all over town.
I think when I was 17 some old guy asked me why I hadn't changed my oil yet* and I said I hadn't got around to it. He ran into his shed and came out with a round piece of wood that said "to it" on it... So I changed the oil in his driveway.
this is a pretty standard performance win to avoid constantly getting the time when you don't need it. Certainly seen this win at work (AWS) but also likely in various other projects repeatedly. But bias for action for getting it done!
I wonder if there’s many other spots where time is being read multiple times instead of reading it once and caching it. All that being said, it’s also surprising how relatively expensive reading time is. Even rdtsc is 50-100ns if I recall correctly, but to put that in perspective that’s basically the performance of a 10-20Mhz chip to do an operation that should be a simple register read of a latch that’s incremented on every clock cycle. yes yes - I know in reality it’s a heck of a lot more complex for lots of reasons, but still, given how frequently we want to time things or obtain a timestamp, it’s surprising to me we don’t have a faster instruction and we have to do a bunch of caching (eg you’d not likely bother caching just the addition of two variables because addition is so fast)
> . All that being said, it’s also surprising how relatively expensive reading time is.
Typically you don't need granularity as fine as you think you do. Reading time at the finest granularity available is going to kill your throughput/latency no matter what because you are probing the hardware, so make sure you need that very fine granularity.
IME, most (99.999%) of the case you need the time, you can get by with coarse granularity. I worked on realtime systems to control a jig in a factory, using embedded 486-class processors, with hard-realtime kernel patches, and even then I had no need to use the non-coarse timers when recording events for observability.
Meanwhile on Win32, GetTickCount performs no system calls at all, simply reading a value from a shared memory page. The operating system is updating that value behind the scenes.
This isn't about performing a syscall to get the time, this is about the kernel handling a syscall and in the process checking the time too many times.
And getting the time without a full syscall is hardly unique to Windows. On Linux, that's almost the entire point of the vDSO.
A giant ship’s engine failed. The ship’s owners tried one ‘professional’ after another but none of them could figure out how to fix the broken engine.
Then they brought in a man who had been fixing ships since he was young.
He carried a large bag of tools with him and when he arrived immediately went to work. He inspected the engine very carefully, top to bottom.
Two of the ship’s owners were there watching this man, hoping he would know what to do. After looking things over, the old man reached into his bag and pulled out a small hammer. He gently tapped something. Instantly, the engine lurched into life. He carefully put his hammer away and the engine was fixed!!!
A week later, the owners received an invoice from the old man for $10,000.
What?! the owners exclaimed. “He hardly did anything..!!!”.
So they wrote to the man; “Please send us an itemised invoice.”
The man sent an invoice that read:
Tapping with a hammer………………….. $2.00
Knowing where to tap…………………….. $9,998.00
Effort is important but experience and knowing where to direct that effort makes all the difference.
The really awful thing about the story you’ve reposted is it’s actually a real story about a really intriguing and amazing person. Why tell the moral of the story when you can tell the real story?
37 comments
[ 3.6 ms ] story [ 85.7 ms ] threadSeems irrelevant. From the sound of it this was something that would always have been easy for him to do. He just never got to do it until now.
Point being- it's pretty unlikely that something bouncing around the back of someone's head for years would have taken 5 minutes to do right away, just as it's unlikely that it would have taken 6 years if they'd started when they first had the thought. I suspect there's been some amount of thinking, and potentially that other work was subtly influenced by this thought so that finally doing the task was "easy".
That being said, on average, it would take 5 minutes if you started doing it the 5th - 10th time you thought about it. Definitely doesn't need 6 years indeed.
So it seems we very often drag our feet when it comes to thoughts that are already well-baked / perfectly fermented.
The plumber replied: “Sure. I’m charging you $1 for the washer and $99 for knowing where to fit it.”
https://www.reddit.com/r/todayilearned/comments/2b4n7a/til_h...
I remember contracting for IBM in the 90s and we had a huge performance boost due to caching and, like the anecdote in this post, there was a slight chance of something going out of date. The IBM guys said "no thanks, IBM prefers bullet proof over fast".
Bulletproof typically means rigid in design. Trading one bag of worms for another.
"You didn't pay me $1k to press three buttons; you paid me $1k for knowing which three buttons to press."
EDIT: oh, i see a version of this was already posted in another comment :)
When I asked my Dad how long something took, he'd say something like "2 years and 20 minutes." 2 years to get around to it. Mom took his habit into the real world and published an ad for the company with a cut out "Round Tuit" and copy, "How many times have you heard someone say, 'I'll do that when I get a round tuit.' Now that you have your very own Round Tuit, there's no limit to what you can accomplish!"
Round Tuits were cut out and appeared on walls all over town.
* The oil was still <5k mi old. People are funny.
Typically you don't need granularity as fine as you think you do. Reading time at the finest granularity available is going to kill your throughput/latency no matter what because you are probing the hardware, so make sure you need that very fine granularity.
IME, most (99.999%) of the case you need the time, you can get by with coarse granularity. I worked on realtime systems to control a jig in a factory, using embedded 486-class processors, with hard-realtime kernel patches, and even then I had no need to use the non-coarse timers when recording events for observability.
In any case, here's the relative costs of the different timers: https://www.lelanthran.com/chap5/content.html
And getting the time without a full syscall is hardly unique to Windows. On Linux, that's almost the entire point of the vDSO.
https://man7.org/linux/man-pages/man7/vdso.7.html
Then they brought in a man who had been fixing ships since he was young. He carried a large bag of tools with him and when he arrived immediately went to work. He inspected the engine very carefully, top to bottom.
Two of the ship’s owners were there watching this man, hoping he would know what to do. After looking things over, the old man reached into his bag and pulled out a small hammer. He gently tapped something. Instantly, the engine lurched into life. He carefully put his hammer away and the engine was fixed!!!
A week later, the owners received an invoice from the old man for $10,000.
What?! the owners exclaimed. “He hardly did anything..!!!”.
So they wrote to the man; “Please send us an itemised invoice.”
The man sent an invoice that read:
Tapping with a hammer………………….. $2.00
Knowing where to tap…………………….. $9,998.00
Effort is important but experience and knowing where to direct that effort makes all the difference.
dirty C&P from: https://medium.com/@oceanbcreative/the-ship-repair-man-story...
https://en.m.wikipedia.org/wiki/Charles_Proteus_Steinmetz
https://www.smithsonianmag.com/history/charles-proteus-stein...