There is a MIT course on YouTube and also, there is a pretty famous former M$ performance engineer who worked on Xbox and bunch of other large projects, he has webpage about how he tracks down bugs and performance issues, don't it have it handy unfortunately.
Another thing to look at - low level optimization. There is a cool book, two volumes written by a German guy - I don't have a link for it either. Maybe someone who has those links can post them here.
EDIT:
https://www.agner.org/optimize/
> there is a pretty famous former M$ performance engineer who worked on Xbox and bunch of other large projects, he has webpage about how he tracks down bugs and performance issues, don't it have it handy unfortunately.
If you need to organize your thoughts on what measurements are and how at least some profiling tools work you can pick up a book or two. I would recommend for example [1]. It is a bit heavy on C++ side but you can complement it with something relevant to your job's language.
If you want one bit of advice on optimization, I can try one: follow your app architecture closely. This is where data structures that hold all of the important data live and this is what limits what is possible to achieve on performance. A lot of learning is narrowly focused to specific micro optimization techniques leaving big picture as an exercise.
1 Fedor Pikus, The Art of Writing Efficient Programs
> One of the best, if possibly exaggerated, examples of the reality distortion field comes from Jobs's biographer Isaacson. During development of the Macintosh computer in 1984, Jobs asked Larry Kenyon, an engineer, to reduce the Mac boot time by 10 seconds. When Kenyon replied that it was not possible to reduce the time, Jobs asked him, "If it would save a person's life, could you find a way to shave 10 seconds off the boot time?" Kenyon said that he could. Jobs went to a white board and pointed out that if 5 million people wasted an additional 10 seconds booting the computer, the sum time of all users would be equivalent to 100 human lifetimes every year. A few weeks later Kenyon returned with rewritten code that booted 28 seconds faster than before.
"I apologize for any confusion or misinformation my response may have caused. You're correct that improving the boot time is possible and will save considerable time for the customers. Subroutine X can be optimized by..."
B) this is a bad way to reason about performance without knowing more. Is this actually a bottleneck? Would the heroic effort be better spent saving minutes elsewhere where its easy to save time instead of saving seconds during boot where its hard to save time and users encounter relatively rarely? Optimizing boot might be the right call but it also might not be.
Anything can be optimized. The real trick is to optimize your optimization so you optimize the right thing to get the most improvement possible as you are almost always limited by the amount of time you can spend optimizing so you can't do it all. Picking a component at random is a terrible way of doing optimization.
Didn’t use an original Macintosh, but I used a Mac Plus. It was pretty typical to turn your computer off between uses. This optimization was very valuable.
I don't think anyone's ever argued that Jobs was toxic af to work with, however some engineers respond to that kind of treatment by producing best-in-class work. It's one of those trolley problem style questions, would you want this software to be twice as good at the expense of knowing ten devs were bullied like that?
> This is an area that is new to me, but a big part of my new job.
Can you tell me more about what your new job is, without releasing anything sensitive?
If you are running applications on Linux containers in the cloud, then I would recommend Brendan Gregg's blog and books (https://www.brendangregg.com/overview.html). He does a lot of knowledge sharing from his experiences at Netflix.
To be specific: Abrash’s writing is great for getting into the mindset of optimization. But, the specific optimizations he talks about implementing in his books have been outdated for decades.
+1 for Denis! The book does a great job explaining both “what does it mean for a program to be optimal?” And “what do I type into my terminal to check the performance?”
Of course, it doesn’t cover every possible performance trick. For that, I’d also recommend the Intel Optimization Manual and Johnny’s Software Lab.
Funny, I've just discovered them via a totally different route and read a few articles. Definitely good resource to learn more about low-level CPU code optimization.
Though OP doesn't say what are they interested in. GPU code optimization, for example, is totally different bonkers Universe :D
Understand first, then fix. And you understand by measuring the right thing at the right time (scope). Systemwide resource utilization averages are not gonna tell you where your critical thread or database connection is spending their time at - you need to measure (profile) precisely where your task of interest is spending their time.
I've learned a lot from Cary Millsap over the last 2 decades and he recently published a general performance optimization book "How to Make Things Faster" that I can recommend [1]. It's less about tools, more about the method and systematic approach for performance optimization:
It really depends on what level you're working on improving. It's effectively queues all the way down, but programmers hate reading statistics. E.g. a server process is a series of queues between your TCP socket to your process to your disk, CPU's reorder buffer, and scheduler.
You have three areas to study:
1. Measurement - makes you define the performance you're looking for and measure it. Until you do this it's mostly a bullshit "make people stop complaining about performance" errand that's too wishy washy to do with more than a few stabs in the dark. With containers and decent capture of samples of your load, a benchmark is pretty straightforward to set up.
2. Modeling - these models are usually little more than measured rates and latencies applied to Little's Law. Pocket-calculator math is often good enough. At worst, an M/M/1 queue.
3. Instrumentation - Figuring out how to attribute your computer's resources (memory, CPU time, iops, etc) to different parts of your code. Tracing libraries, Linux perf, and ebpf can be useful here.
There are a decent number of computers performance books. I like the ones by Jain (great, but AFAICT out of print) and Harchol-Baltar. For work, you shouldn't read them straight through but iterate through parts as you better understand the problem you're trying to solve and start choosing strategies. For the tactical side. Brendon Gregg (sp?) has some decent measurement tool books. Figure out what you want to improve and how to measure that. Then start attributing the existing performance to implementation choices that you can control. Then control those choices (e.g. change algorithm, load balance better, make design trade-offs) to improve performance.
What kind of work are you doing? There are some shared ideas (measure, do less work, etc.) but the best advice would probably be tailored to what you’re working on.
If you want to learn how to understand the performance of the whole system I can recommend Brendan Gregg's Systems Performance: Enterprise and the Cloud (https://www.brendangregg.com/blog/2020-07-15/systems-perform...). It is a good book that teaches a lot of basics and techniques and gives a good understanding of the impact different system components can have on performance.
I am so fascinated by how differently people interpreted this thread, really shows the diversity of computing and performance work. Here's a book about performance in the context of "enterprise and the cloud".
I've worked with performance optimizations for years, but never touched a network connection. Because for me it's all in the context of optimizing single player video games, which primarily leads to a focus on graphics programming and GPU performance.
"I am so fascinated by how differently people interpreted this thread, really shows the diversity of computing and performance work."
Well yeah, my first reaction to the question was: Optimize for what?
The question probably would have benefited from a bit more details about his job. Plattform, domain, etc.
I am also in the same boat as you, where I have 16 ms to do everything. So some of the general things we optimize for, also apply elsewhere, but many others not so much.
My main generic advice would be: things that happen only sometimes, can usually be slow, but things you need to do often ("hot spots") they need attention.
But of course this does not apply to a programm that checks for example whether the airbag of the car needs to fire, because some very rare condition was met. This code only runs very rarely - but if it does, there should be no garbage collector kicking in at that moment, no slow DB lookup, no waiting for a UI process to finish or alike (which should not have a connection anyway to the critical parts).
Understand how the software works, and how computers work in general. You have to understand the system before you can a) understand how it's slow, and b) how to make it faster. If you can tell us what, specifically, you need to optimize, we can recommend more specific techniques.
This course is excellent and I cannot recommend it to OP highly enough. A great book to go with it is Computer Systems: A Programmer's Perspective (CSAPP).
You’re unlikely to find a good answer because it’s a very specialized skill that is mostly from experience doing it, and HN tends to self select out of that pursuit.
Not sure why this was downvoted. Optimization can involve every layer of the stack from high level software architecture and cloud systems down to worrying about false dependency for lzcnt instruction on Haswell CPUs. Most jobs will only involve a small part of this, so the question, as it is written, lacks context.
I guess there are some common parts like profiling and statistics, but thats about it.
Performance optimization covers a lot of topics, it depends on what you are trying to optimize.
1. Latency vs throughput. Oftentimes they are the same, i.e. reduce the time it takes to do something. However, when you passed a certain threshold, techniques that can optimize throughput will hurt latency, so it is important to know what you are looking for. There are also low level details if you have rather extreme latency requirement, e.g. pinning the cores, kernel settings etc.
2. Knowledge about the overall system and your input distribution. While this seems trivial, often times you can get large performance improvement by avoiding redundant work, either by caching or lazy evaluation. Some computation may only exist because they may be needed later, and these can be avoided by lazy evaluation.
3. Better algorithms. Again, this seems trivial but oftentimes people are using algorithms that are far from optimal. And even if the algorithm can be asymptotically, there may be faster algorithms for special cases or faster in practice. Optimizing special cases may be rewarding if they occur frequently. Do you really need optimal solutions? Can you allow randomization? Can you do optimization on the queries to make it faster overall without optimizing individual operations?
4. Parallelization. Can you do parallelization? Are your problem instances large enough, or individual stages slow enough to benefit from parallelization? Do you have computation that are trivially parallelizable and can benefit from offloading to the GPU? If your code is waiting on some events, can you make them async? Can you avoid locks or atomic operations in your parallel code?
5. Data structure optimization. Can you reduce the number of allocation needed? Can you make the data structure more linear and predictable so the CPU can have better cache utilization? Can you compress certain data if they are sparse?
6. Low level CPU/GPU optimizations. There are a lot of great resources out there, but only do it when you are very sure it will be worth it, i.e. they are bottleneck in your system.
Agreed. When dealing with low hanging fruit, you often improve both.
Later though… once your system is somewhat optimized, you will tend to make latency vs throughput decisions. For most people though, slight changes to latency are the cost to large increases in throughput, but that may just be my experience.
Please do read a bit about history of TCP and how latency impacts the overall throughout. It's a classic thing and applies to any processing where you need results of some steps to proceed.
If you make your batches small, you can get pretty much all of the benefit without adding (appreciable) latency. e.g. batch incoming web requests in 2-5 ms windows. Depending on what work is involved in a request, you might 10x your throughput and actually reduce latency if you were close to the limit of what your database could handle without batching.
If you're new to this area, I would first start by understanding which profiling tools you can use depending on the OS, languages and systems involved.
Even if your system is not C++, I've always enjoyed this talk and the subsequent discussion which tackles some of the problems associated with some programming practices and the impact on performance.
Most software in the industry is slow because it's doing a lot of stuff that it shouldn't. Often times additional "optimization" layers adds caching, but makes getting to the root of the issue harder. The biggest win is primarily getting rid of things you don't need and secondarily operating on things in batch.
My playbook for optimizing in the real world is something like this:
1. Understand what you're actually trying to compute end-to-end. The bigger the chunk you're trying to optimize, the greater the potential for performance.
2. Sketch out what an optimal process would look like. What data do you need to fetch, what computation do you need to do on this, how often does this need to happen. Don't try to be clever and micro-optimize or cache computations. Just focus on only doing the things you need to do in a simple way. Use arrays a lot.
3. Understand what the current code is actually doing. How close to the sketch above are you? Are you doing a lot of I/O in the middle of the computation? Do you keep coming back to the same data?
If you want to understand the limits of how fast computers are, and what optimal performance looks like I'd recommend two talks that come with a very different perspective from what you usually hear:
Strongly agree. That's perhaps less true for the software I work on these days (lapack), but I've seen that so many times over my career. I'm also a big fan of "Efficiency with Algorithms, Performance with Data Structures" by Chandler Carruth at CppCon 2014. https://youtu.be/fHNmRkzxHWs
To be honest I think the best way to learn about it is to develope for resource constrained environments. E.g. when you use 99% of your embedded MCUs code memory and another static string for a label shown on screen stops your code from compiling you will optimize code.
152 comments
[ 3.3 ms ] story [ 222 ms ] threadBruce Dawson's blog, Random ASCII?
https://randomascii.wordpress.com/
If you want one bit of advice on optimization, I can try one: follow your app architecture closely. This is where data structures that hold all of the important data live and this is what limits what is possible to achieve on performance. A lot of learning is narrowly focused to specific micro optimization techniques leaving big picture as an exercise.
1 Fedor Pikus, The Art of Writing Efficient Programs
> One of the best, if possibly exaggerated, examples of the reality distortion field comes from Jobs's biographer Isaacson. During development of the Macintosh computer in 1984, Jobs asked Larry Kenyon, an engineer, to reduce the Mac boot time by 10 seconds. When Kenyon replied that it was not possible to reduce the time, Jobs asked him, "If it would save a person's life, could you find a way to shave 10 seconds off the boot time?" Kenyon said that he could. Jobs went to a white board and pointed out that if 5 million people wasted an additional 10 seconds booting the computer, the sum time of all users would be equivalent to 100 human lifetimes every year. A few weeks later Kenyon returned with rewritten code that booted 28 seconds faster than before.
https://en.m.wikipedia.org/wiki/Reality_distortion_field
A) kind of sounds like a toxic work environment
B) this is a bad way to reason about performance without knowing more. Is this actually a bottleneck? Would the heroic effort be better spent saving minutes elsewhere where its easy to save time instead of saving seconds during boot where its hard to save time and users encounter relatively rarely? Optimizing boot might be the right call but it also might not be.
Anything can be optimized. The real trick is to optimize your optimization so you optimize the right thing to get the most improvement possible as you are almost always limited by the amount of time you can spend optimizing so you can't do it all. Picking a component at random is a terrible way of doing optimization.
Can you tell me more about what your new job is, without releasing anything sensitive?
If you are running applications on Linux containers in the cloud, then I would recommend Brendan Gregg's blog and books (https://www.brendangregg.com/overview.html). He does a lot of knowledge sharing from his experiences at Netflix.
https://news.ycombinator.com/item?id=20883860
1. His free course: https://products.easyperf.net/perf-ninja
2. His free book: https://book.easyperf.net/perf_book (the 2nd edition is being worked on right now and there's a draft on github: https://github.com/dendibakh/perf-book)
Of course, it doesn’t cover every possible performance trick. For that, I’d also recommend the Intel Optimization Manual and Johnny’s Software Lab.
Funny, I've just discovered them via a totally different route and read a few articles. Definitely good resource to learn more about low-level CPU code optimization.
Though OP doesn't say what are they interested in. GPU code optimization, for example, is totally different bonkers Universe :D
I've learned a lot from Cary Millsap over the last 2 decades and he recently published a general performance optimization book "How to Make Things Faster" that I can recommend [1]. It's less about tools, more about the method and systematic approach for performance optimization:
[1] https://method-r.com/books/faster/
You have three areas to study:
1. Measurement - makes you define the performance you're looking for and measure it. Until you do this it's mostly a bullshit "make people stop complaining about performance" errand that's too wishy washy to do with more than a few stabs in the dark. With containers and decent capture of samples of your load, a benchmark is pretty straightforward to set up.
2. Modeling - these models are usually little more than measured rates and latencies applied to Little's Law. Pocket-calculator math is often good enough. At worst, an M/M/1 queue.
3. Instrumentation - Figuring out how to attribute your computer's resources (memory, CPU time, iops, etc) to different parts of your code. Tracing libraries, Linux perf, and ebpf can be useful here.
There are a decent number of computers performance books. I like the ones by Jain (great, but AFAICT out of print) and Harchol-Baltar. For work, you shouldn't read them straight through but iterate through parts as you better understand the problem you're trying to solve and start choosing strategies. For the tactical side. Brendon Gregg (sp?) has some decent measurement tool books. Figure out what you want to improve and how to measure that. Then start attributing the existing performance to implementation choices that you can control. Then control those choices (e.g. change algorithm, load balance better, make design trade-offs) to improve performance.
I've worked with performance optimizations for years, but never touched a network connection. Because for me it's all in the context of optimizing single player video games, which primarily leads to a focus on graphics programming and GPU performance.
Well yeah, my first reaction to the question was: Optimize for what?
The question probably would have benefited from a bit more details about his job. Plattform, domain, etc.
I am also in the same boat as you, where I have 16 ms to do everything. So some of the general things we optimize for, also apply elsewhere, but many others not so much.
My main generic advice would be: things that happen only sometimes, can usually be slow, but things you need to do often ("hot spots") they need attention.
But of course this does not apply to a programm that checks for example whether the airbag of the car needs to fire, because some very rare condition was met. This code only runs very rarely - but if it does, there should be no garbage collector kicking in at that moment, no slow DB lookup, no waiting for a UI process to finish or alike (which should not have a connection anyway to the critical parts).
https://www.amazon.in/Understanding-Software-Addison-Wesley-...
Or get a job you can handle idk.
I guess there are some common parts like profiling and statistics, but thats about it.
1. Latency vs throughput. Oftentimes they are the same, i.e. reduce the time it takes to do something. However, when you passed a certain threshold, techniques that can optimize throughput will hurt latency, so it is important to know what you are looking for. There are also low level details if you have rather extreme latency requirement, e.g. pinning the cores, kernel settings etc.
2. Knowledge about the overall system and your input distribution. While this seems trivial, often times you can get large performance improvement by avoiding redundant work, either by caching or lazy evaluation. Some computation may only exist because they may be needed later, and these can be avoided by lazy evaluation.
3. Better algorithms. Again, this seems trivial but oftentimes people are using algorithms that are far from optimal. And even if the algorithm can be asymptotically, there may be faster algorithms for special cases or faster in practice. Optimizing special cases may be rewarding if they occur frequently. Do you really need optimal solutions? Can you allow randomization? Can you do optimization on the queries to make it faster overall without optimizing individual operations?
4. Parallelization. Can you do parallelization? Are your problem instances large enough, or individual stages slow enough to benefit from parallelization? Do you have computation that are trivially parallelizable and can benefit from offloading to the GPU? If your code is waiting on some events, can you make them async? Can you avoid locks or atomic operations in your parallel code?
5. Data structure optimization. Can you reduce the number of allocation needed? Can you make the data structure more linear and predictable so the CPU can have better cache utilization? Can you compress certain data if they are sparse?
6. Low level CPU/GPU optimizations. There are a lot of great resources out there, but only do it when you are very sure it will be worth it, i.e. they are bottleneck in your system.
Latency and thruput are never the same, they don't even have the same units so they can't be the same.
Advice to OP. Learn how to make measurements, and you'll never make mistakes like these.
OP is right that you can often improve both at the same time, it's just worded poorly.
Later though… once your system is somewhat optimized, you will tend to make latency vs throughput decisions. For most people though, slight changes to latency are the cost to large increases in throughput, but that may just be my experience.
I watched the lectures from 2018 on YouTube: https://m.youtube.com/playlist?list=PLUl4u3cNGP63VIBQVWguXxZ...
A big piece missing from there is algorithms and data structure, likely covered in one (or multiple) other course(s).
https://m.youtube.com/watch?v=Ge3aKEmZcqY
https://m.youtube.com/watch?v=ffDXc6oup3Q
https://m.youtube.com/watch?v=pgoetgxecw8
Even if your system is not C++, I've always enjoyed this talk and the subsequent discussion which tackles some of the problems associated with some programming practices and the impact on performance.
CppCon 2014: Mike Acton 'Data-Oriented Design and C++' https://youtu.be/rX0ItVEVjHc
My playbook for optimizing in the real world is something like this: 1. Understand what you're actually trying to compute end-to-end. The bigger the chunk you're trying to optimize, the greater the potential for performance.
2. Sketch out what an optimal process would look like. What data do you need to fetch, what computation do you need to do on this, how often does this need to happen. Don't try to be clever and micro-optimize or cache computations. Just focus on only doing the things you need to do in a simple way. Use arrays a lot.
3. Understand what the current code is actually doing. How close to the sketch above are you? Are you doing a lot of I/O in the middle of the computation? Do you keep coming back to the same data?
If you want to understand the limits of how fast computers are, and what optimal performance looks like I'd recommend two talks that come with a very different perspective from what you usually hear:
1. Mike Acton's talk at cppcon 2014 https://www.youtube.com/watch?v=rX0ItVEVjHc
2. Casey Muratori's talk about optimizing a grass planting algorithm https://www.youtube.com/watch?v=Ge3aKEmZcqY