I guess at that point of ones career they can basically write their own job description and budget, and people will line up to pay them whatever that description says.
I've been clear on HN multiple times that I saw no future in the Solaris/SmartOS ecosystem by 2014, which is one (but not the only) reason I left it. I was right too: https://www.itjobswatch.co.uk/jobs/uk/solaris.do
That seems like an odd conclusion to me. If you've got deep expertise in the spaces he does, and the server market moves mostly to Linux, getting involved there seems unavoidable. Lingering on the dying SmartOS ecosystem, even if it was technically better, wouldn't have served anyone well.
Edit: Ah, as mentioned below, perhaps a joke? I'd attributed the smiley face only to the latter part of the sentence.
The odd conclusion to me is how the last line I've post was interpreted as a serious statement.
Didn't the "Before he got his Netflix special.." or the fact Linux is pretty much the only game in town with no sign of even being seriously challenged served as a hint?
Netflix has hired more great BSD folk, so I haven't needed to do a lot on it for a while. I do enjoy it though, and it's usually much easier to read and understand the kernel code.
Most recently I've been talking to folk about eBPF on BSD. Maybe I should do a blog post about it (force me to go talk to everyone again and get their updates).
Thank you Brendan. I work in systems performance engineering in finance and always have been your humble and fanatic follower.
I learned a lot from your books, posts and talks and use your tools (mostly bpftrace and flame graphs for the JVM) on daily basis. I can't thank you enough for the wealth of wisdom you shared with us, mortals!
I started as a java developer around 20 years ago; in those times it was called enterprise java and at some point I joined a consultancy that had many clients in telco industry. That meant there was a lot of opportunities to get exposure to performance troubleshooting. At first it was only on the JVM level but you quickly realize the virtual machine doesn't run in vacuum an the OS as well as HW are equally important layers of software execution stack.
I devoted a lot of time to learning about systems engineering and modern hardware. When switching jobs it was always with the purpose of getting more and more exposure to performance-focused software engineering.
In my case the last 12 years was was exclusively big banks and one exchange (LMAX, UBS, Goldman Sachs, Morgan Stanley, HSBC). From what I can tell there's more opportunities to get your hands dirty with perf on the buy side (hedge funds, prop trading shops) but I always got lucky with the big market makers and got involved in amazingly interesting things anyway.
I absolutely love this line of work and wouldn't trade it for anything else but it sometimes costs you some grey hair especially if you're dealing with the very bleeding edge of hardware and kernel/OS level bits (think spending weeks chasing misbehaving icache on a heavily overclocked cpu).
This job is very rewarding but requires real passion and constant learning.
I've been following Brendan since the Sun Microsystems days with his Dtrace book which opened my eyes to the world of system call tracing from user-space to kernel. Linux at the time had a hodgepodge of tracing utilities (SystemTap, Ktap etc..) that usually required kernel modules to work properly. Strace was my goto in Linux back then but slowed down the target binary a lot. Thank the universe for eBPF and Brendan's current books that today we have powerful tracing capabilities in Linux that rivals Dtrace in FreeBSD/Illumos. If you want to understand the internals of an operating system and how things work, without being a kernel developer, systems performance engineering is a way to go.
Peformance (beyond analysis) is one of the most interesting things to do beyond building things. It's needed and relevant in many settings and you can (have to) do it without the JD (and IMO you should at least initially).
You'll find plenty of this at any relevant size infrastructure place (there are a lot of them).
The principle to do whatever you love is to "ask for forgivness not permission". Feel free to understand why you're solving what you're solving in order to challenge it and do it right.
I'm currently at a very big company (not faang) and I had the chance to do a little bit of performance analysis.
But unfortunate it was a totally unoptimized thing so only quick low hanging fruits.
Like io bound but they didn't though that component is the issue as it was not CPU bound, bad SQL queries and unoptimized caches.
Nothing big or difficult to analyse :-(
I learned about one person who is only doing java Performance analysis and he said it took a while to be known in the industry good enough that he can do it fulltime.
Hey it’s fine to build a name for yourself while knocking out low hanging fruit. It’s still bringing useful and meaningful results in a professional setting.
I work on performance and I enjoy it even more than writing features, honestly. There’s much more freedom in it in a professional setting because you generally get to define the subtasks under the goal of “make it faster/more efficient” whereas with a feature you’re usually told what specific thing you’re supposed to make, and your flexibility is usually just picking how to implement that. And the problems are almost always challenging and require technical (and occasionally, political) insight to fix. With features you might ship something that’s loved or a dud, but improving the performance of something is basically always a good thing.
I’ve built and run a performance monitoring platform/team at a large tech company. I’ve also worked in security for the past few years.
Both have similar problems: if the business goals don’t align with reducing latency or improved security posture then your detailed analysis will get parked in a ticket queue and promptly forgotten. Your OKRs will be missed. You will also be fighting the tide against new features developed sloppily- as latency will increase and then investment in your team seems like it’s not working to the executives. Fortunately we had a supportive directive but being siloed into an infra org didn’t give us the juice to influence product roadmaps.
We built a lab for testing against devices, did a bunch of work to isolate noise, integrated automated regression testing into CI to combat this. It still didn’t move the needle frankly, as the teams building products were not held accountable for the regressions (and don’t forget that it’s likely the right move for the greater business health!).
Netflix is unique in that latency is very core to their product.
Off topic: I’m a bit surprised about Gregg’s desk (pre-pandemic). I imagine he’s getting a top level salary at Netflix but yet he’s got a small desk in what it looks to me a shared small office (or perhaps is that a mini open space office? Can’t tell).
A number of times people have asked about my desk over the years, and I'm curious as to why! I've visited other tech companies in the bay area, and the desks I see (including for 7-figure salary engineers) are the same as everyone else, in open office layouts. At Netflix it's been open office desks, and all engineers have the same desk.
Does some companies give bigger desks for certain staff, or offices, or is it a country thing (Europe?).
I think given your visibility and profile (compared to many tech workers), the previous poster expected a grand office or vast desk, certainly different from the setup “everyone else” has.
I understand why people might think that, but I'm not sure it'd be great for the culture for me to be different from my coworkers. We're all working on important things, and we all need to collaborate together.
Absolutely, I agree with you there. I’m relatively junior in my team and my team is quite absent of clout within our organisation as we’re a new team.
Our correspondent teams are much more approachable when it comes to collaborating than I’ve experienced in the past - it’s something I find very refreshing.
Ah right, people have asked about that before. I'm just happy with a single normal-sized monitor.
Maybe related, but we once went around the Netflix parking lot with Blippar to look at the KBB value of the cars, and found that I was driving the cheapest car at Netflix (at least until we got sick of looking). Mine was a 2001 Toyota Prius, KBB value ~$1800. Not the cheapest car to ever park at Netflix, just the cheapest on the day!
I don't find the desk itself surprising, but I am surprised by the open office plan. I've found sharing an open space with more than two or three other people extremely distracting. I'm sure it depends a lot on the personalities of the coworkers.
Yes, I helped organize the Netflix cricket practice sessions. Netflix has a cricket team (in engineering) that in the past has played in the GPL: Google Premiership League, along with several other tech companies. Lots of fun!
We practiced on baseball diamonds, and played league games on the Google soccer field.
When I ran the BPF tutorial at USENIX LISA 2019, I had one finger in a splint from breaking it playing cricket. :-)
Used to work at Sun in Ire and i had a massive corner desk cubicle, whiteboard, 3 monitors etc. Then working in banking it was pretty big cubicle areas (NYC/London) now reduced down to more normal desk sizes.
I don't know about larger companies, but I work for a much smaller company and every person has an office with a door. After having my own space, I'm not sure I could go back to an open office plan again.
I haven't had my own office. That feels like such a 1950s Mad Men type of thing. Has to be pretty rare outside of accountants and lawyers today.
But I will say the absolutely best I've had was 3 devs to a single office, with desks oriented so that the monitors face the walls for privacy. We could chat throughout the day, but you could also get real work done. I've done the open floor plan thing. Never again. There is a certain constant level of anxiety with having people floating around you and talking nearby all day long.
I've followed you awesome work on Linux performance tuning over the years. All the best for whatever you're doing next and I hope it ends up in another book for us mere mortals to read :-).
First time I’m really hearing about Brendan Gregg’s work. Not that I’m someone who should know, but I think this puts eBPF on my future studies list. Plus Australia sounds very cool. Currently working towards RHCE certification myself after a long time of swearing I don’t need it. (I want a new, more advanced job so I need it.)
I am so grateful somebody’s using FreeBSD in 2022 in production. I do wonder whether there’s anything that could get it more use. Maybe a certification and backing company like red hat does for Linux. Or something more harebrained like a smartphone app geared towards systems management for the upcoming generation. I’ve heard they’re generally less interested in this stuff.
I say this because it’s awesome to see Netflix having people so deep in the tech stack. But somehow still feels a little fragile.
When engineers like Brendan Gregg complain about American immigration and visa processes, you know something is really really f-ed up in how America treats humans.
Netflix has generally amazing latency, I can feel that the engineers work really hard on fixing all latency issues, except when it totally fails: offline mode.
I always have some films downloaded on my iPad Pro that I want to watch when I don't have internet (travelling on long plane flight), and with Netflix always the best films that I want to watch are ,,expired''. It would take 1 second to ,,renew'' them, but I have to wait until the end of the flight to have internet access again, and then I forget it, as I don't look at downloads anymore. It's really frustrating that I see that I have something downloaded, taking space on my SSD, but can't be used when I need it.
I wish it could just renew the video certificates automatically whenever I access the internet, and generally it would be great to have better analysis tools for offline / crappy internet for companies.
Usually I use Youtube more on the plane, because it doesn't play these games with me.
I have the same experience. It feels like the download experience was only really made for the “I’m watching a series and want to keep watching it on the plane” use case, and not the “I downloaded 10 episodes of Micky Mouse for my 3yo on our last trip a year ago and now she wants to watch them again but I can’t cause…” use case.
Thanks for sharing this. Why do more companies not follow this philosophy?
Perhaps one advantage for Netflix is that they’re largely a one-trick pony. They don’t have 20+ products fighting for attention and engineers within the company.
Brendan is truly one of the best educators in tech engineering. He may not think of himself as such, and he's also obviously a fantastic engineer, but what really stands out to me is his ability to communicate ideas coherently.
So much of the "staff+" engineer world is hot air, and Brendan has always struck a contrast with his straight forward, information-dense blog posts and books, and down-to-earth communication style. He explores, and often defines, the frontiers of what we do and then brings back lessons.
I'm a huge fan, and can't wait to see what he does (and shares) next.
I hope he will be contributing to the performance space as much as he's been at Netflix. His books and blogs have helped many including me better understand the whole stack.
75 comments
[ 0.25 ms ] story [ 204 ms ] threadOne of the game companies that do distribution (valve, epic)
Working with old colleagues (as a C-level?) at oxide.
One of the major cloud providers, very likely microsoft.
https://news.ycombinator.com/item?id=31052035
I can only hope that he brings FreeBSD to his next employer & is allowed to continue his contributions.
The Linux stuff was a money-grab, some say he sold out.. :)
That seems like an odd conclusion to me. If you've got deep expertise in the spaces he does, and the server market moves mostly to Linux, getting involved there seems unavoidable. Lingering on the dying SmartOS ecosystem, even if it was technically better, wouldn't have served anyone well.
Edit: Ah, as mentioned below, perhaps a joke? I'd attributed the smiley face only to the latter part of the sentence.
Didn't the "Before he got his Netflix special.." or the fact Linux is pretty much the only game in town with no sign of even being seriously challenged served as a hint?
Although at one point I thought Brendon not only abandoned solaris but also BSD.
Netflix has hired more great BSD folk, so I haven't needed to do a lot on it for a while. I do enjoy it though, and it's usually much easier to read and understand the kernel code.
Most recently I've been talking to folk about eBPF on BSD. Maybe I should do a blog post about it (force me to go talk to everyone again and get their updates).
I learned a lot from your books, posts and talks and use your tools (mostly bpftrace and flame graphs for the JVM) on daily basis. I can't thank you enough for the wealth of wisdom you shared with us, mortals!
I devoted a lot of time to learning about systems engineering and modern hardware. When switching jobs it was always with the purpose of getting more and more exposure to performance-focused software engineering.
In my case the last 12 years was was exclusively big banks and one exchange (LMAX, UBS, Goldman Sachs, Morgan Stanley, HSBC). From what I can tell there's more opportunities to get your hands dirty with perf on the buy side (hedge funds, prop trading shops) but I always got lucky with the big market makers and got involved in amazingly interesting things anyway.
I absolutely love this line of work and wouldn't trade it for anything else but it sometimes costs you some grey hair especially if you're dealing with the very bleeding edge of hardware and kernel/OS level bits (think spending weeks chasing misbehaving icache on a heavily overclocked cpu).
This job is very rewarding but requires real passion and constant learning.
I even hoped joining Netflix in the future to say hi to him :)
Doing performance analysis feels like the final thing to fully understand the whole stack and I always was curios on how things work.
Let's see were he will go. Perhaps it's easier to follow but I doubt it.
There are companies I'm following for a while because certain positions are just seldom pop up.
When I joined sap I heard the term headcount for the first time but makes it more understandable.
The principle to do whatever you love is to "ask for forgivness not permission". Feel free to understand why you're solving what you're solving in order to challenge it and do it right.
But unfortunate it was a totally unoptimized thing so only quick low hanging fruits.
Like io bound but they didn't though that component is the issue as it was not CPU bound, bad SQL queries and unoptimized caches.
Nothing big or difficult to analyse :-(
I learned about one person who is only doing java Performance analysis and he said it took a while to be known in the industry good enough that he can do it fulltime.
And he has to travel to allot of customers.
https://www.brendangregg.com/blog/2021-05-29/moving-to-austr...
Both have similar problems: if the business goals don’t align with reducing latency or improved security posture then your detailed analysis will get parked in a ticket queue and promptly forgotten. Your OKRs will be missed. You will also be fighting the tide against new features developed sloppily- as latency will increase and then investment in your team seems like it’s not working to the executives. Fortunately we had a supportive directive but being siloed into an infra org didn’t give us the juice to influence product roadmaps.
We built a lab for testing against devices, did a bunch of work to isolate noise, integrated automated regression testing into CI to combat this. It still didn’t move the needle frankly, as the teams building products were not held accountable for the regressions (and don’t forget that it’s likely the right move for the greater business health!).
Netflix is unique in that latency is very core to their product.
Does some companies give bigger desks for certain staff, or offices, or is it a country thing (Europe?).
Our correspondent teams are much more approachable when it comes to collaborating than I’ve experienced in the past - it’s something I find very refreshing.
Maybe related, but we once went around the Netflix parking lot with Blippar to look at the KBB value of the cars, and found that I was driving the cheapest car at Netflix (at least until we got sick of looking). Mine was a 2001 Toyota Prius, KBB value ~$1800. Not the cheapest car to ever park at Netflix, just the cheapest on the day!
We practiced on baseball diamonds, and played league games on the Google soccer field.
When I ran the BPF tutorial at USENIX LISA 2019, I had one finger in a splint from breaking it playing cricket. :-)
Cheers and thanks for everything.
I'm more shocked at the modesty of your monitor
But I will say the absolutely best I've had was 3 devs to a single office, with desks oriented so that the monitors face the walls for privacy. We could chat throughout the day, but you could also get real work done. I've done the open floor plan thing. Never again. There is a certain constant level of anxiety with having people floating around you and talking nearby all day long.
Add WFH to that list!
I am so grateful somebody’s using FreeBSD in 2022 in production. I do wonder whether there’s anything that could get it more use. Maybe a certification and backing company like red hat does for Linux. Or something more harebrained like a smartphone app geared towards systems management for the upcoming generation. I’ve heard they’re generally less interested in this stuff.
I say this because it’s awesome to see Netflix having people so deep in the tech stack. But somehow still feels a little fragile.
I always have some films downloaded on my iPad Pro that I want to watch when I don't have internet (travelling on long plane flight), and with Netflix always the best films that I want to watch are ,,expired''. It would take 1 second to ,,renew'' them, but I have to wait until the end of the flight to have internet access again, and then I forget it, as I don't look at downloads anymore. It's really frustrating that I see that I have something downloaded, taking space on my SSD, but can't be used when I need it.
I wish it could just renew the video certificates automatically whenever I access the internet, and generally it would be great to have better analysis tools for offline / crappy internet for companies.
Usually I use Youtube more on the plane, because it doesn't play these games with me.
these companies will never learn, they literally invent their own problems as soon as they think they can force their bullshit on us.
Perhaps one advantage for Netflix is that they’re largely a one-trick pony. They don’t have 20+ products fighting for attention and engineers within the company.
So much of the "staff+" engineer world is hot air, and Brendan has always struck a contrast with his straight forward, information-dense blog posts and books, and down-to-earth communication style. He explores, and often defines, the frontiers of what we do and then brings back lessons.
I'm a huge fan, and can't wait to see what he does (and shares) next.