HN Explorer

Ask HN: Has anyone here worked on the Windows kernel?

363 points by maybekerneldev ↗ HN
Or any other kernel, professionally?

I've been offered a position at Microsoft to do kernel development work. This would be a big transition to me, coming from a services background (backend only).

The job's main draw to me is doing low-level work. I did some in my very first job, but for the past 10+ years, due to a number of circumstances, I've been in the services world. I really liked being a C programmer and I've kept an eye on things over the years, and did some hobby projects (on x86 and some embedded stuff as well).

There's a lot about my current job that I treasure, despite the work itself not being interesting to me about 99% of the time. It's a remote job, the work-life balance is stellar, and I get 25 days of vacation a year (this is in the US), which allows me to spend a lot of quality time with my wife.

However, I'm considering leaving because I've been having significant motivation and performance[0] issues for the last two years. Through a lot of soul searching and even help from a therapist, I've identified that the source of my issues is the nature of the work itself. Building services is just something that doesn't give me a sense of accomplishment, and I'm not attracted to the stuff at all. Some issues I've identified are:

1. Infrastructure complexity, especially since moving to Kubernetes. I refuse to touch it at this point.

2. Debugging exclusively via metrics and logs, since I can't just attach a debugger to a running server.

3. Designing systems in general. Some people love the challenge of distributed transactions, eventual consistency and all that jazz, but it just rubs my brain the wrong way. I'm not interested at all in that problem space [1].

4. The insane amount of work required to stand up even the smallest microservice: infrastructure provisioning, certificates, security reviews, GDPR compliance, etc.

5. Anything I build will end up paging some poor soul at 3am some day when something is down or under heavy traffic.

So, what I'm wondering is: what are the things that would make me say "ugh" on the day-to-day as a kernel developer? Is there a chance I'll be happier, or would I just be trading one miserable set of problems for other equally miserable problems?

I tried asking that to every person who interviewed me, but I only got somewhat vague answers like "the build can take a long time depending on what you're doing", etc. Someone complained about windbg.

[0] Even though my reviews have been good, I know deep inside I'm not doing even 10% of the good work I could do before.

[1] Ironically, I've acquired a ton of knowledge about it and I'm one of the "go to" people within my org.

173 comments

[ 2.5 ms ] story [ 205 ms ] thread
layer8
As someone who shares your preferences, you should absolutely go for it I would say, even just for the learning experience. Many devs would metaphorically kill to work on an OS kernel team. If it turns out to not be that great after all, you’ll have no difficulty to return to Kubernetes-land ;), or rather more likely, find a different low-level oriented dev job with that Windows kernel experience.
[dead] beebmam
jnwatson
K8s is like 4-wheel drive. It allows one to get stuck further off the main road.
beebmam
I don't see how the analogy is applicable. To illustrate what I mean, you could have also said: "K8s is like a train. It's big and heavy, will carry most things that you need it to", and it would be just as much of a non sequitur. Analogies are usually worthless to an argument (and therefore should be worthless to your opinions), unless one is able to demonstrate why thinking about an idea with an analogy is applicable.

There's zero reason to think that Kubernetes is like a 4-wheel driven car for the same reason to think that Kubernetes is like a train, or Kubernetes is like a virus, or Kubernetes is like a snowman; it's completely unrelated.

MAGZine
What pendantry is this? The only thing that you could take away from OP's comment is that they made a metaphor, and since you can make a metaphor about anything, the chosen metaphor says nothing?

That's like saying, well, you can use words to construct a lie, therefore, all words are untrustworthy.

He's saying that k8s can be powerful, but that that power can be destructive if not wielded correctly.

Personally? k8s is powerful. But you need a team to manage it, and if you're the poor sob who wants to write software but gets stuck managing k8s (due to its complexity) half the time, you're not going to be very happy.

Aloha
I think you're right.

K8's (and other similar tools) allow you to push complexity into places that are harder to troubleshoot. Its another layer of abstraction, and too much abstraction is dangerous in complex systems.

beebmam
That's not what I'm saying.

I said that saying K8s is like X requires that you show how it is like X first, before you can draw any conclusions from that analogy. Otherwise, it is not an argument and should have zero value in a conversation, other than someone stating their (unsupported) opinion.

maxbond
I think what you're missing is that they didn't give you an argument, they told you how they felt about Kubernetes. And that's a totally reasonable thing for them to do. You aren't owed an argument. But if you had asked politely you probably would've gotten one.

When it comes to political discussions, I think it is reasonable to demand that everyone who chooses to participate substantiate their claims and make arguments rather than stating opinions. Misinformation about politics is serious, it can be a matter of life and death.

That standard does not apply to talking smack about Kubernetes on the internet. I can tell you that programming in Python feels like a game of operation and that programming in Rust feels like a breath of fresh air. I don't have to tell you why. You shouldn't try to argue against this, either - it isn't something I can possibly be wrong about, or that you could possibly understand better than I. You can tell me about how you feel like Rust is overhyped garbage (I have no idea how you feel about Rust, it's only an example). But why would you try to tell me how I feel?

What is the consequence if my statement goes unchallenged? Someone tries to learn Rust and is disappointed that it doesn't match the hype? Some engineer starts a new project in Go instead of Python because they don't want to play operation? Life goes on.

If you look at it through that lens, you can make sense of it. They're not saying there's a relevant similarity between Kubernetes and vehicles. They're saying that with more complex tools you make more complex mistakes, that Kubernetes may have solved problems but it also gave us more rope to hang ourselves. And I bet you'd have something interesting and productive to say in response to that idea - you obviously have passionate opinions on this subject.

I'd also note, no one ever said not to use Kubernetes. The general vibe was that Kubernetes is stressful to work with. Again, no one is making an argument - they're venting to other engineers.

beebmam
>I think what you're missing is that they didn't give you an argument

That's fine, and I'm glad that people are admitting that their comment shouldn't be taken as persuasive, or anything other than self-expression. Many people think analogies are persuasive, but they're often nothing more than illogical rhetoric, and it's a critical blunder to be persuaded by an unjustified analogy.

mickael-kerjean
> But you need a team to manage it,

I do run a kubernetes cluster outside work as a side project and it hardly takes me more than 2 hours per month. It mostly run cloud instance of my open source software for customers and is saving a lot of time compared to the manual process of creating dns manually, rp rules for everyone, handling of SSL, monitoring, ... Just to think about the work that would be required to get all this done outside kubernetes makes me sweat and the maintenance aspect of such a solution would make it even worse.

goodpoint
> side project ... 2 hours per month

That's your error: you are evaluating k8s on a use-case that is not representative.

Once you scale up the number of nodes, the number of applications, and the years of uptime the complexity grows exponentially.

Now your team needs to be able to debug each component in the OS stack as well as k8s itself.

In the long run you always pay the price for unnecessary complexity - and with interests.

inkeddeveloper
This rant is completely unrelated and definitely not wanted.
nicce
You can write many books from Kubernetes and still not know it perfectly.
naikrovek
I’ve written entire software packages which do what I need, exactly, rather than try to get Kubernetes to behave, and I am much happier for it.
remflight
Why wouldn’t you take a chance, given how much you dislike what you’re going right now? What have you got to lose?
cschep
Kinda seems like if you hate it you could go back to building services really easily. No real downside to giving it a shot! Unless I'm missing another trap door it feels very reversible. :)

You might even be able to return to your old company if you hate it. Tell them directly "hey I need to give this a shot but I'd love to check-in in a year".

drewcoo
The design is very different than *nix. The entire ecosystem is different. I think experience with both is eye-opening. There may even be a couple of grizzled old graybeards left who followed DaveC over from DEC. I don't have a clue now.

If you can read and enjoy these books, consider it:

https://docs.microsoft.com/en-us/sysinternals/resources/wind...

The ugh!: You will be a tiny cog in a big machine, spending very little time writing code, mostly banging your head over someone else's bugs. And M$FT still has a certain stink to it these days and that may cling to you.

Firmwarrior
Microsoft is a friggin dumpster fire, dude. Some teams are great to work for, but a LOT of them are little crews of treacherous rogues who're only looking out for themselves and always ready to stab someone in the back.

You'll run into a lot of the same issues on the Windows kernel that you're getting irritated with in service development land. A lot of infrastructure is already out there, but figuring out how to use it effectively ends up being almost as much work as just rewriting it from scratch. Things are documented but poorly.

For Microsoft specifically you'll often get stuck because of problems with some other team's code, and then find yourself embroiled in multi-week battles with multiple engineers, managers, project managers, and product managers all at each other's throats.

All that said, the Windows kernel itself is a work of art IMO, and kernel development is a lot of fun. I think that if you want to make the big bucks but still work with c/c++, it's getting to the point where your choices are very limited: Microsoft, Google, Facebook, Apple, or Tableau.

Microsoft was terrible to work for, but they aren't as bad as a bad small company. At least there's HR to report gross violations to, they have a TON of great perks, the money is really good, they have good work-life balance, and it's not too hard to change teams (but if you find yourself wanting to jump to a different team, do it well before performance reviews come along, since your manager will definitely give you a bad review and hamstring your ability to move around within the company.. so you'll end up having to change companies. Not the worst thing if you do so, I guess)

EDIT: I just reread your list of complaints.. timing and consistency in the kernel are WAY more complicated than your average web services. There's also a ludicrous amount of red tape around getting anything done. I'm starting to think that you should look at Microsoft as a stepping stone in your career rather than its final destination.. it's still a good place to work for overall compared to random small companies, it'll give a boost to your resume, but it sounds like what you actually want to work on is game emulators or raspi home automation gizmos in your free time.

greggsy
You basically described any workplace in a large company. Organisations are systems made of loosely interconnected and dependent systems, made up of people. The complexity of business norms, personality and cultural differences, and unrealistic expectations around business processes can be difficult to reign in inside a global company. Some workplaces have worked out a sweet spot, balancing people, HR, tools, culture and purpose.

I’m not telling you or anyone else in these companies to suck it up, but there’s a point where employees and teams in these situations need to step back and realise what they’re dealing with, and adjust their mindset accordingly.

Personally, I like identifying and unfucking bottlenecks. Sometimes explaining why something is so fucked to the people being fucked can go a long way.

notJim
I really disagree with this idea, we shouldn't normalize every workplace being full of back-stabbers and retaliatory jerk managers. Nor is horrible red tape a given, although some bureaucracy is of course required. None of this has been my experience at fairly large, well-known companies, although I'm definitely aware of cases where it has happened to friends.
retcon
Is enabling the same as normalization? Surely normalization is acceptance into culture whereas even in dysfunctional organizations the individual actors all think that they're solving problems and not taking any shit.

The pity is often how the most qualified talent for resolving poor behaviour is so often subordinated into slopping out the pigsty. There needs to be a foundational next step.

Edit: since Fairchild we've been scared of merely assembling raw talent under a thin layer of management capabilities and from this is developed the pseudo matrix management system that e.g. Microsoft operates.

greggsy
It’s not an idea - it’s just a perspective that some people forget: the world is volatile, uncertain, complex and ambiguous, and our organisations are a reflection of that. I know that it’s fashionable to talk about not normalising things, but the reality doesn’t change.

I don’t like it either, but I can’t really do much about it. If you realise you’re in a difficult organisation, you can at least start to take action, or take a different tact (or leave). Being annoyed about the situation is probably just going to make you more and more unhappy because the system won’t (or can’t) change in your desired timeframe.

Firmwarrior
Microsoft CEO Satya Nadella wrote a book right around the time I was planning to leave, and gave everyone a free copy. The book really perfectly nailed what a toxic dumpster fire the company had become, and then it outlined his plan to fix it.

What's funny is that the guy agreed with everything I thought, he saw the problems that were hamstringing me (and everyone else), he had complete authority to fix the problems.. and so far as I can tell, he wasn't having any luck with it

The best fix seems to be to tell people to quit being jerks to each other and focus on making good products and making customers happy, but good luck getting that to happen.

One of the main criteria for performance reviews was suddenly something along the lines of "Teamwork" that captured what a non-toxic, non-Ballmerized good person you were. Of course, all that happened is that the best backstabbers and the weaseliest scumbags manipulated their way into high "Teamwork" scores, and anyone who sat down and focused on productive work/helping others got screwed on that score just like everything else.

Firmwarrior
I’ve been at other big companies where it wasn’t the case. The engineers were all there to get a job done and were willing to help other teams when necessary to make that happen, without lengthy negotiations ahead of time

There’s going to be SOME of that stuff any time you have more than one person in the same building, but a decade under the iron fist of Ballmer has permanently ruined Microsoft as an organization.

jiggawatts
> think that if you want to make the big bucks but still work with c/c++

Judging by the presenters at CPPCon, a significant fraction of highly-skilled and highly-paid C++ devs are employed in the High Frequency Trading (HFT) industry.

They meet OP's criteria in many ways as well, as HFT trading tends to be "very close to the metal", optimised to death, and often involves solving deep technical problems. It's diametrically opposed to modern web development practices.

int_19h
The downside of working on HFT is that the value one provides to society is at best nil, and some would argue that it's negative. Some people care about things like that.
Firmwarrior
I’ve been thinking about getting into HFT.. they’re always bragging about operating on millisecond/microsecond timescales, but they’re doing it on bespoke heavily-monitored systems. Meanwhile here I am shipping the same type of code to millions of people on commodity hardware..

I’m pretty sure that you could donate a big chunk of the million+ ill-gotten dollars to charity and make a lot of good things happen

chasil
The original kernel from which Windows evolved was written in assembler by Dave Cutler for the VAX architecture. After his DECWest team was hired by Microsoft, they reimplemented it in C. The MIPS architecture was the first fully complete port, I believe.

This is documented in Zachary's Showstoppers book:

https://www.flyingpigbooks.com/book/9780759285781

It appears that performance and architecture improvements are very hard to get accepted (and the kernel suffers for it):

https://blog.zorinaq.com/i-contribute-to-the-windows-kernel-...

It is a great piece of engineering, but it is likely for a developer that is more comfortable making small changes within the hierarchy than for any revolutionary ideas (unfortunately).

If Cutler hadn't found his way to Microsoft, then they probably would have ended up on a BSD kernel, as Apple did. As it was, Microsoft sold their Xenix business around the time of Cutler's arrival.

carabiner
Ah, so it is true: https://www.androidpolice.com/wp-content/uploads/2021/07/08/...
Firmwarrior
Man, it is so true, or at least was when I left in 2017

It’s a fractal shape too. You see the same sort of intrapersonal weaponry at the division level, the org level, the group level, and even within teams

Maybe it even continues to a Herman’s Head-style inner conflict for good company men

pjmlp
You forgot NVidia and Adobe, two big hires of well known names in the C++ community and ISO C++ working groups.
Firmwarrior
That’s a good point, I’ve heard good things about NVidia

Presumably Tesla and Uber must have some “native” code being written somewhere in-house too, and someone mentioned the HFT industry

There are a lot of decent jobs out there that will keep you comfortable, too, even if you can’t bring in 300k+.

waynesonfire
i think it's more impressive you were able to land a job in such a vastly different ecosystem. what's your trick against being pigeon-holed?
KerrAvon
If MS or any FAANG wants to hire you for a kernel dev position, go for it; they clearly see a lot of potential in you that maybe you don't see yourself. I want to point out a couple of things that you should be prepared for:

> 2. Debugging exclusively via metrics and logs, since I can't just attach a debugger to a running server.

You often can't do this in kernel/OS development work either. Serial printf logging is often required. It can be a real "my tools to debug my tools are broken" slog.

> 3. Designing systems in general. Some people love the challenge of distributed transactions, eventual consistency and all that jazz, but it just rubs my brain the wrong way. I'm not interested at all in that problem space [1].

I'm not sure I understand this issue. I mean, kernels have subsystems for doing stuff; you might need to design one someday? But it won't be dull-as-dishwater web technology stacks, it'll be you writing data structures directly in C/C++ or Rust if MS goes there.

ccrush
Can you no longer use windbg to attach to a running kernel on a remote box, and debug the remote box from your development machine? I believe that's how kernel debugging is generally done.
ace2358
That’s probably true, but sometimes you have to go one level deeper right? What happens when your debugger has crashed?
Arainach
In 8 years on the Windows team I can't recall the debugger ever crashing. If it does, you open the debugger again and reconnect to the target machine.
hyperrail
Heh, I wish that had ever been the case for me. Sometimes it feels like WinDbg freezes every other command. Plus, getting it to reconnect to a kd after it freezes and I kill it was such an exercise in frustration that if the target machine were my own repro box instead of a stress break remote, I would have just hard-reset the machine or (if it were a VM) restored it from snapshot.
amluto
(I’m a Linux kernel dev)

Tools like windbg probably work great until you are debugging the code that runs when windbg tries to take over. Or you’re debugging something sensitive to interrupts and it’s literally impossible to keep up with a tool like windbg. Or you a debugging something that overwrites windbg in memory because you have a corrupt pointer. Or you triple-fault the machine and it reboots with so much prejudice that windbg doesn’t have a chance. Or you’re poking at hardware that the debugger can’t usefully interact with. Etc.

Basically, in the kernel, a lot of the things that a working kernel does to hold your hand aren’t available. So there’s a lot of time spent just thinking or adding logging statements or otherwise using low tech tools. (Or using high tech instrumentation or sanitizers!)

Kernel development is great :)

muricula
I have debugged the kernels of Windows, Linux and now iOS using internal tools. Windbg generally works, and is far superior to the equivalent tools for other kernels. Like vim it has a baroque syntax and a steep learning curve, but it also has a gui with big buttons to compensate. Yeah, you can get yourself in bad situations, especially in early boot, but I miss it.
amluto
I confess that it’s been years since I used WinDbg. It was quite bare bones back then.

These days I mostly use gdb (which is incredibly buggy for such things) attached to QEMU’s gdbserver. It works except when I doesn’t.

DaiPlusPlus
I thought everyone doing OS/kernel dev would be running code in a VM they can pause, snapshot, and single-step all without a guest debugger?
xmodem
Virtual machines are used extensively by kernel developers but are not an adequate substitute for real hardware in a lot of situations. After all, one of a kernel's jobs is to abstract away differences in underlying hardware.
surajrmal
Having worked on both windows and Linux kernel development, I can easily say that Windows has superior debugging facilities. It's pretty rare to make the debugger fall over. Working in the kernel is certainly not without its own challenges, but I feel like Linux makes it harder than it needs to be.
shultays 

  Serial printf logging is often required
This reminded me using "printk" in an incorrect place in an operating system course assignment. It was actually printk itself that was hanging the kernel!
maybekerneldev
> > 3. Designing systems in general. Some people love the challenge of distributed transactions, eventual consistency and all that jazz, but it just rubs my brain the wrong way. I'm not interested at all in that problem space [1].

>I'm not sure I understand this issue. I mean, kernels have subsystems for doing stuff; you might need to design one someday? But it won't be dull-as-dishwater web technology stacks, it'll be you writing data structures directly in C/C++ or Rust if MS goes there.

Ah, I meant designing distributed systems. Like, figuring out the difference services, storage requirements, databases, then planning out the infrastructure. It's really not my cup of tea. I do enjoy designing systems/subsystems/components as long as they're within a single host :)

booboofixer
I'm way less experienced than you are but sort of started out closer to the kernel world (work involves using C mostly) than the webdev world. It's fun if you're interested in that sort of stuff. The things that mostly make you go "ugh" would be testing your work (can't let the kernel slip up, quality is important), the processes involved at some companies in committing something as simple as one line of code, and the lower upper-bar for salary. Maybe Microsoft is a bit wealthier than where I work though.
cglan
Definitely do not join Microsoft and definitely do not join any windows related teams. Microsoft is hell. Every team is fully siloed. Culture wise it’s a wasteland and teams as a communication platform is awful
darknavi
Hard disagree about the "Hell" portion. Siloed, probably but really dependent on the team/org.

The culture in games is very healthy from my experience and one of the best compensation in the games industry.

Someone1234
Do you care to contextualize this? Did you work for Microsoft and or on Windows? For how long?
cglan
Sure. I've worked at Microsoft primarily, and I'm not speaking for all teams but I have a sibling on another team in another org and he has the same issues

Microsoft is incredibly siloed. Each team is essentially their own mini company and they're mainly guided by large top line metrics but there's no top down overall vision on what something should look like. It's like those party games where everyone has to draw a portion of a drawing. It comes out looking like a disaster even if every individual portion is good. This also means that product management doesn't work with any particular team either. You essentially get some random big metric and are told "make this metric better" without any context. Everyone is duplicating work and there's nowhere to learn from

Microsoft Teams is a disaster in so many ways. In the most obvious way, it's very very slow and a pain to use. This subtly hinders teamwork because no one wants to use teams. In other ways, there's no global search so finding stuff in other orgs is impossible. The teams "channels" are essentially shitty forums that are unintuitive to use. You'll never get a channel about hobbies and stuff and even if you did it's hard to find and they're usually dead. Everyone uses private group chats including each team but these have zero discoverability. What this means is you'll never have a golang channel or something where people share and chat about stuff. Most people have a facebook group where they chat about stuff (wild).

Every team has their own onboarding down to what hardware you should get. In theory this allows for some flexibility but what this actually means is that no one has any idea how you should be onboarded and you essentially are sent to flounder until you pick stuff up.

Everyone is doing everything. Every person is their own product manager, scrum master, manager, and also programmer. There's so much duplicated process overhead it's wild.

They have not handled remote well. They insisted on trying to send me a desktop computer. I asked for a laptop and they couldn't give it to me and they instead sent me a used intern laptop. They gave my sibling a used surface tablet. This is a 2 trillion dollar company and they're unwilling to shell out 2k for a basic workstation computer with 6-8 cores and 32gb of ram. Not a huge ask. Also some stuff is only accessible through a direct hardline in the office. Whether you want to use a desktop or not is irrelevant. It's mainly how cheap they are when it comes to hardware.

EVERYTHING has to be Microsoft software for the most part. If you think nih syndrome is bad at your company, imagine you're at a company where they've been making mostly mediocre versions of other software for the past 30 years. Yeah. I'm not a huge splunk fan but trust me when I say the azure equivalent is much shittier.

The pay isn't top notch. In fact it's pretty bottom barrel for a big company and if you can pass the microsoft interviews you can pass somewhere else. Their interview process is also a nightmare. I went through 4 different recruiters and it took me 2 months between passing to get an offer letter. The whole thing was insane. They initially offered me such a paltry amount it made me laugh.

Everyone is a lifer because anyone else has left. Imagine talking to your boss about docker or talking to him about IntelliJ and he's never heard of it because he's been at Microsoft for 20 years.

There's a lot of weird "not racism" but might as well be where certain ethnic groups have taken over certain orgs and speak in their primarily mother tongue despite being in the US in a US based company. It makes teammwork really hard.

You need a separate laptop to log into any production resource. Production resource is a loose term because that also includes int environments and anything on azure. I have 2 laptops and a desktop that I'm forced to remote into it's ins...

SBArbeit
Sitting on the Microsoft campus at the moment... not a kernel developer. <looks around> yeah, this is not hell. Not even a bit.

Microsoft is great. It's an amazing place to make impacts in many different areas of technology. Work teams used to be more siloed, but that's changed a lot in the last 6-7 years. We regularly work across teams at GitHub and Microsoft to get things done with great cooperation.

As for Teams vs. Slack vs. Discord, it's a personal taste thing. I'd rather have Teams over Slack + Zoom 1000x.

Take the Windows kernel job! Worst case, it's not a fit and you move on, like any other job. Best case, you stick around for a long time and have a great career working on tech that 1B+ people use.

When Robert Downey Jr. was trying to talk Gwyneth Paltrow into joining "Iron Man", he said to her something like, "do you want to work on art house films the rest of your life or do you want to be in something that people actually see?"

Windows is massive... it's the largest code base that I'm aware of. It has a ton of process and procedures and test cycles because... it's Windows. Of course you can't just freelance there. But you can "be in something that people actually see", and most SV startups will never even get 1,000 users, much less 1,000,000,000+, and more if you count the users on Azure services using Windows indirectly.

Mandatum
> When Robert Downey Jr. was trying to talk Gwyneth Paltrow into joining "Iron Man", he said to her something like, "do you want to work on art house films the rest of your life or do you want to be in something that people actually see?"

Sounds like a bastardization of Jobs on convicing John Sculley manufactured by a PR person:

Steve Jobs and John Sculley, then PepsiCo president, were sitting on a balcony overlooking New York’s Central Park. Jobs turned to Sculley and said, “Do you want to sell sugar water for the rest of your life or come with me and change the world?”

https://www.forbes.com/sites/carminegallo/2016/11/12/how-ste...

That's like asking someone who cooks at a fine-dining restaurant to work at McDonalds, "do you want to make pretentious food for 40 cashed up foodies and critics a day, or do you want to make food for hundreds of regular people a day?"

People pretending that Facebook, Google, Microsoft, et al are noble endevours are deranged. It's like rooting for Walmart to win the World Series of Retail.

philwelch
McDonalds corporate actually employs chefs to develop new menu items, and a large part of the process is figuring out how to work within the constraints of being able to consistently reproduce a menu item at thousands of locations, sourcing the necessary ingredients via the McD supply chain. I'm sure it's an interesting problem and a chef working at a fine-dining restaurant may very well find it a nice change of pace compared to feeding a small number of rich, pretentious jerks.
dekhn
I worked at McDonalds some time ago. New managers were sent to "hamburger university" in illinois and come back talking like robots. https://en.wikipedia.org/wiki/Hamburger_University
marktangotango
Say what you will, but McDonalds is pretty remarkable in their consistency, even internationally. A big mac in New York is the same a Big Mac in San Francisco, as in the Midwest, as in Germany.

A pet peeve of mine is pizza joints who serve a different pie every time, depending on who's in the kitchen. If I have a great pizza, I expect to get it again, but rarely do. Consistency is hard.

jdsully
When I was on the Excel team the kernel devs I talked to knew their stuff and were of the highest quality. More importantly they cared and stuck around to see the consequences of their actions.

The windows org has a reputation of being dysfunctional although I never personally observed this, and personally think even if it were true it would be in the higher level parts of the stack.

Kernel dev was my original dream but I fell in love with Excel during my internship. I definitely think you should go for it.

fassssst
I work on Windows (not the kernel) and the best part is that you can work on critical infrastructure without having to be on call. Work life balance is great.
itisme
The answers you will get here at HN will mostly be misplaced hatred against Microsoft, from people who hate MS, because someone else here at HN hates MS.

I think what you should do is, if possible, arrange for a short call with some of the team mates that you are going to work with. Just a casual one about how the kernel team day to day work looks like there. If not with the team, then maybe with your manager.

Unless someone from that team or some ex-kernel team member replies here, all the rest of these posts are just speculative and not worth basing your decision on.

tpmx
I guess you're probably thinking about someone like me - someone who got started in 90s, or perhaps even earlier.

I don't hate the competent developers at Microsoft (of course they exist, there have been source code leaks) - I just very strongly dislike the machine that is Microsoft. I don't think the machine has reformed.

localhost
What level (title is fine)? What team? I can try to help point you at some folks in Windows though it's been a while since I worked in that org. You can reach me via contact info in my profile if you don't mind decloaking.
oldmanhorton
I've worked adjacent to windows for a few years, and it's not perfect. They have mountains of legacy requirements by the very nature of being windows and other comments about siloed culture can often be true, though I think this is true in most large companies. With that being said, windows kernel developers are by and large extremely skilled and passionate, and working on that team is pretty unique within most of software engineering - rarely do you have the ability to impact so many people.

The concern for me would be the fact that windows is no longer a fully offline experience running on people's desks. A large part of the kernel team is working on features that only benefit Azure and you may get some of the same services exposure there as you claim to be so burnt out on.

Asking to discuss the role more with the team to get a better sense of what they're doing, how they're doing it, and who their customers are seems like the path forward IMO.

jiggawatts
> A large part of the kernel team is working on features that only benefit Azure

And the IIS team, and the networking team, and…

keithnz
I once interviewed a guy who worked on various parts of the core windows, naturally being curious I asked a lot of questions about it. I think it would be an interesting experience, but ultimately it seemed very constrained in terms of what you can do, decision making is quite constrained etc. It sounded like you would end up very very focused. Which might be your thing. The guy I interviewed was wanting a lot more freedom. In the embedded world, there's lots of opportunities for low level programming.
jvert
I spent about ten years on the Windows Kernel team. In the 90's, so likely very different than what you would experience today, but probably still a lot of the same problems. (Funny to hear people are still complaining about windbg!)

I think you are coming at this from the wrong perspective. Rather than thinking about how to avoid work you DON'T like, think about what you DO like and then decide if the new job would offer more or less of that.

Personally, I've found that every five years I end up sick of working on the same kind of problem and I have to go work on something completely different. Maybe that's where you're at.

jackling
Don't you find it difficult to transition from one domain to another?
syschick
Disclaimer: I worked at MSFT for 14 years, but that was >10 years ago, in Servers (Exchange, ISS) not kernel. But I've debugged windows stresslab crashes across the network, before windbg got good :)

Is it difficult to transition? Yes, in that if you spend 10 years learning SystemA, then it will take some time with SystemB to build up to the _same level of expertise_ that you enjoyed on SystemA.

The neat thing about learning one OS or system _deeply_ is that the deep knowledge gifts you with frameworks for learning the next-system.

I wrote software on various *nix'es during my university time, then worked on OS/2 (yes, "OS-who?"), Windows, and now I'm at Google, (my work involves 80% Linux 20% Win). It took me some time to re-build my skills on each environment/platform, but it's been a great ride and a lot of fun.

(P.S. earhart@ pointed me to this thread, and says hi to all y'all: Evan, JonO, JVert, LandyW? Dan?)

heleninboodler
The most growth (and fun, in my opinion) is experienced when you hop to a new discipline where you have no idea what you're doing.

Source: 33 years experience hopping all over the spectrum

jvert
I like to learn new things; I find it energizing, especially when I am feeling burned out. Sure, it can be difficult but that is also what makes it rewarding. It's also really interesting to see the different tradeoffs that have been made between different solutions. (Linux vs Windows for example)

Ultimately, it's all just code and problem solving. And you can usually find a way to leverage your expertise in one domain into a different domain.

tarun_anand
I worked in the kernel team many moons ago helping improve the performance and then built the .net clr later. Also worked on a NT Filesystem called Cairo. So I guess I have experienced both kernel dev and backend framework dev.

Kernel code is amazing, especially the parts written by the early members like DaveC, MarkZ and others.

For me the biggest part was working with a group of extremely smart people who were very nearly the best programmers in the world.

I really miss that outside of Microsoft. I would imagine that you can get the same experience if you worked with the Linux kernel dev team or some of the other few places in the world like FAMGA where you can work.

My suggestion is to go for it. After leaving MS I found peace by working with the open source community and open source software.

Hope that helps, happy to discuss further.

beefman
I would love to hear any Cairo stories you may be able to share!
jacobsenscott
I've managed to just to full stack rails apps for years so I've been avoiding most of those issues. But I feel you on the GDPR, and whatever other regulation is there or coming down the pipeline. It really sucks the fun out of programming. I used to be a systems programmer, and was just thinking the other day that would probably be more fun then building web apps these days.
jsolson
I think I can count the number of kernel changes I've submitted on one hand, but I work on core virtualization that involves a lot of pretending to be hardware and (these days) a lot of poking directly at hardware registers.

I would say James Mickens sums things up nicely in "The Night Watch[0]." For example, you mention debugging with logs and metrics -- this snippet came to mind:

     “Yeah, that sounds bad. Have you checked the log files for
     errors?” I said, “Indeed, I would do that if I hadn’t broken every
     component that a logging system needs to log data. I have a
     network file system, and I have broken the network, and I have
     broken the file system, and my machines crash when I make
     eye contact with them. I HAVE NO TOOLS BECAUSE I’VE
     DESTROYED MY TOOLS WITH MY TOOLS. My only logging
     option is to hire monks to transcribe the subjective experience
     of watching my machines die as I weep tears of blood.”
Mind you, I absolutely _love_ working on low-level stuff, and I wouldn't trade the time I get to spend actually doing that for anything. That said, the complexity of modern operating systems, CPU architectures, interconnects, and peripherals creates opportunities for frustration and confusion that honor no bounds of reasonability or decency.

[0]: https://www.usenix.org/system/files/1311_05-08_mickens.pdf

dtgriscom
> I think I can count the number of kernel changes I've submitted on one hand

I know I can; I don't even need any of my fingers.

tomrod
This is breathtakingly brilliant, and I am grateful to now have the wisdomful humor of James Mickens in my life. Thanks!
EvanAnderson
As an aside: James Mickens is a treasure. If anybody reading this hasn't read any of his articles or watched any of his talks before stop now and do it. You won't be sorry.

https://mickens.seas.harvard.edu/wisdom-james-mickens

retcon
Reading this man's homepage he comes across as a insufferable egotist. I couldn't find any trace of didactic value among the self aggrandizing rhetoric not possible to convey with infinitely greater humility and persuasion as well as concision.

Edit: reference source decades of being quite intolerable myself. Takes one to know one. And by my experience incredible good fortune and more decades to repent.

vlmutolo
The page is supposed to read as a joke. That's his schtick. I think his writing is pretty funny.
teekert
Haha, I love this humor, it's the kind I write in a frenzy of enthusiasm and inspiration often fueled by a sudden realization of the crazy state some things are in.

I also have gotten a lot slack for it. But I see it as a way to convey that new feeling of wonder and awe a child has upon entering a new world. Challenge the assumptions, make up some nice words, pretend very hard that your academic title is an actual thing that gives you some blessed superpowers (as is the way that people often treat it!). I love it. It's truth wrapped in humor to soften the blow to the ones that think themselves overly important. The "shocked ones" are the ones that I try to avoid in life anyway, nice discussions begin at the edge of your comfort zone.

RagnarD
The page is extremely obviously full of satire.
sodapopcan
I think even with the edit that this comment is also satire.
whimsicalism
Having interacted with Mickens quite a bit, he is far from intolerable.
x0n
Dude. How can you read "I’ve been a legendary hacker for 98% of my life, but there was a brief period when I did not possess the sum totality of human knowledge." and not realize this is self-parody?
sodapopcan
I did as you suggested. I was not sorry. Thank you!
CodeWriter23
So what you do in that case, add a monochrome adapter to the PC and write to the screen memory. Or send [greatly abbreviated] trace out a serial port.
jsolson
In at least one instance of breaking my tools with my tools, the machines in question were in a data-center. Ordinarily one might be able to grab debugging output over an NC-SI link, but that only works if the breakage doesn't also unexpectedly crash your NIC firmware...
CodeWriter23
If you can’t touch it, you might not be able to debug it. Kudos though for testing^H^H^H^H^H^H^Hdebugging in production.
CodeWriter23
Case in point. Had a client in the late 80’s with an X.25 connection between their office and a Mainframe across town. It was accumulating hundreds of CRC errors per minute. Couldn’t fix remotely so I flew out there. Had a scope hooked up on both sides.

We loaded the connection in varying patterns for about an hour. ZERO CRC errors. Scratching our heads, we decided to punt. Then suddenly the CRC error count started climbing on my scope. I shout into the speakerphone, excitedly, what did you all just do? They’re like “nothing but unplug the scope from the patch panel”. “Please plug it back in”. CRC error count stopped.

Diagnosis: faulty patch panel.

jsolson
When your job is to build infrastructure, "in a data-center" does not necessarily mean it's a production server :)
cperciva
poking directly at hardware registers

Such luxury! I just spent a couple weeks getting FreeBSD booting in the Firecracker VM and most of my debugging was performed by inserting hlt instructions into the FreeBSD kernel and looking at whether virtual CPU halted or hit a triple fault.

jsolson
I take it the Firecrakcer folks haven't built out support for KVM_GUESTDBG_SINGLESTEP yet :)
aasasd
> whether virtual CPU halted or hit a triple fault

Ah, I see the gentleman works directly in binary.

saghm
> I HAVE NO TOOLS BECAUSE I’VE DESTROYED MY TOOLS WITH MY TOOLS

This might be my all-time favorite quote that I never to get use in relevant situations because nobody is around who would get the reference. I think of it almost every time I hear the word "tool"

[deleted]
(comment deleted)
darepublic
> I HAVE NO TOOLS BECAUSE I'VE DESTROYED MY TOOLS WITH MY TOOLS

Only comparable experience I can think of is "breaking" the terminal via bash_profile and then not being able to fix bash_profile via the terminal. Or locking yourself out of a web server via security configuration update.

philistine
I’m an enormous idiot who’s done that recently. If not for the fact I had another window open, I don’t know what I would have done.
thowaway_pandat
chroot from an OS whose terminal is not broken and a shell other than bash
philistine
zsh was the one I broke, so bash would have been preferable.
oblio
Raise your hand if you broke the SSHD server config and then restarted it!

That: uh-oh! moment.

zrobotics
Definitely didn't do that last week, no way. Thankfully, the server I didn't push the wrong credentials to before restarting wasn't the new instance of the test server.
doubled112
I've definitely never done that via Ansible on a bunch of machines at once, nope nope nope.
TimSchumann
Thank you for posting that. Epic read, amazing sense of humor, and I feel like I owe James a beer or three after reading it.
ineedasername
If you're miserable in your job now then switching seems like there's lots of upside and only a little downside (it could potentially be worse than current).

The main thing I'd consider in this is the work/life balance. Will it allow the same stellar level of that? People vary in their priorities. For some it's interesting/fun problems to solve, for some it's impact, but for me it's work/life balance, presuming of course that the work side isn't a hateful stressfest. No amount of remote work or flex schedule will make up for absolutely hellish work environment, and I had something like that once. (I ended up taking a 30% pay cut that I could just barely afford and got the heck out)

So I'm sorry that I can't speak to the kernel work itself, but instead the framework for making the decision. If the work/life balance will remain constant then you're only risking the possibility that the work won't be any better, and hopefully no worse. If the work/life balance for the new job is also uncertain then (for me) that would be a bigger risk consideration.

Finally, you should consider the worst case scenario: The work is worse, the work/life balance is also worse. How easily can you shift to something better? If you have the chops to work on the Windows kernel then I'd wager that if you hit the worst case scenario you could get out of it without too much trouble, but you're the only one who knows whether or not that's true.

Best of luck to you, sláinte, and may the wind be always at your back.

ForgottenEnmity
Is work/life balance a US thing? In my country, for example, the employer can't easily make you work for longer than 8 hours per day, and even then there are big big compensations like extra payment with time off, restrictions on how much overtime per year is allowed and so on. So when people speak of a bad life/work balance, how does that happen in the first place?
hyperrail
Salaried office workers receive far poorer wage and hour protections in the U.S. than you may be used to. Overtime above 40 hours per week is often unpaid, as is the case at Microsoft, and no amount of paid vacation is guaranteed by law. (Workers paid hourly do get overtime pay at 1.5 times the normal wage or higher.)

Microsoft's work/life balance is good in my experience; managers respect your time, product deadlines rarely or never cause long-hour rushes, and if your role lacks an on-call rotation then you won't be woken up in the middle of the night. But other companies may be far worse, and Microsoft is big enough that one person's experience never truly generalizes.

For time off in particular, as of my last experience 2 years ago, MSFT's US policy on paid time off was stingy by American tech company standards:

* There is no scheduled period of the year where everyone gets paid time off. (LinkedIn has 2 such weeks a year but it manages its HR independently of the rest of Microsoft.)

* You accumulate 3 weeks of vacation PTO per year of work for your first 6 years at the company, then 4 weeks per year for years 7-12, then 5 weeks per year for years 13+.

* Unused vacation days expire at the end of the calendar year following the calendar year in which you accumulated them.

* When you leave the company, you get a "vacation cash-out:" you are paid a lump sum equal to all your unused, unexpired vacation days.

* In addition to vacation, each calendar year you get 2 weeks of paid sick leave and 2 "floating holidays" to use like vacation days.

For comparison, at Google you accumulate 4 weeks of vacation per year starting just in year 2, while at Netflix and many smaller companies you get unlimited vacation at the cost of no cash-out.

In practice, there is more leeway at Microsoft on PTO than this policy allows. Even when I had an otherwise poor relationship with my manager, they still looked the other way when I went on Christmas/New Year's week vacation without filing an absence request or using up my vacation days. And I have heard of divisions where after a particularly stressful and overtime-heavy year, the division Vice President told everyone to take such an off-the-books holiday vacation.

willcipriano
Man I envy you, that sort of thing is the dream. If you don't try, it will always be a what if. The services side of things will always be there if you change your mind.
Ombudsman
Bad stuff:

1. There will be a lot of infrastructure complexity in the kernel, just prepare yourself for that. Even worse bugs! You'll be fixing a lot of bugs, or looking at a lot of bugs, and most of these bugs are from other teams who are interacting with your component! Just order a copy of Windows Internals and get yourself familiar with how thing work.

2. Old ass engineering systems. Just as the interviewers said you will spend a lot of your time waiting for Windows builds.

Good stuff:

1. Work life balance is amazing actually. Most of the time there's very little pressure for you to get work done, as long as you're doing something no one really bothers you.

2. Since you said you hate designing systems, good news, everything has been designed for you! Your, job will mostly be implementing new features for a component.

3. Windbg is actually great and I will die on this hill. You might have to print some debug logs but you won't be looking at metrics because there's whole teams dedicated to doing that stuff. There's also tools which quickly spin up VMs for you to do some live kernel debugging.

Meh stuff:

1. The pay, could be better. That might change soon though.

gavinray 

  >  Windbg is actually great and I will die on this hill.
Someone convinced me to try Windbg Preview about 2 years ago and I was in awe. I too will now die on the hill that it's an incredible cross-language debugger capable of a lot more than that.
asveikau
> Windbg is actually great and I will die on this hill

Came here to say this. (Actually kd, but lots of overlap.)

criddell
How did you learn it? I use it occasionally to debug memory dumps after an application crash but I know that WinDbg can do so much more than show me the stack just as the application crashed.
asveikau
I learned it at Microsoft. When debugging issues in Windows, it was pretty common to leave a remote server open and pass it around among colleagues to find the right person who knows what code is failing. You could see what the remote party was typing and doing. It was a pretty social way to learn your way around.
matthewfcarlson
This comment is something that matches my experience. My new company tends to have worse work life balance but higher pay. It’s a trade off. I think the best play is join microsoft and leave after the nice sign off bonus all vests. I can’t complain about the ride, just the drop at the end.