This is a wonderfully told story of two very senior engineers who developed a lifelong friendship and wrote some of the core of the Google search engine I use countless times per day - I really enjoyed this piece.
I'm so glad this this article (unlike so many other representations of Jeff and Sanjay's work) does not overlook Sanjay's contributions.
I don't know what it is; maybe Sanjay is a weird (read: non-English) name, maybe it's that computing culture's obsession with lone hackers leaves no room for a partnership like Jeff and Sanjay's. Anyway, kudos to the New Yorker for not falling into that trap.
I think a big part of it is something the article touches on: Jeff is an extrovert, Sanjay is an introvert. Jeff is the much more visible of the two; the one who people are more likely to have heard speak personally or to have dealt with. It sounds like this is how they both prefer it, but it definitely means that Sanjay's contributions get passed over in the common discourse. I was also glad to see the article avoid that trap.
Ditto on the sentiment. I’m not a Googler but I’ve worked closely with a few, and Jeff was revered accordingly (along with the Chuck Norris-esque jokes). I’d honestly never heard Sanjay mentioned (and didn’t know who he was) until I read this article. I’m glad he was given his due and I’m glad to have learned about his contributions.
Funny thing, that was started by kentonv [0], who worked on open-sourcing protobufs [1], then CapnProto [2], then (with Jade Wang) Sandstorm [3]. He lurks around here :)
Indeed, I wrote the internal web site (built on a pre-release prototype of AppEngine!) that aggregated these jokes, and released it on April 1, 2007. I was not funny enough to come up with the jokes myself; I enlisted some friends to seed the site before "launch".
In retrospect, I regret that Jeff Dean Facts gave people the impression that Jeff is more notable than Sanjay. The only reason it was "Jeff Dean Facts" and not "Sanjay Ghemawat Facts" is because Jeff seemed "funnier". But I dunno, that might just be some form of subtle racism that I failed to recognize at the time. :(
As a college student, it was so weird learning about Jeff Dean's meme status, and it wasn't until after joining Google and coming across things like the original protobuf design doc that I realized how significant Sanjay's contributions are.
Yeah. Maybe that's the outside view, or view from the surface level folks concerned with the next shiny framework, but I think the best engineers are always at least a little enamored with deeper nuts and bolts.
That makes sense. At least, outside of massive companies like Google, who would see the value in a compiler that optimized code for their use even a fraction of a percent better. Then compilers become the "shiny object" for them.
Writing a modern concurrent garbage collector is as complicated as writing a compiler, to a great degree because of their "real time" constraints and concurrency.
these are the kind of articles i like reading, well written, interesting subject and good content length. Too many times i read bold headlines on HN, "Why X is bringing america down", then go to read it and just when they get to the crux of the matter i find the article has ended smh.
Good stuff. Every sport has great partnerships (Jordon - Pippen, Brady - Gronk) and I always wondered if such a thing would even be possible in SE as I've felt overly self conscious and inefficient while pair programming.
Though what really makes this my favorite long read of the year is how it follows their story over the course of Google's rise and the subtle juxtapositions between early Google of "doors across sawhorses" and today with "Level 0-11". Given all the recent news with engineer walkouts and protests it makes you realize that Google have probably done the best job of handling the scaling challenges of being a unicorn and making the necessary cultural adjustments as they grew in size.
What a lovely profile. I worked at Google early enough that I got to know Jeff and Sanjay, although never worked directly with them (to my regret). Every aspect of this article rings true to me, the way they are, the way they work together. I always admired how incredibly collaborative they were. They never competed with each other. Or really with anyone else, either, they just quietly did excellent work and helped others do their work too.
Would highly recommend checking out `The Undoing Project` by Michael Lewis. It's a biography (of sorts) about Amos Tversky and Daniel Kahneman that has excellent stories about their fruitful collaboration in behavioral economics.
Sanjay's "experimental" directory in Google's repo -- a place where users can commit unreviewed code that are just ideas, or park some code that isn't ripe yet -- is full of gems.
Holy moly, a mention of Bogdan! He was a legend of his own. Some of the most consequential changes, like elevating your service to higher QoS levels, which something like e.g. Gmail would require, were gated on his approval and his only. Someone made him a SW Easter egg, but the story already on the Internet gets some basic facts wrong.
“I don’t know why more people don’t do it,” Sanjay said, of programming with a partner.
=======
Maybe because most employers won't pay for that. Two people? For one level of output? You'd have to prove yourself a level 11 googler before most places would give you the luxury of working as a long-term pair.
I don't think that's it. It's typical to have a team of programmers working on a large project, and many managers are willing to try things.
However, pair programming does require a lot of commitment to make it work. I've only seen it work on a team where it was an XP project to start with, so everyone hired was already willing to give it a solid try. Your average team isn't self-selected in this way and won't have the same level of enthusiasm for a new management initiative.
Pivotal Labs has had across-the-board pair programming for years[1]. Although the folks there are undoubtedly talented, I'm fairly certain most aren't at Jeff and Sanjay's level.
I don't know why companies don't do it, but it's not cost alone. Here are some of my hypotheses:
1. It seems like it would be more expensive - which might deter companies from thinking about it
2. If you're not already doing it, it's hard to convince all or even a large fraction of your engineers to do it. It's a completely alien way of working to most people
3. It's not obvious to employees who haven't done it that it's clearly better. And it might not be - I honestly don't know
4. For 100% pairing, you need 2 programmers' schedules to line up exactly, 5 days/week - so goodbye flextime, work from home, remote working and all the other arrangements that people structure their lives around
5. It must be exhausting to sit with, and talk to, another person constantly your entire working day
Yeah, based on what I've seen/ heard, pairing should last no more than half the day. It's just too demanding to maintain all day long, like a cognitive sprint.
Absolutely. Just the bus factor thing was amazing on its own. It's the very rare startup where anybody can take a vacation whenever they need. But that was definitely my experience. Even though I was the first programmer, in pretty short order there was nothing where my absence a blocker. It was great.
i'd think another benefit might be 'better code' overall. has there been any study about the quality of code from these setups? number of bugs or security issues, for example?
Regarding #5, you get used to it. It's intense at first, but after a while, it's just normal. One secret is to take actual breaks. And another is to not over-work. When pairing, there's none of that stuff that might stretch out your day. E.g., checking Hacker News. It's solid work time, and after 8 hours of that I'm definitely done for the day.
After reading this article, I'm surprised that Google doesn't encourage pair programming / developing developer relationships at the level that Jeff and Sanjay have here. One might even argue that the manner in which interviewing and hiring is currently done in the tech industry in general might actively work against building the sorts of relationships where the output is greater than the sum of the inputs.
Their specific choices seem hard to replicate. Very strong friends, both very strong programmers, both willing to work at the same company. I'm sure that parts of their successes can be applied elsewhere, but from the article a lot of it is about having a strong relationship.
In a way, Google does encourage it by requiring all code to be reviewed before check in. If you're pair programming, you already have a reviewer and can check in right away. One-person projects can be more frustrating due to the delay.
It doesn't seem to be enough to encourage widespread pair programming, though.
Every company I've worked at as a software engineer, even my first where we emailed patches to each other like cavepeople (and no we were not using git) has had code reviews before check in. It's not at all the same thing as pair programming though.
> Most of the fragile insights that laid the foundation of a new vision emerged not when the whole group was together, and not when members worked alone, but when they collaborated and responded to one another in pairs
The article gives several examples of this dynamic, but one that came to mind for me was how Monty Python members created sketches for Flying Circus:
> Writing started at 9 am and finished at 5 pm. Typically, Cleese and Chapman worked as one pair isolated from the others, as did Jones and Palin, while Idle wrote alone. After a few days, they would join together with Gilliam, critique their scripts, and exchange ideas. Their approach to writing was democratic. If the majority found an idea humorous, it was included in the show.
Anyone know where folks like Peter Norvig and Urs Hölzle fit on the L1-L11 scale? I assumed that all of the early employees/now legends were at the top level, so I was surprised to see the article state otherwise.
Urs however does have one, he's listed as a Google fellow (L10) if you go looking.
As a general comment, to compare, a director is L8 equivalent, and VP L9, (but on the management ladder) although at VP+ it gets tricky because there are subdivisions.
Also worth noting that the management ladder goes up more. But the total number of people who are >L11 is probably <11, so it's not like there's any data on how exactly that's broken up.
Peter is a director, and Urs is a senior VP. As such, they are now both on the management ladder. I think the article was discussing only people who were on the eng ladder.
Was surprised to hear the audio version of the story has a different title - "Binary stars". I think the editorial board went with a more click friendly title.
Does anyone have a reference for "In 1966, researchers at the System Development Corporation discovered that the best programmers were more than ten times as effective as the worst." ?
Is there a 1966 reference out there somewhere? Or perhaps the actual study was done in '66 and didn't make it into press until '68? I didn't pull the original paper.
(The fact checkers at the New Yorker are typically very sharp.)
edit: I pulled the original paper (10.1145/362851.362858). The studies were done in 1966!
Interesting to see the genesis of the myth of the 10x programmer.
"Interesting to see the genesis of the myth of the 10x programmer."
How could it be a myth? Ever see a professional sports player compete against a non professional player? Why wouldnt this dynamic exist in programming?
Maybe I'm unimaginative, but in sports where performance is objectively comparable, I can't think of any where professionals 10x amateurs.
Amateur marathons are about 4 hours, professional are about 2 hours. Amateur deadlifts are about 500lb, professional are about 1000lb. Amateur long jump might be 6m, professional almost 9m.
Cristiano Ronaldo and Messi are extreme outliers in soccer/football, even compared to other greats. They're probably 10x. Trouble is amateurs don't play in the big leagues, so we don't have data on how a normal person would peform. There is an amusing Ronaldo vs Ronald video. I think it's called "Tested to the Limit."
Occasionally a 1st league team plays a 3rd league. I saw a video of Napoli vs Nowhere go 26-0. They scored about as fast as the game could be reset to kick-off. That's more than 10x.
You are confusing game scoring with some concept of "x times ability". In many games, someone who is only 10% better than opponents can win close to 100% of the games.
To solve problems at scale, paradoxically, you have to know the smallest details -Alan Eustace
Today, Google’s engineers exist in a Great Chain of Being that begins at Level 1. At the bottom are the I.T. support staff. Level 2s are fresh out of college; Level 3s often have master’s degrees. Getting to Level 4 takes several years, or a Ph.D. Most progression stops at Level 5. Level 6 engineers—the top ten per cent—are so capable that they could be said to be the reason a project succeeds; Level 7s are Level 6s with a long track record. Principal Engineers, the Level 8s, are associated with a major product or piece of infrastructure. Distinguished Engineers, the Level 9s, are spoken of with reverence. To become a Google Fellow, a Level 10, is to win an honor that will follow you for life. Google Fellows are usually the world’s leading experts in their fields. Jeff and Sanjay are Google Senior Fellows—the company’s first and only Level 11s.
Yeah it's mostly accurate but drop level 1. Lowest I've seen for technical employee of any kind is 2. New grad is 3. 4 is about 2 years later. 5 is the point where you have "no expectation if advancement" and is the most common level for mid-career people.
Nah. You can do it with anybody. I've been part of long-lived teams that pair. But there we rotated pairing partners once or twice a day, so that over the course of the week you'd pair with everybody.
I think that's much more sustainable for most. Pairing with the same person every day is sort of like sharing a snowbound cabin with them. There are very few people I get along with so well that they wouldn't grate after a few weeks.
103 comments
[ 4.6 ms ] story [ 242 ms ] threadI don't know what it is; maybe Sanjay is a weird (read: non-English) name, maybe it's that computing culture's obsession with lone hackers leaves no room for a partnership like Jeff and Sanjay's. Anyway, kudos to the New Yorker for not falling into that trap.
Funny thing, that was started by kentonv [0], who worked on open-sourcing protobufs [1], then CapnProto [2], then (with Jade Wang) Sandstorm [3]. He lurks around here :)
[0] https://plus.google.com/+KentonVarda/posts/TSDhe5CvaFe
[1] https://github.com/protocolbuffers/protobuf
[2] https://capnproto.org
[3] https://sandstorm.io
Indeed, I wrote the internal web site (built on a pre-release prototype of AppEngine!) that aggregated these jokes, and released it on April 1, 2007. I was not funny enough to come up with the jokes myself; I enlisted some friends to seed the site before "launch".
In retrospect, I regret that Jeff Dean Facts gave people the impression that Jeff is more notable than Sanjay. The only reason it was "Jeff Dean Facts" and not "Sanjay Ghemawat Facts" is because Jeff seemed "funnier". But I dunno, that might just be some form of subtle racism that I failed to recognize at the time. :(
PS. Your list doesn't include my current project, Cloudflare Workers... :) https://blog.cloudflare.com/cloud-computing-without-containe...
As a college student, it was so weird learning about Jeff Dean's meme status, and it wasn't until after joining Google and coming across things like the original protobuf design doc that I realized how significant Sanjay's contributions are.
That bit surprised me. For me it doesn't get much sexier than compilers.
But to call them sexy is another thing ...
They will even mail you a copy in exchange for a subscription. It's magical.
Though what really makes this my favorite long read of the year is how it follows their story over the course of Google's rise and the subtle juxtapositions between early Google of "doors across sawhorses" and today with "Level 0-11". Given all the recent news with engineer walkouts and protests it makes you realize that Google have probably done the best job of handling the scaling challenges of being a unicorn and making the necessary cultural adjustments as they grew in size.
Extreme Pair Programming - Guy Steele and Richard Stallman
Are books extra-long?
=======
Maybe because most employers won't pay for that. Two people? For one level of output? You'd have to prove yourself a level 11 googler before most places would give you the luxury of working as a long-term pair.
However, pair programming does require a lot of commitment to make it work. I've only seen it work on a team where it was an XP project to start with, so everyone hired was already willing to give it a solid try. Your average team isn't self-selected in this way and won't have the same level of enthusiasm for a new management initiative.
I don't know why companies don't do it, but it's not cost alone. Here are some of my hypotheses:
1. It seems like it would be more expensive - which might deter companies from thinking about it
2. If you're not already doing it, it's hard to convince all or even a large fraction of your engineers to do it. It's a completely alien way of working to most people
3. It's not obvious to employees who haven't done it that it's clearly better. And it might not be - I honestly don't know
4. For 100% pairing, you need 2 programmers' schedules to line up exactly, 5 days/week - so goodbye flextime, work from home, remote working and all the other arrangements that people structure their lives around
5. It must be exhausting to sit with, and talk to, another person constantly your entire working day
1. https://pivotal.io/careers
Simultaneously, you get a bunch of benefits. (Most importantly, there's no bus factor of 1 for anything)
Edit: added a missing "the".
It doesn't seem to be enough to encourage widespread pair programming, though.
The article gives several examples of this dynamic, but one that came to mind for me was how Monty Python members created sketches for Flying Circus:
> Writing started at 9 am and finished at 5 pm. Typically, Cleese and Chapman worked as one pair isolated from the others, as did Jones and Palin, while Idle wrote alone. After a few days, they would join together with Gilliam, critique their scripts, and exchange ideas. Their approach to writing was democratic. If the majority found an idea humorous, it was included in the show.
https://en.m.wikipedia.org/wiki/Monty_Python
As a general comment, to compare, a director is L8 equivalent, and VP L9, (but on the management ladder) although at VP+ it gets tricky because there are subdivisions.
Also worth noting that the management ladder goes up more. But the total number of people who are >L11 is probably <11, so it's not like there's any data on how exactly that's broken up.
I'm seeing the year 1968 at http://thecomputerboys.com/?tag=black-art#fn-156-1
Is there a 1966 reference out there somewhere? Or perhaps the actual study was done in '66 and didn't make it into press until '68? I didn't pull the original paper.
(The fact checkers at the New Yorker are typically very sharp.)
edit: I pulled the original paper (10.1145/362851.362858). The studies were done in 1966!
Interesting to see the genesis of the myth of the 10x programmer.
For anyone else, that DOI points to: https://dl.acm.org/citation.cfm?doid=362851.362858 "Exploratory experimental studies comparing online and offline programming performance" from CACM 1968.
How could it be a myth? Ever see a professional sports player compete against a non professional player? Why wouldnt this dynamic exist in programming?
Amateur marathons are about 4 hours, professional are about 2 hours. Amateur deadlifts are about 500lb, professional are about 1000lb. Amateur long jump might be 6m, professional almost 9m.
Occasionally a 1st league team plays a 3rd league. I saw a video of Napoli vs Nowhere go 26-0. They scored about as fast as the game could be reset to kick-off. That's more than 10x.
I haven't read through it yet, but it appears to be a more extensive whitepaper version than the CACM version.
You might know his writing from “You’re Probably Using the Wrong Dictionary” or his amazing article about New York’s subway system told through the lens of the countdown clock: http://www.theatlantic.com/technology/archive/2015/11/why-do...
Highly recommend all his work: http://jsomers.net/
To solve problems at scale, paradoxically, you have to know the smallest details -Alan Eustace
Today, Google’s engineers exist in a Great Chain of Being that begins at Level 1. At the bottom are the I.T. support staff. Level 2s are fresh out of college; Level 3s often have master’s degrees. Getting to Level 4 takes several years, or a Ph.D. Most progression stops at Level 5. Level 6 engineers—the top ten per cent—are so capable that they could be said to be the reason a project succeeds; Level 7s are Level 6s with a long track record. Principal Engineers, the Level 8s, are associated with a major product or piece of infrastructure. Distinguished Engineers, the Level 9s, are spoken of with reverence. To become a Google Fellow, a Level 10, is to win an honor that will follow you for life. Google Fellows are usually the world’s leading experts in their fields. Jeff and Sanjay are Google Senior Fellows—the company’s first and only Level 11s.
I..don't think this is still accurate. New grad SWEs are L3's.
I think that's much more sustainable for most. Pairing with the same person every day is sort of like sharing a snowbound cabin with them. There are very few people I get along with so well that they wouldn't grate after a few weeks.