I think it was a very common perception among application-level SWEs and SREs that TI, platforms, and Borg did not themselves use the stack enough to perceive its flaws.
The most obvious change that came from this video is that Google abandoned the Borgmon readability requirement. At Google, every change needs approval from someone who has passed a detailed style-guide review process in the given language.
Now over multiple changes, it used to require one fairly big one. It's still a pain in the languages that require it--which is all the main ones, but very few of the niche ones.
Many other things changed as well. Much of what the video complains about got automated and better documented. But the company has grown so much, and the product lines have diversified so dramatically, that there are still plenty of places to complain about the overhead.
I've proudly managed to avoid Borgmon in favor of Monarch. Which was new at the time, but worked all right even back then. I have a lot fewer gray hairs because of that. They should have kept and rigidly enforced the Borgmon readability requirement to force people to migrate off that convoluted, idiosyncratic piece of shit.
I've never understood places with rigid style guides policed by people. Its idiotic, because we have computers and in places like google presumably a fair number of them know basic parsing/lex sufficiently that if they can't make a tool like clang-format that automatically reformats on save/commit/whatever then they can use a tool like clang-tidy to toss warnings during a development/CI/whatever phase.
Putting people in charge of formatting/style is just an excuse for wasting time bikeshedding, either the code is wrong and a tool can tell you, or its not wrong.
That's not what readability is. There are plenty of automated tools that will give you results from running lint, ClangTidy and other tools. Readability is mostly about structuring your code well to be easily read. It's about architecting your code within a single file. It's about telling a junior SWE who reinvented the wheel use a library function he/she didn't know about instead.
So the rules can be codified sufficiently to test people on, but they can't be codified for a computer?
The only one that sounds more difficult to codify is telling people of the existence of duplicate functions. But as someone who contributes to the linux kernel, I can tell you right now that the only way that works reliably is to have a very large pool of reviewers. Very experienced engineers frequently miss what people are doing in other parts of the source base, the name might not be what they expect, etc, etc, etc. In the case of linux there are a fair number of duplicates, or similar functions, and people write coccinelle patches to replace them on a fairly regular basis after they have been in the kernel for years.
So, I doubt giving someone a formal gatekeeper flag, really helps vs just having wider change review.
I know of no automated tools available today that can determine if an identifier is accurately and usefully named. They can all tell if you are using the proper case, but that doesn't really tell you anything.
No tool like that tells you if returning a bool instead of an enum is appropriate here, or that a reference vs a pointer makes more sense given the rest of the code.
I'm sure a clever machine learning algorithm could figure that out with a corpus as large as Google's. Maybe. But no tool like that works today.
And not strangely at all, Google does accept "what clang-tidy does" as the canonical way of formatting text. But readability at Google is far more than just formatting.
Readability is frustrating and annoying, but more than just lint.
> I know of no automated tools available today that can determine if an identifier is accurately and usefully named.
Naming isn't very language-specific though [*], and poor naming should be flagged in coding interviews. Any Google engineer should generally have good naming, and should be able to critique others' naming in any language.
[*] Granted there are some language-specific naming conventions, like ! and ? in Ruby methods. But I'd say the majority of languages don't have important naming conventions, and those that do often have lints for them.
This discussion wasn't about hiring but about code reviews. If you use bad names in your code then the reviewer is correct when they tell you to make better names.
"poor naming should be flagged in coding interviews"
Exactly! And Readability process is exactly the training Google uses to ensure someone knows how to look for this issue, and that someone actually is looking for this issue in code reviews.
There is plenty about the Readability process I don't like (particularly that one has to get it through a fairly artificial process), but checking for things like the above (which I just use as one easily understood example) isn't something that happens automatically in an organization as large as Google, with as many engineers as Google.
A common story on HackerNews is just how bad the average programmer is, how crummy code they produce. Taking steps to ensure they do it better is a good thing.
We can argue about the process, for sure, and there are many things I would love to change about it. But "yeah, that should be done in code reviews" is exactly the idea here.
> Very experienced engineers frequently miss what people are doing in other parts of the source base, the name might not be what they expect, etc, etc
Very true. Readability can't help with that, nor is it designed to. It's mostly there to help novices and new hires. Experienced engineers already have readability themselves so they don't need this extra review.
> So the rules can be codified sufficiently to test people on, but they can't be codified for a computer?
Small note: readability isn't a test or quiz you take (asterisk). It's obtained by merging code in the language you want readability for. If you merge code for a language often and the reviewers have very few style-based questions for the code then you will get readability fairly quickly.
> The only one that sounds more difficult to codify is telling people of the existence of duplicate functions. But as someone who contributes to the linux kernel, I can tell you right now that the only way that works reliably is to have a very large pool of reviewers. Very experienced engineers frequently miss what people are doing in other parts of the source base, the name might not be what they expect, etc, etc, etc.
A better example would be knowing when you should use `const std::string&`, `std::string_view` or `char*`. Example: https://abseil.io/tips/1
The best readability advice I have recieved has been:
1. Direct "I was confused by X" or "The recommended way to do A is using B", etc
2. Reasoned: "std::string_view is more efficient and clearer in intention than char ptr, it also improves type safety as it is read only and clear about ownership"
3. Linked to source material where examples are given totw or other examples in the code.
> So the rules can be codified sufficiently to test people on, but they can't be codified for a computer?
I mean, given the halting problem alone, and the absence of general AI, yes, I'd say that...
I'm not sure what to tell you otherwise. I've worked at a place with such strong coding style guidelines, and even though I personally did not agree with every individual point, I accepted the tradeoff because being able to count on consistency in a very large code base was extremely helpful.
And I take it google has done wide ranging scientific studies about the variations in coding styles and language constructs that it is a secret advantage that they know how to write "readable" code? Implying they tried a bunch of diffrent ways until settling on the one true way that allows a diverse set of people with diverse experiences to read it?
Ever heard of COBOL?
Because readability has always been in the eye of the beholder, and codifying it makes it even worse.
I'm not a fan of readability exactly the way Google does it, but I'm pretty happy that Google insists on various aspects of it, like good identifier names.
I don't know of any research off hand, but I'm pretty sure the industry consensus is that good identifier names improve the quality of the code (Go style notwithstanding.) Readability is one way to training engineers to do it.
> Because readability has always been in the eye of the beholder, and codifying it makes it even worse.
This is empirically false. Consistency, even if it is unfavorable to your preferences, is superior to inconsistency. So a codified set of best practices is better than none at all.
There are part of Google's style guides that I would change if I could, but I also prefer having a style guide (and one that goes beyond things that are lintable) than none at all, because consistency across the codebase means that I can usually understand code at a glance, or if not, know at a glance that something unusual is happening. (this is in fact precisely the argument in favor of autoformatters like gofmt/black/prettier, but extended to softer concepts that can't always be formatted: consistent style, even if it isn't your favorite, is superior to inconsistent style).
Consistency is what you get when you have a defined rule set programmatically enforced. If your looking for "readability" via human judgment, then you get a very different result.
A programmatically enforced set of rules is certainly one way to get consistency, but it isn't the only way. You can achieve consistency through culture and training too, and sometimes that's the only way.
It isn't possible to statically analyze if a class/struct is a POD or if the methods enforce invariants. But it's often very easy to do so with a human eye. And there's value in the distinction!
Similarly, forcing someone to justify using a power-feature (operator overloading, templates, metaclasses, whatever) can only be done by a human. There may be cases where the power feature is warranted and the benefits outweigh the cost, but a linter can't know that. (and ultimately all of this comes back to: things look consistent, and when things are inconsistent, that's a strong signal that something unusual is happening and you should pay close attention)
> Consistency, even if it is unfavorable to your preferences, is superior to inconsistency. So a codified set of best practices is better than none at all.
Indeed, but at what cost? Every step of added gatekeeping reduces development velocity, and Google does not have a monopoly on high scale services (cough AWS cough). In an ideal world for every full-time dev writing code, you'd have a full-time reviewer whose job it is to simply review that dev's code. But the real world has budgets and deadlines, and humans become discouraged when changes take too long to complete. So is readability worth the added gatekeeping cost is the question.
You're ignoring the cost of unrestrained development without standards. The impression I've gotten is that a AWS papers over their lack of development gatekeeping with unreasonable oncall loads. All things considered, the "cost" of readability in terms of development velocity is fairly low. If anyone on your team has readability [in the language], they can review your code. This can be problem if only one person has it, but usually once one person gets through the process, you can get more people through fairly quickly.
I'm a bit of a polyglot at Google (I have readability in C++, python, and Javascript/Typescript, and am part-way through the Go and Java processes, which covers basically every popular language at the company), and IMO, the readability process for each language has been a net positive, although it was painful for a while when I had no readability and none of my coworkers did (but there are procedures for getting around that today).
For example, I "donate" my readability, and review random changes from people who don't have readability but need a quick turnaround or aren't yet working to gain readability.
> You're ignoring the cost of unrestrained development without standards.
I'm not. I'm not saying that code should be merged without any stylistic commentary; I'm simply proposing that stylistic commentary be provided by other stakeholders in the service you're contributing to, on top of whatever linter or sanitizer is being used. That is, after all, the goal of a human reviewer. To add _another_ layer of gatekeeping is what I'm questioning.
> The impression I've gotten is that a AWS papers over their lack of development gatekeeping with unreasonable oncall loads.
I don't agree with this perception at all. AWS spends a lot of time building frameworks and libraries which attempt to minimize human gatekeeping to only the necessary areas. So think retries, backoff, timeouts, circuit-breakers, etc. Instead of adding gatekeeping at the code level (with things like code style), instead gatekeeping is added at the architecture and operations stages, which is where most of the work of keeping a production level service up, available, and fast. Operational level gatekeeping in the bounds of a given organization is a much more neatly constrained problem than stylistic gatekeeping and this gatekeeping happens on a per-service basis rather than a per-CL/review basis. This keeps overhead a lot lower.
My impression from being outside of Google (and having a Xoogler for a partner) is that Google loves bureaucracy and process, and that engineers that continue to stay at Google are fine with navigating these processes. Its answer to problems is to add more bureaucracy, another item for the checklist that a person/committee/reviewer needs to check for. That reflex is hard to shake, IMO.
> That is, after all, the goal of a human reviewer. To add _another_ layer of gatekeeping is what I'm questioning.
I guess I'm not following the distinction. For most people, that's exactly what happens. Either you, or the someone on your direct team, will be the person with readability for your code. The readability process just ensures that you don't end up with silos where style diverges too much, and that the stakeholder who proposes stylistic commentary has a baseline knowlege of linguistic best practices.
With few exceptions, the thing people complain about is attaining readability themselves, not getting changes approved.
> I don't agree with this perception at all. AWS spends a lot of time building frameworks and libraries which attempt to minimize human gatekeeping to only the necessary areas. So think retries, backoff, timeouts, circuit-breakers, etc.
I think we're talking a bit past each other. These things are all table stakes defined by frameworks and updated by centralized processes as best practices change. Human reviewers, for the most part, are only going to say "why are you diverging from the default".
But importantly, that sidesteps my statement entirely, which, rephrasing, is that amazon carries along more technical debt than Google. Whether that's a good long-term business decision remains to be seen, but the impression I get from ex-amazon friends is that maintainability is less prioritized, and maintainability is one of the goals of the readability process. Global consistency helps anyone understand and improve anything else, and to an extend avoids haunted graveyards (though...not entirely).
> instead gatekeeping is added at the architecture and operations stages, which is where most of the work of keeping a production level service up, available, and fast. Operational level gatekeeping in the bounds of a given organization is a much more neatly constrained problem
I'm not sure what you mean, but if anything, this sounds more bureaucratic.
> My impression from being outside of Google (and having a Xoogler for a partner) is that Google loves bureaucracy and process, and that engineers that continue to stay at Google are fine with navigating these processes. Its answer to problems is to add more bureaucracy, another item for the checklist that a person/committee/reviewer needs to check for. That reflex is hard to shake, IMO.
FWIW I don't get this impression. It's honestly difficult for me to tell what you even are referring to. Like, other than readability and promo, there's very few processes or bits of bureaucracy that are consistent across the company, and those two have gotten decidedly more streamlined since I joined the company.
Well okay that's half true, I can think of a bunch of places where bureaucracy has increased, but they're all spurned by legislation.
> I guess I'm not following the distinction. For most people, that's exactly what happens. Either you, or the someone on your direct team, will be the person with readability for your code. The readability process just ensures that you don't end up with silos where style diverges too much, and that the stakeholder who proposes stylistic commentary has a baseline knowlege of linguistic best practices.
Separating these two out, even if one person may fulfill two roles, IMO adds to bureaucracy that I don't care for. Ramping up onto a new language sounds like a pretty fraught experience if nobody on your team has readability and the service you're contributing doesn't have a member who has the time to actually work with you, especially if your work is not relevant to theirs. They even touched upon this in the linked video. The whole idea that you can enter a bureaucratic catch-22 at a company seems like an anti-pattern to me, a moment to stop and think; a "modern" engineering organization should try its utmost to accelerate development, with only as many checks as needed for safety and no more.
> With few exceptions, the thing people complain about is attaining readability themselves, not getting changes approved.
That's what I mean by bureaucracy. Even having "another eye" on the code should work. Having a certification process for readability is even more bureaucracy.
> But importantly, that sidesteps my statement entirely, which, rephrasing, is that amazon carries along more technical debt than Google. Whether that's a good long-term business decision remains to be seen, but the impression I get from ex-amazon friends is that maintainability is less prioritized, and maintainability is one of the goals of the readability process. Global consistency helps anyone understand and improve anything else, and to an extend avoids haunted graveyards (though...not entirely).
Ah, that makes a lot more sense. In my career I've never worked at a place that prioritizes code style that much, and I've worked at other high scale places in the past. This seems like a Google specific need. You're right that global consistency helps to save off "there-be-dragons" codebases, but I feel like this is a case of YAGNI. Unless you have engineers constantly cross-cutting across the company, most engineers learn the style guidelines of a team/product/service by ramping up on the team. The optimization for global consistency seems premature to me.
> FWIW I don't get this impression. It's honestly difficult for me to tell what you even are referring to. Like, other than readability and promo, there's very few processes or bits of bureaucracy that are consistent across the company, and those two have gotten decidedly more streamlined since I joined the company.
But promos drive _so much_ of the company culture, at least from what my partner tells me. And the video in the OP certainly makes it sound like Google has lots of bureaucracy, though I'm not sure how many of the requirements outlined in the video were there for hyperbole or not. But especially for services that have low SLOs, it makes no sense to put thought into failover, evacuation, or anything like that.
> Separating these two out, even if one person may fulfill two roles, IMO adds to bureaucracy that I don't care for. Ramping up onto a new language sounds like a pretty fraught experience if nobody on your team has readability and the service you're contributing doesn't have a member who has the time to actually work with you, especially if your work is not relevant to theirs. They even touched upon this in the linked video. The whole idea that you can enter a bureaucratic catch-22 at a company seems like an anti-pattern to me, a moment to stop and think; a "modern" engineering organization should try its utmost to accelerate development, with only as many checks as needed for safety and no more.
Keep in mind this video is ten years old. While yes, trying to write something from scratch in a language no one on your team has any experience in can be fraught (although this raises other questions: what exactly are you doing, in every case I've used a new lang, my team has been able to find people to review if needed), even in that case, the organization has help for you (see again my prior comment about "donating" readability, and my understanding is that the readability granting process actively prioritizes people who "need" the readability because they have few potential reviewers).
> That's what I mean by bureaucracy. Even having "another eye" on the code should work. Having a certification process for readability is even more bureaucracy.
I think initially the primary driver of readability is C++ (but Java + Guice also...needs it). Take a look at https://abseil.io/tips/. That's 70 tips, which is around a third of the internal ones. I have C++ readability and have internalized, some of those. The readability granting process ensures your code is reviewed by someone who knows all (or nearly all) of those tips, and will recognize when you're doing wrong things in your code and help explain how or why you can improve. When I personally went through the C++ readability process, I got mentorship on how to fix bugs in my code, some of which a peer on my team would have caught, and some of which they wouldn't have. I can now recognize those bug prone patterns (which are hard to lint for, in my case they mostly had to do with parallelism) and know about tools to debug them (msan and asan, which an experienced C++ user should know, but on my team of people who had basically no C++ experience at Google? Nah).
That follows into other languages. Being granted readability means you have a familiarity with the language that suggests you'll be able to effectively mentor others and not mislead them. You don't have that otherwise.
> Unless you have engineers constantly cross-cutting across the company, most engineers learn the style guidelines of a team/product/service by ramping up on the team.
Google uses a single repo and builds stuff from HEAD. This has up- and downsides, but one of the upsides is that its very easy to investigate unusual behavior in your dependencies. I was doing something similar just this week, and fixed obscure bugs in like 7 other teams tools that were causing downstream problems that those teams weren't aware of. No bureaucracy. I found the bug, fixed them, and sent the change to the owning team. Imagine other situations, where I'd need to maintain a fork, or file a bug to have them fix it, or coordinate an update with them.
You'll pay the cost no matter what, choosing to do so in a way that also provides value and mentorship makes a lot of sense to me.
> But promos drive _so much_ of the company culture, at least from what my partner tells me. And the video in the OP certainly makes it sound like Google has lots of bureaucracy, though I'm not sure how many of the requirements outlined in the video were there for hyperbole or not.
What counts for readability is not set in stone by some language czar as the One True Way. Everyone knows the style guides can't be perfect which is why they're relatively mutable.
In any case, readability will comment on stuff that cannot easily be quantified, such as when to use a certain object hierarchy or dependency injection, etc...
Oh boy.
So, to start, we have done scientific studies about the costs and benefits of readability (as codified circa 2018, since that was when the last study was).
It was studied for multiple programming languages Google uses.
It was done, at my request, by the engineering productivity research team, who are experts in this kind of research - you can find public papers they publish[1].
For background: I was the person responsible for production programming languages, and I did this precisely because i did not feel at the time there were recent good studies as to whether readability was really worth the cost.
The answer is "yes, it is".
There is an upfront cost, but readability has a meaningful and net positive effect on engineering velocity.
It is large enough to make the cost back quite quickly.
One could go down the rabbit hole of seeing whether you improve (or make worse) the numbers by changing various parts of the style guides, but like I said, what is there now, has in fact been studied scientifically.
Honestly, i'm not sure why you think it wouldn't be. You come off as a little immature when you sort of just assume people have no idea what they are doing and don't think these things through. You are talking about a company with 60k+ engineers. Every hour of all-engineer time you waste is equivalent to having ~30 SWE do nothing for a year.
Maybe there are companies happy to do that. I worked at IBM for a few years when i was much younger ;). All I can say is that as long as i'm involved in developer tools at Google, i'm gonna try not to waste my customers time.
[1] I say this in the hopes you don't go making assumptions about whether the studies were done properly. They were properly controlled for tenure, number of reviews, change size, etc.
I'd love to see the research here. Because searching the software engineering Google papers from 2021-2021, the only somewhat relevant papers I found were case studies (such as [1] and [2]), which were based on Googlers' perception of their own processes. While self-perception can indeed by used to measure efficacy, there's a reason why it's used with great caution in surveys. I'll be honest, I have a hard time believing in readability myself.
We have never published the readability research. I could look into doing so, but it is awfully Google specific. Trying to come up with something generally applicable would be a very different project
Then why do you often need external readability reviewers, especially if you or your team don't contribute to a codebase frequently? Having humans judge other humans' style is a given; it is part of the PR process. But why do you _also_ need someone with a readability rating as opposed to a code owner?
Note: I'm not a Googler nor a Xoogler, but my partner worked at Google for some years and so I've heard her complain a lot about Google and readability.
> But why do you _also_ need someone with a readability rating as opposed to a code owner
FWIW, these can be the same person - if the have appropriate readabilities. The problem is that it's not easy to get readability in the first place, so occasionally teams have to look for people outside.
Right but the fact that these are two separate roles strikes me both as bureaucratic overhead and a fundamental lack of trust in individual engineers. If I were approached to enact something like readability, I would veto it immediately. We hire engineers because we trust them. Part of that trust means that engineers occasionally wade into codebases they don't understand in languages they don't know and need to make contributions, but we trust them to not make a mess. Do some of them make messes? Undoubtedly. Is the magnitude of the mess, multiplied by the number of engineers, worth the creation of an entire readability certification process? I don't think so.
> Is the magnitude of the mess, multiplied by the number of engineers, worth the creation of an entire readability certification process? I don't think so.
And that's an acceptable trade-off for a different sized company. Google arguably has the biggest centralised codebase in the world, and simply has different requirements.
As other have said, most of the overhead is from attaining readability, not the requirement itself.
> And that's an acceptable trade-off for a different sized company. Google arguably has the biggest centralised codebase in the world, and simply has different requirements.
AWS does not operate this way nor do many CDNs operate this way nor do ISPs operate this way. There's other high scale businesses out there. Google isn't the only one. Using Google's scale to justify business practices is a self-fulfilling prophecy.
> AWS does not operate this way nor do many CDNs operate this way nor do ISPs operate this way.
I'm not denying that, but all the same, we're not talking about thousands of separate codebases. It's a single monorepo codebase with consistent styling, the advantage being that people can switch teams or contribute to other Google software under the same style guide.
The overhead isn't a bug, it's a core feature as consistent style makes it that much easier to switch teams.
It's also worth noting that Borgmon readability included an awful lot of "how not to shoot yourself in the foot in strange and unexpected ways". That required someone who had shown they knew of those strange and unexpected ways to review your code.
The hypothetical discussion about readability is pointless.
Let's make it specific. Read https://google.github.io/styleguide/cppguide.html for readability for a language, namely C++. All the things that can be automated, automatic tools have been written for. But, for example, you can't automate "Prefer to use a struct instead of a pair or a tuple whenever the elements can have meaningful names." Because what does it mean for a name to be meaningful?
"Because what does it mean for a name to be meaningful? "
Are you optimizing for someone who already knows all the project lingo, or someone who doesn't know any of it?
Are your engineers native English speakers?
There are a whole bunch of things which make the perfect variable name frequently less than perfect, and putting project insiders in charge likely yields the opposite result.
If you don't know what a vma, pte, pfn, compound_page, young pte, huge page, lru, etc your going to be unable to even begin to understand what that code is doing, despite those all being pretty reasonable variable names and actually fairly industry standard concepts. It gets worse as you move to more esoteric topics. Expanding pte to PageTableEntry might help some subset of users, but at the expense of those that work on the code daily. So who do you optimize for? Is it readable if the only people that can read it already know what it does?
Formatting and readability are two separate concepts (as other replies have pointed out). I'd like to specifically point to a fantastic example of what we mean when we say "readability": https://www.youtube.com/watch?v=wf-BqAjZb8M
Someone with readability in a language, who keeps up with the style recommendations, will generally produce code that is easier to read by other engineers.
I think you can read about some of these changes in Google's SRE and SWE books (even if they don't mention this video in particular), at least the ones most likely to be interesting to someone outside Google.
But dropping Borgmon readability was the most immediate and obvious. It was basically true that no one had Borgmon readability. The policy was a catch-22: you couldn't get readability for the simple/formulaic Borgmon macro invocations that were encouraged and often sufficient. You could only get it for doing something "clever". I got it by writing fancy borgmon rules to paper over a problem that (in hindsight) I should have solved elsewhere.
Another was easing quota management. IMHO the most unbelievable thing in the video was that after Broccoli Man told Panda Woman to get quota in two cells, she just said "done". Besides the hassle in transcribing what you needed into the request system [1], various types of quota were chronically unavailable where you needed them, even in tiny amounts. In 2010, I kept a critical infrastructure service running by regularly IMing major clients' on-calls asking them to donate 0.1 cpu(!) of their quota in some cell or another when I didn't have quite enough to grow. There was a "gray market" mailing list where people would trade resources they couldn't get through the primary system. But eventually, they built a system that for small services would make the quota just happen for you.
Overall, it was a kick in the pants for the most basic infrastructure teams that made them see how unnecessarily hard this is for their internal customers, prompting them to make small things just happen while keeping large things possible. In any large organization, it's healthy to get this kind of feedback regularly. The actual specific changes and technologies are pretty specific to Google in 2010...
[1] Many people managed this very very tediously with spreadsheets. I eventually wrote a tool to generate the requests based on comparing your intended production config with your current quota.
Production priority quota horse trading in the days before it was easy was a real skill. But non-production quota was free and virtually infinite, even in those days.
Or converting between priority levels or users that were under your product. If you ran a large enough service, there was also begging your users for spare quota or warning them that their recent donation couldn't be fully deployed because a necessary change in some secondary borgcfg template had eaten into the service's resources, so it would help everybody if they knew anyone with a few cores laying around...
By the way, do you happen to be the jwb that filed a... creative BT quota request?
Hah, I was the one that played along and made you keep going ("May you be gifted with a long, healthy life and an exquisite taste in ties"?). And I think SAD was one of the teams that occasionally helped us find resources in this or that cluster, along with Analytics and, later, Social.
I seem to remember hearing prod quota disputes described as “monkey knife fights” by SRE. SWE would joke about using corp credit cards on EC2 instances rather than waiting on the outcome.
Thing is, though, something equally ridiculous happens at every large company and at more than a few small companies. Worse in government. The important thing is how they react when it's pointed out.
Every single buzzword uttered in the video has a cloud equivalent except for PCR. These days people just give up on regional (even zonal) outages and just take downtime and blame their cloud provider in the rca so doesn’t matter. The big thing that i think is missing in this thread is just how enormous google infra was even at 2010 so these problems were and still are sort of unique
I think I may have used your tool at some point. There were times in geo where we'd run out of bigtable quota and I needed to hit the "emergency loan" button to keep maps from going down globally.
That's why you had these problems: you should have just let maps go down globally and then pointed the finger at the appropriate target to blame for it.
I'm only half joking about that too. The other half is that sometimes it is better to let the system collapse under its own weight when that's what is required to convince the right people the system is broken.
There's been a big investment in server platforms that strive to enable SWEs to build a new service that follows Best Practices with as little knowledge and handholding as possible. These consist of conformance tests that yell at you while you're coding if you are trying something generally thought to be bad, and semi-automated workflows that help you bring your code to production. When everything works as intended, the production workflows set up a decent set of alerts, acquire resources, configure CI/CD pipelines, and launch your jobs with just a few button presses on your part. (In practice, one of the steps will probably require debugging, but eh, it seems way better than the broccoli man video.)
I feel like people forgot after about five years. I remember wasting a week filling out various pieces of paperwork and submitting byzantine configuration CLs so that some contractor would have permission to view a certain webpage through the corporate proxy. (I think what made me most mad is that regular employees could view the website with no additional configuration. I can understand if I was filling out tickets to get approval, or a security review, but the actual configuration of the proxy had to change to allow this, in addition to getting all of those approvals!) My team didn't make the website, and the contractor didn't work on my team, so I'm honestly not sure why I was involved. I just remember being annoyed about it. I'm sure there are some memes about it in the archive.
That's the scary, and perhaps most immoral, part about Google. More so than other megacorps, it relies on a shadow workforce of contractors to do core parts of its work. It's quite the caste system, and most Google FTEs are oblivious or choose to ignore it.
I know a Google contractor who was denied access to a lactation room at her office (she's the type of contractor who works full-time alongside regular employees). When she tried to fix that, she got sent into an infinite loop of support tickets being bounced around between different departments claiming it was the other one's responsibility. Literally no one was able to fix that for her. Her manager was on a different continent, and wasn't able to help either.
I did a stint contracting for Goldman Sachs a while back. I can relate. Don't think I can say anything more without a team descending on my house from a black helicopter, though.
I once worked at a tiny startup where we were trying to sell a dataset to GS. Before we could even send a sample, they sent over some boilerplate forms for us to sign. I remember two distinct stipulations - anything we sent them was immediately and forever their property, AND they had the right to drug test any of our employees. We ended up not signing so there was no deal. My boss said it was their way of getting rid of us.
Once, I saw the employees get frustrated in a nominally high security environment with the contractors not being able to even directly access their dev env without scheduling a visit days in advance (the contractors were all hours away) while having the employees stare over their shoulder the entire time. They were trying to debug an issue that sometimes came up in dev/prod but not in the contractors' local copy of dev.
The employees set up TeamViewer on their "dev" server and promised the contractors in writing that this all had sufficient permission, this was a dev server, and the credentials on the dev server were not going to get them into anything else that might be troublesome by mistake.
The last of those three statements was accurate. As you might imagine, TeamViewer on a nominally tightly controlled network was not even in the same hemisphere as acceptable...
...and while debugging, the contractors made an incompatible DB schema change on dev to see if it fixed something, only to get a nasty surprise when the employees ran to them within an hour or so asking what they had done, because by "dev" they meant "prod".
I don't really have a great moral for this story, other than maybe "netsec/infosec teams need to actually work with other teams and not just be opaque sources of fiats, or people are going to work around them to get things done instead of trying to work with them, and that's going to end poorly for everyone."
This is paradoxically because historically everything was wide open to anyone so access-control and such isn't super fleshed out for most apps behind the proxy. Random internal app X could have been conceived and built with very little oversight and opening it up to a rotating cast of temporary workers is seen as an unnecessary risk. Broadly used apps (e.g. the bug ticket system) tend to have app-level security-controls and are not blocked by the proxy for contractors.
It was hugely influential in identifying the frustration of getting things done at Google. In my experience it's even more true now than it was back then, the number of things you have to deal with has just grown. I've been at Google since 2006 and I feel like I'm losing my mind with all the complexity.
Out of genuine curiosity, what keeps you at Google for 15 years despite perceived increase in complexity to getting things done? I'm wondering whether the answer is like "there's a lot of complexity, but I like the work I do more" or "I like the people more" or some other reason.
You know, I've made similar remarks on HN before, but you are the first person to actually edit their comment. Amazing. Now I feel compelled to edit mine...
This is not true, Google is no longer capable of changing anything internally, it's like a government in that respect, it can make a new process/program (that might on the surface make some things look better) but underneath the old system is still there.
Well they can launch new semi-independent initiatives under the Alphabet umbrella. They just need to use that to create a web search engine. Something sorely lacking in their current portfolio.
I wish I would have had this video before 2010. I got paged at night every time there was a PCR failover, and I didn’t know what to do with it. This video is better than all the extensive documentation that we had.
That was a major impetus for this video, IIRC. The "MongoDB is web-scale" video went around Google about a month before Broccoli Man and some enterprising Googler figured they could use the same software to make a satire of Google's internal tools.
I had a similar conversation with a heavily-intoxicated MongoDB sales guy in a diner at 1AM after the second day of KubeCon 2019. My concerns were primarily around data consistency issues during denormalization and lack of schema. H pitch was essentially "Who cares?! I'm getting [three-letter agency] to move _everything_ to Mongo because it's so cheap and easy! It's all just JSON! Why does it need a schema?!"
He probably made more than I did that year so maybe he has a point ¯\_(ツ)_/¯
Hey... those of us that worked on Google's internal Bigtable service worked very hard so you didn't have a file to a ticket to set up replication between your Bigtable cells.
A system for alerting based on time-series data, with its own rule language. The language (along with many others) required authors have demonstrated they can adhere to the style guide by going through a process to obtain readability.
It's a language and supporting infrastructure for collecting and querying time series data for monitoring.
It was replaced a long time ago by a new system called monarch, but a few holdouts will probably continue using borgmon until the heat death of the universe.
Ah, the laughs (Xoogler since 2020).
It was a lot easier, at least last year: you'd use "flex" quota from your PA pool (product area) for Spanner and Borg, write some code for your server, a few configs here and there, and you'd be ready and serving.
That video came out a few years before flex appeared I think at a time they were having a sort of “resource crunch” on the heels of growth spur following the GFC.
It was easier to mint and carve out Colossus quota than e.g. Bigtable. I seem to remember that flex for Borg existed, but only in a few locations with enough capacity to back it. You couldn't just retrofit it in clusters where existing, large customers were already granted and using most of the quota.
It wasn't that there was a crunch — that had always existed. There just wasn't all the tooling to implement anything like flex. At least this video was made after "buying Borg quota" was a normal thing. Before it, you had to "buy" regular machines and donate/assimilate them into Borg. Then after X days you'd receive your quota, minus a Borg "tax" of 10% to cover borglet and system daemons' overhead.
Hah, I think that at least at the beginning, the tools wanted to steamroll the Bigtable service, too. As the one that had to vet the changes every week, I had to go explain that it wasn't really neither our quota nor our usage.
For Colossus, sometimes large new clusters would go from very idle to super busy and under provisioned within days or few weeks, AFTER the steamrolling was already done, just because large users moved in. We then had to bring up new curators (masters) quickly, many times with quota that didn't exist. Quite often, when out of options, I ended up taking precious p360 quota from the storage monitoring user to mint production resources for Colossus. Fun times.
So before Borgmon existed, the quota unit was basically an entire machine? No virtualization?
Oh wait, this was probably 2008, VMWare had only just figured out JITed software virtualization a couple years prior. Makes a bit more sense now. And now the containerization thing makes a tad more sense: Google basically skipped over the "use QEMU" (or more recently, Firecracker) phase everyone else is now going through.
Borg was started in 2003 and by 2006-2007 was already the default way to run things, even though isolation wasn't perfect then. I think it started with chroot jails, then fake NUMA, then cgroups, which were written for it. It took years before all of web search moved to dedicated Borg machines (their quota belonged to you and nobody else could run on them) and, eventually, shared ones.
Ah, see, should've given 'em to us! Big rectangular state, you know, #3 machine owner at the time. We didn't charge overhead, probably because it never occurred to us to do it.
People brought machines, we gave 'em quota. Easy enough.
I know the service! The folks in NY had the state flag by their cubicle. My team in 2007-2011 was probably one of the largest users and I think I donated machines to y'all in the old Groningen cluster, before quotas were automated. GFS didn't charge for chunkserver overhead, either, and that mistake took years and lots of pain to fix...
About six years ago I had a resource manager deny me a database instance the very same day it became available for flex in another product area. I tried to "Hey Mister" resources from someone in that group to no avail. Eventually I wrote a high-durability key-value store on top of our source control system and told them they could give me my database or I'd be deploying to prod.
>Eventually I wrote a high-durability key-value store on top of our source control system and told them they could give me my database or I'd be deploying to prod.
That's diabolical. I knew Google should never have dropped the "do not be evil" thing
Ok, replicate it to a second instance. If it breaks we'll page you to fix it.
The funny thing is, for important stuff, we ended up doing similar things to what you see in this video, but for unimportant things, we didn't. I think it was a better system, and it was amusing when we hired people from Google who were confused by the lack of process and approvals.
> I want to serve 5TB of data. Ok, spin up an instance in AWS and put it there... it was amusing when we hired people from Google who were confused by the lack of process and approvals.
In 2003, at a time in Amazon's history when we were particularly frustrated by our speed of software engineering, we turned to Matt Round, an engineering leader who was a most interesting squeaky wheel in that his team appeared to get more done than any other, yet he remained deeply impatient and complained loudly and with great clarity about how hard it was to get anything done. He wrote a six-pager that had a great hook in the first paragraph: "To many of us Amazon feels more like a tectonic plate than an F-16."
Matt's paper had many recommendations... including the maximization of autonomy for teams and for the services operated by those teams by the adoption of REST-style interfaces, platform standardization, removal of roadblocks or gatekeepers (high-friction bureaucracy), and continuous deployment of isolated components. He also called... for an enduring performance indicator based on the percentage of their time that software engineers spent building rather than doing other tasks. Builders want to build, and Matt's timely recommendations influenced the forging of Amazon's technology brand as "the best place where builders can build."
The "approval paralysis" thing happens at a lot of companies, large and small, not just GiantTech. It creeps up on you slowly: 1. A big problem happens that gains the attention of leadership. 2. The problem is root-caused to some risky thing an employee did trying to accomplish XYZ. 3. To correct this, a process is put in place that must be followed when one wants to do XYZ, and (critically) gatekeepers are anointed who must approve the activity. 4. These gatekeepers are inevitably senior already-busy people who become bottlenecks. Now we can't do this critical thing without hounding approvers. 5. Some other big problem happens and the above cycle starts all over again.
Before you know it, every even slightly risky task you need to do through the course of your job requires the blessing of approvers who are well-intentioned, but all so overloaded they don't even answer their E-mail or chats. They sometimes need to be physically grabbed in the hallway in order to unblock your project. Progress grinds to a halt and it still has not stopped production problems--just those particular classes of problems that the approval processes caught.
EDIT: Not sure what the right solution is, but it must be one that doesn't rely on a particular overloaded human doing something. Maybe an automated approval system that produces a paper trail (to help with postmortem and corrective action later) and ensuring all changes can be rolled back effortlessly. Easier said than done, obviously.
It's actually super related, given that (at least in the medical software sector) you won't get anything approved by the FDA before spelling out the entire software development operation in processes.
I've worked at big companies that are mired in process because they would rather spend more time crossing i's and dotting t's than risk breaking something. I can see why.
And I've worked at smaller companies where the clients are small and it's easy to fix things that break. Move fast and break things at a small scale maybe.
But how do you grow to be a big company and still operate like a small company? I can't seem to see an answer.
Solution to such org woes, in part, is discussed by Clayton Christensen in his work, The Innovator's Solutionhttp://web.mit.edu/6.933/www/Fall2000/teradyne/clay.html: Even after correctly identifying potentially disruptive technologies, firms still must circumvent its hierarchy and bureaucracy that can stifle the free pursuit of creative ideas. Christensen suggests that firms need to provide experimental groups within the company a freer rein. "With a few exceptions, the only instances in which mainstream firms have successfully established a timely position in a disruptive technology were those in which the firms' managers set up an autonomous organization charged with building a new and independent business around the disruptive technology." This autonomous organization will then be able to choose the customers it answers to, choose how much profit it needs to make, and how to run its business.
---
Amazon and Cloudflare are good examples of big-orgs trying their best to implement late Prof. Christensen's ideas.
Andy Jassy on Amazon's approach to innovation: https://www.hbs.edu/forum-for-growth-and-innovation/podcasts...: And then if we like the answers to those first four elements, then we ask, can we put a group of single threaded focused people on this initiative, even if it seems like they're overwhelming it with strong senior people, if you try to add really busy people do the existing business and the big new idea, they will always favor the existing business because it's surer bet. So we want to peel people away from the existing business and put them just on the new initiative.
Pace of innovation at Cloudflare https://blog.cloudflare.com/the-secret-to-cloudflare-pace-of...: ...it is not unusual for an initial product idea to start with a team small enough to split a pack of Twinkies and for the initial proof of concept to go from whiteboard to rolled out in days. We intentionally staff and structure our teams and our backlogs so that we have flexibility to pivot and innovate. Our Emerging Technology and Incubation team is a small group of product managers and engineers solely dedicated to exploring new products for new markets. Our Research team is dedicated to thinking deeply and partnering with organizations across the globe to define new standards and new ways to tackle some of the hardest challenges.
Instead of appointing a senior eng to be approver, task the same senior eng with writing down his decision criteria (as text or where it makes sense even as code).
This has advantages for everyone:
1. It lets the engineers who need approval move at their own speed, and plan time for it as a predictable work item like any other, instead of depending on an approver for whom the approvals will usually be at a lower priority and mid-sprint.
2. For the approval policy writer, it turns this into a one time effort with a defined scope that can be planned and prioritized in his/her own backlog, instead of open ended toil that can come at any time, take any time, and not clearly relate to their own current priorities.
3. For the company, writing down the policy brings consistent decision making.
Obviously this requires trust that employees can and will say "no, can't do" when they're tasked with something that is not approvable, which can be culturally difficult (business and otherwise). Checklists (literally a list of checkboxes to click on, "I confirm that...") can help with this.
(As an example of writing down the policy as code: that's any CI/CD pipeline. But it's not limited to engineering decision making - for example, we're using a well-known open source license management tool that promises auto-approval for open source library use depending on policies configured by legal. This works moderately not so well because this particular tool is not great; the idea is sound. We still made it work: now legal wrote down their policies, trained a large number of engineers on them and those are now empowered to make decisions.)
There are many, but the problems are more subtle than this video really gives credit for.
I worked at Google at the time this video was made, and empathized (in fact I had been an SRE for years by that point). Nonetheless, there are flip sides that the video maker obviously didn't consider.
Firstly, why did everything at Google have to be replicated up the wazoo? Why was so much time spent talking about PCRs? The reason is, Google had consciously established a culture up front in which individual clusters were considered "unreliable" and everyone had to engineer around that. This was a move specifically intended to increase the velocity of the datacenter engineering groups, by ensuring they did not have to get a billion approvals to do changes. Consider how slow it'd be to get approval from every user of a Google cluster, let alone an entire datacenter, to take things offline for maintenance. These things had tens of thousands of machines per cluster and that was over a decade ago. They'd easily be running hundreds of thousands of processes, managed by dozens of different groups. Getting them all to synchronize and approve things would be impossible. So Google said - no approvals are necessary. If the SRE/NetOps/HWOPS teams want to take a cluster or even entire datacenter offline then they simply announce they're going to do it in advance, and, everyone else has to just handle it.
This was fantastic for Google's datacenter tech velocity. They had incredibly advanced facilities years ahead of anyone else, partly due to the frenetic pace of upgrades this system allowed them to achieve. The downside: software engineers have to run their services in >1 cluster, unless they're willing to tolerate downtime.
Secondly, why couldn't cat woman just run a single replica and accept some downtime? Mostly because Google had a brand to maintain. When she "just" wanted to serve 5TB, that wasn't really true. She "just" wanted to do it under the Google brand, advertised as a Google service, with all the benefits that brought her. One of the aspects of that brand that we take for granted is Google's insane levels of reliability. Nobody, and I mean nobody, spends serious time planning for "what if Google is down", even though massive companies routinely outsource all their corporate email and other critical infrastructure to them.
Now imagine how hard it'd be to maintain that brand if random services kept going offline for long periods without Google employees even noticing? They could say, sure, this particular service just wasn't important enough for us to replicate or monitor and the DC is under maintenance, we think it'll be back in 3 days, sorry. But customers and users would freak out, and rightly so. How on earth could they guess what Google would or would not find worthy of proper production quality? That would be opaque to them, yet Google has thousands of services. It'd destroy the brand to have some parts that are reliable and others not according to basically random factors nobody outside the firm can understand. The only solution is to ensure every externally visible service is reliable to the same very high degree.
Indeed, "don't trust that service because Google might kill it" is one of the worst problems the brand has, and that's partly due to efforts to avoid corporate slowdown and launch bureaucracy. Google signed off on a lot of dud uncompetitive services that had serious problems, specifically because they hated the idea of becoming a slow moving behemoth that couldn't innovate. Yet it trashed their brand in the end.
A lot of corporate process engineering is like this. It often boils down to tradeoffs consciously made by executives that the individual employee may not care about or value or even know about, but which is good for the group as a whole. Was Google wrong to take an unreliable-DC-but-reliable-services approach? I don't know but I rea...
(I've never worked at google, and maybe this isn't a problem anymore however) It seems like the "solution" here would be to do for Infra what Go did for Concurrency - build an abstraction with sane defaults, and rubber stamp anything that doesn't stray from those defaults. Anything that does - requires further scrutiny.
For example, at the companies where I've been response for infrastructure (admittedly much smaller than google) I've done exactly that (with Kubernetes specific things like PodDisruptionBudgets and defaulting to 2 replicas), and if users use the default helm chart values, they can ship their service by themselves.
They did a lot of stuff like that, but the work to launch a new service wasn't only technical and some of the non automated works was there partly to enforce checkpoints on the other stuff. For example to get your service mapped through the load balancers required you to prove you'd been approved for launch by executives, so the process required filing a ticket. It's probably all different now though.
I should also note that "launch" in Google speak means "make visible to the world". If you only wanted your service to be available for Googlers it was dramatically easier and the infrastructure was entirely automatic with zero approvals being easily possible.
Trust your people! Otherwise don't hire them in the first place.
You want to deploy something to prod? Okay, either call an API or fill out a webform (or message a Slack bot idk) - but the contents will be a checklist of stuff you need to have done.
1. Did you [load test, integration test, whatever]
2. Did your local architect look over and approve the high-level design? (NB: notice how we aren't requiring an architect to sign off, we trust the developer. Because if they lie they're fired lol.)
3. Other stuff. Maybe some taxonomic stuff like tags associated with deployed infrastructure? Swagger endpoints? Go nuts as long as it's stuff actually needed by central planning - the documentation and paper trail aspect is covered here
This is picked up to be ingested into databases, wikis, emails, wherever.
Compare with my last large corp where we had change approval boards where 25+ people sat in a long meeting and essentially just asked if you'd done the above and you'd then be greenlit to go to prod (at the time deployments were pretty manual and required scheduling as well which is obviously suboptimal). I'm just about to move from a small consulting company to a startup/scaleup so it's going to be interesting to see how things move there..
In change management they argue that companies tend to purposely slow down change over time to become more predictable and lock in on the "successful route". That certainly mirrors my experience. The only thing I don't understand is why you hire so many people when you let a few handful people gate everything. You might just as well fire 80% of the workforce.
You cannot have both an organization that fastidiously protects the privacy and security of user data, and one that requires no process to build and launch software. It's just not possible.
Anyway the video is just a joke. I've never worked anywhere where it was as easy to just serve 5TB of static data as at Google. Googlers who want to just host junk under their own authority do not need to shop for quota, set up borgmon, etc.
Right like looking back, they're setting up a production, user facing service. If I want to just store a 5tb blob somewhere, I think that fits in freebie CNS, so I don't even have to provision resources, I just cat the file or whatever (granted, 5tb was a bit bigger 10 years ago).
Having a rule that "your user-facing service needs to be replicated" is a good rule. Replication being difficult was the problem.
Automate as much as possible. Approval gates are there to prevent obvious issues from continuing down the pipeline. If you can automate checks for known issues that you want to prevent from happening, then you should be able to add it as a test step. Then in the catch, log why it failed and point the dev at documentation.
When I first started at Google I got things done a lot faster because I didn't know all those rules existed and nobody stopped me. My service was still plenty fast & reliable.
Eventually it all got rewritten by other people to do things properly like the video says.
I managed to deploy a whole system at google that had the ability take down all of google globally by DoS'ing the network, and ran it casually (IE, starting and stopping it when I felt like it, at the capacity I felt like, with the binary versions I wanted) for 3 years.
In retrospect, this was absolutely crazy! The actual visible outcomes were: 1 cluster drained due to heat rising so fast the alerting thought there was a fire, 1 page to an engineer in the middle of the night (sorry discovery-service) and a whole bunch of complaints about CPU stealing that weren't my fault.
Rachel said something similar “My own ‘solution’ to it after far too much thrashing was just to say ‘we cannot get all N types of quota in the same place so we are at the mercy of whatever happens to be available, and if that dries up, we stop running’. Granted, this was for some internal stuff that was seven or eight levels removed from anything that anyone on the outside might ever see, but still, it was stupid and made me feel so dirty. I'm sure my non-solution probably bit someone later. Sorry, whoever.” — https://rachelbythebay.com/w/2021/10/30/5tb/
Isn't that things are supposed to be? Prototype and then polish later. I think teams should work in layer like this, but I guess with big company they're just too lazy and assume you're smart enough and have time to take care of everything.
Right. Time was of the essence with this service, and it was still very reliable and scalable in its original simple form, because it leaned on a lot of existing Google infrastructure. Later on it got upgraded to follow the rules better.
we still had our processes though. Radar was my least favorite, but they replaced the ant eater app with one that was at least partially usable right before i left.
IMO/IME it's the clash between tooling, systems and and processes designed for running long-term highly scalable and reliable services maintained by teams in multiple geographical locations and used by billions of people; and greenfield projects that just want to get things done at an early stage.
Requiring multi-cluster/region, the quota/resource economy system, handling PCRs, code review, readability approval for complex configuration languages (and the existence of such complex languages in the first place) ... all of that makes sense in a vacuum and all were built to handle real problems and are likely written in the blood of a near-miss outage. But it also all comes crashing down on you when you're doing things from scratch for a relatively simple usecase that no-one really designed for.
I can't tell if this comment is implying that my comment is unclear, or if you're agreeing with the first line of my comment.
In either case, though, it's an inside joke precisely because it's more relatable to those who are (or were) inside. In particular, I think it would be most funny to someone who was at Google about a decade ago; when I left Google in 2017 things had already changed enough that this didn't ring quite as true for new hires.
That said, GCP is not very representative of what the internal platform looked like circa 2010. (Or even of what the internal platform looks like now, as far as I know.)
A few years ago there was a Swedish tourist at a hotel where I was on vacation. He had a blue-yellow hat with "SWE" written on it in Courier font. I felt an urge to steal his hat because it looked better than most of the Google-branded swag I got as a Google SWE :)
Or: These are really good points for a visibly-user-facing post-alpha service, but isn't it a bit overengineered for an experimental internal service whose clients can tolerate the risk of occasional downtime?
Is there a page that documents this anecdote? I’d like to link to it next time i use this phrase. Ironically, googling for it doesn’t turn up anything relevant.
Google has officially apologized. The person in question had to take their blog offline due to bad behavior from readers (unrelated to Google AFAIK). Overall, this is a dark chapter. It's also been scrubbed internally. Not obliterated, but you won't accidentally bump into it as a Noogler.
As much as I think people should take responsibility for their own actions, it's probably for the better to let this one rest now. Who caused it is irrelevant at this point, though. We (Xooglers, and Googlers) can take responsibility for our actions, and not continue perpetuating it.
I use the phrase to describe how leveling is not about what junior people think it’s about. I’m not clear about the responsibility you’re talking about.
Yeah, that was my reaction. I get the need for all this reliability/failover, but it's horrible failure of abstraction/separation of concerns.
There's no reason the serving team should have to learn how to do all of those things on the checklist, since it can be done by anyone who's already learned the infra. You're expecting them to learn all kinds of stuff outside of their specialty, when they should be able to kick the app over the wall and let infra ensure that the app is deployed in two separate PCR zones with the failover plan etc, which should itself be mostly automated.
> when they should be able to kick the app over the wall and let infra ensure that the app is deployed in two separate PCR zones with the failover plan etc, which should itself be mostly automated
Not entirely - the developers should actively participate in designing the actual failover scenario and making sure the application can handle that (anything from being okay with some downtime due to the failover happening to designing an actual multi-region multi-master application). Making assumptions like 'infra will handle it' is a great way to not only get unexpected outages (because the developers assumed there would be no downtime because failover is magic, or that writes will never be lost) but to also introduce tensions between teams (because you now have an outside team having to wrangle an application into reliability when the original authors don't give a crap about it).
I get and agree with your point, the tooling and processes should definitely be simplified/automated when possible, and developers deserve a working platform that just works. The whole point of a platform team is to abstract away the mundane to let people do their job. But reliability is everyone's job, not just the infra's team, and developers must understand the tradeoffs and technology involved in order to not design broken systems.
A) It's doing a horrible job conveying it. A dev does need to be concerned on how to handle failover, but only at a certain abstraction level. They should be required to specify something in the form "given server A fails and has to pass to B, what do you do?" That does not require you to know the terminology about PCRs and how to make decisions about which cells (or whatever) to pick on deployment, or avoiding the "gotcha" about making sure the two servers are in different PCR zones.
At that point, it's just following a checklist that needs no knowledge of the specifics of the app, and, to the extent that it's accurately representing how Google was, is indicative of bad processes.
B) Many things should be infra's job, as they're cleanly orthogonal to what dev's are doing. For example, how to apply a security patch to a DB. That's unrelated to the operation of the app.
I do get your point though, and I wouldn't say something like this about e.g. testing (which was the short, "reasonable" part of the video!) -- the devs have intimate knowledge of what counts as passing and failing and should be writing tests, and not 100% passing it over to QA. But that's precisely because such concerns are deeply tied in to the thing they are concerned with. "SQL 3.4.1 vs 3.4.2" is not.
1. There are more engineers making more divergent architectural solutions such that there is never a single place where you can make changes across the group.
2. Failures keep happening, so process is instituted with many checkboxes for engineers to work through.
3. Engineers on the small scale stuff get stack ranked against the engineers on the big scale stuff. Everyone needs to show that they can do the work and are "fungible". This leads to small internal systems having the same operational standard as large public facing systems.
I don't see what that's replying to. Nothing in that list would justify demanding that the app's team have knowledge or preference about which PCR zones to pick and which will just have to be corrected when they inevitably pick the wrong one.
The point is that every team gets to set their own failure modes. I know of multiple tier-1 services which diverge from at least one best practice.
Think of the scenario where a cloud provider needs to evacuate an az. There is no API which would allow the compute team to force migrate tens of thousands of apps and guarantee that they both are not effected and maintain their redundancy guarantees.
Internal services at google are in the same boat. However google knows about the hard edges and forces everyone to deal with all of that complexity - there is no api which the serving team could plug into which will avoid this overhead.
That still at no point requires the application's team to make decisions about which two PCR zones to pick and which cells within it to pick, which [decision] can still be cleanly abstracted away, and would still be a mixing of unrelated concerns, and so your comments are still orthogonal to the point I was bringing up here.
While what you say is true, I think GP is ultimately correct. You can have a system define a convention and allow bypassing it, instead of forcing everyone to start from scratch. In fact, this is the approach that pretty much any modern service at Google will use.
Half a companies lifetime isn't a "<small number> of years ago" for that company. You can't compare tech ecosystems today to those in 2010, so many things has gotten standardized since then, Google was at the forefront back then.
Unlike modern companies Google had to build out everything themselves since nobody had built those systems or even had experience building such systems. That takes time, but today all of the things Google learned is common knowledge in papers and similar.
If you disagree mention one company that had a one button script that abstracted away things like where the data is stored to ensure failsafe, data replication etc, in 2010. I don't think there was any, just the fact that Google made it relatively easy to launch such services, just that you had to manually configure the replication script and the zones your data should be stored in wasn't really a big deal.
> Where does "these are really good points, but why don't we have tooling which sets everything up automatically?" fit on the scale?
My guess it fits nowhere because the L5s don't have the ability to automate it, and the L6s think it's trivial and as it's done sparingly then it doesn't justify the work to do things differently.
More like you aren't going to get promoted for automating someone else's toil. Also, now who's going to support it, better deprecate it since the library changed / got deprecated / it's tuesday.
> More like you aren't going to get promoted for automating someone else's toil.
Lots of people were promoted for automating these things. They built easy to use services, got extra headcount since they became important and climbed the ranks. So not sure why you'd think that.
It may be different at other companies, but at Google building stuff that many other engineers depends on is a major way to get promoted. Of course if you automate something and nobody uses your automation tooling then you wont get promoted, but if your work gets used by basically every new engineer you'll climb the ranks quickly.
And yet it's been a decade since this video and practically everything it mentions is a non-problem now.
No one is spinning up new borgmon instances. Spanner is replicated by default. Only very low level services need to care about PCRs. If you use one of the approved frameworks it will set up practically all the production configuration for you. Basic alerting for your service is automated, just turn it on, picking cells to run in is automated, scaling your service is automated, etc.
Actually getting quota remains a problem... :-p
Anyway I would argue we can and do have nice things, and that has happened precisely through the efforts of a huge number of people at all levels.
Edit to add: of course, there are always new problems to complain about! It's the march of progress after all.
Yes. If someone were to make this video today, it wouldn't be about production jobs and PCRs, it would be about privacy reviews and branding approvals.
That would be 'Xoogler' because Google's engineering and broader corporate culture does not reward work like that and so when you realize that, you leave.
In general, Googlers have very little idea how far behind the rest of the industry they are when it comes to tooling.
I got the impression, based on a blog post by Eric Lawrence [1], that Google's developer tooling was top-notch (except for devs working on open-source projects like Chromium). Did it get worse since 2017, or are you talking about a different kind of tooling?
I think that was more or less the intended response. And ten years on, most of these things are automated. This video was a kick in the pants internally.
These kind of organizational problems happen everywhere, that doesn't bug me. What bugs me is when leadership knows about it and doesn't care. After low-level engineers stick their professional neck out to complain in internal town halls and through feedback forms, and leadership gives some bullshit answer that doesn't address or even acknowledge the problem. It would be less infuriating if they just said "I don't give a shit." It's the weasel words and pretending the problem doesn't exist that infuriates me. A lot of the time it doesn't even take much work at all to begin addressing the issue, like a working group for continuous improvement of highly-painful high-value processes. You don't even have to solve it. Just attempt to address it.
I work at a global corporation with 50,000 employees. Even though I've never been at Google I felt every pain point this video was getting at because our company is trying to implement all of this stuff right now.
"Oh you want to go to production? Here's a list from A-XX stating what you need to accomplish that." Thing is I thought they actually handled this gracefully when I started because lots of requirements were tiered with various criteria you had to meet to move up (mostly for brownie points).
But then one day the Tech Execs lose their minds and decide "everything needs to meet all criteria for every single process." You want to create an S3 bucket to store data? That will be a week of submitting paperwork and another month of meetings and approvals from various teams you've never heard of. Plus you have to register your schema, implement data quality checks, unit tests, regression tests, get a PR and CO approved for your central config change, remediate any CVEs in the tooling that you used, and build all of this using our in-house CI/CD platform we created because we're just soooo special. Now you're allowed to launch. Oh wait, NO because we've put the entire corporation on hold from launching new systems for the last calendar year because we're still trying to agree on the final process everyone needs to follow to go to production.
It's surreal how universally so many orgs makes the same mistake of trying to throw more and more process at problems.
In my previous role, the secdevops groups (matrixed teams) were building custom terraform modules for our devs to use in order to easily deploy compliant AWS infrastructure - and devs could only deploy via terraform/CI-CD. While TF specifically states that custom modules are not meant to be used as wrappers, I thought it was a clever way to try getting security "out of the way" while still enforcing best practices.
"We do not recommend writing modules that are just thin wrappers around single other resource types. If you have trouble finding a name for your module that isn't the same as the main resource type inside it, that may be a sign that your module is not creating any new abstraction and so the module is adding unnecessary complexity. Just use the resource type directly in the calling module instead."
If you write different versions of the terraform modules that do some corporate specific magic, I think that would be okay under this rule. It's when you're writing a module that doesn't do any useful magic that they want you to stop and think.
> It's surreal how universally so many orgs makes the same mistake of trying to throw more and more process at problems.
It's hard to find the right balance. You want a bit of process, but not too much.
But it's one of the hardest problems in the existence of humanity and whoever solves it should probably get all the Nobel prizes available (including peace and chemistry!).
Well we're small, but our development is currently starting to build new products and new extensions to their products. I'm pretty happy that everyone is pretty much onboard with our situation #1.
There are a few hard requirements, but most of the requirements we as operations put up are tied to the guaranteed service level agreement to the customer and possibly overall user count.
If there is just an entirely lax service level agreement there might be no need to invest time in clustering non-trivial applications, or implementing more monitoring than a simple HTTP check. On the other hand, if you're selling some 99.95 24/7 with penalties to a customer, the list of must-dos suddenly grows a lot.
The nice thing of approaching it like this: It allows a gradual increase in operational rigidity and robustness. A product team doesn't hit a wall of requirements for their first productive customer. They rather have to incorporate more requirements as the service becomes more successful. Or they don't if the idea doesn't work.
Literally every point you made applies also at my workplace. The optimist in me hopes we work at the same place, but I fear that your last statement might just be the truth :-)
My team has issues deploying builds to test machines. It's like 15 steps and takes an hour. The tooling is atrocious and recently got even worse.
We eventually found the team responsible for this (the org structure is hard to penetrate because no one answers emails). They said they had no idea anyone was dissatisfied. Then they said that it was a low priority so they didn't care and nothing would be done.
In my experience, you can usually convince an engineer that their stuff has a problem and they need to fix it. But it's often impossible to convince management if they aren't on the hook for user satisfaction.
At Google back then "leadership" might as well not even show up. It was super bottom-up, and _you_, not "leadership" were supposed to identify and fix issues. No "leadership" would stop you, either, at least in most cases. I don't believe that in all my years there anyone ever told me what to do. It was very easy to start projects, shut down projects, get headcount, get resources (if your business case is sufficiently persuasive to others). Not a complete free for all, but certainly _a lot_ more freedom than you'd normally see in companies of that size. And (IMO) people used that freedom and autonomy pretty well.
That kinda deteriorated over time, culminating with Sundar "McKinsey" Pichai, and then went rapidly downhill from there, and now I flat out reject their recruiters, based on the feedback from friends still employed there.
At 2:05 the green dude asks if you think your users are scum and do you hate them.
The funny thing is Google as an org ends up hating their users "accidently" anyway because of their history of pulling the rug under the services/APIs etc.
If the users had properly set up a PCR notification about the change and registered it to a bigdata instance then they would never be surprised about service discontinuations. The moral of the story is that you can't fix stupid users. /s
I’m so confused, isn’t this just like basic highly available infrastructure mixed with a toxic SRE culture?
I want to serve 5TB!
Okay grab two instances in different patching zones, create a bucket in our replicated RADOS storage that can hold your data or create a table/db in our Postgres cluster, write your app with tests, add an entry in to the load balancer, add an entry in our big ole distributed job scheduler if you need cron, and submit a PR against the infra repo to add Prometheus metrics and alerts.
And when your done with that set up CI/CD because you shouldn’t assume that instances are reliable and if you don’t give us the code to do a deploy we can’t recreate your app when the VM goes belly up and we’ll have to page you.
Are people not used to what it really takes to “just run some code?”
1. It is rare for the details of how to actually accomplish each of those steps to be both documented and the documentation made accessible.
2. If you can describe it that succinctly, it really ought to be automated. If it can't be automated... then you left something out of your instructions, which goes back to point (1).
Like the steps to do all of this are automated, but we can't read your mind. All of this is basically boils down to submit a PR against some repo that says "there shall be two instances in these regions, there shall be a database in this cluster, there shall be a bucket with this name, etc etc" that the SRE team reviews and merges, which triggers an infra deploy.
People with HA production experience can easily vibe with points made by Broccoli Man. Yes, these things make a lot of sense if you actually want to get code running reliably, especially at scale (organizational and userbase).
But we must not forget how this can look from the point of view of someone who hasn't had to deal with a page due to an entire datacenter going offline, who's not aware of all the hundreds of small things that can go wrong by doing the 'obvious' thing. I think the video is more of a way to poke fun at the optics of this (and some of the overly arcane stuff involved), rather than at the idea of high availability being useless. At least that's how I've always felt about it, a way to remind SREs to respect their internal users (simplify! automate! document!) and that what makes sense to them might look ridiculous to others.
It totally makes sense for Gmail, but at Google "serve 5TB" means something like sort your manager's inbox, something that someone somewhere has an interest in doing, or trying, but of no real consequence for failure.
IIRC, the impetus for Jon Orwant creating the video was him wanting to make 5TB of data publicly available (the US patent dataset? it was before my time) and all the hassles that were involved.
As a friend of mine explained why she left Google a year or so ago, "I got tired of emailing 30 people to try to figure out who owned a single variable."
A TL I worked with once had a simple but effective strategy for that:
"Remove it and see who complains."
I did that (with the impenetrably named "PrefetchExperiment", last touched by a branch that lost previous file history in 2007). Turned out it was the source data for Google's DNS to figure out how to route queries to the lowest-latency datacenter, based on their geographic location. In about a month, it would've taken down all Google services. Oops.
It was a very effective way of figuring out who owned the variable and writing a big long comment explaining what it's there for and which team to contact before changing it, though.
Facebook's recent "DNS problem" was a process for checking routing failover capacity on the backbone for maintenance ended up taking down the backbone links. As a result of the servers being disconnected from the backbone they pulled their BGP advertisements since they considered their location to be unhealthy (no connection to the backbone).
FB's problem was the lack of routing reachability on its backbone triggering the lack of routing reachability information being sent to the larger internet, this in turn caused problems for DNS not the other way around.
> What exactly are these "peer bonuses"? Is it real? Is it what I think it is?
Each month, you can nominate another employee for a small bonus. This is designed to be given to coworkers who have gone above and beyond what was expected from them.
> Do people actually use them as bargain chips?
From my experience it's so over-the-top absurd that it would be difficult to have someone interpret such an offer as anything other than a joke or a meta-joke.
Yes they are real, it's in the low hundreds of bucks range. They must be approved by the recipients manager. Its also limited how many a employee can send (but the limit is fairly high). There is also "kudos" which comes with no money, but has no limits or approvals required.
They are intended to be used for going above and beyond, not for stuff that falls within the scope of ones job. Using them as bargaining chip is explicitly against policy.
People don't use them as bargaining chips most of the time--it is explicitly against policy. I'm sure it happens some times.
What they do do is send one when someone else does something nice (like fix a bug from a project they have left or whatever else). If you ever need something similar again, the person you peer bonused has warm fuzzies about the experience and a hint that they might get it again.
People also use peer bonuses during perf time to demonstrate that the work they are doing impacts other people enough for a somewhat uncommmon thank you.
You can send a small, semi-official "thanks for a job well done" to someone else and it comes with a few bucks attached. People joke about using them nefariously (as people tend to joke), but I've only seen them used appropriately.
544 comments
[ 4.2 ms ] story [ 396 ms ] threadThis video was hugely influential on changing the way Google does internal tools and operations.
Now over multiple changes, it used to require one fairly big one. It's still a pain in the languages that require it--which is all the main ones, but very few of the niche ones.
Many other things changed as well. Much of what the video complains about got automated and better documented. But the company has grown so much, and the product lines have diversified so dramatically, that there are still plenty of places to complain about the overhead.
borgmon was a truly weird system.
Putting people in charge of formatting/style is just an excuse for wasting time bikeshedding, either the code is wrong and a tool can tell you, or its not wrong.
The only one that sounds more difficult to codify is telling people of the existence of duplicate functions. But as someone who contributes to the linux kernel, I can tell you right now that the only way that works reliably is to have a very large pool of reviewers. Very experienced engineers frequently miss what people are doing in other parts of the source base, the name might not be what they expect, etc, etc, etc. In the case of linux there are a fair number of duplicates, or similar functions, and people write coccinelle patches to replace them on a fairly regular basis after they have been in the kernel for years.
So, I doubt giving someone a formal gatekeeper flag, really helps vs just having wider change review.
No tool like that tells you if returning a bool instead of an enum is appropriate here, or that a reference vs a pointer makes more sense given the rest of the code.
I'm sure a clever machine learning algorithm could figure that out with a corpus as large as Google's. Maybe. But no tool like that works today.
And not strangely at all, Google does accept "what clang-tidy does" as the canonical way of formatting text. But readability at Google is far more than just formatting.
Readability is frustrating and annoying, but more than just lint.
Naming isn't very language-specific though [*], and poor naming should be flagged in coding interviews. Any Google engineer should generally have good naming, and should be able to critique others' naming in any language.
[*] Granted there are some language-specific naming conventions, like ! and ? in Ruby methods. But I'd say the majority of languages don't have important naming conventions, and those that do often have lints for them.
I'm not going to turn a candidate away because they use shorthand names when writing code under time pressure on a whiteboard.
Exactly! And Readability process is exactly the training Google uses to ensure someone knows how to look for this issue, and that someone actually is looking for this issue in code reviews.
There is plenty about the Readability process I don't like (particularly that one has to get it through a fairly artificial process), but checking for things like the above (which I just use as one easily understood example) isn't something that happens automatically in an organization as large as Google, with as many engineers as Google.
A common story on HackerNews is just how bad the average programmer is, how crummy code they produce. Taking steps to ensure they do it better is a good thing.
We can argue about the process, for sure, and there are many things I would love to change about it. But "yeah, that should be done in code reviews" is exactly the idea here.
Very true. Readability can't help with that, nor is it designed to. It's mostly there to help novices and new hires. Experienced engineers already have readability themselves so they don't need this extra review.
Small note: readability isn't a test or quiz you take (asterisk). It's obtained by merging code in the language you want readability for. If you merge code for a language often and the reviewers have very few style-based questions for the code then you will get readability fairly quickly.
> The only one that sounds more difficult to codify is telling people of the existence of duplicate functions. But as someone who contributes to the linux kernel, I can tell you right now that the only way that works reliably is to have a very large pool of reviewers. Very experienced engineers frequently miss what people are doing in other parts of the source base, the name might not be what they expect, etc, etc, etc.
A better example would be knowing when you should use `const std::string&`, `std::string_view` or `char*`. Example: https://abseil.io/tips/1
The best readability advice I have recieved has been:
1. Direct "I was confused by X" or "The recommended way to do A is using B", etc
2. Reasoned: "std::string_view is more efficient and clearer in intention than char ptr, it also improves type safety as it is read only and clear about ownership"
3. Linked to source material where examples are given totw or other examples in the code.
I mean, given the halting problem alone, and the absence of general AI, yes, I'd say that...
I'm not sure what to tell you otherwise. I've worked at a place with such strong coding style guidelines, and even though I personally did not agree with every individual point, I accepted the tradeoff because being able to count on consistency in a very large code base was extremely helpful.
It's more about good usage of idiomatic language constructs, which still requires good human judgement to evaluate.
Ever heard of COBOL?
Because readability has always been in the eye of the beholder, and codifying it makes it even worse.
I don't know of any research off hand, but I'm pretty sure the industry consensus is that good identifier names improve the quality of the code (Go style notwithstanding.) Readability is one way to training engineers to do it.
This is empirically false. Consistency, even if it is unfavorable to your preferences, is superior to inconsistency. So a codified set of best practices is better than none at all.
There are part of Google's style guides that I would change if I could, but I also prefer having a style guide (and one that goes beyond things that are lintable) than none at all, because consistency across the codebase means that I can usually understand code at a glance, or if not, know at a glance that something unusual is happening. (this is in fact precisely the argument in favor of autoformatters like gofmt/black/prettier, but extended to softer concepts that can't always be formatted: consistent style, even if it isn't your favorite, is superior to inconsistent style).
Edit: you can look at Google's C-style guide for some examples, https://google.github.io/styleguide/cppguide.html#Structs_vs...
It isn't possible to statically analyze if a class/struct is a POD or if the methods enforce invariants. But it's often very easy to do so with a human eye. And there's value in the distinction!
Similarly, forcing someone to justify using a power-feature (operator overloading, templates, metaclasses, whatever) can only be done by a human. There may be cases where the power feature is warranted and the benefits outweigh the cost, but a linter can't know that. (and ultimately all of this comes back to: things look consistent, and when things are inconsistent, that's a strong signal that something unusual is happening and you should pay close attention)
Indeed, but at what cost? Every step of added gatekeeping reduces development velocity, and Google does not have a monopoly on high scale services (cough AWS cough). In an ideal world for every full-time dev writing code, you'd have a full-time reviewer whose job it is to simply review that dev's code. But the real world has budgets and deadlines, and humans become discouraged when changes take too long to complete. So is readability worth the added gatekeeping cost is the question.
I'm a bit of a polyglot at Google (I have readability in C++, python, and Javascript/Typescript, and am part-way through the Go and Java processes, which covers basically every popular language at the company), and IMO, the readability process for each language has been a net positive, although it was painful for a while when I had no readability and none of my coworkers did (but there are procedures for getting around that today).
For example, I "donate" my readability, and review random changes from people who don't have readability but need a quick turnaround or aren't yet working to gain readability.
I'm not. I'm not saying that code should be merged without any stylistic commentary; I'm simply proposing that stylistic commentary be provided by other stakeholders in the service you're contributing to, on top of whatever linter or sanitizer is being used. That is, after all, the goal of a human reviewer. To add _another_ layer of gatekeeping is what I'm questioning.
> The impression I've gotten is that a AWS papers over their lack of development gatekeeping with unreasonable oncall loads.
I don't agree with this perception at all. AWS spends a lot of time building frameworks and libraries which attempt to minimize human gatekeeping to only the necessary areas. So think retries, backoff, timeouts, circuit-breakers, etc. Instead of adding gatekeeping at the code level (with things like code style), instead gatekeeping is added at the architecture and operations stages, which is where most of the work of keeping a production level service up, available, and fast. Operational level gatekeeping in the bounds of a given organization is a much more neatly constrained problem than stylistic gatekeeping and this gatekeeping happens on a per-service basis rather than a per-CL/review basis. This keeps overhead a lot lower.
My impression from being outside of Google (and having a Xoogler for a partner) is that Google loves bureaucracy and process, and that engineers that continue to stay at Google are fine with navigating these processes. Its answer to problems is to add more bureaucracy, another item for the checklist that a person/committee/reviewer needs to check for. That reflex is hard to shake, IMO.
I guess I'm not following the distinction. For most people, that's exactly what happens. Either you, or the someone on your direct team, will be the person with readability for your code. The readability process just ensures that you don't end up with silos where style diverges too much, and that the stakeholder who proposes stylistic commentary has a baseline knowlege of linguistic best practices.
With few exceptions, the thing people complain about is attaining readability themselves, not getting changes approved.
> I don't agree with this perception at all. AWS spends a lot of time building frameworks and libraries which attempt to minimize human gatekeeping to only the necessary areas. So think retries, backoff, timeouts, circuit-breakers, etc.
I think we're talking a bit past each other. These things are all table stakes defined by frameworks and updated by centralized processes as best practices change. Human reviewers, for the most part, are only going to say "why are you diverging from the default".
But importantly, that sidesteps my statement entirely, which, rephrasing, is that amazon carries along more technical debt than Google. Whether that's a good long-term business decision remains to be seen, but the impression I get from ex-amazon friends is that maintainability is less prioritized, and maintainability is one of the goals of the readability process. Global consistency helps anyone understand and improve anything else, and to an extend avoids haunted graveyards (though...not entirely).
> instead gatekeeping is added at the architecture and operations stages, which is where most of the work of keeping a production level service up, available, and fast. Operational level gatekeeping in the bounds of a given organization is a much more neatly constrained problem
I'm not sure what you mean, but if anything, this sounds more bureaucratic.
> My impression from being outside of Google (and having a Xoogler for a partner) is that Google loves bureaucracy and process, and that engineers that continue to stay at Google are fine with navigating these processes. Its answer to problems is to add more bureaucracy, another item for the checklist that a person/committee/reviewer needs to check for. That reflex is hard to shake, IMO.
FWIW I don't get this impression. It's honestly difficult for me to tell what you even are referring to. Like, other than readability and promo, there's very few processes or bits of bureaucracy that are consistent across the company, and those two have gotten decidedly more streamlined since I joined the company.
Well okay that's half true, I can think of a bunch of places where bureaucracy has increased, but they're all spurned by legislation.
Separating these two out, even if one person may fulfill two roles, IMO adds to bureaucracy that I don't care for. Ramping up onto a new language sounds like a pretty fraught experience if nobody on your team has readability and the service you're contributing doesn't have a member who has the time to actually work with you, especially if your work is not relevant to theirs. They even touched upon this in the linked video. The whole idea that you can enter a bureaucratic catch-22 at a company seems like an anti-pattern to me, a moment to stop and think; a "modern" engineering organization should try its utmost to accelerate development, with only as many checks as needed for safety and no more.
> With few exceptions, the thing people complain about is attaining readability themselves, not getting changes approved.
That's what I mean by bureaucracy. Even having "another eye" on the code should work. Having a certification process for readability is even more bureaucracy.
> But importantly, that sidesteps my statement entirely, which, rephrasing, is that amazon carries along more technical debt than Google. Whether that's a good long-term business decision remains to be seen, but the impression I get from ex-amazon friends is that maintainability is less prioritized, and maintainability is one of the goals of the readability process. Global consistency helps anyone understand and improve anything else, and to an extend avoids haunted graveyards (though...not entirely).
Ah, that makes a lot more sense. In my career I've never worked at a place that prioritizes code style that much, and I've worked at other high scale places in the past. This seems like a Google specific need. You're right that global consistency helps to save off "there-be-dragons" codebases, but I feel like this is a case of YAGNI. Unless you have engineers constantly cross-cutting across the company, most engineers learn the style guidelines of a team/product/service by ramping up on the team. The optimization for global consistency seems premature to me.
> FWIW I don't get this impression. It's honestly difficult for me to tell what you even are referring to. Like, other than readability and promo, there's very few processes or bits of bureaucracy that are consistent across the company, and those two have gotten decidedly more streamlined since I joined the company.
But promos drive _so much_ of the company culture, at least from what my partner tells me. And the video in the OP certainly makes it sound like Google has lots of bureaucracy, though I'm not sure how many of the requirements outlined in the video were there for hyperbole or not. But especially for services that have low SLOs, it makes no sense to put thought into failover, evacuation, or anything like that.
Keep in mind this video is ten years old. While yes, trying to write something from scratch in a language no one on your team has any experience in can be fraught (although this raises other questions: what exactly are you doing, in every case I've used a new lang, my team has been able to find people to review if needed), even in that case, the organization has help for you (see again my prior comment about "donating" readability, and my understanding is that the readability granting process actively prioritizes people who "need" the readability because they have few potential reviewers).
> That's what I mean by bureaucracy. Even having "another eye" on the code should work. Having a certification process for readability is even more bureaucracy.
I think initially the primary driver of readability is C++ (but Java + Guice also...needs it). Take a look at https://abseil.io/tips/. That's 70 tips, which is around a third of the internal ones. I have C++ readability and have internalized, some of those. The readability granting process ensures your code is reviewed by someone who knows all (or nearly all) of those tips, and will recognize when you're doing wrong things in your code and help explain how or why you can improve. When I personally went through the C++ readability process, I got mentorship on how to fix bugs in my code, some of which a peer on my team would have caught, and some of which they wouldn't have. I can now recognize those bug prone patterns (which are hard to lint for, in my case they mostly had to do with parallelism) and know about tools to debug them (msan and asan, which an experienced C++ user should know, but on my team of people who had basically no C++ experience at Google? Nah).
That follows into other languages. Being granted readability means you have a familiarity with the language that suggests you'll be able to effectively mentor others and not mislead them. You don't have that otherwise.
> Unless you have engineers constantly cross-cutting across the company, most engineers learn the style guidelines of a team/product/service by ramping up on the team.
Google uses a single repo and builds stuff from HEAD. This has up- and downsides, but one of the upsides is that its very easy to investigate unusual behavior in your dependencies. I was doing something similar just this week, and fixed obscure bugs in like 7 other teams tools that were causing downstream problems that those teams weren't aware of. No bureaucracy. I found the bug, fixed them, and sent the change to the owning team. Imagine other situations, where I'd need to maintain a fork, or file a bug to have them fix it, or coordinate an update with them.
You'll pay the cost no matter what, choosing to do so in a way that also provides value and mentorship makes a lot of sense to me.
> But promos drive _so much_ of the company culture, at least from what my partner tells me. And the video in the OP certainly makes it sound like Google has lots of bureaucracy, though I'm not sure how many of the requirements outlined in the video were there for hyperbole or not.
I mean the entire thing i...
In any case, readability will comment on stuff that cannot easily be quantified, such as when to use a certain object hierarchy or dependency injection, etc...
It was studied for multiple programming languages Google uses.
It was done, at my request, by the engineering productivity research team, who are experts in this kind of research - you can find public papers they publish[1].
For background: I was the person responsible for production programming languages, and I did this precisely because i did not feel at the time there were recent good studies as to whether readability was really worth the cost.
The answer is "yes, it is".
There is an upfront cost, but readability has a meaningful and net positive effect on engineering velocity.
It is large enough to make the cost back quite quickly.
One could go down the rabbit hole of seeing whether you improve (or make worse) the numbers by changing various parts of the style guides, but like I said, what is there now, has in fact been studied scientifically.
Honestly, i'm not sure why you think it wouldn't be. You come off as a little immature when you sort of just assume people have no idea what they are doing and don't think these things through. You are talking about a company with 60k+ engineers. Every hour of all-engineer time you waste is equivalent to having ~30 SWE do nothing for a year.
Maybe there are companies happy to do that. I worked at IBM for a few years when i was much younger ;). All I can say is that as long as i'm involved in developer tools at Google, i'm gonna try not to waste my customers time.
[1] I say this in the hopes you don't go making assumptions about whether the studies were done properly. They were properly controlled for tenure, number of reviews, change size, etc.
[1]: https://research.google/pubs/pub47025/
[2]: https://research.google/pubs/pub43835/
Note: I'm not a Googler nor a Xoogler, but my partner worked at Google for some years and so I've heard her complain a lot about Google and readability.
FWIW, these can be the same person - if the have appropriate readabilities. The problem is that it's not easy to get readability in the first place, so occasionally teams have to look for people outside.
And that's an acceptable trade-off for a different sized company. Google arguably has the biggest centralised codebase in the world, and simply has different requirements.
As other have said, most of the overhead is from attaining readability, not the requirement itself.
AWS does not operate this way nor do many CDNs operate this way nor do ISPs operate this way. There's other high scale businesses out there. Google isn't the only one. Using Google's scale to justify business practices is a self-fulfilling prophecy.
I'm not denying that, but all the same, we're not talking about thousands of separate codebases. It's a single monorepo codebase with consistent styling, the advantage being that people can switch teams or contribute to other Google software under the same style guide.
The overhead isn't a bug, it's a core feature as consistent style makes it that much easier to switch teams.
Let's make it specific. Read https://google.github.io/styleguide/cppguide.html for readability for a language, namely C++. All the things that can be automated, automatic tools have been written for. But, for example, you can't automate "Prefer to use a struct instead of a pair or a tuple whenever the elements can have meaningful names." Because what does it mean for a name to be meaningful?
Are you optimizing for someone who already knows all the project lingo, or someone who doesn't know any of it?
Are your engineers native English speakers?
There are a whole bunch of things which make the perfect variable name frequently less than perfect, and putting project insiders in charge likely yields the opposite result.
Take: https://elixir.bootlin.com/linux/latest/source/mm/khugepaged...
If you don't know what a vma, pte, pfn, compound_page, young pte, huge page, lru, etc your going to be unable to even begin to understand what that code is doing, despite those all being pretty reasonable variable names and actually fairly industry standard concepts. It gets worse as you move to more esoteric topics. Expanding pte to PageTableEntry might help some subset of users, but at the expense of those that work on the code daily. So who do you optimize for? Is it readable if the only people that can read it already know what it does?
Someone with readability in a language, who keeps up with the style recommendations, will generally produce code that is easier to read by other engineers.
Also probably the privacy review could be a bigger bottleneck these days ;-)
But dropping Borgmon readability was the most immediate and obvious. It was basically true that no one had Borgmon readability. The policy was a catch-22: you couldn't get readability for the simple/formulaic Borgmon macro invocations that were encouraged and often sufficient. You could only get it for doing something "clever". I got it by writing fancy borgmon rules to paper over a problem that (in hindsight) I should have solved elsewhere.
Another was easing quota management. IMHO the most unbelievable thing in the video was that after Broccoli Man told Panda Woman to get quota in two cells, she just said "done". Besides the hassle in transcribing what you needed into the request system [1], various types of quota were chronically unavailable where you needed them, even in tiny amounts. In 2010, I kept a critical infrastructure service running by regularly IMing major clients' on-calls asking them to donate 0.1 cpu(!) of their quota in some cell or another when I didn't have quite enough to grow. There was a "gray market" mailing list where people would trade resources they couldn't get through the primary system. But eventually, they built a system that for small services would make the quota just happen for you.
Overall, it was a kick in the pants for the most basic infrastructure teams that made them see how unnecessarily hard this is for their internal customers, prompting them to make small things just happen while keeping large things possible. In any large organization, it's healthy to get this kind of feedback regularly. The actual specific changes and technologies are pretty specific to Google in 2010...
[1] Many people managed this very very tediously with spreadsheets. I eventually wrote a tool to generate the requests based on comparing your intended production config with your current quota.
By the way, do you happen to be the jwb that filed a... creative BT quota request?
AAAAA+ GREAT CUSTOMER WOULD SELL TO AGAIN
I'm only half joking about that too. The other half is that sometimes it is better to let the system collapse under its own weight when that's what is required to convince the right people the system is broken.
I know a Google contractor who was denied access to a lactation room at her office (she's the type of contractor who works full-time alongside regular employees). When she tried to fix that, she got sent into an infinite loop of support tickets being bounced around between different departments claiming it was the other one's responsibility. Literally no one was able to fix that for her. Her manager was on a different continent, and wasn't able to help either.
send them CP
The employees set up TeamViewer on their "dev" server and promised the contractors in writing that this all had sufficient permission, this was a dev server, and the credentials on the dev server were not going to get them into anything else that might be troublesome by mistake.
The last of those three statements was accurate. As you might imagine, TeamViewer on a nominally tightly controlled network was not even in the same hemisphere as acceptable...
...and while debugging, the contractors made an incompatible DB schema change on dev to see if it fixed something, only to get a nasty surprise when the employees ran to them within an hour or so asking what they had done, because by "dev" they meant "prod".
I don't really have a great moral for this story, other than maybe "netsec/infosec teams need to actually work with other teams and not just be opaque sources of fiats, or people are going to work around them to get things done instead of trying to work with them, and that's going to end poorly for everyone."
If they don't, they call it "comfy job with awesome paycheck and not a lot of pressure" :-p
https://cheezburger.com/5821507840/his-and-hers-shower-head
I think the technical term is Golden cage :)
lots and lots of money
And bonus lean startup video: https://www.youtube.com/watch?v=3J9KhpgYVB0
NSFWish; it gets a bit personal around 3:11
He probably made more than I did that year so maybe he has a point ¯\_(ツ)_/¯
"Mongo DB Is Web Scale" - https://www.youtube.com/watch?v=b2F-DItXtZs
The rest does seem about accurate though.
Rumour had it that the Borgmon readability requirement was removed when Sergey saw this video. I don't know if this is true but that's what I heard.
https://sre.google/sre-book/practical-alerting/
It was replaced a long time ago by a new system called monarch, but a few holdouts will probably continue using borgmon until the heat death of the universe.
Oh wait, this was probably 2008, VMWare had only just figured out JITed software virtualization a couple years prior. Makes a bit more sense now. And now the containerization thing makes a tad more sense: Google basically skipped over the "use QEMU" (or more recently, Firecracker) phase everyone else is now going through.
Borg was started in 2003 and by 2006-2007 was already the default way to run things, even though isolation wasn't perfect then. I think it started with chroot jails, then fake NUMA, then cgroups, which were written for it. It took years before all of web search moved to dedicated Borg machines (their quota belonged to you and nobody else could run on them) and, eventually, shared ones.
People brought machines, we gave 'em quota. Easy enough.
So glad to not be doing that any more.
That's diabolical. I knew Google should never have dropped the "do not be evil" thing
I want to serve 5TB of data.
Ok, spin up an instance in AWS and put it there.
I want it production ready.
Ok, replicate it to a second instance. If it breaks we'll page you to fix it.
The funny thing is, for important stuff, we ended up doing similar things to what you see in this video, but for unimportant things, we didn't. I think it was a better system, and it was amusing when we hired people from Google who were confused by the lack of process and approvals.
Quoting from Velocity in Software Engineering https://queue.acm.org/detail.cfm?id=3352692:
In 2003, at a time in Amazon's history when we were particularly frustrated by our speed of software engineering, we turned to Matt Round, an engineering leader who was a most interesting squeaky wheel in that his team appeared to get more done than any other, yet he remained deeply impatient and complained loudly and with great clarity about how hard it was to get anything done. He wrote a six-pager that had a great hook in the first paragraph: "To many of us Amazon feels more like a tectonic plate than an F-16."
Matt's paper had many recommendations... including the maximization of autonomy for teams and for the services operated by those teams by the adoption of REST-style interfaces, platform standardization, removal of roadblocks or gatekeepers (high-friction bureaucracy), and continuous deployment of isolated components. He also called... for an enduring performance indicator based on the percentage of their time that software engineers spent building rather than doing other tasks. Builders want to build, and Matt's timely recommendations influenced the forging of Amazon's technology brand as "the best place where builders can build."
...leading up to the creation of AWS.
Before you know it, every even slightly risky task you need to do through the course of your job requires the blessing of approvers who are well-intentioned, but all so overloaded they don't even answer their E-mail or chats. They sometimes need to be physically grabbed in the hallway in order to unblock your project. Progress grinds to a halt and it still has not stopped production problems--just those particular classes of problems that the approval processes caught.
EDIT: Not sure what the right solution is, but it must be one that doesn't rely on a particular overloaded human doing something. Maybe an automated approval system that produces a paper trail (to help with postmortem and corrective action later) and ensuring all changes can be rolled back effortlessly. Easier said than done, obviously.
But most software isn't that critical.
I've worked at big companies that are mired in process because they would rather spend more time crossing i's and dotting t's than risk breaking something. I can see why.
And I've worked at smaller companies where the clients are small and it's easy to fix things that break. Move fast and break things at a small scale maybe.
But how do you grow to be a big company and still operate like a small company? I can't seem to see an answer.
Solution to such org woes, in part, is discussed by Clayton Christensen in his work, The Innovator's Solution http://web.mit.edu/6.933/www/Fall2000/teradyne/clay.html: Even after correctly identifying potentially disruptive technologies, firms still must circumvent its hierarchy and bureaucracy that can stifle the free pursuit of creative ideas. Christensen suggests that firms need to provide experimental groups within the company a freer rein. "With a few exceptions, the only instances in which mainstream firms have successfully established a timely position in a disruptive technology were those in which the firms' managers set up an autonomous organization charged with building a new and independent business around the disruptive technology." This autonomous organization will then be able to choose the customers it answers to, choose how much profit it needs to make, and how to run its business.
---
Amazon and Cloudflare are good examples of big-orgs trying their best to implement late Prof. Christensen's ideas.
Andy Jassy on Amazon's approach to innovation: https://www.hbs.edu/forum-for-growth-and-innovation/podcasts...: And then if we like the answers to those first four elements, then we ask, can we put a group of single threaded focused people on this initiative, even if it seems like they're overwhelming it with strong senior people, if you try to add really busy people do the existing business and the big new idea, they will always favor the existing business because it's surer bet. So we want to peel people away from the existing business and put them just on the new initiative.
Pace of innovation at Cloudflare https://blog.cloudflare.com/the-secret-to-cloudflare-pace-of...: ...it is not unusual for an initial product idea to start with a team small enough to split a pack of Twinkies and for the initial proof of concept to go from whiteboard to rolled out in days. We intentionally staff and structure our teams and our backlogs so that we have flexibility to pivot and innovate. Our Emerging Technology and Incubation team is a small group of product managers and engineers solely dedicated to exploring new products for new markets. Our Research team is dedicated to thinking deeply and partnering with organizations across the globe to define new standards and new ways to tackle some of the hardest challenges.
---
Also read: Clayton Christensen and Stephen Kaufman on "Resources, Process, and Priorities": https://personal.utdallas.edu/~chasteen/Christensen%20-%202n...
Instead of appointing a senior eng to be approver, task the same senior eng with writing down his decision criteria (as text or where it makes sense even as code).
This has advantages for everyone:
1. It lets the engineers who need approval move at their own speed, and plan time for it as a predictable work item like any other, instead of depending on an approver for whom the approvals will usually be at a lower priority and mid-sprint.
2. For the approval policy writer, it turns this into a one time effort with a defined scope that can be planned and prioritized in his/her own backlog, instead of open ended toil that can come at any time, take any time, and not clearly relate to their own current priorities.
3. For the company, writing down the policy brings consistent decision making.
Obviously this requires trust that employees can and will say "no, can't do" when they're tasked with something that is not approvable, which can be culturally difficult (business and otherwise). Checklists (literally a list of checkboxes to click on, "I confirm that...") can help with this.
(As an example of writing down the policy as code: that's any CI/CD pipeline. But it's not limited to engineering decision making - for example, we're using a well-known open source license management tool that promises auto-approval for open source library use depending on policies configured by legal. This works moderately not so well because this particular tool is not great; the idea is sound. We still made it work: now legal wrote down their policies, trained a large number of engineers on them and those are now empowered to make decisions.)
I worked at Google at the time this video was made, and empathized (in fact I had been an SRE for years by that point). Nonetheless, there are flip sides that the video maker obviously didn't consider.
Firstly, why did everything at Google have to be replicated up the wazoo? Why was so much time spent talking about PCRs? The reason is, Google had consciously established a culture up front in which individual clusters were considered "unreliable" and everyone had to engineer around that. This was a move specifically intended to increase the velocity of the datacenter engineering groups, by ensuring they did not have to get a billion approvals to do changes. Consider how slow it'd be to get approval from every user of a Google cluster, let alone an entire datacenter, to take things offline for maintenance. These things had tens of thousands of machines per cluster and that was over a decade ago. They'd easily be running hundreds of thousands of processes, managed by dozens of different groups. Getting them all to synchronize and approve things would be impossible. So Google said - no approvals are necessary. If the SRE/NetOps/HWOPS teams want to take a cluster or even entire datacenter offline then they simply announce they're going to do it in advance, and, everyone else has to just handle it.
This was fantastic for Google's datacenter tech velocity. They had incredibly advanced facilities years ahead of anyone else, partly due to the frenetic pace of upgrades this system allowed them to achieve. The downside: software engineers have to run their services in >1 cluster, unless they're willing to tolerate downtime.
Secondly, why couldn't cat woman just run a single replica and accept some downtime? Mostly because Google had a brand to maintain. When she "just" wanted to serve 5TB, that wasn't really true. She "just" wanted to do it under the Google brand, advertised as a Google service, with all the benefits that brought her. One of the aspects of that brand that we take for granted is Google's insane levels of reliability. Nobody, and I mean nobody, spends serious time planning for "what if Google is down", even though massive companies routinely outsource all their corporate email and other critical infrastructure to them.
Now imagine how hard it'd be to maintain that brand if random services kept going offline for long periods without Google employees even noticing? They could say, sure, this particular service just wasn't important enough for us to replicate or monitor and the DC is under maintenance, we think it'll be back in 3 days, sorry. But customers and users would freak out, and rightly so. How on earth could they guess what Google would or would not find worthy of proper production quality? That would be opaque to them, yet Google has thousands of services. It'd destroy the brand to have some parts that are reliable and others not according to basically random factors nobody outside the firm can understand. The only solution is to ensure every externally visible service is reliable to the same very high degree.
Indeed, "don't trust that service because Google might kill it" is one of the worst problems the brand has, and that's partly due to efforts to avoid corporate slowdown and launch bureaucracy. Google signed off on a lot of dud uncompetitive services that had serious problems, specifically because they hated the idea of becoming a slow moving behemoth that couldn't innovate. Yet it trashed their brand in the end.
A lot of corporate process engineering is like this. It often boils down to tradeoffs consciously made by executives that the individual employee may not care about or value or even know about, but which is good for the group as a whole. Was Google wrong to take an unreliable-DC-but-reliable-services approach? I don't know but I rea...
(I've never worked at google, and maybe this isn't a problem anymore however) It seems like the "solution" here would be to do for Infra what Go did for Concurrency - build an abstraction with sane defaults, and rubber stamp anything that doesn't stray from those defaults. Anything that does - requires further scrutiny.
For example, at the companies where I've been response for infrastructure (admittedly much smaller than google) I've done exactly that (with Kubernetes specific things like PodDisruptionBudgets and defaulting to 2 replicas), and if users use the default helm chart values, they can ship their service by themselves.
I should also note that "launch" in Google speak means "make visible to the world". If you only wanted your service to be available for Googlers it was dramatically easier and the infrastructure was entirely automatic with zero approvals being easily possible.
You want to deploy something to prod? Okay, either call an API or fill out a webform (or message a Slack bot idk) - but the contents will be a checklist of stuff you need to have done.
1. Did you [load test, integration test, whatever]
2. Did your local architect look over and approve the high-level design? (NB: notice how we aren't requiring an architect to sign off, we trust the developer. Because if they lie they're fired lol.)
3. Other stuff. Maybe some taxonomic stuff like tags associated with deployed infrastructure? Swagger endpoints? Go nuts as long as it's stuff actually needed by central planning - the documentation and paper trail aspect is covered here
This is picked up to be ingested into databases, wikis, emails, wherever.
Compare with my last large corp where we had change approval boards where 25+ people sat in a long meeting and essentially just asked if you'd done the above and you'd then be greenlit to go to prod (at the time deployments were pretty manual and required scheduling as well which is obviously suboptimal). I'm just about to move from a small consulting company to a startup/scaleup so it's going to be interesting to see how things move there..
Anyway the video is just a joke. I've never worked anywhere where it was as easy to just serve 5TB of static data as at Google. Googlers who want to just host junk under their own authority do not need to shop for quota, set up borgmon, etc.
Having a rule that "your user-facing service needs to be replicated" is a good rule. Replication being difficult was the problem.
Manual processes suck for everyone involved.
Matt went on to study theology, and he's started a church community in Scotland: https://www.linkedin.com/in/mattround/
"Leader Company Name: Hope City Church Edinburgh Dates Employed: Sep 2017 – Present Driving a new church start-up."
[1] https://news.ycombinator.com/item?id=21786729
Here's hoping Google doesn't get mad about it - though after 12(ish) years there's really nothing secret in that video.
EDIT: it has been explained to me: https://rachelbythebay.com/w/2021/10/30/5tb/
In retrospect, this was absolutely crazy! The actual visible outcomes were: 1 cluster drained due to heat rising so fast the alerting thought there was a fire, 1 page to an engineer in the middle of the night (sorry discovery-service) and a whole bunch of complaints about CPU stealing that weren't my fault.
Those were the good old days.
Noogler: Haha, this video is so funny!
L4 SWE: (Crying because the video is so true)
L5 SWE: Haha, this video is so funny! I should show it to my interns, this will be a good training for them.
L6+ SWE: Why do people think this is funny? This Broccoli Man guy makes some really good points...
Exactly. This wasn't too relatable, even though I have the GCP Certified Architect cert.
Requiring multi-cluster/region, the quota/resource economy system, handling PCRs, code review, readability approval for complex configuration languages (and the existence of such complex languages in the first place) ... all of that makes sense in a vacuum and all were built to handle real problems and are likely written in the blood of a near-miss outage. But it also all comes crashing down on you when you're doing things from scratch for a relatively simple usecase that no-one really designed for.
In either case, though, it's an inside joke precisely because it's more relatable to those who are (or were) inside. In particular, I think it would be most funny to someone who was at Google about a decade ago; when I left Google in 2017 things had already changed enough that this didn't ring quite as true for new hires.
That said, GCP is not very representative of what the internal platform looked like circa 2010. (Or even of what the internal platform looks like now, as far as I know.)
Its fairly simple to extract the gist of what these systems from the script.
A few years ago there was a Swedish tourist at a hotel where I was on vacation. He had a blue-yellow hat with "SWE" written on it in Courier font. I felt an urge to steal his hat because it looked better than most of the Google-branded swag I got as a Google SWE :)
My life is a waste but the money is too good...
As much as I think people should take responsibility for their own actions, it's probably for the better to let this one rest now. Who caused it is irrelevant at this point, though. We (Xooglers, and Googlers) can take responsibility for our actions, and not continue perpetuating it.
There's no reason the serving team should have to learn how to do all of those things on the checklist, since it can be done by anyone who's already learned the infra. You're expecting them to learn all kinds of stuff outside of their specialty, when they should be able to kick the app over the wall and let infra ensure that the app is deployed in two separate PCR zones with the failover plan etc, which should itself be mostly automated.
Not entirely - the developers should actively participate in designing the actual failover scenario and making sure the application can handle that (anything from being okay with some downtime due to the failover happening to designing an actual multi-region multi-master application). Making assumptions like 'infra will handle it' is a great way to not only get unexpected outages (because the developers assumed there would be no downtime because failover is magic, or that writes will never be lost) but to also introduce tensions between teams (because you now have an outside team having to wrangle an application into reliability when the original authors don't give a crap about it).
I get and agree with your point, the tooling and processes should definitely be simplified/automated when possible, and developers deserve a working platform that just works. The whole point of a platform team is to abstract away the mundane to let people do their job. But reliability is everyone's job, not just the infra's team, and developers must understand the tradeoffs and technology involved in order to not design broken systems.
A) It's doing a horrible job conveying it. A dev does need to be concerned on how to handle failover, but only at a certain abstraction level. They should be required to specify something in the form "given server A fails and has to pass to B, what do you do?" That does not require you to know the terminology about PCRs and how to make decisions about which cells (or whatever) to pick on deployment, or avoiding the "gotcha" about making sure the two servers are in different PCR zones.
At that point, it's just following a checklist that needs no knowledge of the specifics of the app, and, to the extent that it's accurately representing how Google was, is indicative of bad processes.
B) Many things should be infra's job, as they're cleanly orthogonal to what dev's are doing. For example, how to apply a security patch to a DB. That's unrelated to the operation of the app.
I do get your point though, and I wouldn't say something like this about e.g. testing (which was the short, "reasonable" part of the video!) -- the devs have intimate knowledge of what counts as passing and failing and should be writing tests, and not 100% passing it over to QA. But that's precisely because such concerns are deeply tied in to the thing they are concerned with. "SQL 3.4.1 vs 3.4.2" is not.
1. There are more engineers making more divergent architectural solutions such that there is never a single place where you can make changes across the group.
2. Failures keep happening, so process is instituted with many checkboxes for engineers to work through.
3. Engineers on the small scale stuff get stack ranked against the engineers on the big scale stuff. Everyone needs to show that they can do the work and are "fungible". This leads to small internal systems having the same operational standard as large public facing systems.
Think of the scenario where a cloud provider needs to evacuate an az. There is no API which would allow the compute team to force migrate tens of thousands of apps and guarantee that they both are not effected and maintain their redundancy guarantees.
Internal services at google are in the same boat. However google knows about the hard edges and forces everyone to deal with all of that complexity - there is no api which the serving team could plug into which will avoid this overhead.
Edit: It might help to check out my comment here, where I clarify what a dev should vs shouldn't have to worry about: https://news.ycombinator.com/item?id=29085638
Half a companies lifetime isn't a "<small number> of years ago" for that company. You can't compare tech ecosystems today to those in 2010, so many things has gotten standardized since then, Google was at the forefront back then.
Unlike modern companies Google had to build out everything themselves since nobody had built those systems or even had experience building such systems. That takes time, but today all of the things Google learned is common knowledge in papers and similar.
If you disagree mention one company that had a one button script that abstracted away things like where the data is stored to ensure failsafe, data replication etc, in 2010. I don't think there was any, just the fact that Google made it relatively easy to launch such services, just that you had to manually configure the replication script and the zones your data should be stored in wasn't really a big deal.
My guess it fits nowhere because the L5s don't have the ability to automate it, and the L6s think it's trivial and as it's done sparingly then it doesn't justify the work to do things differently.
And this is why we can't have nice things.
Lots of people were promoted for automating these things. They built easy to use services, got extra headcount since they became important and climbed the ranks. So not sure why you'd think that.
It may be different at other companies, but at Google building stuff that many other engineers depends on is a major way to get promoted. Of course if you automate something and nobody uses your automation tooling then you wont get promoted, but if your work gets used by basically every new engineer you'll climb the ranks quickly.
No one is spinning up new borgmon instances. Spanner is replicated by default. Only very low level services need to care about PCRs. If you use one of the approved frameworks it will set up practically all the production configuration for you. Basic alerting for your service is automated, just turn it on, picking cells to run in is automated, scaling your service is automated, etc.
Actually getting quota remains a problem... :-p
Anyway I would argue we can and do have nice things, and that has happened precisely through the efforts of a huge number of people at all levels.
Edit to add: of course, there are always new problems to complain about! It's the march of progress after all.
But the quota issues haven't changed a bit.
In general, Googlers have very little idea how far behind the rest of the industry they are when it comes to tooling.
I am a Xoogler.
[1]; https://textslashplain.com/2017/02/01/google-chrome-one-year...
"Oh you want to go to production? Here's a list from A-XX stating what you need to accomplish that." Thing is I thought they actually handled this gracefully when I started because lots of requirements were tiered with various criteria you had to meet to move up (mostly for brownie points).
But then one day the Tech Execs lose their minds and decide "everything needs to meet all criteria for every single process." You want to create an S3 bucket to store data? That will be a week of submitting paperwork and another month of meetings and approvals from various teams you've never heard of. Plus you have to register your schema, implement data quality checks, unit tests, regression tests, get a PR and CO approved for your central config change, remediate any CVEs in the tooling that you used, and build all of this using our in-house CI/CD platform we created because we're just soooo special. Now you're allowed to launch. Oh wait, NO because we've put the entire corporation on hold from launching new systems for the last calendar year because we're still trying to agree on the final process everyone needs to follow to go to production.
It's surreal how universally so many orgs makes the same mistake of trying to throw more and more process at problems.
Followed by the inevitable ranting about “shadow IT”, AKA the requirements gathering they really should have done.
What do you mean with this?
from https://www.terraform.io/docs/language/modules/develop/index...
If you write different versions of the terraform modules that do some corporate specific magic, I think that would be okay under this rule. It's when you're writing a module that doesn't do any useful magic that they want you to stop and think.
It's hard to find the right balance. You want a bit of process, but not too much.
But it's one of the hardest problems in the existence of humanity and whoever solves it should probably get all the Nobel prizes available (including peace and chemistry!).
There are a few hard requirements, but most of the requirements we as operations put up are tied to the guaranteed service level agreement to the customer and possibly overall user count.
If there is just an entirely lax service level agreement there might be no need to invest time in clustering non-trivial applications, or implementing more monitoring than a simple HTTP check. On the other hand, if you're selling some 99.95 24/7 with penalties to a customer, the list of must-dos suddenly grows a lot.
The nice thing of approaching it like this: It allows a gradual increase in operational rigidity and robustness. A product team doesn't hit a wall of requirements for their first productive customer. They rather have to incorporate more requirements as the service becomes more successful. Or they don't if the idea doesn't work.
We eventually found the team responsible for this (the org structure is hard to penetrate because no one answers emails). They said they had no idea anyone was dissatisfied. Then they said that it was a low priority so they didn't care and nothing would be done.
In my experience, you can usually convince an engineer that their stuff has a problem and they need to fix it. But it's often impossible to convince management if they aren't on the hook for user satisfaction.
That kinda deteriorated over time, culminating with Sundar "McKinsey" Pichai, and then went rapidly downhill from there, and now I flat out reject their recruiters, based on the feedback from friends still employed there.
The funny thing is Google as an org ends up hating their users "accidently" anyway because of their history of pulling the rug under the services/APIs etc.
I want to serve 5TB!
Okay grab two instances in different patching zones, create a bucket in our replicated RADOS storage that can hold your data or create a table/db in our Postgres cluster, write your app with tests, add an entry in to the load balancer, add an entry in our big ole distributed job scheduler if you need cron, and submit a PR against the infra repo to add Prometheus metrics and alerts.
And when your done with that set up CI/CD because you shouldn’t assume that instances are reliable and if you don’t give us the code to do a deploy we can’t recreate your app when the VM goes belly up and we’ll have to page you.
Are people not used to what it really takes to “just run some code?”
1. It is rare for the details of how to actually accomplish each of those steps to be both documented and the documentation made accessible.
2. If you can describe it that succinctly, it really ought to be automated. If it can't be automated... then you left something out of your instructions, which goes back to point (1).
But we must not forget how this can look from the point of view of someone who hasn't had to deal with a page due to an entire datacenter going offline, who's not aware of all the hundreds of small things that can go wrong by doing the 'obvious' thing. I think the video is more of a way to poke fun at the optics of this (and some of the overly arcane stuff involved), rather than at the idea of high availability being useless. At least that's how I've always felt about it, a way to remind SREs to respect their internal users (simplify! automate! document!) and that what makes sense to them might look ridiculous to others.
Someone made a bit of internal-only snark, and "I just want to serve 5TB" became an in-joke for turning easy problems into exercises in frustration.
Some of these things have, actually, been addressed.
The most straightforward is, "I just want do this incredibly simple thing; why is it so hard?"
But there's also the level of, "Googlers are so engineeringly pampered that they think serving 5 terabytes is the equivalent of Hello World."
And then there's another level of, "Well, isn't it? After all, this is Google and this is $YEAR."
5 TB of data made available, not 5 TB of transfer/bandwidth/etc.
"Remove it and see who complains."
I did that (with the impenetrably named "PrefetchExperiment", last touched by a branch that lost previous file history in 2007). Turned out it was the source data for Google's DNS to figure out how to route queries to the lowest-latency datacenter, based on their geographic location. In about a month, it would've taken down all Google services. Oops.
It was a very effective way of figuring out who owned the variable and writing a big long comment explaining what it's there for and which team to contact before changing it, though.
Also I love the idea of removing it and seeing who complains.
FB's problem was the lack of routing reachability on its backbone triggering the lack of routing reachability information being sent to the larger internet, this in turn caused problems for DNS not the other way around.
Each month, you can nominate another employee for a small bonus. This is designed to be given to coworkers who have gone above and beyond what was expected from them.
> Do people actually use them as bargain chips?
From my experience it's so over-the-top absurd that it would be difficult to have someone interpret such an offer as anything other than a joke or a meta-joke.
https://blog.bonus.ly/a-look-at-googles-peer-to-peer-bonus-s...
They are intended to be used for going above and beyond, not for stuff that falls within the scope of ones job. Using them as bargaining chip is explicitly against policy.
akin to the usual corporate "thank you" gift card, but more money and generally easier to distribute
What they do do is send one when someone else does something nice (like fix a bug from a project they have left or whatever else). If you ever need something similar again, the person you peer bonused has warm fuzzies about the experience and a hint that they might get it again.
People also use peer bonuses during perf time to demonstrate that the work they are doing impacts other people enough for a somewhat uncommmon thank you.
Here's where the company is now (the original domain is used for something else now):
https://en.wikipedia.org/wiki/Nawmal