SRE has gotten so good that people don't even think about it. It is default expectation that any online service is basically five nine availability or better. If it is worse than five nine is the only time it gets talked about.
SRE is really one of the only well solidified parts of tech as far as career and expectations go. The author seems to look at the same info and come to a different conclusion.
I’m curious what services you interact with that have five nines availability. Most services that I interact with, AWS APIs as an example, top out at an SLA of four nines for availability.
OMG you guys are missing the point. THe article is not talking about what is a good SLA level. Its that SRE "could be so much more" My point is no it cannot. SRE is now a commodity, the expectation is that the system is up pretty much always. Even two nine is only an hour or so a year downtime. Jeeze
There is some nuance there. Because SLA violations typically have real-money impact, they're often backed by internal objectives (SLOs) that are quite a bit tighter. So if an API promises four nines of availability in its SLA, there is probably an internal target that's at least four-and-a-half nines, if not five.
Which is to say, I would not be surprised if many APIs with a four-nines SLA were actually closer to five nines.
From a planning perspective, you should probably stick to the public guarantees (i.e. the SLA), though. Although things can always go horribly wrong, like in the big Atlassian deletion snafu recently.
I'm saying SRE/SLA has become a commodity. No one thinks about it. They just expect the systems to be up. The article is about how SRE can be more than "downtime" I propose it cannot and they are wrong.
I guess I made a poor point. I am saying that SRE has basically solidified into the expectation of the high 99 percentile as the bottom level of expected downtime. Even two nine is like 1hr per year.
There really isn't much to discuss in the industry of SRE. You basically have to be flawless. Its like the janitor. Miss the garbage pickup one day and heads roll. If you have SRE's and the service is down you are done. There is no longer any nuance which is why the CEO guy in the OP doesn't think about SRE anymore than the garbage collection.
A lot of the metrics for outage impact are available, and if you really want to Do Science you can use whatever A/B Canary/experiment framework you have to simulate outages (which, arguably, are not perfect because of down-detectors) on users to understand their behavior.
I tend to agree that most people don't care about transient downtime beyond 2 or 3 nines for reputation. Covid has proven how bad people are at reasoning about 0.5% probabilities so there's essentially no chance that they are noticing (en mass) the difference between 2.5 or 3 nines of reliability. Know your customer, as the article points out; more complex customers will know and measure actual availability because they need it for their own reliability calculations.
This brings me to my final guess, that SLOs are primarily useful because they provide a risk model that upstream and downstream consumers can reason about. SLOs are a way of building even larger distributed and interdependent systems that can still be somewhat reliable. Precisely what SLIs are chosen and the limits are a little less important than the fact that they are published publicly. There's an economy of reliability in large systems and like any market information about the market is key to its efficiency. If everyone just trusted that everyone was doing their best at maintaining dependencies most of those beliefs would be very unanchored.
A side benefit of SLOs is postmortems and communication; people working on codependent products and systems need to talk to each other from time to time and understand the true nature of the interfaces between their systems. Postmortems and SLO negotiations give a concrete framework to do that within, metrics drive quantitative understanding of the systems instead of handwaving.
I think this is the point where we're getting too serious about the discipline. We need to be a little more grounded in reality. SRE is basically a highly polished, best-practice, telemetry-and-automation-focused sysadmin. And a sysadmin doesn't really have a say in the business. They can make suggestions, like error budgets. But what about sales contracts? The contracts define what kind of guarantees you have to give and on what timeline. And what about customer experience? How is SRE supposed to quantify if the users are happy if they can't tell the UX and customer support how to build the product to better collect user happiness? Or tell the devs how to design their apps to fail less often?
If you keep optimizing and focusing on all the ways to improve satisfaction, you literally end up an executive director of product. If we really want to make products that users are happy with, we need to teach executives how to achieve it, not SREs. SREs shouldn't be driving the bus from the back row. They should be rotating the tires and checking the fuel and oil gauges and telling the driver when to stop for gas. Let's teach executives how to set everyone up to succeed, and then SRE doesn't have to get all existential just to keep the bus running.
> If we really want to make products that users are happy with, we need to teach executives how to achieve it, not SREs. SREs shouldn't be driving the bus from the back row. They should be rotating the tires and checking the fuel and oil gauges and telling the driver when to stop for gas.
To extend on this analogy a bit, the theory is that you might have business OKRs that are driven by SLOs.
For example, you might have a wishy washy goal of having happy customers. If you're in the shipping business, one aspect of being happy with the service would be to receive deliveries on time (or earlier!) so you might set an OKR where the objective is "Deliveries are made in a timely manner" where key results might be the number of deliveries made on time and inversely, the number of deliveries that are not returned.
By pitting those two metrics (speed and state of the delivery) together, you can ideally capture that an increase in speedy deliveries is not just achieved by people throwing packages out of moving vehicles as one or the other will decline.
Of course, simply setting these metrics doesn't mean very much but this is where we can drop down to a lower level and make use of SLOs. An SLO is just some number with a target after all, it doesn't have to be technical by any means.
So, how can we help drive our goal of timely deliveries? Well, we might set an OKR that says something like "99% of vehicles should have their oil checked every 3 months".
On the face of it, that has nothing at all to do with vehicle deliveries but inspecting historical data might determine that failed deliveries were often correlated with poorly serviced vehicles so by ensuring they're serviced, we can indirectly improve customer happiness by something that seems entirely unrelated.
This is all in theory of course and assumes your business is able to communicate across layers and share some sort of overall vision which is very hard! It also assumes someone (in the middle?) is able to bridge the gap to co-locate SLOs with OKRs and what not.
Anyway, that's how I understand the "dream state" but have I ever seen it actually accomplished? I can't say I have!
> I think this is the point where we're getting too serious about the discipline.
In my opinion, we are not serious enough. The role of the cited sysadmin is dying or dead in the modern company.
As an SRE, you are expected to directly contribute to the product to improve SLOs as well as several other dimensions you quoted (like user happiness).
I've personally performed code refactoring to reduce build job duration slowing down pipelines for _everybody_ in a component I was unfamiliar with. The code review was then performed by a code owner and the PR merged.
> The contracts define what kind of guarantees you have to give and on what timeline.
That's why you commit to ship product increments, not hard deadlines. See the failed launch of healthcare.gov, for example.
> How is SRE supposed to quantify if the users are happy?
The same way a dev would: review with stakeholders, "show-and-tell" where you collect feedback and improve your domain knowledge by focusing on the customer problem instead of your own vision of the solution.
> Or tell the devs how to design their apps to fail less often?
This is precisely why SREs would provide value by direct contribution.
> you literally end up an executive director of product.
We are all ambassadors of the product and can contribute past our designated "boundaries". That's what cross-functional is all about. No, you cannot make business decisions but if it is code, you have very high leverage.
> If we really want to make products that users are happy with, we need to teach executives how to achieve it, not SREs.
This has not happened yet and I suspect it never will.
> SREs shouldn't be driving the bus from the back row [..] Let's teach executives how to set everyone up to succeed [..].
Agreed. Do you have a concrete proposal? Without fundamental organizational change, this is impossible. Conway's law still applies, so does its inverse. Meaning you can break down silos (including at executive level) by migrating from component to cross-functional feature teams.
My proposal would be to form an industry panel made up of experts in each of the discrete roles that make up a typical software product value chain. Ask that panel to produce the following:
1. Adapt industry best practices into a training program to teach executives what is necessary for a software product with happy customers. Very simple and to the point: here are the components of the value chain, here is their common problems, here is what each job needs to do to prevent them, here is how they have to work together. The purpose isn't to micro-manage each department, but to know how to identify when they aren't operating correctly and how to put people in place who know #2 below.
2. Take the above and break it down for each part of the value chain. Have the managers/directors of each part learn what their own department's best practice requires, its pitfalls, strategies. Teach them how inter-departmental overlap is required for happy customers, how departments should set each other up for success and collaborate, how "ownership" is tenuous and demands collaboration and compromise. They're all tributaries of a river.
Charge money for people to go through that training and get a certificate. The money goes back into the whole thing. This becomes a baseline that shows you understand how to work on software products with best practice geared towards customer satisfaction.
I particularly liked this article [1]. Certification is IMHO people selling just about anything as a silver bullet/snake oil.
There is no such magical solution, because one needs to adapt the Agile template to their organization's needs and that is where real skill is necessary.
SRE has an identity crisis. It means too many things to too many people. In some companies, it is an operational help desk. In a few companies, SRE is rumored to be a different aspect of software engineering.
Pay is a rough signal on what a company expects from SRE. If SREs get paid as much or more than Software Engineers, it is a good sign. Yet even that signal is not accurate. I know highly paid SREs stuck in Ops Hell. You could not pay me enough money to do a job like that. It is also not sustainable because eventually the company will fire you and replace you with a lower paid employee who can do that job just as well.
I strongly oppose organizations that have SRE separate from Software Engineers. I strongly believe that SREs must be fully integrated into Software Engineering teams (aka the embedded model), with the same reporting lines as their product colleagues.
Keeping SRE organizationally separate has done a great disservice to SRE. SRE 2.0 needs to tear down the mini empires that SRE 1.0 built and get back to software engineering.
Imo embedded mode is no better - usually just a token ops person that everyone on the team throws over the fence to.
I’d say better signal is if sre has its own separate org structure which roots at the cto/vp of eng level. Soon as you have sres reporting to product eng managers it’s beginning of an end.
In the several examples of separate SRE orgs that I've seen, SRE builds their own little empires with no alignment to the product. A cynic would guess that the goal of SRE management is to hire enough SREs so there can be senior managers and directors and eventually a VP with pay packages to boot.
They come to Product from the mountain-top with their holy scriptures (the Google SRE book), prescribing surface solutions to problems with deep roots (pray it away with SLOs).
The entire point of the unembedded model is to create the management structure that isn't strictly beholden to the whims of product. An embedded model can't work unless the company leadership wants to make it work, and product leadership won't care enough (as demonstrated by: everything pre-sre). An embedded model is designed to fail, because the incentives are misaligned.
Like, what your asking is "hey, random engineer or engineering manager who is an 'SRE', make the VP of product prioritize your work and staff your team correctly!". That's not something anyone can fix, no matter how good, unless the management is willing to listen, and continue to listen and prioritize appropriately.
Greybeard sysadmin turned SRE-mask ops dude here, and this is correct. I've seen inside literally hundreds of orgs in my career, and almost always the real disconnect exists at the c-suite and board level. For a long time I blamed them for this dysfunction, but it tooks some years and maturity to start thinking more about how I can speak their language to reach their lofty, double-scheduled minds. Its a failure on both ends.
This is where a good CTO/CIO with actual leadership qualities will make or break an org, as they should usually be the SRE-esque visionaries, leaders, defenders. When that doesnt happen, and lets face it, it doesnt happen very often, you get cycles of "try this other ops/admin/swe/sre structure and management method" until it falls apart and new management tries something else or they start rolling heads right under the CTO/CIO until it becomes an 1000 and 1 nights. One of the biggest warning signs of this is when tech competent middle managers (worth their weight in gold) start jumping ship enmasse.
Isnt this a bit reductive? The other side of the fence is “How would an independent SRE org be able to fund/influence/prioritize inside the product org.” And Im pretty sure the answer there is
> A dedicated model can't work unless the company leadership wants to make it work. Product reliability and performance is not something anyone can fix, no matter how good, unless the management is willing to listen, and continue to listen and prioritize appropriately.
So you have the same answer, incentives and top down buy in. And if you have incentives at management layers, and top down buy in, then youve solved the root people + business problem. And now how/why do you choose between options?
> Isnt this a bit reductive? The other side of the fence is “How would an independent SRE org be able to fund/influence/prioritize inside the product org.” And Im pretty sure the answer there is
No, the answer is that the SRE org is unilaterally able to stop supporting the product/implement a production freeze if the product org is not doing enough reliability work (or falls out of SLO, or whatever). That's literally the trick, you take the ability to deploy out of the product org's hands and put it into the hands of an organization whose job is reliability, and whose management up to the VP level will back them up on that!
It is far easier to have an organization with the incentive "reliability" and an organization with the incentive "features" and have senior management deal with conflicts when they arise, than to have every middle manager try to correctly split their prioritization in the correct way.
> the SRE org is unilaterally able to stop supporting the product/implement a production freeze
Who exactly does this. And by what mechanism do you enforce:
> take the ability to deploy out of the product org's hands
If it is not "incentives and top down buy in" from above/at both orgs, what is it? Are you literally proposing some reality where VP Alice nonconsensually tells peer VP Bob "you're missing your roadmap ship dates. Im taking away your orgs prod creds because your lines arent up and to the right enough?" Because when there's a conflict between the two it's either going to get resolved by negotiation and common understanding, at and above Alice & Bob, or Bob is going to tell Alice to stay in her lane and get bent.
Edit: For the above, Im interested in the medium/long term. Yes, if you have hived off deployment gates to a different team I believe the could tactically say "no" once. I do not believe they could maintain that position.
Unilateral actions only work when the power dynamics support the person taking action. And those power dynamics are roughly more senior management, broad cultural consensus, and revenue, in my experience. So I'm really curious what enables the dedicated SRE orgs unilateral force of action if its not incentives & top down buy in.
This isn't hypothetical to me. Im a principal systems engineer at AWS who spent years reporting directly to a VP with a billion dollar business, and now work on operations & incident management in the same VPs now larger org running multiple billion dollar businesses. I was there when a billion dollar business took a year off the feature roadmap to focus on availability & performance. No one "told" us to do that, but it was supported as the best thing for the business. I've also stopped working on teams when they weren't making progress with operational investments.
> Who exactly does this. And by what mechanism do you enforce:
That's one of the key pieces of what makes SRE "SRE" as opposed to an ops team. They are given the power to stop deployment. If they can't, they're not SREs.
> Are you literally proposing some reality where VP Alice nonconsensually tells peer VP Bob "you're missing your roadmap ship dates. Im taking away your orgs prod creds because your lines arent up and to the right enough?"
Yes, and no. No because "roadmap ship dates" aren't reliability. I'm describing a reality where VP Alice nonconsentually tells VP Bob "you've had three outages in the past three weeks, we're out of SLO and so you no longer get to deploy new versions until we have remediations in place to prevent future outages". (and of course they both agreed to this)
Of course, Bob, being a rational human probably starts taking actions after the first outage, and so it never gets to this point, but yes, that Alice can tell Bob "no more deployments" is literally the point.
> They are given the power to stop deployment. If they can't, they're not SREs
Google is the only place where this is even discussed. I cannot name any other company that does this. Not Apple, Meta's Production Engineers, Netflix, Amazon's SysDevEngs, Stripe SRE, Uber, Pinterest, and more. This must mean that nobody else does real SRE!
> Alice can tell Bob "no more deployments" is literally the point.
Fantastic. Why can she say & enforce this.
You seem to be going a long ways to describe “what” while explicitly ignoring “why” and “how.”
To be clear my thesis is that success in both operating models is built on a combination of top down incentive/buy in and broad consensus expressed as a cultural norm. Im not being intentionally obtuse, but Im yet to see any other hypothesis or method of action whicb would explain the differentiation of ability to execute.
Directly, the answer is the permissions system. In my neck of the woods, devs are quite literally incapable of pushing things to production on their own. The automated tooling handles it 99% of the time, but in the case that they turn it off, only the SREs have production access necessary to actually push new things.
But I think you're speaking organizationally, and the answer there is that that agreement was made by SVPs and is enforced by the PRR process. When you adopt SRE support, you hand over the keys, so to speak.
> Fantastic. Why can she say & enforce this.
Because there's nothing Bob can do except escalate, and SRE leadership will support Alice all the way up to the SVP level.
> To be clear my thesis is that success in both operating models is built on a combination of top down incentive/buy in and broad consensus expressed as a cultural norm.
My point is that the process creates the cultural norm, and enforces everywhere. That is, your org might have these good processes, and that's laudable. But if Andy turned around tomorrow and said "hey, this year your first priority needs to be shipping new features, because we're falling behind", would your org maintain that commitment? Would every exec? The same question goes for every exec speaking to their subordinates.
Its much more difficult to turn to the guy whose job is reliability and reports directly to the CEO and say "hey, your focus this year is shipping features". Like maybe you could say "your focus is shipping features reliably this year", and that should take the focus off of some other things that the SRE org is involved in, but I'm not convinced.
Or briefly, your approach is betting on there being a cohesive company culture of reliability that permeates every team, org, and manager. We both know that's not true, at Amazon or Google or any other company. They're all big enough to have different cultures in different parts. A distinct org and reporting structure, and split responsibilities, ensures that leadership has to be really explicit about such things ("lower your SLOs"), instead of just quietly deprioritizing enforcing or reporting on them. (And I'm generally a fan of multiple individuals with well scoped and distinct responsibility who are charged with reaching consensus over a single individual who makes a decision).
Isn’t reliability a feature of the product? I am currently building out a team for a new product and reliability is quite high on our feature list. We have a embedded SREs who are working on improving our product reliability full time. We share the operational workload between the SWEs and SREs instead of just throwing all the operational problems at the SREs. This gives developers an appreciation for reliability and stability and makes sure that the SREs actually have the time and resources to improve the product instead of being a help desk.
It sounds like you needed a separate chain of command for SREs because your product leadership was incompetent.
If your stack is complex enough and/or team is large enough the reliability work can be smeared so thin or be so intricate that it becomes nobody’s responsibility. Hence separate command chain that Google and some others use
If an organization decides that reliability is important (which honestly isn't always the case), then the question becomes whether they can trust Product to include reliability work or not. If not, if reliability is so essential to the business that the business is willing to sacrifice velocity for it, then a separate SRE reporting structure can exist to slow down Product when reliability objectives aren't being met. If reliability is negotiable, then an embedded structure means Product can make its own prioritization decisions and move quickly when it needs to.
The embedded structure doesn't mean that SRE always, necessarily gets overruled. It means that SRE leadership needs observability into when SRE is getting deprioritized by Product, so that consistent failure by Product to respect a core business goal (reliability) can reach executives who will, ahem, clarify for Product that they need to spend some time prioritizing reliability.
Even with an embedded model, SRE leadership is responsible, first and foremost, for setting up tooling like observability and auto-healing (i.e. Kubernetes) for the company's workloads, which gets you about 90% of the way there. If Product can't keep their service online because they never wrote tests etc., then the way you "bolt reliability on the side" is, SRE leadership gives the exec the raw lack-of-availability numbers (see: observability tooling), then the exec tells Product to stop fudging around and prioritize making the availability numbers better. Then Product management freezes feature development and takes a vacation for a sprint (or two) while Product developers focus on bolstering their QA.
Yes SLOs are kind of like OKRs of sre management - they only ever as good as the objectives and objectives are often not set very well by senior leadership. Ultimately that stems from the fact that reliability is not valued by the market. You see a lot of the same things in security (outside of the boneheaded compliance cya).
A recent talk I saw suggested that SREs report into the COO (Chief Operations Officer) and not the CTO or VP of Engg. I thought that’s a good idea.
The COO is interested in the stability of ongoing operations whereas the other two roles are about launching new things (Unless I have gotten the definition of COO wrong).
organizationally that's a bad idea at a lot of places because the CTO will usually have more oversight and a bigger budget to match, and SREs are closer to SWEs than traditional ops
> SRE has an identity crisis. It means too many things to too many people. In some companies, it is an operational help desk. In a few companies, SRE is rumored to be a different aspect of software engineering.
SRE is Sysadmins but with better contracts between teams, and automation focused.
I haven't had much trouble with that definition, does away with all the cruft of the "Production" talk by Benjamin Treynor Sloss and the majority of the SRE books (which are a prescription of a method of creating those contracts).
That said: "DevOps" and Systems Administration in general has had an identity crisis for a while. People don't like the idea of "Sysadmin" for some unknown reason, but give the same role to someone and call them "DevOps" and suddenly all is well.
Same is now true of SRE.
Someone on HN said to me: "DevOps is how you get Engineering to respect Systems Administration".. Maybe the same is true of SRE?
I definitely feel the identity crisis of DevOps is much, much worse than that of SRE- but I guess that depends on your circles.
> SRE is Sysadmins but with better contracts between teams, and automation focused.
That's not even close to true, which I think is OPs point. For instance, I have an extensive SRE-SE background, but I'm also a software engineer. If I was tasked with "sysadmin" stuff, or shepherding peoples Jenkins jobs every day I'd probably lose my mind.
There are many types of systems administrators; when we talk about as SRE in particular there some overlap with what I was doing as a “Linux Systems Administrator” in a Platform Engineering team, in 2009.
Automating migrations, distributed job control, instrumenting the message queue system, managing cache invalidation across our entire fleet of memcache servers (which was ugly hacked with MySQL, not proud), as well as “maintenance” programs to upgrade “versions” (multi-tenant SaaS) and warm caches, dark launches, canary deployments… well you get the picture.
Of course I was responsible for the reliability of the systems these things ran on, but there was a really-really strong focus on automation as part of the job, and this was when puppet was quite new, we were stuck with tools like cfengine.
Coddling Jenkins sounds like build or release engineering. Which is also a separate thing.
People seem to forget that systems administration has a focus on software development too- I mean for goodness sake terraform, ansible and salt all come from sysadmins; I suspect it’s because of some very foolish admins proudly exclaiming they don’t code. But they were always the bottom rung and end up being “promoted” to management at some point.
I get more of what you're saying now, but I still think that reinforces OPs point because, to me, what you described is a Systems Engineer not a Systems Administrator. OP is asking for some structuring as to what these roles require in terms of knowledge. To me, "Administrator", "Engineer", "Technician", etc all prescribe different degrees of knowledge and also usually correlate with responsibility.
As for "SRE" vs "DevOps" vs "Systems Engineer", I'm not so sure. For instance, I typically expect SREs to be able to write application software and orchestration software. I have similar expectations of someone who uses the title Systems Engineer but I might expect less application software development experience. DevOps I think I'd rather forget is a word.
Systems engineer is a title that seems to have fallen somewhat out of favor, but it wasn't unusual that it was a less hands-on role than sysadmin/SRE. Sometimes close to a system integrator but could also be what is now known as an infrastructure architect.
Who do you think did the SRE work before that name was invented? Most likely it was your friendly sysadmin. It is the type of role that is not much noticed when done well, and probably that's why some feel it underappreciated enough to keep inventing new job titles. It's still the same job, pretty much the role that requires privileged access.
I was raised in the South and grew up across the midwest, so that may color my perspective.
The place where I encountered sysadmins the most was people who managed a small collection of assets; the keyword being managed. I didn't see them inventing entirely new systems. Working across DCs, I would see Systems Engineers building these new systems, being escalation support for them, and at times coding up entirely new infrastructure applications.
I don't really buy the idea that administrators are underappreciated, I just don't think it accurately captures what a systems person on a product team actually does, much less the distinction between a SRE-SE and a systems person.
My base argument is that it's not the same job at all. A good Systems Engineer does not guarantee a good SRE-SE because the job is that much different. One of the notable distinctions is writing application code, which is much different from Terraform, Ansible, Salt etc
I have to disagree. SRE=SE=SA. What they exactly do depends on the job. Some can write application code, some cannot. But they are all the same thing. I've worked at SRE jobs where there was no application code writing, and I've worked at SE jobs where there was application code writing.
Reliability Engineering is much wider than the scope of a sysadmin. SREs are software engineers whose focus on the reliability of systems and platforms. Could be sysadmins; could be traditional SWEs; knowledge of both is required.
One of the functions that SRE provides in our organization is building out the platform and solving cross-cutting problems. How do you do that without some reporting structure? Shared ownership and guilds can only go so far since each team has its own priorities and incentives (and will rarely choose to yield engineering time to platform things if they can devote it to product things).
I’m not arguing against an embedded model, but I do think there’s tension that’s not easily resolved.
In my division, there is an infrastructure team and product teams. Those platform type projects are handled by the infrastructure team. In a product only team, it will always be difficult to work on such projects.
Ultimately, the answer ties back into the identity question: what is SRE?
In our org, we have SRE who handle platform and SRE who are embedded on product teams, but all report up through the SRE org. One of the goals is to spread more infrastructure knowledge to the engineering teams to reduce their dependence on SRE so that SRE is mostly doing platform work and special infra-intense projects.
I’m not saying this is the one true way or even a particularly good way, and I agree about the identity crisis (same as with DevOps half a decade ago).
Former Senior SRE here. Was doing everything: working with engineering teams to build a microservice authoring framework. Putting out fires and responding to outages. Adding monitoring/metrics/logging to all the things. Building productivity tools for engineers. Spinning up new software in sandboxed environments for evaluation. Onboarding and offboarding new engineers. Coordinating downtime for software upgrades. Refactoring our infra and making it more IaC. Security audits.
Our team was basically a dumping ground for anything that required experience in many different types systems and/or privileged access. On top of that, we had projects were shoe-horned into roadmaps. So while 90% of our duties were reactive to what the org needed, we also had the pressure of trying to meet deadlines for normal software engineering work. It was unhealthy and I wanted very much to not be defined as an "SRE"
Add on top of this managing multiples of you and having to do the CFOs work for the CEO and litterally manage to the penny the instances we spun up and dealing with all the expectations of ops WRT to developing tooling on top of tooling.
My company expects SRE to be synonymous with cloud administrator, acting as the gateway to doing anything AWS-related.
They've also asked us to troubleshoot for database backup processes, write Crystal Reports on user activity, and be Slack admins. Some of the developers believe that we do manual production deployments of their builds (we don't). I get requests all the time from users asking if I can make accounts on various systems for them.
Agreed about having SRE fully integrated into feature teams in terms of day-to-day work, seating charts, team building events, etc.
From a management perspective I think it still makes sense for SRE to have a different reporting structure. Feature teams generally aren't rewarded for investing in long-term code and production health. If you report to a feature team manager there will often be downward pressure to focus less on production health and more on helping features ship quickly.
Giving SRE a degree of independent oversight serves as a system of checks and balances against those feature team pressures.
> SRE has an identity crisis. It means too many things to too many people. In some companies, it is an operational help desk. In a few companies, SRE is rumored to be a different aspect of software engineering.
SWE has an identity crisis. It means too many things to too many people.
Just kidding, our entire industry is filled with titles that mean nothing. I agree, we should fix that.
> I strongly oppose organizations that have SRE separate from Software Engineers
Can confirm, we have an SRE team that's totally separate from engineering and it's a huge and constant bottleneck. We've literally re-architected parts of the application to avoid having to go through SRE for things.
Bastard Operator From Hell -- A famous IT trope. Which was typically a sysad for a mainframe/Mini-Computer... Usually on a University Campus....
Think Lazlo from Wierd Science, but an asshole.
I actually worked with a BOFH. (Sadly he had a heart attack on the CalTrain Station in Mountain View, and died.)
He was the AS/400 admin for a place where we manufactured all of SUNs Software (meaning made the SUNos CDs manuals packaging etc) and this guy was a BOFH
Sales folk would call up to the helpdesk and request a password reset.
This one time, he set the users password to "STUPID!" ...
We use an "embedded model" with separate teams. I run an SRE organization with 10 individuals. At any given moment 3-4 are embedded on teams pursuing some objective. The rest are doing incident response and SRE specific projects from the big pile of sprint work.
We did this because in our case most development problems didn't have a reliability dimension. We build mathematical models. Do I need an SRE involved in the discussion of switching from one statistical approach to another? In most cases no. But when that team deems themselves limited by their current walled garden and want to explore/operationalize new technology then we need an embedded SRE to facilitate that.
I remember a common, puerile early adulthood "game" in which one would add the words "in bed" to the text of a fortune cookie. I feel like proclamations about the industry by people who were and are at Google need a similar party trick: add "at Google" to the end of each sentence. SRE is Google's answer to Google's problem; it says so right in the first revelatory header of the introductory chapter of the book: "Google’s Approach to Service Management: Site Reliability Engineering".
Inherent in the text of technical books that reveal what people did is an attempt to persuade the reader why they should do it, too. In some ways this reveals the central, myopic conceit at the heart of Google: if you don't have Google's problems, you should. And I'm not sure why anyone who isn't Google should just accept this premise.
That's not to say that the premise is unacceptable. I'm sure that there are people who want to participate in a thing that looks like Google, except that they'd rather start it from scratch than join the one that already exists. Perhaps those people want to have Google's problems.
Now we have a piece, expanded from a talk, by one of the editors of the books that persuaded organizations that, if only they would squint their eyes a bit and turn their head, they could make the problems of their organization orders of magnitude smaller by so many metrics look like the problems of Google. And now, perhaps, Google's answer to Google's problem wasn't quite right, because it didn't involve...something? It could be philosophy or human-centered design, and I wouldn't be surprised to find Google bereft of either, but it certainly isn't going to be found in a paper by Google (ostensibly, a Google answer to a Google question) that predates the first book by nearly a decade. It isn't going to be found in people for whom these things are afterthoughts rather than guiding principles. Instead, shouldn't we be asking whether we let SRE as a Google answer to a Google problem be a solution outside of Google in the first place?
Thank you for this - you put into words very well all of the problems I had with that book. In fact, I found myself very frustrated with the authors, wondering if they had any familiarity with all the rest of the decades of work that had been done in related fields outside Google.
It's a telltale sign of a naive blogger. If you think anyone outside your profession will read your writing (or code!), always expand acronyms on first use. I'll fix this one for everyone:
It’s natural, in the swirling chaos of the past few years, to take a step back and wonder just where everything is going. Though I’ve certainly been doing my fair share of that, I’ve been also thinking about defining precisely where things are. Answering that question for SITE RELIABILITY ENGINEERING (SRE) is not easy.
But your question was probably about the job, not the acronym. Carry on.
90% of SREs and SRE managers haven't read the SRE book(s).
99.9% of folks hiring SREs or starting SRE teams haven't read the SRE book.
The SRE book (and its sequels) say quite plainly what SRE is and isn't. They also say that not every org is going to be exactly like google so no, "we're not google" isn't an excuse.
the E in SRE is for engineering. As in software engineering. SREs are software engineers. Or should be. If your SREs don't know basic SWE principles, they're not SREs. If your org isn't applying software engineering principles to minimizing operational complexity at scale, your org isn't doing SRE.
I'm constantly shocked by how hard these things are to grasp, even for most SREs. If the problems I (occasionally) get to solve weren't more interesting than most regular product work, I'd get out of "SRE" entirely.
I think this myth exists because Google was (is?) famously obsessed with SWE. But if you actually read the SRE books and look at the actual discipline of SRE ("what's the difference between SWE and SRE?"), SRE is quite blatantly just operations management. The website is a power plant, and the SRE runs the power plant. You don't build parts to run a power plant, you use software (as in manipulate/control/operate) to run it. You act quickly when the numbers go out of line, you write reports and control how much power is going in and out, respond to surges and dips, etc.
For whatever reason, Google decided to tell people that the same person who's building the klaxon and the concrete wall and the pipes for the power plant, and the person who's operating the power plant, are one in the same. But that's clearly bunk. Building a part and running a system are completely different disciplines, and anyone who does both will only be half good at both. Humans are shit at multitasking and there's few true polymaths out there. Show me a master programmer and I'll show you an amateur woodworker.
I also don't believe software engineering principles will help you reduce operational complexity. If anything, software engineering tends to either make things either inefficient or subtly complicated. Reducing operational complexity comes from the discipline of operations, which isn't engineering. Non-tech companies have known about these distinctions for like a hundred years. Deming applied scientific rigor and analysis to come up with better practices, but he didn't have to design any widgets to do it.
> For whatever reason, Google decided to tell people that the same person who's building the klaxon and the concrete wall and the pipes for the power plant, and the person who's operating the power plant, are one in the same. But that's clearly bunk. Building a part and running a system are completely different disciplines, and anyone who does both will only be half good at both
Depending on the team, SREs can absolutely involved with "building the system", especially the klaxon ;) Examples include designing and implementating metrics used to make make decisions in business logic and or exposed to customers/users, writing routing components like mixers and proxies, developing data pipelines, etc. At Google many SRE teams build and run entire multi-tenant systems with no pure SWEs involved at all.
Healthy SRE teams should be spending 20% of their time on operations. On my team its actually the devs who do most of the operations work. They take the pager during business hours and we route most maintenance tickets to them.
The website is NOT a power plant, it's just code. In software, "operations management" is basically infrastructure automation, incident response and build and release. All of these require some software development or at least code literacy and familiarity with software development practices. If there's large overlap in technical skill between the operators and the builders, then it makes more sense to see them as the same but focussing on different problems.
It's probably useful to talk about what Operations Management is first. It's a business discipline that touches on many parts of a business. It is defined as "the management of an organization's productive resources or its production system, which converts inputs into the organization's products and services". You can get a PhD in Operations Management.
In tech, software and data is the "productive resources", and the "production system" is the actual system you build out of those resources: the website, API, etc. You don't have to write any software to build and manage that production system. Maybe that's unusual to people in tech today, but it's a fact that you don't have to write a single line of code to build and operate such a system. Heroku, PagerDuty, DataDog, Splunk, Octopus, AWS, etc, all are products built with the sole purpose of enabling operations without the need to write code. You can assemble logging, alerting, monitoring, web server, networking, database, deployment, etc, without ever writing a single line of code, and have it be highly available and highly reliable.
The title will vary (Systems Engineer, Operations Engineer, DevOps Engineer, Site Reliability Engineer, Systems Administrator) but the job is the same: to use Operations Management techniques to ensure the products and services are productive. You can use software development practices for all of this, sure! But they are absolutely not a requirement to accomplish the goal. And many other roles in the company are involved in Operations (QA, PM, DM, etc) and may or may not use code. The business doesn't care about code, it cares that its resources are being used properly and the production line is operating nominally.
In terms of the distinction between builders and operators: you could say that a construction worker and a custodial worker are part of the same occupation because a lot of their skills overlap. They both need to understand how the building works and may need to build/repair parts of it at times. But they're still two different disciplines that require different training, experience, and day to day responsibilities, and as such we don't lump them into the same category.
"You don't have to write any software to build and manage that production system."
It depends on the scale and complexity of your application. At some scale/complexity, it absolutely requires writing software because your IAAS provider doesn't provide you with automation that covers 100% of your operational needs and even they recommend using infrastructure as code tools to manage your infra.
If your production system is a CRUD service with 3 application nodes and a managed PostgreSQL instance then you do not need to write software to manage it. But if your application is that simple, then I'd suggest you probably don't need a software developer to build it (Wordpress, Wix).
Construction vs custodian is not a fair analogy because their training and evaluation doesn't really overlap. The training and eval for both "dev" vs "systems" engineers is very similar; most have CS degrees and have to do some leet coding to get the job. Devs generally need to be better at algos, systems engineers need better understanding of networking, os, system design.
THere's at least one big issue here, which is that you're pretending that a website is like a building or a dam. If that were the case, a company like Google would have a (relatively) small team of SWEs who "built" things, and a much larger team of SREs who maintained them over their operational lifecycle once the SWEs were done building the thing. But that isn't the case.
Software systems (at least in competitive consumer markets) are constantly changing and evolving. To use the dam analogy, there's constantly a team of people making the dam taller or wider or deeper, even while the dam is running and producing power.
All the SRE teams I've worked with have done a bunch of things that go beyond "operations". They are usually consulted at the design stage, to make sure that the thing is going to be built reliably. They're also responsible for ensuring ongoing reliability as all new features are added. That means that the features themselves don't impact reliability, and that the process of adding new features doesn't impact reliability. None of this work has a reasonable analogue in your dam analogy, except perhaps as some combination of consultant and regulatory body.
> I also don't believe software engineering principles will help you reduce operational complexity.
This isn't a goal of SRE, in my opinion nor in anything I can recall reading. The goal of applying software engineering principles is to accept the increased complexity in exchange for a reduced operational burden.
There's layers of that effect, and the right one depends on largely on your operational burden. Sysadmins shun complexity, so systems are simple but doing mass updates requires a lot of manpower. DevOps embraces some complexity like Ansible or manually orchestrated containers making it easier to do mass updates, but still a burden. SRE embraces complexity, in exchange for a dramatic reduction in manual effort on many tasks.
The idea is that at certain scales (or reliability requirements), it becomes cheaper to hire a small number of expensive people that can manage complex systems than it is to hire a large number of people each managing a simple system.
Software engineering arises because it can effectively trade complexity for reduced operational burdens in exactly the areas you want. You don't have to migrate to a new infrastructure orchestration tool, you can just write an orchestration tool on top of what's there (which I've actually seen done). Was it perfect? No. Was it cheaper than migrating a half million containers to Kubernetes? Yes.
Operations management tends to be very inflexible. They have a set of tools, and anything outside those tools is either a no go or will require replacing an old tool at the cost of months of effort.
its not that this is hard to grasp. it's not. in fact, many of the people i've consulted on RE have read all or parts of these books. IMO, it's mostly comes down to selective interpretation.
it's like telling a homeowner that they need to spend $1000/year on an annual maintenance item to prevent a _possible_ $15k repair bill every five years.
For some, $1000/yr is too expensive for them. So they take their bets or skimp. (People who think you can do SRE without being SWEs because they "can't code")
For others, $1000/yr is affordable, but because the $15k bill is "unlikely", they skimp. (People who think you can do SRE without being SWEs because even though they can code, they'd rather separate those jobs.)
In a purist approach there are clear guidelines for DevOps and SRE but these are seldom followed. In all the cases I have seen personally, for a developer both DevOps and SRE are support teams. Major cultural changes in an organisation especially at the development end is very hard to make. What has worked for me is having Ops people with some development experience work as DevOps Engineers who work with Dev team for doing whatever is required for making development and release easier. And the SRE team work with development teams for doing whatever is required for making production support easier. Once we have a fairly seamless and well defined pipeline between committing code to supporting in production, the DevOps and SRE Engineers can merge with Dev or Production Support teams depending on their skill set
From 2008-2018, I went all over the world handling incidents and attempting to teach companies how to do the kinds of things the author is talking about. It was multi-diciplinary and very demanding, both in terms of technical and organizational (political and communication) skills.
One of the core observations I made was that few companies were willing to actually get better. Rather than figure out what skills they lacked organizationally, or how to improve their processes, or how to align their goals and teams, or any of the lessons they could have learned, it was way easier and cheaper to just keep on status quo. SRE as a practice, a position, a skill - whatever it could have been, it instead failed to really exist apart from maybe a hero here and there who chose to care about it. SLOs (or whatever name might have been used) did not exist to actually measure or inform anything. They were a kind of placebo - come up with something that looks measurable, find a way to make sure we always score well, and make sure there is a wall between these metrics and anyone's compensation / job performance.
An example of this, at a very prominent health insurer. I identified issues that could have been solved for, let's say well under $10MM. Rather than do any of that work, the company threw about $50MM in new hardware at the problem. In another case, at a national name Pharma company, about every two years I went back out to do the same basic consulting engagement, because they lacked the leadership to develop the skills in house or to figure out how to align their teams with the work they actually did. Still another case (prominent bank), they had this serious problem that blocked their commercial lenders from doing their job... they finally called me in after about a year of fooling around, I got the problem fixed, rolled out a bunch of tooling and training, but I guarantee they never used any of it... again - problem outright blocked key bank personnel from doing their job and yet it took them a year to do anything about it.
So... I guess what I'm saying is that yes SRE work can be fun and rewarding, but it's just not something most companies have the maturity to really do. I read the Google SRE book way back when, but I didn't find it particularly insightful. Likewise, I'm not really sure what to take away from this blog - for me, it doesn't answer any questions about a clear path forward and leads me to believe not much has changed since I left that kind of work years ago.
Somewhat relatedly, I converted from SWE to SRE for three reasons. 1. SWE interviews are torture and no matter what I accomplish at my job, the next job I have to re-prove everything all over again. SRE interviews do not typically have this style of interview (I've heard Meta and Alphabet do however). 2. SRE focuses on lower level abstractions and therefore seems less prone to product oversight which is tiresome. SRE work is typically longer lived and deadlines less tight. 3. SRE jobs seem even more remote friendly than traditional SWE roles because of reason 2.
I'm much better as a SWE but I'm paid more as an SRE and have much more freedom. I don't actually like ops work that much but I'll take that trade off to guarantee the other three advantages above.
All that said, in my short time as an SRE I've noticed that it's very much in an identity crises because there's not really a standard job. Infrastructure is very different company to company and some companies expect SREs to be able to code as well as SWEs whereas others don't expect you know how to code at all.
In my experience, an SRE is someone who's principal goal is to maintain infrastructure and ensure code is able to be developed, built and shipped. In the "better" SRE roles, SREs are able to build platform tooling rather than just ops.
This was the most unfortunate section of this blog for me. Some of the companies that need SRE the most, and have the money to actually fund this practice, care the least because they are the only game in town.
My take on SRE:
1. It is an extremely important and extremely technical practice in desperate need of good non-technical PR (example: most SLOs I've helped craft are guesses from engineering because they couldn't get data from product to support real numbers)
2. It is damn hard to hire for on skills alone (good SREs are good sysadmins _and_ good SWEs, i.e. two hard types of candidates rolled into one)
3. The hires are damned expensive (see [2]), and
4. Ops-aligned BUs that are traditionally given SRE headcount are (still) considered cost centers, which, combined with [1], makes [3] difficult to justify and [2] difficult to obtain as a result.
I have never been enamored with SRE culture, and definitely not the name. Reliability isn't the only thing to worry about - as the article sales, latency is just as if not more important. I also say that systems should be responsive, functional, reliable, and secure, and not necessarily in that order.
102 comments
[ 0.20 ms ] story [ 202 ms ] threadSRE is really one of the only well solidified parts of tech as far as career and expectations go. The author seems to look at the same info and come to a different conclusion.
https://cloud.google.com/compute/sla
Which is to say, I would not be surprised if many APIs with a four-nines SLA were actually closer to five nines.
From a planning perspective, you should probably stick to the public guarantees (i.e. the SLA), though. Although things can always go horribly wrong, like in the big Atlassian deletion snafu recently.
There really isn't much to discuss in the industry of SRE. You basically have to be flawless. Its like the janitor. Miss the garbage pickup one day and heads roll. If you have SRE's and the service is down you are done. There is no longer any nuance which is why the CEO guy in the OP doesn't think about SRE anymore than the garbage collection.
SLAs in most cases are bullshit and if anything really bad happens (Atlassian) they are just prepared to pay out class lawsuit.
I tend to agree that most people don't care about transient downtime beyond 2 or 3 nines for reputation. Covid has proven how bad people are at reasoning about 0.5% probabilities so there's essentially no chance that they are noticing (en mass) the difference between 2.5 or 3 nines of reliability. Know your customer, as the article points out; more complex customers will know and measure actual availability because they need it for their own reliability calculations.
This brings me to my final guess, that SLOs are primarily useful because they provide a risk model that upstream and downstream consumers can reason about. SLOs are a way of building even larger distributed and interdependent systems that can still be somewhat reliable. Precisely what SLIs are chosen and the limits are a little less important than the fact that they are published publicly. There's an economy of reliability in large systems and like any market information about the market is key to its efficiency. If everyone just trusted that everyone was doing their best at maintaining dependencies most of those beliefs would be very unanchored.
A side benefit of SLOs is postmortems and communication; people working on codependent products and systems need to talk to each other from time to time and understand the true nature of the interfaces between their systems. Postmortems and SLO negotiations give a concrete framework to do that within, metrics drive quantitative understanding of the systems instead of handwaving.
- also a Sysadmin, really expensive.
at least that's the attitude I'm trying to wrench out of the minds of everyone at $currentJob
- a current Service Restart Engineer
I build self-service, provide training, provide documentation, provide portal, provide permissions.
3am, and the overseas team are still pinging me asking me to restart their thing.
Eliminating toil: https://sre.google/sre-book/eliminating-toil/ Every team tech and non tech everywhere should read this
Incident Management: https://sre.google/workbook/incident-response/
Post Mortems: https://sre.google/sre-book/postmortem-culture/ These 2 chapters are necessary reading.
SLOS: https://sre.google/sre-book/service-level-objectives/ Disclaimer: I think the SLO chapter is good to know the theory around, but SLOs specifically Ive used very few times.
95% of companies will never reach that scale or deal with those issues.
If you keep optimizing and focusing on all the ways to improve satisfaction, you literally end up an executive director of product. If we really want to make products that users are happy with, we need to teach executives how to achieve it, not SREs. SREs shouldn't be driving the bus from the back row. They should be rotating the tires and checking the fuel and oil gauges and telling the driver when to stop for gas. Let's teach executives how to set everyone up to succeed, and then SRE doesn't have to get all existential just to keep the bus running.
They can easily tell if the users aren’t happy which is usually same time their pagers go off
To extend on this analogy a bit, the theory is that you might have business OKRs that are driven by SLOs.
For example, you might have a wishy washy goal of having happy customers. If you're in the shipping business, one aspect of being happy with the service would be to receive deliveries on time (or earlier!) so you might set an OKR where the objective is "Deliveries are made in a timely manner" where key results might be the number of deliveries made on time and inversely, the number of deliveries that are not returned.
By pitting those two metrics (speed and state of the delivery) together, you can ideally capture that an increase in speedy deliveries is not just achieved by people throwing packages out of moving vehicles as one or the other will decline.
Of course, simply setting these metrics doesn't mean very much but this is where we can drop down to a lower level and make use of SLOs. An SLO is just some number with a target after all, it doesn't have to be technical by any means.
So, how can we help drive our goal of timely deliveries? Well, we might set an OKR that says something like "99% of vehicles should have their oil checked every 3 months".
On the face of it, that has nothing at all to do with vehicle deliveries but inspecting historical data might determine that failed deliveries were often correlated with poorly serviced vehicles so by ensuring they're serviced, we can indirectly improve customer happiness by something that seems entirely unrelated.
This is all in theory of course and assumes your business is able to communicate across layers and share some sort of overall vision which is very hard! It also assumes someone (in the middle?) is able to bridge the gap to co-locate SLOs with OKRs and what not.
Anyway, that's how I understand the "dream state" but have I ever seen it actually accomplished? I can't say I have!
In my opinion, we are not serious enough. The role of the cited sysadmin is dying or dead in the modern company.
As an SRE, you are expected to directly contribute to the product to improve SLOs as well as several other dimensions you quoted (like user happiness).
I've personally performed code refactoring to reduce build job duration slowing down pipelines for _everybody_ in a component I was unfamiliar with. The code review was then performed by a code owner and the PR merged.
> The contracts define what kind of guarantees you have to give and on what timeline.
That's why you commit to ship product increments, not hard deadlines. See the failed launch of healthcare.gov, for example.
> How is SRE supposed to quantify if the users are happy?
The same way a dev would: review with stakeholders, "show-and-tell" where you collect feedback and improve your domain knowledge by focusing on the customer problem instead of your own vision of the solution.
> Or tell the devs how to design their apps to fail less often?
This is precisely why SREs would provide value by direct contribution.
> you literally end up an executive director of product.
We are all ambassadors of the product and can contribute past our designated "boundaries". That's what cross-functional is all about. No, you cannot make business decisions but if it is code, you have very high leverage.
> If we really want to make products that users are happy with, we need to teach executives how to achieve it, not SREs.
This has not happened yet and I suspect it never will.
> SREs shouldn't be driving the bus from the back row [..] Let's teach executives how to set everyone up to succeed [..].
Agreed. Do you have a concrete proposal? Without fundamental organizational change, this is impossible. Conway's law still applies, so does its inverse. Meaning you can break down silos (including at executive level) by migrating from component to cross-functional feature teams.
1. Adapt industry best practices into a training program to teach executives what is necessary for a software product with happy customers. Very simple and to the point: here are the components of the value chain, here is their common problems, here is what each job needs to do to prevent them, here is how they have to work together. The purpose isn't to micro-manage each department, but to know how to identify when they aren't operating correctly and how to put people in place who know #2 below.
2. Take the above and break it down for each part of the value chain. Have the managers/directors of each part learn what their own department's best practice requires, its pitfalls, strategies. Teach them how inter-departmental overlap is required for happy customers, how departments should set each other up for success and collaborate, how "ownership" is tenuous and demands collaboration and compromise. They're all tributaries of a river.
Charge money for people to go through that training and get a certificate. The money goes back into the whole thing. This becomes a baseline that shows you understand how to work on software products with best practice geared towards customer satisfaction.
There is no such magical solution, because one needs to adapt the Agile template to their organization's needs and that is where real skill is necessary.
[1] https://www.simplethread.com/agile-at-20-the-failed-rebellio...
Pay is a rough signal on what a company expects from SRE. If SREs get paid as much or more than Software Engineers, it is a good sign. Yet even that signal is not accurate. I know highly paid SREs stuck in Ops Hell. You could not pay me enough money to do a job like that. It is also not sustainable because eventually the company will fire you and replace you with a lower paid employee who can do that job just as well.
I strongly oppose organizations that have SRE separate from Software Engineers. I strongly believe that SREs must be fully integrated into Software Engineering teams (aka the embedded model), with the same reporting lines as their product colleagues.
Keeping SRE organizationally separate has done a great disservice to SRE. SRE 2.0 needs to tear down the mini empires that SRE 1.0 built and get back to software engineering.
I’d say better signal is if sre has its own separate org structure which roots at the cto/vp of eng level. Soon as you have sres reporting to product eng managers it’s beginning of an end.
They come to Product from the mountain-top with their holy scriptures (the Google SRE book), prescribing surface solutions to problems with deep roots (pray it away with SLOs).
SRE 2.0 needs to make the embedded model work.
Like, what your asking is "hey, random engineer or engineering manager who is an 'SRE', make the VP of product prioritize your work and staff your team correctly!". That's not something anyone can fix, no matter how good, unless the management is willing to listen, and continue to listen and prioritize appropriately.
This is where a good CTO/CIO with actual leadership qualities will make or break an org, as they should usually be the SRE-esque visionaries, leaders, defenders. When that doesnt happen, and lets face it, it doesnt happen very often, you get cycles of "try this other ops/admin/swe/sre structure and management method" until it falls apart and new management tries something else or they start rolling heads right under the CTO/CIO until it becomes an 1000 and 1 nights. One of the biggest warning signs of this is when tech competent middle managers (worth their weight in gold) start jumping ship enmasse.
> A dedicated model can't work unless the company leadership wants to make it work. Product reliability and performance is not something anyone can fix, no matter how good, unless the management is willing to listen, and continue to listen and prioritize appropriately.
So you have the same answer, incentives and top down buy in. And if you have incentives at management layers, and top down buy in, then youve solved the root people + business problem. And now how/why do you choose between options?
No, the answer is that the SRE org is unilaterally able to stop supporting the product/implement a production freeze if the product org is not doing enough reliability work (or falls out of SLO, or whatever). That's literally the trick, you take the ability to deploy out of the product org's hands and put it into the hands of an organization whose job is reliability, and whose management up to the VP level will back them up on that!
It is far easier to have an organization with the incentive "reliability" and an organization with the incentive "features" and have senior management deal with conflicts when they arise, than to have every middle manager try to correctly split their prioritization in the correct way.
Who exactly does this. And by what mechanism do you enforce:
> take the ability to deploy out of the product org's hands
If it is not "incentives and top down buy in" from above/at both orgs, what is it? Are you literally proposing some reality where VP Alice nonconsensually tells peer VP Bob "you're missing your roadmap ship dates. Im taking away your orgs prod creds because your lines arent up and to the right enough?" Because when there's a conflict between the two it's either going to get resolved by negotiation and common understanding, at and above Alice & Bob, or Bob is going to tell Alice to stay in her lane and get bent.
Edit: For the above, Im interested in the medium/long term. Yes, if you have hived off deployment gates to a different team I believe the could tactically say "no" once. I do not believe they could maintain that position.
Unilateral actions only work when the power dynamics support the person taking action. And those power dynamics are roughly more senior management, broad cultural consensus, and revenue, in my experience. So I'm really curious what enables the dedicated SRE orgs unilateral force of action if its not incentives & top down buy in.
This isn't hypothetical to me. Im a principal systems engineer at AWS who spent years reporting directly to a VP with a billion dollar business, and now work on operations & incident management in the same VPs now larger org running multiple billion dollar businesses. I was there when a billion dollar business took a year off the feature roadmap to focus on availability & performance. No one "told" us to do that, but it was supported as the best thing for the business. I've also stopped working on teams when they weren't making progress with operational investments.
That's one of the key pieces of what makes SRE "SRE" as opposed to an ops team. They are given the power to stop deployment. If they can't, they're not SREs.
> Are you literally proposing some reality where VP Alice nonconsensually tells peer VP Bob "you're missing your roadmap ship dates. Im taking away your orgs prod creds because your lines arent up and to the right enough?"
Yes, and no. No because "roadmap ship dates" aren't reliability. I'm describing a reality where VP Alice nonconsentually tells VP Bob "you've had three outages in the past three weeks, we're out of SLO and so you no longer get to deploy new versions until we have remediations in place to prevent future outages". (and of course they both agreed to this)
Of course, Bob, being a rational human probably starts taking actions after the first outage, and so it never gets to this point, but yes, that Alice can tell Bob "no more deployments" is literally the point.
Google is the only place where this is even discussed. I cannot name any other company that does this. Not Apple, Meta's Production Engineers, Netflix, Amazon's SysDevEngs, Stripe SRE, Uber, Pinterest, and more. This must mean that nobody else does real SRE!
Sweet. Given by whom
> Alice can tell Bob "no more deployments" is literally the point.
Fantastic. Why can she say & enforce this.
You seem to be going a long ways to describe “what” while explicitly ignoring “why” and “how.”
To be clear my thesis is that success in both operating models is built on a combination of top down incentive/buy in and broad consensus expressed as a cultural norm. Im not being intentionally obtuse, but Im yet to see any other hypothesis or method of action whicb would explain the differentiation of ability to execute.
Directly, the answer is the permissions system. In my neck of the woods, devs are quite literally incapable of pushing things to production on their own. The automated tooling handles it 99% of the time, but in the case that they turn it off, only the SREs have production access necessary to actually push new things.
But I think you're speaking organizationally, and the answer there is that that agreement was made by SVPs and is enforced by the PRR process. When you adopt SRE support, you hand over the keys, so to speak.
> Fantastic. Why can she say & enforce this.
Because there's nothing Bob can do except escalate, and SRE leadership will support Alice all the way up to the SVP level.
> To be clear my thesis is that success in both operating models is built on a combination of top down incentive/buy in and broad consensus expressed as a cultural norm.
My point is that the process creates the cultural norm, and enforces everywhere. That is, your org might have these good processes, and that's laudable. But if Andy turned around tomorrow and said "hey, this year your first priority needs to be shipping new features, because we're falling behind", would your org maintain that commitment? Would every exec? The same question goes for every exec speaking to their subordinates.
Its much more difficult to turn to the guy whose job is reliability and reports directly to the CEO and say "hey, your focus this year is shipping features". Like maybe you could say "your focus is shipping features reliably this year", and that should take the focus off of some other things that the SRE org is involved in, but I'm not convinced.
Or briefly, your approach is betting on there being a cohesive company culture of reliability that permeates every team, org, and manager. We both know that's not true, at Amazon or Google or any other company. They're all big enough to have different cultures in different parts. A distinct org and reporting structure, and split responsibilities, ensures that leadership has to be really explicit about such things ("lower your SLOs"), instead of just quietly deprioritizing enforcing or reporting on them. (And I'm generally a fan of multiple individuals with well scoped and distinct responsibility who are charged with reaching consensus over a single individual who makes a decision).
It sounds like you needed a separate chain of command for SREs because your product leadership was incompetent.
If an organization decides that reliability is important (which honestly isn't always the case), then the question becomes whether they can trust Product to include reliability work or not. If not, if reliability is so essential to the business that the business is willing to sacrifice velocity for it, then a separate SRE reporting structure can exist to slow down Product when reliability objectives aren't being met. If reliability is negotiable, then an embedded structure means Product can make its own prioritization decisions and move quickly when it needs to.
The embedded structure doesn't mean that SRE always, necessarily gets overruled. It means that SRE leadership needs observability into when SRE is getting deprioritized by Product, so that consistent failure by Product to respect a core business goal (reliability) can reach executives who will, ahem, clarify for Product that they need to spend some time prioritizing reliability.
The COO is interested in the stability of ongoing operations whereas the other two roles are about launching new things (Unless I have gotten the definition of COO wrong).
SRE is Sysadmins but with better contracts between teams, and automation focused.
I haven't had much trouble with that definition, does away with all the cruft of the "Production" talk by Benjamin Treynor Sloss and the majority of the SRE books (which are a prescription of a method of creating those contracts).
That said: "DevOps" and Systems Administration in general has had an identity crisis for a while. People don't like the idea of "Sysadmin" for some unknown reason, but give the same role to someone and call them "DevOps" and suddenly all is well.
Same is now true of SRE.
Someone on HN said to me: "DevOps is how you get Engineering to respect Systems Administration".. Maybe the same is true of SRE?
I definitely feel the identity crisis of DevOps is much, much worse than that of SRE- but I guess that depends on your circles.
That's not even close to true, which I think is OPs point. For instance, I have an extensive SRE-SE background, but I'm also a software engineer. If I was tasked with "sysadmin" stuff, or shepherding peoples Jenkins jobs every day I'd probably lose my mind.
There are many types of systems administrators; when we talk about as SRE in particular there some overlap with what I was doing as a “Linux Systems Administrator” in a Platform Engineering team, in 2009.
Automating migrations, distributed job control, instrumenting the message queue system, managing cache invalidation across our entire fleet of memcache servers (which was ugly hacked with MySQL, not proud), as well as “maintenance” programs to upgrade “versions” (multi-tenant SaaS) and warm caches, dark launches, canary deployments… well you get the picture.
Of course I was responsible for the reliability of the systems these things ran on, but there was a really-really strong focus on automation as part of the job, and this was when puppet was quite new, we were stuck with tools like cfengine.
Coddling Jenkins sounds like build or release engineering. Which is also a separate thing.
People seem to forget that systems administration has a focus on software development too- I mean for goodness sake terraform, ansible and salt all come from sysadmins; I suspect it’s because of some very foolish admins proudly exclaiming they don’t code. But they were always the bottom rung and end up being “promoted” to management at some point.
As for "SRE" vs "DevOps" vs "Systems Engineer", I'm not so sure. For instance, I typically expect SREs to be able to write application software and orchestration software. I have similar expectations of someone who uses the title Systems Engineer but I might expect less application software development experience. DevOps I think I'd rather forget is a word.
Who do you think did the SRE work before that name was invented? Most likely it was your friendly sysadmin. It is the type of role that is not much noticed when done well, and probably that's why some feel it underappreciated enough to keep inventing new job titles. It's still the same job, pretty much the role that requires privileged access.
The place where I encountered sysadmins the most was people who managed a small collection of assets; the keyword being managed. I didn't see them inventing entirely new systems. Working across DCs, I would see Systems Engineers building these new systems, being escalation support for them, and at times coding up entirely new infrastructure applications.
I don't really buy the idea that administrators are underappreciated, I just don't think it accurately captures what a systems person on a product team actually does, much less the distinction between a SRE-SE and a systems person.
My base argument is that it's not the same job at all. A good Systems Engineer does not guarantee a good SRE-SE because the job is that much different. One of the notable distinctions is writing application code, which is much different from Terraform, Ansible, Salt etc
I’m not arguing against an embedded model, but I do think there’s tension that’s not easily resolved.
Ultimately, the answer ties back into the identity question: what is SRE?
I’m not saying this is the one true way or even a particularly good way, and I agree about the identity crisis (same as with DevOps half a decade ago).
Our team was basically a dumping ground for anything that required experience in many different types systems and/or privileged access. On top of that, we had projects were shoe-horned into roadmaps. So while 90% of our duties were reactive to what the org needed, we also had the pressure of trying to meet deadlines for normal software engineering work. It was unhealthy and I wanted very much to not be defined as an "SRE"
Add on top of this managing multiples of you and having to do the CFOs work for the CEO and litterally manage to the penny the instances we spun up and dealing with all the expectations of ops WRT to developing tooling on top of tooling.
They've also asked us to troubleshoot for database backup processes, write Crystal Reports on user activity, and be Slack admins. Some of the developers believe that we do manual production deployments of their builds (we don't). I get requests all the time from users asking if I can make accounts on various systems for them.
I was given a programming test for this job :(
From a management perspective I think it still makes sense for SRE to have a different reporting structure. Feature teams generally aren't rewarded for investing in long-term code and production health. If you report to a feature team manager there will often be downward pressure to focus less on production health and more on helping features ship quickly.
Giving SRE a degree of independent oversight serves as a system of checks and balances against those feature team pressures.
SWE has an identity crisis. It means too many things to too many people.
Just kidding, our entire industry is filled with titles that mean nothing. I agree, we should fix that.
Can confirm, we have an SRE team that's totally separate from engineering and it's a huge and constant bottleneck. We've literally re-architected parts of the application to avoid having to go through SRE for things.
Bastard Operator From Hell -- A famous IT trope. Which was typically a sysad for a mainframe/Mini-Computer... Usually on a University Campus....
Think Lazlo from Wierd Science, but an asshole.
I actually worked with a BOFH. (Sadly he had a heart attack on the CalTrain Station in Mountain View, and died.)
He was the AS/400 admin for a place where we manufactured all of SUNs Software (meaning made the SUNos CDs manuals packaging etc) and this guy was a BOFH
Sales folk would call up to the helpdesk and request a password reset.
This one time, he set the users password to "STUPID!" ...
I fired him for that.
He had a heart attack shortly thereafter....
:-(
We did this because in our case most development problems didn't have a reliability dimension. We build mathematical models. Do I need an SRE involved in the discussion of switching from one statistical approach to another? In most cases no. But when that team deems themselves limited by their current walled garden and want to explore/operationalize new technology then we need an embedded SRE to facilitate that.
Inherent in the text of technical books that reveal what people did is an attempt to persuade the reader why they should do it, too. In some ways this reveals the central, myopic conceit at the heart of Google: if you don't have Google's problems, you should. And I'm not sure why anyone who isn't Google should just accept this premise.
That's not to say that the premise is unacceptable. I'm sure that there are people who want to participate in a thing that looks like Google, except that they'd rather start it from scratch than join the one that already exists. Perhaps those people want to have Google's problems.
Now we have a piece, expanded from a talk, by one of the editors of the books that persuaded organizations that, if only they would squint their eyes a bit and turn their head, they could make the problems of their organization orders of magnitude smaller by so many metrics look like the problems of Google. And now, perhaps, Google's answer to Google's problem wasn't quite right, because it didn't involve...something? It could be philosophy or human-centered design, and I wouldn't be surprised to find Google bereft of either, but it certainly isn't going to be found in a paper by Google (ostensibly, a Google answer to a Google question) that predates the first book by nearly a decade. It isn't going to be found in people for whom these things are afterthoughts rather than guiding principles. Instead, shouldn't we be asking whether we let SRE as a Google answer to a Google problem be a solution outside of Google in the first place?
99.9% of folks hiring SREs or starting SRE teams haven't read the SRE book.
The SRE book (and its sequels) say quite plainly what SRE is and isn't. They also say that not every org is going to be exactly like google so no, "we're not google" isn't an excuse.
the E in SRE is for engineering. As in software engineering. SREs are software engineers. Or should be. If your SREs don't know basic SWE principles, they're not SREs. If your org isn't applying software engineering principles to minimizing operational complexity at scale, your org isn't doing SRE.
I'm constantly shocked by how hard these things are to grasp, even for most SREs. If the problems I (occasionally) get to solve weren't more interesting than most regular product work, I'd get out of "SRE" entirely.
For whatever reason, Google decided to tell people that the same person who's building the klaxon and the concrete wall and the pipes for the power plant, and the person who's operating the power plant, are one in the same. But that's clearly bunk. Building a part and running a system are completely different disciplines, and anyone who does both will only be half good at both. Humans are shit at multitasking and there's few true polymaths out there. Show me a master programmer and I'll show you an amateur woodworker.
I also don't believe software engineering principles will help you reduce operational complexity. If anything, software engineering tends to either make things either inefficient or subtly complicated. Reducing operational complexity comes from the discipline of operations, which isn't engineering. Non-tech companies have known about these distinctions for like a hundred years. Deming applied scientific rigor and analysis to come up with better practices, but he didn't have to design any widgets to do it.
Depending on the team, SREs can absolutely involved with "building the system", especially the klaxon ;) Examples include designing and implementating metrics used to make make decisions in business logic and or exposed to customers/users, writing routing components like mixers and proxies, developing data pipelines, etc. At Google many SRE teams build and run entire multi-tenant systems with no pure SWEs involved at all.
Healthy SRE teams should be spending 20% of their time on operations. On my team its actually the devs who do most of the operations work. They take the pager during business hours and we route most maintenance tickets to them.
My difficulty is that mandated separation of responsibilities within our org is preventing us from embedding ops in dev.
Anyone successfully fought against this and have tips?
They shadow-IT’d their way to launch and we’re hugely successful, not the business is largely re-orging to better fit the paradigm.
Was a big gamble. The wrong person could have left a mountain of tech-debt.
In tech, software and data is the "productive resources", and the "production system" is the actual system you build out of those resources: the website, API, etc. You don't have to write any software to build and manage that production system. Maybe that's unusual to people in tech today, but it's a fact that you don't have to write a single line of code to build and operate such a system. Heroku, PagerDuty, DataDog, Splunk, Octopus, AWS, etc, all are products built with the sole purpose of enabling operations without the need to write code. You can assemble logging, alerting, monitoring, web server, networking, database, deployment, etc, without ever writing a single line of code, and have it be highly available and highly reliable.
The title will vary (Systems Engineer, Operations Engineer, DevOps Engineer, Site Reliability Engineer, Systems Administrator) but the job is the same: to use Operations Management techniques to ensure the products and services are productive. You can use software development practices for all of this, sure! But they are absolutely not a requirement to accomplish the goal. And many other roles in the company are involved in Operations (QA, PM, DM, etc) and may or may not use code. The business doesn't care about code, it cares that its resources are being used properly and the production line is operating nominally.
In terms of the distinction between builders and operators: you could say that a construction worker and a custodial worker are part of the same occupation because a lot of their skills overlap. They both need to understand how the building works and may need to build/repair parts of it at times. But they're still two different disciplines that require different training, experience, and day to day responsibilities, and as such we don't lump them into the same category.
It depends on the scale and complexity of your application. At some scale/complexity, it absolutely requires writing software because your IAAS provider doesn't provide you with automation that covers 100% of your operational needs and even they recommend using infrastructure as code tools to manage your infra.
If your production system is a CRUD service with 3 application nodes and a managed PostgreSQL instance then you do not need to write software to manage it. But if your application is that simple, then I'd suggest you probably don't need a software developer to build it (Wordpress, Wix).
Construction vs custodian is not a fair analogy because their training and evaluation doesn't really overlap. The training and eval for both "dev" vs "systems" engineers is very similar; most have CS degrees and have to do some leet coding to get the job. Devs generally need to be better at algos, systems engineers need better understanding of networking, os, system design.
Software systems (at least in competitive consumer markets) are constantly changing and evolving. To use the dam analogy, there's constantly a team of people making the dam taller or wider or deeper, even while the dam is running and producing power.
All the SRE teams I've worked with have done a bunch of things that go beyond "operations". They are usually consulted at the design stage, to make sure that the thing is going to be built reliably. They're also responsible for ensuring ongoing reliability as all new features are added. That means that the features themselves don't impact reliability, and that the process of adding new features doesn't impact reliability. None of this work has a reasonable analogue in your dam analogy, except perhaps as some combination of consultant and regulatory body.
This isn't a goal of SRE, in my opinion nor in anything I can recall reading. The goal of applying software engineering principles is to accept the increased complexity in exchange for a reduced operational burden.
There's layers of that effect, and the right one depends on largely on your operational burden. Sysadmins shun complexity, so systems are simple but doing mass updates requires a lot of manpower. DevOps embraces some complexity like Ansible or manually orchestrated containers making it easier to do mass updates, but still a burden. SRE embraces complexity, in exchange for a dramatic reduction in manual effort on many tasks.
The idea is that at certain scales (or reliability requirements), it becomes cheaper to hire a small number of expensive people that can manage complex systems than it is to hire a large number of people each managing a simple system.
Software engineering arises because it can effectively trade complexity for reduced operational burdens in exactly the areas you want. You don't have to migrate to a new infrastructure orchestration tool, you can just write an orchestration tool on top of what's there (which I've actually seen done). Was it perfect? No. Was it cheaper than migrating a half million containers to Kubernetes? Yes.
Operations management tends to be very inflexible. They have a set of tools, and anything outside those tools is either a no go or will require replacing an old tool at the cost of months of effort.
> 90% of SREs and SRE managers haven't read the SRE book(s).
> 99.9% of folks hiring SREs or starting SRE teams haven't read the SRE book.
These are free to read online, for those that are wondering:
https://sre.google/books/
it's like telling a homeowner that they need to spend $1000/year on an annual maintenance item to prevent a _possible_ $15k repair bill every five years.
For some, $1000/yr is too expensive for them. So they take their bets or skimp. (People who think you can do SRE without being SWEs because they "can't code")
For others, $1000/yr is affordable, but because the $15k bill is "unlikely", they skimp. (People who think you can do SRE without being SWEs because even though they can code, they'd rather separate those jobs.)
One of the core observations I made was that few companies were willing to actually get better. Rather than figure out what skills they lacked organizationally, or how to improve their processes, or how to align their goals and teams, or any of the lessons they could have learned, it was way easier and cheaper to just keep on status quo. SRE as a practice, a position, a skill - whatever it could have been, it instead failed to really exist apart from maybe a hero here and there who chose to care about it. SLOs (or whatever name might have been used) did not exist to actually measure or inform anything. They were a kind of placebo - come up with something that looks measurable, find a way to make sure we always score well, and make sure there is a wall between these metrics and anyone's compensation / job performance.
An example of this, at a very prominent health insurer. I identified issues that could have been solved for, let's say well under $10MM. Rather than do any of that work, the company threw about $50MM in new hardware at the problem. In another case, at a national name Pharma company, about every two years I went back out to do the same basic consulting engagement, because they lacked the leadership to develop the skills in house or to figure out how to align their teams with the work they actually did. Still another case (prominent bank), they had this serious problem that blocked their commercial lenders from doing their job... they finally called me in after about a year of fooling around, I got the problem fixed, rolled out a bunch of tooling and training, but I guarantee they never used any of it... again - problem outright blocked key bank personnel from doing their job and yet it took them a year to do anything about it.
So... I guess what I'm saying is that yes SRE work can be fun and rewarding, but it's just not something most companies have the maturity to really do. I read the Google SRE book way back when, but I didn't find it particularly insightful. Likewise, I'm not really sure what to take away from this blog - for me, it doesn't answer any questions about a clear path forward and leads me to believe not much has changed since I left that kind of work years ago.
I'm much better as a SWE but I'm paid more as an SRE and have much more freedom. I don't actually like ops work that much but I'll take that trade off to guarantee the other three advantages above.
All that said, in my short time as an SRE I've noticed that it's very much in an identity crises because there's not really a standard job. Infrastructure is very different company to company and some companies expect SREs to be able to code as well as SWEs whereas others don't expect you know how to code at all.
In my experience, an SRE is someone who's principal goal is to maintain infrastructure and ensure code is able to be developed, built and shipped. In the "better" SRE roles, SREs are able to build platform tooling rather than just ops.
From the SRE vs DevOps definition I just read from the top of google -> this description sounds more like a regular DevOps role instead of SRE.
But IMHO, SRE is just a Google way of naming regular SystemAdmin DevOps role.
I see no difference. I did exactly the same tasks as an SRE and as a Senior DevOps.
Could anyone with 20+ years of experience explain why exactly we still distinguish those ?
SRE is supposed to be a superset of DevOps but how?
Steve Yegge has said in his tech talks on YT that DevOps is often "all ops" and not necessarily involving any dev.
This was the most unfortunate section of this blog for me. Some of the companies that need SRE the most, and have the money to actually fund this practice, care the least because they are the only game in town.
My take on SRE:
1. It is an extremely important and extremely technical practice in desperate need of good non-technical PR (example: most SLOs I've helped craft are guesses from engineering because they couldn't get data from product to support real numbers)
2. It is damn hard to hire for on skills alone (good SREs are good sysadmins _and_ good SWEs, i.e. two hard types of candidates rolled into one)
3. The hires are damned expensive (see [2]), and
4. Ops-aligned BUs that are traditionally given SRE headcount are (still) considered cost centers, which, combined with [1], makes [3] difficult to justify and [2] difficult to obtain as a result.