I meant ECS with fargate launch-type, of course. IaaC support is you can use single CFN template to deploy autoscaled service, with load balancing and run time parameters. No kubectl or helm needed.
I’m sure it doesn’t have “full” feature parity, but do you really need 100% of k8s’ features? ECS is a lot simpler to work with and has plenty of features for most systems.
I don't think you need to take the advice of "use Kubernetes" as a literal "there are no other options", but more as along the lines of "use an existing implementation that you will be able to hire engineers to maintain, rather than a kludge of batch scripts and home rolled abstractions"
> Im sticking with Fargate, full, native IaaC support, feature parity with k8s
Fargate by itself doesn't provide anything that k8s does, ecs does. You can run k8s on fargate via EKS, for example.
> Or plain EC2s in ASG.
This is exactly what this post is saying not to do. ASG's are a subset of the features of k8s. k8s does load balancing, service discovery, deployments, health checks for example. You may not need all of those, and that's fine, but most containerised applications IME benefit from the _majority_ of k8s features.
> I don't think you need to take the advice of "use Kubernetes" as a literal "there are no other options"
It looks pretty literal there. And why everyone talks about service discovery? Put an LB in front of your app instances and give it DNS name. Here, sorted, no service discovery needed, service is reachable at this URI only.
> When you spin more instances as load grows, what updates the DNS?
If you're using cloud LBs, this happens automatically. If not, you can have the instances register themselves in DNS when they turn on. You can also have an out of band system register things in DNS based on rules (or maybe you already have software that supports registering based on health checks so you can add DNS entries from a fixed list of static IPs when they become healthy/powered on)
> When you have more than one service interacting, or when your LB restarts / fails over, what updates the DNS?
You can have the LBs communicate peer-to-peer so they can update DNS when they become unavailable (if A and B can't reach C, they remove its DNS record). Some care needs to be taken to prevent things like split brain but there's established patterns for cluster formation. Something like keepalived could be used. You can also use VIPs instead of DNS
You end up with service discovery either way. Either you discover a load balancer or you discover an endpoint directly. Load balancers allow you to more granularly route traffic with the client being aware of the server topology. This is good when you're exposing services to the web/public clients. On the other hand, load balancers add a certain overhead and can become bottlenecks for high-traffic services
> you can have the instances register themselves in DNS when they turn on. You can also have an out of band system register things in DNS based on rules (or maybe...
Yeah, don't do this. This is _exactly_ what the article is saying not to do. It's a nightmare to maintain, totally non standard.
> You can have the LBs communicate peer-to-peer so they can update DNS when they become unavailable <...>
Or you could use a system that doesn't require you to write custom P2P traffic for a solved problem. Doing this instead of using an existing system is an interesting decision.
Because service discovery is pretty much the next problem that people find themselves facing once they've decided to not use kubernetes or ECS or something, and it's also something that it solves very well, without relying on DNS with all its quirks. To use my favourite saying, "it's always DNS".
> Put an LB in front of your app instances and give it DNS name. Here, sorted, no service discovery needed, service is reachable at this URI only.
Dumping a service behind a load balancer and relying on DNS is a heavy handed approach. Sure it's "simple", for some definitions of the word simple, but so is writing 3
60 lines of yaml and/or terraform to spin up an eks or ECS cluster
> I don't think you need to take the advice of "use Kubernetes" as a literal "there are no other options", but more as along the lines of "use an existing implementation that you will be able to hire engineers to maintain
Yeah, exactly this actually :) I didn't want to get too verbose in the little bullet points (I already use parentheticals way too often lol), but it'd be more accurate to have written "Use kubernetes, or some other tool that takes care of container networking, lifecycle, health checks etc for you" -- I bet if you squint even platforms like GCP app engine would fit the bill here
Great insights! I came to some similar conclusions as well.
I really like the one about failing on missing config parameters. I used to set default params all the time (thinking I was clever).
Another big win in the last few years has been feature flagging. I use it extensively to tune performance when in production. I can control batch sizes, delays, retries, and API key rotation.
> Use Helm. Or some other tool for managing Kubernetes manifests, I’m not picky – the important thing is that you ~never directly use kubectl apply, edit, or delete. The resource lifecycle needs to be findable in version control.
I'd say `kubectl apply` fits better with a Git oriented flow than the imperative `helm install` flow. Nothing about helm guarantees or forces you to use version control. Actually the opposite. If `helm install` fails. You need to `helm delete` first. It has no way of declaratively managing updates at all.
Have a CI job call `kubectl apply -k ./deploy --prune` on every commit is way nicer than helm in my opinion.
I think Helm can be nice to template mildly complex stuff you cannot reach simply with Kustomize, but it's true there are lot of things you can't see until you apply them, even when `helm lint` doesn't fail.
If you need this templating, a think you can do is use helm for that but not for install/deployment: render the charts with `helm template` and then apply them with `kubectl apply` and/or verify the changes with `kubectl diff`.
IIRC Argo CD can show you the differences between the running helm chart and the new version you want to apply if you haven't enabled automated syncing, but I only use Argo CD with Kustomize, so I can't say for sure right now.
edit: by reading the article again I think the spirit of that point is not 'use just helm' but 'don't apply stuff directly'. Long term, if there's more than one person working in Kubernetes, that's prone to failure and configuration drift. Doing the templating in whatever language or system that fills your needs and then using something like Argo CD or Octopus to apply them can remove most of those problems.
> by reading the article again I think the spirit of that point is not 'use just helm' but 'don't apply stuff directly'
Yep, this is what I meant! Just "if you're using some kind of manager for your resources, which you probably are, then ONLY use that manager, don't just `apply` stuff out of band" is the idea :)
I might take a pass later at making that more clear
Helm templates tend to be too complex, in my opinion. On the other hand, having 50 services, deployments, ingresses, etc which have names/labels copy/pasted in 3-5 different places is also extremely complex--especially when you want to make a change to update all of them. Kustomize seems too basic--maybe JSONNet or something fills this gap
Because we can't begin to agree on how to do this in a standard way. The whole idea of git oriented flows seems to be an afterthought rather than being considered an important part of a standard production k8s deployment from the start. Here are some competing options I see after a quick glance in this thread:
And because this is an advanced topic (most of the above require a large investment reading docs and github issues), I'm guessing many/most people never get beyond ad-hoc kubectl commands and letting their cluster run wild.
Funny that most of those things don't compete: helm template and helm install are two entirely different things. Argo CD is just a way to deliver manifests to your cluster - manifests could originate from helm template, kustomize, raw manifests or JSONNet.
kubectl apply -k is...well...Kustomize...
There is a big distinction between first party software and third-party software. Helm is fine for first party software because you control the chart, but awful for 3rd party software because charts often don't have level of customization you need.
Reinstalling shit with kustomize that was installed with helm is like 1/4 of my time at work.
Kustomize won't get you 100% though because something has to generate base files first.
I'd like to second Kustomize. It's not getting enough popularity, IMO. It's also great because it doesn't break tooling by mixing in a templating language on top of a configuration language. If people need more than Kustomize supports, there's always Cue or Pulumi; I'd never recommend Helm, personally.
I tried helmfile, but helm state weirdness became enough of an issue that I investigated `kubectl apply -k` (which uses `kustomize`), and after I got it working it felt like a significant improvement over helm+helmfile. (But I still use `helm` to generate manifests.)
Big +1 to “ Never give up on local testing”. The current code base I’m dealing with has many tests that require interaction with a dev environment on the cloud, and occasionally these tests fail due to a timeout or some other thing not related to actually testing the code (and instead reveals that I forgot to refresh my MFA).
Additionally the dependency thing can be huge; we ran into a weird bug for installing a particular dependency on our CI system so our test there keeps failing, but being able to run it locally let’s us know that the changes did not break our actual code tests.
For golang, but would work in any language. What's really worked is the idea that your main loads configuration and injects all of the configured interfaces into your handler.
You then can build all of the testing on mocks/stubs to test the behavior. If you access a database you access in through the interface which can mimic the appropriate behavior for your code in test vs production. If you need to you can do local integration testing of the db access layer.
With AWS SAM you can invoke functions locally, run api gateway and use step functions. There's also a 3rd party project to run various AWS services locally, can't remember the project name.
There's also SAM sync that syncs changes you make quickly, so if you have your own dev copy in AWS you can quickly test changes.
There's also a fair amount of lambdas where you can just invoke the lambda handler yourself without anything special other than a compatible version of python/node/etc. Variations of things like:
thanks for saying that! people are pretty skeptical about localstack when they hear about it for the first time, and don't understand how we could ever emulate something remotely resembling AWS. tbh sometimes i'm baffled myself, it's pretty crazy what it can already do (i work there).
Honestly, my answer is: don't use serverless, or at least have a non-serverless way of running your code locally, preferably outside of a container/VM. Otherwise you have no way of running or testing your code locally, much less attaching a debugger; IMHO this can slow down dev cycles so much that it cancels out the time saved by adopting a serverless architecture.
Most people building stuff quickly in serverless seem to eschew the concept of local entirely, and it's something I've recently started doing too. I make a code change locally and it gets picked up by the deployment of my stack associated with my feature branch the second I save the file.
It's an uncomfortable concept at first but I find it's helped me build things faster and has ultimately led to less rabbit hole chasing across multiple categories of issues.
Serverless deployments aren't painfully slow like they are with Kubernetes or other container orchestration platforms. At least with AWS lambda you can change and deploy your code in <10 seconds and have it taking traffic (In an infinite number of "environments"). So developing locally is kind of pointless.
> Serverless deployments aren't painfully slow like they are with Kubernetes
What are you doing with your K8s cluster/containers that is making your deployments slow? My cluster at work pulls down containers faster than my local, and deployments are usually swapping within ~1 minute of being committed in git/the container build finishing…
He's not discouraging automated testing; he's just saying that we should prefer integration tests (using services running locally) to unit tests of individual functions. Look up "testing trophy" for more about this philosophy.
Totally with you on the value of integration tests. I'm familiar with the trophy and the triangle, but he's saying this:
'..with unit tests notably coming in last place – i.e. “only if you have some time”.'
The "only if you have time" part is what I disagree with. If we're doing TDD then I don't see how you can avoid writing unit tests, or even deprioritize them.
> The "only if you have time" part is what I disagree with. If we're doing TDD then I don't see how you can avoid writing unit tests, or even deprioritize them.
Perhaps the author doesn't recommend TDD? They don't suggest anywhere that they do. It seems to me that the industry these days sees TDD as a tool for occasional use, not the dogma once proposed.
Yeah, that advice was bad and looked like it was written by someone that's done too many deployments and not enough development.
He's certainly right that deployment validation is the best test at some level. But it completely leaves off the most important aspect. That deployment validation, and the mistakes that you inevitably find, should directly guide your test development to prevent regression.
If you do this, future component upgrades and deployments get easier. Its well worth the extra effort.
Yeah as some of the responses to your reply said, agreed, and I've always found value in unit tests, and the "too many deployments and not enough development" comment from jsight rings loud in my head. Deploying to a real non-prod environment as a "test" can be fine for a team, but to do so while not testing for how you expect the code to behave... maybe that works for some folks and if you're moving super fast, but I wouldn't be super comfortable with it myself.
I totally disagree with the author on testing. Integration tests are more trouble than they are worth in my experience, and unit tests are really useful.
+1 unit tests are essential for software evolution like refactors, updating business logic, and updating dependencies. More generally, good testing requires a defense-in-depth strategy and unit testing, integration testing, and canary testing all have a role to play. All other things being equal, it's much better to catch a bug before you even push to the the source repo versus when it has one foot out the door to your customers.
> I’ve been doing this “reliability” stuff for a little while now (~5 years), at companies ranging from about 20 developers to over 2,000
I've been doing it for 20+ years including running critical services in FAANGs.
> Use Docker
> Use Kubernetes.
Hard pass. There's a reason why most FAANGs developed their own packages management and deployment system and keep using them. They are simpler, less bloated, easier to debug.
> Deploy everything all the time
On your local testbed, maybe, unless you already know a commit is broken. On production, absolutely not.
> There's a reason why most FAANGs developed their own packages management and deployment system and keep using them. They are simpler, less bloated, easier to debug.
And for every other company that isn't one of those 5, rolling your own deployment system won't be simpler, leaner or easier to debug. Nor will it, in all likelihood, be as stable, feature rich, observable, etc.
> There's a reason why most FAANGs developed their own packages management and deployment system and keep using them. They are simpler, less bloated, easier to debug.
FAANG-appropriate solutions aren’t always appropriate for the rest of the world.
> There's a reason why most FAANGs developed their own packages management and deployment system and keep using them. They are simpler, less bloated, easier to debug.
Most FAANGs developed their computing infrastructure before Kubernetes gained popularity. After all the investment into building it and fine tuning for their software services (like ads), they probably have a lot more reasons to not migrate to something different. Not just because their systems are "simpler, less bloated, easier to debug".
> Hard pass. There's a reason why most FAANGs developed their own packages management and deployment system and keep using them. They are simpler, less bloated, easier to debug.
That seems like an odd thing to say when Kubernetes came out of Google and they based it on Borg, their internal orchestrator.
FAANG create many things themselves as they're trend setters. They have the size to create them. Having worked on in-house solutions, it is incredibly difficult to maintain adequate documentation, training, and resources to keep it going.
More often than not, this ends up being a business decision about balancing risk with potential loss of revenue (you don't want to take your stuff down, but you also don't want to delay new features that could bring more business)
Fast growing companies trying to enter the market tend to favor "deploy more often, break more often" while companies with large, established bases tend to favor "more testing, more deliberate changes"
I would say the inverse is a better rule "you should be /able/ to deploy any time"
Why, deploying everything all the time is fine, it's what FB, for instance, does. If you don't, there is a hazard of accumulating too much change per deployment, which becomes more dangerous to deploy, and more problematic to roll back.
But it does not mean "deploy every change to 100% of the fleet instantly". Use canaries, use slow, monitored rollout. With smaller changes, it's relatively painless and routine.
Hmm, I think "config" is too vague here. I've seen numerous incidents caused by NGINX config changes that would have been far worse if we had instant 100% rollouts, but I'm not sure this is the kind of config change the author is referring to.
> Run 3 of everything
What is a "thing" in this example? Is it an instance of a service? A database? Something else?
If you're using Kubernetes and have autoscaling, what does "3 of everything" even mean? It sounds like the author is basically talking about databases..?
> What is a "thing" in this example? Is it an instance of a service? A database? Something else?
No, you just run three of everything. While only databases and similarly distributed data storages really need exactly three (or sometimes, just an odd number larger than two) for quorum and network partition detection, there is a good possibility that you'd have three availability zones because of your databases. And since you already have three availability zones, it's only logical to put every service in every zone at least once, so running "three of everything" becomes a rule of the thumb, easy to remember, easy to implement.
> If you're using Kubernetes and have autoscaling, what does "3 of everything" even mean? It sounds like the author is basically talking about databases..?
If you use autoscaling, you still need to start somewhere. And this somewhere is "three of everything".
I disagree. If we're talking about distributed systems, then one thing is guaranteed - network is not going to be reliable. And if we have 10's or 100's of services, this policy means that at the smallest blip the whole thing collapses like a house of cards. If the concern is "it's hard to troubleshoot", well then perhaps implement better logging ("connectivity to service X has been unreliable with X% failure rate over the last Xhrs" instead of "connection terminated", or no logging at all).
The stage of testing is important here. Early on, I agree with the author. Catch code/config bugs early with no wiggle room for retries. But later, testing in a live system that lives with unreliability, you need those configs that allow for it. This enters the chaos testing phase, where you can assume with some degree that the code works deterministically, but now you have to test how it works in non-deterministic settings. Or more likely, why it failed in retrospect and how it recovered previously. This is much harder.
You are 100% correct. It's crucial to understand and manage the failure modes of your system around transient network failures and permanent bottlenecks.
Retries are a must in most systems, but need to be planned, otherwise you DoS your own network or services.
This particular concern costed some 10 folks 2 mandays per head last week across the globe and due to consequences of the issue got escalated to higher management.
Different topic a bit - messaging & routing ecosystem working 10 years without flaw suddenly started exhibiting slowness and randomly would just stop, needing restart of client. We debugged like crazy, java messaging system by me on one end, Tibco ems system on the other. Tibco refused to help due to server being out of support (note for us/US team). We had network guys on 13h call too, but they didn't have as much experience with WANs.
After 2 days, they discovered some internal backbone network system between US and Switzerland just failed out of blue in the worst way possible - dropped some +-20% of the packets, so things kept chugging along somehow, till they didn't (some Acks on transaction commits rarely didn't happen and then all got blocked without a hint why).
2 lessons - don't always doubt yourself and your skills when SHTF. And don't take things like servers, OS, network for granted. Don't expect some clever monitoring already in place will figure out issues for you.
> If we're talking about distributed systems, then one thing is guaranteed - network is not going to be reliable. And if we have 10's or 100's of services, this policy means that at the smallest blip the whole thing collapses like a house of cards.
With RPC, I believe the author is talking about retries at the application level. There are already enough retries in the TCP layer below it that happen with exponential backoff. Tuning that and also your HTTP library's timeout settings is possible if you happen to have a unique enough network that the defaults don't work.
But very likely, your slowness or problems will exist in the application layer – either on "your" side (your service is tied up doing something too long) or on the other side (their service is tied up doing something too long). The correct fix is to "Fix the flaky service!" as the author recommends, and this can take many forms – spin up more copies of the service, or fix any CPU or I/O resource problems.
Slapping on another layer of "just retry" on top of all the other retries at the application layer is what the author is recommending against – this is because you will end up inventing a newer, complicated model of a distributed system.
TCP retries won't help if your backend is restarting, if failover switching is happening, if your overloaded cluster has just been scaled up to add nodes, etc.
A reasonable application-level retry policy (exponential randomized delays, limited attempts) would turn these from a service disruption for the client into a mere delay, often pretty short.
> TCP retries won't help if your backend is restarting, if failover switching is happening, if your overloaded cluster has just been scaled up to add nodes, etc.
Yeah, this is where the nuance begins.
You are correct that TCP is not always sufficient. Perhaps where we differ is that in my experience, it still helps for this to be a feature of the framework or infrastructure that the applications are running on (e.g. a retry budget in the service mesh, or a load balancer) rather than scattered around in the application itself. At some point it becomes a word game – you could say that the service mesh is also kind of an "application" itself, but the core principle is that the retries should be kept in a few simple, common places that are rarely tuned.
Otherwise, you will find that N developers who are tasked with figuring out something like this will scatter N different version of your exponential randomized delay policy all across your codebase. It is always possible to avoid this with enough code review discipline, but once the trend starts, it's much harder to say "No, you need to fix this the right way".
It’s pretty important to implement a mechanism to prevent a thundering herd if you can, and TCP doesn’t help here. Accidental synchronization can have many many causes.
In addition, despite fantasies, tcp in particular is an instance level concept and reserving the ability to shoot instances without draining first is a good design.
Application-level user-perceived reliability is an end to end concern, not a transport layer concern.
My experience is that you need to plan for retries at the system level anyway (above application; think “dead letter queue processing” or equivalent).
When you don’t (as we sometimes haven’t), you end up with people having to write scripts or perform manual actions to poorly approximate what the system should have done, often with thousands of orders (or whatever entity matters to your company) in a limbo state.
TCP eventually gives up though, right? Like if you want to tolerate network partitions that lays days or weeks, during which the other node might have received a different address, then you're gonna have to move that logic up the stack.
I was just about to post the exact same thing. 9 times out of 10 the issue isn't a broken service. It's going to be something environmental. You absolutely must have retries. Set alerts for elevated numbers of retries so you can respond when it can't recover on it's own but don't leave these out. You'll just generate alert fatigue for whoever is on call in your teams.
The answer is to "kick retries up the stack"; when you fail to reach a service, you return a 503 and have your clients retry, to avoid a case where every service in the stack starts retrying all at once and causes a massive increase in traffic.
IMHO you should only add retries if it proves necessary in practice to reach your SLA, if you're building something that isn't itself triggered by an RPC, or if you're performing an operation that can never be made idempotent.
I like the idea of pushing the retry up to the caller, but in a lot of apps I've seen they've built up an in-memory object-graph from the results of calling out to other services. Wouldn't failing due to one bad service and asking the caller to retry result in every other service being unnecessarily hit?
I had a tech lead who vehemently agreed with the parent commenter (retry from the top), but I ended up learning different lessons.
* Differentiate retryable and non-retryable errors. If the service can't return success because the DB it queries is borked, it should send a non-retryable error. Then it won't get overwhelmed by retries from upstream.
* Retry configuration should have sane defaults. Even "retry once" is too often; many services aren't overprovisioned for 100% increase in traffic. What ended up working here for us was having the retry module collect req/sec statistics, then only allowing 20% of that number in retries/second. Individual requests can be retried twice. That was small enough to not push over any services, but enough retries to compensate for garden-variety unavailability.
* Services shouldn't serve requests first-come-first-served under load. When SHTF, fairness means everyone has to suffer long delays, often much longer than rpc timeout. Instead, serve them in an unfair order - the most recent requests to come in are the most likely to have a caller still interested in them. Answer those!
* Use headless services in kubernetes. By exposing the replicaset to the client, the client can load balance itself more intelligently. Retries should go to different replicas than the failing request. Furthermore, you can perform request hedging to different replicas than the lagging request.
* Define a degraded form of your in-memory object-graph. If a feature is optional to the core business flow, it shouldn't take down your whole product. This one is a lot more involved. We needed custom monitoring for degraded responses, in-memory collection and storage of "guesses" to substitute for degraded portions of the object graph, as well as some other work I can't think of right now. This does enable an organization to compartmentalize better, having faster, less fearful deploys of newer initiatives.
It seems like knowing the difference between a retryable and non-retryable error is itself difficult (perhaps impossible in practice).
DB is unreachable? Is it the DB or my host? If it’s the DB, maybe it’s not retryable, but if it’s me (and I’m load-balanced), it is probably retryable.
Sometimes you want the exact opposite, though. Consider an endpoint that makes 100 behind-the-scenes requests (say, to S3). You absolutely want to retry at the lowest level, not the highest level. You could fail on the 99th request. If you kick it up, the caller will retry and you'll do those 99 requests again, instead of just retrying the one that failed. With enough requests, there's a point where you're unlikely ever to succeed without one of the calls failing if you restart from the beginning every time. I don't think you can "one size fits all" this, and that's one of the reasons retries are hard.
I use S3 as an example because it has a comparatively high random failure rate. You MUST implement timeouts and retries if you're using S3 as a backend.
That's why I said "except if your SLA requires it." If you've agreed--with other teams, with outside clients, or just as a project goal--that your service should work on 99.9% of calls, and you find in practice that you need to retry S3 calls to meet that target, then adding retries is reasonable.
If the problem is just that the 99 calls could overload downstream services on retries, the ideal solution is to add rate-limiting, though admittedly that is an imprecise science.
There's never a reason that a human needs to read an error message caused merely by Kubernetes restarting a pod in the normal course of operations.
Passing every error up the stack causes people to be confused by 500 status errors for trivially retry-able calls. Browsers (and TCP) natively implement retries despite there being no SLAs in place.
If a client retries it’s not necessary for all 100 requests to be attempted again. The system could be designed to only repeat the unsuccessful operations.
You do want the policy on retries to live and be evaluated at the outermost level where your business logic lives. If that’s living across an RPC boundary, then you’re stuck making this weird trade off and where I think this back and forth is happening - people have different mental models of the specific service they have familiarity with that they’re using to test the recommendation against and because there’s a trade off to be made, the advice can be correct in some contexts and wrong in others. If you can encode your policy and it’s evaluation generically so it flows through the stack, that’s not terrible although it becomes hard to manage certain other kinds of SLAs (eg bounding latency of your overall operation or the latency of a specific suboperation).
> Consider an endpoint that makes 100 behind-the-scenes requests
Don't do that; Don't ever make middleware retry.
An API endpoint should be quick, and if you can't figure out how to make it quick, use batch processing metaphors (submit task/query task) and a task-runner instead. The task-runner can retry indefinitely, but a user needs to be given feedback or they will push the refresh button.
Lisp's conditions and restarts are great for this, so you can make the policy decision at a high level while not fully unrolling the stack and allowing retries to be done in the original context if that's the verdict.
Retries are complicated to implement properly, you have some many timers in an HTTP request ( tcp hanshake, tls, headers, send bytes etc ... ) Let say you have 15sec to do an API call you have to take all the timers in consideration for the retry to be ffective and also to cancel the request if you go over 15sec.
The network is unreliable. Technology has become pretty good, to the point that we have become spoiled and take the reliability for granted. But it is at its very core an unreliable medium. Applications should expect that and survive accordingly.
Now, before packets go out to the network, or after they have made their way inside your network, they are in an environment that you have a degree of control. Errors and retries should be monitored. If they increase and remain elevated, they should be investigated. But guess what, if your services are resilient, you should have some time to investigate before things start breaking down - as they would if you treated everything as if it was _localhost_
Case in point: our app started breaking in horrendous ways once we deployed it in China and it tried to cross over the Great Firewall. Most issues would have been survivable if they just backed off and retried. The GFW doesn't usually like the first attempt to/from a new address, but it will usually allow the traffic after a while. We had to pay a company a not insignificant chunk of cash to get better connectivity (that would have been fine as an improvement, but not to make the system work at all).
Retries are at the core of everything we do. TCP has retries (if the author followed their own advice, they should switch to UDP!). Kubernetes has a whole bunch of retries before reaching error states: CrashLoopBackOff, ImagePullBackOff. Your ethernet card has retries. But it also tracks errors.
Track errors. And have retries. For as long as it makes sense to retry for your use-case.
> well then perhaps implement better logging
Not logging. Metrics! Logs are good for troubleshooting. For signaling issues, metrics are better. Errors can be a simple counter that you increment. Have Prometheus or similar scrape that and throw alerts as necessary.
If you do structured binary logging, then it is better than doing unstructured/text logging and metrics separately. With structured binary logs you can extract whatever metrics you want to your hearts content and you have the freedom to turn metrics processing on/off as needed.
Yup, better advice (imo) is put metrics on all incoming requests and any external services. Add request count, request time buckets, and error counters. A lot of APM agents will instrument this automatically but it's still easy to do yourself. Some frameworks have this functionality built in
I'm with you. I recognize that "fail hard and restart" can be a valid design, but if you aren't careful you're going to hit some form of thundering herd problem at some point.
> If your service can’t load the config on startup for any reason, it should just crash
Not always the case. You don't want to carry on as usual as if everything is normal, but running in a degraded state or dedicated failure mode that you can coax information out of is often quite useful.
It almost seems like the author err's on the side of KISS to a fault. Writing off or failing to recognize things like retry-logic, metrics, logging, failure modes, etc.
It strikes me that the author may have dealt with a lot of projects that are far from feature complete and may not have seen one taken one all the way to rock-solid. And by rock-solid I mean a system that auto-remediates to a certain extent and lets you ask questions of when the sky is starting to fall.
That said most of the advice is alright, especially for a relatively young project. Rome wasn't built in a day and much of the failure behavior I tend to plumb into projects gets added only after a problem is encountered.
In practice, due to the way network / system failures tend to work at scale, failure of a first retry is generally strongly correlated with failure of a second retry. Thus a second retry can be more problematic than a first (especially if each retry causes load). From that you can infer that a single retry at the highest level is the right approach (most of the time as always YMMV). It's worth measuring this for your own services in production with real workloads by including a metric that captures how often a first and a second retry succeed.
When you don't choose zero / one as your multiplier, there's a strong risk of implementing a retry strategy that is multiplicative. E.g. given 3 layers with a try and then 3 retries at each layer you cause a potential 64x (4x4x4) amplification of any failure at the lowest level. Retries are an easy way to overload a service that would otherwise recover from a problematic situation.
Adaptive retry using a token bucket / circuit breaker approach are reasonable second alternative to zero/one.
In practice, for resilient systems, you can actually go even further than zero retries when you have shared knowledge of an outage to the downstream service (due to concurrent calls from the same source). You can choose not make the call altogether, and look at only making a small amount of calls to the service to let it recover sanely. Obviously this is only useful for calls that are optional parts of a call chain. An example implementation skip a percentage of calls based on the percentage of failing calls (e.g. if 50% of the calls are failing due to an overloaded downstream service, backing off to only make around 50% of the call volume is directionally appropriate).
Better logging is always appreciated regardless of situation ;)
Hah. That made me smile. But that's my issue with most "kids" these days. The code is in rust, takes dark wizard incantations to understand, a treasure map to even compile, ... And then the rest is built on yolo-yaml reproducible like that time I did pancakes that didn't suck.
With Ansible, you first start a machine, then build a configuration on it.
With Docker, you first build the configuration, then start the machine (not a real VM, but an isolated enough slice).
The upside of a Docker image is that you build it once, and then start instantly, no need to wait for Ansible to run through steps.
The downside, but also an upside, of Docker is that you can't make persistent changes inside it while it's running, and usually you avoid such changes. You always can restart it from a known state.
Of course Ansible can handle more aspects than just application software configuration, basically any aspect at all. This is its upside, but also a downside: you can make an unintended system change along with some innocent-looking operation by mistake, and nothing will stop you, there's no separation of concerns.
Yes. Although, typically some od those other dependencies come with docker images themselves (such as Apache). Also, depending on your definition of dependency, some app dependencies need to live inside the image too (ie shared libraries, or app deps)
Typically, if using bare nodes, you’ll see OS level concerns (disk, networking, init systems, etc) managed via a config management system, while app level concerns (lib dependencies, env variables, port setups, etc) managed via a container.
Yes, it's possible to configure some aspects of your machine via Ansible, and run Docker containers on it.
You can of course run a non-dockerized, locally configured Apache, and make it talk to backend code that lives in containers which just expose TCP ports or Unix sockets.
It allows you to package all the dependency tree horrors of a Node app, or of a large Django app, into a container once, at build time, and just put them on a host where you run them. You're guaranteed to run exactly the same code on your prod box(es) which you've tested on your dev and CI boxes, with all dependencies, libraries, etc guaranteed the same, never reinstalled.
Eventually you may discover that it's convenient to put the Apache into a container, too. Suddenly it stops depending on the distro your host is running.
You may also not need to run Docker proper on your prod machine(s); in simpler cases systemd will start / restart your containers just as well. During development though, docker-compose is invaluable, it allows you to locally run nearly your prod configuration of services, at a tiny scale.
One challenge both containers and scripts tackle is reproducibility and in that case it is solving that problem.
With Linux competency and some thought put into design, you can find a balance between reproducibility and ease of use with Ansible. Easier for me and those who agree with me to configure and debug a Linux host vs. debugging and configuring Docker to be as flexible as running it straight on the OS.
It’s not an impossible task to harmonize configurations with Ansible. It definitely takes more thought than Docker and that’s where the competencies of the team count.
Do you test your ansible roles? If not, you absolutely should be. Check out molecule, which coincidentally uses docker. It'll definitely make your ansible roles better, and possibly help you learn docker and how it does something different from what ansible does.
No SSH is better than any SSH :) When your infra matures enough, you'll bake machine images (AMIs in AWS) and start machines from this image. And containers already operate this way, thats why they popular.
You'll find yourself in the world of pain the next time you need to remove something. Over time, your playbook will be full with stuff that shouldn't be on the machines. At some point you'll consider re-installing the machines from scratch from time to time. At some later point you'd consider re-installing machines every time your Ansible playbook changes. And at that point you'll have reimplemented containers. Poorly.
I agree with most of these points. They are a good starting point. I'd probably steer clear of kubernetes until you actually really really need it. ECS is good enough for most things, and a fucktonne more simple to look after.
but, where I hard diverge is the lack of metrics. Everything should be generating metrics. Logs are great, but they are crap for giving you near realtime trends.
Everytime your container gets a connection: increment a counter. Every message processed, counter. every message failed, counter. Every KB of data sent out, counter. Every service call, counter by service.
metric all the things, in a sensible, mostly automated way, with a decent schema.
Then you can combine all those metrics into a dashboard that shows your system performing against business goals.
This list has things I agree with and things I don’t. You could probably make another similar list that didn’t overlap at all.
That’s because the real way to build software that’s reliable has very little to do with a list of features/tools it has to do with designing for the sad paths at the same time as the happy paths.
For each failure modality the app should have a conscious decision about what the app will do when the bad thing happens. This can be everything from “this will cause an outage” to “this will route to an alternate system”. What’s important is figuring out up front what those decisions mean for your data, your operations, your business and your users.
Without that all the structured logs and version controlled configs in the world are just window dressing.
I agree with this. But I really see this from a business perspective. I think k8s works better at companies with large, complex engineering teams and infra. A lot of companies simply aren't ready, nor will they benefit from, introducing k8s into their stack.
GKE/EKS/AKS lower the bar by a huge lot. There is a ton of value in having just one platform, toolchain, deployment language, set of templates/blueprints, set of dashboards, set of alerts, target for hardening/securing, etc. pp. One of the big upsides of this for me is that it becomes a lot more tractable to get ordinary devs and others involved since you only have to polish things and train people once.
I've found it's quite neat for e.g. data scientists to be able to toss a quick yaml into git and have their thing running to their specifications in non-prod. Things like flux and Kyverno make sure they are boxed in tightly enough that they can't cause a lot of damage, and if that thing works out, someone adds CI in front of the yaml and some templating to the yaml itself and off we go to prod, and the result is still quite comprehensible, so people are enabled at least to ask the good questions to the right people. That's what I see as one of the killer applications of Kubernetes, to build a lingua franca interface for all things ops that doesn't change (much) for different service levels and even totally different needs, like frontend devs and data scientists.
It's probably not a great choice for a three-person startup trying to gain traction by moving fast, but IMO the threshold where k8s starts making sense is a lot lower than many here appear to think.
> Assuming you have more than one service and more than one instance, you either need or will need stuff like service discovery, autoscaling, and deployment versioning, and you do not want to manage all that by yourself.
To paraphrase, "If you have anything but a tiny amount of complexity to your architecture, you must add in a high degree of complexity". It sounds insane, right? Like, we're skipping some middle steps.
Yes, Kubernetes solves those problems, BUT ALSO the care and feeding of Kubernetes clusters themselves is a significant time investment. Manual scaling sucks, but if you have a good automation game, it's not that bad. Often what people think is a need to scale extremely rapidly is really an organizational problem they are trying to automate around (example: An ECommerce site sends out coupons to 10 million customers without telling the devops teams).
While this is true (btw, Azure as well has a managed offering) the complexity is not the only issue. Cost!
The article is basically advocating, "if you have more than one service, you need K8S", and sticking your app on Kubernetes in the cloud (managed or not) is a huge cost upgrade from a single or handful of servers.
You're right, but the higher your service count gets, the more you should probably be on k8s. Large scale ECS or Lambda stuff is much worse to maintain
1. The higher your service count gets, the more you should probably be on some container orchestrator. There is more than one.
2. The number of services you have to manage to justify moving to a container orchestrator is most certainly not just > 1. That is crazy. I don't know the right number, as it probably varies according to how mature your organization is and how well the parts work together. But it is definitely higher than > 1!
My last point is: If all teams worked together smoothly, or one team had infinite capacity, or if automation was perfect, etc. etc. etc. we wouldn't need containers, let alone container orchestration. These are solutions to people problems, not technology problems.
I think better phrasing for this would be "develop 12-factor apps". At scale, Kubernetes is great.
When things are still small and simple, an Ansible Playbook and a Docker container can take a company pretty far while still being able to move to k8s pretty easily.
Time and time again, commenters suggest that the reason to use Kubernetes is to enable easy scaling, but this is not the case in my organization. Kubernetes is valuable because it is a standardized and flexible deployment model.
>The highest-value-per-time-spent kind of test is just pushing your change to staging (or better, prod!) and showing it does what you wanted and doesn’t break everything
Ah yes, the time honoured testing in prod strategy
Well, "this but unironically", really! One of the main tenets of how I hold SRE is that you should be confident in sending pretty much _anything_ to prod, because you trust the system you've built around it to catch problems before they lead to an outage. And in a world like that, with good shadow/canary deployments etc -- there's no better way to validate your change, right?
I have worked at some places where SRE guys were a joy to work with. But at my current place they take a cowboy approach and make my life harder. So I'm currently not interested in building software like an SRE.
That used be my perception as well. I have now had back to back jobs where there is some maverick in the SRE team. A single guy who modifies some configuration at 2AM for services your team supposedly owns can ruin your week. Some people think lack of bureaucracy is a get out of jail card, and they are often right.
As someone who has managed a team of SREs that kind of behavior would land someone on a PIP if done repeatedly. Sounds like you had a run of bad luck in teams.
A PIP would be nice tbh. At my previous place they PIP'd one of those guys, but they ended up moving him to my team, where we taught him a bit about processes.
> Use Helm. Or some other tool for managing Kubernetes manifests, I’m not picky – the important thing is that you ~never directly use kubectl apply, edit, or delete. The resource lifecycle needs to be findable in version control.
I have to partly disagree with that one. I find tools like Helm to obscure things that should be readily visible. My favoured method is to keep manifests in full (which you can source from `helm template`!) as pure yaml files and version that. If possible freeze versions and go through regular patch cycles to review updates. That you apply them through `kubectl apply` or through Argo is irrelevant. I treat the repo as the state and the running cluster as stateless. If it's borked, just redeploy. I don't see it as useful to care too much about the in-cluster resource life-cycle. But I completely agree that resources need to be version-controlled.
Agree 100%. Plus Helm runs against a basic tenet of microservices (the usual architecture for the deployed apps in k8s these days). People use to bundle services together when using Helm and the like which, in time, couples services together.
But don't you actually want a certain amount of coupling in the operations part? After all you need to ensure they run in a place and in a manner that allows them to find and talk to each other and usually in a combination determined by the goal of a specific deployment, i.e. sometimes you might not need a rail-cargo-service because the customer only ships by truck, etc. Then scaling/autoscaling (if any) needs to be compatible, versions need to be within a certain range, any central data store must be coordinated as well, not to speak of service meshes, chaos experiments and the like. It's a good thing to develop services with minimal coupling, but that stage has different risks and goals from devops/ops, at least in my experience on both sides of the dev/ops transition zone.
> But don't you actually want a certain amount of coupling in the operations part?
In my opinion, for sure. There's a balance between "too coupled" and "too de-coupled" that should be stricken rather than too far on either side. It's good to say that this is also contextual; some projects may be fine with either more or less coupling than others, and that's OK.
Agree with this. Helm is a great tool for making really terrible abstractions over well designed native configuration. I either end up having to fork charts and fix or literally just writing my own half the time.
It can be used for good and it can be used for bad. Less is more with Helm otherwise it will create bad.
I'd prefer to deploy full manifests as well, but it's not my impression that you can entirely obtain those though "helm template". Certain variables, like "Release.namespace" are only available when actually being applied, AFAIK.
You will get a manifest, but it will usually be missing certain parts of it.
I completely agree with the philosophy of just redeploying the cluster if it's borked. I'm using NixOS myself for the task, and was trying to obtain full manifests though "helm template" originally -- so I'd love to know, if I was just missing something.
My memory didn't quite serve me right, so it's not exactly as I described, and I can see, that as you say, providing the namespace to the template command does work.
The problem for me is, that setting the namespace in that way with "helm template" does not seem to add it to any manifests not specifically specifying the namespace to .Release.Namespace.
The rancher 2.6.8 chart does not set this for all manifests, but does with some, so when I set namespace though template, and deployed it all through the manifests folder, I got some objects in default namespace (because they had none specified), and some in the intended namespace; resulting in an installation that did not work.
As another reply to my comment suggested, this can of course be handled with post-processing of the result of "helm template", though, at the time, I was not certain the problem was limited to this namespace issue, so I didn't feel lucky enough to go down that route. :-)
That could definitely work. And I considered it a bit but didn't feel confident that the problem would be limited to the namespace, so it felt like the wrong tool for the job at the time. :-)
Thank you for the suggestion though. It's comforting to hear that it may actually be a viable approach.
We've deployed our helm charts with Spinnaker. Spinnaker has a nice UI that shows which charts are deployed, which environment variables were used, and the manifest files themselves.
> I find tools like Helm to obscure things that should be readily visible.
Goodness, is this ever true. Particularly with Prometheus Operator and all the monitoring bits that go around that. Dealing with this infrastructure breaks a number of the points in the article, like "Deploy everything all the time." and "No in-code fallbacks for configs."
A previous team built this monitoring infrastructure, so when I had to go back in and re-deploy, a bunch of the Helm charts were broken (YAML errors and the like). It hadn't been re-deployed in likely 4-6 months.
Then a lot of the components don't rely on default configs, but the default configs are there nevertheless. So another team was troubleshooting an issue, and they reached the conclusion that the config for AlertManager was empty, but it's not. The config for AlertManager is in a different directory from the default config. Then an issue with Prom2Teams came up, and Prom2Teams gives an error for its default configuration file that it doesn't have permission to load that file--Prom2Teams runs as a user, the file is readable only by root. So another team came to the conclusion that Prom2Teams can't read its configuration file. But that's not the file it's actually using to configure the service; it's just a default.
So two red herrings as a result of default config files that aren't being used at all, compounded by Helm obscuring components that should be visible, and ultimately stemming from the inherent complexity of the system.
But in reality, there are issues that make this worse which are unrelated to Helm, Kubernetes, and the Prometheus stack.
I recently discovered `kustomize` and the `kubectl apply -k` flag (which uses `kustomize`), which makes keeping full manifests pretty straightforward. There's only one or two things I dislike about `kustomize`, but those are things that can be worked around.
Kustomize has the best local dev environment for k8s that I have found; hot-reloading your cluster as you edit manifests gives a very tight development feedback loop.
For my money it is worth spending the time to grok the slightly funky overlay semantics, at least for teams with infra focus / dedicated SRE.
However, I’m not certain I recommend it for small teams doing full-stack DevOps, I.e. engineers deploying their own code that aren’t k8s experts; if you only work with the Kustomize layer infrequently it can be a bit annoying/unintuitive.
Note you can still use a two-step GitOps process where the Kustomize scripts write to your raw config repo; I think this is a good middle option that keeps the infra legible, while allowing developers to get the ergonomics of a bit of dynamic logic in their deploy pipelines. (Eg parameters for each environment).
> My favoured method is to keep manifests in full (which you can source from `helm template`!) as pure yaml files and version that.
I do something like this but I normally find that helm charts are not parametrized as I want and have to manually modify the output manifests. When updating from helm it can be challenging for other team members to understand what bits we want to take from the new helm template output and which ones we don’t. How do you deal with this?
Sometimes I update the helm chart to fit our use case, but it’s still hard if that is not merged upstream (because that means maintaining our own version of the helm chart)
Isn't that the problem that kustomize is designed to solve? Flux even has a first-class declaration for "take this thing, then kustomize it". The `helm template` into git pattern could be extended to "helm template, write kustomize files, then version control both" since it would capture the end state as well as the diffs that were applied on top of the vanilla chart
I think the "maintaining our own version of the helm chart" is only painful if the helm chart itself is moving around a lot, versus they're using helm releases to carry changes to "--set image.tag=$(( tag + 1 ))" type thing
> Never give up on local testing. It keeps dev cycle time much shorter than needing to rely on (and fiddle with) CI or remote workspaces.
Excellent advice for improving productivity and testability. I wish more teams would be sure to follow this by not adopting technologies that are impossible to test locally.
The "S" in "SRE" is pronounced "ess" which would take an "an" as the article. The title is grammatically correct. You would also say "an FBI agent" or sending "an SOS message."
No, "an SRE" is correct here; it's actually "a SRE" that would be bad grammar.
Whether you use "a" or "an" depends on whether the next word starts with a vowel sound, not with a vowel letter; since the letter "S" is pronounced "ess" in an acronym, you would write "an SRE," but "a Site Reliability Engineer."
I disagree so much with discarding unit tests, if you’re not dead sure of the base units, then you have a combinatoric explosion down the road.
How do you even design code without unit tests :D
Discarding unit tests is such terrible advice. The author clearly never discovered regression bugs at build time, or developed using a dynamically typed language.
Also, I would have expected to see "Use only strongly typed languages" as one of the bullet items. I get the sense that the author doesn't actually have a ton of enterprise experience.
Ah hello, author here! I wrote this up last week and posted it to Reddit (didn’t expect to find it on HN!) and I should probably say the same thing I did over there — this is a list of things I have learned are important, based on my own experiences. I’m sure others have had different experiences leading them to different opinions (sometimes even opposite to mine!), which is good, and I’m enjoying reading all the viewpoints here :)
I disagree with this. Logs should only be used by human operators; if you want something machine-readable, you should expose metrics via some other channel or record errors in a dedicated service.
Sure, some tools will let you read structured logs quite cleanly, but it will never be as nicely formatted as something designed, first and foremost, for humans. Of course, you may need to (say) tack on a request ID to avoid logs from different threads/goroutines/etc from getting interleaved, but that doesn't require everything to be in JSON.
I suspect that you will find a lot of resistance to the idea. At least I have. Machines should be consuming metrics, not logs, unless there aren't any alternatives. That statement never seems to be controversial, but people... just don't do it, for whatever reason.
Also, the author seems to want to do a poor man's version of distributed tracing.
There's some logs which you want machine readable that are not necessarily metrics.
Let's say you have 10 services in serial, and you want to get precise information on how that request was performed - because your customer reported it performed super bad. To do that you might need to get logs from all those 10 services and align them. If they all are human readable and you can filter by a common request-id, that's easy to do. And you can even have tooling on top of it that automatically determines what might have happened by looking at other fields of the log entries.
Metrics are mostly just pushed to a metrics system, and that moment you don't care about them or the relation to an end-to-end request anymore. So while all services might have emitted a metric for the request, looking at the overall metrics won't tell you if outliers in the services refer to the same end-to-end request or different ones.
A colleague of mine once shared his lay sociological theory about dev vs ops, and if taken for what it is -- an essentialization -- it's an interesting perspective.
The idea is that ops people have inherited a blue-collar culture, whereas devs have inherited an office-worker/academic culture.
Ops people conceive of their work as fundamentally operational: progress is measured in terms of actions taken, and while automation is greatly valued, there is nothing inherently "messy" with one-off fixes; the objective is to get things working now. The pathological case for this mindset is that of constantly being on the back-foot, responding to incidents with one-off fixes without recognizing that many of them share a common cause that could be addressed.
Dev people conceive of their work as fundamentally intellectual: progress is attained when the problem is correctly conceived, at which point the solution follows naturally. While writing code is greatly valued, most effort should be spent understanding the problem; the objective is to solve it correctly, once and for all. The pathological case for this mindset is that of over-engineering by an ivory-tower idealist, disconnected from the messiness of real-world praxis.
In nearly all orgs I've seen, the proportion of first-generation college grads is greater in ops teams than in dev teams. So too is the proportion of people who come from blue-collar families, or are mechanically inclined (ex: look around and see who tinkers with cars). Likewise, the proportion of people holding graduate degrees is greater in the dev crowd (ex: look around and see who's into math).
It should hopefully be clear that neither is superior to the other. The point is rather that the divide between dev and ops is partly sociological, which means it is largely based on values. Ops will tend to over-value "honest hard work" and dev will tend to over-value "clear articulate thought". There is also some latent, historical tension between these sociological groups, which has a funny way of masquerading as a technical problem. It is helpful to view arguments about "just ship it" vs "design it the right way" through this lens.
Far from being a cute "just so" story, it's been my experience that this dynamic is very important for two reasons: (1) it's harder than you might expect to foster the sense of common destiny required for real "devops"-style collaboration, and (2) each side of the dev-ops divide has a lot to gain from learning when the other side's mindset is helpful, and how to cultivate it in themselves.
I think there's definitely a large nugget of truth here, even if it's not universally true or accepted wisdom.
I would enjoy reading it expanded on even more, 'ops' people tend to be almost universally coming from non-university backgrounds, for example, in my experience.
i like this observation. its an interesting dichotomy even if you dont accept the sociological bit. the same thing occurs in machining and fabrication i think- theres a role for "dog meat" and a role for precision. true wisdom is valuing both and knowing when which is appropriate. i really like your neutral characterization of both attitudes, its very hard not to be partial to one.
Interesting idea but perhaps you are confusing attitudes to code with attitudes in general.
eg Ops tends to be very attached to getting the infrastructure correct whereas developers care about that less. Developers will write an application that handles all sorts of data edge cases but doesn't react well to network errors.
imo this is seeing this a bit too black and white. Like, ops is _paid_ to get things back on track ASAP. Whereas devs are generally given more time and have the space to think about the elegant solution. And, sometimes one-off issues are just that - one and done problems that probably won't ever arise again. In that case, especially on a lean team, is it really worth the time to elegantly solve an edge case? Especially when 6 other things are on fire? Just my two cents.
> Avoid state like the plague
This really depends on the problem domain. Application state is fine in many areas, and in the case the state is non trivial managing it in the application is a lot easier that pushing to databases.
233 comments
[ 4.7 ms ] story [ 64.3 ms ] threadGives nightmares too. Im sticking with Fargate, full, native IaaC support, feature parity with k8s. Or plain EC2s in ASG.
Also, what exactly is "full, native IaaC support" in this case, compared to e.g. EKS?
> Im sticking with Fargate, full, native IaaC support, feature parity with k8s
Fargate by itself doesn't provide anything that k8s does, ecs does. You can run k8s on fargate via EKS, for example.
> Or plain EC2s in ASG.
This is exactly what this post is saying not to do. ASG's are a subset of the features of k8s. k8s does load balancing, service discovery, deployments, health checks for example. You may not need all of those, and that's fine, but most containerised applications IME benefit from the _majority_ of k8s features.
It looks pretty literal there. And why everyone talks about service discovery? Put an LB in front of your app instances and give it DNS name. Here, sorted, no service discovery needed, service is reachable at this URI only.
When you have more than one service interacting, or when your LB restarts / fails over, what updates the DNS?
Here's where service discovery enters the picture. It needs not be excessively complex.
If you're using cloud LBs, this happens automatically. If not, you can have the instances register themselves in DNS when they turn on. You can also have an out of band system register things in DNS based on rules (or maybe you already have software that supports registering based on health checks so you can add DNS entries from a fixed list of static IPs when they become healthy/powered on)
> When you have more than one service interacting, or when your LB restarts / fails over, what updates the DNS?
You can have the LBs communicate peer-to-peer so they can update DNS when they become unavailable (if A and B can't reach C, they remove its DNS record). Some care needs to be taken to prevent things like split brain but there's established patterns for cluster formation. Something like keepalived could be used. You can also use VIPs instead of DNS
You end up with service discovery either way. Either you discover a load balancer or you discover an endpoint directly. Load balancers allow you to more granularly route traffic with the client being aware of the server topology. This is good when you're exposing services to the web/public clients. On the other hand, load balancers add a certain overhead and can become bottlenecks for high-traffic services
Yeah, don't do this. This is _exactly_ what the article is saying not to do. It's a nightmare to maintain, totally non standard.
> You can have the LBs communicate peer-to-peer so they can update DNS when they become unavailable <...>
Or you could use a system that doesn't require you to write custom P2P traffic for a solved problem. Doing this instead of using an existing system is an interesting decision.
https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGui...
You might have missed the first 5 paragraphs of the blog post
Because service discovery is pretty much the next problem that people find themselves facing once they've decided to not use kubernetes or ECS or something, and it's also something that it solves very well, without relying on DNS with all its quirks. To use my favourite saying, "it's always DNS".
> Put an LB in front of your app instances and give it DNS name. Here, sorted, no service discovery needed, service is reachable at this URI only.
Dumping a service behind a load balancer and relying on DNS is a heavy handed approach. Sure it's "simple", for some definitions of the word simple, but so is writing 3 60 lines of yaml and/or terraform to spin up an eks or ECS cluster
Yeah, exactly this actually :) I didn't want to get too verbose in the little bullet points (I already use parentheticals way too often lol), but it'd be more accurate to have written "Use kubernetes, or some other tool that takes care of container networking, lifecycle, health checks etc for you" -- I bet if you squint even platforms like GCP app engine would fit the bill here
[1]: https://github.com/DapperLib/Dapper
I really like the one about failing on missing config parameters. I used to set default params all the time (thinking I was clever).
Another big win in the last few years has been feature flagging. I use it extensively to tune performance when in production. I can control batch sizes, delays, retries, and API key rotation.
bookmarked!
I'd say `kubectl apply` fits better with a Git oriented flow than the imperative `helm install` flow. Nothing about helm guarantees or forces you to use version control. Actually the opposite. If `helm install` fails. You need to `helm delete` first. It has no way of declaratively managing updates at all.
Have a CI job call `kubectl apply -k ./deploy --prune` on every commit is way nicer than helm in my opinion.
There's a lot of valid complaints floating around YAML but what's worse than dealing with YAML is dealing with templated YAML.
kubectl brings everything to the surface. Helm is an unnecessary abstraction.
If you need this templating, a think you can do is use helm for that but not for install/deployment: render the charts with `helm template` and then apply them with `kubectl apply` and/or verify the changes with `kubectl diff`.
IIRC Argo CD can show you the differences between the running helm chart and the new version you want to apply if you haven't enabled automated syncing, but I only use Argo CD with Kustomize, so I can't say for sure right now.
edit: by reading the article again I think the spirit of that point is not 'use just helm' but 'don't apply stuff directly'. Long term, if there's more than one person working in Kubernetes, that's prone to failure and configuration drift. Doing the templating in whatever language or system that fills your needs and then using something like Argo CD or Octopus to apply them can remove most of those problems.
Yep, this is what I meant! Just "if you're using some kind of manager for your resources, which you probably are, then ONLY use that manager, don't just `apply` stuff out of band" is the idea :)
I might take a pass later at making that more clear
I don't want complexity in my manifests. I want them to be to the point.
Renaming in multiple places is very easy with find-replace.
It also allows to calculate a checksum for a config map which then triggers a restart of a deployment.
It allows to do a proper release do to working conventions.
It also makes it easy to pull out configuration options.
I think kubectl apply is the worst option.
kubectl apply -k is...well...Kustomize...
There is a big distinction between first party software and third-party software. Helm is fine for first party software because you control the chart, but awful for 3rd party software because charts often don't have level of customization you need.
Reinstalling shit with kustomize that was installed with helm is like 1/4 of my time at work.
Kustomize won't get you 100% though because something has to generate base files first.
And I'm quite happy, if not by far most happy about my helm + kustomize + argocd workflow.
None of it is complicated but works tremendously well.
Whenever I can, I use helmfile[0] for storing variables for helm since it does add a declarative layer on top of helm.
0 - https://github.com/roboll/helmfile
Additionally the dependency thing can be huge; we ran into a weird bug for installing a particular dependency on our CI system so our test there keeps failing, but being able to run it locally let’s us know that the changes did not break our actual code tests.
Most seem to use docker, here’s one for Google Cloud functions: https://cloud.google.com/functions/docs/running/overview
If you use something like “cloud run” then it’s containers anyway.
You then can build all of the testing on mocks/stubs to test the behavior. If you access a database you access in through the interface which can mimic the appropriate behavior for your code in test vs production. If you need to you can do local integration testing of the db access layer.
There's also SAM sync that syncs changes you make quickly, so if you have your own dev copy in AWS you can quickly test changes.
node -e "console.log(require('./index').handler(require('./event.json')));"
[1]: https://github.com/localstack/localstack
It's an uncomfortable concept at first but I find it's helped me build things faster and has ultimately led to less rabbit hole chasing across multiple categories of issues.
What are you doing with your K8s cluster/containers that is making your deployments slow? My cluster at work pulls down containers faster than my local, and deployments are usually swapping within ~1 minute of being committed in git/the container build finishing…
'..with unit tests notably coming in last place – i.e. “only if you have some time”.'
The "only if you have time" part is what I disagree with. If we're doing TDD then I don't see how you can avoid writing unit tests, or even deprioritize them.
It's also kind of weird to separate them like that, as Fowler discussed here: https://martinfowler.com/articles/2021-test-shapes.html
Perhaps the author doesn't recommend TDD? They don't suggest anywhere that they do. It seems to me that the industry these days sees TDD as a tool for occasional use, not the dogma once proposed.
He's certainly right that deployment validation is the best test at some level. But it completely leaves off the most important aspect. That deployment validation, and the mistakes that you inevitably find, should directly guide your test development to prevent regression.
If you do this, future component upgrades and deployments get easier. Its well worth the extra effort.
> I’ve been doing this “reliability” stuff for a little while now (~5 years), at companies ranging from about 20 developers to over 2,000
I've been doing it for 20+ years including running critical services in FAANGs.
> Use Docker > Use Kubernetes.
Hard pass. There's a reason why most FAANGs developed their own packages management and deployment system and keep using them. They are simpler, less bloated, easier to debug.
> Deploy everything all the time
On your local testbed, maybe, unless you already know a commit is broken. On production, absolutely not.
And for every other company that isn't one of those 5, rolling your own deployment system won't be simpler, leaner or easier to debug. Nor will it, in all likelihood, be as stable, feature rich, observable, etc.
FAANG-appropriate solutions aren’t always appropriate for the rest of the world.
Most FAANGs developed their computing infrastructure before Kubernetes gained popularity. After all the investment into building it and fine tuning for their software services (like ads), they probably have a lot more reasons to not migrate to something different. Not just because their systems are "simpler, less bloated, easier to debug".
That seems like an odd thing to say when Kubernetes came out of Google and they based it on Borg, their internal orchestrator.
FAANG create many things themselves as they're trend setters. They have the size to create them. Having worked on in-house solutions, it is incredibly difficult to maintain adequate documentation, training, and resources to keep it going.
More often than not, this ends up being a business decision about balancing risk with potential loss of revenue (you don't want to take your stuff down, but you also don't want to delay new features that could bring more business)
Fast growing companies trying to enter the market tend to favor "deploy more often, break more often" while companies with large, established bases tend to favor "more testing, more deliberate changes"
I would say the inverse is a better rule "you should be /able/ to deploy any time"
But it does not mean "deploy every change to 100% of the fleet instantly". Use canaries, use slow, monitored rollout. With smaller changes, it's relatively painless and routine.
Hmm, I think "config" is too vague here. I've seen numerous incidents caused by NGINX config changes that would have been far worse if we had instant 100% rollouts, but I'm not sure this is the kind of config change the author is referring to.
> Run 3 of everything
What is a "thing" in this example? Is it an instance of a service? A database? Something else?
If you're using Kubernetes and have autoscaling, what does "3 of everything" even mean? It sounds like the author is basically talking about databases..?
No, you just run three of everything. While only databases and similarly distributed data storages really need exactly three (or sometimes, just an odd number larger than two) for quorum and network partition detection, there is a good possibility that you'd have three availability zones because of your databases. And since you already have three availability zones, it's only logical to put every service in every zone at least once, so running "three of everything" becomes a rule of the thumb, easy to remember, easy to implement.
> If you're using Kubernetes and have autoscaling, what does "3 of everything" even mean? It sounds like the author is basically talking about databases..?
If you use autoscaling, you still need to start somewhere. And this somewhere is "three of everything".
I disagree. If we're talking about distributed systems, then one thing is guaranteed - network is not going to be reliable. And if we have 10's or 100's of services, this policy means that at the smallest blip the whole thing collapses like a house of cards. If the concern is "it's hard to troubleshoot", well then perhaps implement better logging ("connectivity to service X has been unreliable with X% failure rate over the last Xhrs" instead of "connection terminated", or no logging at all).
Retries are a must in most systems, but need to be planned, otherwise you DoS your own network or services.
Different topic a bit - messaging & routing ecosystem working 10 years without flaw suddenly started exhibiting slowness and randomly would just stop, needing restart of client. We debugged like crazy, java messaging system by me on one end, Tibco ems system on the other. Tibco refused to help due to server being out of support (note for us/US team). We had network guys on 13h call too, but they didn't have as much experience with WANs.
After 2 days, they discovered some internal backbone network system between US and Switzerland just failed out of blue in the worst way possible - dropped some +-20% of the packets, so things kept chugging along somehow, till they didn't (some Acks on transaction commits rarely didn't happen and then all got blocked without a hint why).
2 lessons - don't always doubt yourself and your skills when SHTF. And don't take things like servers, OS, network for granted. Don't expect some clever monitoring already in place will figure out issues for you.
With RPC, I believe the author is talking about retries at the application level. There are already enough retries in the TCP layer below it that happen with exponential backoff. Tuning that and also your HTTP library's timeout settings is possible if you happen to have a unique enough network that the defaults don't work.
But very likely, your slowness or problems will exist in the application layer – either on "your" side (your service is tied up doing something too long) or on the other side (their service is tied up doing something too long). The correct fix is to "Fix the flaky service!" as the author recommends, and this can take many forms – spin up more copies of the service, or fix any CPU or I/O resource problems.
Slapping on another layer of "just retry" on top of all the other retries at the application layer is what the author is recommending against – this is because you will end up inventing a newer, complicated model of a distributed system.
TCP retries won't help if your backend is restarting, if failover switching is happening, if your overloaded cluster has just been scaled up to add nodes, etc.
A reasonable application-level retry policy (exponential randomized delays, limited attempts) would turn these from a service disruption for the client into a mere delay, often pretty short.
Yeah, this is where the nuance begins.
You are correct that TCP is not always sufficient. Perhaps where we differ is that in my experience, it still helps for this to be a feature of the framework or infrastructure that the applications are running on (e.g. a retry budget in the service mesh, or a load balancer) rather than scattered around in the application itself. At some point it becomes a word game – you could say that the service mesh is also kind of an "application" itself, but the core principle is that the retries should be kept in a few simple, common places that are rarely tuned.
Otherwise, you will find that N developers who are tasked with figuring out something like this will scatter N different version of your exponential randomized delay policy all across your codebase. It is always possible to avoid this with enough code review discipline, but once the trend starts, it's much harder to say "No, you need to fix this the right way".
TCP, the service mesh, the company wide http client, application retries, and a database backed queueing library that everyone uses.
Imagine what happens when a service is flaky and each one of those layers retries 3-5 times.
In addition, despite fantasies, tcp in particular is an instance level concept and reserving the ability to shoot instances without draining first is a good design.
Application-level user-perceived reliability is an end to end concern, not a transport layer concern.
When you don’t (as we sometimes haven’t), you end up with people having to write scripts or perform manual actions to poorly approximate what the system should have done, often with thousands of orders (or whatever entity matters to your company) in a limbo state.
IMHO you should only add retries if it proves necessary in practice to reach your SLA, if you're building something that isn't itself triggered by an RPC, or if you're performing an operation that can never be made idempotent.
* Differentiate retryable and non-retryable errors. If the service can't return success because the DB it queries is borked, it should send a non-retryable error. Then it won't get overwhelmed by retries from upstream.
* Retry configuration should have sane defaults. Even "retry once" is too often; many services aren't overprovisioned for 100% increase in traffic. What ended up working here for us was having the retry module collect req/sec statistics, then only allowing 20% of that number in retries/second. Individual requests can be retried twice. That was small enough to not push over any services, but enough retries to compensate for garden-variety unavailability.
* Services shouldn't serve requests first-come-first-served under load. When SHTF, fairness means everyone has to suffer long delays, often much longer than rpc timeout. Instead, serve them in an unfair order - the most recent requests to come in are the most likely to have a caller still interested in them. Answer those!
* Use headless services in kubernetes. By exposing the replicaset to the client, the client can load balance itself more intelligently. Retries should go to different replicas than the failing request. Furthermore, you can perform request hedging to different replicas than the lagging request.
* Define a degraded form of your in-memory object-graph. If a feature is optional to the core business flow, it shouldn't take down your whole product. This one is a lot more involved. We needed custom monitoring for degraded responses, in-memory collection and storage of "guesses" to substitute for degraded portions of the object graph, as well as some other work I can't think of right now. This does enable an organization to compartmentalize better, having faster, less fearful deploys of newer initiatives.
DB is unreachable? Is it the DB or my host? If it’s the DB, maybe it’s not retryable, but if it’s me (and I’m load-balanced), it is probably retryable.
I use S3 as an example because it has a comparatively high random failure rate. You MUST implement timeouts and retries if you're using S3 as a backend.
If the problem is just that the 99 calls could overload downstream services on retries, the ideal solution is to add rate-limiting, though admittedly that is an imprecise science.
Passing every error up the stack causes people to be confused by 500 status errors for trivially retry-able calls. Browsers (and TCP) natively implement retries despite there being no SLAs in place.
Don't do that; Don't ever make middleware retry.
An API endpoint should be quick, and if you can't figure out how to make it quick, use batch processing metaphors (submit task/query task) and a task-runner instead. The task-runner can retry indefinitely, but a user needs to be given feedback or they will push the refresh button.
In fact, this subthread seems to be recreating many of the network protocol design ideas from the ancient days.
> I disagree
I too disagree with that too, pretty strongly.
The network is unreliable. Technology has become pretty good, to the point that we have become spoiled and take the reliability for granted. But it is at its very core an unreliable medium. Applications should expect that and survive accordingly.
Now, before packets go out to the network, or after they have made their way inside your network, they are in an environment that you have a degree of control. Errors and retries should be monitored. If they increase and remain elevated, they should be investigated. But guess what, if your services are resilient, you should have some time to investigate before things start breaking down - as they would if you treated everything as if it was _localhost_
Case in point: our app started breaking in horrendous ways once we deployed it in China and it tried to cross over the Great Firewall. Most issues would have been survivable if they just backed off and retried. The GFW doesn't usually like the first attempt to/from a new address, but it will usually allow the traffic after a while. We had to pay a company a not insignificant chunk of cash to get better connectivity (that would have been fine as an improvement, but not to make the system work at all).
Retries are at the core of everything we do. TCP has retries (if the author followed their own advice, they should switch to UDP!). Kubernetes has a whole bunch of retries before reaching error states: CrashLoopBackOff, ImagePullBackOff. Your ethernet card has retries. But it also tracks errors.
Track errors. And have retries. For as long as it makes sense to retry for your use-case.
> well then perhaps implement better logging
Not logging. Metrics! Logs are good for troubleshooting. For signaling issues, metrics are better. Errors can be a simple counter that you increment. Have Prometheus or similar scrape that and throw alerts as necessary.
> If your service can’t load the config on startup for any reason, it should just crash
Not always the case. You don't want to carry on as usual as if everything is normal, but running in a degraded state or dedicated failure mode that you can coax information out of is often quite useful.
It almost seems like the author err's on the side of KISS to a fault. Writing off or failing to recognize things like retry-logic, metrics, logging, failure modes, etc.
It strikes me that the author may have dealt with a lot of projects that are far from feature complete and may not have seen one taken one all the way to rock-solid. And by rock-solid I mean a system that auto-remediates to a certain extent and lets you ask questions of when the sky is starting to fall.
That said most of the advice is alright, especially for a relatively young project. Rome wasn't built in a day and much of the failure behavior I tend to plumb into projects gets added only after a problem is encountered.
Edit: grammar
When you don't choose zero / one as your multiplier, there's a strong risk of implementing a retry strategy that is multiplicative. E.g. given 3 layers with a try and then 3 retries at each layer you cause a potential 64x (4x4x4) amplification of any failure at the lowest level. Retries are an easy way to overload a service that would otherwise recover from a problematic situation.
Adaptive retry using a token bucket / circuit breaker approach are reasonable second alternative to zero/one.
In practice, for resilient systems, you can actually go even further than zero retries when you have shared knowledge of an outage to the downstream service (due to concurrent calls from the same source). You can choose not make the call altogether, and look at only making a small amount of calls to the service to let it recover sanely. Obviously this is only useful for calls that are optional parts of a call chain. An example implementation skip a percentage of calls based on the percentage of failing calls (e.g. if 50% of the calls are failing due to an overloaded downstream service, backing off to only make around 50% of the call volume is directionally appropriate).
Better logging is always appreciated regardless of situation ;)
Heh nah, I’ve started learning ansible and it’s so much easier for me to understand.
For some reason docker is a little to complex for my purposes.
Ansible just runs ssh commands , no extra layer of abstraction
I found that the fewest abstractions the better, because at some point you’ll have to debug both the abstraction and the bottom layer.
With Ansible, you first start a machine, then build a configuration on it.
With Docker, you first build the configuration, then start the machine (not a real VM, but an isolated enough slice).
The upside of a Docker image is that you build it once, and then start instantly, no need to wait for Ansible to run through steps.
The downside, but also an upside, of Docker is that you can't make persistent changes inside it while it's running, and usually you avoid such changes. You always can restart it from a known state.
Of course Ansible can handle more aspects than just application software configuration, basically any aspect at all. This is its upside, but also a downside: you can make an unintended system change along with some innocent-looking operation by mistake, and nothing will stop you, there's no separation of concerns.
So it’s possible to pair Ansible with docker?
Like only the app on docker but all the runtime, Apache, dependencies configured using Ansible ?
Typically, if using bare nodes, you’ll see OS level concerns (disk, networking, init systems, etc) managed via a config management system, while app level concerns (lib dependencies, env variables, port setups, etc) managed via a container.
You can of course run a non-dockerized, locally configured Apache, and make it talk to backend code that lives in containers which just expose TCP ports or Unix sockets.
It allows you to package all the dependency tree horrors of a Node app, or of a large Django app, into a container once, at build time, and just put them on a host where you run them. You're guaranteed to run exactly the same code on your prod box(es) which you've tested on your dev and CI boxes, with all dependencies, libraries, etc guaranteed the same, never reinstalled.
Eventually you may discover that it's convenient to put the Apache into a container, too. Suddenly it stops depending on the distro your host is running.
You may also not need to run Docker proper on your prod machine(s); in simpler cases systemd will start / restart your containers just as well. During development though, docker-compose is invaluable, it allows you to locally run nearly your prod configuration of services, at a tiny scale.
We use them because then what’s running in prodcution is 100% the same as is running locally when testing the application.
You can hardly get there using Ansible.
With Linux competency and some thought put into design, you can find a balance between reproducibility and ease of use with Ansible. Easier for me and those who agree with me to configure and debug a Linux host vs. debugging and configuring Docker to be as flexible as running it straight on the OS.
It’s not an impossible task to harmonize configurations with Ansible. It definitely takes more thought than Docker and that’s where the competencies of the team count.
Only if you’re running the same kernel.
No SSH is better than any SSH :) When your infra matures enough, you'll bake machine images (AMIs in AWS) and start machines from this image. And containers already operate this way, thats why they popular.
but, where I hard diverge is the lack of metrics. Everything should be generating metrics. Logs are great, but they are crap for giving you near realtime trends.
Everytime your container gets a connection: increment a counter. Every message processed, counter. every message failed, counter. Every KB of data sent out, counter. Every service call, counter by service.
metric all the things, in a sensible, mostly automated way, with a decent schema.
Then you can combine all those metrics into a dashboard that shows your system performing against business goals.
That’s because the real way to build software that’s reliable has very little to do with a list of features/tools it has to do with designing for the sad paths at the same time as the happy paths.
For each failure modality the app should have a conscious decision about what the app will do when the bad thing happens. This can be everything from “this will cause an outage” to “this will route to an alternate system”. What’s important is figuring out up front what those decisions mean for your data, your operations, your business and your users.
Without that all the structured logs and version controlled configs in the world are just window dressing.
> Use Kubernetes.
Dear God no lol
If you're running a startup that's starting to scale dramatically, I think maybe this is valid advice.
But nowadays you can keep things dead simple and still be able to accomplish most business use-cases.
Just having a Kube available I can run a quick kron on is valuable.
I agree with this. But I really see this from a business perspective. I think k8s works better at companies with large, complex engineering teams and infra. A lot of companies simply aren't ready, nor will they benefit from, introducing k8s into their stack.
I've found it's quite neat for e.g. data scientists to be able to toss a quick yaml into git and have their thing running to their specifications in non-prod. Things like flux and Kyverno make sure they are boxed in tightly enough that they can't cause a lot of damage, and if that thing works out, someone adds CI in front of the yaml and some templating to the yaml itself and off we go to prod, and the result is still quite comprehensible, so people are enabled at least to ask the good questions to the right people. That's what I see as one of the killer applications of Kubernetes, to build a lingua franca interface for all things ops that doesn't change (much) for different service levels and even totally different needs, like frontend devs and data scientists.
It's probably not a great choice for a three-person startup trying to gain traction by moving fast, but IMO the threshold where k8s starts making sense is a lot lower than many here appear to think.
> Assuming you have more than one service and more than one instance, you either need or will need stuff like service discovery, autoscaling, and deployment versioning, and you do not want to manage all that by yourself.
To paraphrase, "If you have anything but a tiny amount of complexity to your architecture, you must add in a high degree of complexity". It sounds insane, right? Like, we're skipping some middle steps.
Yes, Kubernetes solves those problems, BUT ALSO the care and feeding of Kubernetes clusters themselves is a significant time investment. Manual scaling sucks, but if you have a good automation game, it's not that bad. Often what people think is a need to scale extremely rapidly is really an organizational problem they are trying to automate around (example: An ECommerce site sends out coupons to 10 million customers without telling the devops teams).
AWS and GCP will do this for you for a small markup
The article is basically advocating, "if you have more than one service, you need K8S", and sticking your app on Kubernetes in the cloud (managed or not) is a huge cost upgrade from a single or handful of servers.
1. The higher your service count gets, the more you should probably be on some container orchestrator. There is more than one. 2. The number of services you have to manage to justify moving to a container orchestrator is most certainly not just > 1. That is crazy. I don't know the right number, as it probably varies according to how mature your organization is and how well the parts work together. But it is definitely higher than > 1!
My last point is: If all teams worked together smoothly, or one team had infinite capacity, or if automation was perfect, etc. etc. etc. we wouldn't need containers, let alone container orchestration. These are solutions to people problems, not technology problems.
When things are still small and simple, an Ansible Playbook and a Docker container can take a company pretty far while still being able to move to k8s pretty easily.
Easy scalability is a nice side effect of that.
Ah yes, the time honoured testing in prod strategy
I have to partly disagree with that one. I find tools like Helm to obscure things that should be readily visible. My favoured method is to keep manifests in full (which you can source from `helm template`!) as pure yaml files and version that. If possible freeze versions and go through regular patch cycles to review updates. That you apply them through `kubectl apply` or through Argo is irrelevant. I treat the repo as the state and the running cluster as stateless. If it's borked, just redeploy. I don't see it as useful to care too much about the in-cluster resource life-cycle. But I completely agree that resources need to be version-controlled.
In my opinion, for sure. There's a balance between "too coupled" and "too de-coupled" that should be stricken rather than too far on either side. It's good to say that this is also contextual; some projects may be fine with either more or less coupling than others, and that's OK.
It can be used for good and it can be used for bad. Less is more with Helm otherwise it will create bad.
This (and CNI) are the rough bits of Kubernetes.
You will get a manifest, but it will usually be missing certain parts of it.
I completely agree with the philosophy of just redeploying the cluster if it's borked. I'm using NixOS myself for the task, and was trying to obtain full manifests though "helm template" originally -- so I'd love to know, if I was just missing something.
When we generate the templates we use -n to override the namespace.
The command looks like this helm template CHART_NAME CHART_PATH -f CHART_INPUT_PATH/config.yaml --output-dir CHART_OUTPUT_PATH/manifests -n NAMESPACE --include-crds --render-subchart-notes --kube-version KUBE_VERSION
The problem for me is, that setting the namespace in that way with "helm template" does not seem to add it to any manifests not specifically specifying the namespace to .Release.Namespace.
The rancher 2.6.8 chart does not set this for all manifests, but does with some, so when I set namespace though template, and deployed it all through the manifests folder, I got some objects in default namespace (because they had none specified), and some in the intended namespace; resulting in an installation that did not work.
As another reply to my comment suggested, this can of course be handled with post-processing of the result of "helm template", though, at the time, I was not certain the problem was limited to this namespace issue, so I didn't feel lucky enough to go down that route. :-)
Thank you for the suggestion though. It's comforting to hear that it may actually be a viable approach.
Goodness, is this ever true. Particularly with Prometheus Operator and all the monitoring bits that go around that. Dealing with this infrastructure breaks a number of the points in the article, like "Deploy everything all the time." and "No in-code fallbacks for configs."
A previous team built this monitoring infrastructure, so when I had to go back in and re-deploy, a bunch of the Helm charts were broken (YAML errors and the like). It hadn't been re-deployed in likely 4-6 months.
Then a lot of the components don't rely on default configs, but the default configs are there nevertheless. So another team was troubleshooting an issue, and they reached the conclusion that the config for AlertManager was empty, but it's not. The config for AlertManager is in a different directory from the default config. Then an issue with Prom2Teams came up, and Prom2Teams gives an error for its default configuration file that it doesn't have permission to load that file--Prom2Teams runs as a user, the file is readable only by root. So another team came to the conclusion that Prom2Teams can't read its configuration file. But that's not the file it's actually using to configure the service; it's just a default.
So two red herrings as a result of default config files that aren't being used at all, compounded by Helm obscuring components that should be visible, and ultimately stemming from the inherent complexity of the system.
But in reality, there are issues that make this worse which are unrelated to Helm, Kubernetes, and the Prometheus stack.
For my money it is worth spending the time to grok the slightly funky overlay semantics, at least for teams with infra focus / dedicated SRE.
However, I’m not certain I recommend it for small teams doing full-stack DevOps, I.e. engineers deploying their own code that aren’t k8s experts; if you only work with the Kustomize layer infrequently it can be a bit annoying/unintuitive.
Note you can still use a two-step GitOps process where the Kustomize scripts write to your raw config repo; I think this is a good middle option that keeps the infra legible, while allowing developers to get the ergonomics of a bit of dynamic logic in their deploy pipelines. (Eg parameters for each environment).
I do something like this but I normally find that helm charts are not parametrized as I want and have to manually modify the output manifests. When updating from helm it can be challenging for other team members to understand what bits we want to take from the new helm template output and which ones we don’t. How do you deal with this?
Sometimes I update the helm chart to fit our use case, but it’s still hard if that is not merged upstream (because that means maintaining our own version of the helm chart)
Isn't that the problem that kustomize is designed to solve? Flux even has a first-class declaration for "take this thing, then kustomize it". The `helm template` into git pattern could be extended to "helm template, write kustomize files, then version control both" since it would capture the end state as well as the diffs that were applied on top of the vanilla chart
I think the "maintaining our own version of the helm chart" is only painful if the helm chart itself is moving around a lot, versus they're using helm releases to carry changes to "--set image.tag=$(( tag + 1 ))" type thing
Excellent advice for improving productivity and testability. I wish more teams would be sure to follow this by not adopting technologies that are impossible to test locally.
Acronyms don't excuse poor grammar.
Whether you use "a" or "an" depends on whether the next word starts with a vowel sound, not with a vowel letter; since the letter "S" is pronounced "ess" in an acronym, you would write "an SRE," but "a Site Reliability Engineer."
Also, I would have expected to see "Use only strongly typed languages" as one of the bullet items. I get the sense that the author doesn't actually have a ton of enterprise experience.
I disagree with this. Logs should only be used by human operators; if you want something machine-readable, you should expose metrics via some other channel or record errors in a dedicated service.
Sure, some tools will let you read structured logs quite cleanly, but it will never be as nicely formatted as something designed, first and foremost, for humans. Of course, you may need to (say) tack on a request ID to avoid logs from different threads/goroutines/etc from getting interleaved, but that doesn't require everything to be in JSON.
Ex:
[2022-01... INFO] trace=abcd123 <msg>
There is a structure which can be parsed.
I suspect that you will find a lot of resistance to the idea. At least I have. Machines should be consuming metrics, not logs, unless there aren't any alternatives. That statement never seems to be controversial, but people... just don't do it, for whatever reason.
Also, the author seems to want to do a poor man's version of distributed tracing.
Let's say you have 10 services in serial, and you want to get precise information on how that request was performed - because your customer reported it performed super bad. To do that you might need to get logs from all those 10 services and align them. If they all are human readable and you can filter by a common request-id, that's easy to do. And you can even have tooling on top of it that automatically determines what might have happened by looking at other fields of the log entries.
Metrics are mostly just pushed to a metrics system, and that moment you don't care about them or the relation to an end-to-end request anymore. So while all services might have emitted a metric for the request, looking at the overall metrics won't tell you if outliers in the services refer to the same end-to-end request or different ones.
The idea is that ops people have inherited a blue-collar culture, whereas devs have inherited an office-worker/academic culture.
Ops people conceive of their work as fundamentally operational: progress is measured in terms of actions taken, and while automation is greatly valued, there is nothing inherently "messy" with one-off fixes; the objective is to get things working now. The pathological case for this mindset is that of constantly being on the back-foot, responding to incidents with one-off fixes without recognizing that many of them share a common cause that could be addressed.
Dev people conceive of their work as fundamentally intellectual: progress is attained when the problem is correctly conceived, at which point the solution follows naturally. While writing code is greatly valued, most effort should be spent understanding the problem; the objective is to solve it correctly, once and for all. The pathological case for this mindset is that of over-engineering by an ivory-tower idealist, disconnected from the messiness of real-world praxis.
In nearly all orgs I've seen, the proportion of first-generation college grads is greater in ops teams than in dev teams. So too is the proportion of people who come from blue-collar families, or are mechanically inclined (ex: look around and see who tinkers with cars). Likewise, the proportion of people holding graduate degrees is greater in the dev crowd (ex: look around and see who's into math).
It should hopefully be clear that neither is superior to the other. The point is rather that the divide between dev and ops is partly sociological, which means it is largely based on values. Ops will tend to over-value "honest hard work" and dev will tend to over-value "clear articulate thought". There is also some latent, historical tension between these sociological groups, which has a funny way of masquerading as a technical problem. It is helpful to view arguments about "just ship it" vs "design it the right way" through this lens.
Far from being a cute "just so" story, it's been my experience that this dynamic is very important for two reasons: (1) it's harder than you might expect to foster the sense of common destiny required for real "devops"-style collaboration, and (2) each side of the dev-ops divide has a lot to gain from learning when the other side's mindset is helpful, and how to cultivate it in themselves.
I would enjoy reading it expanded on even more, 'ops' people tend to be almost universally coming from non-university backgrounds, for example, in my experience.
eg Ops tends to be very attached to getting the infrastructure correct whereas developers care about that less. Developers will write an application that handles all sorts of data edge cases but doesn't react well to network errors.
IMO, the curious thing is that in addition to focusing on slightly different problems, there exists a cultural difference with respect to work values.