I like to do short lived credentials using Vault (e.g. vault can create say db access credentials dynamically), but for things like API keys where I can't do that..? Is the Vault KV store the source of truth?
At least in cloud providers, they have secret vaults accessible to their customers. The individual secrets are stored in source code but they're encrypted. We've used SOPS as a valuable way to manage these secrets. You can certainly stand up your own secretserver or equiv but may not have all the same.integratuon bells and whistles.
i think notion uses nix in production. I can't remember what their ci/cd pipeline and version control system is outside of that, or if it was even mentioned in that one comment i saw about it
Ok for when deployment is completely and always automated, but for that _one special case_ maybe keep a copy of the old that you can revert to until you're _really_ sure of no unwanted effects. In the meantime, find out how & why that local change got made and what can be done to automate it next time
Wow and I thought functions should only perform a single task. I need to keep up with the times! Apparently you need an entire deployable app and API to do anything these days. I guess it makes sense. How else could we justify so many software engineers!?
Understaffed because with the microservices fashion you can get 100 engineers to be satisfied and think they are doing meaningful work while producing the output of 10 engineers. The industry believes the fashion is the only way to work by now, and shortage results.
(Of course, there are several other ways to get 100 engineers to have the output of 10; but I think microservices makes the engineers a lot less frustrated when it happens compared to more traditional alternatives)
What I meant is: If people are working at 1/10th of their potential they would tend to get frustrated.
With microservices, you make 10x as much work in such a way that the work seems meaningful and is perhaps even fullfilling. It takes some effort to see that the complexity you are solving day to day is accidental complexity, not essential complexity.
If you're losing 10x performance then you're clearly using microservices poorly. Our team of like 6 people produces a crap ton of output. We've got something like 18 well written services that need basically no ongoing support. If for some reason we're doing too much, those services can be shipped to other teams to maintain or enhance. The one and only amazingly valuable function of microservices is the ability to easily move tasks to other teams without a significant amount of hassle. The monolith doesn't work well in multi-team setups, so you're either doing less or have huge teams with their own sets of problems. Companies do it, but it's so much worse IMHO.
To your original point though, correlation doesn't equal causation, and just because you could abuse a technique to placate your wrong-sized org using this technique doesn't mean that it's the purpose of said technique.
"Using microservices poorly" -- this comes up a lot when I discuss microservices with people who believe in them. I think it comes close to the True Scotsman fallacy -- if microservices ended up causing a mess, the explanation is always that one wasn't using microservices "properly".
I could say the same of a monolith -- if it is a big ball of mud then it is not a Proper Monolith ...
Granted, my only experience with microservices was a ball of mud mess microservices. I am sure we will both agree they were NOT doing microservices correctly -- BUT -- they did not know that going in did they?
If microservices is defined by definition to only be about the successsful applications of the technique, then it is not very helpful for those choosing architecture for a new system..
Re shifting services to other teams:
We have a medium-sized-service architecture, not monolith nor microservice. One or two services per team mostly.
By far, the hardest part of onboarding and doing development is learning the business domain and have people understand WHAT to do, why to do it, etc. -- the HOW is usually simple.
Code usually covers only the HOW part. Just shipping parts of what we do to another team and expect them to be on top of things quickly doesn't seem realistic regardless of the size of services..
Teams are homes for business and domain knowledge first. A home for the code is just a byproduct.
I think GP was (half) joking that this shortage is caused by everybody cargo culting overly complicated dev practices. I'd say there's some truth in that.
Due to Parkinson’s law (“work expands so as to fill the time available for its completion”) it’s likely that all teams will always appear to be slightly understaffed.
I think this advice really depends on your scaling needs. If you need to scale your services up, it’s a lot easier to do that if each service only does one thing.
It also depends on how much functionality you consider to be “one thing”.
I never understood when people talk about microservices and scaling (for traffic).
I thought microservices is a solution to scale development teams, not for traffic.
If you have a horizontally scalable monolith, it can scale pretty much as far as you want. If you split services along functional boundaries (i.e., vertical) then a split from 1 to 2 services will in the extreme best case scenario give you 2x scaleup; further splits give you less. So: If load is the issue, work on horizontal scaling, not microservices.
There are many non-organizational reasons to use microservices. Here are some examples:
As you add more functionality to a system, it gains different workloads, which have different needs (e.g. different storage solutions) and need to scale at different rates (e.g. one part of the system is very memory hungry and another is very compute heavy).
As your system gets larger, it's usually more important that its various components are well isolated, both on software and hardware axes, e.g. container isolation vs isolating across different physical zones/regions. Sometimes that isolation happens for security/compliance reasons, e.g. PCI.
As your business grows, you will need to build functional solutions that require different toolchains and platforms. You will acquire other businesses that have built on different platforms. You will not require all of your acquisitions to rewrite all of their software into your monolith.
With extra work, you can solve for many of these problems within a monolithic architecture, but those solutions will typically introduce similar complexities as microservices (e.g. talking to many datastores, composing your monolith of many coordinating processes, configuring independent nodes of your monolith differently, provisioning independent nodes appropriately, etc).
My experience in software is that we draw false dichotomies between approaches and the reality is that they will tend to converge when you put in the effort to consider and address the shortcomings of each approach.
I think I should have avoided the word "monolith". I just meant non-microservice architectures.
Microservices have their own challenges apart from network communication etc between medium sized services that yiu refer to.
For instance lack of easily available consistency, sometimes needing to basically implement multi-service transactions (see SAGA pattern) and so on... if services are a bit larger, such problems occur much more seldom.
Storage in general, and transactional boundaries in particular, are one of the biggest banes of distributed computing. It's still more art than science to determine where and how you slice an entity graph across datastores and services. Sadly, these decisions also bleed through API.
A giant, completely stateless monolith that takes no memory will scale as far as you want, but said monolith doesn't exist. Every new route you handle has memory costs, and if you are caching any computations, you also have to think about that. You also are going to have interesting situations if every server could end up keeping connections to 15 different kinds of databases.
At some point, you have servers, that say, pre compute complex internationalization logic, and therefore are almost databases, and you want to keep them separate. Other times splitting requests that have very different memory vs cpu requirements into different boxes helps. And this is just for performance/costs.
The scariest issues are all operational though: Suddenly my giant fleet is misbehaving, and a rollback means 10K servers get hit, and I am not exactly sure what is going on. Having the functionality separated a little will do a lot of good.
Now, from there to needing microservices is a pretty big gap, but the lone uberservice can get very scary under the wrong circumstances, and the probability approaches 1 as the codebase gets really big.
Right, I agree with this. I think "monolith" in my example was a standin for "non-microservice".
I am all for dividing services after technical requirements; and certainly any non-O(1) memory requirements for instance one may want to look at a dedicated service.
When we get into the things you say, the definition of "service" gets more important too. Is it separate DB, or only separate k8s pod or VM, or what... If the only constraint is to make computation efficient and robust, the solution space is bigger if one does not add the "should fullfill the microservices ideology" as another constraint.
With microservices on k8s I think it is pretty common for each VM to have many open connections to 15+ databases though, so not sure if I got that particular point.
I ~hate~ dislike this advice.
If you can't deploy on a Friday, you need to fix your deployment strategy. By removing Friday from when you can deploy, you're wasting 1/5 of your available days.
Note: deploy != Release[1]. Use flags, canaries etc.
This one made me laugh. I've been places that only allow deployments on Fridays because it gives the whole weekend to fix things if they break.
It's a good interview question as a candidate. If you ask the interviewer when they deploy and they say only Friday (or worse only once a month) then perhaps look elsewhere for your own sanity because it's a sign of serious malfunction either organizationally, technically, or both.
Depending on your role, that is. If your desired position is straight dev with minimal to no ops work as possible, then yeah, red flag. However, if you're an SRE/DevOps-type person, setting up a continuous deployment system so they can deploy more often than that is a perfect landing task to dig your teeth into. Different strokes for different folks.
If there's a very strong "only during standard work hours" usage pattern with SLA penalties for downtime, adapting deployment patterns to that reality can maybe be sensible.
Don't discount a job because of one deployment per month---it really depends upon the service. I joke that a busy year for me involved four deployments to production, but "production" for me wasn't a website, or even a web-based service, but a service involved in the call path for phone calls. Our customer is [1] the Oligarchic Cell Phone company and the SLAs are pretty severe.
I do have to ask---where do you people work where you have multiple deployments per week (or even per day)? To me, that sounds insane!
[1] Still is, even though I left a few months ago, and not because of the lack of deployments, but the shoving of Enterprise Agile [2] on the company by new management.
Any SaaS company that that has trusted automated testing on the backend, big enough to have a platform team so that there's a continuous deployment (CD) system should be able to do that. It's less about the actual cadence and more about the fact that, given a bug or request for a new feature, developers are empowered to go and do a release.
I am on a tools team, so the "customers" for our team are the company's developers. For changes that might cause an extended outage if things go sideways, we generally prefer to do those after-hours or on weekends so that we don't have all the dev teams sitting idle during work hours if something goes wrong on our end.
The upshot is that this is fairly rare and we do not have an on-call rotation. If most anything breaks over the weekend, nobody is going to notice or care until Monday morning rolls around.
I'm a huge advocate for CI/CD pipelines and my team owns a lot of them. We're confident enough to deploy anytime but we choose to limit deploys to our team's business hours and not on Fridays. Why? Because we think the return going from deploying 4 days/week to 5 days/week is outweighed by the stress and morale hit of ruined weekend plans if something weird happens. There's probably situations where that extra speed makes a difference but for us deploying to all regions safely can take a full day anyways so it's pretty normal to have multiple changes flowing at the same time.
I understand that, but would counter with that it sounds like deploy == release, and that if that weren't the case, you could deploy more often.
However, I will admit it is a trade-off; some engineering time does have to be spent to get there, and perhaps that engineering time is better used elsewhere right now.
What is the difference between deploying and releasing? Everywhere I’ve worked, those are the same thing. We deploy probably 10 to 30 times a workday. As for Friday deployments, I tend to avoid deploying big changes on Friday but medium and small are probably fine Friday unless it’s the very end of the day.
> Deploy is the act of moving software into an environment and running it.
> Release is the process of making a feature visible to a user or subset of users.
In our case, we deploy often (10s of times per day), and make use of feature flags for releasing. Keeping an eye on data in Honeycomb before and after a deployment and before and after a flag is changed is part of the engineers job.
If something doesn't look like it's working as intended, ideally it is disabled with a flag, rather than needing to figure out how a rollback can be done (often it can't), or rushing a fix and roll forward.
This is a pretty small distinction in my experience, and I think you'd find that most people would go with "if it goes into prod, it counts as a release".
Regardless of how confident you are on your feature-flagging functionality, everyone's going to be unimpressed if suddenly your "minor" deploy has caused a bunch of things to go haywire at 4:30pm on a Friday and now people have to stay back and fix things.
If you put new code into production, that is a deployment and an engineering release.
If some of that new code is a flag that controls what customers see, then from a product perspective it’s not a “product release”. Is that what you mean?
The point is, even feature-flagged code can break things unexpectedly.
Hating it seems a little strong. I'm sure that any team far along enough on the quality spectrum can just read this and say "we've moved beyond this worry". The post is titled "general guidance", not "absolute truths". Adjust expectations accordingly.
The point of not deploying on friday is to reduce the risk of getting paged over the weekend. It's a quality of life move for the oncall team. No deployment strategy will change the fact that deployments are the leading cause of outages.
If you can't afford to give up 1/5 of your available deployment days you have a problem somewhere in your CI/CD system.
Over a long enough time, with enough deployments, unlikely things are guaranteed to happen! In the long run, setting a given feature loose on Monday instead of Friday is unlikely to move the needle in terms of business results, and eliminates a category of risk. I'm not superstitious about much, but No Friday Deployments is one of my rules
I agree. We have some advocates that really want to say we deploy on Friday, but even they don’t seem super pushy about a Friday deployment when they have weekend plans, and the org as a whole isn’t keen either. It’s high risk and low reward. I’ve also burned myself on this more than once.
CI/CD, flags, canaries don't catch everything, and can still cause outages to others. We try and do pretty heavy CI/CD where I work, but not everyone does (we, like everyone, has old systems). It's actually quite easy for us to have the well behaved systems honor release hours or not depending how their release history has gone, or coverage,etc... but they're well behaved, so they usually have great tests, and they're not usually panicked about rolling out after hours, they have their sh*t together.
The reason we have core hours release only without director approval (aka director approval required outside core hours) is so you don't piss off another team by paging them after hours, and so you aren't trying to shove out a thing on a system that doesn't have good coverage or by turning off the safeties. In a large company I've noticed many engineers assume urgency even where there isn't. As an approver myself, most of the time someone wants to rush is because they've not even had the convo with their manager on if it's worth the risk, they are assuming urgency because that's when the sprint ends or what some TPM added to a jira ticket 4 months ago.
I admit that sounds risky itself (the engineers not having the right risk training) but this is why we have a policy and tooling... most of the times I've dug in they're just very new and worried about perception as a new employee, so my job is to shepherd them through having that convo with their managers which inevitably has the managers saying "yes it can totally wait till monday", and the change is inevitibly a bit more hot than it should be due to accidental deadline pressure.
I get that people really want to flex that they can deploy on Friday afternoon and NOTHING CAN GO WRONG, but it’s still foolish and flaunts Murphy’s Law. It can wait.
There isn't a bottleneck in the amount of commits you can ship per day. You just get more changes to roll out on Monday.
I do disagree with the absolute statement of not doing it, but I definitely do a risk analysis whenever I ship a change on Friday and avoid anything risky and just push it off until Monday morning.
Not all changes can be put behind flags and canaries don't really fix the issue (unless you are okay in blocking important fixes from being rolled out due to your bad change killing the canary)
Seems reasonable, but if you work for a large company, you can't guarantee that a major release (which is a production change) won't cause any unexpected harm. I have worked with quite bit organizations, and deployed on Friday and wasted my entire Friday-night and Saturday morning, rolling back the +130 components that an app had.
If you are a small company, or you do not do extra weekend shifts, I understand your point. Elsewhere, you just want to live an adventure every Friday.
Don't forget "pets vs cattle", thinking of servers as ephemeral and working towards quickly being able to scale up/down based on demand. So often I see people "lift and shift" from a dedicated server model into the cloud and never convert their pets into cattle. This reduces flexibility later, not to mention makes it harder to respond to patching needs, scaling, and moving to optimize latency or costs.
It should still not be administered as a pet. In fact, it's even more important when you have a single instance of some importance to make it entirely rebuildable and replaceable.
If you manage to run your entire service on a single (heavy weight) server, uptimes will usually be excellent. Reliability for a single server is high. It gets lower for each server you add, unless you're adding redundancy. But that adds complexity, and that hurts reliability too.
My point: a single dedicated server may be a more reliable, simpler and much cheaper solution than the cloud provides.
But you are introducing the complexities of managing a fleet. This may or may not be an improvement in overall complexity, cost, or reliability. This decision should be made per-project, and neither option is a good fit for all projects.
These things are inherent complexities in any situation with backup/DR. You've already got to figure this out. When one machine is degraded and another is trying to recover from db logs, that's a cluster.
> This decision should be made per-project
This is one of the least project-dependent things ever, akin to source control. If you don't have scripted installs you don't know what you're running and you can't reliably test it or upgrade it.
You don't need to run a cluster, or have auto-scaling groups, just because you can.
> But you are introducing the complexities of managing a fleet.
Depends on who manages the fleet. An EC2 instance provisioned by Elastic Beanstalk is quite easy, the problem is EBS can be an utter PITA of edge cases and weird failure modes.
They can fail much less often than you’d think. StackOverflow runs on 10-ish dedicated servers. There’s plenty of other services out there that don’t autoscale and are much more reliable than anyone needs them to be.
I've replaced pets with ASGs that do nothing but sit there unless the EC2 instance fails. Its amazing because you no longer have to baby the pet, it just recovers in a nice clean state, same massive instance..
A cattle server needs no special maintenance. Yes, if you only have one server then by definition it'll require special maintenance, but that maintenance should be as generic as practicable so that if you ever need to set up a second one then you don't need to do anything special. That's the distinction between pets and cattle, not the tooling: you can configure cattle by hand if you have a repeatable and non-special cased setup script to follow, and you can use Ansible or Puppet or whatever to set up the unique environment your pet server needs. The important thing is that the setup process, maintenance, etc is standardized and documented. Automated is a bonus that should be relatively easy if you're doing it right
There might be an earlier source, but I first ran across the pets versus cattle nomenclature in Tom Limoncelli's _Handbook of System and Network Administration_ - which is a really, really good read for anyone going deep into ops space (like a cloud engineer should be).
As an ex-FAANG engineer, this is FAANG advice. Pets are just fine. Most companies arent FAANG and don't need that class of solution.
An R620 plugged into a switch in a colo, a bash script via cron, or a cloudflare worker are just fine for a lot of use cases. The only time it stops being fine is when you can't afford to do your pet -> cattle migration as you scale up. But I don't think this is a common death for companies.
If you call "cattle" a cloudflare worker or lambda function - fine. But when we are talking about multiple redundant servers with load balancing across them, you really need to justify the cost of that vs the value you squeeze out. Sometimes you're squeezing the juice out of the rind.
Treating servers as disposable is about more than just scale. It helps avoid creating snowflake servers, makes DR more predictable, and makes creating dev environments much easier.
Also, just knowing what you're running at the time is great. Changing things manually means you'll forget some of them when you need to do a bigger change. Or when you're seeing up server no.2. We've got great tooling these days, so whether you're going with nixos, docker, puppet, shell scripts for installation, or something else, the cattle approach gives you benefits.
I'd say that until you're FANG-level or at least hockey-sticking, it's actually totally okay for some of your things to be pets and that cattleizing literally everything is actually a premature optimization. API boxes which you have 100 of? Absolutely cattleize. That one bespoke server that runs that one service that is totally weird? Just let it be weird for a while. Work on more important things.
I think there’s degrees of cattleisation, I have in the past deployed crappy vendor software which needs a unique license key for each running instance. We discussed some sort of license service which would allow keys to be checked out when a container is started, but in the end settled on an auto scaling group with 1 instance for each running container with the key in an environment variable.
That got us the comfort of knowing if the host machine died in the night the task would be rescheduled without a bunch of extra engineering to be able to scale arbitrarily when we only needed a couple of instances running.
Is this on AWS where instances get reaped and can't be transparently migrated to a different physical machine like that can on GCP, or is my info out of date? (Just haven't been on AWS for my latest projects.) Because yeah, that would also keep me up at night if random instances will randomly get deleted. My proposal that pets are actually totally okay relies on that not being the case.
My real, pointed question is: what was the opportunity cost of cattelizing that vs any of the other work you could have done in that time? Because given a fresh instance but I'm allowed to use the rest of my infra, setting that machine back up in the morning should just be to instantiating the right docker container on the host and copying the license file in. Depending on what the vendor software does, I probably still wouldn't ASG that until it had actually happened more often than once a month - there are so many more things to stress about! Even if this were the splunk license checker, losing some log files would be quite disappointing but not the absolute end of the world - if you're losing api calls to stripe or the equivalent that's a different matter!
$PREVIOUS_COMPANY used to have pages and pages and pages of HOWTOs in Confluence re what to click on in the AWS console for the stuff that had to be done manually.
Doubled up as amazing training / onboarding material for folks that maybe hadn't worked intimately with AWS before (including myself).
There is a lot to unpack in this statement. First, this sentiment is already covered by the article under:
> Git should be your only source of truth. Discard any local files or changes, what's not pushed into the repository, does not exist.
Second, don't fool yourself into thinking, just because you wrote a script or configured some provisioning service, that means you've pulled yourself out of the valley of nondeterministic unreproducible production environments. Pets can and do live in git. A majority of the systems I've worked on in both FAANG and non-FAANG in my career couldn't be reproduced from the source repo in the case of a catastrophic disaster (I.e. us-east-1 disappears) or a year of being stable enough to not deploy even though their "source of truth" lives in git. The world moves out from under your repo and many of the steps you take for granted when writing your automation aren't understood by the next engineer running the automation. Then one day you have to go upgrade this one repo and realize it's IaC is no longer compatible with the infra in the real world and you burn a day trying to figure out what's wrong with a 1000LoC YAML file.
> Then one day you have to go upgrade this one repo and realize it's IaC is no longer compatible with the infra in the real world
Thats yet another reason to avoid pets. When I say pets I mean relatively big pieces of SW, usually 3rd party, that need non-trivial configuration. Like Jenkins or Windows or some security gizmos.
> Then one day you have to go upgrade this one repo and realize it's IaC is no longer compatible with the infra in the real world and you burn a day trying to figure out what's wrong with a 1000LoC YAML file.
While I'm all too familiar with this problem from the NodeJS ecosystem, it's rare to encounter it in Terraform in my experience. The only exception that needs manual intervention in a "cold boot" scenario is AWS Cloudfront using certificates provisioned from AWS ACM - the provider will assume that certificates are already valid when creating the CF distribution and error out if ACM hasn't issued the certificate yet.
> But when we are talking about multiple redundant servers with load balancing across them, you really need to justify the cost of that vs the value you squeeze out.
Sometimes you can justify using a thing for the wrong reasons.
I recently attached 1x NLB to each of our Swarm clusters to migrate to automatically managed certificates directly attached to the NLB (Digital Ocean).
$COMPANY has maybe ~3 users accessing each production application at a time. So the NLB itself is utterly pointless.
But Engineering no longer have to fix the certificates each quarter after users see an insecure browser warning and email us about it.
Unfortunately most base OSes are designed around being pets - the FreeBSD manual is all about a world where you get a single machine, give it a cute name, install a web server, install a C compiler and the OS source, build a custom kernel for some reason…
This may make sense if you can only afford a single machine you want to hyperoptimize but it’s not a good workflow for anyone. Not even BSD kernel developers.
All that is scriptable. I ran thousands of FreeBSD machines. When you get a new one, you run the script that sets up users, pulls the source, builds the custom kernel, installs the packages, etc. A little bit of automation goes a long way. (make world is a reasonable, although not particularly comprehensive, burn-in test too, but if you were in a hurry, you could build in one place and distribute)
You shouldn’t have a compiler on your webserver at all, and building a kernel should happen on a different machine than installing it. (And why isn’t the official kernel good enough?)
Beyond the security surface that’s just asking to brick the machine; if you never make any mistakes it’s because you never tried anything very interesting with it in the first place.
Furthermore, I claim that “you can write a script to edit /etc” is not how an OS should support configuration. Instead of being possible to keep a copy of the config in a script somewhere else, it should be impossible to do anything but that. So basically /etc and UNIX are bad ‘cause they’re not declarative.
Declarative won't necessarily give you cattle. What you want is a continuous automated build process, and immutable images. That way you have a reliable image that has been tested and you know works, you have documented the build steps, and get an error when the build fails. Over time you can make it more declarative, if being imperative is adding a lot of toil.
The important thing to remember is that improvements should happen gradually in small chunks; you don't have to have a perfect cattle system out of the gate, but you should be working your way there.
> You shouldn’t have a compiler on your webserver at all, and building a kernel should happen on a different machine than installing it.
More likely than not, if they've got execution permission on your webserver, they can send down a compiler to run. Plus or minus a c compiler doesn't open up much (anything?) they couldn't have compiled elsewhere or done in the scripting language or shell code or ... Unless you're running a static website or something. Plus, it's presumptious that all my servers are webservers. :p Do my other servers (other than build servers) get to have compilers because they're less naughty?
> And why isn’t the official kernel good enough?
Too many drivers, not enough local patches that are worth having but not worth pushing upstream. (Going upstream is a good metaphor, it often takes a lot of effort)
> Beyond the security surface that’s just asking to brick the machine; if you never make any mistakes it’s because you never tried anything very interesting with it in the first place.
That's what the console is for. Serial and IPMI consoles are way more convenient than vga consoles, but I've certainly screwed up bunches of times and had to use a console to sort things out.
> So basically /etc and UNIX are bad ‘cause they’re not declarative.
So you declare. ;) If it helps, I ran the script from Make, which is declarative (although not really the way we used it), and we mostly ran Erlang, which is claimed to be declaritive in Erlang: The Movie, but seems to be useful, so I'm not sure. I've never quite understood how the goal of declaritive system management matches with reality; it feels like a very leaky abstraction to me; the part that changes the system is necessary complexity, and hiding it away adds additional complexity. Maybe if you build a read-only root filesystem declaritively and run that? Which you can if you want? While you're saying I shouldn't do something the way I did it, my team managed a bunch of machines and a complex system with very few people, so I think the method works.
But I think about things differently than a lot of other people; I'm comfortable hot loading code to change uninterupted services, wheras others like to move traffic to new servers (and hope it all moves). You could transfer existing connections to your new servers, and that would be a lot of fun (and probably require a custom kernel), but nobody does that. Instead you either serve old connections with old code, or forceably terminate old connections. If I'm going to hot load code, it makes an immutable system directly opposed to operability. Why would I be interested in making my system less operable?
Adding on: farmera don't kill their livestock at the first sign of injury or illness; they won't do heroic interventions like you might do with a pet, but they'll certainly do simple fixes on the live cattle in the hopes of returning to health and retaining the value.
Just the other day I had to perform some maintenance on a long-running VM hosting some monitoring software. A backup VM is supposedly always running and ready to handle the workload in case of downtime. The switchover seemed to go fine, at first.
Turns out, someone long ago had manually added a cron job to the primary server without adding it to the backup server, without documenting what it does, what permissions it needs, how it works, or why it's needed. This was only discovered after some manager in a different department complained that he stopped receiving the daily report to his e-mail inbox.
If whoever deployed the report generation script took an extra hour to document what the script did, or even better, added it to VCS as part of the provisioning process for the server and re-deployed the server to ensure that the process works as expected, a day's worth of headache could have been averted.
Some replies are saying this is only for "as-scale/FAANG".
It may only be absolutely necessary there, but it's helpful even for smaller folks.
Over the years, even Debian LTS goes out of support and new features and software should be installed. There's moving systems, doing restores, things breaking and wanting to "reset" to a known working state. Any time you can do something simple with docker or even just (short) step-by-step build scripts, that's a huge win.
I have playbooks for deploying a system, but with npm installs, bower installs, secrets to be hand copied from multiple places, etc, it feels more like pets and it's NOT simple to deploy.
Just about any Certificate is worthwhile depending on your reasons. Best case I've seen them used for is to help you break into new technology areas, EG. you want to work as an SRE for AWS services, having a few AWS Certificates under your belt might be just enough to get you that interview (plus you'd kill at AWS trivia nights).
There's a lot of conceptual carryover between cloud platform offerings, so getting _any_ of the big 3 (GCP, AWS, Az) is likely to help you out a ton of you're new to the space. Much like how your first programming language took much longer than your second through fifth ones, learning your first platform well enough to get employed is much more challenging than filing the serials number off and learning the new quirks of the other two.
In the absence of further information as to your career goals, I'd lightly recommend AWS. It came into existence years before the others and can offer SLAs that approach "this S3 bucket will outlive you in the event of thermonuclear war".
Azure is where I have lived so far in my career and it seems to be catering more towards enterprise and government needs. I actually imagine finding an Azure shop is harder than an AWS shop if you haven't already worked at one before, but it's a pretty sweet gig otherwise.
GCP goes the other direction from what I've seen - much more startup-oriented, as the newest kid on the block itself. It looked nice from the last time I played around with it.
Kubernetes exists as a useful "stage 2" if you want to go further down the pipeline, as a technology whose business raison d'etre is to commoditize cloud providers. _In theory_ a Kubernetes cluster can be engineered to run unaltered on any of the big 3, since they all offer k8s clusters.
It's also totally cool to say that's okay thanks, I'll stick with simple architectures and a focus on getting MVPs out the door rapidly. For me $DAYJOB is spent between Azure and k8s, but my side projects start the same way every time - SQLite, Django, and _maybe_ Docker Compose to sidecar Litestream if I'm feeling extra infra-inclined that day. Really there's no reason to get dogmatic about anything in a space with so many options.
> A good monitoring system, well-organized repository, fault-tolerance workloads and automation mechanisms are the basis of any architecture.
Monitoring/alarming, and knowing what to monitor. Also, properly instrument your services or whatever it is you have. Take time to reflect on what are the signals that tell you operational health. An error metric alone is useless if you don’t know the denominator. Also be careful to avoid adding noisy metrics that cause panic for no reason.
I’m not sure what fault tolerance means in this context. Very handwavy statement. I think if you have dependencies, have a plan and understanding of which ones tipping over will bring down your service or how you can build resiliency. For example, some feature on your page requires talking to a recommendations service. If the service goes down, can you call back to a generic list of hard coded recommendations or some static asset?
As for automation: yeah, have test workflows built into your CI/CD harness. And avoid manual steps there requiring human intervention. Use canaries to test certain functions are up and running as expected, etc
Maybe I was a bit vague in the fault tolerance statement.
What I mean is to have a high availability in your services, e.g: using AWS ASG for your servers, having more than one replica for your Kubernetes pods.
If one of the servers/pods fail, the process behind detects them as "unhealthy" (having a nice monitoring/alarming as you mentioned) and replaces them with a new server with the same software characteristics so, for the end-user, SO, your client. Nothing has changed, the load just moved to a single instance for about 5-10 mins until a new server was deployed.
> If you need to build an architecture which involves microservices, I am sure that your cloud provider has a solution that fits better than Kubernetes. E.g: ECS for AWS.
Thank you! So many people running unnecessary things on Kubernetes
On the other hand, K8S provides you with orchestration abstraction across AWS, GCP, Azure, VMWare, bare metal.
There are distinct advantages to that in terms of both development (running a local K8S cluster is relatively easy) and deployment.
ECS has no distinct advantages over K8S (or EKS in AWS land). Particularly now that there are CRDs for K8S that allow you to deploy AWS functionality (eg ALBs, TGs) from K8S.
> If you need to build an architecture which involves microservices, I am sure that your cloud provider has a solution that fits better than Kubernetes. E.g: ECS for AWS. Kubernetes is a fantastic toolkit, but only shines when all that it has to offer, gets used.
As far as FAAS goes, I think more people need to go check out Cloud Run as a Knative implementation. Having used it for sometime now it feels like a near-perfect FAAS solution. The only gripe I have is that versioning is a bit dopey. But hey, if I can have autoscaling services with absolute impunity over how my HTTP interface is shaped (looking at you AWS lambda) and without needing to worry about Kubernetes headaches, I’m perfectly happy to embed version names in service domains.
Papas=PaaS, or? If Papas I am unfamiliar with the term.
If PaaS, isn’t Gcloud itself the PaaS? For instance cloud run, the product inside Gcloud, is ephemeral and stateless, which wouldn’t be at all good for trying to make a DB.
I believe PaaS predates FaaS, there used to be only IaaS, PaaS and SaaS. In my book App Engine and Cloud Run are PaaS, while Cloud Functions is FaaS. OTOH, there might not be any super clean definition
So many "aaS"es. It's definitely a spectrum. Thanks for pointing out App Engine, I will probably be using this in the future now that I read the documentation!
Truth is an interesting concept. It's often subjective and has many forms. Within the context of the cloud, almost all cloud services are only mutable, so "truth" is whatever the current state of the cloud actually is. Whatever is in Git is merely idealism.
Whatever you are maintaining, read the docs completely first. And I mean cover to cover. Not just the one chapter you need to get a PoC up and running. You will wish you had later, and it will come in handy many times over your career. Consider it an investment in your future.
Read books on microservices before you implement them. Whatever two-line quip you read on a blog will not be as good as reading several whole books from experts.
Docker multi-stage builds won't work in some circumstances. Build optimization eventually gets complex, the more you rely on builds to be "advanced".
Thanks for the alternative microservices quip, it was better than the original. Indeed, I find that “microservices should only perform a single task” is a really dangerous way to phrase it because we have no idea what the article means by “task.” The classic microservices separation is to separate an ordering service from a shipping service, is each of those one “task”? Or at the most extreme, is saving an order distinct from returning the list of your outstanding orders? Even when people graduate to a language of DDD and refine, often they settle on “one microservice per bounded context” where “bounded context” means “separated however I want it separated at the time,” and has no consistent principle behind... This despite the fact that I think Eric was quite explicit in his explanation of the idea, he meant a mapping of the software idea to the fussy complex world of businesspeople and business language: perhaps a better way to phrase this is that it's one microservice per archetype of user, “we have people from the warehouses who all speak the same shipping jargon, we should have a microservice specifically for them which speaks their language,” and I think most developers target their microservices smaller than that, in which case it is definitely not “one microservice per bounded context”.
Don't get me started on how “strong coupling” is shorthand for “coupled in ways I don't like” etc. ... Sometimes I feel like I'm on an episode of “whose line is it anyway?”, where everything is made up and the points don't matter.
All guidance assumes it's for a thinking person. You should look at what could be learned, not insist that there be one unambiguous message and that it be earth-shaking.
> I find that “microservices should only perform a single task” is a really dangerous way to phrase it because we have no idea what the article means by “task.”
First off, it's not dangerous. It's just loosely defined.
Second, it's not important to know their task size to still know that two different (to them!) things shouldn't be shoved together.
If the two tasks fit together so well that it's a mere 5% extra code to do them both, then maybe they are two sides of the same coin. But if "one simple thing" is painful to implement maybe it's really two separate things under one blanket.
> often they settle on “one microservice per bounded context” where “bounded context” means “separated however I want it separated at the time,”
Right, because sometimes 'bounded context' is what Legal tells you about data residency and sometimes it's about optimal latency.
> Don't get me started on how “strong coupling” is shorthand for “coupled in ways I don't like”
Strong coupling is generally easy to define, relative to loose coupling, in any given language/platform. The value of engineering around it depends on the value of doing the thing which requires it multiplied by how often you have to do it.
> First off, it's not dangerous. It's just loosely defined.
I'm not sure about this. Some people might take it at face value and end up with dozens if not hundreds of services for a domain that doesn't quite need them. Ergo, the project that they work on might have the development velocity plummet, because it has too much accidental complexity and ops overhead.
As anyone who has ever seen SOLID or DRY taken as gospel, many people out there would probably agree with the claim that some general advice can definitely be dangerous.
> Some people might take it at face value and end up with dozens if not hundreds of services
You're worried that someone junior enough to run with random advice is going to be able to rearchitect their entire codebase into hundreds of microservices and deploy it to production without testing its performance/stability/etc?
I think they should try. It's a great way to learn why, and why not.
> many people out there would probably agree with the claim that some general advice can definitely be dangerous.
It's just mouth noises which aren't dangerous. We need to be able to hear other people's opinions (advice is just an opinion after all) without immediately adopting it.
> You're worried that someone junior enough to run with random advice is going to be able to rearchitect their entire codebase into hundreds of microservices and deploy it to production without testing its performance/stability/etc?
No, I'm worried that someone senior will do that, because they want microservices on their CV. This has happened in the past and will continue happening in the industry with every new technology or approach - as I'm sure many have seen projects get burned due to being an early adopter of Kubernetes or Kafka, or even microservices in general.
As for how much testing should be done vs how much actually is done, you'd be surprised. While some orgs absolutely do things right, for a large part of them out there it's basically the Wild West - ship stuff and hope it works well enough. The entire e-health system of my country is a good example of this (though I cannot comment on the architecture itself): https://www-lsm-lv.translate.goog/raksts/zinas/zinu-analize/...
Some excerpts:
In December, the availability of e-health was disrupted for seven days (with one exception - on weekdays), but it actually did not work for five more weekdays. For example, on December 7, none of the e-health services were available for more than four hours (4 h 13 min) during the working day. January looks a little better - there are disruptions on six working days, while in February e-health is disrupted for three days, but for four days it can be said that the system does not work.
"If there are three, four interruptions for more than 10-15 minutes during an hour, then of course the user can be lucky to hit during the time when the system is working, but there are users who can only hit during the times when the system is not working, so I also marked red," explains Edgars Goba, deputy director of the National Health Service for information and communication technologies.
One of the users of the system, family doctor Ingars Burlaks says: "I have been working with e-health from the very beginning, since it appeared. Unfortunately, those problems that were there from the beginning, those chronic problems, like the system hanging up or getting kicked out of the system when you get a screen with an exclamation mark, those problems persist."
I've brought it up before, because it's one of the locally more well known examples. Furthermore, you'd think that it'd be the kind of system that should work: it's supposed to help with national healthcare, with millions spent on developing it, for the population of a small country. And yet, that's not the case.
> I think they should try. It's a great way to learn why, and why not.
This is something I can get behind for internal pilot projects, side projects, or hackathons and such. This is not something that I can get behind for projects with potential contractual penalties.
> It's just mouth noises which aren't dangerous. We need to be able to hear other people's opinions (advice is just an opinion after all) without immediately adopting it.
As long as no adopting advice at face value and no cargo culting follows, sure.
> As long as no adopting advice at face value and no cargo culting follows, sure.
I advise you to take all advice with a grain of salt. :) It's just what someone else thinks, it's not something magic.
> I'm worried that someone senior will do that, because they want microservices on their CV
Heh. I was going to say that the only way this advice is "dangerous" is if a point-haired boss hears about it. Someone padding a resume is the danger though, not the technology they choose.
> This is something I can get behind for internal pilot projects, side projects, or hackathons and such. This is not something that I can get behind for projects with potential contractual penalties.
Just don't merge it into the main branch.
> The entire e-health system of my country is a good example of this [...]
> I advise you to take all advice with a grain of salt. :) It's just what someone else thinks, it's not something magic.
Agreed, of course. Or maybe even suggestions that have some caution codified into them, such as "innovation tokens" from: https://boringtechnology.club/ Moderation in all things and so on.
> Heh. I was going to say that the only way this advice is "dangerous" is if a point-haired boss hears about it.
This is also true. I would definitely suggest that there are industry wide trends that urge people to rush towards "the next big thing" or sometimes adopt certain technologies or approaches because "others are doing it, so clearly we also have to". Perhaps a lot of it can simply be summed up by hype cycles: https://en.wikipedia.org/wiki/Gartner_hype_cycle
> Just don't merge it into the main branch.
The problem the way I see it, is that people don't just experiment with new architectures within the confines of an existing system. Typically they give in to the "second system effect" in regards to their former tech choices when starting a new project, which can have long term implications and choices that are really hard to change down the line: https://en.wikipedia.org/wiki/Second-system_effect
But hey, at the opposite end of that spectrum, maybe I shout at the cloud too much; exploring new stuff is certainly desirable - there were certainly benefits to being an early adopter of something like Docker or 12 Factor App principles, even though not without seeing what works and what doesn't in non-prod projects first.
> The problem the way I see it, is that people don't just experiment with new architectures within the confines of an existing system. Typically they give in to the "second system effect"
Have you tried to advise them otherwise?
> Or maybe even suggestions that have some caution codified into them
But then every suggestion is couched in a ton of warnings. Not only is this a bit useless, but if someone does start to depend on it they're at risk of a suggestion which lost its warning. Like, say the warning runs onto the next page and that page got lost.
There are an infinite number of possible suggestions and all could potentially rabbit-hole a team to death. Why not embrace immutability? Why not go serverless? Why not migrate to on-prem hardware? Why not use a memory-safe language?
It seems far better to build some sort of suggestibility-immunity by listening to others, and their suggestions, until you can understand but not feel compelled to copy their ideas.
"Microservices" are just "services". I use the business object/entity as the service boundary. So I don't have an "ordering" vs "shipping" service. I have an Orders service and a Shipments service.
Orders are related to Fulfillments are related to Shipments. They are "coupled" in that an Order will trigger Fulfillment, and Fulfillment will trigger Shipment (there's a Payments service in there somewhere too).
while I wouldn't wish the bootstrap problem on my worst enemy, I think the idealism helps for at least versioning configuration changes, and partial component-level tear-downs and bring ups (you don't need this often, but when you do, you do).
also, with k8s, nothing like deleting the wrong object or making a change and not knowing what it was, N revisions ago.
The only truth is the memory and disk contents of the devices that make up your cloud. Everything else is an abstraction of that, which discards data and potentially is out of sync with reality.
* When choosing internal names and identifiers (e.g. DNS) do not include org hierarchy of the team. Chances are the next reorg is coming faster than the lifetime of the identifier and renaming is often hard.
* The industry leading tools will contain bugs. From Linux kernel to deploy tooling, there are bugs everywhere. Part of your job is to identify and work around them until upstream patches make it to you if ever.
* Maintaining a patched fork is usually more expensive than setting up a workaround
* Your hyperscaler cloud provider has plenty of scalability limitations. Some of which are not documented. If you want to do something out of the ordinary make sure to check with your account rep before wasting engineering time.
* Bought SaaS will break production in the middle of the night. Your own team will have the best context and motivation to fix/workaround them. When choosing a vendor, include the visibility into their internal monitoring as a factor for disaster recovery (exported metrics and logs of their control plane for example)
> * Your hyperscaler cloud provider has plenty of scalability limitations. Some of which are not documented. If you want to do something out of the ordinary make sure to check with your account rep before wasting engineering time.
If only they'd tell you. We had this exact issue on AWS. Seemingly random packet drops. Metrics on both clients and servers were ok, latency specifically was very low when it worked.
Call up support "yeah, you're running into our connection limit". "Oh. What's that limit?" "yeah, I can't tell you that". His solution was that, since this was somehow related to connection tracking in the security group, I could set this to allow all/all, and set up filtering at the NACL level. Turns out I could do it for this particular issue.
This was before there was a possibility to monitor this [0]. Called up our customer manager. "Let me check". A few days later, "yeah, that's not something we divulge".
Ran into that one too, but my service rep didn't mention the possibility of configuring connectionless firewall rules. I'm still bitter many years later.
It wasn't the service rep who suggested that, but the support guy after I opened a ticket. I think this is an internal policy issue because he actually gave me the impression of trying to be helpful.
The service rep has been, overall, useless, when he wasn't actually less informed than I was (my knowledge being based on the AWS website).
> When choosing internal names and identifiers (e.g. DNS) do not include org hierarchy of the team.
Naming in general is hard. If you name stuff based on location, use an identifier that won't change, like provider datacenter names, street addresses or customer building codes, not the current tenant or purpose of use.
For products, come up with an internal product/project name and stick to it in everything that is not immediately visible to the customer. At one point you could see the current and three previous names of our product if you popped out an iframe and opened the inspector (logo with name, page title, URL and prefixed log messages).
> Maintaining a patched fork is usually more expensive than setting up a workaround
When your bosses demand additional features a single customer requested, you absolutely have to make them understand that the functionality must be added to the main product.
- anything my company doesn't create or own, is just called whatever it is
- logical components that aren't specific to the org chart can be whatever you want
- anything org chart specific gets a randomly generated code name. an internal website allows you to register and look up any code name across the company. this allows you to find who the hell owns server "ws-prod" in a 6 year old account that nobody seems to maintain. instead you find "peanutcar-ws-prod" and can then look up who registered "peanutcar". (this also prevents the bu-org-group-product-subproduct-env mess that eventually runs over character limits)
- doesn't matter if the rest of the company doesn't do it, I do it for what I manage. later on if it gets adopted, fine, but if not, at least I won't ever have to rename my crap.
> Microservices should only perform a single task. If you are not able to achieve that isolation, maybe you should switch back to a monolithic architecture. Do not get fooled by the current trends, microservices are not meant for everything.
I feel like this is spectacularly bad advice. "Do not get fooled by shades of grey, things are meant to be either black or white!"
EVERYTHING costs money. Tag every resource. Come up with ways to show cost avoidance and cost savings. This is will be appreciated more by management than any code you can bang out.
I love monitoring but after a few decades working I still haven't found a good way to monitor everything. Still a mix of email, pagerduty, prometheus, cloudwatch, websites, kibana consoles. Surely there is a good way to do this? I figure some of the new BI dashboards would be good but haven't seen much usage.
"Learn to say: I do not know about this/that. You cannot know everything that gets presented to you. The bad habit comes when the same technological asset appears for a second time and you still do not know how it works or what it does."
Absolutely. I've seen so many junior engineers / devs go on about it like this:
Someone higher up: Could you please look at this problem? I need it fixed ASAP.
Jr. Engineer, presented with a problem he's never seen before: No problem, I will look into it!
Someone higher up (the next day): Did you fix the problem?
Jr. Engineer: Sorry, I haven't still gotten around to look at it / I'm still working on it / etc.
Someone higher up: We really need it fixed today, please prioritize it and give me a call when it is fixed.
Jr. Engineer works on the problem all night, feeling stressed out, not wanting to let down his seniors.
151 comments
[ 3.5 ms ] story [ 100 ms ] threadCompletely agree with that.
I like to do short lived credentials using Vault (e.g. vault can create say db access credentials dynamically), but for things like API keys where I can't do that..? Is the Vault KV store the source of truth?
Edit: now that I think of it, for generated short lived passwords we also use SSM but for anything set by a human it’s in version control…
Probably.
Completely agree
> only
Fine, but can substitute "git" as appropriate
> discard any local files or changes
Ok for when deployment is completely and always automated, but for that _one special case_ maybe keep a copy of the old that you can revert to until you're _really_ sure of no unwanted effects. In the meantime, find out how & why that local change got made and what can be done to automate it next time
I feel like this could used as one of those "How to 10x career" articles - and be better than all of them.
(Of course, there are several other ways to get 100 engineers to have the output of 10; but I think microservices makes the engineers a lot less frustrated when it happens compared to more traditional alternatives)
Can you explain this reasoning?
With microservices, you make 10x as much work in such a way that the work seems meaningful and is perhaps even fullfilling. It takes some effort to see that the complexity you are solving day to day is accidental complexity, not essential complexity.
To your original point though, correlation doesn't equal causation, and just because you could abuse a technique to placate your wrong-sized org using this technique doesn't mean that it's the purpose of said technique.
Seems like this falls under "doing work", not creating output. You're doing a lot of work.
> The monolith doesn't work well in multi-team setups..
There are other ways of splitting work between multiple people and teams without introducing a network call.
I could say the same of a monolith -- if it is a big ball of mud then it is not a Proper Monolith ...
Granted, my only experience with microservices was a ball of mud mess microservices. I am sure we will both agree they were NOT doing microservices correctly -- BUT -- they did not know that going in did they?
If microservices is defined by definition to only be about the successsful applications of the technique, then it is not very helpful for those choosing architecture for a new system..
Re shifting services to other teams:
We have a medium-sized-service architecture, not monolith nor microservice. One or two services per team mostly.
By far, the hardest part of onboarding and doing development is learning the business domain and have people understand WHAT to do, why to do it, etc. -- the HOW is usually simple.
Code usually covers only the HOW part. Just shipping parts of what we do to another team and expect them to be on top of things quickly doesn't seem realistic regardless of the size of services..
Teams are homes for business and domain knowledge first. A home for the code is just a byproduct.
It also depends on how much functionality you consider to be “one thing”.
I thought microservices is a solution to scale development teams, not for traffic.
If you have a horizontally scalable monolith, it can scale pretty much as far as you want. If you split services along functional boundaries (i.e., vertical) then a split from 1 to 2 services will in the extreme best case scenario give you 2x scaleup; further splits give you less. So: If load is the issue, work on horizontal scaling, not microservices.
What am I missing?
As you add more functionality to a system, it gains different workloads, which have different needs (e.g. different storage solutions) and need to scale at different rates (e.g. one part of the system is very memory hungry and another is very compute heavy).
As your system gets larger, it's usually more important that its various components are well isolated, both on software and hardware axes, e.g. container isolation vs isolating across different physical zones/regions. Sometimes that isolation happens for security/compliance reasons, e.g. PCI.
As your business grows, you will need to build functional solutions that require different toolchains and platforms. You will acquire other businesses that have built on different platforms. You will not require all of your acquisitions to rewrite all of their software into your monolith.
With extra work, you can solve for many of these problems within a monolithic architecture, but those solutions will typically introduce similar complexities as microservices (e.g. talking to many datastores, composing your monolith of many coordinating processes, configuring independent nodes of your monolith differently, provisioning independent nodes appropriately, etc).
My experience in software is that we draw false dichotomies between approaches and the reality is that they will tend to converge when you put in the effort to consider and address the shortcomings of each approach.
Microservices have their own challenges apart from network communication etc between medium sized services that yiu refer to.
For instance lack of easily available consistency, sometimes needing to basically implement multi-service transactions (see SAGA pattern) and so on... if services are a bit larger, such problems occur much more seldom.
At some point, you have servers, that say, pre compute complex internationalization logic, and therefore are almost databases, and you want to keep them separate. Other times splitting requests that have very different memory vs cpu requirements into different boxes helps. And this is just for performance/costs.
The scariest issues are all operational though: Suddenly my giant fleet is misbehaving, and a rollback means 10K servers get hit, and I am not exactly sure what is going on. Having the functionality separated a little will do a lot of good.
Now, from there to needing microservices is a pretty big gap, but the lone uberservice can get very scary under the wrong circumstances, and the probability approaches 1 as the codebase gets really big.
I am all for dividing services after technical requirements; and certainly any non-O(1) memory requirements for instance one may want to look at a dedicated service.
When we get into the things you say, the definition of "service" gets more important too. Is it separate DB, or only separate k8s pod or VM, or what... If the only constraint is to make computation efficient and robust, the solution space is bigger if one does not add the "should fullfill the microservices ideology" as another constraint.
With microservices on k8s I think it is pretty common for each VM to have many open connections to 15+ databases though, so not sure if I got that particular point.
I ~hate~ dislike this advice. If you can't deploy on a Friday, you need to fix your deployment strategy. By removing Friday from when you can deploy, you're wasting 1/5 of your available days.
Note: deploy != Release[1]. Use flags, canaries etc.
[1]: https://andydote.co.uk/2022/11/02/deploy-doesnt-mean-release...
Edit: hate is far too stronger word for this
Would that still be required if a deployment and a release are decoupled entirely, or is it unavoidable? Genuinely interested!
It's a good interview question as a candidate. If you ask the interviewer when they deploy and they say only Friday (or worse only once a month) then perhaps look elsewhere for your own sanity because it's a sign of serious malfunction either organizationally, technically, or both.
Don't discount a job because of one deployment per month---it really depends upon the service. I joke that a busy year for me involved four deployments to production, but "production" for me wasn't a website, or even a web-based service, but a service involved in the call path for phone calls. Our customer is [1] the Oligarchic Cell Phone company and the SLAs are pretty severe.
I do have to ask---where do you people work where you have multiple deployments per week (or even per day)? To me, that sounds insane!
[1] Still is, even though I left a few months ago, and not because of the lack of deployments, but the shoving of Enterprise Agile [2] on the company by new management.
[2] Which is anything but Agile.
The upshot is that this is fairly rare and we do not have an on-call rotation. If most anything breaks over the weekend, nobody is going to notice or care until Monday morning rolls around.
However, I will admit it is a trade-off; some engineering time does have to be spent to get there, and perhaps that engineering time is better used elsewhere right now.
> Deploy is the act of moving software into an environment and running it.
> Release is the process of making a feature visible to a user or subset of users.
In our case, we deploy often (10s of times per day), and make use of feature flags for releasing. Keeping an eye on data in Honeycomb before and after a deployment and before and after a flag is changed is part of the engineers job.
If something doesn't look like it's working as intended, ideally it is disabled with a flag, rather than needing to figure out how a rollback can be done (often it can't), or rushing a fix and roll forward.
Regardless of how confident you are on your feature-flagging functionality, everyone's going to be unimpressed if suddenly your "minor" deploy has caused a bunch of things to go haywire at 4:30pm on a Friday and now people have to stay back and fix things.
If some of that new code is a flag that controls what customers see, then from a product perspective it’s not a “product release”. Is that what you mean?
The point is, even feature-flagged code can break things unexpectedly.
If you can't afford to give up 1/5 of your available deployment days you have a problem somewhere in your CI/CD system.
The reason we have core hours release only without director approval (aka director approval required outside core hours) is so you don't piss off another team by paging them after hours, and so you aren't trying to shove out a thing on a system that doesn't have good coverage or by turning off the safeties. In a large company I've noticed many engineers assume urgency even where there isn't. As an approver myself, most of the time someone wants to rush is because they've not even had the convo with their manager on if it's worth the risk, they are assuming urgency because that's when the sprint ends or what some TPM added to a jira ticket 4 months ago.
I admit that sounds risky itself (the engineers not having the right risk training) but this is why we have a policy and tooling... most of the times I've dug in they're just very new and worried about perception as a new employee, so my job is to shepherd them through having that convo with their managers which inevitably has the managers saying "yes it can totally wait till monday", and the change is inevitibly a bit more hot than it should be due to accidental deadline pressure.
I do disagree with the absolute statement of not doing it, but I definitely do a risk analysis whenever I ship a change on Friday and avoid anything risky and just push it off until Monday morning.
Not all changes can be put behind flags and canaries don't really fix the issue (unless you are okay in blocking important fixes from being rolled out due to your bad change killing the canary)
If you are a small company, or you do not do extra weekend shifts, I understand your point. Elsewhere, you just want to live an adventure every Friday.
Not saying it's the way to go for every situation.
My point: a single dedicated server may be a more reliable, simpler and much cheaper solution than the cloud provides.
I literally do not care about any particular server. If it fails, it kills itself, and a new server comes up and serves traffic.
No alerts, no pages, nothing.
These things are inherent complexities in any situation with backup/DR. You've already got to figure this out. When one machine is degraded and another is trying to recover from db logs, that's a cluster.
> This decision should be made per-project
This is one of the least project-dependent things ever, akin to source control. If you don't have scripted installs you don't know what you're running and you can't reliably test it or upgrade it.
You don't need to run a cluster, or have auto-scaling groups, just because you can.
Depends on who manages the fleet. An EC2 instance provisioned by Elastic Beanstalk is quite easy, the problem is EBS can be an utter PITA of edge cases and weird failure modes.
In what context? Individual instances fail frequently.
I've replaced pets with ASGs that do nothing but sit there unless the EC2 instance fails. Its amazing because you no longer have to baby the pet, it just recovers in a nice clean state, same massive instance..
Pets are a artifact of laziness.
I was with you until this unnecessary point.
An R620 plugged into a switch in a colo, a bash script via cron, or a cloudflare worker are just fine for a lot of use cases. The only time it stops being fine is when you can't afford to do your pet -> cattle migration as you scale up. But I don't think this is a common death for companies.
If you call "cattle" a cloudflare worker or lambda function - fine. But when we are talking about multiple redundant servers with load balancing across them, you really need to justify the cost of that vs the value you squeeze out. Sometimes you're squeezing the juice out of the rind.
That got us the comfort of knowing if the host machine died in the night the task would be rescheduled without a bunch of extra engineering to be able to scale arbitrarily when we only needed a couple of instances running.
My real, pointed question is: what was the opportunity cost of cattelizing that vs any of the other work you could have done in that time? Because given a fresh instance but I'm allowed to use the rest of my infra, setting that machine back up in the morning should just be to instantiating the right docker container on the host and copying the license file in. Depending on what the vendor software does, I probably still wouldn't ASG that until it had actually happened more often than once a month - there are so many more things to stress about! Even if this were the splunk license checker, losing some log files would be quite disappointing but not the absolute end of the world - if you're losing api calls to stripe or the equivalent that's a different matter!
They are not, if they are being configured by hand, using mouse or cli
Doubled up as amazing training / onboarding material for folks that maybe hadn't worked intimately with AWS before (including myself).
> Git should be your only source of truth. Discard any local files or changes, what's not pushed into the repository, does not exist.
Second, don't fool yourself into thinking, just because you wrote a script or configured some provisioning service, that means you've pulled yourself out of the valley of nondeterministic unreproducible production environments. Pets can and do live in git. A majority of the systems I've worked on in both FAANG and non-FAANG in my career couldn't be reproduced from the source repo in the case of a catastrophic disaster (I.e. us-east-1 disappears) or a year of being stable enough to not deploy even though their "source of truth" lives in git. The world moves out from under your repo and many of the steps you take for granted when writing your automation aren't understood by the next engineer running the automation. Then one day you have to go upgrade this one repo and realize it's IaC is no longer compatible with the infra in the real world and you burn a day trying to figure out what's wrong with a 1000LoC YAML file.
> Then one day you have to go upgrade this one repo and realize it's IaC is no longer compatible with the infra in the real world
Thats yet another reason to avoid pets. When I say pets I mean relatively big pieces of SW, usually 3rd party, that need non-trivial configuration. Like Jenkins or Windows or some security gizmos.
While I'm all too familiar with this problem from the NodeJS ecosystem, it's rare to encounter it in Terraform in my experience. The only exception that needs manual intervention in a "cold boot" scenario is AWS Cloudfront using certificates provisioned from AWS ACM - the provider will assume that certificates are already valid when creating the CF distribution and error out if ACM hasn't issued the certificate yet.
Sometimes you can justify using a thing for the wrong reasons.
I recently attached 1x NLB to each of our Swarm clusters to migrate to automatically managed certificates directly attached to the NLB (Digital Ocean).
$COMPANY has maybe ~3 users accessing each production application at a time. So the NLB itself is utterly pointless.
But Engineering no longer have to fix the certificates each quarter after users see an insecure browser warning and email us about it.
100% worth it for $12 pcm (per swarm cluster).
Cattle is the best approach. Practice it and make it your default.
This may make sense if you can only afford a single machine you want to hyperoptimize but it’s not a good workflow for anyone. Not even BSD kernel developers.
Beyond the security surface that’s just asking to brick the machine; if you never make any mistakes it’s because you never tried anything very interesting with it in the first place.
Furthermore, I claim that “you can write a script to edit /etc” is not how an OS should support configuration. Instead of being possible to keep a copy of the config in a script somewhere else, it should be impossible to do anything but that. So basically /etc and UNIX are bad ‘cause they’re not declarative.
The important thing to remember is that improvements should happen gradually in small chunks; you don't have to have a perfect cattle system out of the gate, but you should be working your way there.
More likely than not, if they've got execution permission on your webserver, they can send down a compiler to run. Plus or minus a c compiler doesn't open up much (anything?) they couldn't have compiled elsewhere or done in the scripting language or shell code or ... Unless you're running a static website or something. Plus, it's presumptious that all my servers are webservers. :p Do my other servers (other than build servers) get to have compilers because they're less naughty?
> And why isn’t the official kernel good enough?
Too many drivers, not enough local patches that are worth having but not worth pushing upstream. (Going upstream is a good metaphor, it often takes a lot of effort)
> Beyond the security surface that’s just asking to brick the machine; if you never make any mistakes it’s because you never tried anything very interesting with it in the first place.
That's what the console is for. Serial and IPMI consoles are way more convenient than vga consoles, but I've certainly screwed up bunches of times and had to use a console to sort things out.
> So basically /etc and UNIX are bad ‘cause they’re not declarative.
So you declare. ;) If it helps, I ran the script from Make, which is declarative (although not really the way we used it), and we mostly ran Erlang, which is claimed to be declaritive in Erlang: The Movie, but seems to be useful, so I'm not sure. I've never quite understood how the goal of declaritive system management matches with reality; it feels like a very leaky abstraction to me; the part that changes the system is necessary complexity, and hiding it away adds additional complexity. Maybe if you build a read-only root filesystem declaritively and run that? Which you can if you want? While you're saying I shouldn't do something the way I did it, my team managed a bunch of machines and a complex system with very few people, so I think the method works.
But I think about things differently than a lot of other people; I'm comfortable hot loading code to change uninterupted services, wheras others like to move traffic to new servers (and hope it all moves). You could transfer existing connections to your new servers, and that would be a lot of fun (and probably require a custom kernel), but nobody does that. Instead you either serve old connections with old code, or forceably terminate old connections. If I'm going to hot load code, it makes an immutable system directly opposed to operability. Why would I be interested in making my system less operable?
Adding on: farmera don't kill their livestock at the first sign of injury or illness; they won't do heroic interventions like you might do with a pet, but they'll certainly do simple fixes on the live cattle in the hopes of returning to health and retaining the value.
Just the other day I had to perform some maintenance on a long-running VM hosting some monitoring software. A backup VM is supposedly always running and ready to handle the workload in case of downtime. The switchover seemed to go fine, at first.
Turns out, someone long ago had manually added a cron job to the primary server without adding it to the backup server, without documenting what it does, what permissions it needs, how it works, or why it's needed. This was only discovered after some manager in a different department complained that he stopped receiving the daily report to his e-mail inbox.
If whoever deployed the report generation script took an extra hour to document what the script did, or even better, added it to VCS as part of the provisioning process for the server and re-deployed the server to ensure that the process works as expected, a day's worth of headache could have been averted.
It may only be absolutely necessary there, but it's helpful even for smaller folks.
Over the years, even Debian LTS goes out of support and new features and software should be installed. There's moving systems, doing restores, things breaking and wanting to "reset" to a known working state. Any time you can do something simple with docker or even just (short) step-by-step build scripts, that's a huge win.
I have playbooks for deploying a system, but with npm installs, bower installs, secrets to be hand copied from multiple places, etc, it feels more like pets and it's NOT simple to deploy.
Can anyone recommend some certifications that are worthwhile? I realize that this is a very broad ask, but the advise is also rather broad.
EKS?? EKS? EC2? EBS..? ELB..? Ah no way it was /data egress/ of course.
There's a lot of conceptual carryover between cloud platform offerings, so getting _any_ of the big 3 (GCP, AWS, Az) is likely to help you out a ton of you're new to the space. Much like how your first programming language took much longer than your second through fifth ones, learning your first platform well enough to get employed is much more challenging than filing the serials number off and learning the new quirks of the other two.
In the absence of further information as to your career goals, I'd lightly recommend AWS. It came into existence years before the others and can offer SLAs that approach "this S3 bucket will outlive you in the event of thermonuclear war".
Azure is where I have lived so far in my career and it seems to be catering more towards enterprise and government needs. I actually imagine finding an Azure shop is harder than an AWS shop if you haven't already worked at one before, but it's a pretty sweet gig otherwise.
GCP goes the other direction from what I've seen - much more startup-oriented, as the newest kid on the block itself. It looked nice from the last time I played around with it.
Kubernetes exists as a useful "stage 2" if you want to go further down the pipeline, as a technology whose business raison d'etre is to commoditize cloud providers. _In theory_ a Kubernetes cluster can be engineered to run unaltered on any of the big 3, since they all offer k8s clusters.
It's also totally cool to say that's okay thanks, I'll stick with simple architectures and a focus on getting MVPs out the door rapidly. For me $DAYJOB is spent between Azure and k8s, but my side projects start the same way every time - SQLite, Django, and _maybe_ Docker Compose to sidecar Litestream if I'm feeling extra infra-inclined that day. Really there's no reason to get dogmatic about anything in a space with so many options.
The number of issues I've seen that turn out to be documented features... (or, more accurately, things just being configured incorrectly)
Monitoring/alarming, and knowing what to monitor. Also, properly instrument your services or whatever it is you have. Take time to reflect on what are the signals that tell you operational health. An error metric alone is useless if you don’t know the denominator. Also be careful to avoid adding noisy metrics that cause panic for no reason.
I’m not sure what fault tolerance means in this context. Very handwavy statement. I think if you have dependencies, have a plan and understanding of which ones tipping over will bring down your service or how you can build resiliency. For example, some feature on your page requires talking to a recommendations service. If the service goes down, can you call back to a generic list of hard coded recommendations or some static asset?
As for automation: yeah, have test workflows built into your CI/CD harness. And avoid manual steps there requiring human intervention. Use canaries to test certain functions are up and running as expected, etc
If one of the servers/pods fail, the process behind detects them as "unhealthy" (having a nice monitoring/alarming as you mentioned) and replaces them with a new server with the same software characteristics so, for the end-user, SO, your client. Nothing has changed, the load just moved to a single instance for about 5-10 mins until a new server was deployed.
Thank you! So many people running unnecessary things on Kubernetes
There are distinct advantages to that in terms of both development (running a local K8S cluster is relatively easy) and deployment.
ECS has no distinct advantages over K8S (or EKS in AWS land). Particularly now that there are CRDs for K8S that allow you to deploy AWS functionality (eg ALBs, TGs) from K8S.
Have a good logging & rollback strategy well communicated across stakeholders
As far as FAAS goes, I think more people need to go check out Cloud Run as a Knative implementation. Having used it for sometime now it feels like a near-perfect FAAS solution. The only gripe I have is that versioning is a bit dopey. But hey, if I can have autoscaling services with absolute impunity over how my HTTP interface is shaped (looking at you AWS lambda) and without needing to worry about Kubernetes headaches, I’m perfectly happy to embed version names in service domains.
If PaaS, isn’t Gcloud itself the PaaS? For instance cloud run, the product inside Gcloud, is ephemeral and stateless, which wouldn’t be at all good for trying to make a DB.
I believe PaaS predates FaaS, there used to be only IaaS, PaaS and SaaS. In my book App Engine and Cloud Run are PaaS, while Cloud Functions is FaaS. OTOH, there might not be any super clean definition
Whatever you are maintaining, read the docs completely first. And I mean cover to cover. Not just the one chapter you need to get a PoC up and running. You will wish you had later, and it will come in handy many times over your career. Consider it an investment in your future.
Read books on microservices before you implement them. Whatever two-line quip you read on a blog will not be as good as reading several whole books from experts.
Docker multi-stage builds won't work in some circumstances. Build optimization eventually gets complex, the more you rely on builds to be "advanced".
Don't get me started on how “strong coupling” is shorthand for “coupled in ways I don't like” etc. ... Sometimes I feel like I'm on an episode of “whose line is it anyway?”, where everything is made up and the points don't matter.
> I find that “microservices should only perform a single task” is a really dangerous way to phrase it because we have no idea what the article means by “task.”
First off, it's not dangerous. It's just loosely defined.
Second, it's not important to know their task size to still know that two different (to them!) things shouldn't be shoved together.
If the two tasks fit together so well that it's a mere 5% extra code to do them both, then maybe they are two sides of the same coin. But if "one simple thing" is painful to implement maybe it's really two separate things under one blanket.
> often they settle on “one microservice per bounded context” where “bounded context” means “separated however I want it separated at the time,”
Right, because sometimes 'bounded context' is what Legal tells you about data residency and sometimes it's about optimal latency.
> Don't get me started on how “strong coupling” is shorthand for “coupled in ways I don't like”
Strong coupling is generally easy to define, relative to loose coupling, in any given language/platform. The value of engineering around it depends on the value of doing the thing which requires it multiplied by how often you have to do it.
I'm not sure about this. Some people might take it at face value and end up with dozens if not hundreds of services for a domain that doesn't quite need them. Ergo, the project that they work on might have the development velocity plummet, because it has too much accidental complexity and ops overhead.
That's how you get humorous parody videos like these ("Microservices" by KRAZAM): https://www.youtube.com/watch?v=y8OnoxKotPQ
As anyone who has ever seen SOLID or DRY taken as gospel, many people out there would probably agree with the claim that some general advice can definitely be dangerous.
You're worried that someone junior enough to run with random advice is going to be able to rearchitect their entire codebase into hundreds of microservices and deploy it to production without testing its performance/stability/etc?
I think they should try. It's a great way to learn why, and why not.
> many people out there would probably agree with the claim that some general advice can definitely be dangerous.
It's just mouth noises which aren't dangerous. We need to be able to hear other people's opinions (advice is just an opinion after all) without immediately adopting it.
No, I'm worried that someone senior will do that, because they want microservices on their CV. This has happened in the past and will continue happening in the industry with every new technology or approach - as I'm sure many have seen projects get burned due to being an early adopter of Kubernetes or Kafka, or even microservices in general.
As for how much testing should be done vs how much actually is done, you'd be surprised. While some orgs absolutely do things right, for a large part of them out there it's basically the Wild West - ship stuff and hope it works well enough. The entire e-health system of my country is a good example of this (though I cannot comment on the architecture itself): https://www-lsm-lv.translate.goog/raksts/zinas/zinu-analize/...
Some excerpts:
I've brought it up before, because it's one of the locally more well known examples. Furthermore, you'd think that it'd be the kind of system that should work: it's supposed to help with national healthcare, with millions spent on developing it, for the population of a small country. And yet, that's not the case.> I think they should try. It's a great way to learn why, and why not.
This is something I can get behind for internal pilot projects, side projects, or hackathons and such. This is not something that I can get behind for projects with potential contractual penalties.
> It's just mouth noises which aren't dangerous. We need to be able to hear other people's opinions (advice is just an opinion after all) without immediately adopting it.
As long as no adopting advice at face value and no cargo culting follows, sure.
I advise you to take all advice with a grain of salt. :) It's just what someone else thinks, it's not something magic.
> I'm worried that someone senior will do that, because they want microservices on their CV
Heh. I was going to say that the only way this advice is "dangerous" is if a point-haired boss hears about it. Someone padding a resume is the danger though, not the technology they choose.
> This is something I can get behind for internal pilot projects, side projects, or hackathons and such. This is not something that I can get behind for projects with potential contractual penalties.
Just don't merge it into the main branch.
> The entire e-health system of my country is a good example of this [...]
I'll check it out. Thanks.
Agreed, of course. Or maybe even suggestions that have some caution codified into them, such as "innovation tokens" from: https://boringtechnology.club/ Moderation in all things and so on.
> Heh. I was going to say that the only way this advice is "dangerous" is if a point-haired boss hears about it.
This is also true. I would definitely suggest that there are industry wide trends that urge people to rush towards "the next big thing" or sometimes adopt certain technologies or approaches because "others are doing it, so clearly we also have to". Perhaps a lot of it can simply be summed up by hype cycles: https://en.wikipedia.org/wiki/Gartner_hype_cycle
> Just don't merge it into the main branch.
The problem the way I see it, is that people don't just experiment with new architectures within the confines of an existing system. Typically they give in to the "second system effect" in regards to their former tech choices when starting a new project, which can have long term implications and choices that are really hard to change down the line: https://en.wikipedia.org/wiki/Second-system_effect
But hey, at the opposite end of that spectrum, maybe I shout at the cloud too much; exploring new stuff is certainly desirable - there were certainly benefits to being an early adopter of something like Docker or 12 Factor App principles, even though not without seeing what works and what doesn't in non-prod projects first.
Have you tried to advise them otherwise?
> Or maybe even suggestions that have some caution codified into them
But then every suggestion is couched in a ton of warnings. Not only is this a bit useless, but if someone does start to depend on it they're at risk of a suggestion which lost its warning. Like, say the warning runs onto the next page and that page got lost.
There are an infinite number of possible suggestions and all could potentially rabbit-hole a team to death. Why not embrace immutability? Why not go serverless? Why not migrate to on-prem hardware? Why not use a memory-safe language?
It seems far better to build some sort of suggestibility-immunity by listening to others, and their suggestions, until you can understand but not feel compelled to copy their ideas.
Orders are related to Fulfillments are related to Shipments. They are "coupled" in that an Order will trigger Fulfillment, and Fulfillment will trigger Shipment (there's a Payments service in there somewhere too).
It definitely is loosely defined and the rules do get stretched to fit opinions though.
also, with k8s, nothing like deleting the wrong object or making a change and not knowing what it was, N revisions ago.
I learned from using software like Photoshop and Ableton Live, that you shouldn't underestimate the complexity of any software you use.
Take a few days or weeks, if you can, to read docs or do high quality courses on the topic and it will make your life easier in the long run.
* When choosing internal names and identifiers (e.g. DNS) do not include org hierarchy of the team. Chances are the next reorg is coming faster than the lifetime of the identifier and renaming is often hard.
* The industry leading tools will contain bugs. From Linux kernel to deploy tooling, there are bugs everywhere. Part of your job is to identify and work around them until upstream patches make it to you if ever.
* Maintaining a patched fork is usually more expensive than setting up a workaround
* Your hyperscaler cloud provider has plenty of scalability limitations. Some of which are not documented. If you want to do something out of the ordinary make sure to check with your account rep before wasting engineering time.
* Bought SaaS will break production in the middle of the night. Your own team will have the best context and motivation to fix/workaround them. When choosing a vendor, include the visibility into their internal monitoring as a factor for disaster recovery (exported metrics and logs of their control plane for example)
If only they'd tell you. We had this exact issue on AWS. Seemingly random packet drops. Metrics on both clients and servers were ok, latency specifically was very low when it worked.
Call up support "yeah, you're running into our connection limit". "Oh. What's that limit?" "yeah, I can't tell you that". His solution was that, since this was somehow related to connection tracking in the security group, I could set this to allow all/all, and set up filtering at the NACL level. Turns out I could do it for this particular issue.
This was before there was a possibility to monitor this [0]. Called up our customer manager. "Let me check". A few days later, "yeah, that's not something we divulge".
---
[0] For those who don't know, it's now possible to keep an eye on refused connections (at least on Linux). https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/monitori... -> conntrack_allowance_exceeded
The service rep has been, overall, useless, when he wasn't actually less informed than I was (my knowledge being based on the AWS website).
Naming in general is hard. If you name stuff based on location, use an identifier that won't change, like provider datacenter names, street addresses or customer building codes, not the current tenant or purpose of use.
For products, come up with an internal product/project name and stick to it in everything that is not immediately visible to the customer. At one point you could see the current and three previous names of our product if you popped out an iframe and opened the inspector (logo with name, page title, URL and prefixed log messages).
> Maintaining a patched fork is usually more expensive than setting up a workaround
When your bosses demand additional features a single customer requested, you absolutely have to make them understand that the functionality must be added to the main product.
- anything my company doesn't create or own, is just called whatever it is
- logical components that aren't specific to the org chart can be whatever you want
- anything org chart specific gets a randomly generated code name. an internal website allows you to register and look up any code name across the company. this allows you to find who the hell owns server "ws-prod" in a 6 year old account that nobody seems to maintain. instead you find "peanutcar-ws-prod" and can then look up who registered "peanutcar". (this also prevents the bu-org-group-product-subproduct-env mess that eventually runs over character limits)
- doesn't matter if the rest of the company doesn't do it, I do it for what I manage. later on if it gets adopted, fine, but if not, at least I won't ever have to rename my crap.
I feel like this is spectacularly bad advice. "Do not get fooled by shades of grey, things are meant to be either black or white!"
Absolutely. I've seen so many junior engineers / devs go on about it like this:
Someone higher up: Could you please look at this problem? I need it fixed ASAP.
Jr. Engineer, presented with a problem he's never seen before: No problem, I will look into it!
Someone higher up (the next day): Did you fix the problem?
Jr. Engineer: Sorry, I haven't still gotten around to look at it / I'm still working on it / etc.
Someone higher up: We really need it fixed today, please prioritize it and give me a call when it is fixed.
Jr. Engineer works on the problem all night, feeling stressed out, not wanting to let down his seniors.