> Microservices only pay off when you have real scaling bottlenecks, large teams, or independently evolving domains. Before that? You’re paying the price without getting the benefit: duplicated infra, fragile local setups, and slow iteration. For example, Segment eventually reversed their microservice split for this exact reason — too much cost, not enough value.
Basically this. Microservices are a design pattern for organisations as opposed to technology. Sounds wrong but the technology change should follow the organisational breakout into multiple teams delivering separate products or features. And this isn't a first step. You'll have a monolith, it might break out into frontend, backend and a separate service for async background jobs e.g pdf creation is often a background task because of how long it takes to produce. Anyway after that you might end up with more services and then you have this sprawl of things where you start to think about standardisation, architecture patterns, etc. Before that it's a death sentence and if your business survives I'd argue it didn't because of microservices but inspite of them. The dev time lost in the beginning, say sub 200 engineers is significant.
Some resume driven developers will choose microservices for startups as a way to LARP a future megacorp job. Startup may fail, but they at least got some distributed system experience. It takes extremely savvy technical leadership to prevent this.
I've also seen the top down version where senior leadership like a CIO/CTO wants to put a huge "modernization" project on their resume and they don't care if it is impossible to maintain or falls over after they move on.
Oh yeah, I've come to realise that the people at the top are just as bad. I was at a place where "consultants decided" that the company's two backbone systems should be replaced with off-the-shelf solutions. Very obviously a resume-padding project from the CIO who was in their first large company job, but which would have almost certainly crippled the company if it went ahead.
Thankfully the business and finance people at the company decided it would be an expensive and stupid idea it was shut down.
In my experience, it seems the majority of folks know the pitfalls of microservices, and have since like... 2016? Maybe I'm just blessed to have been at places with good engineering, technical leadership, and places that took my advice seriously, but I feel like the majority of folks I've interacted with all have experienced some horror story with microservices that they don't want to repeat.
I feel like it's only in the last 5 years in the tech publicity sphere that I've seen pushback against microservices, only it feels like only the last year or two where I see it to the exclusion of influencers pushing microservices.
Things are different in the embedded space so I don't have personal experience with any of it.
Pushback was always there from the start. The first edition of O'Reilly's "Building microservices" recommended _against_ microservices, unless you absolutely tried scaling your monolith and team beforehand.
Any organization stuck in microservice hell fully deserves the punishment.
> In my experience, it seems the majority of folks know the pitfalls of microservices, and have since like... 2016?
Majority maybe, but not everyone. I was at a place in 2018 where a guy turned up declaring that microservices would solve all of the company's performance issues (he'd heard that AWS S3 was made up of over 500 microservices so we must do the same, but he obviously had no idea about the depth of features in S3 so hit take was more is better).
So we got like 30 microservices in the space of year which gave the company 10x complexity, complicated and fragile local development and... dubious wins like faster email sending, but no improvements to true bottle-necks of the system.
So true. It was in March that I saw on HN an advertisement for a vibe coder with 3 years experience. I believe the term "vibe coding" was invented a month before! Buzzword hiring is as bad as resume driven development.
Hiring will need to change to stop resume-driven development (can't eliminate it completely though), because you're likely to only get monolith roles if you only work on monoliths. Only being able to speak about microservices puts you in the "talk the talk, not walk the walk" category.
It would also nice to have less fear-driven career advice like "your skills go out of date" which drives people to try adopting the latest things.
Keyword driven and filtered application processes also heavily incentivize adding into projects whatever is being posted on jobs sites. If microservices are part of a company's standard template for developer postings, people who want to work at that company will find a way to get it on their resume.
It could also just be plain old overengineering. Like using Django and leaning on all of the magic contained within it just to implement a simple API that could instead be a very small Flask or FastAPI app.
Unforeseen scope creep is the reason to utilise Django over Flask, I feel.
Also, you can pick and choose what to use in Django similarly to Flask - it just has a higher initial learning curve.
Once you get to sufficient levels of complicated, leaning on established, documented, community supported design patterns and abstractions helps vs. sorting out your imports, making bespoke design choices, and doing a bunch of non-core value producing work.
I will forever beat the Django over Flask drum. Nothing ever stays simple. Maybe today it is one single view function, but tomorrow someone is going to want to add authentication, emails, whatever. Django might have features you do not need, but it would be an unusual situation where Django is holding you back vs Flask.
> You'll have a monolith, it might break out into frontend, backend and a separate service for async background jobs
And when you break these out, you don't actually have to split your code at all. You can deploy your normal monolith with a flag telling it what role to play. The background worker can still run a webserver since it's useful for healthchecks and metrics and the loadbalancer will decide what "roles" get real traffic.
If you are building the same binary for all microservices you lose the dependency-reduction benefit microservices provide, since your build will still break because of some completely unrelated team's code.
If it is possible for that other team to merge a broken build, you are doing it wrong.
If you are concerned about someone else breaking your thing, good! You were going to eventually break it yourself. Write whatever testing gives you confidence that someone else's changes won't break your code, and, bonus, now you can make changes without breaking your code.
Even if it builds successfully, I've never worked anywhere where automated tests prevented 100% of problems and I doubt I ever will. For most systems of sufficient complexity you are testing in prod, even if you did a lot of testing before prod as well.
That's even more true for microservices, though, since I have yet to see a microservice architecture that automatically runs end to end tests before deploying.
The post I was replying to said "your build will still break": that's what I was taking issue with. In this day and age there is no reason our trunk build should ever be broken.
> I have yet to see a microservice architecture that automatically runs end to end tests before deploying.
One of the big tenets of independent services is that your APIs are contracts that don't change behaviour. As long as each individual service doesn't introduce breaking changes, the system as a whole should work as expected. If it doesn't this is indicative of either 1) a specific service lacking test coverage, or 2) doing something wrong i.e. directly reading from a microservices' database without going through an API.
Yes, I suspect some of the back and forth is the fuzziness of the term "broken build", whether that means the code literally doesn't compile or it does but the code does the wrong thing.
I agree that you can prevent merges that cause compilation errors in nearly all cases!
A very large code base full of loosely related functionality makes it more and more likely a change in one part will break another part in unexpected ways.
> If it is possible for that other team to merge a broken build, you are doing it wrong.
This assertion is unrealistic and fails to address the problem. The fact that builds can and do break is a very mundane fact of life. There are whole job classes dedicated to mitigate the problems caused by broken builds, and here you are accusing others of doing things wrong. You cannot hide this away by trying to shame software developers for doing things that software developers do.
> Write whatever testing gives you confidence that someone else's changes won't break your code, and, bonus, now you can make changes without breaking your code.
That observation is so naive that casts doubt on whether you have any professional experience developing software. There are a myriad of ways any commit can break something that goes well beyond whether it compiles or not. Why do you think that companies, including FANGs, still hire small armies of QAs to manually verify if things still work once deployed? Is everyone around you doing things wrong, and you're the only beacon of hope? Unreal.
I haven't seen a broken build in at least nine years, not since I left the company with a merge process built out of bash scripts that took three hours and required manual hand-holding.
I am genuinely curious what situations you are seeing where builds are making it through CI and then don't compile.
It isn't always worth investing in quality, but when it is it is entirely possible to write essentially bug-free software. I've gone seven months without a bug in production and the one we saw we had a signed letter from product saying "I am okay if this feature breaks, because I think writing the tests that can verify this integration was going to take too long."
FAANG companies aren't prioritizing writing software well: they are prioritizing managing 50,000 engineers. Which is a much harder problem, but the management solutions that work for that preclude the techniques that let us write bug-free software.
One of the great things about startups is that it is trivial to manage five engineers, so there is no reason we have to write software badly.
Of course, if people wrote bug-free code, then there would be no bug !
Bug-free code in the actual code, or bug-free code in the test code, this is the same story.
If you write stuff and never have any bug, then either:
- you are lying
- you do not write much
- you only write really simple things
- you are Jesus, came back from heaven to shine his light on us, poor souls
The more complicated, intricate stuff you have, the more bugs you'll get (and only time will allow you to fix that).
Tests are great do define how you think it should work, and to ensure it keeps doing that way. Take the time to think about the third point on the bullet list above.
In the DVCS era, we have inexpensive branching. Do as thou wilt on your topic or epic branches. Rebase them against main/master before merging upwards. Fix what must be fixed first.
Main/master branch should never fail CI. If it does, there is something seriously wrong with your branch lifecycle and/or deployment process.
If you test defensively[0], CI will catch the vast majority of functional bugs. Anything that is missed suggests at least two bugs: one in tests, and one in business logic.[0]
A reliable deployment pipeline is outside of CI, but can be kept straightforward and minimal to constrain the scope of failures.
Bugs happen, and systems complexify. It is possible to manage both of those risks down to near-zero by the time code reaches production release candidate stage.
In some industries, this is more important than others, though -- obviously the goal is to match quality to business needs.
But I agree with our thread predecessor: I haven't seen a broken build make it anywhere near production in many years, and it's not because of the snarky dismissal that provoked my original response.
[0] In some situations, proper tests are not possible, and in others they are not practical. And I acknowledge that I'm omitting things like visual design/layout bugs, which is probably not reasonable.
I disagree. I write complex code, and it is essentially bug-free.
And no, I'm not Jesus, I just care a lot about quality and have spent the last 20 years finding ways and strategies to improve it.
Reducing the number of bugs does not mean being a god that writes bug-free code on the first draft. It means being able to detect and fix issues as early as possible. In my case I aim to always do that before letting myself push any code to git.
IMO, it only comes down to how much someone really cares about the quality, but here are some examples of what can be done and is very effective:
- Plan ahead your functional and technical design
- Carefully research existing code to confirm the feasibility of the design
- Use a statically typed language
- Use advanced static-analysis tools
- Avoid magic, write explicit code. Especially avoid runtime checks such as reflection. Ideally, everything should be checked statically one way or another.
- Never let a code path/branch/corner case be unhandled, however unlikely it is (and go back to step one to refine the design if a code path has been forgotten in the current design)
- Always have automated testing. The bare minimum is to unit-test all business logic, including all possible code paths. Ideally e2e tests are nice, but not always a good investment. Tests must be 100% independent and never depend on an external environment, otherwise it's going to be flaky at some point.
- Always manually test every feature and path related to my changes (especially don't skip testing the ones that I think are going to be ok) before pushing anything to git.
- Warnings and "optional" notices are unacceptable and must always be fixed (or disabled), otherwise the list will just keep growing, which reduces the visibility of any issue and normalizes having problems.
- Have a CI integration that applies all the automated checks mentioned in this list and make everything mandatory.
Each one of those actions does on it's own significantly reduce the number of bugs. If you combine them all, you can effectively reduce the number of bugs to pretty-much zero. And since the earlier you find a bug, the cheaper is it overall to fix, I've also found-out that in terms of productivity it's always worth the investment (despite many people pretending the opposite).
You have a point, but I wouldn't say this is a big deal unless there is a mammoth dependency somewhere that slow down things to a crawl. Then maybe that one part of the codebase can be broken into its own separate service.
But even then there are ways around this kind of problem with dynamic linking pre-built binaries and caching, but it is extra complexity that could be worse than managing multiple services. Docker cache can usually handle this pretty well though.
There are plenty of tech reasons for microservices. e.g. scaling high traffic services separately and separating low priority functionality from critical paths. I would agree that this is usually not a smart thing to do in a small org, but I have seen times where splitting out a high load path into a microservice has been very much worth it at a startup.
This is a great optimization once you have high traffic services
Building this way before you have any traffic at all is a great way to build the wrong abstractions because your assumptions about where your load will be might be wrong
Microservices are a technical solution to regional availability and pairing problems, and they start with a spreadsheet telling you when to make them based on requirements vs. cost. They're slow, expensive threads you should have a really good reason to use.
> Building this way before you have any traffic at all is a great way to build the wrong abstractions
These services only make sense to think about within specific traffic contexts. It'd be impossible to build the right abstraction.
I saw one startup with about fifty engineers, and dozens of services. They had all of the problems that the post describes. Getting anything done was nearly impossible until you were in the system for at least six months and knew how to work around all the issues.
Here’s the kicker: They only had a few hundred MAUs. Not hundreds of thousands. Hundreds of users. So all this complexity was for nothing. They burned through $50M in VC money then went under. It’s a shame because their core product was very innovative and well architected, but it didn’t matter.
I don't think I learned basically anything about "fancy architecture" from my undergraduate courses except, ironically, reasoning about coupling and overhead.
I don't remember one solitary lecture on CI/CD, microservices, or even just deployment in general, in Uni. The closest that our comp. sci. classes ever came to touching on anything but the code itself was making us use SVN.
I've never heard of or seen a Software Development bachelors degree?
I've seen Information Systems programs, that are usually CE, after-hours tracks. Neither Harvard, Yale, nor MIT have a software dev one, just Comp. Sci. I'm calling BS (no pun intended) on "software dev degrees" as a thing distinct from CS in any widespread fashion.
I’m not sure where’s the downside. The engineers got paid, they managed to put “founder” on their cvs, and enjoyed the ride. Now they are
more prepared for their next adventure.
The only ones who lost money were the investors, but nobody cares about them.
I have worked with a company with ~100k MAUs with ~4 teams, even then it often feels the system is over-microserviced (about two dozen services I think).
Definitely some stuff makes sense (especially since it has a lot of IoT stuff), but micro-service added was used mostly as a way to develop new stuff without having to deal with the legacy monolith. The core of the application could easily be a single service backed by one big RDBMS with a few ancillary services around it.
The legacy monolith is still there kicking and screaming, it didn't need "breaking up" it needed (and I assume still do) need a major refactoring.
I thought the linked article about how Khan Academy eventually migrated to multiple services was a good example of when introducing micro services is a good idea:
They had already scaled the mono service about as far as it could go and had a good sense of what the service boundaries should be based on experience.
> Microservices only pay off when you have (...) independently evolving domains.
I don't see any major epiphany in this. In fact, it reads like a tautology. The very definition of microservice is that it's an independently evolving domain. That's a basic requirement.
I put my team through this as an inexperienced lead about 15 years ago. We were a team of less than a dozen who had a nice single solution file that you could build and run the entire stack from. At the end we were looking at roughly a dozen services all which required orchestration to get them running and working together. First hand lessons in YAGNI and "do the simplest thing that works" which have stuck with me the rest of my career.
Sounds right, no? Service is what people provide, implied in the scope of a macro economy. Microservice then implies the same type of service, but within the micro economy of a single business.
I've tended to use microservices in limited cases where the system had to serve a few requests that had radically different performance requirements, particularly memory utilization. I had a PHP server for instance that served exactly one URL for which PHP was not a good fit and a specialized server in another language for that one URL gave like 1000x better performance and money savings in terms of not needing a much bigger PHP server.
Using Spring or Guava in the Java world it is frequent that people write "services" that are simply objects that implement some interface which are injected by the framework. In a case like that you can imagine a service could have either an in-process implementation or an out-of-process implementation (e.g. via a web endpoint or some RPC.) Frameworks like that normally are thinking at the level of "let's initialize one application in one address space at a time" but it would be nice to see something oriented towards managing applications that live in various address spaces.
Trouble is that some people get this squee when they hear they can use JDK 9 for this project and JDK 10 for another project and JDK 11 for another project and they'd rather die than eschew the badly broken Python 3.5 for something better. If you standardized absolutely everything I think you could be highly productive with microservices because you wouldn't have to face gear switching or deal with teams who just don't know that XML serialization worked completely differently in JDK 7 vs JDK 8 thus the services they make don't quite communicate properly, etc.
> Microservices are a design pattern for organisations as opposed to technology
Very true in my experience. The main benefit is letting small groups of people work independently without stepping on each other’s toes. Although I’ve worked on a project where multiple teams owned micro services that were supposed to be standardized with each other, and it just lead to endless meetings and requirements churn since nobody was willing to work on the other teams service but everyone had an opinion on what the cross-team standard should be. Learning the diplomatic way to say “mind your own business” was more important than any technical skills for getting code merged.
> Microservices are a design pattern for organisations as opposed
> to technology ... breakout into multiple teams
I agree, but just saying "multiple teams" has led many eng directors to think "I have two squads now --> omg they cannot both be in the same monolith".
When both squads are 5 people each.
And the squads re-org (or "right size") every 9 months to re-prioritize on the latest features.
Five years go by, 7 team/re-org changes, all of which made sense, but thank god we didn't microservice on the 2nd/3rd/4th/5th/6th team boundaries. :grimmacing:
We should stay "stable, long-lived teams" -- like you need to have a team that exists with the same ownership and mandate for ~18 months to prove its a stable entity worth forming your architecture around.
They have their place. In my experience, a good rule of thumb[0] is if there are actual benefits from being a standalone service.
For example, we have a authentication microservice at work. It makes sense that it lives outside of the main application, because its used in a multiple different contexts and the service boundary allows for it to be more responsive to changes, upgrades and security fixes than having it be part of the main app, and it deploys differently than the application. It also adds enough intentional friction that we don't accidentally put logic where it doesn't belong as part of the user authentication process. It has helped keep the code focused on only primary concerns.
That said, you can't apply any of these patterns blindly, as is so often the case. A good technical leader should push back when the benefits don't actually exist. The real issue is lack of experience making technical decisions on merits.
This includes high level executive leaders in the organization. At a startup especially, they are still often involved in many technical decisions. You'd be surprised (well maybe not!) how the highest leadership in a company at a startup will mandate things like using microservices and refuse to listen to anything running counter to such things.
Its an international forum, there may be at least 1 person who hasn't encountered this colloquialism before. It hinders nothing yet may be informative to someone who's unfamiliar.
And I dont see how the link detracts from the post at all. For ESL people, like me, it can be quite helpful to have such a link. (i find myself looking up such phrases quite often)
Most “benefits” assumed from separation can be achieved with clear interfaces and modular monoliths, without the cognitive and operational tax microservices impose.
> It also adds enough intentional friction that we don't accidentally put logic where it doesn't belong as part of the user authentication process.
Preventing misplaced logic is a matter of good code structure, well defined software development processes and team discipline - not something that requires splitting into a separate microservice, and definitely not something that you want to solve on system architecture level.
>Most “benefits” assumed from separation can be achieved with clear interfaces and modular monoliths, without the cognitive and operational tax microservices impose.
Perhaps yes. Every situation should be evaluated on merits. This came across that there is also an assumption that we didn't try other solutions first - we absolutely did. Microservice is the best solution to solving the problems we needed solved in this case. Even better than modular monolith with clear interfaces.
>without the cognitive and operational tax microservices impose
When done correctly, I don't think there is a tax. Most operational questions should be automated away once discovered. The only 'tax' is that it lives separately from the larger application and is deployed independently, but I haven't seen in practice this add any notable overhead.
>Preventing misplaced logic is a matter of good code structure, well defined software development processes and team discipline
All true, and a microservice can aid all of these things too, but isn't the solution you should reach for when solving for these things and these things alone in my opinion. That said, myself and others have observed there is time saved on enforcing discipline around this issue once we separated the code away from the main application. I can't deny that hasn't been a good thing, because it has. It would be leaving information out about the benefits we'v experienced, and I see no reason to do that.
All told, completely dismissing the value of microservices as a potential solution is no different than completely dismissing other solutions in favor of microservices. Things have their place, there are pros and cons to them, and should be evaluated relative to their merit for the situation.
You may find you never implement microservices, or implement very few, or perhaps the needs of an organization is as such that its a pattern used most of the time, but the technical merits of doing so - with any decision of this nature, not limited to microservices - should have a backing justification that includes why other solutions don't fit
> Every situation should be evaluated on merits. This came across that there is also an assumption that we didn't try other solutions first - we absolutely did.
I completely agree. But this a little bit contradicts with your original comment that caught my eye:
> In my experience, a good rule of thumb[0] is if there are actual benefits from being a standalone service.
A rule of thumb is, by nature, a generalization — it simplifies decision making through heuristics. Benefits on the other hand always subjective, they can be interpreted in a given context.
And based on my experience, there will always be some benefits that can be used to justify factoring something out into a separate service. The challenge is that it's often easy to overemphasize those benefits, even when they don't outweigh the downsides. Your example with the auth service and the added friction is, in my view, a good illustration of a justification that might sound reasonable but can lead to unnecessary complexity. (Just to be clear, my intent here isn't to judge your decisions - I understand these trade-offs are often nuanced - and that's why again there is no good rule of thumb for this)
>I completely agree. But this a little bit contradicts with your original comment that caught my eye:
Its a good rule of thumb that you may want to evaluate if a microservice is appropriate. Thats the point in context[0]. If you think it might be a relevant solution, that is a pretty good heuristic that evaluating it may be worth the time.
What I seem to be coming up against when discussing this is people conflate worthwhile evaluation with worthwhile solution. Thats where the nuance and details need to live is in solution discovery, but how do you arrive at the right solution if you don't first evaluate what solutions might fit a given problem?
I stand by it being a good rule of thumb. If you think there are actual benefits - not perceived, but quantifiable benefits - of something being a standalone independent service, you might have a case for a microservice. I feel its a good heuristic for narrowing solution evaluation.
It doesn't equate to saying microservice is the solution if it meets that one criteria, only don't rule it out.
>And based on my experience, there will always be some benefits that can be used to justify factoring something out into a separate service. The challenge is that it's often easy to overemphasize those benefits, even when they don't outweigh the downsides. Your example with the auth service and the added friction is, in my view, a good illustration of a justification that might sound reasonable but can lead to unnecessary complexity.
Bias is hard to overcome. Technical decision makers need to be keenly aware of this. I wish it was easier to identify when this comes into the situation as it plays a bigger role in all this than is often realized. I go to great lengths to validate my thoughts when making big technical decisions for this reason, and deciding on something like this is a big technical decision that deserves that approach, in my opinion.
One good quality of a good technical decision maker is taking the time to quantify why its a good solution and why other solutions are not, and that must hold up to technical scrutiny of your peers. This challenges any assumptions missed. Ideally, your peers should be able to see through anything that wasn't given enough evaluation, like if the benefits of doing something is overemphasized.
>good illustration of a justification that might sound reasonable but can lead to unnecessary complexity.
I agree, which is why I state elsewhere that it shouldn't be used solely for this purpose, but I'm not going to leave out a real benefit we saw. If there is a persistent organizational problem that engineers seem to want to put code inside of something that is in broader context inappropriate or adds complexity where it shouldn't, you may benefit from such friction and its okay to evaluate that aspect too.
[0]: By which I mean in the context of the article, which I think dismisses microservices as a potential solution with prejudice
> By which I mean in the context of the article, which I think dismisses microservices as a potential solution with prejudice
> What I seem to be coming up against when discussing this is people conflate worthwhile evaluation with worthwhile solution.
> It doesn't equate to saying microservice is the solution if it meets that one criteria, only don't rule it out.
I read the article, it's not against microservice architecture itself, but points out that you shouldn't treat them as a starting point or a best practice.
There is even a section named "When Microservices Do Make Sense".
It also highlights examples where microservices make sense ("Their post is a good example of how microservices can work when you have the organizational maturity and operational overhead to support them."), so I really do not see where does anyone rule out, dismiss, come up against anything.
I somewhat understand and agree what you are trying to argue against, but that's definitely not in this article.
> Its a good rule of thumb that you may want to evaluate if a microservice is appropriate.
This is too general to be practically useful — it applies to almost everything in life.
The largest benefit of microservices has always been lifecycle management, and "clear interfaces" in "modular monoliths" does not in fact solve that. If you update the logging library in a monolith, everyone takes that updates even if it breaks half the teams.
That's a "large" organization problem. But large is actually, not that big (about 5-10 scrum teams before this is a very large problem).
It also means on critical systems separating high risk and low risk changes are not possible.
Like all engineering decisions, this is a set of tradeoffs.
> The largest benefit of microservices has always been lifecycle management, and "clear interfaces" in "modular monoliths" does not in fact solve that.
What lifecycle are we really talking about? There are massive monoliths - like the Linux kernel or PostgreSQL - with long lifespans, clear modularity, and thousands of contributors, all without microservices. Lifecycle management is achievable with good architecture, not necessarily with service boundaries.
> If you update the logging library in a monolith, everyone takes that updates even if it breaks half the teams.
This is a vague argument. In a microservice architecture, if multiple systems rely on the structure or semantics of logs — or any shared behavior or state - updating one service without coordination can just as easily break integrations. It’s not the architecture that protects you from this, but communication, discipline, and tests.
> It also means on critical systems separating high risk and low risk changes are not possible.
Risk can be isolated within a monolith through careful modular design, feature flags, interface boundaries, and staged rollouts. Microservices don’t eliminate risk - they often just move it across a network boundary, where failures can be harder to trace and debug.
I’m not against microservices. But the examples given in the comment I responded to reflect the wrong reasons (at least based on what I’ve seen in 15+ years across various workplaces) for choosing or avoiding a microservice architecture.
Microservices don’t solve coupling or modularity issues — they just move communication from in-process calls to network calls. If a system is poorly structured as a monolith, it will likely be a mess as microservices too — just a slower, harder-to-debug one.
Btw, do any good, modern CI tools support incremental rollout of multiple in-flight changes on monoliths? As in patch A is live, team B wants to rollout A+B and team C wants to rollout A+C. Ideally, A+B+C will eventually go live.
Do cloud/paas providers deeply support this flow anymore? Every dashboard would need to compare across multiple live versions and I haven't tried that in a while.
I'd say this is a job for feature flags. That way you always have exactly one live version of the code, but still retain the ability to hide WIP from users until it's ready.
If you're instead doing this with feature branches or something like that, then by definition you don't have CI. You have NI: Never Integration.
Because, to an approximation, there's never any point in time where all of the code you're working on is integrated together so that everyone has a chance to see how what they're doing interacts with what everyone else is doing. And yes, it is possible for a branch to successfully auto-merge and produce something that compiles and passes all automated tests, and still introduce a horrible regression defect because of an unanticipated interaction between two different changes on two different feature branches. I don't see it happen often, but when it does it usually creates such a big production SNAFU that even once every 5 years is still way too often for my taste.
I don't dispute that good CI is facilitated by committing early and often but difference do feature flags really make? You still need to roll out features whether you tie that to versions or not.
If you're looking for performance regressions as you roll out feature flags, do you get any kind of support for that from built in dashboards? How do you get clean metrics if you only have one version and a bunch of teams turning knobs?
SOA/Microservices have a lot of headaches but you do get the option for very tightly scoped deployments or feature rollouts.
Monkey patching is a great technique for hacking rudimentary testability into legacy software as part of your preparations for refactoring it for maintainability.
But when I see a plan to use it that doesn't include a plan for how to stop using it again ASAP, I get very worried.
grug mention grug brain. grug also have grug brain. grug like grug. grugs together strong unless too many grugs then Overgrug think 9 grugs make baby grug in one month and grug not think it work like that
This is also why app backends don't really need statically typed languages, no matter how big the company is. You have a well-defined API on the front, and you have a well-defined DB schema on the back, that's good enough.
The static typing makes even less sense at finer code scopes, like I don't need to keep asserting that a for-loop counter is an int.
"Need"? Probably not. But unlike microservices they don't really have downsides (at least not with modern IDEs and the automatic refactorings they support) and they do offer some benefits.
Statically-types languages are a form of automatically-verified documentation, and an opportunity to name semantic properties different modules have in common. Both of those are great, but it is awkward that it is usually treated as an all-or-nothing matter.
Almost no language offers what I actually want: duck typing plus the ability to specify named interfaces for function inputs. Probably the closest I've found is Ruby with a linter to enforce RDoc comments on any public methods.
I'm fine with types in shared libs, just not in the app layer code, where the cost outweighs the benefit. I think you can do the in-between you describe with Typescript, but every time I've been on a team that says "oh you can use `any`," one day they disallow it. Especially in a big corp where someone turns it into a metric and a promo target.
Statically typed languages, when used correctly, save engineering time both as you extend your service and when thing go wrong as the compiler helps you check that the code you've written, to some degree, meets your specification of the problem domain. With a weak type system you can't specify much of the problem domain without increased labour but with a more expressive type system (and a team that understands how to use it) you can embed enough of the domain specification that implementing part of the business logic incorrectly or violating protocols turns into compile errors instantly rather than possibly leaking to production.
As for your comment on `any`, the reason why one doesn't want to fall back on such is that you throw out most of the gains of using static types with such a construct when your function likely doesn't work with `any` type (I've never seen a function that works on absolutely anything other than `id :: a -> a` and I argue there isn't one even with RTTI).
Instead you want to declare the subset of types valid for your function using some kind of discriminated union (in rust this is `enum`, zig `union(enum)`, haskell ADTs/GADTs, etc etc) where you set a static bound on the number of things it can be. You use the type system to model your actual problem instead of fighting against it or "lying" (by saying `any`) to the compiler.
The same applies to services, APIs, protocols, and similar. The more work the compiler can help you with staying on spec the less work you have to do later when you've shipped a P1-Critical bug by mistake and none of your tests caught it.
The type system almost never catches a bug that proper testing would miss. And if the code has such nasty untested edge cases that you don't even notice a wrong type going somewhere, it'd probably behave wrongly even with the right types.
Indeed "any" breaks type checking all around it, but it can be contained more easily in a helper func with a simple return type. Most common case is your helper does a SQL query, and it's tedious and redundant to specify the type of rows returned when the SQL is already doing that.
It saves development time because if I change an API my language server can immediately notify me about all the now-broken call sites, and I don't have to wait for tests to run to find out about all of them.
The type system doesn't replace unit/snapshot/property/simulation tests as it's only job is specification. The type system is meant to be used in addition to testing to reduce the set of possible inputs to a smaller domain such that it's easier to reason about what is possible and what isn't. The same would be true even if you go as far as formal verification of programs, you always need to test even when you have powerful static types!
For example `foo :: Semigroup a, Traversable t => t a -> a` I already know that whatever is passed to this function can be traversed and have it's sum computed. It's impossible to pass something which doesn't satisfy both of those constraints as this is a specification given at the type level which is checked at compile-time. The things that cannot be captured as part of a type (bounded by the effort of specification) are then left to be captured by tests which only need to handle the subset of things which the above doesn't capture (e.g `Semigroup` specifies that you can compute the sum but it doesn't prevent `n + n = n` from being the implementation of `+`, that must be captured by property tests).
Another example, suppose you're working with time:
tick :: Monad m => m Clock.Present
zero :: Clock.Duration
seconds :: Uint -> Clock.Duration
minutes :: Uint -> Clock.Duration
hours :: Uint -> Clock.Duration
add :: Clock.Duration -> Clock.Present -> Clock.Future
sub :: Clock.Duration -> Clock.Present -> Clock.Past
is :: Clock.Duration -> Clock.Duration -> Bool
until :: Clock.Future -> Clock.Present -> Clock.Duration
since :: Clock.Past -> Clock.Present -> Clock.Duration
timestamp :: Clock.Present -> Clock.Past
compare :: Clock.Present -> Clock.Foreign.Present -> Order
data Order = Ahead Clock.Duration | Equal | Behind Clock.Duration
From the above you can tell what each function should do without looking at the implementation and you can probably write tests for each. Here the interface guides you to handle time in a safer way and tells a story `event = add (hours 5) present` where you cannot mix the wrong type of data ``until event `is` zero``. This is actual code that I've used in a production environment as it saves the team from shooting themselves in the foot with passing a `Clock.Duration` where a `Clock.Present` or `Clock.Future` should have been. Without a static type system you'd likely end up with a mistake mixing those integers up and not having enough test coverage to capture it as the space you must test is much larger than when you've constrained it to a smaller set within the bounds of the backing integer of the above.
In short, types are specifications, programs are proofs that the specification has a possible implementation, and tests ensure it behaves correctly for that the specification cannot constrain (or it would be too much effort to constrain it with types).
As for SQL, I'd rather say the issue is that the SQL schema is not encoded within your type system and thus when you perform a query the compiler cannot help you with inferring the type from the query. It's possible (in zig [1] at least) to derive the type of a prepared SQL query at compile-time so you write SQL as normal and zig checks that all types line up. It's not that types cannot do this, your tool just isn't expressive enough. F# [2] is capable of this through type providers where the database schema is imported making the type system aware of your SQL table layouts solving the "redundant specification" problem completely./
So with all of that, I assume (and do correct me if I'm wrong) that your view on what types can do is heavily influenced by typescript itself and you've yet to explore more expressive type systems (if so I do recommend trying Elm to see how you can work in an environment where `any` doesn't even exi...
> `foo :: Semigroup a, Traversable t => t a -> a` I already know that whatever is passed to this function can be traversed and have it's sum computed. It's impossible to pass something which doesn't satisfy both of those constraints as this is a specification given at the type level which is checked at compile-time.
To add to your point, I don't think foo can even be implemented (more accurately: is not total) because neither `Semigroup a` or `Traversable t` guarantee a way to get an `a`.
I think you'd need either `Monoid a` which has `mempty`, or `(Foldable1 t, Traversable t)` which guarantees that there's at least one `a` available.
Yep, I missed it as I don't often work in haskell anymore but with the correction the rest of the above still stands (haskell syntax is still the most efficient I'm aware of for talking about this). Also, it being unimplementable is also a nice property of having such a type system as you can prove that your specification is impossible to implement (more applicable in Lean, ATS2, Idris, Agda) or make invalid states impossible to even write (consider state transition rules at the type level).
> The type system almost never catches a bug that proper testing would miss.
This is true, but the difference is you don't have to write a compiler, it's already written for you. The testing, you have to write, and do so correctly.
A lot of the woes of statically typed languages can be mitigated with tooling. Don't want to repetitively create types from an OpenAPI spec? Generate the code. Don't want to create types from SQL records? Generate the code. Don't want to write types everywhere? Deduce them.
You get all the benefits of static typing, but none of the work. It's so advanced these days that lots of statically-typed languages look dynamically-typed when you see the code. But they're not, everything has a type if you hover over them. The type deduction is just that good.
You need tests either way. It's hard to write a test that checks behavior but misses a wrong-type. Simply running the problematic code will most likely throw an exception.
The type deduction is not so automatic in most languages, TS included. Rust has the most automatic one I've seen, and of course that kind of language needs static typing. But still, it's more explicit than needed for a typical web backend.
SQL type autogen is limited to full rows, so any query returning an aggregate or something isn't going to work with that. Even for full rows, it's eh. Usually I just see that encouraging people to do local computations that should be in SQL.
The argument isn't that with static types you don't need tests but rather with static types you can focus on testing what actually matters which is the fuction itself along with it's integration into the program. Tests won't cover 100% of the surface area of a function taking `any` as that is by definition infinite with the constraint on the shape specified in the body of the function dependent on the path taken at runtime (it might never be hit). I urge you to take a look at languages other than TS for this as TS is in the strange place of bolting on a static type system to rather messy language as what may be issues with TS may not be for the rest of the space of languages with static type checking (e.g haskell effectively requires no type annotations at all in theory but in practice it's often better to use them and with extensions it's required).
To add to that, try looking at Go, Elm, Zig, C#, mercury, and Idris to see different type systems at play rather than assuming TS and Rust are all that's available in this design space.
> Simply running the problematic code will most likely throw an exception.
The issue here is "most likely" which during a large refactor (and due to a wide spread use of duck typing) can hide many faults within the application that go without being noticed until a user hits that path. Static types turn this into a compile-error thus it won't ever reach production. If you have 100% path coverage and test every possible type that can be passed and ensure truthy values don't change behaviour (e.g `if ("")`, `if ([])`, ...) when passed (need to test the space where all of them evaluate to `false` in addition to `true`) you're still short of the number of variations that can be passed to your function and thus may end up in a situation in future where the wrong behaviour causes failure at runtime post deployment. This is not to say you should use types only and not tests (still test even with static types) but rather that the domain of possible things passed is reduced to a testable set of values instead of always being the largest path-dependent set possible.
> It's hard to write a test that checks behavior but misses a wrong-type.
It's incredibly easy to write such tests as most don't test the negative space of such functions nor is it easy to know what the minimal shape of an object may be for your function as it depends on which logic it hits inside. If a truthy value is used or a branch is made on a field or method call of the object then anything unused in the consequent branch is not checked by your test. You'd effectively have to resort to fuzzing to find what the minimal shape is for each possible path to have an idea of what your function doesn't catch in tests to get even close to static type checking (which is still far off as in some cases the set is so large it will never finish / enumerate all possible shapes).
> SQL type autogen is limited to full rows, so any query returning an aggregate or something isn't going to work with that.
Ask _why_ tooling doesn't work with that and you may notice it's due to a lack of specification or just insufficient tooling. Take F# [1] as the example here which can do far more [2] and doesn't suffer from the mentioned problem. F# is a language with static type checking.
In my experience of dealing with loosey goosey languages (JS, PHP, Perl), 80%+ of the errors you will receive are because of the type system.
In PHP, if you look at your logs you'll see almost all the errors are checking an array with an index that doesn't exist. This is because people are using arrays as objects and then using strings as members. If you just use an object, this type of error is impossible.
So, we have to write `??` everywhere because anything can always be null and then it can break stuff.
And then you have errors with passing empty string vs null vs empty array to functions and getting unpredictable behavior. So you need to constantly check everything.
If you actually open up a function in a dynamically typed language, take your pick, you'll see something like this:
If you don't include checks like that everywhere, your code will break. You just don't know it isn't broken yet.
And, btw, something like PHP `empty()` is not a silver bullet either. Because it considers like a dozen different values to be empty. Which comes with it's own set of problems.
I forgot to add that "like I don't need to keep asserting that a for-loop counter is an int." is exactly what is happening with a dynamically typed language that is exactly what the runtime ends up doing unless it has a built-in range type to avoid that overhead and the loop variable cannot change while looping. With a static type checker that can be eliminated upfront as the compiler knows that it's an int and not suddenly a string as it's impossible to change the variable's type once it's defined thus all of the overhead of RTTI can be erased.
Javascript has to check on each iteration that the item is an int and for arrays that the length of those arrays hasn't changed underneath along with a bunch of other things as the language doesn't guarantee that it can't change. Even the JIT compiler in use has to check "is this path still that I expect" as at any point the type of the variable used in the loop can change to something else which invalidates the specialization the JIT compiler emitted for the case of it being an int. When you don't use languages with static types you push all of this work on runtime which makes the program slower for every check it needs to do while offering none of the advantages you have with static types.
Thus with a for loop in say C you don't assert that it is an int each time you statically constrain it to be an int as the loop condition can well be based on something else even if int is the more common to use. For example in zig `for` only takes slices and integer ranges with any other type being a compile-error:
var example: [1 << 8]usize = @splat(0);
for (&example, 0..) |*num, int| num.* = int;
This is not more work than what you'd do in a dynamically typed language.
You're probably on the early part of the curve where anything works - small team, simple product, no scale - come back when one or two of these changes...
I know about a org with ~2-3 devs who decided microservices would be cool. I warned not to go that way because they would surely face delivery and other issues which they wouldn't have when building the solution based on a architecture archetype which could be a better fit for the team and solution, which I evidently decided should be a modular monolith. (the codebase at that point was already a monolith, in fact, but had a large amount of tech debt due to the breakneck speed in which features needed to be released)
They ignored me and went the microservices way.
Guess what?
2 years later the rebuild of the old codebase was done.
3 years later and they are still fighting delivery and other issues they would never have had if they didn't ignore me and just went for the "lame" monolith.
Moral of this short story: I can personally say everything this article says is pretty much true.
> 3 years later and they are still fighting delivery and other issues
Having added a fancy new technology and a "successful" project to their resume, they're supposed to move on to the next job before the consequences of their actions are fully obvious.
The best use case is promotion! Welcome to big tech, where all the teams get reshuffled every few months and every microservice exists because some dev needed a promotion. The greater the ratio of microservices to devs, the better your manager looks! (Dev work-life balance be damned, we pay you to ruin your life.)
Yep, I remember working at a place where one team churned out a large majority of the microservices and the other three teams just kept on doing their thing.
The microservice team were especially terrible because the did all the initial work and basked in the "glory", but when it came to maintaining the services, they wanted nothing to do with it.
At some point it just doesn't scale for the team that owns all the micro services. It's poor organizational decision making to have this setup.
Sure a monolith that does 1000 things might not be ideal, but 100 repos for 100 micro services owned by a team of 5-10 devs is unmanageable on the other end of the spectrum. Oh and everyone forgets the orchestration layer.
1 team supporting multiple services is not great, but a monolith with more than 50 developers working on it (no matter how you split your teams) isn't great either.
That's why I don't like the term "microservice", as it suggests each service should be very small. I don't think it's the case.
You can have a distributed system of multiple services of a decent size.
I know "services of a decent size" isn't as catchy as "go for one huge monolith!" or "microservices!" but that's the sensible way to approach things.
Folder and file structure and separation of concerns doesn't change the fact that if you have one deployable artifact, it's all sharing the same runtime when deployed. Which means the underlying versions of Java/Go/Python/etc, or core shared libraries, all need to be updated at the same time. All the code is far more coupled than it first seems.
Monorepo is orthogonal to services though. You can have a monorepo with multiple services in it.
Even with a monorepo, you will hit a point where you have 1, 10, 100 million lines of e.g. Python, realize you should upgrade from 3.8 to 3.14 because it's EOL, and feel a lot of pain as you have to do a big-bang, all-at-once change, fixing every single breaking change, including from libraries which you also have to update. There's no way around this in current mainstream languages.
That is not really an issue I've had with Java, but I would absolutely agree that Python is wildly unsuited as a production backend language.
I don't think it's much better if you have to spend a year and a half updating 400+ different repos, though. It's much easier to use an operationalized language that knows backwards compatibility matters.
I was at AWS RDS when they upgraded the shared control plane code from Java 7 to 8. IIRC it was about 6 months for 5-10 developers more or less full-time. Absolutely massive timesink. The move to separate services happened shortly after that.
> I don't think it's much better if you have to spend a year and a half updating 400+ different repos, though.
There's two things going for separate services (which may or may not be separate repos; remember a single repo can have multiple services):
1. You can do it piecemeal. 90% of your services will be 15-minute changes: update versions in a few files, let automated tests run, it's good to go. The 10% that have deeper compatibility issues can be addressed separately without holding back the rest. You can't separate this if you have a single deployable artifact.
2. Complexity is superlinear with respect to lines of code. Upgrading a single 1mLOC service isn't 10x harder than updating ten 100kLOC services, it's more like 20, 30x harder. Obviously this is hard to measure, but there's a reason these massive legacy codebases get stuck on ancient versions of dependencies. (And a reason companies pay out the ass for Oracle's extended Java 8 support, which they still offer.)
Part of the problem is that many current programmers came up through functional programming or framework-based development. Microservices are often the first time they encountered modular programming or encapsulation, and so they equate "literally any architecture" with "microservices".
I've worked on monoliths with 400+ developers that were great, but it takes skills that people who have only ever worked in orgs that mandate microservice just don't have.
Functional programming precludes encapsulation, so it doesn't scale indefinitely the way fractal paradigms can. Eventually, the complexity becomes overwhelming.
One effective solution to that is introducing microservices: programmers can still write entirely functional code, but have encapsulation in the form of services. They have to be micro, though, because conventionally-sized services are still big enough to strain the paradigm.
But I see junior engineers who aren't expected to think about the "architecture", by which they mean the modular design. They are handed a spec and they implement it, Mythical Man Month style. That treats organizing lines of code and organizing services as two completely-distinct activities, and depending on the company junior engineers are often not exposed to modular design until five or ten years into their careers.
You’re suffering from a misunderstanding there. Functional programming is all about encapsulation, starting at the individual function boundary (closures can encapsulate state) and then at every layer above that.
Functional languages have some of the most rigorous module systems available. In fact Java adopted such a system recently, showing the weaknesses in its previous support for encapsulation via classes and packages.
I think he means stateless, at some point a system needs to have some internal state to keep things running and that state can be really hard to manage when it is mixed with all other functions in the system that also have state.
Even if that state is kept outside the service itself (like a database or event queue) it can still be really hard to reason about when said state-stores are shared across a huge codebase. Changes to a part of the state can have negative effects in completely unrelated functionality.
And of course, there is nothing blocking a monolith from isolating/modularizing their state-stores, but it tends to not happen unless the architecture forces it to happen or through strong tech leadership.
This is really what it comes down to right here. The real challenge is Conway's Law. Both the software architecture and the org chart need to be designed with Conway's Law in mind. If that hasn't happened then deciding between microservices and monolith is ultimately just deciding how you will be punished for your mistake.
People misunderstanding Conway's Law is a big part of the problem for sure. The law says nothing about team boundaries: it talks about communication pathways.
The paranoid socialist in me thinks big companies like team-sized microservices because it lets them prevent workers from talking to each other without completely ruling out producing running software.
When companies instead encourage forums for communication across team boundaries, it unlocks completely different architectural patterns.
The most common alternative to organizing teams by service boundaries is to organize teams around the business problems to be solved. That is a lot easier to budget for than trying to staff by microservice boundary, doesn't have the coordination and planning overhead, and it means you aren't reliant on up-front planning to get to a functional solution or design.
In high-uncertainty greenfield development, Explore projects or Lean Startup-style experimentation, having developer be close to the users they are serving is very efficient.
It also lets those companies reteam frequently, without needing to change the software to match the new team boundaries, which is very helpful when growing the team.
We joked about a microservice at a previous job in the same way - the frontend guys didn't want to include a dependency for generating a random ID so the microservice guy decided to build yet another microservice that did nothing but return random IDs.
> but a monolith with more than 50 developers working on it (no matter how you split your teams) isn't great either.
Why can the game industry etc somehow manage this fine, but the only place where it's actually possible to adapt this kind of artificial separation over the network, it's somehow impossible not do it beyond an even lower number of devs than for a large game? Suggests confirmation bias to me.
The main problem with microservices is that it's preemptive, split whatever you want when it makes sense after-the-fact, but to intentionally split everything up before-the-fact is madness.
How many of those game developers are actually art and asset developers?
How many times have AAA releases been total crap?
How many times have games been delayed by months or years?
How many times have games left off features like local LAN play, and instead implemented a 'microservice' as a service for online play?
How many times have the console manufactures said "Yea, actually you have the option of running a client server architecture with as many services you want?"
No. I'm saying there is no real correlation to the quality of microservices and the quality of monorepos in games and the amount of work required to build each one as a quality software object.
Comparing a game to almost any other piece of software, especially web based software, is how you end up with broken abstractions and bad analogies.
The point is that it's clearly not a major issue to have large teams working on the same monolithic codebase, the problems are just solved differently or are just vastly overstated in the first place.
> How many times have AAA releases been total crap?
> How many times have games been delayed by months or years?
What are we arguing here? Because I can think of many microservice apps that are crap as well, and have no velocity in development.
> How many times have games left off features like local LAN play, and instead implemented a 'microservice' as a service for online play?
This is entirely irrelevant. We're talking about the trade-offs of separating networked services that could otherwise be one unit. You're saying "why do games have servers then" which is a befuddling question with an obvious answer.
That's like saying my web server is a Microservice because it's not run in my clients browser. It makes no sense.
Note that the game's industry uses the term 'developer' differently. If a game has X developers, the vast majority of those people are not programmers. Engines also do a lot to empower non-programmers to implement logic in video games, taking lots of the workload off of programmers.
Maybe look up the game credits, I sometimes do and I often see like 10 UI programmers (in games with a ton to 2d UI), 5 gameplay programmers, 5 environment scripting, etc. Sure it is not small amount of people, but it is not an army.
Also those programmers seem to be neatly segregated in different areas of the project which I imagine work similarly to boundaries between the microservices at keeping the logic isolated between teams.
: I do AM surprised just how much QA people are credited in games, QA for major games sure do feel like an army.
I work on a monolith with ~1500 developers and it works pretty great.
The secret is that you're able to break a monolith apart, just like you can with microservices. You have APIs and modules of the monolith are responsible for their own thing. APIs are your contracts, just like in a microservice architecture.
The difference is that you can check if APIs are broken at compile time. In addition, you can view the API right in your IDE. In addition, your API isn't returning wishy-washy json with a half-assed OpenAPI spec - it's returning real types in a full-featured type system. And, cherry on top - you don't have to communicate over the network. Oh my god, you don't realize how many bugs and thousands of hours are wasted just working around that until you no longer have to. It's an immediate productivity boost.
But the best part is probably deployments. It's just so, so much more straightforward with one codebase.
I mean, "GREAT" until you need to do any kind of refactoring, or the company grows, or shrinks, or reorgs, or you have a feature that needs to change more than one service.
The "one team per microservice" makes code-enclosure style code ownership possible, but it is the least efficient way I have ever seen software written.
I've long wanted to hack an IDE so people are only allowed to change the Java objects they created, and then put six Java programmers in a room and make them write an application, yelling back and forth across the room. "CAN YOU ADD A VERSION OF THAT METHOD THAT ACCEPTS THIS NEW CLASS?" "SURE THING! TRY THAT?"
People discount the costs of microservices because they makes management's job easier, especially when companies have adopted global promotion processes. But unless they are solving a real technical constriant, they are a shitty way to work as an engineer.
As long as that team built those microservices to solve whatever problem they're responsible for solving I think it's better to let the problem domain dictate how many you need. Better to have seams that make sense in terms of the surrounding code than to have them in arbitrary places based on the org chart.
The trouble comes when some political wind blows and reshuffles the org chart, and now you're responsible for some services that only made sense in the context of a political reality that no longer exists.
I’m guessing you’re thinking of a certain kind of application (web apps perhaps?), where a monolith can make sense. But that’s but the only kind of application.
We have dozens of service components that are all largely independent of each other - combining them together would be purely a packaging decision, and wouldn’t really simplify much. In some cases, it wouldn’t make sense or even be possible at all.
An example is our execution agent, which executes customer workflows - that’s completely independent both conceptually and from a security perspective. Each agent instance executes a single flow at a time, for resource consumption and security reasons, which entails an ecosystem of services to manage that - messaging, data ingestion at scale (100K flows per day, multi-petabyte “hot” datastore for active data), orchestration, and other supporting services such as data access and network routing.
All of our teams support multiple services, and many of them qualify as microservices.
I suspect a lot of the issues teams encounter with microservices stem from a lack of cohesive understanding of microservices.
If people on the team continue to think about the "system" as a monolith (what they already know and are comfortable with), you'll hit friction ever step of the way from design all the way out to deployment. Microservices throw out a lot of traditional assumptions and designs, which can be hard for people to subscribe to.
I think there has to be adequate "buy-in" throughout the org for it to be successful. Turning an existing mono into microservices is very likely to meet lots of internal resistance as people have varying levels of being "with it", so-to-speak.
One place I worked at got sold on microservices by Thoughtworks, along with a change to Java as the main language to be used.
As one would expect, they made bank from their consulting endeavor and rode off into the sunset while the rest of us wasted several years of our careers rewriting ugly but functional monolithic code into distributed Java based microservices. We could have been working on features and product but essentially were justifying a grift, adding new and novel bugs as we rebuilt stable APIs from scratch.
The company went under not long after the project was abandoned. Nobody, of course, would be held to account for it. I will no longer touch a tech consultancy like TW with a 10 foot barge pole.
> 2 years later the rebuild of the old codebase was done.
>
> 3 years later and they are still fighting delivery and other issues they would never have had if they didn't ignore me and just went for the "lame" monolith.
I had a similar experience setting up the Infra for an 8-12 microservice application. The project had been dragging and no one really understood what they were doing. When I started asking scale questions, the answer came back that this was an internal admin UI for 9-5 business that would have 5-10 users.
I always tell people if they can’t handle writing decent libraries they also won’t handle microservices. Especially when a 3 person team cranks out 15 microservices, ideally with different languages.
I've seen microservices get introduced at companies... it never solved a real problem, it was more to scratch a developer's itch, or cargo cult ideas. It started to fall apart when they tried to figure out how to get an order service to fetch the prices of a product from the product pricing service, only to realise they need to hold onto the product price at the time of placing the order (it was a high volume / short product life cycle type of e-commerce), so uhh.. maybe we should duplicate this product into the order service? And then it would need to end up at a payment or invoicing service, more data duplication. And everything had to go through a central message bus to avoid web-like sprawl.
The other one was a microservice architecture in front of the real problem, a Java backend service that hid the real real problem, one or more mainframes. But the consultants got to play in their microservices garden, which was mostly just a REST API in front of a Postgres database that would store blobs of JSON. And of course these microservices would end up needing to talk to each other through REST/JSON.
I've filed this article in my "microservices beef" bookmarks folder if I ever end up in another company that tries to do microservices. Of course, that industry has since moved on to lambdas, which is microservices on steroids.
Problem is that, come recruiting time, interview gatekeepers are filtering out candidates who don't have the shiny words of the season, see micro services, unit tests, lots of abstractions, etc. It's like a dating app game. Everyone knows is overblown but they are still playing the game. The idea that not every company needs to make the same architectural and technological decisions is a concept way too complex for interview gatekeepers.
Are unit tests a shiny fad? Second time I've seen it mentioned in this thread. Is there some other type of testing I should be doing, or have I been doing it all wrong for the last two decades?
For unit testing to pay off, it requires having modular units to test.
Programmers coming up through frameworks or functional programming often don't have those, and so the techniques OO unit testers use don't translate well at all. If the first "unit" you build is a microservice, the first possible "unit" test is the isolation test for that service.
I have watched junior engineers crawl over glass to write tests for something because they didn't know how to write testable code yet, and then the tests they write often make refactoring a-la-Martin-Fowler's-book impossible.
(And that is leaving aside the consultancies that want to be able to advertise "100% test coverage!" but don't actually care if the tests make software harder to maintain in the long run because they aren't going to be there.)
Eventually we'll be able to acknowledge that there are a lot of different skills in our profession, and that writing good code isn't about being "smart": it's about knowing how to write code well. But until then people will keep blaming the tools they don't know how to use.
I am so mad that the mockists stole the word "unit test" for their thing. The original definition of a unit test was writing "integration" tests for each of the sub-components of a system.
(Mockist tests are fine for people who really want them, as long as you delete them before checking in the code.)
Back in the day when I used to make "webapps" - our integration tests were an epic faff.
Often times using things like cucumber to describe a set of interactions, in the days before headless chrome it'd drive selenium which would literally open Firefox on your desktop and start navigating the site.
It did it's job but the feedback loop was slow.
Today I mainly write libraries, cli tools and the occasional small HTTP API. I still have unit and integration tests but they are all just standard pytest functions. I can run each type individually, my definition of an "integration" test is either something that talks to the outside world in some capacity, or something that is really slow (mostly running an ml model these days).
I much prefer today's approach, but admittedly I work on things with far fewer points of interaction which I think greatly simplifies the work.
Has anyone tried something like Polylith which lets you build all your code like normal functions for local dev and testing and then seemlessly pull out parts to network services as needed?
Totally agree. Micro services unnecessarily makes thing complicated for small teams. IMHO it solves the problem of velocity ONLY when a large engineering team is slowed down due to too much release & cross cutting dependencies on a monolith. Although I see people solving with modular monoliths, merge queues and CODEOWNERS effectively.
Few cases where microservices makes sense probably when we have a small and well bounded use-case like webhooks management, notifications or may be read scaling on some master dataset
They're a way to assign ownership to individuals or teams on a granular basis, rather than at the repo-level. You can assign entire folders or individual files to people.
Microservices are the software architecture analog to Conway's Law. You can't help but introduce some sort of significant architecture boundary at the boundary between teams, and while that doesn't have to be "microservices" that's certainly a very attractive option. But on the flip side, introducing those heavier-weight boundaries on to yourself, internal to a team, can be very counterproductive.
I can't prove this scales up forever but I've been very happy with making sure that things are carefully abstracted out with dependency injection for anything that makes sense for it to be dependency-injected, and using module boundaries internally to a system as something very analogous to microservices, except that it doesn't go over a network. This goes especially well with using actors, even in a non-actor-focused language, because actors almost automatically have that clean boundary between them and the rest of the world, traversed by a clean concept of messages. This is sometimes called the Modular Monolith.
Done properly, should you later realize something needs to be a microservice, you get clean borders to cut along and clean places to deal with the consquences of turning it into a network service. It isn't perfect but it's a rather nice cost/benefit tradeoff. I've cut out, oh, 3 or 4 microservices out of monoliths in the past 5 years or so. It's not something I do everyday, and I'm not optimizing my modular monoliths for that purpose... I do modular monoliths because it is also just a good design methodology... but it is a nice bonus to harvest sometimes. It's one of the rare times when someone comes and quite reasonably expects that extracting something into a shared service will be months and you can be like "would you like a functioning prototype of it next week"?
Conway's law is about communication, not team boundaries. There is no requirement that we introduce a significant architectural boundary at the boundary between teams: companies choose to do so to avoid having cross-team communication.
The only way for significant architectural boundaries at team boundaries to not result in incredibly painful software, especially for a growing team, is to let the software organize the teams. Which means reorging the company whenever you need to refactor, and somehow guessing right about how many changes each component will need in the coming year.
It also means you can't have product and engineers explore a problem together, or manage by objective with OKRs since engineers aren't connected to business outcomes.
I know that all the ex-Amazonians are convinced this is the only way to build software, but it really, really isn't.
"Conway's law is about communication, not team boundaries."
I'm a spirit of the law sort of person, not the letter of the law. I don't care how you draw your internal organizational diagram; communication barriers are your team barriers.
It's for management to read the flow over time and keep the de jure boundaries somewhat sync'd with the de facto boundaries, but in the meantime the de facto ones are what will get written into your software.
My friend briefly worked at a company where every API was a lambda. Each lambda had a git repo. Lambdas would often call into other lambdas. In order to make a feature, it might involve touching 10+ lambdas. They had over 200 lambdas after a year. Total nightmare
I think the major issue that I see, and I could be wrong is that if you want to change some underlying functionality between you and a dependent function to do that you would need to change all the intermediate functions only so that you could call that dependent function and layers deep.
I have played around with architectures like this, but I allowed the caller to patch in a dependent function in the call with those function overlay overrides were passed from function to function.
Every service boundary you have to cross is a point of friction and a potential source of bugs and issues so by having more microservices you just have more than go wrong, by definition.
A service needs to maintain an interface for compatibility reasons. Each microservice needs to do that and do integration testing with every service they interact with. If you can't deploy a microservice without also updating all its dependencies then you don't have an independent service at all. You just have a more complicated deployment with more bugs.
The real problem you're trying to solve is deployment. If a given service takes 10 minutes to restart, then you have a problem. Ideally that should be seconds. But more ideally, you should be able to drain traffic from it then replace it however long it takes and then slowly roll it out checking for canary changes. Even more ideally, this should be largely automated.
Another factor: build times. If a service takes an hour to compile, that's going to be a huge impediment to development speed. What you need is a build system that caches hermetic artifacts so this rarely happens.
With all that above, you end up with what Google has: distributed builds, automated deployment and large, monolithic services.
In 2016-17, I was involved with a rather large mictoservice-heavy rewrite project tha didn't go particularly well. The main reason was that microservices were actually a good fit for the _planned_ organisational structure, but not for the one that was eventually put in place. When you go from 4 vertically integrated independent teams to 2 backend devs, 2 frontend devs, and 1 "devops" without stopping 5 minutes to rethink the architecture, of course shit will happen.
You only need microservices for massive scale or to enable micromanagement of teams, but that doesn't mean you have to give up on clear module boundaries.
You can get the architectural benefits of microservices by using message-passing-style Object-Oriented programming. It requires the discipline not to reach directly into the database, but assuming you just Don't Do That a well-encapsulated "object" is a microservice that runs in the same virtual machine as the other mircoservices.
Java is the most mainstream language that supports that: whenever you find yourself reaching for a microservice, instead create a module, namespace the database tables, and then expose only the smallest possible public interface to other modules. You can test them in isolation, monitor the connections between them, and bonus: it is trivial to deploy changes across multiple "services" at the same time.
Years ago I attended a local meetup where the CTO of a local startup gave a presentation on their, mostly successful, microservice rollout.
In the Q&A after ward, another local startup CTO asked about problems their company was having with their microservices.
The successful CTO asked two questions: "How big is your microservices tooling team?" and "How big is your Dev Ops Team?"
His point was, if you're development team is not big enough to afford dedicated teams to tooling and dev ops, it's not big enough to afford microservices.
Was in org with 10 people devops dedicated team, it was smooth, also as a dev could push requests for their repos... but also only 3 devops and they were so busy my requirement for basic stuff was burried in backlog. You can develop but still need to maintain from the to time.
The biggest wins for microservices aren't really technical, they're organizational. They force you to break a problem down and allow each team to own a piece of it, including end to end delivery. This allows specialization of labor which is a key driver of productivity - including an ability to experiment and innovate. Every change is incremental by default, and well-documented external APIs are the only way to talk to other domains- no shared databases, filesystems, or internal APIs. It's not free and definitely takes some discipline and tooling to enforce shared standards (every service should have metrics, logging, tracing, discovery, testing, CI/CD, etc) but you'd need to build that muscle with a monolith as well.
Yeah, I keep telling people at work that we need to figure out how to make it easier for teams to manage their own DBs. There are so many teams trying to shove their data into some other team's DB.
It sounds like you're currently struggling against provisioning new database instances (or clusters) when the delineation is actually at the database within that instance
Compare:
psql --host=team1.example -d team1
# versus
psql --host=the-db.example -d team1
Sometimes one can get away with schema level split, which can allow cross schema references in a read only setup, but carries coupling concerns
psql --host=the-db.example -d my_db -U team1 "select * from team1.orders o inner join team2.products p ..."
I don't mean to troubleshoot over HN comments, as the problem space is filled with nuance and trade-offs, but intended it as "for your consideration"
Thanks, but it's something else. When I say people shove their data into other teams' DBs, I really mean into other teams' schemas... and they don't even have direct SQL access to their data anymore, rather the teams set up some CRUD API which they have to painfully maintain.
We aren't using Postgres but a more specialized DBMS. Everyone basically does the first thing in your example, sharing one cluster. The real issue is it's a big corp with red tape around stuff, people don't want responsibilities like compliance, and people are rightfully scared of keeping things performant on a fancy DBMS. Our team isn't scared, we did it, but it wasn't easy either.
To clarify, team A has some data in a DB, team B comes along asking to store their own data in there too, team A sets up CRUD API through which team B writes their stuff into team A's DB then gets it back.
Could kept infra as a code, logging, auth and so on in packages, gRPC or message queues for communication, telemetry, monitoring/alerts and more stuff as a code too... got to the point creating new service was just new repo, name, port a resource utilization.
Agree with organizational win, also smaller merge requests in the team were superb.
Around 5-10 devs, monolith, we ran into conflicts more often, deployment, bigger merge requests, releasing by feature was problematic, microservices made team more productive, but rules about tests/docs/endpoints/code were important.
The problem is the "micro" part. Service oriented architecture is generally the way to go, but the service boundaries should be defined by engineering constraints, not as arbitrarily small.
Where I work, they consider a service managed full-time by a team of 2-8 people a "microservice." Before that, they had a monolith shared by a dept of ~120.
Micro services show their benefits in a large organization.
It’s a tool to solve people issues. They can remove bureaucratic hurdles and allow devs to somewhat be autonomous again.
In a small startup, you really don’t gain much from them. Unless if the domain really necessitates them, eg. the company uses Elixir but all of the AI toolings are written in Python/Go.
If your application has different load or resource requirements, you should build separate services, even in a startup.
You can put most of your crud and domain logic in a monolith, but if you have a GPU workload or something that has very different requirements - that should be its own thing. That pattern shouldn't result in 100 services to maintain, but probably only a few boundaries.
Bias for monolith for everything, but know when you need to carve something out as its own.
I was starting from the assumption that Elixir does something they need, but yeah in most cases Golang would cover the same thing. Even then, you probably have separate Golang and Python, or just two separate Python services.
microservices can also cause organizational dependencies and coordination that wouldn't otherwise be necessary. i've seen it create at least as many people issues as solve them. one seemingly innocuous example is the policy of 'everybody just uses whatever services they want', which can hugely increase the ongoing maintenance requirements and seems to require that everyone learn everything in order to be functional. which never happens, which means you're always chasing people down.
I probably just haven't checked these comment threads enough yet because I'm surprised I haven't seen this posted, but even though this is a bit old now, https://youtu.be/y8OnoxKotPQ, there is a reason it resonated with so many. It's spot on with the downsides microservices can inflict.
I've certainly seen microservices be a total disaster in large (and small) organizations. I think it's especially important that larger organizations have standards around cross-cutting concerns (e.g. authorization, logging, service-to-service communication, etc.) before they just should "OK, microservices, and go!"
microservices are a gigantic waste of time. like TDD.
it takes skill and taste to use only enough of each. unfortunately a lot of VC $$$ has been spent by cloud companies and a whole generation or two of devs are permasoiled by the micro$ervice bug.
don't do it gents. monolith, until you literally cannot go further, then potentially, maybe, reluctantly, spin out a separate service to relieve some pressure.
Ha! I always feel more than a little embarrassed when it happens, but I can't sit idly by while TDD is slandered, especially from so seemingly oblique an angle!
While I agree with you regarding microservices (eg language abstractions provide 80% of the encapsulation SOA provides for 20% of the overhead) and I readily acknowledge that 100% test coverage is a quixotic fantasy, I really can't imagine writing reliable software without debuggers, print-statements, or a REPL—all of which TDD replaces in my workflow.
How, I wonder, do you observe the behavior of the program if not through tests? By playing with it? Manually reproducing state? Or, do you simply wait until after the program is written to test its functionality?
I wonder what mental faculties I lack that facilitate your TDD-less approach. Can it be learned?
Write the program, run the program, observe. Or, write the program, write the tests, run the program, observe for breakages. Neither are TDD, the practice of writing tests first before any actual implementation is written, which I agree with the parent is dogma, impractical, and unrealistic.
I can't imagine a small team following ALL the rules of microservices benefiting much at all. It makes no sense.
For large orgs where each service has a dedicated team it starts to make sense... but then it becomes clear that microservices are an organizational solution.
There was a point in time (circa 2019-2020) when the madness got so severe that every new feature ended up as a microservice backed by a DB with a single table (plus a couple tables for API keys, migration tracking, etc.)
I love it when all my CRUD has to be abstracted over HTTP. /s
271 comments
[ 1364 ms ] story [ 4802 ms ] threadBasically this. Microservices are a design pattern for organisations as opposed to technology. Sounds wrong but the technology change should follow the organisational breakout into multiple teams delivering separate products or features. And this isn't a first step. You'll have a monolith, it might break out into frontend, backend and a separate service for async background jobs e.g pdf creation is often a background task because of how long it takes to produce. Anyway after that you might end up with more services and then you have this sprawl of things where you start to think about standardisation, architecture patterns, etc. Before that it's a death sentence and if your business survives I'd argue it didn't because of microservices but inspite of them. The dev time lost in the beginning, say sub 200 engineers is significant.
Thankfully the business and finance people at the company decided it would be an expensive and stupid idea it was shut down.
Things are different in the embedded space so I don't have personal experience with any of it.
Any organization stuck in microservice hell fully deserves the punishment.
Should the URLs contain a version; like /api/v1/ ?
FWIU OpenAPI API schema enable e.g. MCP service discovery, but not multi-API workflows or orchestrations.
(Edit: "The Arazzo Specification - A Tapestry for Deterministic API Workflows" by OpenAPI; src: https://github.com/OAI/Arazzo-Specification .. spec: https://spec.openapis.org/arazzo/latest.html (TIL by using this comment as a prompt))
Majority maybe, but not everyone. I was at a place in 2018 where a guy turned up declaring that microservices would solve all of the company's performance issues (he'd heard that AWS S3 was made up of over 500 microservices so we must do the same, but he obviously had no idea about the depth of features in S3 so hit take was more is better).
So we got like 30 microservices in the space of year which gave the company 10x complexity, complicated and fragile local development and... dubious wins like faster email sending, but no improvements to true bottle-necks of the system.
It would also nice to have less fear-driven career advice like "your skills go out of date" which drives people to try adopting the latest things.
Also, you can pick and choose what to use in Django similarly to Flask - it just has a higher initial learning curve.
Once you get to sufficient levels of complicated, leaning on established, documented, community supported design patterns and abstractions helps vs. sorting out your imports, making bespoke design choices, and doing a bunch of non-core value producing work.
And when you break these out, you don't actually have to split your code at all. You can deploy your normal monolith with a flag telling it what role to play. The background worker can still run a webserver since it's useful for healthchecks and metrics and the loadbalancer will decide what "roles" get real traffic.
If you are concerned about someone else breaking your thing, good! You were going to eventually break it yourself. Write whatever testing gives you confidence that someone else's changes won't break your code, and, bonus, now you can make changes without breaking your code.
The post I was replying to said "your build will still break": that's what I was taking issue with. In this day and age there is no reason our trunk build should ever be broken.
One of the big tenets of independent services is that your APIs are contracts that don't change behaviour. As long as each individual service doesn't introduce breaking changes, the system as a whole should work as expected. If it doesn't this is indicative of either 1) a specific service lacking test coverage, or 2) doing something wrong i.e. directly reading from a microservices' database without going through an API.
How is that any different from an API in a monolith not changing behavior?
I agree that you can prevent merges that cause compilation errors in nearly all cases!
A very large code base full of loosely related functionality makes it more and more likely a change in one part will break another part in unexpected ways.
This assertion is unrealistic and fails to address the problem. The fact that builds can and do break is a very mundane fact of life. There are whole job classes dedicated to mitigate the problems caused by broken builds, and here you are accusing others of doing things wrong. You cannot hide this away by trying to shame software developers for doing things that software developers do.
> Write whatever testing gives you confidence that someone else's changes won't break your code, and, bonus, now you can make changes without breaking your code.
That observation is so naive that casts doubt on whether you have any professional experience developing software. There are a myriad of ways any commit can break something that goes well beyond whether it compiles or not. Why do you think that companies, including FANGs, still hire small armies of QAs to manually verify if things still work once deployed? Is everyone around you doing things wrong, and you're the only beacon of hope? Unreal.
I am genuinely curious what situations you are seeing where builds are making it through CI and then don't compile.
It isn't always worth investing in quality, but when it is it is entirely possible to write essentially bug-free software. I've gone seven months without a bug in production and the one we saw we had a signed letter from product saying "I am okay if this feature breaks, because I think writing the tests that can verify this integration was going to take too long."
FAANG companies aren't prioritizing writing software well: they are prioritizing managing 50,000 engineers. Which is a much harder problem, but the management solutions that work for that preclude the techniques that let us write bug-free software.
One of the great things about startups is that it is trivial to manage five engineers, so there is no reason we have to write software badly.
Of course, if people wrote bug-free code, then there would be no bug !
Bug-free code in the actual code, or bug-free code in the test code, this is the same story.
If you write stuff and never have any bug, then either:
The more complicated, intricate stuff you have, the more bugs you'll get (and only time will allow you to fix that).Tests are great do define how you think it should work, and to ensure it keeps doing that way. Take the time to think about the third point on the bullet list above.
In the DVCS era, we have inexpensive branching. Do as thou wilt on your topic or epic branches. Rebase them against main/master before merging upwards. Fix what must be fixed first.
Main/master branch should never fail CI. If it does, there is something seriously wrong with your branch lifecycle and/or deployment process.
A reliable deployment pipeline is outside of CI, but can be kept straightforward and minimal to constrain the scope of failures.
Bugs happen, and systems complexify. It is possible to manage both of those risks down to near-zero by the time code reaches production release candidate stage.
In some industries, this is more important than others, though -- obviously the goal is to match quality to business needs.
But I agree with our thread predecessor: I haven't seen a broken build make it anywhere near production in many years, and it's not because of the snarky dismissal that provoked my original response.
[0] In some situations, proper tests are not possible, and in others they are not practical. And I acknowledge that I'm omitting things like visual design/layout bugs, which is probably not reasonable.
And no, I'm not Jesus, I just care a lot about quality and have spent the last 20 years finding ways and strategies to improve it.
Reducing the number of bugs does not mean being a god that writes bug-free code on the first draft. It means being able to detect and fix issues as early as possible. In my case I aim to always do that before letting myself push any code to git.
IMO, it only comes down to how much someone really cares about the quality, but here are some examples of what can be done and is very effective:
- Plan ahead your functional and technical design
- Carefully research existing code to confirm the feasibility of the design
- Use a statically typed language
- Use advanced static-analysis tools
- Avoid magic, write explicit code. Especially avoid runtime checks such as reflection. Ideally, everything should be checked statically one way or another.
- Never let a code path/branch/corner case be unhandled, however unlikely it is (and go back to step one to refine the design if a code path has been forgotten in the current design)
- Always have automated testing. The bare minimum is to unit-test all business logic, including all possible code paths. Ideally e2e tests are nice, but not always a good investment. Tests must be 100% independent and never depend on an external environment, otherwise it's going to be flaky at some point.
- Always manually test every feature and path related to my changes (especially don't skip testing the ones that I think are going to be ok) before pushing anything to git.
- Warnings and "optional" notices are unacceptable and must always be fixed (or disabled), otherwise the list will just keep growing, which reduces the visibility of any issue and normalizes having problems.
- Have a CI integration that applies all the automated checks mentioned in this list and make everything mandatory.
Each one of those actions does on it's own significantly reduce the number of bugs. If you combine them all, you can effectively reduce the number of bugs to pretty-much zero. And since the earlier you find a bug, the cheaper is it overall to fix, I've also found-out that in terms of productivity it's always worth the investment (despite many people pretending the opposite).
You lost what little credibility you had left.
But even then there are ways around this kind of problem with dynamic linking pre-built binaries and caching, but it is extra complexity that could be worse than managing multiple services. Docker cache can usually handle this pretty well though.
You'll need services. They're hard. If something is hard but it needs to be done, you should get good at it.
Like every fad, there a backlash from people seeing the fad fall apart when used poorly.
Services are a good pattern with trade offs. Weigh the trade offs, just don't do things to do them.
This is a great optimization once you have high traffic services
Building this way before you have any traffic at all is a great way to build the wrong abstractions because your assumptions about where your load will be might be wrong
> Building this way before you have any traffic at all is a great way to build the wrong abstractions
These services only make sense to think about within specific traffic contexts. It'd be impossible to build the right abstraction.
Here’s the kicker: They only had a few hundred MAUs. Not hundreds of thousands. Hundreds of users. So all this complexity was for nothing. They burned through $50M in VC money then went under. It’s a shame because their core product was very innovative and well architected, but it didn’t matter.
Way too many companies believe they're really just temporarily embarrassed BigTech.
I've seen Information Systems programs, that are usually CE, after-hours tracks. Neither Harvard, Yale, nor MIT have a software dev one, just Comp. Sci. I'm calling BS (no pun intended) on "software dev degrees" as a thing distinct from CS in any widespread fashion.
https://www.harvard.edu/programs/
https://admissions.yale.edu/majors-and-academic-programs
https://facts.mit.edu/degrees-majors/
Definitely some stuff makes sense (especially since it has a lot of IoT stuff), but micro-service added was used mostly as a way to develop new stuff without having to deal with the legacy monolith. The core of the application could easily be a single service backed by one big RDBMS with a few ancillary services around it.
The legacy monolith is still there kicking and screaming, it didn't need "breaking up" it needed (and I assume still do) need a major refactoring.
https://blog.khanacademy.org/go-services-one-goliath-project...
They had already scaled the mono service about as far as it could go and had a good sense of what the service boundaries should be based on experience.
I don't see any major epiphany in this. In fact, it reads like a tautology. The very definition of microservice is that it's an independently evolving domain. That's a basic requirement.
Sounds right, no? Service is what people provide, implied in the scope of a macro economy. Microservice then implies the same type of service, but within the micro economy of a single business.
Using Spring or Guava in the Java world it is frequent that people write "services" that are simply objects that implement some interface which are injected by the framework. In a case like that you can imagine a service could have either an in-process implementation or an out-of-process implementation (e.g. via a web endpoint or some RPC.) Frameworks like that normally are thinking at the level of "let's initialize one application in one address space at a time" but it would be nice to see something oriented towards managing applications that live in various address spaces.
Trouble is that some people get this squee when they hear they can use JDK 9 for this project and JDK 10 for another project and JDK 11 for another project and they'd rather die than eschew the badly broken Python 3.5 for something better. If you standardized absolutely everything I think you could be highly productive with microservices because you wouldn't have to face gear switching or deal with teams who just don't know that XML serialization worked completely differently in JDK 7 vs JDK 8 thus the services they make don't quite communicate properly, etc.
Very true in my experience. The main benefit is letting small groups of people work independently without stepping on each other’s toes. Although I’ve worked on a project where multiple teams owned micro services that were supposed to be standardized with each other, and it just lead to endless meetings and requirements churn since nobody was willing to work on the other teams service but everyone had an opinion on what the cross-team standard should be. Learning the diplomatic way to say “mind your own business” was more important than any technical skills for getting code merged.
It's not wrong at all, literally Conway's law: https://en.wikipedia.org/wiki/Conway's_law
I agree, but just saying "multiple teams" has led many eng directors to think "I have two squads now --> omg they cannot both be in the same monolith".
When both squads are 5 people each.
And the squads re-org (or "right size") every 9 months to re-prioritize on the latest features.
Five years go by, 7 team/re-org changes, all of which made sense, but thank god we didn't microservice on the 2nd/3rd/4th/5th/6th team boundaries. :grimmacing:
We should stay "stable, long-lived teams" -- like you need to have a team that exists with the same ownership and mandate for ~18 months to prove its a stable entity worth forming your architecture around.
For example, we have a authentication microservice at work. It makes sense that it lives outside of the main application, because its used in a multiple different contexts and the service boundary allows for it to be more responsive to changes, upgrades and security fixes than having it be part of the main app, and it deploys differently than the application. It also adds enough intentional friction that we don't accidentally put logic where it doesn't belong as part of the user authentication process. It has helped keep the code focused on only primary concerns.
That said, you can't apply any of these patterns blindly, as is so often the case. A good technical leader should push back when the benefits don't actually exist. The real issue is lack of experience making technical decisions on merits.
This includes high level executive leaders in the organization. At a startup especially, they are still often involved in many technical decisions. You'd be surprised (well maybe not!) how the highest leadership in a company at a startup will mandate things like using microservices and refuse to listen to anything running counter to such things.
[0]: https://en.wikipedia.org/wiki/Rule_of_thumb
> It also adds enough intentional friction that we don't accidentally put logic where it doesn't belong as part of the user authentication process.
Preventing misplaced logic is a matter of good code structure, well defined software development processes and team discipline - not something that requires splitting into a separate microservice, and definitely not something that you want to solve on system architecture level.
Perhaps yes. Every situation should be evaluated on merits. This came across that there is also an assumption that we didn't try other solutions first - we absolutely did. Microservice is the best solution to solving the problems we needed solved in this case. Even better than modular monolith with clear interfaces.
>without the cognitive and operational tax microservices impose
When done correctly, I don't think there is a tax. Most operational questions should be automated away once discovered. The only 'tax' is that it lives separately from the larger application and is deployed independently, but I haven't seen in practice this add any notable overhead.
>Preventing misplaced logic is a matter of good code structure, well defined software development processes and team discipline
All true, and a microservice can aid all of these things too, but isn't the solution you should reach for when solving for these things and these things alone in my opinion. That said, myself and others have observed there is time saved on enforcing discipline around this issue once we separated the code away from the main application. I can't deny that hasn't been a good thing, because it has. It would be leaving information out about the benefits we'v experienced, and I see no reason to do that.
All told, completely dismissing the value of microservices as a potential solution is no different than completely dismissing other solutions in favor of microservices. Things have their place, there are pros and cons to them, and should be evaluated relative to their merit for the situation.
You may find you never implement microservices, or implement very few, or perhaps the needs of an organization is as such that its a pattern used most of the time, but the technical merits of doing so - with any decision of this nature, not limited to microservices - should have a backing justification that includes why other solutions don't fit
I completely agree. But this a little bit contradicts with your original comment that caught my eye:
> In my experience, a good rule of thumb[0] is if there are actual benefits from being a standalone service.
A rule of thumb is, by nature, a generalization — it simplifies decision making through heuristics. Benefits on the other hand always subjective, they can be interpreted in a given context.
And based on my experience, there will always be some benefits that can be used to justify factoring something out into a separate service. The challenge is that it's often easy to overemphasize those benefits, even when they don't outweigh the downsides. Your example with the auth service and the added friction is, in my view, a good illustration of a justification that might sound reasonable but can lead to unnecessary complexity. (Just to be clear, my intent here isn't to judge your decisions - I understand these trade-offs are often nuanced - and that's why again there is no good rule of thumb for this)
Its a good rule of thumb that you may want to evaluate if a microservice is appropriate. Thats the point in context[0]. If you think it might be a relevant solution, that is a pretty good heuristic that evaluating it may be worth the time.
What I seem to be coming up against when discussing this is people conflate worthwhile evaluation with worthwhile solution. Thats where the nuance and details need to live is in solution discovery, but how do you arrive at the right solution if you don't first evaluate what solutions might fit a given problem?
I stand by it being a good rule of thumb. If you think there are actual benefits - not perceived, but quantifiable benefits - of something being a standalone independent service, you might have a case for a microservice. I feel its a good heuristic for narrowing solution evaluation.
It doesn't equate to saying microservice is the solution if it meets that one criteria, only don't rule it out.
>And based on my experience, there will always be some benefits that can be used to justify factoring something out into a separate service. The challenge is that it's often easy to overemphasize those benefits, even when they don't outweigh the downsides. Your example with the auth service and the added friction is, in my view, a good illustration of a justification that might sound reasonable but can lead to unnecessary complexity.
Bias is hard to overcome. Technical decision makers need to be keenly aware of this. I wish it was easier to identify when this comes into the situation as it plays a bigger role in all this than is often realized. I go to great lengths to validate my thoughts when making big technical decisions for this reason, and deciding on something like this is a big technical decision that deserves that approach, in my opinion.
One good quality of a good technical decision maker is taking the time to quantify why its a good solution and why other solutions are not, and that must hold up to technical scrutiny of your peers. This challenges any assumptions missed. Ideally, your peers should be able to see through anything that wasn't given enough evaluation, like if the benefits of doing something is overemphasized.
>good illustration of a justification that might sound reasonable but can lead to unnecessary complexity.
I agree, which is why I state elsewhere that it shouldn't be used solely for this purpose, but I'm not going to leave out a real benefit we saw. If there is a persistent organizational problem that engineers seem to want to put code inside of something that is in broader context inappropriate or adds complexity where it shouldn't, you may benefit from such friction and its okay to evaluate that aspect too.
[0]: By which I mean in the context of the article, which I think dismisses microservices as a potential solution with prejudice
> What I seem to be coming up against when discussing this is people conflate worthwhile evaluation with worthwhile solution.
> It doesn't equate to saying microservice is the solution if it meets that one criteria, only don't rule it out.
I read the article, it's not against microservice architecture itself, but points out that you shouldn't treat them as a starting point or a best practice.
There is even a section named "When Microservices Do Make Sense". It also highlights examples where microservices make sense ("Their post is a good example of how microservices can work when you have the organizational maturity and operational overhead to support them."), so I really do not see where does anyone rule out, dismiss, come up against anything.
I somewhat understand and agree what you are trying to argue against, but that's definitely not in this article.
> Its a good rule of thumb that you may want to evaluate if a microservice is appropriate.
This is too general to be practically useful — it applies to almost everything in life.
That's a "large" organization problem. But large is actually, not that big (about 5-10 scrum teams before this is a very large problem).
It also means on critical systems separating high risk and low risk changes are not possible.
Like all engineering decisions, this is a set of tradeoffs.
What lifecycle are we really talking about? There are massive monoliths - like the Linux kernel or PostgreSQL - with long lifespans, clear modularity, and thousands of contributors, all without microservices. Lifecycle management is achievable with good architecture, not necessarily with service boundaries.
> If you update the logging library in a monolith, everyone takes that updates even if it breaks half the teams.
This is a vague argument. In a microservice architecture, if multiple systems rely on the structure or semantics of logs — or any shared behavior or state - updating one service without coordination can just as easily break integrations. It’s not the architecture that protects you from this, but communication, discipline, and tests.
> It also means on critical systems separating high risk and low risk changes are not possible.
Risk can be isolated within a monolith through careful modular design, feature flags, interface boundaries, and staged rollouts. Microservices don’t eliminate risk - they often just move it across a network boundary, where failures can be harder to trace and debug.
I’m not against microservices. But the examples given in the comment I responded to reflect the wrong reasons (at least based on what I’ve seen in 15+ years across various workplaces) for choosing or avoiding a microservice architecture.
Microservices don’t solve coupling or modularity issues — they just move communication from in-process calls to network calls. If a system is poorly structured as a monolith, it will likely be a mess as microservices too — just a slower, harder-to-debug one.
> grug wonder why big brain take hardest problem, factoring system correctly, and introduce network call too
> seem very confusing to grug
0: https://grugbrain.dev/#grug-on-microservices
Make small changes in the monolith a time, though.
Do cloud/paas providers deeply support this flow anymore? Every dashboard would need to compare across multiple live versions and I haven't tried that in a while.
If you're instead doing this with feature branches or something like that, then by definition you don't have CI. You have NI: Never Integration.
Because, to an approximation, there's never any point in time where all of the code you're working on is integrated together so that everyone has a chance to see how what they're doing interacts with what everyone else is doing. And yes, it is possible for a branch to successfully auto-merge and produce something that compiles and passes all automated tests, and still introduce a horrible regression defect because of an unanticipated interaction between two different changes on two different feature branches. I don't see it happen often, but when it does it usually creates such a big production SNAFU that even once every 5 years is still way too often for my taste.
If you're looking for performance regressions as you roll out feature flags, do you get any kind of support for that from built in dashboards? How do you get clean metrics if you only have one version and a bunch of teams turning knobs?
SOA/Microservices have a lot of headaches but you do get the option for very tightly scoped deployments or feature rollouts.
But when I see a plan to use it that doesn't include a plan for how to stop using it again ASAP, I get very worried.
The static typing makes even less sense at finer code scopes, like I don't need to keep asserting that a for-loop counter is an int.
Statically-types languages are a form of automatically-verified documentation, and an opportunity to name semantic properties different modules have in common. Both of those are great, but it is awkward that it is usually treated as an all-or-nothing matter.
Almost no language offers what I actually want: duck typing plus the ability to specify named interfaces for function inputs. Probably the closest I've found is Ruby with a linter to enforce RDoc comments on any public methods.
As for your comment on `any`, the reason why one doesn't want to fall back on such is that you throw out most of the gains of using static types with such a construct when your function likely doesn't work with `any` type (I've never seen a function that works on absolutely anything other than `id :: a -> a` and I argue there isn't one even with RTTI).
Instead you want to declare the subset of types valid for your function using some kind of discriminated union (in rust this is `enum`, zig `union(enum)`, haskell ADTs/GADTs, etc etc) where you set a static bound on the number of things it can be. You use the type system to model your actual problem instead of fighting against it or "lying" (by saying `any`) to the compiler.
The same applies to services, APIs, protocols, and similar. The more work the compiler can help you with staying on spec the less work you have to do later when you've shipped a P1-Critical bug by mistake and none of your tests caught it.
Indeed "any" breaks type checking all around it, but it can be contained more easily in a helper func with a simple return type. Most common case is your helper does a SQL query, and it's tedious and redundant to specify the type of rows returned when the SQL is already doing that.
For example `foo :: Semigroup a, Traversable t => t a -> a` I already know that whatever is passed to this function can be traversed and have it's sum computed. It's impossible to pass something which doesn't satisfy both of those constraints as this is a specification given at the type level which is checked at compile-time. The things that cannot be captured as part of a type (bounded by the effort of specification) are then left to be captured by tests which only need to handle the subset of things which the above doesn't capture (e.g `Semigroup` specifies that you can compute the sum but it doesn't prevent `n + n = n` from being the implementation of `+`, that must be captured by property tests).
Another example, suppose you're working with time:
From the above you can tell what each function should do without looking at the implementation and you can probably write tests for each. Here the interface guides you to handle time in a safer way and tells a story `event = add (hours 5) present` where you cannot mix the wrong type of data ``until event `is` zero``. This is actual code that I've used in a production environment as it saves the team from shooting themselves in the foot with passing a `Clock.Duration` where a `Clock.Present` or `Clock.Future` should have been. Without a static type system you'd likely end up with a mistake mixing those integers up and not having enough test coverage to capture it as the space you must test is much larger than when you've constrained it to a smaller set within the bounds of the backing integer of the above.In short, types are specifications, programs are proofs that the specification has a possible implementation, and tests ensure it behaves correctly for that the specification cannot constrain (or it would be too much effort to constrain it with types).
As for SQL, I'd rather say the issue is that the SQL schema is not encoded within your type system and thus when you perform a query the compiler cannot help you with inferring the type from the query. It's possible (in zig [1] at least) to derive the type of a prepared SQL query at compile-time so you write SQL as normal and zig checks that all types line up. It's not that types cannot do this, your tool just isn't expressive enough. F# [2] is capable of this through type providers where the database schema is imported making the type system aware of your SQL table layouts solving the "redundant specification" problem completely./
So with all of that, I assume (and do correct me if I'm wrong) that your view on what types can do is heavily influenced by typescript itself and you've yet to explore more expressive type systems (if so I do recommend trying Elm to see how you can work in an environment where `any` doesn't even exi...
To add to your point, I don't think foo can even be implemented (more accurately: is not total) because neither `Semigroup a` or `Traversable t` guarantee a way to get an `a`.
I think you'd need either `Monoid a` which has `mempty`, or `(Foldable1 t, Traversable t)` which guarantees that there's at least one `a` available.
This is true, but the difference is you don't have to write a compiler, it's already written for you. The testing, you have to write, and do so correctly.
A lot of the woes of statically typed languages can be mitigated with tooling. Don't want to repetitively create types from an OpenAPI spec? Generate the code. Don't want to create types from SQL records? Generate the code. Don't want to write types everywhere? Deduce them.
You get all the benefits of static typing, but none of the work. It's so advanced these days that lots of statically-typed languages look dynamically-typed when you see the code. But they're not, everything has a type if you hover over them. The type deduction is just that good.
The type deduction is not so automatic in most languages, TS included. Rust has the most automatic one I've seen, and of course that kind of language needs static typing. But still, it's more explicit than needed for a typical web backend.
SQL type autogen is limited to full rows, so any query returning an aggregate or something isn't going to work with that. Even for full rows, it's eh. Usually I just see that encouraging people to do local computations that should be in SQL.
The argument isn't that with static types you don't need tests but rather with static types you can focus on testing what actually matters which is the fuction itself along with it's integration into the program. Tests won't cover 100% of the surface area of a function taking `any` as that is by definition infinite with the constraint on the shape specified in the body of the function dependent on the path taken at runtime (it might never be hit). I urge you to take a look at languages other than TS for this as TS is in the strange place of bolting on a static type system to rather messy language as what may be issues with TS may not be for the rest of the space of languages with static type checking (e.g haskell effectively requires no type annotations at all in theory but in practice it's often better to use them and with extensions it's required).
To add to that, try looking at Go, Elm, Zig, C#, mercury, and Idris to see different type systems at play rather than assuming TS and Rust are all that's available in this design space.
> Simply running the problematic code will most likely throw an exception.
The issue here is "most likely" which during a large refactor (and due to a wide spread use of duck typing) can hide many faults within the application that go without being noticed until a user hits that path. Static types turn this into a compile-error thus it won't ever reach production. If you have 100% path coverage and test every possible type that can be passed and ensure truthy values don't change behaviour (e.g `if ("")`, `if ([])`, ...) when passed (need to test the space where all of them evaluate to `false` in addition to `true`) you're still short of the number of variations that can be passed to your function and thus may end up in a situation in future where the wrong behaviour causes failure at runtime post deployment. This is not to say you should use types only and not tests (still test even with static types) but rather that the domain of possible things passed is reduced to a testable set of values instead of always being the largest path-dependent set possible.
> It's hard to write a test that checks behavior but misses a wrong-type.
It's incredibly easy to write such tests as most don't test the negative space of such functions nor is it easy to know what the minimal shape of an object may be for your function as it depends on which logic it hits inside. If a truthy value is used or a branch is made on a field or method call of the object then anything unused in the consequent branch is not checked by your test. You'd effectively have to resort to fuzzing to find what the minimal shape is for each possible path to have an idea of what your function doesn't catch in tests to get even close to static type checking (which is still far off as in some cases the set is so large it will never finish / enumerate all possible shapes).
> SQL type autogen is limited to full rows, so any query returning an aggregate or something isn't going to work with that.
Ask _why_ tooling doesn't work with that and you may notice it's due to a lack of specification or just insufficient tooling. Take F# [1] as the example here which can do far more [2] and doesn't suffer from the mentioned problem. F# is a language with static type checking.
[1]: https://learn.microsoft.com/en-us/dotnet/fsharp/tutorials/ty...
[2]: https://fsprojects.github.io/SQLProvider/
In PHP, if you look at your logs you'll see almost all the errors are checking an array with an index that doesn't exist. This is because people are using arrays as objects and then using strings as members. If you just use an object, this type of error is impossible.
So, we have to write `??` everywhere because anything can always be null and then it can break stuff.
And then you have errors with passing empty string vs null vs empty array to functions and getting unpredictable behavior. So you need to constantly check everything.
If you actually open up a function in a dynamically typed language, take your pick, you'll see something like this:
``` if (arg === '' || arg === null || arg === undefined || arg === []) ... ```
If you don't include checks like that everywhere, your code will break. You just don't know it isn't broken yet.
And, btw, something like PHP `empty()` is not a silver bullet either. Because it considers like a dozen different values to be empty. Which comes with it's own set of problems.
Javascript has to check on each iteration that the item is an int and for arrays that the length of those arrays hasn't changed underneath along with a bunch of other things as the language doesn't guarantee that it can't change. Even the JIT compiler in use has to check "is this path still that I expect" as at any point the type of the variable used in the loop can change to something else which invalidates the specialization the JIT compiler emitted for the case of it being an int. When you don't use languages with static types you push all of this work on runtime which makes the program slower for every check it needs to do while offering none of the advantages you have with static types.
Thus with a for loop in say C you don't assert that it is an int each time you statically constrain it to be an int as the loop condition can well be based on something else even if int is the more common to use. For example in zig `for` only takes slices and integer ranges with any other type being a compile-error:
This is not more work than what you'd do in a dynamically typed language.If you don't understand the benefit of xyz then don't do it.
Our microservice implementation is great. It scales with no maintenance, and when you have three people that makes a difference.
They ignored me and went the microservices way.
Guess what?
2 years later the rebuild of the old codebase was done.
3 years later and they are still fighting delivery and other issues they would never have had if they didn't ignore me and just went for the "lame" monolith.
Moral of this short story: I can personally say everything this article says is pretty much true.
Having added a fancy new technology and a "successful" project to their resume, they're supposed to move on to the next job before the consequences of their actions are fully obvious.
Rather than 1 micro service per team, which many devs.. it was some team that owns 20 services, generally way more services than developers.
It's probably just how non-lean Mag7 were in peak vs how lean most other orgs that try to ape them are.
The microservice team were especially terrible because the did all the initial work and basked in the "glory", but when it came to maintaining the services, they wanted nothing to do with it.
Sure a monolith that does 1000 things might not be ideal, but 100 repos for 100 micro services owned by a team of 5-10 devs is unmanageable on the other end of the spectrum. Oh and everyone forgets the orchestration layer.
That's why I don't like the term "microservice", as it suggests each service should be very small. I don't think it's the case.
You can have a distributed system of multiple services of a decent size.
I know "services of a decent size" isn't as catchy as "go for one huge monolith!" or "microservices!" but that's the sensible way to approach things.
Even with a monorepo, you will hit a point where you have 1, 10, 100 million lines of e.g. Python, realize you should upgrade from 3.8 to 3.14 because it's EOL, and feel a lot of pain as you have to do a big-bang, all-at-once change, fixing every single breaking change, including from libraries which you also have to update. There's no way around this in current mainstream languages.
I don't think it's much better if you have to spend a year and a half updating 400+ different repos, though. It's much easier to use an operationalized language that knows backwards compatibility matters.
> I don't think it's much better if you have to spend a year and a half updating 400+ different repos, though.
There's two things going for separate services (which may or may not be separate repos; remember a single repo can have multiple services):
1. You can do it piecemeal. 90% of your services will be 15-minute changes: update versions in a few files, let automated tests run, it's good to go. The 10% that have deeper compatibility issues can be addressed separately without holding back the rest. You can't separate this if you have a single deployable artifact.
2. Complexity is superlinear with respect to lines of code. Upgrading a single 1mLOC service isn't 10x harder than updating ten 100kLOC services, it's more like 20, 30x harder. Obviously this is hard to measure, but there's a reason these massive legacy codebases get stuck on ancient versions of dependencies. (And a reason companies pay out the ass for Oracle's extended Java 8 support, which they still offer.)
I've worked on monoliths with 400+ developers that were great, but it takes skills that people who have only ever worked in orgs that mandate microservice just don't have.
Functional programming precludes encapsulation, so it doesn't scale indefinitely the way fractal paradigms can. Eventually, the complexity becomes overwhelming.
One effective solution to that is introducing microservices: programmers can still write entirely functional code, but have encapsulation in the form of services. They have to be micro, though, because conventionally-sized services are still big enough to strain the paradigm.
But I see junior engineers who aren't expected to think about the "architecture", by which they mean the modular design. They are handed a spec and they implement it, Mythical Man Month style. That treats organizing lines of code and organizing services as two completely-distinct activities, and depending on the company junior engineers are often not exposed to modular design until five or ten years into their careers.
Since when? Maybe we have different definitions of "encapsulation" but this clause seems nonsensical to me. FP is huge on encapsulation
Functional languages have some of the most rigorous module systems available. In fact Java adopted such a system recently, showing the weaknesses in its previous support for encapsulation via classes and packages.
Even if that state is kept outside the service itself (like a database or event queue) it can still be really hard to reason about when said state-stores are shared across a huge codebase. Changes to a part of the state can have negative effects in completely unrelated functionality.
And of course, there is nothing blocking a monolith from isolating/modularizing their state-stores, but it tends to not happen unless the architecture forces it to happen or through strong tech leadership.
If we try to read the commenter’s mind, at best they may have been talking about a hypothetical goal that no-one actually advocates.
The paranoid socialist in me thinks big companies like team-sized microservices because it lets them prevent workers from talking to each other without completely ruling out producing running software.
When companies instead encourage forums for communication across team boundaries, it unlocks completely different architectural patterns.
In high-uncertainty greenfield development, Explore projects or Lean Startup-style experimentation, having developer be close to the users they are serving is very efficient.
It also lets those companies reteam frequently, without needing to change the software to match the new team boundaries, which is very helpful when growing the team.
Even if you're using micro services, it's usually best to have them in the same repo organized into different directories.
No matter how many people you have, you really should minimize working on the same files concurrently. This is trivial with most languages
Why can the game industry etc somehow manage this fine, but the only place where it's actually possible to adapt this kind of artificial separation over the network, it's somehow impossible not do it beyond an even lower number of devs than for a large game? Suggests confirmation bias to me.
The main problem with microservices is that it's preemptive, split whatever you want when it makes sense after-the-fact, but to intentionally split everything up before-the-fact is madness.
How many times have AAA releases been total crap?
How many times have games been delayed by months or years?
How many times have games left off features like local LAN play, and instead implemented a 'microservice' as a service for online play?
How many times have the console manufactures said "Yea, actually you have the option of running a client server architecture with as many services you want?"
Comparing a game to almost any other piece of software, especially web based software, is how you end up with broken abstractions and bad analogies.
> How many times have games been delayed by months or years?
What are we arguing here? Because I can think of many microservice apps that are crap as well, and have no velocity in development.
> How many times have games left off features like local LAN play, and instead implemented a 'microservice' as a service for online play?
This is entirely irrelevant. We're talking about the trade-offs of separating networked services that could otherwise be one unit. You're saying "why do games have servers then" which is a befuddling question with an obvious answer.
That's like saying my web server is a Microservice because it's not run in my clients browser. It makes no sense.
Also those programmers seem to be neatly segregated in different areas of the project which I imagine work similarly to boundaries between the microservices at keeping the logic isolated between teams.
: I do AM surprised just how much QA people are credited in games, QA for major games sure do feel like an army.
The secret is that you're able to break a monolith apart, just like you can with microservices. You have APIs and modules of the monolith are responsible for their own thing. APIs are your contracts, just like in a microservice architecture.
The difference is that you can check if APIs are broken at compile time. In addition, you can view the API right in your IDE. In addition, your API isn't returning wishy-washy json with a half-assed OpenAPI spec - it's returning real types in a full-featured type system. And, cherry on top - you don't have to communicate over the network. Oh my god, you don't realize how many bugs and thousands of hours are wasted just working around that until you no longer have to. It's an immediate productivity boost.
But the best part is probably deployments. It's just so, so much more straightforward with one codebase.
The "one team per microservice" makes code-enclosure style code ownership possible, but it is the least efficient way I have ever seen software written.
I've long wanted to hack an IDE so people are only allowed to change the Java objects they created, and then put six Java programmers in a room and make them write an application, yelling back and forth across the room. "CAN YOU ADD A VERSION OF THAT METHOD THAT ACCEPTS THIS NEW CLASS?" "SURE THING! TRY THAT?"
People discount the costs of microservices because they makes management's job easier, especially when companies have adopted global promotion processes. But unless they are solving a real technical constriant, they are a shitty way to work as an engineer.
The trouble comes when some political wind blows and reshuffles the org chart, and now you're responsible for some services that only made sense in the context of a political reality that no longer exists.
1. when the requirements are better served by a different language/location/environment/platform.. or by deploying a 3rd party app.
2. some of the services need to quickly scale up and down, and you have enough traffic for it to be worth it.
3. if you have a tight SLA for parts of the app but not all of it.
We have dozens of service components that are all largely independent of each other - combining them together would be purely a packaging decision, and wouldn’t really simplify much. In some cases, it wouldn’t make sense or even be possible at all.
An example is our execution agent, which executes customer workflows - that’s completely independent both conceptually and from a security perspective. Each agent instance executes a single flow at a time, for resource consumption and security reasons, which entails an ecosystem of services to manage that - messaging, data ingestion at scale (100K flows per day, multi-petabyte “hot” datastore for active data), orchestration, and other supporting services such as data access and network routing.
All of our teams support multiple services, and many of them qualify as microservices.
If people on the team continue to think about the "system" as a monolith (what they already know and are comfortable with), you'll hit friction ever step of the way from design all the way out to deployment. Microservices throw out a lot of traditional assumptions and designs, which can be hard for people to subscribe to.
I think there has to be adequate "buy-in" throughout the org for it to be successful. Turning an existing mono into microservices is very likely to meet lots of internal resistance as people have varying levels of being "with it", so-to-speak.
As one would expect, they made bank from their consulting endeavor and rode off into the sunset while the rest of us wasted several years of our careers rewriting ugly but functional monolithic code into distributed Java based microservices. We could have been working on features and product but essentially were justifying a grift, adding new and novel bugs as we rebuilt stable APIs from scratch.
The company went under not long after the project was abandoned. Nobody, of course, would be held to account for it. I will no longer touch a tech consultancy like TW with a 10 foot barge pole.
It was Ruby.
Sounds to me like every startup.
WHY? Just why?
The other one was a microservice architecture in front of the real problem, a Java backend service that hid the real real problem, one or more mainframes. But the consultants got to play in their microservices garden, which was mostly just a REST API in front of a Postgres database that would store blobs of JSON. And of course these microservices would end up needing to talk to each other through REST/JSON.
I've filed this article in my "microservices beef" bookmarks folder if I ever end up in another company that tries to do microservices. Of course, that industry has since moved on to lambdas, which is microservices on steroids.
Programmers coming up through frameworks or functional programming often don't have those, and so the techniques OO unit testers use don't translate well at all. If the first "unit" you build is a microservice, the first possible "unit" test is the isolation test for that service.
I have watched junior engineers crawl over glass to write tests for something because they didn't know how to write testable code yet, and then the tests they write often make refactoring a-la-Martin-Fowler's-book impossible.
(And that is leaving aside the consultancies that want to be able to advertise "100% test coverage!" but don't actually care if the tests make software harder to maintain in the long run because they aren't going to be there.)
Eventually we'll be able to acknowledge that there are a lot of different skills in our profession, and that writing good code isn't about being "smart": it's about knowing how to write code well. But until then people will keep blaming the tools they don't know how to use.
Less mocking, more bang for the buck.
(Mockist tests are fine for people who really want them, as long as you delete them before checking in the code.)
Often times using things like cucumber to describe a set of interactions, in the days before headless chrome it'd drive selenium which would literally open Firefox on your desktop and start navigating the site.
It did it's job but the feedback loop was slow.
Today I mainly write libraries, cli tools and the occasional small HTTP API. I still have unit and integration tests but they are all just standard pytest functions. I can run each type individually, my definition of an "integration" test is either something that talks to the outside world in some capacity, or something that is really slow (mostly running an ml model these days).
I much prefer today's approach, but admittedly I work on things with far fewer points of interaction which I think greatly simplifies the work.
Just use regular sized services
https://github.com/polyfy/polylith
Few cases where microservices makes sense probably when we have a small and well bounded use-case like webhooks management, notifications or may be read scaling on some master dataset
Here's more at Github's docs: https://docs.github.com/en/repositories/managing-your-reposi...
I can't prove this scales up forever but I've been very happy with making sure that things are carefully abstracted out with dependency injection for anything that makes sense for it to be dependency-injected, and using module boundaries internally to a system as something very analogous to microservices, except that it doesn't go over a network. This goes especially well with using actors, even in a non-actor-focused language, because actors almost automatically have that clean boundary between them and the rest of the world, traversed by a clean concept of messages. This is sometimes called the Modular Monolith.
Done properly, should you later realize something needs to be a microservice, you get clean borders to cut along and clean places to deal with the consquences of turning it into a network service. It isn't perfect but it's a rather nice cost/benefit tradeoff. I've cut out, oh, 3 or 4 microservices out of monoliths in the past 5 years or so. It's not something I do everyday, and I'm not optimizing my modular monoliths for that purpose... I do modular monoliths because it is also just a good design methodology... but it is a nice bonus to harvest sometimes. It's one of the rare times when someone comes and quite reasonably expects that extracting something into a shared service will be months and you can be like "would you like a functioning prototype of it next week"?
The only way for significant architectural boundaries at team boundaries to not result in incredibly painful software, especially for a growing team, is to let the software organize the teams. Which means reorging the company whenever you need to refactor, and somehow guessing right about how many changes each component will need in the coming year.
It also means you can't have product and engineers explore a problem together, or manage by objective with OKRs since engineers aren't connected to business outcomes.
I know that all the ex-Amazonians are convinced this is the only way to build software, but it really, really isn't.
I'm a spirit of the law sort of person, not the letter of the law. I don't care how you draw your internal organizational diagram; communication barriers are your team barriers.
It's for management to read the flow over time and keep the de jure boundaries somewhat sync'd with the de facto boundaries, but in the meantime the de facto ones are what will get written into your software.
I have played around with architectures like this, but I allowed the caller to patch in a dependent function in the call with those function overlay overrides were passed from function to function.
Apologies, used sst
Every service boundary you have to cross is a point of friction and a potential source of bugs and issues so by having more microservices you just have more than go wrong, by definition.
A service needs to maintain an interface for compatibility reasons. Each microservice needs to do that and do integration testing with every service they interact with. If you can't deploy a microservice without also updating all its dependencies then you don't have an independent service at all. You just have a more complicated deployment with more bugs.
The real problem you're trying to solve is deployment. If a given service takes 10 minutes to restart, then you have a problem. Ideally that should be seconds. But more ideally, you should be able to drain traffic from it then replace it however long it takes and then slowly roll it out checking for canary changes. Even more ideally, this should be largely automated.
Another factor: build times. If a service takes an hour to compile, that's going to be a huge impediment to development speed. What you need is a build system that caches hermetic artifacts so this rarely happens.
With all that above, you end up with what Google has: distributed builds, automated deployment and large, monolithic services.
You can get the architectural benefits of microservices by using message-passing-style Object-Oriented programming. It requires the discipline not to reach directly into the database, but assuming you just Don't Do That a well-encapsulated "object" is a microservice that runs in the same virtual machine as the other mircoservices.
Java is the most mainstream language that supports that: whenever you find yourself reaching for a microservice, instead create a module, namespace the database tables, and then expose only the smallest possible public interface to other modules. You can test them in isolation, monitor the connections between them, and bonus: it is trivial to deploy changes across multiple "services" at the same time.
In the Q&A after ward, another local startup CTO asked about problems their company was having with their microservices.
The successful CTO asked two questions: "How big is your microservices tooling team?" and "How big is your Dev Ops Team?"
His point was, if you're development team is not big enough to afford dedicated teams to tooling and dev ops, it's not big enough to afford microservices.
It makes Database Per Customer type apps really easy, and that is something alot of SaaS products could benefit from.
Compare:
Sometimes one can get away with schema level split, which can allow cross schema references in a read only setup, but carries coupling concerns I don't mean to troubleshoot over HN comments, as the problem space is filled with nuance and trade-offs, but intended it as "for your consideration"We aren't using Postgres but a more specialized DBMS. Everyone basically does the first thing in your example, sharing one cluster. The real issue is it's a big corp with red tape around stuff, people don't want responsibilities like compliance, and people are rightfully scared of keeping things performant on a fancy DBMS. Our team isn't scared, we did it, but it wasn't easy either.
Agree with organizational win, also smaller merge requests in the team were superb.
Around 5-10 devs, monolith, we ran into conflicts more often, deployment, bigger merge requests, releasing by feature was problematic, microservices made team more productive, but rules about tests/docs/endpoints/code were important.
It’s a tool to solve people issues. They can remove bureaucratic hurdles and allow devs to somewhat be autonomous again.
In a small startup, you really don’t gain much from them. Unless if the domain really necessitates them, eg. the company uses Elixir but all of the AI toolings are written in Python/Go.
You can put most of your crud and domain logic in a monolith, but if you have a GPU workload or something that has very different requirements - that should be its own thing. That pattern shouldn't result in 100 services to maintain, but probably only a few boundaries.
Bias for monolith for everything, but know when you need to carve something out as its own.
At scale, you're 100% correct.
Remember the whole topic here is avoiding this tax
I've certainly seen microservices be a total disaster in large (and small) organizations. I think it's especially important that larger organizations have standards around cross-cutting concerns (e.g. authorization, logging, service-to-service communication, etc.) before they just should "OK, microservices, and go!"
it takes skill and taste to use only enough of each. unfortunately a lot of VC $$$ has been spent by cloud companies and a whole generation or two of devs are permasoiled by the micro$ervice bug.
don't do it gents. monolith, until you literally cannot go further, then potentially, maybe, reluctantly, spin out a separate service to relieve some pressure.
While I agree with you regarding microservices (eg language abstractions provide 80% of the encapsulation SOA provides for 20% of the overhead) and I readily acknowledge that 100% test coverage is a quixotic fantasy, I really can't imagine writing reliable software without debuggers, print-statements, or a REPL—all of which TDD replaces in my workflow.
How, I wonder, do you observe the behavior of the program if not through tests? By playing with it? Manually reproducing state? Or, do you simply wait until after the program is written to test its functionality?
I wonder what mental faculties I lack that facilitate your TDD-less approach. Can it be learned?
How is that even possible?
Sounds hard!
Jokes (?) aside, the scientific method is a loop, so it probably doesn't matter long at which phase you enter.
Like TDD, theyre great if done in the right way for the right reasons.
For large orgs where each service has a dedicated team it starts to make sense... but then it becomes clear that microservices are an organizational solution.
I love it when all my CRUD has to be abstracted over HTTP. /s