I thought Google at least was a bunch of micro services which is why they needed something like Borg and eventually open sourced a version of it called K8s.
Total BS.
Google has tons of services.
They are lots of them but they don't have a `fileRead` service and `fileWrite` service. Rather, they have a `gfs` service that can read/write/modify etc. everything related to files.
> Scale is measured in millions of instances: On 2022/12/21, the microservice topology contained 18,500 active services and over 12 million service instances.
They do go on to say that microservice as a concept is poorly defined. Are you suggesting that they mostly have 18500 of what we’d normally consider monoliths?
I am literally working on a project where the backend team consists of three coders, and the backend currently consists 33 microservices. Common tasks for this platform involves calling around 7+ microservices.
Thankfully I am working on the frontend side of things, but this is going to be hell when when we launch.
Sounds like you're making it hard for yourselves on purpose. May I ask why you (the team) chose this over other technology and what was the decision process at the start the project?
This isn't even a joke. Technology in software development has been progressing horizontally. Nothing is improving but abstractions and patterns are changing constantly.
Engineers need something to talk about in their OKRs. One strategy is to refactor things into microservices.
> Thankfully I am working on the frontend side of things
Honestly, given the opinion of frontend on this forum, I'm surprised that you're thankful! I am 100% certain everyone here would take any kind of backend work over the hype-driven mess that is frontend development.
I moved from backend to frontend years ago, when "hype-driven mess" was a much more accurate description of the field than it is today. There's simply many more interesting challenges in web application front-ends than backends, especially if you need spacial design- things where canvas or webgl excel.
Comparatively, there are far fewer backend positions available that are more than basic CRUD, even if it those CRUD operations are gussied up with a bunch of needless complexity juggling sources of truth across a bunch of different services.
There's plenty of drudgery out there, and some interesting work in every field, but on the whole I personally have found frontend product development- product configurators for engineering / manufacturing, network visualization, visual design tool development, and data driven interfaces to be more fulfilling work.
To each their own, but maybe you might want to be a little more aware of the people that you work with when you feel like saying something condescending.
Edit: FWIW, you'd have to be blind to not see hype driven development in backend, especially in HN. "Rewrite it in rust" is the meme that continues to keep on giving.
A dozen micro services per developer on rickety infrastructure serving data for a pixel generator built on top of 5 years worth of JS hype cycles. Why are we doing this to ourselves? It is just so stupid.
> I am 100% certain everyone here would take any kind of backend work over the hype-driven mess that is frontend development
The back-end is becoming an even more hype-driven mess these days. What with k8s and dozens of micro-services and controllers and telemetry-everything. And people don't even listen nowadays if you try to propose something simpler.
The front-end is at-least constrained by browser limitations and the client device. One can only do hanky-panky upto to a point. But at the backend, "throw another cloud machine" at it has become the de-facto solution for most performance issues.
One of our projects has ~15 backend engineers and ~30 microservices. It also requires a separate platform team. A k8s cluster with several nodes. It has incidents like every week because the whole thing is like a house of cards. A small feature requires changes in ~5 microservices and a lot of coordination between the teams. They always need to think about things like circuit breakers, timeouts, APIs (schema synchronization, versioning), event queues, distributed transactions, sagas, distributed locking, k8s configs, Git repository management, distributed tracing etc. etc.
On the other hand, there's another project, which is a monolith. Around ~15 engineers. Incidents happen very rarely. The infrastructure is simple: just one beefy server without k8s (but there's a standby replica of course), so the team often does maintenance of the server on their own instead of a separate SRE/production team. The monolith is modular so it enjoys same advantages as microservices when it comes to module isolation etc. Highload functionality is moved to separate Go services only when needed (currently 2 or 3 satellite microservices). All changes happen in the same codebase so features are released faster and it's easier to reason about.
I'm currently on the second project and whenever I attend architecture reviews of the first project, I'm shocked how overcomplicated the first project has become. A simple addition of a new object property can end up in 5-10 engineers arguing for hours whose microservices are affected and how they should best communicate.
I have been fighting this in my current position, with some success with some teams and far less success with others (who are now fighting with the distributed monolith they have created). Ironically the ex-faanmg folks are all pro-monolith.
I think most people fail to realize microservices are what you get from an iterative process. You pull out of the monolith the things you need to scale eventually.
Microservices need plenty of basic infrastructure that you don't want to mantain unless you need to. Most companies don't have an adequate platform to support the pattern anyway.
Didn't I mention the Citadel pattern, "trunk and branches"? Neither did I suggest that somehow starting with a monolith will force you to write better code.
But it will definitely be easier to debug and refactor by those who come after.
Microservices is mostly an RDD (Resume Driven Development) practice I've found. Really, as the article says correctly and hilariously, unless you're a greenfield or exceptionally large, you end up with a fleet of shitty golf carts.
My personal anecdote is, watching from the sidelines with face cupped in hands, watching a monolith being broken into hundreds of microservices at the cost of decreasing performance, increasing complexity and 90% of the traffic still hitting the monolith. Of course there's no time to do anything for the customers any more because everyone is updating software versions in all the microservices and promoting that "it'll all be fine if we just move one more thing to a microservice". The architectural vision will be complete in 2130 at this rate.
I want out from this demented ego driven fashion show.
I have seen managers propose microservices to their own bosses just to show their own bosses that their team will embark on something Netflix does - so we should do too.
Mirrors my experience, except some teams also decided to implement the new services in different languages...
After a downsizing, we are now left with 15 services, 5 different programming languages and complicated kubernetes setup. We have less than 10 developers that has to maintain this. In addition we also have a separate flow for managing data and BI.
Kubernetes is a bit of the new OpenStack. It's like an all-you-can-eat buffet of complexity concealed behind interfaces.
Either some startups go rampage with it and deploy needlessly complex stacks because the tooling makes it easy, hence a headlong rush into more hidden complexity to solve problems. Or they use it for the hype and merely deploy a couple monoliths with it, which Compose / Terraform would do just as well without the maintenance burden.
I have avoided, by luck alone, microservices, except the occasional small service where it makes sense (think AWS Lambda to covert doc to pdf for spikey loads) and I am glad. It means I don’t get far in interviews at small companies that bizarrely want microservices experience (as opposed to problem solving experience!)
> Microservices is mostly an RDD (Resume Driven Development) practice I've found
I agree. The worst I've seen is with contractors/consultants, because the incentives are particularly bad.
They often stay with the project only for a limited time and "grokked customer's business/domain logic" is not something you can put on your resume and the typical 6 months are too short for that anyway. But there's enough time to introduce a schema registry or other piece of tech which can look pretty good on your CV. Jump onto a different customer, rinse and repeat. I call this "technology-oriented mindset" vs. "goal oriented mindset".
When proposing a system design - have two. One using a monolith and second using microservices. Put the first in production and the second in your CV and everybody goes home happy.
We broke our monolith ~2015. 8 years later we still have the original monolith plus the original mircoservices are now monoliths too which need splitting. Upgrades were a nightmare for first 5 years, but we solved it with nitpicking code reviews where non-trival changes are blocked forever until everyone's ready.
At least we solved the problem of decreasing performance the old fashioned way: by throwing more RAM and CPU power at the problem.
> I want out from this demented ego driven fashion show.
The author of TFA kinda hits on this in the first paragraph:
> absurdity of the state of the current tech culture. We laugh, and
yet bringing this up in a serious conversation is tantamount to
professional heresy, rendering you borderline un-hirable
As soon as I read that, I felt I knew what the rest of the essay would
be about, and I was right. It's about culture.
A culture of insecurity, wannabeism, fads, in-groups cliques,
so-called "best practices", megalomaniacal levels of scaling,
over-reach..... let's all admit shall we (at least privately) , that
"tech" has an _ugly_ culture, that we've lost those early joys of
solving problems. We've replaced that with a culture of making
problems so we can look good by "solving" them. Just realising that
would bring enormous economic benefits.
In wider human culture, tech is looked down on, for good reasons. At
worst it is a proxy battleground for personal whims and ideologies in
which one group of people set out to impose upon another larger group,
their particular idea of how the world should 'work'. Code becomes our
poor man's mathematics, apparently adding rigour and substance to what
is arbitrary and manifold. The helpless dependency of those who must
use what we create, makes that a very abusive relationship.
It's a shame that Gerald Weinberg's concept of Egoless programming
came at a time before the "boom", and so couldn't address the
devastating effects of ego-driven development on the wider culture.
It's more about the internal dynamics of the development team.
Microservices are a solution to a social problem, not a technical one.
A team of N engineers requires N² coordination. Large teams get mired in endless meetings, email, design reviews. Small teams are more effective, but struggle to maintain large systems.
Splitting a system into subsystems allows each team to focus on their piece of the puzzle while minimizing the amount of peer-to-peer coordination.
Yes, microservices add complexity and overhead, but this approach enables a large organization to build and iterate on large systems quickly.
Not really. You can't really reduce the blast radius of crashes or bad deployments. You need to have the discipline of a good CI/CD instead of siloed but decoupled workflows.
Just keeping things neat doesn't go nearly as far as a separate process on separate machines. Monolith might be better but I don't think it's a situation where you can have it all.
This argument always confuses me. It depends on what you’re doing but if you’re doing web services, as most here are, the crash is limited to the request being served.
The blast radius is a single request, right?
In every likelihood it _is_ a separate process on a separate machine.
The software should be compatible with both the old and new schema until all of your database servers have moved over to the new schema. All rollouts are going to be somewhat staggered even if you go straight to 100%.
If schema migrations take non-zero time and you want zero-downtime deployments, then the both service versions need to be compatible with either the new schema or the old schema, regardless of service size.
The same problem exists in microservoce land, only you now have a load of separate teams managing their own database, perhaps with different solutions.
I think you misunderstood what I said, which is you still have databases and ergo need to manage migrations and such, only that this now typically falls to the specific teams.
That said, loads of folks do microservices with a shared dB.
Even for a web service, imagine a small feature is causing the app to crash to OOM, or too many file handles or something that will take out the whole process and not just the one request.
If you have long running requests (something beside small rest calls like large data transfers) then any being served by that same process are taken down.
Yes, only it takes a while to figure out the right level of abstraction for your organization. It's difficult to start right out of the box with the right type of modularization.
Usually the most important designation of the separate system is that it uses a separate database. You only have immediate consistency within one service.
Each service is authoritative on its own data, and it works neatly. Any medium to large size system is going to have natural places to separate that, or else will buckle under complexity or even hardware constraints if you try to keep it all in one DB. I've seen it repeatedly.
Nah this doesn't work out. Because requirements are so chaotic and modular design isn't an axiomatic science things will for sure go wrong with the design both because you got the design wrong and because the requirements shifted to a point where another design was better.
You encapsulated data under one service and you find that it's actually utilized much more in another service. This happens a lot.
The longer you can centralize your storage the better and easier everything is.
Exactly. And having more than one source of truth for your data complicates a lot of things.
Where I work we run a massive MySQL database that has a pretty simple sharding rule, so I don't believe that you need to split the database even at scale.
It's not primarily about the size of the DB, it's about what's in it. If your massive MySQL DB is only used by one team, then that's fine. If separate teams' services are speaking through shared tables on it, that becomes a mess of an API easily.
We have thousands of developers on it. I’m not saying it’s a silver bullet, but it works and it works well. Also, the service writing on it is only one (monolith).
Except I've seen this work at dozens of companies?
This thread is very painful, feels like CS101 coming in to tell us why we don't need relational databases and that we can just shove it all inside mongo.
It's not the thread it's most likely you. I don't know if people tell you but dealing with you is likely to be extremely painful if you say rude garbage like this to peoples faces.
Multiple databases can "work" but you get the problem I described above.
You know when you go to a company and eventually you see idiot practices or technical debt? Unfortunately this belongs to that category. You seeing it at dozens of companies doesn't validate anything except for the fact that bad patterns are common.
It also validates the existence of people who often do things poorly and don't know it (aka you).
Multiple DBs does work, and large orgs do it this way with good results. But I'm not here to insult people for disagreeing. It's just software, nothing to get heated over.
No worries. You didn't call the thread painful. That kind of comment causes heat. It doesn't belong here.
Of course it "works". But you can do better.
Splitting data into several dbs when one db suffices is called "extra work". Good results can arise from "extra work". But the extra work wasn't necessary.
You have to think like this: Sure multiple dbs work. But would 1 db in it's place work just as well?
The top commenter here from Netflix one of the people who promoted the whole microservices thing is telling startups to go monolith for a reason.
Our large org has been familiar with the 1 DB approach for over a decade. At some point the need for microservices became clear, and years after that, they actually switched. Even our subteam is feeling the need to divide up our service because its scope has kept expanding.
Startups shouldn't start with microservices. I've worked for (or led) a few different ones, and I've always created a monolith while avoiding other kinds of premature scaling (sometimes even "folder, bro" is overkill). Doesn't mean I'd run things that way with 200 people owning systems with evolving requirements for 10+ years, or even three separate dev teams of 5 people.
Right. But you likely split the db for performance not just for decoupling data and code. Microservices is an option for scaling. My argument is more against it as a way to decouple and organize logic.
Performance wasn't the issue; mono DB was actually fast enough. Multiple teams' services sharing tables became effectively a fragile API between them with hidden dependencies. Small new business requirements were causing too many breaking changes, and the org relied too heavily on a few experts who understood it all. Plus, schema changes were behind a ton of red tape since they affected everyone.
Also if by "decoupling data and code" you mean decoupling data from code, that's not the goal. It's to decouple one team's data from another team's data, and to properly abstract it.
The only reason our mono-DB even lasted this long was because previously, very few requirements were changing over time. So when people say their big org is monolithic, I can believe it. It works in some scenarios, but probably not most.
I know, but our DBMS doesn't support schemas. Instead, it's easy to spin up more "databases" that all share one instance but have separate ACLs, quota, backups, etc. Postgres has something similar but with less isolation. It doesn't really matter how exactly you slice it, point is you aren't sharing tables.
If you have centralized data then you have broken the abstraction and are in for a world of hurt. You shouldn't need centralized data, just a service front door where that service is the master of that part of the data.
At the point where you have to write an API call instead of simply joining to a table, you have introduced orders of magnitude more complexity, failure points, testing challenges, race conditions, coherency issues, code duplication etc into your system.
I don't mind stateless microservices too much. This forms more of a hub and spoke model (many services talking to 1 database). But the minute you bring separate state into the equation it's a chaos factory.
Nah but there are real instances where you have to use a different database for performance profiles.
The biggest one in web is regular entity databases vs. timeseries databases designed for analytics.
For this case entity data is just synced to to time series database every so often. Not super chaotic. It does make sense to divide services along those lines.
Though I would argue a folder still works for the web app part.
Joining a table is nice until that table is owned by a separate team. Becomes either gridlock or constant breakage, performance can get precarious, and testing involves populating a whole fake DB vs just mocking a few RPCs.
I really like relational DBs. I'll make a system rely heavily on complex joins. But I'd rather call some other team's API than share DBs with them.
The problem microservices try (tried?) to solve isn't about namespaces, it is about too tight coupling between code. Whether that tightly coupled code sits in a subdirectory or in a different repo doesn't matter.
It can be benefitial to maintain well defined interfaces at boundaries between certain parts of your code. It can also produce a lot of work and add complexity. But beyond a certain scale adding systemic boundaries and honoring them isn't something you should avoid.
Devs who do microservices just tend to go too far too early.
>The problem microservices try (tried?) to solve isn't about namespaces, it is about too tight coupling between code.
You can use folders to decouple code by making code private within that folder. Only publicize parts of the code with your languages version of exporting.
Or you can make it private behind an entire new repo and behind an entire service. Only publicize parts of the code through api interfaces like http and make everything 10x harder for the sake of decoupling code.
Many languages misses that abstraction to make things private in a folder (namespace) hierarchy to outside consumers. This is something I like to see solution to (integrated with your IDE of choice)
> Splitting a system into subsystems allows each team to focus on their piece of the puzzle while minimizing the amount of peer-to-peer coordination.
This does not happen at all. When you break a system into subsystems, all the previous connections that get remapped to new connections between subsystems still need to happen, in order to solve the fundamental problem that the system solves — except now instead of just making the connection directly, there has to be a "cross-functional" meeting between teams and a complicated communication layer between the systems. And if somehow you find a breakdown that requires minimal connections between subsystems, then those connections wouldn't have existed in the original system either, and the N² problem doesn't exist.
If that's the experience then you're doing services wrong. Each service should have its own datastore and a single API. The interface between services should be a single connection.
There should be maybe one meeting where the caller defines what they need the service to return to them.
The problem with this is that you have to be really damned careful how you split things up. If your separate data stores end up having to be joined together later on because some new business feature requires them to be cross-referenced, you're painted into one of two corners:
1. Merge the two services (and their data stores) into one and cause havoc downstream of either service
2. Burn through your network latency/throughput budget trying to reinvent a DB join across RPC boundaries (and god forbid if you can't batch multiple lookups in a single API call!)
Exactly. Software developers will never learn that the separation of concerns is a myth. In reality, UI, business logic and data are deeply entangled. Hence, moving these things apart makes everything worse.
your "argument" was literally just telling the person that they were naive for thinking things could ever be handled by one meeting.
turns out, they inarguably have a whole lot of experience at high levels in the biggest companies in the world. Either they're straight up lying, or it is in fact possible to resolve things with a single meeting. Why wouldn't that be the case?
I can tell you many other wisdoms: code should be tested, code shouldn't contain bugs, everything should be done before deadline, etc.
The thing is in reality it's not that easy.
> Either they're straight up lying, or it is in fact possible to resolve things with a single meeting.
Something can be done in a single meeting but not everything. But you gave me only two alternatives without anything between so there is another useful link for you https://en.m.wikipedia.org/wiki/False_dilemma
oh sweet summer child; have you not heard of the fallacy fallacy?
snark aside, just because a logical fallacy exists in an argument, it does not mean the argument is untrue. My experience echoes jedburg's - services should maintain their own datastore and provide api access. This allows them to be decoupled and allows the service to alter their datastore as needed. When multiple services leverage the same underlying datastore, things can get overly coupled. This becomes increasingly important as organizations grow because the coupling means more overhead communicating and aligning around changes.
Jedburg is an expert because he has seen this happen in multiple companies. I have too, but I don't have the speaking/consulting breadth of experience that they have. Credentialing and appeals to authority work because we are squishy humans and we can't verify everything so we lean in on others. That does not mean that authorities are always right, but when their experience and advice mirrors other's experience and advice, there is something to it.
And for the record, in public companies (two of them), I have exactly seen this play out: a well defined api that does not allow others to muddle in their datastore leading to _single_ meetings where new functionality is discussed and agreed upon and later implemented. I have also experienced the pain of multiple teams sharing a datastore when the datastore now needs to be altered which led to dozens of meetings across several teams and a rollout plan that was measured in months and quarters due to coordination needs. I can firmly say that the latter is a much harder (and expensive!) problem to solve in each case I've experienced.
I do not like appeal to authority either. But based on my experience, most of these issues were resolved in one to two meetings (I said maybe one, after all).
The team that needs data goes to the team that has that data and says "can you make an API that vends the data that I need?". Then they make it. And then maybe there is another meeting with a follow up, or perhaps an email asking for the data to be presented slightly differently.
But if you come to a team with a good use case and some examples, it shouldn't be too hard for them to create an API that vends the data appropriately.
This of course is all predicated on a good microservices architecture where you already have systems in place to handle API discovery and communication between services and proper data separation.
> Of all the things wrong with micro-services this isn't one of them.
It is, exacerbated by the over-use.
> The "complicated" communication layer is always either REST/JSON or GRPC.
Going over network, which is slower, unreliable, poorly typed. It's orders of magnitude more difficult to refactor a published REST API in comparison to e.g. Java interface within a monolith.
Microservices are generally far from easy to debug. In the best case scenario you have the whole stack locally and can put breakpoints, add logging immediately. But that happens rarely, debug port is often blocked (security), you can't easily modify the code to add some diagnostics without a complex CI dance etc.
But I've never worked on a monolith project that I could run entirely locally.
And this idea that APIs are slow, unreliable and unable be to be strongly typed is nonsense. This is 2023. We have plenty of tooling and techniques to make this robust.
For a monolith it's having to run the entire platform locally to fix one bug.
That includes databases, caching, auth, ML models, mock API endpoints etc all pre-populated with sample data then the actual application which for complex, JVM based ones can often fail to launch if there isn't enough memory to pre-allocate.
Many systems I have worked on all of that would simply not fit on a 16GB MBP.
The (monolith) product I currently work on has about 200 engineers working on it (backend + frontend + devops). I'd say it's of medium size, certainly not small. And yes, we can run it locally easily. The monolith starts up in like 20 seconds which I consider quite acceptable.
There are few narrowly defined responsibilities which are handled by dedicated services, and it is often more awkward to get them working locally. But because they are so specialized, you need them only rarely.
> And this idea that APIs are slow, unreliable and unable be to be strongly typed is nonsense.
Network APIs are inherently slower and more unreliable in comparison to an in-process method call. Some typing solutions are there, but they are way more awkward (and in some ways weaker) than statically typed monolith interfaces.
It's all fun & games until product wants to add a feature that doesn't map cleanly to your micro servies architecture. Then you end up hard coding you services into a macrolith. Good times.
Yep, I work in a large org that used to be a monolith (which means single DB really). Was a mess for the reasons you'd expect. Even our subteam of 10 needs to split things up more.
I've never seen teams organized around microservices like that. What I've seen, again and again, is one huge team where "everyone is responsible for all the microservices" (meaning, no-one is responsible for anything).
On a theory level I would agree with you - I've just never seen that happen in practice.
I'm not that much of a supporter of microservices, but my experience is the opposite: in every company I've worked for that used microservices, each team had their own set of microservices they were responsible for.
It should be like that but in practice sometimes other team just too busy to implement required feature. I can say my PM that we have to wait for a month but in reality I will have to implement it myself.
The complexities of software development could be solved with this one weird trick - if only programmers remembered FOLDERS. What an absurd thing to suggest.
Sure teams in large organizations allow other teams to randomly create folders? Also I've rarely seen an internal code base that is easy to grasp. You can create your own microservice and set up an API in 1/5th of the time it takes to understand a foreign code base and make a small adjustment.
FT.com did a recorded seminar session on their microservices architecture and one of the benefits they extolled is if someone wanted to improve on a feature they could just make it all over again and replace the old microservice with a new one. No need to look at the last persons code, just blow it away like it never existed.
I gathered their site is actually a black box filled with hundreds of black boxes of microservices. All a mystery, they either work or they don't and if they don't they fail gracefully quickly.
I'm honestly not even super convinced that small teams struggle to maintain large systems. I've been on a team that was only 7 good engineers maintaining a 3.5 million line project that had both a web UI and thick client. It supported 2 different databases and had a horizontally scalable job runner.
At one point it was 35 engineers, but layoffs took it down to 7, at which point we started to get a lot more done. There was just so much less time spent keeping everyone aligned. So many fewer meetings, sign-offs, reviews, plannings, retrospectives, management meetings, etc. Developers had a lot more agency, so they just got stuff done. Technical debt repayment became 50% of our time, as we easily knocked out features in the other half of the time. We kept ruthlessly cutting complexity, so it got faster to add new features.
I'm sure some projects just need more bodies, but I think there's an upper bound to how much complexity can be added to a system in a given unit of time. Adding developers over a threshold will result in the same amount of features per week, just everyone does a little less and spends a little more time on communication.
Repeat up to thousands of developers where adding a single field takes months.
>At one point it was 35 engineers, but layoffs took it down to 7, at which point we started to get a lot more done.
Years ago I did two back to back contracts for two different pharmaceutical companies. They were both about the same size but one had an IT group that was ten times the size of the other. You can guess which project was late and painful.
Microservices are not a solution to the problem you describe. Nothing in your problem description requires the micro part.
Microservices are about splitting the application into very fine-grained sub-applications. It's not about modularity in general, it's about making things as modular as possible (up to one function per service). That's why they are micro. Otherwise we'd just call them "services" and nobody would have any problem with that.
>Splitting a system into subsystems allows each team to focus on their piece of the puzzle while minimizing the amount of peer-to-peer coordination.
Assuming coupling is reasonable. If you have a "distributed monolith", you still end up with all the meetings because every microservice change risks breaking interfaces other people are using.
In the context of coupling, I'd argue the same applies to monoliths. Multiple teams can successfully work on a monolith given architecture where they're not constantly stepping in each other's toes (each team works mostly in their own modules/classes/packages)
Ummm, extremely large teams develop monolith (single binary) things called OSes or databases etc. The modularity for scaling developers cross-communication comes from.... modules! duh! Aka libraries.
Exactly, I missed this completely in the article. In my experience, microservices attempt to solve organizational problems and not technical problems. There are technical downsides to microservices that may be outweighed by organizational benefits. With monoliths you might have a large amount of hidden opportunity cost that technically never surface.
I do agree that this is less of a problem for startups so it doesn't make sense to start with a complex microservice architecture. But in large organizations, especially corporates, it absolutely makes sense.
"Splitting a system into subsystems allows each team to focus on their piece of the puzzle while minimizing the amount of peer-to-peer coordination."
The coordination is still there because a microservice team does not live in a vacuum. They build services based on demand from other teams that typically build web apps, mobile apps, sometimes server-to-server.
Hence, the "independent" team now becomes a roadblock for higher order features.
There's always something hot to hate on. I'm getting flashbacks to the days where everyone was going "nosql is trash" because they all cargo culted to mongodb back in the day and then tried to do olap on it.
Heck, this isn't even the first time SOA has been in the frame of hate. Member SOAP? Member how everyone more or less jumps between doing everything on the server vs everything on the client every year? We've been having that battle ever since networking or things clients have been a thing, the "sportsification" of that dichotomy is older than I am.
I like the other person's take in this thread about how you can largely get the semantic benefits of micro services by making a well crafted monolith. I honestly agree, but I think the followup is "aren't people who poorly break up service boundaries going to go do regardless if the interface is a network endpoint or a function call/dependency injected class?"
> "aren't people who poorly break up service boundaries going to go do regardless if the interface is a network endpoint or a function call/dependency injected class?"
Today there are several tools we can use that enforces boundaries in a monolith (Spring Modulith for example). If the project uses one of these tools, it is harder to accidentally cross boundaries and you get many of the same benefits as you get with a micro service.
The big advantage is that if you find out that you made a mistake, your only dependencies are within the same service and is easier to refactor. In a micro service oriented architecture, changes might impact several services and teams that needs to be coordinated. I'm not saying that refactoring a monolith can't be time consuming, but you have at least a better control of the flow of data between modules.
An interesting angle would be how to get out of this mess.
Here's one: In politics we see national and supranational governments take on the job of making large decisions. Style guides and best practices on whether to use microservices or not are, at best, made at a company level.
Would it be interesting if these architecture decisions were made, or at least kept in check by a body that is larger than a company?
Perhaps we could have some laws that dictate that you must (at least partially) understand how software works before you can buy it. Especially for government contracts.
The second you think you are smarter and more informed than the people on the ground is the second you revealed yourself to be a fool.
Because having spent decades working for government and enterprises I can assure you that we aren't all stupid and need laws to protect us from ourselves. Instead we are often placed in really challenging and unique circumstances that drive the architecture and design of what gets built.
For example micro-services often works well in places where you have distributed teams that for security, governance or logistical reasons need to work independently and can't collaborate effectively with a monolith architecture. Or where certain components e.g. authentication, payments need a hard separation from the rest of the platform.
Obviously not all in government are stupid, I was not implying this. I was rather suggesting that some are, and that something might be done to avoid problems with that.
Also, microservices can be great, but when used in the wrong context, obviously are not. We are not discussing that either.
Edit: perhaps you may have misunderstood my comment, and that may be due to my clumsy way of stating it. I was actually suggesting that companies learn from governments.
The small org I work for has two distinct AWS accounts and nobody knows whats in each one, let alone whats in the regions for each one. Everyone is an IAM user that a third party IT company creates after emailing helpdesk a few business days prior.
both AWS Accounts have a bunch of lambdas randomly doing necessary things in a variety of our development environments, but mostly production
a single AWS account is fully capable of catering to multiple development environments
This article is off the rails (to borrow the authors amtrak metaphor).
The author posits that if you make pile of crap microservices then all you had to do instead was make a monolith and magically it’ll all be fine.
You can put the same kind of design and engineering work in that results in a pile of crap microservices but if you target a monolith instead then you’ll be golden.
The author enjoys stroking his own ego as he goes (the comment about full stack js devs for example). And yet, there’s very little here in terms of actual engineering. Want some measurements or some data to back up the waffle? Well the author says tough! You just get a diatribe instead.
Keeping the cost to change a system low by managing complexity is a fine goal, but that’s not what’s being proposed here. This article could have been better if it recognised this. This article could have been better if it gave some data - hell even anecdata, a single motivating example, would have been a start.
On my team, I’ll take on a bright enthusiastic front end dev who decided they want to spread their wings and grow into a full stack dev over someone who believes they already know everything and has no growing left to do.
Agreed the dichotomy presented is so reductive that it makes you question the author’s credibility. Any architecture that isn’t well maintained will become crushing over time.
Just use Elixir/Phoenix and get all the scaling benefits of Microservices running seamlessly across datacenters while maintaining the sanity running a single app on your localhost that is fast to test, easy to reason about and deploy!
Having worked in several places that adopted Microservices with nowhere near the Netflix or Uber scale just because it's trendy I can attest to the mess and glacial pace of both learning and development for new team members. If you want to move fast and make something people want, don't pretend you need "webscale", you don't. You're building an apartment block for a few people not a city for millions.
The beauty of Erlang/OTP is that it's designed to scale to millions without any extra effort from the individual contributors or fancy DevOps.
i don't see how erlang / otp or elixir / phoenix is relevant to solve the problem most companies are facing when hitting their first "web scale" problem : bottleneck at the DB level.
Agreed. The DB is the bottleneck way before the code architecture. But that is a "solved" problem from a deployment perspective. All the Cloud providers have managed Postgres that scales to petabytes and millions of transactions per second. And the part being discussed in the article is the Microservices. Where Elixir/Phoenix shines at delivering the benefits of Microservices i.e. scaling a specific function in your codebase to multiple processors/instances while still being easy+fast to run on a single Dev machine without hundreds of Docker processes.
Erlang/OTP (and therefore all BEAM Languages such as Elixir) have a supervision tree which means that each module in a monolithic application can be run independently in as many processors/machines as required. Essentially you get Kubernetes + Docker for free (virtually zero overhead) inside the Erlang VM. See: https://blog.appsignal.com/2021/08/23/using-supervisors-to-o...
I am one of the biggest proponents of microservices. I helped build the platform at Netflix, I've literally traveled around the world extolling the virtues of microservices.
But I also advise a lot of startups, and you know what I tell them nearly every time?
Build a monolith.
It's so much easier to start out with one codebase and one database, and that will scale for a while. Especially if you use a key/value store like DynamoDB (although you will lose relational functionality that can be helpful at the start). And did you know that you can deploy a monolith to Lambda and still get all the benefits of Lambda without building services?
And then, when you start growing, that's when you break out an independently scalable part of the system into a microservice (with it's own data store!). The one that may need to scale independently, or that you want to be able to deploy separately.
Microservices takes at least 25% of your engineering time just maintaining the platform. It's not worth it unless you can recoup that 25% in efficiency.
"A while" is underselling it. As long as you have people who are half-decent with SQL, "just put Postgres on a big db server" will get you to 50 million row tables before you have to start thinking about even hiring a real DBA.
Unpopular opinion: if you're skimping on ops/DBA resources (as you may need to do in a startup), then MySQL is a better default. By all means use postgres if your use case demands it, but personally I find the ops story for MySQL takes less engineering overhead.
I agree with this "unpopular opinion". Worked with both MySQL and postgres based mid-scale apps of several thousands of users. Postgres is so deeply lauded here at HN yet requires two more magnitudes of operations work to keep it up and running. Vacuuming sucks hard.
That is a very specific use case, and might only be a small subset of your actual data. If you don't have these specific requirements (eg. CRUD apps), you can save yourself a lot of unnecessary headaches by defaulting to MySQL.
My main point is attempting to counter the narrative popular on HN that postgres should be an automatic default. For sure there are many aspects in which postgres is superior, I absolutely do not debate that, especially when it comes to developer experience. But there is much more to it than that when it comes to delivering business value. That's where ops and DBA concerns start to matter, and IMO MySQL is so far ahead in this regard that it outweighs all the other hideous warts of working with it, when you consider the bigger picture of the business as a whole.
Yes, and most successful companies who started in the last ~20 years started ( and many continue ) with a monolith and a MySQL database.
Only the mega-cap ones started to pursue other options mostly due to their type of business and bucket-loads of "free" VC money with explicit orders to burn it and get "unicorn" status - which involves hiring thousands of developers in record time and the whole thing turns into a zoo. Which is an organizational problem, mostly not a tech one.
Other than the ones we pretend are the whole Universe, there are thousands and thousands of medium to big companies with billions of revenue who started their product with a monolith and a MySQL database and many still do just that.
50M is something that can easily be done with just a dev who understands that sql is more than select, insert and update with the manual, google and chatgpt.
You can get really damn far with a fat postgres box.
It's not even funny how many software engineers just don't know that, SQL is crazy fast and performant, if you have basic understanding about it - I was once refactoring (or, rather, getting rid of) a microservice that was just a JSON blob storage on top of Postgres, without having any schema for blobs, with 100s of 1000s of them, no indices, and main complaint was - it's slow.
The problem with these discussions is that you can always get a lot out of any given architecture, structure, db approach or whatnot. Can. There's just a lot of daylight between "can" and "likely will."
Ultimately, everything has its limitations and tradeoffs. If we respect them, it's generally a smooth ride. Problem is that we rarely do... within a company (startup or otherwise) under real conditions. There's also a dynamic where we build until the point where something stops us. Tech debt, complexity, over-engineering, under-engineering, feature bloat or antagonism between early decisions and current goals.
There's a self-regulating aspect to this. If architecture is spot on, perfect for the task at hand we can move faster to reach the point where it no longer is.
My point is that postgres is, compared to almost everything else, easy to get from "can" to "likely will" with just somebody with a brain, a manual and google. In absolute terms it of course depends, but the point is relative.
A brain, a manual and google is usually a pretty good way generally to make solid decisions, respect the limits of your chosen stack.
What happens when there is >1 brain involved... or when brainless, manualless decisions eventually get made... or two pivots from now...
I'm not disagreeing with your approach. I agree with it, especially as starting point. I'm cautioning that resilience against complexity isn't about how easy it is to make good decisions when you understand the spec, read the manual and calmly proceed. Complexity and fragility accumulate when one or all of these are absent. How easily you can (and thus inevitably will) make a mess... not how easily you can keep it clean.
IRL situations with regular rdbs, a very common trend seems to be long term drift between schema and spec. The flexibility and approachability of postgres often enables a lot of kludge eventually.
Data stores have this dichotomy between "look how easy" and "is limiting factor" that speaks to difficulties we don't know how to articulate or isolate.
Of course if there are lots of bozos in the startup then even getting to and maintaining 50 million rows is going to be very difficult.
But if you have a small group of folks that are at least as competent as the folks at WhatsApp pre-acquistion, then there really shouldn't be any doubt whatsoever.
> a small group of folks that are at least as competent as the folks at WhatsApp pre-acquistion
That's a success case... beware survivorship bias.
> if there are lots of bozos in the startup
No arguing that quality engineers are fundamental to quality engineering. That said... by this standard, there's no point in having this entire discussion. Every good db/store out there is good. They all work very well if used as they should be, with due respect to tradeoffs. Yet, almost everyone has db problems. Almost every one of these problems occur well within the technical limits of postgres or whatnot.
"It shouldn't be a problem" when it usually is irl is tunnel vision. There is an empirical reality disagreeing with you. Walking into it with "this shouldn't be a problem unless everyone is a moron" is bad strategy. If you can't think of reasons why architecture can and will become a problem, then just assume that you (or some of you, some of the time) are morons, and try to make it moron proof.
I've been on both the sysadmin, development, and hiring sides, and with data models at scales of 50M+ records, devs who "understand that sql is more than select [...]" are rare in my experience, as they're a cross between db admins and developers.
Administrating (in particular, query planning and production operations) databases with tables sizes with magnitude of 10M and more records is challenging, and requires a skill set that is very different from pure development.
One won't get "really damn far with a fat postgres box", unless they're doing very simple SELECTs, which is not the case with modern web apps.
> Administrating (in particular, query planning and production operations) databases with tables sizes with magnitude of 10M and more records is challenging, and requires a skill set that is very different from pure development.
Is it more or less challenging than the alternatives? Is it less challenging enough to add a new tech to your stack, add the required knowledge to the team, etc?
I mean, knowing "enough-to-perform-CRUD" SQL is table stakes for developing on the back-end, but knowing $CURRENT-FLAVOUR-NOSQL (of which there are multiple products, all with such substantial differences that there is no knowledge transfer between using them) isn't, so there's going to be ramp-up time for every dev, and then every dev that is added to the team.
I'm not disputing your argument, I'm just pointing out that, sometimes, it's easier and faster to upskill your PostgreSQL developer to "scale the DB" than it is to teach them how to properly use, maintain, architect and code for DynamoDB and others.
It's not just $CURRENT-FLAVOUR-NOSQL. It's also doing custom transactions and locking on top and/or thinking in terms of eventual consistency. It's so, so much more complex than just SELECT FOR UPDATE/BEGIN TRANSACTION.
The problem isn't storing or inserting 50M rows, it querying 50M rows in non trivial ways. And the difference in performance between doing that 'right' and 'wrong' is orders of magnitude.
I've scaled 300M rows in just one of many similarly sized tables to 1M users... in 2007... on a single box with spinning rust in it. Heck, my laptop could handle 100x the production load.
It amazes me that my comment (while admittedly flippant) got voted down.
It really is true that your phone can update a 50M row table about 10K times per second!
That people are incredulous of this is in itself a stunning admission that developers these days don't have the faintest idea what computers can or cannot actually do.
Just run the numbers: 50M rows with a generous 1 KB per row is 50 GB. My iPhone has 1TB of flash storage that has a random access latency of something like 50 microseconds, which equates to 200K IOPS. An ordinary NVMe laptop SSD can now do 2M. Writing even 10K random locations every second is well within mobile device capability, with 50% headroom to "scale". At 1 KB per row, this is just 10 MB/s, which is hilariously low compared to the device peak throughput of easily a few GB/s.
It's not that good usually, e.g. PostgreSQL writes data in pages (8 KB by default), and changing 10K random rows in the 50M rows table can be quite close to the worst case of 1 changed page per changed row, so 8x of your estimate. Also need to multiply x2 to account for WAL writes. Also indexes. It's not hard to hit a throughput limit, especially with HDDs or networked storage. Although local SSDs are crazy fast indeed.
Agreed: 80MB/s for the random 8K page updates. However, transaction logs in modern databases are committed to disk in batches, and each log entry is smaller than a page size. So a nice round number would be 100 MB/s for both.[1]
For comparison, that's about 1 gigabit per second in the era of 200 Gbps networking becoming common. That's a small fraction of SSD write throughput of any modern device, mobile or not. Nobody in their right mind would use HDD storage if scaling was in any way a concern.
[1] Indexes add some overhead to this, obviously, but tend to be smaller than the underlying tables.
Eh, intelligent table design should knock most of that out. If you've got an 8 page query implementing a naïve solution to the knapsack problem (I've seen this in the wild) several mistakes have been made.
You’ve got a hell of a resume. And been accidentally incredibly convincing in getting many people into many early messes.
Jokes aside, thanks for at least coming around to advise startups sanely.
I’d personally never advocate for microservices until at the scale of Netflix or Amazon or Reddit, and even then only with in-house expertise at your level. Otherwise it’s a nightmare.
Thanks for everything, especially your contributions of sanity.
Aww, thanks for the kind words. I apologize if I caused you any harm with my talks. I did in fact start out saying everyone should use microservices, but I pulled back as I saw how damaging that can be to a small startup, or even a large enterprise that doesn't actually need it.
There is a way to splitting data. The service which owns the data always does the writes and others who need reads on that data can store replicas. ofc the complication then will be in replicating data, but this will enable services to massively scale and eliminate SPOFs
The other part of this pattern you mention (replicating data) seems solved in many cases by the data warehouse patterns of the last few years. Stuff it all in Snowflake or BigQuery then readers can query as they see fit. Query engines like Trino can paper over data storage heterogeneity / breakdowns in centralization. I'm not a fan of the "lakehouse" terminology, but it is the thing.
There are downsides to coupling more loosely with data consumers, but it keeps service owners moving without wasting time vending data.
And then the order table will need to set user ids as a foreign key and you will have two sources of truth lol
Will be fun when you decide that you want to have organizations to be linked to orders instead of users.
All this to say, data management and having more than one source of truth complicates your world a lot. The responsibility for writing the data has to be solved at the appropriate layer: only one entity writing it.
At several places, the biggest challenge to engineering velocity and the ability to innovate is that models are joined in interesting ways that prevent them from being decoupled. A user model tied to the package model preventing independent scaling of either. Every module reaching into all the tables. Joins, joins, and joins. Vast scans. The queries become inefficient and even the mighty postgres slows down at 2k rps. It is just too tempting to reach into another team's datastore. Now the two teams are in lockstep and can't alter their own datastore because others have assumptions on how it is stored and they access it directly.
At SendGrid, we had an few tables to deal with IP management. Over time, the like 5 tables were in use by 15+ services owned by different teams with competing priorities. When we finally had to scale the database, it took over three quarters of working with other teams while we supported both legacy and the new hotness.
It is hard to see early on where whole teams can sprout up and have domain ownership and to know which areas will be common/platform-like for other teams. But as soon as two services share a table, you should see a train coming at you.
There is severe hindsight bias in play here. How many tables weren't problematic despite all those joins and just chugged along scaling fine? How many startups were bogged down by unnecessary complexity or trying to keep a semblance of consistency instead of shipping features and just didn't reach the scale when performance becomes a problem?
Joins are good, actually, and normalisation should be the default. It's much easier to keep things consistent by default rather than trying to make them somewhat consistent afterwards. When something becomes a problem there is a bunch of tools (from denormalisation or materialised views to specialised datastores) to deal with that, but before that it's just premature optimisation.
> Respectfully disagree. At this point you start refactoring your monolith into components and actually look at performance measurements via tracing.
No matter how hard you modularize and instrument your service, that won't give you regional deployments, which you absolutely need to have ti be able to drive down customer-facing latencies from many hundreds of milliseconds to a few tens of milliseconds.
How do you pull that off with a monolith when you have a global userbase?
Do the next simplest thing: shard geographically. If you really need to handle users transiting geos, that's still dramatically simpler than meegroserbusses.
I wrote this 8 years ago: Microservices vs. "air-gapped" modules https://www.linkedin.com/pulse/maintainable-software-archite.... You can achieve the same end as a microservice using a module, by simply having lint rules that prevent you from importing other application level modules. This way the only possible comms interface is to pass events with a payload of primitive typed parameters.
An interesting idea. Sort of a good half way. But one issue I see is that you can still have a shared data store. That means you can accidentally (or intentionally) use the database to pass back-channel messages —- have one module store data and another read it.
That could lead to hard to find bugs. Did this ever come up for you?
I think systems like this are prone to buckling under product pressure. If it takes a few minutes to override a lint rule it's bound to happen sooner or later. With a microservice system the amount of effort is genuinely higher to add cross-dependencies.
The best "microservice" based system I've ever worked on was full ten years before the term had been invented. It had multiple services that read and wrote to the same database, but each used a common dynamically linked library that was the common interface to access that database (no naked SQL was allowed -- you had to #include <foo> and then do Foo* foo = Foo::findById("xxx"); etc. Worked remarkably well. The above common library was very carefully maintained and any addition had to be reviewed and approved. The approach seemed to work well.
That's the services model that Facebook uses and parts of Amazon use. A single data store surrounded by a bunch of small services with a prescribed data access layer.
That's really more of a monolith though. One of the key tenants of microservices is data separation -- each service has its own data store.
I think if we also "air-gap" (loosely) the dependencies, we get a typical monorepo in, say, JS or Golang? That is, a module in a monorepo is a special case of your airgapped modules?
Not exactly sure what you mean, but in my model, the modules all get built into a single runtime artifact/executable. In the case of monorepos as I understand them, the "mono"ness is in the version control system, but each code base builds to its own artifact or executable, which have to communicate with each other over a network interface.
How would you split the total overhead between monolith and monorepo?
(sorry this ran away with me)
A dumb example is that if I start with my single codebase running on a single server, I am likely to have a single git repo (foo).
Then I have a genius idea and put all the email handling code into foo.mail and soon I have foo.web and foo.payments.
All is fine as long as I am just checking out HEAD each time. The code running in the runtime is still one big set of code.
If I get creative and put in load balancers it's still a monolith.
But if I split out the web servers from the email servers, then I kind of start to see microservices appear.
I am trying not to be pedantic, but I am truly interested in experienced views on where the pain really starts to appear.
At this point (server to server comms), I should look at mTLS, and centralised logging and all the good stuff to manage microservices.
But how much pain was there before?
What if I was a large company and so hired a dev or two per repo (you know to get that 9 women one month effect). Co-ordinating multiple devs over different repos, even with one monolithic runtime, is painful (experience tells me).
So I am interested in where the break points are, and whether there are easier paths up the mountain?
Splitting out the webserver (assuming it is the main entrypoint for users) seems more like an infrastructure choice than application architecture and having an independent email-sending-service looks more like replacing a third-party offer (like turboSMTP) with an in-house service.
I do not think that this is what people mean by microservices.
I get that. I was trying to describe a small path from monolith to microservice without inventing yet another student and teacher database. But the even the step you mention matters. going to a third party service is very similar to reaching out to your own internal service. Make it a "customer" service.
Is the pain in microservices (or APIs and mTLS) or is the pain in managing a team that now has to do customer stuff only ? or email stuff only?
Common path to pain right there — services based on nouns and entities rather than verbs and business processes.
Having a single "customer" service means anything involving customer data has to talk to that single service. If that service has a problem, every other service stops working. You basically have a tightly coupled distributed monolith, so network boundaries instead of function calls between modules.
Think instead of splitting that data to separate services around business processes like billing, shipping, outbound marketing, whatever. Each service owns the slice of customer data required for its processes and can run fully autonomously.
…until you need to market to people who didn't get a shipping last month, delay shipping until at least one recharge attempt, or ship before charging for customers you're reasonably sure will pay anyway.
Business changes, and putting rigid firewalls in place without seeing an established pattern of change and without established need for that is a bad idea.
One global database buys you a lot (easy data access, easy transaction consistency, simple locks, even transactional notifications), and you can take it very far performance wise, throwing that away just in case you might need more performance later seems unwise.
> …until you need to market to people who didn't get a shipping last month, delay shipping until at least one recharge attempt, or ship before charging for customers you're reasonably sure will pay anyway.
These are all very simple things to handle when services encode processes instead of entities
> Business changes
Yes, and then you throw away the code for that outdated process and encode the new one.
> One global database buys you a lot
Absolutely, this is the correct choice for probably 95+% of companies. I will always argue in favour of a monolith and single database.
If someone is dead set on microservices though, they'll avoid a world of pain by actually designing them properly
At Netflix we had both noun and verb services. We had the "play movie" service, which was very much verb, as well as the listing service for example. But we also had the subscriber service, which was the source of truth for any account data, and the catalog service, which held all the info about the movies.
Yes, subscriber was a Tier 0 service and required a strong reliability and scaling story. They had more rigorous deployment setup and were usually over-scaled for traffic. But they also vended a library that services used to connect to them. That library had intelligent caching and fallbacks, so that even if the service was down, in a lot of cases the system could keep running. And the catalog was the same.
Microservices is very intertwined with how your engineering teams are built. We had a team that worked on subscriber data, one that worked on catalog data, one that worked on making movies play, one that built listings. So we had services to match.
Thank you. I sometimes feel microservices are an evolution of Conways law. How a company sees its internal organisation is how it will want to arrange its microservices and vice versa.
I remember trying to build out a "barium bullet" through very complex intertwined systems - following known customer data and looking for their footprints in all linked systems. It gave some insights on how to redesign systems - like someone else said here, removing complexity is ridiculously hard.
That seems like a great approach for Netflix and investing in a resilient, redundant Tier 0 service like that was likely worth it there
Most orgs don't operate on that scale though and resources would likely be better spent elsewhere
> Microservices is very intertwined with how your engineering teams are built.
I wish more people understood this. Microservices are much more about scaling your organization than about scaling technology. At a certain size, teams can't collaborate effectively enough and splitting out different services for each team makes a lot of sense
Wise words. They boil down to a very simple truth that was as accurate half a century ago as it is today:
Make things as simple as possible, and as complex as necessary.
I can always make something more complex than it is now. As you say, we can take out things from a monolith and make them into a service. It can be hard to do so, sure. But nowhere near as hard, as trying to get complexity OUT of a system once it's in. Everyone who ever tried to revert a bunch of microservices back into a Monolith knows exactly what I am talking about. It usually amounts to the same work as a ground-up rebuild.
Absolutely agree, I've seen far too many companies spend far too much time working out the infrastructural relationships of microservices. Spending very little time working on the actual applications needs itself.
Just chipping in with thoughts on DynamoDB, (although I have worked on much smaller scale systems)
I am a long term dev, done lots of SQL, but for the past few years I have been using DynamoDB, and I am using it for my new startup (So I rate it).
Cons
- You have to be very aware of your query patterns, and not having ad-hoc queries is a pain.
Plus sides
- With on demand billing, its free if you aren't using it
- Built correctly, it will scale
- No Schema upgrades (This one is massive for me)
On the last point, I really do appreciate not having to worry about keeping schemas upto date across all devs and environments.
We use quite a simple pattern of a table per entity, as opposed to single table design, because it allows us to just use the API at the highest level of abstraction, where you just write your objects to dynamoDB.
(You can still do lower level requests to poke values and such like)
At Amazon, relational databases are banned unless you get an explicit exemption from senior leadership. This is the primary reason why. Too many cases of schema upgrades causing outages.
The problem with DDB or other NoSQL applications, like you say, is how much you need to consider your query patterns. The last major project I worked on using DDB, we spent a couple of days just thinking through our query patterns so we could come up with the right database design and data structures. (We still believe it was the right choice, though.)
Wise. Every time I’ve started a project trying to make an amazing platform with everything segregated with micro services etc. I’ve spent so much more time on the _platform_ than on the actual product.
Now I go with SQLite and some basic python script and pivot from there. Ironically that’s how I used to do it before the micro service fad.
I honestly believe that doing small services is long term better approach than a monolith. But as you I strongly believe it's not where you want to start.
But I do not agree on is more time consuming than the other. It's just time spent on different matters. For the sake of money spent, I have not seen any data on how a monolith outperforms small services. For maintaining the platform, I think 25% sounds a bit high.
Money spent should be measured in many different aspects. One is definitely productivity. And I have seen more stale monoliths than I have seen stale small services. Whether it's better or not is not up to me to judge. I just know what I prefer.
Having been exposed to small service architecture where it has been working really well and very poor, there are few things that stand out.
- Conquer and divide (your monolith over time).
- Responsibility boundaries are easier to cope with for developers in a small service since they often don't have generic and yagni abstractions.
- Less code to comprehend for a small service and the cognitive load decreases.
- Build time, test time, deployment time.
- Should lean up against a direct business measurement and value.
- Group chatting services into one.
I have yet to see a monolith that over time is not really deteriorating, but I haven't worked with SO or Shopify. We could also argue that Cobol and Fortran is still of good use but time has also changed leaving that style of system development exposed as old and dusty.
But like any software development occurrence, it takes responsibility, mandate and proper leadership to get things done in a decent manner. So if you start by making 3 services that has to chat to find a user profile you probably don't know what you're doing. And 25 people (the Threads image) is not a small team IMO.
The problem is not with the devs but with the investors I believe. They're looking for unicorns, unicorns attract million of customers, so they need to work at scale, hence microservices.
Well, in the previous cycle it was clear that the investors were looking for buzzwords and marketing hype because they often don't understand and frankly dont care - they just wanted to make you look good enough to sell to the next guy, as they were all assuming you were unprofitable in and unprofitable out.
They all just want a story that you are going to be the next google so they can sell that story to the next guy.
Will that hold in the non zero interest rate world? The AI hype doesn't make me think the world has changed that much.
Most people don't understand the point of microservices. They look at the idea and are attracted by the power of modular interfaces, customizable scalability and independent deployment, without really thinking of the additional engineering overhead that are required to unleash these capabilities.
I've seen ex-Netflix software engineers taking jobs elsewhere and proposing microservices for systems that would receive little to no benefit from them. In practice, the implementation of microservices in these contexts become a costly solution looking for a problem.
> It's so much easier to start out with one codebase and one database, and that will scale for a while.
Lots of businesses don't even need to scale anyway; Netflix is a "high customer, low revenue per customer" type of business, but there's lots of "low customer, high revenue per customer" businesses too, perhaps even more than the first one. These are often the type of products where you could quite literally run production on your laptop if you wanted to.
At the last place I worked they built all this microservice bonanza for ... 300 customers... If they ever got a few thousand customers it would be quite successful, and tens of thousands would be hugely successful. What usually happens is that they don't spend that "25% of your engineering time just maintaining the platform", so the platform was impossible to run locally, and production was hanging together with duct tape.
(Aside: in a recent interview I was asked to design a system for "500 million concurrent users" – this is for a business that's mostly in the low-customer/high-revenue type. I still don't know if the test was to point out that 500M is an utterly bonkers number – about a 10% of the internet connected people on the planet – or that they really thought this was somehow a vaguely realistic number for a startup. I answered with "build the simplest thing possible, a monolith, and focus on features and making sure you're building something people want". I didn't get hired, so I guess they were somehow under the misapprehension that you need to design things for 500M users right from the start?)
As an example, you'd write your entire API as a flask app, and then deploy that app to Lambda. Then send all requests to that one Lambda. As long as your startup time is quick (and your datastore is elsewhere, like in DynamoDB) then it will work great for quite a while. Lambda will basically run your app enough times to handle all the requests.
You have to be careful when you design it such that you don't rely on subsequent requests coming to the same machine, but you can also design it so that if they do come to the same machine it will work, using a tiered cache.
> And then, when you start growing, that's when you break out an independently scalable part of the system into a microservice (with it's own data store!).
Wasn't this the rule of thumb for microservices from the very start? That the main driving force is to ensure separate teams work independently on stand-alone projects they own exclusively? Scaling tends to only become an issue when the project grows considerably.
I don't think so. One of the very first things that people learn when onboarding into microservices is the microservices tax, whose corollary is don't onboard onto microservices if there isn't a clear significant win that justifies all the tradeoffs.
Moreover, the main selling point of microservices is organizational. That's also right in the first lesson on microservices. If you're not working in a large team who owns a large project and wants to lower managerial overhead by splitting up the team, you need to try very hard to find a justification to peel off specific responsibilities from a service into an independent microservice. That's microservices 101.
Those who jump onto the microservicss bandwagon without having a concrete justification have only themselves to blame, and not any particular architectural style. They are making the piss poor decisions and blaming abstract concepts for their own mistakes.
> break out an independently scalable part of the system into a microservice … The one that may need to scale independently, or that you want to be able to deploy separately.
How often would you estimate that security concerns are a valid reason to isolate functionality into a microservice? Separation of concerns, reduced attack surface and blast radius, etc
>Microservices takes at least 25% of your engineering time just maintaining the platform. It's not worth it unless you can recoup that 25% in efficiency.
My take is, microservices multiply your operational problems so you need a really solid platform/infrastructure/developer experience team otherwise you'll introduce a bunch of new headaches. These include things like CI/CD (building, storing artifacts, testing artifacts, promoting artifacts between environments), observability (metrics, logs, distributed tracing, error monitoring), distributed systems quirks like cascading failures, release/change management and communication, service discovery/routing, automated infra provisioning for things like datastores. Unless you have a really good handle on all these pieces (which I think most startups don't), you end up in an operational nightmare spending tons of time trying to keep the system going.
Of course, the next step is throwing on 3rd party products to try to solve this which adds even more complexity. Throw in Datadog for observability, Istio service mesh for networking/traffic, ArgoCD/kustomize/Helm/Kubernetes to manage all this infra, etc etc
I agree - microservices are a technical solution to a people problem. Stevey's seminal "Platforms Rant" (https://gist.github.com/chitchcock/1281611) touches on this - the reason why Dread Pirate Bezos mandated services was not because Kubernetes was cool[1], but because their teams were too interconnected, moving too slowly, and Conway's law was getting in the way.
Splitting into services siloed each team and allowed them to move independently and by side effect, faster. Not due to inherent properties of (micro) services but because one goes faster by removing the things that slow you down.
As a startup, you do not have this problem. You likely will _never_ have this problem as your default future state is 'dead'.
Do the simplest thing that can possibly work - build a monolith using tools you know that give you the ability to predictably, rapidly iterate on the product.
After you hit some semblance of product/market fit and need to scale, you can do that. Scaling is a solved problem. Premature scaling is not.
[1. This is a joke. Kubernetes wasn't even a thought in Google's eye at this point in history]*
> the reason why Dread Pirate Bezos mandated services was not because Kubernetes was cool[1], but because their teams were too interconnected, moving too slowly, and Conway's law was getting in the way.
Not quite. Amazon's problem was that they were experiencing too much impedance between teams, and some teams were even siloing themselves to the extent they were creating problems for everyone around them. Bezos' diktat was intended to break through a bunch of petty office politics bullshit that was dragging down the company and preventing teams from delivering their work. The diktat boiled down to basically ordering everyone to grant access to the service they owned and provided to external teams that need it, no exception, and would be held liable if they failed to provide it, intentionally or not
The issue isn't blindly adopting microservices or a monolith as a religion or bikeshedding thereabouts.
Architectural and ops concerns lead to dividing an overall service into sufficient and necessary units of abstraction. To efficiently run more than 1 service leads to standardization of automation of the concerns of application platform infrastructure:
- stateful data backup, protection recovery
- configuration management
- OS security
- authentication, authorization, audit, encryption, and identity management
- monitoring
- analytics
- rate limiting
- A/B, etc. feature tests
- sharded deployment
Chopping up 1 service into many more services doesn't make the above concerns go away. Neither does collapsing many into 1.
The internal pieces (and often interfaces between business units) need to be broken down into decoupled units of abstraction. If people want to call that "microservices" or a "monolith", it's kind of irrelevant. Containers of abstraction should serve a purpose rather than hinder it.
It depends on your programming language and how many libraries you are importing. If you use Python you can get startup times less than 200ms. Also, if your app is active, most startups won't be cold (Lambda reuses the VM a bunch of times before recycling it if it's staying active). You can also add a health check API and just poke that once a minute to keep it warm if you want (at the cost of increased invocations).
But yes, if you're doing something that requires a lot of setup, then this pattern isn't for you. But it turns out most web apps are simple enough it can be done this way, especially if you're already using a pattern where the frontend is doing a lot of the heavy lifting and your API is mostly just marshaling data from the client to the database with some security and small logic in between.
The name microservice is a fanatical view of how a service should work. That is the problem. If you say you made a service, then it is not a "micro"-service, so you are lacking "experience" with microservices :) Therefor we must build microservices to have jobs.
In reality, you need services that are engineered properly. They can be monolith or not, can share databases or not, can live on the same server or not... depends on the situation. But, as soon as you say the work "micro"service, you are doomed because it won't be a microservice if it does not adhere to millions of articles on the internet saying how to it should behave...
A post like this pops up every few months arguing one way or the other.
I have lived in absolute nightmare monolith systems with so much cyclic dependency that you couldn't move without breaking something. So monolith isn't the answer. I have also seen many instances where people have completely missed the underlying principles of microservices and have ended up with an equal nightmare.
Neither is a hero or panacea, what matters is to understand what you are building and why and get the interfaces correct. Then understand the principles of different architectures before moving forward.
If you fail on this, you get predictably a nightmare. There is plenty of nightmare code sitting on this planet everywhere we look. My recommendation isn't to vilify either architecture but to understand their strengths and weaknesses.
If you have an expert in design / architecture and know the domain problem / scope / boundaries, go for microservice. Otherwise, monolith.
A good microservice is better than a good monolith since the boundaries are clearer. However a bad microservice are many times worse than bad monolith since on top of the wrong scope, intertwined boundaries and other problem from bad design that exists in monolith, now you're faced with additional infra challenges.
I feel this, working on a small team moving things to microservices. My primary problem is observability. It's become a huge chore figuring what, exactly, is going wrong in production when something goes wrong. It's not enough to tail the logs of some distributed application, I need to tail the logs of several distributed applications where there messages are interspersed with one another. I suppose when we get some way to visualize these traces - tooling - it'll be okay. But, small team, limited human bandwidth, and we don't have this tooling in place yet.
The monolith, in contrast, had NewRelic integrated years ago. There were performance problems with this monolith which have been mostly solved through indexes and a couple of materialized views. Trivial to figure out what is going wrong. The code may be old and full of race conditions, but solving problems isn't difficult.
I dread dealing with multiple separate database instances each backing their own microservice when it comes time to upgrade those databases instances. I was hoping for a single database instance with multiple databases, but that particular architecture isn't on the menu. :\",
Are you not using cloud? Because the cloud providers provide centralized logging, so all you need to do is pass a request id between services and include that I’d in log entries, and you can trace requests across services.
Take a look at the OTEL (Open Telemetry) tooling and libraries. Or Grafana stack/offering with Prometheus, Tempo and Loki. Centralized logging and service calls/code execution tracing is not exactly new. It is often an afterthought.. and then you get yourself is this kinds of unpleasant situations.
And since you didn't implement correct tooling from the start, your team is even smaller and more limited... because you have little to zero idea on what your services are up to.
As per db instances... you upgrade them one by one. Unless there's some really bad bugs present (security or otherwise) there's no rush in upgrading stuff just because.
Been working for 6 month at a company with ~150 engineers that uses Microservices.
As a simple software engineer I love it, It allowed me to easily get right in to parts of the system and conceptually understand the boundaries between different systems and business unites.
But it sure adds complexity - complexity that I fortunately do not need to deal with... Yet.
I don't quite understand the mentioning of node at the start of the article as a cause of this. You can write a monolith in node just as well as you can write a web of microservices. That said I agree with the sentiment of this article, it's very frustrating to deal with in any company. Especially when these microservices aren't versioned under the same repo and/or are hosted at different providers causing unnecessary latency.
How would you sell Node to other old guys? Especially compared to Go.
I try to favor simple, efficient software. I used to dislike PHP because it was so messy and inefficient, until newer frameworks like Django made it look like it was slim and snappy.
Maybe, in this race to the bottom, Node stands out in a way I hadn't seen, so I'm genuinely interested.
> How would you sell Node to other old guys? Especially compared to Go.
JavaScript has a huge business case in the front-end. Going another direction is possible, but you're really fighting against the grain to do so.
Go node/bun/deno and you can build back-end and front-end with one language. This has pretty significant advantages. Particularly if your goal is small teams (or solo) with "full stack" style devs. There are fewer siloed specialists required in the org.
Language concepts, syntax and patterns are a significant part of the cognitive load needed to develop.
If you're in a siloed org with specialists anyway, there is minimal draw.
I've done things like have a backend I brought into the front end to mock an immature system. Shared code between the back end and front end for offline-first systems. You just can't do that kind of stuff when you don't share a language between front-end/back-end.
They argue that Node tricks frontenders into thinking they can do server-side, then says they 'huff and puff' when this is pointed out to them. It's an incredibly patronizing argument to make.
One piece that I removed from the draft was about how I wanted to join in on the Node fun. In 2014, when I was writing very simple async Python Tornado code with "await", I saw the mess that was Node asynchronous development, and I was truly bewildered by why it was considered revolutionary and cool.
I also know people that spend 40k euro a month to run the database of their rails app on the biggest instance they can find on aws. The traffic is seasonal and the workload is mostly one way payment traffic. But hey, they are still making money so good for them. I've personally been exploring a more flexible architecture for another project using a combination of cloudflare workers, storage and k/v store. The hardest thing about it currently is that you have to engineer everything yourself to be specific to your problem. Either way, both monolith and microservices have their challenges, I hope to find something that can avoid both but it feels like a stab in the dark.
I can agree with this sentiment now after working at a handful of companies and many interpretations of micro-services oriented architecture.
The one thing I don’t miss about monoliths, especially at low budget/low rent firms, is that the project often exchanges so many hands/owners/teams between bottom bid contractors and “rockstars/ninjas”. This often leads to the usual god classes, half assed abstractions, massive utility classes, and half assed code refactors.
Having taken part in building the Wix microservices architecture, I have to say that I understand and accept the critique of the article. Microservices is not a magic architecture, it solves some problems with the price of others.
When we stared Wix, we stared as a monolith - at 2006. In 2008 we split this monolith into two services due to different SLA - building websites and serving websites.
In 2011 we stared splitting the monoliths into micro services to solve software delivery pipeline - ability to move fast and scale with number of developers. Today we have about 20,000 deployments of new software every year, with over 1000 developers.
At Wix we are constantly investing a lot to maintain the microservices architecture and make it productive for our developers. We are making tools and utilities to return DRY into microservices architecture (think CDC support, GDPR support, Monitoring support, Scaling, Access Control, etc.).
My takeaways -
* Microservices do not come for free. You have to invest in making it work for you
* When building a startup - build a monolith. Break it down later when you understand what you need.
* We as an industry do not understand how to build micro services. There are a lot of fundamental aspects of microservices that are not commonly known / understood or ready as of the shelf solutions.
The author would disagree with you; I think they think it's more like:
Node engineers: "lEt'S bUiLd SoMe MiCrOsErViCeS!!!!! YAY I'M CODING!!"
Chad UNIX architect who won't code in any language that doesn't give you the tools to blow your own foot off just printing a string to stdout: "No let's keep adding functionality to this one codebase until it's as complex as a suite of microservices."
390 comments
[ 5.1 ms ] story [ 1033 ms ] thread> Scale is measured in millions of instances: On 2022/12/21, the microservice topology contained 18,500 active services and over 12 million service instances.
They do go on to say that microservice as a concept is poorly defined. Are you suggesting that they mostly have 18500 of what we’d normally consider monoliths?
My theory is to make devops as simple as possible and push complexity into application code.
This includes a rule to use cloud VMs but use the minimum possible cloud services, ideally none, especially not things like lambda/ cloud functions.
There's a second trade off. Overall complexity. The application as a whole is more complex when split into two than when existing as one.
So three things happen:
1. Complexity of the individual microservices goes down.
2. Dev ops complexity goes up.
3. Overall complexity of the entire system in general goes up.
Then trying to work out where the problem is spans both DevOps and development and suddenly you are in a world of hurt.
Erlang and Elixir are unusual in that you can have independent “microservices” running within a monolith.
Thankfully I am working on the frontend side of things, but this is going to be hell when when we launch.
Engineers need something to talk about in their OKRs. One strategy is to refactor things into microservices.
Honestly, given the opinion of frontend on this forum, I'm surprised that you're thankful! I am 100% certain everyone here would take any kind of backend work over the hype-driven mess that is frontend development.
Comparatively, there are far fewer backend positions available that are more than basic CRUD, even if it those CRUD operations are gussied up with a bunch of needless complexity juggling sources of truth across a bunch of different services.
There's plenty of drudgery out there, and some interesting work in every field, but on the whole I personally have found frontend product development- product configurators for engineering / manufacturing, network visualization, visual design tool development, and data driven interfaces to be more fulfilling work.
To each their own, but maybe you might want to be a little more aware of the people that you work with when you feel like saying something condescending.
Edit: FWIW, you'd have to be blind to not see hype driven development in backend, especially in HN. "Rewrite it in rust" is the meme that continues to keep on giving.
A dozen micro services per developer on rickety infrastructure serving data for a pixel generator built on top of 5 years worth of JS hype cycles. Why are we doing this to ourselves? It is just so stupid.
The back-end is becoming an even more hype-driven mess these days. What with k8s and dozens of micro-services and controllers and telemetry-everything. And people don't even listen nowadays if you try to propose something simpler.
The front-end is at-least constrained by browser limitations and the client device. One can only do hanky-panky upto to a point. But at the backend, "throw another cloud machine" at it has become the de-facto solution for most performance issues.
On the other hand, there's another project, which is a monolith. Around ~15 engineers. Incidents happen very rarely. The infrastructure is simple: just one beefy server without k8s (but there's a standby replica of course), so the team often does maintenance of the server on their own instead of a separate SRE/production team. The monolith is modular so it enjoys same advantages as microservices when it comes to module isolation etc. Highload functionality is moved to separate Go services only when needed (currently 2 or 3 satellite microservices). All changes happen in the same codebase so features are released faster and it's easier to reason about.
I'm currently on the second project and whenever I attend architecture reviews of the first project, I'm shocked how overcomplicated the first project has become. A simple addition of a new object property can end up in 5-10 engineers arguing for hours whose microservices are affected and how they should best communicate.
I think most people fail to realize microservices are what you get from an iterative process. You pull out of the monolith the things you need to scale eventually.
Microservices need plenty of basic infrastructure that you don't want to mantain unless you need to. Most companies don't have an adequate platform to support the pattern anyway.
But it will definitely be easier to debug and refactor by those who come after.
My personal anecdote is, watching from the sidelines with face cupped in hands, watching a monolith being broken into hundreds of microservices at the cost of decreasing performance, increasing complexity and 90% of the traffic still hitting the monolith. Of course there's no time to do anything for the customers any more because everyone is updating software versions in all the microservices and promoting that "it'll all be fine if we just move one more thing to a microservice". The architectural vision will be complete in 2130 at this rate.
I want out from this demented ego driven fashion show.
Why make only one service when you can bill more.
After a downsizing, we are now left with 15 services, 5 different programming languages and complicated kubernetes setup. We have less than 10 developers that has to maintain this. In addition we also have a separate flow for managing data and BI.
Either some startups go rampage with it and deploy needlessly complex stacks because the tooling makes it easy, hence a headlong rush into more hidden complexity to solve problems. Or they use it for the hype and merely deploy a couple monoliths with it, which Compose / Terraform would do just as well without the maintenance burden.
I agree. The worst I've seen is with contractors/consultants, because the incentives are particularly bad.
They often stay with the project only for a limited time and "grokked customer's business/domain logic" is not something you can put on your resume and the typical 6 months are too short for that anyway. But there's enough time to introduce a schema registry or other piece of tech which can look pretty good on your CV. Jump onto a different customer, rinse and repeat. I call this "technology-oriented mindset" vs. "goal oriented mindset".
At least we solved the problem of decreasing performance the old fashioned way: by throwing more RAM and CPU power at the problem.
Microservices was invented nearly 20 years ago.
At what point is this argument that it is some new, risky approach that is going to wow hiring managers going to stop.
It's actually ridiculous now.
The author of TFA kinda hits on this in the first paragraph:
> absurdity of the state of the current tech culture. We laugh, and yet bringing this up in a serious conversation is tantamount to professional heresy, rendering you borderline un-hirable
As soon as I read that, I felt I knew what the rest of the essay would be about, and I was right. It's about culture.
A culture of insecurity, wannabeism, fads, in-groups cliques, so-called "best practices", megalomaniacal levels of scaling, over-reach..... let's all admit shall we (at least privately) , that "tech" has an _ugly_ culture, that we've lost those early joys of solving problems. We've replaced that with a culture of making problems so we can look good by "solving" them. Just realising that would bring enormous economic benefits.
In wider human culture, tech is looked down on, for good reasons. At worst it is a proxy battleground for personal whims and ideologies in which one group of people set out to impose upon another larger group, their particular idea of how the world should 'work'. Code becomes our poor man's mathematics, apparently adding rigour and substance to what is arbitrary and manifold. The helpless dependency of those who must use what we create, makes that a very abusive relationship.
It's a shame that Gerald Weinberg's concept of Egoless programming came at a time before the "boom", and so couldn't address the devastating effects of ego-driven development on the wider culture. It's more about the internal dynamics of the development team.
[1] 1971, The Psychology of Computer Programming
A team of N engineers requires N² coordination. Large teams get mired in endless meetings, email, design reviews. Small teams are more effective, but struggle to maintain large systems.
Splitting a system into subsystems allows each team to focus on their piece of the puzzle while minimizing the amount of peer-to-peer coordination.
Yes, microservices add complexity and overhead, but this approach enables a large organization to build and iterate on large systems quickly.
Just keeping things neat doesn't go nearly as far as a separate process on separate machines. Monolith might be better but I don't think it's a situation where you can have it all.
The blast radius is a single request, right?
In every likelihood it _is_ a separate process on a separate machine.
With micro-services if you screw up a deployment the service is down and if your platform is well designed the system will be degraded but not down.
With a monolith the whole system is down.
No, you are just at reduced capacity.
The moral of the story is not to roll out to 100% right away.
Bit hard to do schema evolution with a staggered rollout.
That said, loads of folks do microservices with a shared dB.
In theory, but when the getUser service is down, your app is probably broken as well.
If you have long running requests (something beside small rest calls like large data transfers) then any being served by that same process are taken down.
Good CD is even more important if there are more services to deploy
If the codebase is different you can force the separation not just ask nicely to keep the code well modularized.
for example: golang - multi-module workspaces javascript - yarn workspaces
With this setup, you can create modules a, b and c. And then add restrictions about which which modules are available to a given module
How about moving things into different folders. Have you thought about that?
Why do you have to modularize it with a whole new repo, a whole new docker set up? Just use a folder bro.
You encapsulated data under one service and you find that it's actually utilized much more in another service. This happens a lot.
The longer you can centralize your storage the better and easier everything is.
Where I work we run a massive MySQL database that has a pretty simple sharding rule, so I don't believe that you need to split the database even at scale.
This thread is very painful, feels like CS101 coming in to tell us why we don't need relational databases and that we can just shove it all inside mongo.
Multiple databases can "work" but you get the problem I described above.
You know when you go to a company and eventually you see idiot practices or technical debt? Unfortunately this belongs to that category. You seeing it at dozens of companies doesn't validate anything except for the fact that bad patterns are common.
It also validates the existence of people who often do things poorly and don't know it (aka you).
Of course it "works". But you can do better.
Splitting data into several dbs when one db suffices is called "extra work". Good results can arise from "extra work". But the extra work wasn't necessary.
You have to think like this: Sure multiple dbs work. But would 1 db in it's place work just as well?
The top commenter here from Netflix one of the people who promoted the whole microservices thing is telling startups to go monolith for a reason.
Startups shouldn't start with microservices. I've worked for (or led) a few different ones, and I've always created a monolith while avoiding other kinds of premature scaling (sometimes even "folder, bro" is overkill). Doesn't mean I'd run things that way with 200 people owning systems with evolving requirements for 10+ years, or even three separate dev teams of 5 people.
Also if by "decoupling data and code" you mean decoupling data from code, that's not the goal. It's to decouple one team's data from another team's data, and to properly abstract it.
The only reason our mono-DB even lasted this long was because previously, very few requirements were changing over time. So when people say their big org is monolithic, I can believe it. It works in some scenarios, but probably not most.
Use a schema bro.
You can do this with two schemas. Think of it as folders in databases. Then give different permissions to users per schema.
PS: I'm not saying using one DB with microservices, it's still with the monolith
If you have multiple things sharing one data store, having ownership over the structure of the data is very difficult.
Because this is the standard architecture you will find for most platforms.
At the point where you have to write an API call instead of simply joining to a table, you have introduced orders of magnitude more complexity, failure points, testing challenges, race conditions, coherency issues, code duplication etc into your system.
I don't mind stateless microservices too much. This forms more of a hub and spoke model (many services talking to 1 database). But the minute you bring separate state into the equation it's a chaos factory.
The biggest one in web is regular entity databases vs. timeseries databases designed for analytics.
For this case entity data is just synced to to time series database every so often. Not super chaotic. It does make sense to divide services along those lines.
Though I would argue a folder still works for the web app part.
I really like relational DBs. I'll make a system rely heavily on complex joins. But I'd rather call some other team's API than share DBs with them.
It can be benefitial to maintain well defined interfaces at boundaries between certain parts of your code. It can also produce a lot of work and add complexity. But beyond a certain scale adding systemic boundaries and honoring them isn't something you should avoid.
Devs who do microservices just tend to go too far too early.
You can use folders to decouple code by making code private within that folder. Only publicize parts of the code with your languages version of exporting.
Or you can make it private behind an entire new repo and behind an entire service. Only publicize parts of the code through api interfaces like http and make everything 10x harder for the sake of decoupling code.
I say again, use a folder bro.
This will be my new go-to response when discussing microservices
This does not happen at all. When you break a system into subsystems, all the previous connections that get remapped to new connections between subsystems still need to happen, in order to solve the fundamental problem that the system solves — except now instead of just making the connection directly, there has to be a "cross-functional" meeting between teams and a complicated communication layer between the systems. And if somehow you find a breakdown that requires minimal connections between subsystems, then those connections wouldn't have existed in the original system either, and the N² problem doesn't exist.
There should be maybe one meeting where the caller defines what they need the service to return to them.
1. Merge the two services (and their data stores) into one and cause havoc downstream of either service
2. Burn through your network latency/throughput budget trying to reinvent a DB join across RPC boundaries (and god forbid if you can't batch multiple lookups in a single API call!)
Oh sweet summer child
turns out, they inarguably have a whole lot of experience at high levels in the biggest companies in the world. Either they're straight up lying, or it is in fact possible to resolve things with a single meeting. Why wouldn't that be the case?
The thing is in reality it's not that easy.
> Either they're straight up lying, or it is in fact possible to resolve things with a single meeting.
Something can be done in a single meeting but not everything. But you gave me only two alternatives without anything between so there is another useful link for you https://en.m.wikipedia.org/wiki/False_dilemma
snark aside, just because a logical fallacy exists in an argument, it does not mean the argument is untrue. My experience echoes jedburg's - services should maintain their own datastore and provide api access. This allows them to be decoupled and allows the service to alter their datastore as needed. When multiple services leverage the same underlying datastore, things can get overly coupled. This becomes increasingly important as organizations grow because the coupling means more overhead communicating and aligning around changes.
Jedburg is an expert because he has seen this happen in multiple companies. I have too, but I don't have the speaking/consulting breadth of experience that they have. Credentialing and appeals to authority work because we are squishy humans and we can't verify everything so we lean in on others. That does not mean that authorities are always right, but when their experience and advice mirrors other's experience and advice, there is something to it.
And for the record, in public companies (two of them), I have exactly seen this play out: a well defined api that does not allow others to muddle in their datastore leading to _single_ meetings where new functionality is discussed and agreed upon and later implemented. I have also experienced the pain of multiple teams sharing a datastore when the datastore now needs to be altered which led to dozens of meetings across several teams and a rollout plan that was measured in months and quarters due to coordination needs. I can firmly say that the latter is a much harder (and expensive!) problem to solve in each case I've experienced.
The team that needs data goes to the team that has that data and says "can you make an API that vends the data that I need?". Then they make it. And then maybe there is another meeting with a follow up, or perhaps an email asking for the data to be presented slightly differently.
But if you come to a team with a good use case and some examples, it shouldn't be too hard for them to create an API that vends the data appropriately.
This of course is all predicated on a good microservices architecture where you already have systems in place to handle API discovery and communication between services and proper data separation.
Of all the things wrong with micro-services this isn't one of them.
The "complicated" communication layer is always either REST/JSON or GRPC.
Both of which are simple, easy to debug, proven and require nothing more than a simple discussion over an API contract.
It is, exacerbated by the over-use.
> The "complicated" communication layer is always either REST/JSON or GRPC.
Going over network, which is slower, unreliable, poorly typed. It's orders of magnitude more difficult to refactor a published REST API in comparison to e.g. Java interface within a monolith.
Microservices are generally far from easy to debug. In the best case scenario you have the whole stack locally and can put breakpoints, add logging immediately. But that happens rarely, debug port is often blocked (security), you can't easily modify the code to add some diagnostics without a complex CI dance etc.
But I've never worked on a monolith project that I could run entirely locally.
And this idea that APIs are slow, unreliable and unable be to be strongly typed is nonsense. This is 2023. We have plenty of tooling and techniques to make this robust.
What are some examples of monolith components that couldn’t be run locally?
Im guessing third-party integrations, but perhaps there are other things.
That includes databases, caching, auth, ML models, mock API endpoints etc all pre-populated with sample data then the actual application which for complex, JVM based ones can often fail to launch if there isn't enough memory to pre-allocate.
Many systems I have worked on all of that would simply not fit on a 16GB MBP.
There are few narrowly defined responsibilities which are handled by dedicated services, and it is often more awkward to get them working locally. But because they are so specialized, you need them only rarely.
> And this idea that APIs are slow, unreliable and unable be to be strongly typed is nonsense.
Network APIs are inherently slower and more unreliable in comparison to an in-process method call. Some typing solutions are there, but they are way more awkward (and in some ways weaker) than statically typed monolith interfaces.
On a theory level I would agree with you - I've just never seen that happen in practice.
They solve "people problem" by converting trivial technical problem into complex distributed system problem.
Well, now you have a _Problem_.
FT.com did a recorded seminar session on their microservices architecture and one of the benefits they extolled is if someone wanted to improve on a feature they could just make it all over again and replace the old microservice with a new one. No need to look at the last persons code, just blow it away like it never existed.
I gathered their site is actually a black box filled with hundreds of black boxes of microservices. All a mystery, they either work or they don't and if they don't they fail gracefully quickly.
https://www.youtube.com/watch?v=_qakAUjXiek
At one point it was 35 engineers, but layoffs took it down to 7, at which point we started to get a lot more done. There was just so much less time spent keeping everyone aligned. So many fewer meetings, sign-offs, reviews, plannings, retrospectives, management meetings, etc. Developers had a lot more agency, so they just got stuff done. Technical debt repayment became 50% of our time, as we easily knocked out features in the other half of the time. We kept ruthlessly cutting complexity, so it got faster to add new features.
I'm sure some projects just need more bodies, but I think there's an upper bound to how much complexity can be added to a system in a given unit of time. Adding developers over a threshold will result in the same amount of features per week, just everyone does a little less and spends a little more time on communication.
Repeat up to thousands of developers where adding a single field takes months.
Years ago I did two back to back contracts for two different pharmaceutical companies. They were both about the same size but one had an IT group that was ten times the size of the other. You can guess which project was late and painful.
Microservices are about splitting the application into very fine-grained sub-applications. It's not about modularity in general, it's about making things as modular as possible (up to one function per service). That's why they are micro. Otherwise we'd just call them "services" and nobody would have any problem with that.
Assuming coupling is reasonable. If you have a "distributed monolith", you still end up with all the meetings because every microservice change risks breaking interfaces other people are using.
In the context of coupling, I'd argue the same applies to monoliths. Multiple teams can successfully work on a monolith given architecture where they're not constantly stepping in each other's toes (each team works mostly in their own modules/classes/packages)
I do agree that this is less of a problem for startups so it doesn't make sense to start with a complex microservice architecture. But in large organizations, especially corporates, it absolutely makes sense.
The coordination is still there because a microservice team does not live in a vacuum. They build services based on demand from other teams that typically build web apps, mobile apps, sometimes server-to-server.
Hence, the "independent" team now becomes a roadblock for higher order features.
There's always someone saying that everything is a different tool in the toolbox.
The reality is some tools are honest to god pieces of shit. That's where the interesting content is.
The whole fair and balanced viewpoint is played out, boring and obvious.
Heck, this isn't even the first time SOA has been in the frame of hate. Member SOAP? Member how everyone more or less jumps between doing everything on the server vs everything on the client every year? We've been having that battle ever since networking or things clients have been a thing, the "sportsification" of that dichotomy is older than I am.
I like the other person's take in this thread about how you can largely get the semantic benefits of micro services by making a well crafted monolith. I honestly agree, but I think the followup is "aren't people who poorly break up service boundaries going to go do regardless if the interface is a network endpoint or a function call/dependency injected class?"
Today there are several tools we can use that enforces boundaries in a monolith (Spring Modulith for example). If the project uses one of these tools, it is harder to accidentally cross boundaries and you get many of the same benefits as you get with a micro service.
The big advantage is that if you find out that you made a mistake, your only dependencies are within the same service and is easier to refactor. In a micro service oriented architecture, changes might impact several services and teams that needs to be coordinated. I'm not saying that refactoring a monolith can't be time consuming, but you have at least a better control of the flow of data between modules.
Here's one: In politics we see national and supranational governments take on the job of making large decisions. Style guides and best practices on whether to use microservices or not are, at best, made at a company level.
Would it be interesting if these architecture decisions were made, or at least kept in check by a body that is larger than a company?
Perhaps we could have some laws that dictate that you must (at least partially) understand how software works before you can buy it. Especially for government contracts.
The second you think you are smarter and more informed than the people on the ground is the second you revealed yourself to be a fool.
Because having spent decades working for government and enterprises I can assure you that we aren't all stupid and need laws to protect us from ourselves. Instead we are often placed in really challenging and unique circumstances that drive the architecture and design of what gets built.
For example micro-services often works well in places where you have distributed teams that for security, governance or logistical reasons need to work independently and can't collaborate effectively with a monolith architecture. Or where certain components e.g. authentication, payments need a hard separation from the rest of the platform.
Obviously not all in government are stupid, I was not implying this. I was rather suggesting that some are, and that something might be done to avoid problems with that.
Also, microservices can be great, but when used in the wrong context, obviously are not. We are not discussing that either.
Edit: perhaps you may have misunderstood my comment, and that may be due to my clumsy way of stating it. I was actually suggesting that companies learn from governments.
both AWS Accounts have a bunch of lambdas randomly doing necessary things in a variety of our development environments, but mostly production
a single AWS account is fully capable of catering to multiple development environments
this is hilariously Byzantine
The author posits that if you make pile of crap microservices then all you had to do instead was make a monolith and magically it’ll all be fine.
You can put the same kind of design and engineering work in that results in a pile of crap microservices but if you target a monolith instead then you’ll be golden.
The author enjoys stroking his own ego as he goes (the comment about full stack js devs for example). And yet, there’s very little here in terms of actual engineering. Want some measurements or some data to back up the waffle? Well the author says tough! You just get a diatribe instead.
Keeping the cost to change a system low by managing complexity is a fine goal, but that’s not what’s being proposed here. This article could have been better if it recognised this. This article could have been better if it gave some data - hell even anecdata, a single motivating example, would have been a start.
On my team, I’ll take on a bright enthusiastic front end dev who decided they want to spread their wings and grow into a full stack dev over someone who believes they already know everything and has no growing left to do.
Ask me how I know.
"The author posits that if you make pile of crap monolith then all you had to do instead was make microservices and magically it’ll all be fine."
Having worked in several places that adopted Microservices with nowhere near the Netflix or Uber scale just because it's trendy I can attest to the mess and glacial pace of both learning and development for new team members. If you want to move fast and make something people want, don't pretend you need "webscale", you don't. You're building an apartment block for a few people not a city for millions. The beauty of Erlang/OTP is that it's designed to scale to millions without any extra effort from the individual contributors or fancy DevOps.
You can just horizontally scale a monolith to achieve this.
But I also advise a lot of startups, and you know what I tell them nearly every time?
Build a monolith.
It's so much easier to start out with one codebase and one database, and that will scale for a while. Especially if you use a key/value store like DynamoDB (although you will lose relational functionality that can be helpful at the start). And did you know that you can deploy a monolith to Lambda and still get all the benefits of Lambda without building services?
And then, when you start growing, that's when you break out an independently scalable part of the system into a microservice (with it's own data store!). The one that may need to scale independently, or that you want to be able to deploy separately.
Microservices takes at least 25% of your engineering time just maintaining the platform. It's not worth it unless you can recoup that 25% in efficiency.
My main point is attempting to counter the narrative popular on HN that postgres should be an automatic default. For sure there are many aspects in which postgres is superior, I absolutely do not debate that, especially when it comes to developer experience. But there is much more to it than that when it comes to delivering business value. That's where ops and DBA concerns start to matter, and IMO MySQL is so far ahead in this regard that it outweighs all the other hideous warts of working with it, when you consider the bigger picture of the business as a whole.
Only the mega-cap ones started to pursue other options mostly due to their type of business and bucket-loads of "free" VC money with explicit orders to burn it and get "unicorn" status - which involves hiring thousands of developers in record time and the whole thing turns into a zoo. Which is an organizational problem, mostly not a tech one.
Other than the ones we pretend are the whole Universe, there are thousands and thousands of medium to big companies with billions of revenue who started their product with a monolith and a MySQL database and many still do just that.
You can get really damn far with a fat postgres box.
Ultimately, everything has its limitations and tradeoffs. If we respect them, it's generally a smooth ride. Problem is that we rarely do... within a company (startup or otherwise) under real conditions. There's also a dynamic where we build until the point where something stops us. Tech debt, complexity, over-engineering, under-engineering, feature bloat or antagonism between early decisions and current goals.
There's a self-regulating aspect to this. If architecture is spot on, perfect for the task at hand we can move faster to reach the point where it no longer is.
My point is that postgres is, compared to almost everything else, easy to get from "can" to "likely will" with just somebody with a brain, a manual and google. In absolute terms it of course depends, but the point is relative.
What happens when there is >1 brain involved... or when brainless, manualless decisions eventually get made... or two pivots from now...
I'm not disagreeing with your approach. I agree with it, especially as starting point. I'm cautioning that resilience against complexity isn't about how easy it is to make good decisions when you understand the spec, read the manual and calmly proceed. Complexity and fragility accumulate when one or all of these are absent. How easily you can (and thus inevitably will) make a mess... not how easily you can keep it clean.
IRL situations with regular rdbs, a very common trend seems to be long term drift between schema and spec. The flexibility and approachability of postgres often enables a lot of kludge eventually.
Data stores have this dichotomy between "look how easy" and "is limiting factor" that speaks to difficulties we don't know how to articulate or isolate.
But if you have a small group of folks that are at least as competent as the folks at WhatsApp pre-acquistion, then there really shouldn't be any doubt whatsoever.
That's a success case... beware survivorship bias.
> if there are lots of bozos in the startup
No arguing that quality engineers are fundamental to quality engineering. That said... by this standard, there's no point in having this entire discussion. Every good db/store out there is good. They all work very well if used as they should be, with due respect to tradeoffs. Yet, almost everyone has db problems. Almost every one of these problems occur well within the technical limits of postgres or whatnot.
"It shouldn't be a problem" when it usually is irl is tunnel vision. There is an empirical reality disagreeing with you. Walking into it with "this shouldn't be a problem unless everyone is a moron" is bad strategy. If you can't think of reasons why architecture can and will become a problem, then just assume that you (or some of you, some of the time) are morons, and try to make it moron proof.
Yes, I agree just repeating tautologies is unlikely to be meaningful.
Administrating (in particular, query planning and production operations) databases with tables sizes with magnitude of 10M and more records is challenging, and requires a skill set that is very different from pure development.
One won't get "really damn far with a fat postgres box", unless they're doing very simple SELECTs, which is not the case with modern web apps.
Is it more or less challenging than the alternatives? Is it less challenging enough to add a new tech to your stack, add the required knowledge to the team, etc?
I mean, knowing "enough-to-perform-CRUD" SQL is table stakes for developing on the back-end, but knowing $CURRENT-FLAVOUR-NOSQL (of which there are multiple products, all with such substantial differences that there is no knowledge transfer between using them) isn't, so there's going to be ramp-up time for every dev, and then every dev that is added to the team.
I'm not disputing your argument, I'm just pointing out that, sometimes, it's easier and faster to upskill your PostgreSQL developer to "scale the DB" than it is to teach them how to properly use, maintain, architect and code for DynamoDB and others.
That's fine. You only ( ;) ) need one on the team.
It amazes me that my comment (while admittedly flippant) got voted down.
It really is true that your phone can update a 50M row table about 10K times per second!
That people are incredulous of this is in itself a stunning admission that developers these days don't have the faintest idea what computers can or cannot actually do.
Just run the numbers: 50M rows with a generous 1 KB per row is 50 GB. My iPhone has 1TB of flash storage that has a random access latency of something like 50 microseconds, which equates to 200K IOPS. An ordinary NVMe laptop SSD can now do 2M. Writing even 10K random locations every second is well within mobile device capability, with 50% headroom to "scale". At 1 KB per row, this is just 10 MB/s, which is hilariously low compared to the device peak throughput of easily a few GB/s.
For comparison, that's about 1 gigabit per second in the era of 200 Gbps networking becoming common. That's a small fraction of SSD write throughput of any modern device, mobile or not. Nobody in their right mind would use HDD storage if scaling was in any way a concern.
[1] Indexes add some overhead to this, obviously, but tend to be smaller than the underlying tables.
That's your reason right there, you used rust! It's both performant and secure!
:/
You’ve got a hell of a resume. And been accidentally incredibly convincing in getting many people into many early messes.
Jokes aside, thanks for at least coming around to advise startups sanely.
I’d personally never advocate for microservices until at the scale of Netflix or Amazon or Reddit, and even then only with in-house expertise at your level. Otherwise it’s a nightmare.
Thanks for everything, especially your contributions of sanity.
We all make mistakes!
At least here in the UK almost all of them have microservices listed as a requirement.
Given that at best like 5% of companies would benefit from microservices, that's a terrible spot our industry is in.
Respectfully disagree. At this point you start refactoring your monolith into components and actually look at performance measurements via tracing.
Do not do microservices when you're growing (or ever, most of the time).
And by the love of god, don't split your data. Data is much more complex to manage than code.
I think people feel that if they introduce a new data store, they won’t have to deal with the existing nearly unusable massive data store.
Of course that just exacerbates the problem.
There are downsides to coupling more loosely with data consumers, but it keeps service owners moving without wasting time vending data.
An ETL approach from the start will save you a lot of headache.
You can't have 42 different classes directly poking the User-table for example. You need one clear location that has the responsibility for the data.
If you move the User-table to a different database schema, other places CANNOT touch it because they won't have access to it =)
Will be fun when you decide that you want to have organizations to be linked to orders instead of users.
All this to say, data management and having more than one source of truth complicates your world a lot. The responsibility for writing the data has to be solved at the appropriate layer: only one entity writing it.
Why should your order manager service be tightly-coupled to your IAM service when the only responsibility it has is to store a single ID?
You might be databasing wrong if that results in two sources of truth.
Most data problems can be fixed on the frontend with a few relatively simple graphql queries. /s
At SendGrid, we had an few tables to deal with IP management. Over time, the like 5 tables were in use by 15+ services owned by different teams with competing priorities. When we finally had to scale the database, it took over three quarters of working with other teams while we supported both legacy and the new hotness.
It is hard to see early on where whole teams can sprout up and have domain ownership and to know which areas will be common/platform-like for other teams. But as soon as two services share a table, you should see a train coming at you.
Joins are good, actually, and normalisation should be the default. It's much easier to keep things consistent by default rather than trying to make them somewhat consistent afterwards. When something becomes a problem there is a bunch of tools (from denormalisation or materialised views to specialised datastores) to deal with that, but before that it's just premature optimisation.
No matter how hard you modularize and instrument your service, that won't give you regional deployments, which you absolutely need to have ti be able to drive down customer-facing latencies from many hundreds of milliseconds to a few tens of milliseconds.
How do you pull that off with a monolith when you have a global userbase?
That could lead to hard to find bugs. Did this ever come up for you?
That's really more of a monolith though. One of the key tenants of microservices is data separation -- each service has its own data store.
(sorry this ran away with me)
A dumb example is that if I start with my single codebase running on a single server, I am likely to have a single git repo (foo).
Then I have a genius idea and put all the email handling code into foo.mail and soon I have foo.web and foo.payments.
All is fine as long as I am just checking out HEAD each time. The code running in the runtime is still one big set of code.
If I get creative and put in load balancers it's still a monolith.
But if I split out the web servers from the email servers, then I kind of start to see microservices appear.
I am trying not to be pedantic, but I am truly interested in experienced views on where the pain really starts to appear.
At this point (server to server comms), I should look at mTLS, and centralised logging and all the good stuff to manage microservices.
But how much pain was there before?
What if I was a large company and so hired a dev or two per repo (you know to get that 9 women one month effect). Co-ordinating multiple devs over different repos, even with one monolithic runtime, is painful (experience tells me).
So I am interested in where the break points are, and whether there are easier paths up the mountain?
I do not think that this is what people mean by microservices.
Is the pain in microservices (or APIs and mTLS) or is the pain in managing a team that now has to do customer stuff only ? or email stuff only?
Common path to pain right there — services based on nouns and entities rather than verbs and business processes.
Having a single "customer" service means anything involving customer data has to talk to that single service. If that service has a problem, every other service stops working. You basically have a tightly coupled distributed monolith, so network boundaries instead of function calls between modules.
Think instead of splitting that data to separate services around business processes like billing, shipping, outbound marketing, whatever. Each service owns the slice of customer data required for its processes and can run fully autonomously.
Business changes, and putting rigid firewalls in place without seeing an established pattern of change and without established need for that is a bad idea.
One global database buys you a lot (easy data access, easy transaction consistency, simple locks, even transactional notifications), and you can take it very far performance wise, throwing that away just in case you might need more performance later seems unwise.
These are all very simple things to handle when services encode processes instead of entities
> Business changes
Yes, and then you throw away the code for that outdated process and encode the new one.
> One global database buys you a lot
Absolutely, this is the correct choice for probably 95+% of companies. I will always argue in favour of a monolith and single database.
If someone is dead set on microservices though, they'll avoid a world of pain by actually designing them properly
Yes, subscriber was a Tier 0 service and required a strong reliability and scaling story. They had more rigorous deployment setup and were usually over-scaled for traffic. But they also vended a library that services used to connect to them. That library had intelligent caching and fallbacks, so that even if the service was down, in a lot of cases the system could keep running. And the catalog was the same.
Microservices is very intertwined with how your engineering teams are built. We had a team that worked on subscriber data, one that worked on catalog data, one that worked on making movies play, one that built listings. So we had services to match.
I remember trying to build out a "barium bullet" through very complex intertwined systems - following known customer data and looking for their footprints in all linked systems. It gave some insights on how to redesign systems - like someone else said here, removing complexity is ridiculously hard.
Most orgs don't operate on that scale though and resources would likely be better spent elsewhere
> Microservices is very intertwined with how your engineering teams are built.
I wish more people understood this. Microservices are much more about scaling your organization than about scaling technology. At a certain size, teams can't collaborate effectively enough and splitting out different services for each team makes a lot of sense
Make things as simple as possible, and as complex as necessary.
I can always make something more complex than it is now. As you say, we can take out things from a monolith and make them into a service. It can be hard to do so, sure. But nowhere near as hard, as trying to get complexity OUT of a system once it's in. Everyone who ever tried to revert a bunch of microservices back into a Monolith knows exactly what I am talking about. It usually amounts to the same work as a ground-up rebuild.
I am a long term dev, done lots of SQL, but for the past few years I have been using DynamoDB, and I am using it for my new startup (So I rate it).
Cons - You have to be very aware of your query patterns, and not having ad-hoc queries is a pain.
Plus sides - With on demand billing, its free if you aren't using it - Built correctly, it will scale - No Schema upgrades (This one is massive for me)
On the last point, I really do appreciate not having to worry about keeping schemas upto date across all devs and environments.
We use quite a simple pattern of a table per entity, as opposed to single table design, because it allows us to just use the API at the highest level of abstraction, where you just write your objects to dynamoDB. (You can still do lower level requests to poke values and such like)
At Amazon, relational databases are banned unless you get an explicit exemption from senior leadership. This is the primary reason why. Too many cases of schema upgrades causing outages.
The problem with DDB or other NoSQL applications, like you say, is how much you need to consider your query patterns. The last major project I worked on using DDB, we spent a couple of days just thinking through our query patterns so we could come up with the right database design and data structures. (We still believe it was the right choice, though.)
Now I go with SQLite and some basic python script and pivot from there. Ironically that’s how I used to do it before the micro service fad.
But I do not agree on is more time consuming than the other. It's just time spent on different matters. For the sake of money spent, I have not seen any data on how a monolith outperforms small services. For maintaining the platform, I think 25% sounds a bit high.
Money spent should be measured in many different aspects. One is definitely productivity. And I have seen more stale monoliths than I have seen stale small services. Whether it's better or not is not up to me to judge. I just know what I prefer.
Having been exposed to small service architecture where it has been working really well and very poor, there are few things that stand out.
- Conquer and divide (your monolith over time). - Responsibility boundaries are easier to cope with for developers in a small service since they often don't have generic and yagni abstractions. - Less code to comprehend for a small service and the cognitive load decreases. - Build time, test time, deployment time. - Should lean up against a direct business measurement and value. - Group chatting services into one.
I have yet to see a monolith that over time is not really deteriorating, but I haven't worked with SO or Shopify. We could also argue that Cobol and Fortran is still of good use but time has also changed leaving that style of system development exposed as old and dusty.
But like any software development occurrence, it takes responsibility, mandate and proper leadership to get things done in a decent manner. So if you start by making 3 services that has to chat to find a user profile you probably don't know what you're doing. And 25 people (the Threads image) is not a small team IMO.
Good luck
They all just want a story that you are going to be the next google so they can sell that story to the next guy.
Will that hold in the non zero interest rate world? The AI hype doesn't make me think the world has changed that much.
Something like "uniform-services" or "harmonized-services". Vertically-distributed harmonized services.
I've seen ex-Netflix software engineers taking jobs elsewhere and proposing microservices for systems that would receive little to no benefit from them. In practice, the implementation of microservices in these contexts become a costly solution looking for a problem.
Lots of businesses don't even need to scale anyway; Netflix is a "high customer, low revenue per customer" type of business, but there's lots of "low customer, high revenue per customer" businesses too, perhaps even more than the first one. These are often the type of products where you could quite literally run production on your laptop if you wanted to.
At the last place I worked they built all this microservice bonanza for ... 300 customers... If they ever got a few thousand customers it would be quite successful, and tens of thousands would be hugely successful. What usually happens is that they don't spend that "25% of your engineering time just maintaining the platform", so the platform was impossible to run locally, and production was hanging together with duct tape.
(Aside: in a recent interview I was asked to design a system for "500 million concurrent users" – this is for a business that's mostly in the low-customer/high-revenue type. I still don't know if the test was to point out that 500M is an utterly bonkers number – about a 10% of the internet connected people on the planet – or that they really thought this was somehow a vaguely realistic number for a startup. I answered with "build the simplest thing possible, a monolith, and focus on features and making sure you're building something people want". I didn't get hired, so I guess they were somehow under the misapprehension that you need to design things for 500M users right from the start?)
I did not know. Does anyone have pointers or examples on this?
You have to be careful when you design it such that you don't rely on subsequent requests coming to the same machine, but you can also design it so that if they do come to the same machine it will work, using a tiered cache.
Wasn't this the rule of thumb for microservices from the very start? That the main driving force is to ensure separate teams work independently on stand-alone projects they own exclusively? Scaling tends to only become an issue when the project grows considerably.
I don't think so. One of the very first things that people learn when onboarding into microservices is the microservices tax, whose corollary is don't onboard onto microservices if there isn't a clear significant win that justifies all the tradeoffs.
Moreover, the main selling point of microservices is organizational. That's also right in the first lesson on microservices. If you're not working in a large team who owns a large project and wants to lower managerial overhead by splitting up the team, you need to try very hard to find a justification to peel off specific responsibilities from a service into an independent microservice. That's microservices 101.
Those who jump onto the microservicss bandwagon without having a concrete justification have only themselves to blame, and not any particular architectural style. They are making the piss poor decisions and blaming abstract concepts for their own mistakes.
How often would you estimate that security concerns are a valid reason to isolate functionality into a microservice? Separation of concerns, reduced attack surface and blast radius, etc
My take is, microservices multiply your operational problems so you need a really solid platform/infrastructure/developer experience team otherwise you'll introduce a bunch of new headaches. These include things like CI/CD (building, storing artifacts, testing artifacts, promoting artifacts between environments), observability (metrics, logs, distributed tracing, error monitoring), distributed systems quirks like cascading failures, release/change management and communication, service discovery/routing, automated infra provisioning for things like datastores. Unless you have a really good handle on all these pieces (which I think most startups don't), you end up in an operational nightmare spending tons of time trying to keep the system going.
Of course, the next step is throwing on 3rd party products to try to solve this which adds even more complexity. Throw in Datadog for observability, Istio service mesh for networking/traffic, ArgoCD/kustomize/Helm/Kubernetes to manage all this infra, etc etc
Splitting into services siloed each team and allowed them to move independently and by side effect, faster. Not due to inherent properties of (micro) services but because one goes faster by removing the things that slow you down.
As a startup, you do not have this problem. You likely will _never_ have this problem as your default future state is 'dead'.
Do the simplest thing that can possibly work - build a monolith using tools you know that give you the ability to predictably, rapidly iterate on the product.
After you hit some semblance of product/market fit and need to scale, you can do that. Scaling is a solved problem. Premature scaling is not.
[1. This is a joke. Kubernetes wasn't even a thought in Google's eye at this point in history]*
Not quite. Amazon's problem was that they were experiencing too much impedance between teams, and some teams were even siloing themselves to the extent they were creating problems for everyone around them. Bezos' diktat was intended to break through a bunch of petty office politics bullshit that was dragging down the company and preventing teams from delivering their work. The diktat boiled down to basically ordering everyone to grant access to the service they owned and provided to external teams that need it, no exception, and would be held liable if they failed to provide it, intentionally or not
Architectural and ops concerns lead to dividing an overall service into sufficient and necessary units of abstraction. To efficiently run more than 1 service leads to standardization of automation of the concerns of application platform infrastructure:
- stateful data backup, protection recovery
- configuration management
- OS security
- authentication, authorization, audit, encryption, and identity management
- monitoring
- analytics
- rate limiting
- A/B, etc. feature tests
- sharded deployment
Chopping up 1 service into many more services doesn't make the above concerns go away. Neither does collapsing many into 1.
The internal pieces (and often interfaces between business units) need to be broken down into decoupled units of abstraction. If people want to call that "microservices" or a "monolith", it's kind of irrelevant. Containers of abstraction should serve a purpose rather than hinder it.
https://www.pgcon.org/2016/schedule/attachments/437_not-your...
Could I get more info on this? All I've found so far is that it's an anti-pattern. Isn't the startup time a killer?
It depends on your programming language and how many libraries you are importing. If you use Python you can get startup times less than 200ms. Also, if your app is active, most startups won't be cold (Lambda reuses the VM a bunch of times before recycling it if it's staying active). You can also add a health check API and just poke that once a minute to keep it warm if you want (at the cost of increased invocations).
But yes, if you're doing something that requires a lot of setup, then this pattern isn't for you. But it turns out most web apps are simple enough it can be done this way, especially if you're already using a pattern where the frontend is doing a lot of the heavy lifting and your API is mostly just marshaling data from the client to the database with some security and small logic in between.
In reality, you need services that are engineered properly. They can be monolith or not, can share databases or not, can live on the same server or not... depends on the situation. But, as soon as you say the work "micro"service, you are doomed because it won't be a microservice if it does not adhere to millions of articles on the internet saying how to it should behave...
I am sure there is points in the article someplace, but it is hard to resolve with all the hypocritical half-digested corn chunks.
Have a glorious day =)
Many users of microservices would be better off with a monolith. Many users of NoSQL DBs would be better off with Postgres.
A good microservice is better than a good monolith since the boundaries are clearer. However a bad microservice are many times worse than bad monolith since on top of the wrong scope, intertwined boundaries and other problem from bad design that exists in monolith, now you're faced with additional infra challenges.
The monolith, in contrast, had NewRelic integrated years ago. There were performance problems with this monolith which have been mostly solved through indexes and a couple of materialized views. Trivial to figure out what is going wrong. The code may be old and full of race conditions, but solving problems isn't difficult.
I dread dealing with multiple separate database instances each backing their own microservice when it comes time to upgrade those databases instances. I was hoping for a single database instance with multiple databases, but that particular architecture isn't on the menu. :\",
And since you didn't implement correct tooling from the start, your team is even smaller and more limited... because you have little to zero idea on what your services are up to.
As per db instances... you upgrade them one by one. Unless there's some really bad bugs present (security or otherwise) there's no rush in upgrading stuff just because.
As a simple software engineer I love it, It allowed me to easily get right in to parts of the system and conceptually understand the boundaries between different systems and business unites.
But it sure adds complexity - complexity that I fortunately do not need to deal with... Yet.
I try to favor simple, efficient software. I used to dislike PHP because it was so messy and inefficient, until newer frameworks like Django made it look like it was slim and snappy. Maybe, in this race to the bottom, Node stands out in a way I hadn't seen, so I'm genuinely interested.
JavaScript has a huge business case in the front-end. Going another direction is possible, but you're really fighting against the grain to do so.
Go node/bun/deno and you can build back-end and front-end with one language. This has pretty significant advantages. Particularly if your goal is small teams (or solo) with "full stack" style devs. There are fewer siloed specialists required in the org.
Language concepts, syntax and patterns are a significant part of the cognitive load needed to develop.
If you're in a siloed org with specialists anyway, there is minimal draw.
I've done things like have a backend I brought into the front end to mock an immature system. Shared code between the back end and front end for offline-first systems. You just can't do that kind of stuff when you don't share a language between front-end/back-end.
Is it opinionated and condescending? Absolutely.
- Performance tuning - Migrate old data out of the DB - Delete unused indexes - Improve queries so they return less unnecessary data
The problem is usually one massive table or 2-3 unoptimised queries.
No need to rewrite the application to use NoSQL
With seasonal load in AWS it’s really easy to change the DB instance size manually. No need for auto scaling.
The one thing I don’t miss about monoliths, especially at low budget/low rent firms, is that the project often exchanges so many hands/owners/teams between bottom bid contractors and “rockstars/ninjas”. This often leads to the usual god classes, half assed abstractions, massive utility classes, and half assed code refactors.
When we stared Wix, we stared as a monolith - at 2006. In 2008 we split this monolith into two services due to different SLA - building websites and serving websites.
In 2011 we stared splitting the monoliths into micro services to solve software delivery pipeline - ability to move fast and scale with number of developers. Today we have about 20,000 deployments of new software every year, with over 1000 developers.
At Wix we are constantly investing a lot to maintain the microservices architecture and make it productive for our developers. We are making tools and utilities to return DRY into microservices architecture (think CDC support, GDPR support, Monitoring support, Scaling, Access Control, etc.).
My takeaways -
* Microservices do not come for free. You have to invest in making it work for you
* When building a startup - build a monolith. Break it down later when you understand what you need.
* We as an industry do not understand how to build micro services. There are a lot of fundamental aspects of microservices that are not commonly known / understood or ready as of the shelf solutions.
Noobs: microservices can be useful
Advanced and cynical: microservices are a complexity nightmare
Advanced and experienced: microservices can be useful
Node engineers: "lEt'S bUiLd SoMe MiCrOsErViCeS!!!!! YAY I'M CODING!!"
Chad UNIX architect who won't code in any language that doesn't give you the tools to blow your own foot off just printing a string to stdout: "No let's keep adding functionality to this one codebase until it's as complex as a suite of microservices."
(According to the author, at least.)