I don't understand why people treat microservices or monoliths as either or. If I find my various projects using the same functionality over and over, then I'll split that off as a microservice to serve my monolith projects of various sizes.
It's like functions: don't turn code into a function until you need it in three different places.
I agree but the trend still seems to prefer one or the other and that's why it's so important to talk about the potential downsides instead of closing our eyes and saying everything is gonna be better if we do microservices.
Maybe it's just the activation energy. Until you have the setup for making a microservice, you're kinda limited to a monolith. Once you pay the setup cost though, making incrementally more services is trivial so you end up on the other end of the spectrum.
In my opinion implementing strong interfaces and good modularization is something, that we should talk about more, than doing Microservices. In the end it might be easy to rip of a Microservice, when needed, if the code is well structured.
> In my opinion implementing strong interfaces and good modularization is something, that we should talk about more, than doing Microservices.
This is my position as well. Strong interfaces, good logical separation of function under the covers, etc. should allow splitting off things at whatever level of micro you prefer.
This is what I do in practice. I've seen it called a "distributed monolith".
One of the good reasons to spend time with Erlang or Elixir is it'll force you to learn how to write your programs with a variety of actors. Actors are generally easy cut points for turning into microservices if necessary. As with many programming languages I appreciate not being forced to use that paradigm everywhere, but it's great to be forced to do it for a while to learn how, so you can more comfortably adapt the paradigm elsewhere. My Go code is not 100% split into actors everywhere, but it definitely has actors embedded into it where useful, and even on my small team I can cite multiple cases where "actors" got pulled out into microservices. It wasn't "trivial", but it was relatively easy.
That's where a lot of the fun of Elixir comes from I think. It's viscerally satisfying to split off functionality into what almost feels like an independent machine that happens to be in the same codebase. It clicked in a way normal object oriented programming never did for me, I guess since it's not feasible to mint 10,000 genservers to use as more complicated structs.
haha I was thinking the same thing. The code base of my startup is a monolith but in reality its a fork on request webserver sending messages to a collection of genservers actings as services.
In essence, creating a microservice with elixir is about the same amount of effort as adding a controller in rails.
Do you know what kind of software has an absurd level of horizontal scalability by default? Web servers.
The idea of splitting your web servers into multi-tiered web servers for scalability is, well, weird. Yet, somehow it's the main reason people keep pushing it. Even this article repeats this.
There's nothing on microservices that adds scalability. They make it convenient to deal with data partitioning, but it's an ergonomics change, not one of capability.
I guess the idea is to split the workload into parts so if one part of the workload is proving to use more resources, it can then get a larger dedicated pool of resources.
Plus being able to choose different tech for different types of work, and also being able to split between teams when your org gets more vast.
Though I guess this is more about using services than micro services, per-say.
Hum... You won't model scalability and concurrency using Turing machines. The Turing model is all about theoretical computation and those two parameters are all about real limits.
What you have on your desk isn't equivalent to a Turing machine.
I never got the “scalability” argument of microservices. You can trivially deploy multiple instances of your monolithic web application - chances are you’re already doing so by running multiple workers/threads in your application server. Spreading that to other machines is trivial.
The real issue is in scaling the data store. Microservices typically work around that problem by each having their own separate database, but nothing prevents your monolith from also talking to different DBs.
Monolith becomes a problem when it becomes too big to build/test/deploy/debug in a sane fashion.
And when you have alternatives, great. But for example game devs don't. Or operating system devs. Though of course these are all active areas of ongoing research (for decades!).
If the problem/subject were that easy we would have already solved it and it wouldn't be a hot topic.
And since it's likely a nonlinear problem it's hard to map the problem-solution space, hard to get a good mental model for it, so we use the next best thing: stories. We have success stories and parables on what not to do. (And whole conferences dedicated to telling them :))
Building, ok, but for testing or debugging microservices aren't really an alternative. Just like scalability, they add no new capability here. The most they can do is adapt better to some set of procedures than a monolith.
For deploying they are nothing but a very large hindrance.
Is it not better to debug one microservice than the whole monolith? Easy to run locally, easy to reproduce errors, etc. (My last two weeks was working in such an environment hunting a production bug. Debugging it was a pleasure. Clear interfaces, etc. Of course when a small feature requires changing 3-4 microservices the pleasure is definitely less :D)
It's much easier to test them in separation also. Deployments are low risk, not much need for coordination with other teams, etc.
There are components that have a heavier load, so we run more of those. Not really a big difference compared to simply running a few more monoliths and configuring them to only handle certain requests... but that's already a given here.
I don't have strong opinions either way. Teams/companies should do what works for them, and they should spend some time on improving it so they can be more effective. (Because most people/teams/companies have a "good enough" complacency inertia/bias.) And most solutions work in both setups. (Eg. making builds fast requires a divide and conquer strategy anyway. Do you build modules/packages separately and link them in the end? Do you build containers separately? Same thing. Do whatever works.)
> Is it not better to debug one microservice than the whole monolith?
As long as your bug is well behaved and decided to keep itself into a single microservice... and you somehow know what microservice it is, then yes, it's much easier.
It's in fact exactly in Horizontal scalability that it becomes important to have control over exactly what resources you need to scale :)
Let's take a typical monolith. You'll be serving endpoints that:
- Are CPU intensive
- Are memory intensive
- Are I/O intensive
They are almost always heavy on some part, but not all. And definitely they are not uniformly all relying on the same amount of resources.
Now, you scale out horizontally, adding instances of the same size because each of these monolithic instances need to be capable of serving the entire domain without impacting tail latencies. Some part of your application becoming more resource intensive? You'll be bumping up the sizes of the entire monolith because you are essentially needing to do the equivalent of provisioning for peak load.
Contrast this with microservices (which I'm not arguing are a silver bullet), you can run your memory intensive part of memory optimized instances, and run your compute intensive parts on compute optimized instances.
That's the part about scalability.
Now, the more interesting part (for me, personally) is reliability. You are decreasing your blast radius of a bad component/thread/process taking down your entire monolith service, and instead compartmentalizing it into limited subparts of your entire API or application.
Finally, as you mentioned, microservices do help guide you towards better choices when it comes to structuring your data. You can do this with monoliths as well (and should), but it doesn't come naturally, and having a single data store is the main reason I see teams run into scalability issues :)
---
Addition: Some thing I don't see many people talk about is the ability to address tech-debt in Microservices. My experience has been that it is an enormous benefit to have made you domain smaller for when it comes to making changes that require sweeping across the whole codebase. Examples include upgrading language versions (e.g. Scala 2 -> 3, Haskell 8.10 -> 9.2), upgrading frameworks, introducing stricter compilation checks (e.g. TypeScript and strict: true).
These easily end up being either insurmountable or year-long projects in a monolith, where it's very hard to incrementally benefit from the work because they are often by nature all or nothing changes.
It's scalability with regard to parallelism if work streams and business units. One module can respond to market faster if all the other business units and teams don't have to weigh in
It seems over time some nuance has been lost in translation on this. Microservices weren't 'more scalable' than monoliths, they were 'more appropriately scalable'. In other words, you could scale parts independently and shape your infrastructure more effectively for your workload. e.g. If your bottleneck is logins, you scale the LoginService and don't need extra copies of the AppointmentService running to keep up.
Sure, but I think the point is that it's unlikely the bottleneck will be in the web server, but in whatever database you're using for your logins. Just because you have a single program, it doesn't mean it is required to use a single database.
But even if the webserver is the problem, it feels unlikely that you'll end up saving much in infrastructure cost by scaling just the login service rather than just deploying more copies of your webserver.
Not saying it is impossible, but it's unlikely for that to be a good justification to adopt microservices. I'm not saying that there aren't good reasons to use microservices, but I agree that scalability is not a good selling point. I'd actually argue that it's harder to build scalable microservices than monoliths.
> Just because you have a single program, it doesn't mean it is required to use a single database.
I would challenge that. Obviously a well written "monolith" with well insulated modules, separation of concerns etc can use different databases but I heard too many times when I ask "why can't we shard the database" the answer "the data model is one and it's way easier to develop on one single data model that on N models inside the monolith".
Even this premise is fundamentally mistaken. If you have a monolith, and the LoginModule is overloaded, you add more hardware, and it's the LoginModule which makes use of the additional resources. The AppointmentModule isn't going to suddenly start using more resources just because they're available.
Not really, the load gets spread out across your nodes so if you go from 1 -> 2 then the new AppointmentModule is handing 50% of the previous load. Thats just CPU. RAM tends to be stickier and you can easily cost yourself a lot more than "needed" to overprovision in this manner.
How/why would the AppointmentModule suddenly start using CPU? Where does it get called from? Unless a web request comes in that requires it, it shouldn't be called at all. If all your incoming requests call the LoginModule, the AppointmentModule shouldn't enter the picture.
I agree with you for the most part, but to be devil's advocate, I believe the argument is that the data stores don't tend to horizontally scale the same way except in the most 'cdn' like data flows.
I have a theory that auto-wired Dependency Injection in a single DI container is partly to blame for monolith spaghetti. Once an app reaches a certain size and anything can depend on anything else, reasoning about the whole can become difficult.
I think there is value in wiring a monolith together in such a way that each course grained subcomponent exposes a constrained interface into the rest of the system (payments, orders, shipping, customers etc) before needing to break it into distributed micro-services.
Note: I quite like Dependency Injection, I just think the 1 giant bag of dependencies can lead to complexity at scale.
I wouldn't blame (badly implemented) DI for all the problems of monoliths.
I think the real main issue is lack of discipline when dividing the application into modules.
Spaghetti is basically defined as an application where real modularisation does not exist and everything talks to everything.
It is much easier to work with an application when you can abstract parts of it when you are solving your problem. You effectively work on much smaller part of the application.
Spaghetti == you effectively have to take into account possibility of the piece of code you look at interacting with any other piece of code in the application.
Well modularised application == you only need to take into account the contents of your current module and the interface of the other modules you are using.
One reason why microservices sort of work (when done well) is because they force people to think about APIs and how those services talk to each other.
In most cases you could just put these microservices as modules in a monolithic application and expend the effort on ensuring APIs and application structure.
I have successfully rolled back couple microservice initiatives by integrating services into monoliths. This usually results in the team getting back a lot of their time because their environment suddenly became much simpler. Less applications to manage, less network communication, less possible ways for things to break, less frameworks, less resources needed to run the application, less processes (like processes around deployment, release management, etc.), less boilerplate code, and so on. The list is very long.
Of course, when you work on a large monolith vs a lot of small microservices, it is now important to be able to structure your applications. But there is also an opportunity for improvement.
Sure, if you were ready to make an entire separate application for it it will not be worse when you turn it into one package of a monolith?
Also you can build nested structures with modules (like a larger module consisting of smaller submodules).
Divide and conquer. This is how abstraction works in a nutshell. You start with a large problem, divide it into smaller parts (modules), each part you treat as a separate problem dividing it into smaller modules, and so on.
There exists no need to have a flat application consisting of hundreds of sibling modules.
An application might consist of larger modules, then some of those might have smaller submodules (packages), then those might have even smaller ones (classes), then you have methods, then you have statements, then you have function calls, etc.
A large monolith does not have to succumb to spaghetti or ravioli-type structuring. A good developer should know how to structure an application of any size.
This is exactly what Shopify does with their monolithic Rails app [1]. I worked at a company with ~200 engineers that used the same general architecture and I really enjoyed it. We got a lot of the benefits (clear interfaces, teams able to work on their system without having to understand the whole platform, build optimization, etc) without any of the operational headaches that come with microservices.
I notice there's a lot of comments here saying this exact same thing, and it's also what seems most sensible to me. Yet microservices are getting all the hype. Should be hype a thorough modular design more?
They also maximize the number of engineers and managers needed to oversee the whole thing. People are unlikely to speak against something that guarantees their employment.
The nice thing about microservices is that you don't need iron discipline to maintain modular boundaries, instead its just how things work.
I like to analogize it to assembly vs structured programming languages, or C vs Java- you can write the same programs in either, you can write bugs in either, but there are whole classes of pitfalls that get erased by moving from one language to another.
That’s not true. People in industry commonly refer to the consequence of their misguided decision to embark on a microservices adventure as a “distributed monolith”.
> Once an app reaches a certain size and anything can depend on anything else, reasoning about the whole can become difficult.
I have experienced this pain so many times and in so many different varieties. Often, you cannot meaningfully subdivide the problem space without ruining the logical semantics per the business (i.e. bowl of spaghetti).
An alternative is to embrace the reality that circular dependencies are actually inevitable and appropriate ways to model many things.
The example scenario I like to use is that of a typical banking customer. One customer likely has a checking and savings account. For each of those accounts, there are potentially multiple customers (joint ownership). Neither of these logical business types will ever "win" in the real world DI graph. Certainly you can start to invent bullshit like AccountCustomer and CustomerAccount, but that only gets you 1 layer deeper into an infinitely-recursive rabbit hole of pain and suffering. There also exists the relational path, but I have heavily advocated for that elsewhere and it is not always applicable when talking about code-time BL method implementation. Being able to model things just as they are in the real world is a big key to success in the more complicated problem domains.
Instead of trying to control what depends on what, I shifted my thinking to:
> What needs to start up and in what order?
Turns out, most things don't really care in practice. The only thing I have to explicitly initialize before everything else in my current code base is my domain model working set (recovery from snapshot/event logs) and settings. I decided to not use DI for any business services. Instead, all services become a static class with static members that can be invoked from anywhere. This also includes the domain model instance which is used as the in-memory working set. This type just contains an array of every subordinate domain type (Customers, Accounts, etc.). By having the working set available as a public static type, every service can directly access it without requiring method injection. If I was working with a different problem domain (or certain bounded context within this one), I might prefer method-level injection.
Yes - according to every book on programming style you ever read, this is an abominable practice. Unit testing this would be difficult/impossible. But you know what? It works. It's simple. I can teach a novice how to add a new service in an hour. A project manager stumbling into AccountService might actually walk away enlightened. You can circularly-reference things at runtime if you need to. I've got some call graphs that bounce back and forth between customer & account services 5+ times. And it totally makes sense to model it that way too as far as the business is concerned. Everyone is happy.
I think I agree with you as well. Although it is hard for me to picture exactly how you've structured your dependency graph. In any case, extracting the business logic into some sort of static method / class has definitely been one of the only useful things I can carry across projects that works in nearly all use cases. It also makes unit testing the actual business logic extremely easy. That said, you can end up with a static method that takes 20 parameters, which is always fun. But, in those cases, you are lefty dealing with complexity that is intrinsic to the business, rather than complexity introduced through some bad architectural decision, so at least it is isolated.
> That said, you can end up with a static method that takes 20 parameters, which is always fun.
I used to get upset about this, but now I embrace it. If a business method requires certain things to operate, then modeling those things as arguments to the method is totally reasonable. Some business is complicated and messy so we would expect more arguments to be involved. Trying to sweep reality under the rug just makes things 10x harder elsewhere.
In theory it all sounds great and makes total sense but when the company grows to a billion dollar firm and hires engineers to work on the said monolith thats when things breakdown. The founding team was a closet knit team and ensure there is no spaghetti code but the moment you move on from that closely knit group its hard to enforce constraints.
Have worked at multiple companies that started out as a monolith and are still running the monolith in some form or the other while breaking it down into micro services.
I’ve never been part of a monolith that used a DI framework, but I’ve seen quite a few monoliths fail (as in “the project becomes too convoluted and iteration slows to a crawl until the project is canceled or effectively rewritten”) and I certainly believe that one important reason microservices do well is that they enforce the modularity that you describe. That said, a lot of critics of microservices describe similar issues of indiscernible chaos, so either I’ve been very fortunate or microservice critics are gaslighting us. :)
There was another article about building this structure into a single, self-contained executable and decide via command line arguments which piece(s) to use at startup. For some reason I remember "microlith", but it must be another clever word combination because I can not find any relevant HN posts, just one about archaeology...
You can run a single copy of the resulting binary (eg. for testing) that spins up all sub-components, or copy it to multiple machines and start individual parts. The ones which happen to run together can use in-process communication as well, others will have to dial in via remoting.
At that point it becomes useful to look at some indirection/decoupling patterns like the Mediator pattern.
Then instead of having a IService with 15 methods with a construction with 20 dependencies listed (multiply it with say 50 IServices in your project - mmmm spagheti project), you end up with a IHandler with a single method and a constructor that only has say 3 injected dependencies, only what it needs. The trade off is, you now have 100's of small Handler files, some people don't like that BUT you now have a pleasant git commit history per file and most of the code of a Handler file fits on one screen and it is easy to digest. You can also let a handler trigger other handlers if it needs to (via the same mediator). It also fits with SRP.
Auto-wiring 95% of your dependencies still stays intact as the above plumbing will need it to work.
I have done the full theme park ride on monolith->microservices->monolith. Both have ups and downs.
The most important thing I learned is that "microservices" in absolutely no way necessitates "bullshit spread across multiple cloud vendors and other scenarios involving more than 1 computer". What part of microservices says things must be separated by way of an arbitrary wire protocol?
We now have a "monolith" (process) that is comprised of many "microservices" (class files), each of which is responsible for its own database, business logic, types, etc. These services are able to communicate amongst themselves using the simple path of direct method invocation.
If you are looking to scale up your capacity or even make things more resilient, microservices vs monolith is really not the conversation you need to be having. If you are trying to better organize your codebase, you might be in the right place but you also need to be super careful with how you proceed.
We wasted a solid 3 years trying to make microservices solve all of our problems. Looking back, we spent more time worrying about trying to put Humpty Dumpty back together again with JSON APIs and other technological duct tape than what our customers were complaining about.
Currently at my day job, we have a monolithic code base with multiple entry points. If you update a library, you have to update the client or keep the changes backwards compatible. I really prefer this over multiple micro services.
What exactly do you mean by multiple entry points? Do you have multiple processes which run independently but are co-located in the same repository or are you talking about something else?
You can have more than one Main function, just pick which one to use when you compile. In our case it’s PHP, so the API entry point uses a different index than jobs, for example. They can be deployed differently and scaled differently, but operationally it’s just deploying the same code/configuration everywhere and the only difference is routing. When you are writing code, you very, very rarely have to worry about which context you are writing for.
But there's nothing in that about using different entrypoints or routing.
FWIW, variations on the theme:
1. A single binary with a single entrypoint, which can play multiple roles simultaneously (UI, API, scheduled jobs), but where different kinds of request are routed to different pools of instances
2. A single binary with multiple embedded configurations, selecting via command line arguments etc, each for a single different role (UI, admin console, data ingestion)
3. A single (Java) binary with multiple entrypoints (main methods), each playing a different role (live calculations, batch calculations, data recording)
4. A single (C++) codebase building multiple binaries (via CMake add_executable), each playing a different role (calculating prices for potatoes, calculating prices for oranges)
5. A single repository with multiple completely separate applications, with some shared submodules, each built separately, playing a different role (receiving transactions, validating transactions, reporting transactions)
That last one is probably not an example of what you are talking about, but it's closer to the one before than to the first in the list. There is a sort of "ring species" [1] shape to this variation.
We've been doing this too for some "logical" services. For example, we might have a service which has a REST API and but also needs to do long running processing in response to requests via said REST API. The code for both lives in one repo and can share code, data structure definitions, databases etc. One container is made but we deploy it twice with different args. One is set to run the REST API,and the other runs the processing. Both are closely related, but in the cloud we can scale and monitor them separately. It gives a lot of the benefits of "standard" microservices with much less of the code and repo level screwing around. It relaxes the general microservices approach that: one service == one git repo == one database == one container == one deployment in GCP/etc.
Farming out libraries or subprojects to separate teams is an ok way to go as long as you don't mind a single language/build system. Its not as flexible but you maybe get some perf out of it.
You don't get the convenience of a single datastore with simple transactions but it sounds like you prefer the flexibility of services owning their own data.
As it turns out there's no silver bullet. Do what works for you.
If you're a startup, I think it's actually a pretty good design constraint. You're going to end up with a single system to optimize (inevitably in your 110% time because there's no platform team) and a leg-up in shipping with fast and well-tested infra (or at least better tested because it's what you're using everywhere else). Dropping-in and tweaking stuff across many parts of the system is way easier if it's consistent, easily testable wherever you are, and you know that tests for every dependency are being run on each PR.
On the opposite extreme you're talking multiple PRs, languages, test suites and deploys for every new feature. I've watched startups do this and die before finding something people want to buy.
It's not as stupid as you make it sound. It is after all how normal manufacturing works. What you build is constrained by your assembly line. If you want to build something new, start off with redesigning your assembly line. It's time well-invested that pays back in heaps later on.
> If you want to build something new, start off with redesigning your assembly line. It's time well-invested that pays back in heaps later on.
We applied this ideology to our software project. Our build/deploy toolchain is 100% in-house code. Turns out, there's actually not a lot of it required because you can build a monster .net project into a single exe distribution with 1 simple command line operation. This applies to most other ecosystems as well. Everything else is basic file operations and moving shit between computers. No reason to vendor either of these problems out.
The benefits of having bespoke CI tooling are difficult to overstate. We've got automated processes where a project manager can multiselect 20 different customers, select a commit hash, and schedule an email containing a link to a zip of the resulting build that will be delivered to each customer's configured IT contact upon completion - all with a single button press. This process also entails injection of customer-specific configuration that is pulled at runtime from another system into each customer's distribution. The email templates are configured on a per customer basis and email delivery is delayed based upon project parameters.
I can't imagine wiring something like this up in Jenkins or GH Actions.
Sounds like the only hard part is a nice interface to select the customer IDs. You can just pass along that data to select build configs or what have you.
How is the rest of the process not applicable to GH or Jenkins?
You could even have a mono-repo that holds services which run as separate processes, if you need that. That way you still get the benefits of sharing code, types, version numbers, build processes, etc, which seem like the main headaches with the usual approach.
We passed right by this mode of operation while migrating from microservices=>monolith. Everything is a separate process but running on the same box. Ran that way for quite some time. We pulled services into the main process one-by-one. It went about as smoothly as one could have hoped for.
I've inherited a project like this - lots of common code in a shared library that gets used by multiple services.
Doesn't work too badly but things have been split out more than they need to be.
Pretty clearly the pros and cons come down to the application and necessary stability. Need ultra stable? More and smaller microservices usually better. Need fairly stable? Bigger services is fine. Need it to work generally? Build it however you can.
> Need ultra stable? More and smaller microservices usually better.
I don't think I agree with this statement. "More and smaller" also means potentially more routing and communication paths, thus increased latency and complexity.
Pretty sure microservices will be remembered as a horribly convoluted stop gap once we have better language modularization.
They encourage modular patterns which is usually good, but all that plumbing will eventually become unnecessary. I can’t help but remember building 6 versions of each class in the old ejb days whenever I hear microservices hype.
OSGi was supposed to solve this problem. You could start by running all bundles together in the same JVM, then split them across a cluster later if required. But it never quite took off beyond an implementation detail of various application servers.
> PHP, hear me out, each file is basically an independent hot swappable 'service'.
I think that what the OP is referring to is the fact that you can map URLs to PHP files pretty easily and therefore changing what an endpoint does can be as simple as just swapping out a single file for another one.
However, that tends to fall apart with most modern frameworks, like Symfony, Laravel, Lumen and Slim, where you will typically invoke a main application file which deals with all of the bootstrapping first (reading configuration, DB connections, logging etc.) and then will hand off control over to a router, which will usually map to a controller of some sort, which may have the actual business/view logic spread over other referenced files, the way as other similar technologies like (Ruby on Rails) do it.
Of course, that's not a bad thing, especially since it allows you to follow some of the RESTful conventions, like having path variables without doing rewrites at the web server level, such as /users/12345/photos versus /users/photos.php?id=12345
That said, the whole "mostly single file, maybe with a few common imports" approach can also work fine, as long as your endpoints are kept relatively small. I haven't seen it scale up to large projects that well, at least most of the time.
Even with language modularisation you lose the ability to provision modules independently if they are bundled together in a single service.
Microservices and monoliths are opposing extremes on a scale. There is a range somewhere in the middle which is entirely sensible.
I feel like I've found my greatest success with my service abstractions by separating things out based on resource and availability needs of the functionality.
If you maintain the same interface (same classes, methods, types, etc) it seems to me a module ought to be hot-reloadable within the same process. This is already possible with a loosely-coupled plugin architecture and I could see it being an integrated part of a language.
Yeah, the biggest problem that people get into with microservices is that they allow the communication structure to dictate the app structure. A monolith gives you some refactorability because you can run what would have been "integration" tests locally, as you massage your module boundaries to match the problem that you're solving. So a monolith can become a clean monolith, and then a clean monolith can maybe become microservices if you need the scale attributes.
The basic problem is that before you know where the module boundary is, you cannot have a clear module boundary. So three popular approaches emerge:
(a) DDD "milliservices". Basically define the different sorts of users, come up with a clustering of the different sorts of people that you think will be using your app. People are considered to be in the same cluster if they use the same jargon to refer to things, or different clusters if they both use some word but they both mean subtly different things by it.
(b) Gut-feeling microservices that become feature microservices. I should probably have an "auth service," I don't know what it does but I'm going to be doing auth so that's probably a service. I need to import a git library to contact GitHub, probably there should be a "git service" that handles all communication to GitHub or other git repos. Each of these things exposes some swagger/openAPI docs, maybe we should have a docs service? -- that sort of thing. The danger is that the things that are easy to break off are usually not the core competency of the product, and so there emerges some sort of "core services," one or two big honkers that basically are monoliths. People are aware that they aren't supposed to keep adding to the core services and so new services emerge named after feature requests, hello "sharing service," hello "wallet service." Except the core usually is tightly coupled to these new feature services and they all kind of connect to each other. The idea of splitting the core services into other services to fit the newfound module boundaries becomes complicated by all these ties to nearby features, you are not actually loosely coupled because you do RPCs and you probably code in the expectation of success into these RPCs like method calls.
(c) Every noun becomes a "nanoservice". This is a service that watches just one or two tables (or NoSQL document types or what have you) and exposes a CRUD API for that noun, plus a couple auxiliary verbs to do actual business needs with those nouns. So if you were implementing Git there would be a file service, a tree service, a branch service, a commit service, probably a working tree service... But you just have to "know" that logs and rebases and cherrypicks live in CommitService while diffing for some reason lives in TreeService and adding a file to your working tree requires first creating it in FileService and then handing that link off to the WorkingTreeService which will make needed calls to TreeService, creating your own tree is exposed via TreeService's CRUD but the developers tell you that down that road There Be Dragons and you should not have been messing with that.
Any of these three can be successful but only insofar as you can create new module boundaries and move module boundaries and test to make sure that your users will not notice any performance regressions. Of them I would only recommend option (a), because it gives a really nice place for these tests to live and an intuition that each test should document a user journey for the system.
A company I worked at went all in on option c - there was a random number service (that just wrapped a random number generation library), and sending an email was split across multiple services - one to pull the email request from the database, one to generate the email from the template, one to actually send the email and one to save the templated email to the database.
In reality it worked - each service only talked to the next so in that sense it was okay. But each service was its own binary running in its own container (someone went to a conference and heard that S3 was made up of 500 microservices, to they decided we needed as many microservices as possible).
The bit where it didn't work so well was making changes and testing those changes - the original developers hacked it together then didn't want to touch it after that, so there was no dev environment (just staging environments to hijack) and no functional/e2e tests.
My current team has this pattern. I think we've ended up with two core services, with a fairly thin channel between them, but a constellation of tightly-coupled peripheral services around each core.
To be honest, it works pretty well. It wasn't intentional, and it grew in exactly the haphazard way you describe. But it's not a disaster, or at least doesn't feel like one.
Maybe we should rebrand this as "natural microservices"?
It doesn't make sense for me personally. So I applied all DDD logic and implemented it in my app.
In dotnet there is a useful feature where functionality from referenced projects is included in your main WebApp. Which makes everything pretty clean.
I use application logic over 2 "gateways". Where a gateway is a dummy solution that contains Swagger, global things for that project ( eg. auth verification) and uses the controllers/logic from the referenced projects.
- SPA - Angular that contains a Ocelot gateway on the same domain ( = no cors issues)
- ApiGateway - Contains all the backend api logic and custom integrations with partners.
- ShopGateway - Is included in the Frontend.
- Frontend - the shop itselve. Currently in the process of making this db-less ( everything over API / messages or using the gateway. The gateway is currently a referenced project ( see up), so everything is pretty fast)
In the meantime, i can make "reasonable" quick adjustments and the logic is really flexible.
Note: Db migration from "1 project" to this is not complete yet :( . But it's one of the bigger shifts and i'm already 90% finished before i can start the actual shift to MartenDb from EF / Dapper )
If you want a quick summary on the logic behind, DDD would be the answer.
"What part of microservices says things must be separated by way of an arbitrary wire protocol?"
That's not a requirement, cf. Erlang/OTP.
But to qualify as a microservice, you need it decoupled enough that you can independently deploy and/or restart individual services. Pretty sure that's a minimum to call yourself microservice-based, and hard to do by just using classes in C++ or something.
Arguably you should also have quite a bit of control over dependencies and their versions. If your service isn't ready for python3, it can keep using python2, or whatever. Taking this to the extreme would generally mean a black box that serializes everything over a network, so that any service can be implemented on any stack. I'm not sure that's fundamental to the idea of microservices, though.
I've come to the conclusion that microservices as usually implemented are simply a Conway's law concern.
We haven't had a great track record defining the semantics of component boundaries, but http is both simple enough that it constricts code flow in a particular way leaving a lot of misunderstandings and bike shedding discussions off of the table, and is the ligua franca for external services owned by other orgs so your engineers can be universally expected to understand those semantics. There is a nice decoupling of letting teams make technology choices though. Letting that drift on a per team basis is kind of nice to allow experimentation with different technologies without having to get the whole org onboard.
Maybe I need to go on the talk giving, consultant circuit reselling a restricted view of DI as "hybrid service architecture! Write your code as microservices but DevOps can colocate your apps in the same process if that's more efficient!" Lolz
It's also quite a nerd trap as we developers love to separate components into as small of boxes as possible, but just like I've seen super deep object hierarchies that didn't gain you anything at the end of the day, it's easy to fall into the trap of spinning every tiny bit of code into its own service.
I'm creating my own backend arch right now from scratch, and really trying to look at each network traversal from a "what does drawing this boundary actually get me" perspective. And I'm only splitting that out because the third party protocol being implemented is inherently separate services in an interesting way.
Spot on. Microservices is a way of scaling people. There are other ways of scaling systems.
I would suggest that http not be the go-to mechanism for communicating between microservices. Each should be able to operate to some degree without hard dependencies. If you build using microservices with RPC over http or what-have-you replacing function calls, you've done it wrong. You can usually tell if your 'bounded context' is of the form noun-service or adjective-noun-service. Usually the context boundary should be a process (workflow kind) that serves a user type. Reliable one-way async datastreams are more useful for keeping microservices fed with the information it needs, and it can decide how to utilize that information in a way that makes sense for that service. If you have core 'source of truth' services used by many others, you have a distributed monolith.
There are plenty of monoliths that scale in terms of number of contributors and organizational changes. For example, the Linux kernel. I've worked on massive monoliths that were 20 years old and changed hands and org structures tons of times and nobody ever thought "this needs to be microservices!"
Microservices are not the only way to modularize code.
Also, there is TONS of bike-shedding with regard to HTTP. Which HTTP method do you use? How do you structure your paths? What does the body of the request look like? How to we translate failures into the various HTTP error codes? Do you send a 200 or a 201 status for this particular request? I could go on and on... Even among well established HTTP APIs, there are differences.
> Letting [technology choices] drift on a per team basis is kind of nice to allow experimentation with different technologies without having to get the whole org onboard.
This is very much a double edged sword. I've seen this happen and people just tend to avoid services in languages/frameworks that they don't know. And even if they don't avoid them, they are less productive in them. We have a critical services written in Rust and one guy at the company knows Rust. We have 2 services written in a particular Java reactive framework which only a few people know how to use. Any newcomers to these services have to spend a lot of time learning a new thing.
> There are plenty of monoliths that scale in terms of number of contributors and organizational changes. For example, the Linux kernel. I've worked on massive monoliths that were 20 years old and changed hands and org structures tons of times and nobody ever thought "this needs to be microservices!"
> Microservices are not the only way to modularize code.
Nobody said that it was the only way to modularize code.
However, but to be frank, most of those require greater discipline than most of the engineering teams going to microservices have. These teams don't do a great job at microservices, but probably would do a worse job with a monolith.
> Also, there is TONS of bike-shedding with regard to HTTP. Which HTTP method do you use? How do you structure your paths? What does the body of the request look like? How to we translate failures into the various HTTP error codes? Do you send a 200 or a 201 status for this particular request? I could go on and on... Even among well established HTTP APIs, there are differences.
You can bike shed anything, that's why it's called bikeshedding. Referring to nobody being able to contribute to the design review of a nuclear reactor so the committee just fights over the color of the bikeshed. That being said, the state space of an HTTP API is strictly smaller than a local library's API, and you can generally expect backend engineers to at least know what complete the state space of options looks like and what the tradeoffs are.
> This is very much a double edged sword. I've seen this happen and people just tend to avoid services in languages/frameworks that they don't know. And even if they don't avoid them, they are less productive in them. We have a critical services written in Rust and one guy at the company knows Rust. We have 2 services written in a particular Java reactive framework which only a few people know how to use. Any newcomers to these services have to spend a lot of time learning a new thing.
The other end I've seen is that having the whole org on one tech stack is a single organizational point of failure. It's not an issue for smaller orgs, but if you have more than say, 200 devs working on the same product, you want a little experimentation and a lower barrier to new technology. If something is abandoned you can simply replace it, since it's not your literal entire stack. And forcing your devs to be polylingual I've found makes them better in all of the language they use, even their primary language.
There's something to be said about the benefits of having a serialization boundary between parts of the code that's a natural consequence of microservices, but it's something that could be employed in a monolith just as well. Then if at any point you actually need to split out a module into its own process or machine, it would be trivial to do.
I'm watching a government customer take simple, cohesive systems developed by a small team (4-5 people) and split it up into tiny little pieces scattered across different hosting platforms and even clouds.
Why?
Because it's "fun" for the architects and pads out their resume.
Just now, I'm watching a "digital interactions" project that will have dozens of components across two clouds, including multiple ETL steps. In all seriousness, half of that could be replaced by a database index, and the other half with a trigger.
They're seriously going to deploy clusters of stuff on Kubernetes for 100 MB of data to make sure it "scales"... to 200 MB. Maybe. Eventually.
What kills me is that now that they've made the decision to over-engineer the thing, my consultancy firm can't bid on the tender because we don't have the appropriate experience building over-engineered monstrosities!
The simple and effective solutions to problems we've delivered in the past are disqualifying us from work.
You guessed it: my next project will be an architect's wet dream and will be over engineered just so that we can say on tender applications that "yes, we have the relevant experience".
I'm involved in a (greenfield) project right now where the lead insists on gluing a bunch of Google Sheets/Forms/Docs/whatnot together with triggers and custom APIs (roadmap TBD) instead of just building a simple website in $framework.
Do you feel those hosted solutions are worse than rolling your own website? Why stand up your own website and architecture if you can use off the shelf solutions? I am genuinely curious.
For me it's a judgement call based on how easy to integrate the hosted solution is, how much lock-in there is, how much flexibility I think we'll need, and how fast we need to ship.
One definite disadvantage is testability (which drives stability and ease of refactoring). I can write an automated end-to-end test that will enter some text into a CMS on my site, hit submit, then verify that a post was published. If the CMS is google sheets, I'm going to violate some terms of service and possibly get a blacklisted IP if I do that. Instead I'm going to mock out a response according to the API contract sheets gives me. If they change the API in a subtle way, it may break.
An advantage is that google sheets is probably more robust than my CMS and I'll never be able to look at (or have to fix) their code.
Flexibility cuts both ways: I can't add features to sheets. So if there's a field I really need to validate on the client, I'm out of luck. But also nobody is going to ask for that feature because I obviously can't add it, so that may be good assuming there's a workaround.
Yes. The project has a (dynamic) website as one of the requirements so we will be building one anyways. And if the lead gets his way, a multiple additional APIs to interface with the Google apps.
This reflects my opinion on the current state of the software industry.
The seemingly-infinite VC money being thrown around around means you can create a startup, raise some money and provide a comfortable salary & industry experience for yourself and your friends for a few years, regardless of whether the company "makes it" in the end or whether the business problem is even solvable.
At this point solving the business problem is no longer the primary objective. The longer you can drag out the process of "solving" this problem the longer you & your employees can enjoy the salary and build "experience". So instead of a simple solution that needs a team of 5, you end up with an engineering playground that needs 50 people (and associated managers, scrum masters, etc) spread across many teams just to keep the lights on. The outcome becomes endless busywork, self-inflicted problems to brag about solving on your engineering blog (every startup has got to have one obviously) and next AWS conference to attract more employees looking for this kind of environment and make yourself look "serious" in the eyes of VCs that keep bankrolling the disaster.
The business problem, if it gets solved at all, is a secondary concern since the VCs happily keep throwing more money into the dumpster fire and some bigger idiot might even buy out the company regardless of its inefficiency.
I'm watching a government customer take simple, cohesive systems developed by a small team (4-5 people) and split it up into tiny little pieces scattered across different hosting platforms and even clouds.
Governments are generally driven by a set of requirements that most of us have no inkling of. Distributed deployments are often not just for performance or resilience, but also politics and/or playing the vendors off against each other. It seems daft to us mere minions but there's often N-level chess shenanigans going on near the top.
Lifecycle management is also a big part of it, I think.
My search engine is a hybrid architecture, with some monoliths and some microservices.
* Index - 5+ minute start time and uses 60% of system RAM.;
* Search Server (query parser, etc.) - 30 second start time due to having to load a bunch of term frequency data, low resources, ephemeral state
* Assistant Server (dictionary lookups, spell checks, etc.) - fast start, medium resources, stateless
* Crawler - only runs sometimes, high resources, statefull
* Archive Server- fast start, low resources, ephemeral state
* Public API gateway - 5 second start time, low resources, ephemeral state
+ a few others
A lot of the ways it's divided is along the lines of minimizing disruption when deploying changes. I don't have the server resources to run secondary instances of any of these, so I'm working with what I've got. If I patch the crawler, I don't want the search function to go down. If I patch the search function, I don't want to have to wait 5 minutes to restart the index.
It would certainly be cleaner to say break apart the Index service in terms of design, as it does several disparate things, but those things have a resource synergy which means I can't, not without buying another server and running a small 100 GbE network between them. Seems silly for a living room operation.
We have a big codebase where I work and it’s written as a set of micro services. I’m a bit of a noob but I thought that micro services were always containerized. Turns out we don’t do that and it all runs in a very bespoke and poorly documented VM. Now can we call this a microservices based architecture? I feel like we cannot, it feels like worst of both worlds…
> Starting a greenfield project with microservices is easy.
That's an overstatement if I've ever heard one. Staring a greenfield project with microservices is a nightmare. In a greenfield monolith you can code and deploy your dependencies together. In greenfield microservices? Oh, right. Set them all up for deployment, figure out netwroking and access between them, any change requires the fix-test-deploy dance etc.
Companies that aren’t B2D have a hard time with monolithic architecture. They simply have too many disparate needs, teams, vendors, and timelines to try to force everyone to share a codebase, unit of deployment, and set of endpoints. The supply side and demand side of any business has vastly different objectives, SLAs and priorities. Should we have to wait two weeks to deploy a minor patch to the Purchase Order portal because there is a risky ecommerce deployment that needs to be finalized first? Do we need to scale up the web servers to batch process some nightly data?
Surely there are excesses in the realm of microservice architecture… “every library is a microservice…” is advice I’ve read in the past. Service discovery / routing isn’t easy. Naming things is hard.
Personally I prefer the term Service and try to make no prescriptions as to what size it will end up being.
The example given, of adding new instances of a session service that consumes from a Kafka topic, is completely wrong.
Kafka producers use a partition key of your choice, so the UserLoggedInEvent and UserActivityEvent that relate to the same userId will always be written to same partition. This is how horizontal scaling of Kafka consumers works, without ordering problems. Anyone that isn't aware of this has very limited experience with Kafka and event-driven microservices.
I respect the author's attempt to give some balance, but I think some parts aren't well informed.
It's important to understand the benefits of microservices AND the costs. Small focused projects have small, fast test suites, can be wholly owned and developed by a single team amongst many, can be released and deployed without fear of breaking unrelated features across a vast platform, can be retired easily when obsolete. They also introduce the complexity of communication across many systems to complete an end-to-end journey, and require incredibly careful design and carefully chosen boundaries and responsibilities if you want to allow each one to evolve independently. This takes time and experience, and many organisations get themselves into a big mess with no governance, consistency or cohesion across a confusing sea of services and teams.
There's no free lunch, but let's get beyond fashion-driven lurching from one extreme to another. This cycle of having a few years where the costs of an approach aren't acknowledged, to a few years where the benefits aren't acknowledged, is very lame.
> This cycle of having a few years where the costs of an approach aren't acknowledged, to a few years where the benefits aren't acknowledged, is very lame.
Well said. I do sympathize with some of the criticisms but only insofar as people who didn't have experience with microservices get sold on a lot of hype, the tradeoffs aren't made clear, and they get burned. The cycle seems to be "This is the silver bullet for programming complexity!" for a few years, followed by a few years of "This isn't a silver bullet!" before the Next Big Thing (TM) comes along and the cycle starts over.
I’m going to be the contrarian in this discussion. I think microservices are awesome and I’ve deployed architectures in anger in both Java and Go. 100% would (and will) do it again.
Most of the problems people are talking about seem to be around using microservices inappropriately, without sufficient preparation, or when an application is not complex enough to benefit from it.
Microservices are not a panacea any more than any other architecture. Properly implemented, they have significant strengths vs monoliths, but require experience to get right.
Properly implemented monoliths built by experienced developers are likely to strongly resemble microservices but without some of the advantages that I enjoy.
As others have said, it’s not all or nothing, one or the other, but I don’t get the hate for microservices here.
I'm guessing more than a few here have seen and been burned by colleagues jumping on the microservice bandwagon: plenty of developers think microservices = good regardless of how the system is architected.
If you don’t know how to manage the complexity of a monolith (by using modules/libraries/clean interfaces) then you will have even more problems managing the complexity of micro-services. Since micro-services include all the complexity of the monolith with added networking and deployment complexity.
Microservices mix several different needs: scalability, function decomposition, and team structure. If you want them all you might want microservices, but there are better alternatives when you only want one or two of them.
People have different opinions on microservices because microservices is great when done right, but can easily screw things up if people don't split the bounded contexts carefully. It's also more expensive to correct these mistakes because there are APIs between them instead of just code, so many projects would live on with them painfully forever.
You know what's underrated and underused? Encapsulation and common sense. You can't just do microservices for sake of microservices, but if you see that one of domains in your monolith is growing faster than others it MIGHT be good idea to split it to microservices. Every decision has to be reasoned by something. If you "don't know" or you haven't done basic designing like event storming, then you should start with monolith because it's easier to take step back.
Choosing architecture has to be conscious not based on trends or beliefs. Even small app can be easy to mantain and develop in miroservices manner if you know WHY you decided to split it up. Don't do (or don't stop doing this) because you read some comments on hacker news or reddit :)
138 comments
[ 2.9 ms ] story [ 207 ms ] threadIt's like functions: don't turn code into a function until you need it in three different places.
But then, I'm firmly in the old-skool "don't split the monolith until you absolutely have to" camp.
This is my position as well. Strong interfaces, good logical separation of function under the covers, etc. should allow splitting off things at whatever level of micro you prefer.
One of the good reasons to spend time with Erlang or Elixir is it'll force you to learn how to write your programs with a variety of actors. Actors are generally easy cut points for turning into microservices if necessary. As with many programming languages I appreciate not being forced to use that paradigm everywhere, but it's great to be forced to do it for a while to learn how, so you can more comfortably adapt the paradigm elsewhere. My Go code is not 100% split into actors everywhere, but it definitely has actors embedded into it where useful, and even on my small team I can cite multiple cases where "actors" got pulled out into microservices. It wasn't "trivial", but it was relatively easy.
In essence, creating a microservice with elixir is about the same amount of effort as adding a controller in rails.
This results in K8s-driven-development instead of microservices.
The idea of splitting your web servers into multi-tiered web servers for scalability is, well, weird. Yet, somehow it's the main reason people keep pushing it. Even this article repeats this.
There's nothing on microservices that adds scalability. They make it convenient to deal with data partitioning, but it's an ergonomics change, not one of capability.
Plus being able to choose different tech for different types of work, and also being able to split between teams when your org gets more vast.
Though I guess this is more about using services than micro services, per-say.
Isn't that a bit of an empty argument. Literally anything Turing complete is an ergonomics change when compared to anything else Turing complete.
We are writing for the same computer. There's nothing different (fundamentally) about the kernel and my program.
What you have on your desk isn't equivalent to a Turing machine.
The real issue is in scaling the data store. Microservices typically work around that problem by each having their own separate database, but nothing prevents your monolith from also talking to different DBs.
And when you have alternatives, great. But for example game devs don't. Or operating system devs. Though of course these are all active areas of ongoing research (for decades!).
If the problem/subject were that easy we would have already solved it and it wouldn't be a hot topic.
And since it's likely a nonlinear problem it's hard to map the problem-solution space, hard to get a good mental model for it, so we use the next best thing: stories. We have success stories and parables on what not to do. (And whole conferences dedicated to telling them :))
For deploying they are nothing but a very large hindrance.
It's much easier to test them in separation also. Deployments are low risk, not much need for coordination with other teams, etc.
There are components that have a heavier load, so we run more of those. Not really a big difference compared to simply running a few more monoliths and configuring them to only handle certain requests... but that's already a given here.
I don't have strong opinions either way. Teams/companies should do what works for them, and they should spend some time on improving it so they can be more effective. (Because most people/teams/companies have a "good enough" complacency inertia/bias.) And most solutions work in both setups. (Eg. making builds fast requires a divide and conquer strategy anyway. Do you build modules/packages separately and link them in the end? Do you build containers separately? Same thing. Do whatever works.)
As long as your bug is well behaved and decided to keep itself into a single microservice... and you somehow know what microservice it is, then yes, it's much easier.
Let's take a typical monolith. You'll be serving endpoints that: - Are CPU intensive - Are memory intensive - Are I/O intensive
They are almost always heavy on some part, but not all. And definitely they are not uniformly all relying on the same amount of resources.
Now, you scale out horizontally, adding instances of the same size because each of these monolithic instances need to be capable of serving the entire domain without impacting tail latencies. Some part of your application becoming more resource intensive? You'll be bumping up the sizes of the entire monolith because you are essentially needing to do the equivalent of provisioning for peak load.
Contrast this with microservices (which I'm not arguing are a silver bullet), you can run your memory intensive part of memory optimized instances, and run your compute intensive parts on compute optimized instances.
That's the part about scalability.
Now, the more interesting part (for me, personally) is reliability. You are decreasing your blast radius of a bad component/thread/process taking down your entire monolith service, and instead compartmentalizing it into limited subparts of your entire API or application.
Finally, as you mentioned, microservices do help guide you towards better choices when it comes to structuring your data. You can do this with monoliths as well (and should), but it doesn't come naturally, and having a single data store is the main reason I see teams run into scalability issues :)
---
Addition: Some thing I don't see many people talk about is the ability to address tech-debt in Microservices. My experience has been that it is an enormous benefit to have made you domain smaller for when it comes to making changes that require sweeping across the whole codebase. Examples include upgrading language versions (e.g. Scala 2 -> 3, Haskell 8.10 -> 9.2), upgrading frameworks, introducing stricter compilation checks (e.g. TypeScript and strict: true).
These easily end up being either insurmountable or year-long projects in a monolith, where it's very hard to incrementally benefit from the work because they are often by nature all or nothing changes.
But even if the webserver is the problem, it feels unlikely that you'll end up saving much in infrastructure cost by scaling just the login service rather than just deploying more copies of your webserver.
Not saying it is impossible, but it's unlikely for that to be a good justification to adopt microservices. I'm not saying that there aren't good reasons to use microservices, but I agree that scalability is not a good selling point. I'd actually argue that it's harder to build scalable microservices than monoliths.
I would challenge that. Obviously a well written "monolith" with well insulated modules, separation of concerns etc can use different databases but I heard too many times when I ask "why can't we shard the database" the answer "the data model is one and it's way easier to develop on one single data model that on N models inside the monolith".
I think there is value in wiring a monolith together in such a way that each course grained subcomponent exposes a constrained interface into the rest of the system (payments, orders, shipping, customers etc) before needing to break it into distributed micro-services.
Note: I quite like Dependency Injection, I just think the 1 giant bag of dependencies can lead to complexity at scale.
I think the real main issue is lack of discipline when dividing the application into modules.
Spaghetti is basically defined as an application where real modularisation does not exist and everything talks to everything.
It is much easier to work with an application when you can abstract parts of it when you are solving your problem. You effectively work on much smaller part of the application.
Spaghetti == you effectively have to take into account possibility of the piece of code you look at interacting with any other piece of code in the application.
Well modularised application == you only need to take into account the contents of your current module and the interface of the other modules you are using.
One reason why microservices sort of work (when done well) is because they force people to think about APIs and how those services talk to each other.
In most cases you could just put these microservices as modules in a monolithic application and expend the effort on ensuring APIs and application structure.
I have successfully rolled back couple microservice initiatives by integrating services into monoliths. This usually results in the team getting back a lot of their time because their environment suddenly became much simpler. Less applications to manage, less network communication, less possible ways for things to break, less frameworks, less resources needed to run the application, less processes (like processes around deployment, release management, etc.), less boilerplate code, and so on. The list is very long.
Of course, when you work on a large monolith vs a lot of small microservices, it is now important to be able to structure your applications. But there is also an opportunity for improvement.
Sure, if you were ready to make an entire separate application for it it will not be worse when you turn it into one package of a monolith?
Also you can build nested structures with modules (like a larger module consisting of smaller submodules).
Divide and conquer. This is how abstraction works in a nutshell. You start with a large problem, divide it into smaller parts (modules), each part you treat as a separate problem dividing it into smaller modules, and so on.
There exists no need to have a flat application consisting of hundreds of sibling modules.
An application might consist of larger modules, then some of those might have smaller submodules (packages), then those might have even smaller ones (classes), then you have methods, then you have statements, then you have function calls, etc.
A large monolith does not have to succumb to spaghetti or ravioli-type structuring. A good developer should know how to structure an application of any size.
[1]: https://shopify.engineering/shopify-monolith
I like to analogize it to assembly vs structured programming languages, or C vs Java- you can write the same programs in either, you can write bugs in either, but there are whole classes of pitfalls that get erased by moving from one language to another.
I have experienced this pain so many times and in so many different varieties. Often, you cannot meaningfully subdivide the problem space without ruining the logical semantics per the business (i.e. bowl of spaghetti).
An alternative is to embrace the reality that circular dependencies are actually inevitable and appropriate ways to model many things.
The example scenario I like to use is that of a typical banking customer. One customer likely has a checking and savings account. For each of those accounts, there are potentially multiple customers (joint ownership). Neither of these logical business types will ever "win" in the real world DI graph. Certainly you can start to invent bullshit like AccountCustomer and CustomerAccount, but that only gets you 1 layer deeper into an infinitely-recursive rabbit hole of pain and suffering. There also exists the relational path, but I have heavily advocated for that elsewhere and it is not always applicable when talking about code-time BL method implementation. Being able to model things just as they are in the real world is a big key to success in the more complicated problem domains.
Instead of trying to control what depends on what, I shifted my thinking to:
> What needs to start up and in what order?
Turns out, most things don't really care in practice. The only thing I have to explicitly initialize before everything else in my current code base is my domain model working set (recovery from snapshot/event logs) and settings. I decided to not use DI for any business services. Instead, all services become a static class with static members that can be invoked from anywhere. This also includes the domain model instance which is used as the in-memory working set. This type just contains an array of every subordinate domain type (Customers, Accounts, etc.). By having the working set available as a public static type, every service can directly access it without requiring method injection. If I was working with a different problem domain (or certain bounded context within this one), I might prefer method-level injection.
Yes - according to every book on programming style you ever read, this is an abominable practice. Unit testing this would be difficult/impossible. But you know what? It works. It's simple. I can teach a novice how to add a new service in an hour. A project manager stumbling into AccountService might actually walk away enlightened. You can circularly-reference things at runtime if you need to. I've got some call graphs that bounce back and forth between customer & account services 5+ times. And it totally makes sense to model it that way too as far as the business is concerned. Everyone is happy.
I used to get upset about this, but now I embrace it. If a business method requires certain things to operate, then modeling those things as arguments to the method is totally reasonable. Some business is complicated and messy so we would expect more arguments to be involved. Trying to sweep reality under the rug just makes things 10x harder elsewhere.
Have worked at multiple companies that started out as a monolith and are still running the monolith in some form or the other while breaking it down into micro services.
You can run a single copy of the resulting binary (eg. for testing) that spins up all sub-components, or copy it to multiple machines and start individual parts. The ones which happen to run together can use in-process communication as well, others will have to dial in via remoting.
Then instead of having a IService with 15 methods with a construction with 20 dependencies listed (multiply it with say 50 IServices in your project - mmmm spagheti project), you end up with a IHandler with a single method and a constructor that only has say 3 injected dependencies, only what it needs. The trade off is, you now have 100's of small Handler files, some people don't like that BUT you now have a pleasant git commit history per file and most of the code of a Handler file fits on one screen and it is easy to digest. You can also let a handler trigger other handlers if it needs to (via the same mediator). It also fits with SRP.
Auto-wiring 95% of your dependencies still stays intact as the above plumbing will need it to work.
The most important thing I learned is that "microservices" in absolutely no way necessitates "bullshit spread across multiple cloud vendors and other scenarios involving more than 1 computer". What part of microservices says things must be separated by way of an arbitrary wire protocol?
We now have a "monolith" (process) that is comprised of many "microservices" (class files), each of which is responsible for its own database, business logic, types, etc. These services are able to communicate amongst themselves using the simple path of direct method invocation.
If you are looking to scale up your capacity or even make things more resilient, microservices vs monolith is really not the conversation you need to be having. If you are trying to better organize your codebase, you might be in the right place but you also need to be super careful with how you proceed.
We wasted a solid 3 years trying to make microservices solve all of our problems. Looking back, we spent more time worrying about trying to put Humpty Dumpty back together again with JSON APIs and other technological duct tape than what our customers were complaining about.
I think it's a variety of cookie-cutter scaling:
https://paulhammant.com/2011/11/29/cookie-cutter-scaling/
But there's nothing in that about using different entrypoints or routing.
FWIW, variations on the theme:
1. A single binary with a single entrypoint, which can play multiple roles simultaneously (UI, API, scheduled jobs), but where different kinds of request are routed to different pools of instances
2. A single binary with multiple embedded configurations, selecting via command line arguments etc, each for a single different role (UI, admin console, data ingestion)
3. A single (Java) binary with multiple entrypoints (main methods), each playing a different role (live calculations, batch calculations, data recording)
4. A single (C++) codebase building multiple binaries (via CMake add_executable), each playing a different role (calculating prices for potatoes, calculating prices for oranges)
5. A single repository with multiple completely separate applications, with some shared submodules, each built separately, playing a different role (receiving transactions, validating transactions, reporting transactions)
That last one is probably not an example of what you are talking about, but it's closer to the one before than to the first in the list. There is a sort of "ring species" [1] shape to this variation.
[1] https://en.wikipedia.org/wiki/Ring_species
You don't get the convenience of a single datastore with simple transactions but it sounds like you prefer the flexibility of services owning their own data.
As it turns out there's no silver bullet. Do what works for you.
Imagine designing your entire architecture around your build system.
On the opposite extreme you're talking multiple PRs, languages, test suites and deploys for every new feature. I've watched startups do this and die before finding something people want to buy.
We applied this ideology to our software project. Our build/deploy toolchain is 100% in-house code. Turns out, there's actually not a lot of it required because you can build a monster .net project into a single exe distribution with 1 simple command line operation. This applies to most other ecosystems as well. Everything else is basic file operations and moving shit between computers. No reason to vendor either of these problems out.
The benefits of having bespoke CI tooling are difficult to overstate. We've got automated processes where a project manager can multiselect 20 different customers, select a commit hash, and schedule an email containing a link to a zip of the resulting build that will be delivered to each customer's configured IT contact upon completion - all with a single button press. This process also entails injection of customer-specific configuration that is pulled at runtime from another system into each customer's distribution. The email templates are configured on a per customer basis and email delivery is delayed based upon project parameters.
I can't imagine wiring something like this up in Jenkins or GH Actions.
How is the rest of the process not applicable to GH or Jenkins?
I don't think I agree with this statement. "More and smaller" also means potentially more routing and communication paths, thus increased latency and complexity.
They encourage modular patterns which is usually good, but all that plumbing will eventually become unnecessary. I can’t help but remember building 6 versions of each class in the old ejb days whenever I hear microservices hype.
That's an understatement. It was soul destroying awfulness.
I wish more languages/compilers/runtimes had a similar capability.
I think that what the OP is referring to is the fact that you can map URLs to PHP files pretty easily and therefore changing what an endpoint does can be as simple as just swapping out a single file for another one.
However, that tends to fall apart with most modern frameworks, like Symfony, Laravel, Lumen and Slim, where you will typically invoke a main application file which deals with all of the bootstrapping first (reading configuration, DB connections, logging etc.) and then will hand off control over to a router, which will usually map to a controller of some sort, which may have the actual business/view logic spread over other referenced files, the way as other similar technologies like (Ruby on Rails) do it.
Of course, that's not a bad thing, especially since it allows you to follow some of the RESTful conventions, like having path variables without doing rewrites at the web server level, such as /users/12345/photos versus /users/photos.php?id=12345
That said, the whole "mostly single file, maybe with a few common imports" approach can also work fine, as long as your endpoints are kept relatively small. I haven't seen it scale up to large projects that well, at least most of the time.
Microservices and monoliths are opposing extremes on a scale. There is a range somewhere in the middle which is entirely sensible.
I feel like I've found my greatest success with my service abstractions by separating things out based on resource and availability needs of the functionality.
The basic problem is that before you know where the module boundary is, you cannot have a clear module boundary. So three popular approaches emerge:
(a) DDD "milliservices". Basically define the different sorts of users, come up with a clustering of the different sorts of people that you think will be using your app. People are considered to be in the same cluster if they use the same jargon to refer to things, or different clusters if they both use some word but they both mean subtly different things by it.
(b) Gut-feeling microservices that become feature microservices. I should probably have an "auth service," I don't know what it does but I'm going to be doing auth so that's probably a service. I need to import a git library to contact GitHub, probably there should be a "git service" that handles all communication to GitHub or other git repos. Each of these things exposes some swagger/openAPI docs, maybe we should have a docs service? -- that sort of thing. The danger is that the things that are easy to break off are usually not the core competency of the product, and so there emerges some sort of "core services," one or two big honkers that basically are monoliths. People are aware that they aren't supposed to keep adding to the core services and so new services emerge named after feature requests, hello "sharing service," hello "wallet service." Except the core usually is tightly coupled to these new feature services and they all kind of connect to each other. The idea of splitting the core services into other services to fit the newfound module boundaries becomes complicated by all these ties to nearby features, you are not actually loosely coupled because you do RPCs and you probably code in the expectation of success into these RPCs like method calls.
(c) Every noun becomes a "nanoservice". This is a service that watches just one or two tables (or NoSQL document types or what have you) and exposes a CRUD API for that noun, plus a couple auxiliary verbs to do actual business needs with those nouns. So if you were implementing Git there would be a file service, a tree service, a branch service, a commit service, probably a working tree service... But you just have to "know" that logs and rebases and cherrypicks live in CommitService while diffing for some reason lives in TreeService and adding a file to your working tree requires first creating it in FileService and then handing that link off to the WorkingTreeService which will make needed calls to TreeService, creating your own tree is exposed via TreeService's CRUD but the developers tell you that down that road There Be Dragons and you should not have been messing with that.
Any of these three can be successful but only insofar as you can create new module boundaries and move module boundaries and test to make sure that your users will not notice any performance regressions. Of them I would only recommend option (a), because it gives a really nice place for these tests to live and an intuition that each test should document a user journey for the system.
I'm in this comment and i don't like it.
My current team has this pattern. I think we've ended up with two core services, with a fairly thin channel between them, but a constellation of tightly-coupled peripheral services around each core.
To be honest, it works pretty well. It wasn't intentional, and it grew in exactly the haphazard way you describe. But it's not a disaster, or at least doesn't feel like one.
Maybe we should rebrand this as "natural microservices"?
While we do Microservices at work ( distrubited team and Conway's law does make sense: https://ardalis.com/conways-law-ddd-and-microservices/ )
It doesn't make sense for me personally. So I applied all DDD logic and implemented it in my app.
In dotnet there is a useful feature where functionality from referenced projects is included in your main WebApp. Which makes everything pretty clean.
I use application logic over 2 "gateways". Where a gateway is a dummy solution that contains Swagger, global things for that project ( eg. auth verification) and uses the controllers/logic from the referenced projects.
- SPA - Angular that contains a Ocelot gateway on the same domain ( = no cors issues)
- ApiGateway - Contains all the backend api logic and custom integrations with partners.
- ShopGateway - Is included in the Frontend.
- Frontend - the shop itselve. Currently in the process of making this db-less ( everything over API / messages or using the gateway. The gateway is currently a referenced project ( see up), so everything is pretty fast)
In the meantime, i can make "reasonable" quick adjustments and the logic is really flexible.
Overview of the solution: https://ibb.co/k5wn07x
Note: Db migration from "1 project" to this is not complete yet :( . But it's one of the bigger shifts and i'm already 90% finished before i can start the actual shift to MartenDb from EF / Dapper )
If you want a quick summary on the logic behind, DDD would be the answer.
That's not a requirement, cf. Erlang/OTP.
But to qualify as a microservice, you need it decoupled enough that you can independently deploy and/or restart individual services. Pretty sure that's a minimum to call yourself microservice-based, and hard to do by just using classes in C++ or something.
Arguably you should also have quite a bit of control over dependencies and their versions. If your service isn't ready for python3, it can keep using python2, or whatever. Taking this to the extreme would generally mean a black box that serializes everything over a network, so that any service can be implemented on any stack. I'm not sure that's fundamental to the idea of microservices, though.
Does it mean calling a method of a class?
We haven't had a great track record defining the semantics of component boundaries, but http is both simple enough that it constricts code flow in a particular way leaving a lot of misunderstandings and bike shedding discussions off of the table, and is the ligua franca for external services owned by other orgs so your engineers can be universally expected to understand those semantics. There is a nice decoupling of letting teams make technology choices though. Letting that drift on a per team basis is kind of nice to allow experimentation with different technologies without having to get the whole org onboard.
Maybe I need to go on the talk giving, consultant circuit reselling a restricted view of DI as "hybrid service architecture! Write your code as microservices but DevOps can colocate your apps in the same process if that's more efficient!" Lolz
It's also quite a nerd trap as we developers love to separate components into as small of boxes as possible, but just like I've seen super deep object hierarchies that didn't gain you anything at the end of the day, it's easy to fall into the trap of spinning every tiny bit of code into its own service.
I'm creating my own backend arch right now from scratch, and really trying to look at each network traversal from a "what does drawing this boundary actually get me" perspective. And I'm only splitting that out because the third party protocol being implemented is inherently separate services in an interesting way.
I would suggest that http not be the go-to mechanism for communicating between microservices. Each should be able to operate to some degree without hard dependencies. If you build using microservices with RPC over http or what-have-you replacing function calls, you've done it wrong. You can usually tell if your 'bounded context' is of the form noun-service or adjective-noun-service. Usually the context boundary should be a process (workflow kind) that serves a user type. Reliable one-way async datastreams are more useful for keeping microservices fed with the information it needs, and it can decide how to utilize that information in a way that makes sense for that service. If you have core 'source of truth' services used by many others, you have a distributed monolith.
Microservices are not the only way to modularize code.
Also, there is TONS of bike-shedding with regard to HTTP. Which HTTP method do you use? How do you structure your paths? What does the body of the request look like? How to we translate failures into the various HTTP error codes? Do you send a 200 or a 201 status for this particular request? I could go on and on... Even among well established HTTP APIs, there are differences.
> Letting [technology choices] drift on a per team basis is kind of nice to allow experimentation with different technologies without having to get the whole org onboard.
This is very much a double edged sword. I've seen this happen and people just tend to avoid services in languages/frameworks that they don't know. And even if they don't avoid them, they are less productive in them. We have a critical services written in Rust and one guy at the company knows Rust. We have 2 services written in a particular Java reactive framework which only a few people know how to use. Any newcomers to these services have to spend a lot of time learning a new thing.
> Microservices are not the only way to modularize code.
Nobody said that it was the only way to modularize code.
However, but to be frank, most of those require greater discipline than most of the engineering teams going to microservices have. These teams don't do a great job at microservices, but probably would do a worse job with a monolith.
> Also, there is TONS of bike-shedding with regard to HTTP. Which HTTP method do you use? How do you structure your paths? What does the body of the request look like? How to we translate failures into the various HTTP error codes? Do you send a 200 or a 201 status for this particular request? I could go on and on... Even among well established HTTP APIs, there are differences.
You can bike shed anything, that's why it's called bikeshedding. Referring to nobody being able to contribute to the design review of a nuclear reactor so the committee just fights over the color of the bikeshed. That being said, the state space of an HTTP API is strictly smaller than a local library's API, and you can generally expect backend engineers to at least know what complete the state space of options looks like and what the tradeoffs are.
> This is very much a double edged sword. I've seen this happen and people just tend to avoid services in languages/frameworks that they don't know. And even if they don't avoid them, they are less productive in them. We have a critical services written in Rust and one guy at the company knows Rust. We have 2 services written in a particular Java reactive framework which only a few people know how to use. Any newcomers to these services have to spend a lot of time learning a new thing.
The other end I've seen is that having the whole org on one tech stack is a single organizational point of failure. It's not an issue for smaller orgs, but if you have more than say, 200 devs working on the same product, you want a little experimentation and a lower barrier to new technology. If something is abandoned you can simply replace it, since it's not your literal entire stack. And forcing your devs to be polylingual I've found makes them better in all of the language they use, even their primary language.
I'm watching a government customer take simple, cohesive systems developed by a small team (4-5 people) and split it up into tiny little pieces scattered across different hosting platforms and even clouds.
Why?
Because it's "fun" for the architects and pads out their resume.
Just now, I'm watching a "digital interactions" project that will have dozens of components across two clouds, including multiple ETL steps. In all seriousness, half of that could be replaced by a database index, and the other half with a trigger.
They're seriously going to deploy clusters of stuff on Kubernetes for 100 MB of data to make sure it "scales"... to 200 MB. Maybe. Eventually.
What kills me is that now that they've made the decision to over-engineer the thing, my consultancy firm can't bid on the tender because we don't have the appropriate experience building over-engineered monstrosities!
The simple and effective solutions to problems we've delivered in the past are disqualifying us from work.
You guessed it: my next project will be an architect's wet dream and will be over engineered just so that we can say on tender applications that "yes, we have the relevant experience".
Gotta play the game by the rules...
Fingers crossed he doesn't discover k8s...
One definite disadvantage is testability (which drives stability and ease of refactoring). I can write an automated end-to-end test that will enter some text into a CMS on my site, hit submit, then verify that a post was published. If the CMS is google sheets, I'm going to violate some terms of service and possibly get a blacklisted IP if I do that. Instead I'm going to mock out a response according to the API contract sheets gives me. If they change the API in a subtle way, it may break.
An advantage is that google sheets is probably more robust than my CMS and I'll never be able to look at (or have to fix) their code.
Flexibility cuts both ways: I can't add features to sheets. So if there's a field I really need to validate on the client, I'm out of luck. But also nobody is going to ask for that feature because I obviously can't add it, so that may be good assuming there's a workaround.
The seemingly-infinite VC money being thrown around around means you can create a startup, raise some money and provide a comfortable salary & industry experience for yourself and your friends for a few years, regardless of whether the company "makes it" in the end or whether the business problem is even solvable.
At this point solving the business problem is no longer the primary objective. The longer you can drag out the process of "solving" this problem the longer you & your employees can enjoy the salary and build "experience". So instead of a simple solution that needs a team of 5, you end up with an engineering playground that needs 50 people (and associated managers, scrum masters, etc) spread across many teams just to keep the lights on. The outcome becomes endless busywork, self-inflicted problems to brag about solving on your engineering blog (every startup has got to have one obviously) and next AWS conference to attract more employees looking for this kind of environment and make yourself look "serious" in the eyes of VCs that keep bankrolling the disaster.
The business problem, if it gets solved at all, is a secondary concern since the VCs happily keep throwing more money into the dumpster fire and some bigger idiot might even buy out the company regardless of its inefficiency.
Governments are generally driven by a set of requirements that most of us have no inkling of. Distributed deployments are often not just for performance or resilience, but also politics and/or playing the vendors off against each other. It seems daft to us mere minions but there's often N-level chess shenanigans going on near the top.
My search engine is a hybrid architecture, with some monoliths and some microservices.
* Index - 5+ minute start time and uses 60% of system RAM.;
* Search Server (query parser, etc.) - 30 second start time due to having to load a bunch of term frequency data, low resources, ephemeral state
* Assistant Server (dictionary lookups, spell checks, etc.) - fast start, medium resources, stateless
* Crawler - only runs sometimes, high resources, statefull
* Archive Server- fast start, low resources, ephemeral state
* Public API gateway - 5 second start time, low resources, ephemeral state
+ a few others
A lot of the ways it's divided is along the lines of minimizing disruption when deploying changes. I don't have the server resources to run secondary instances of any of these, so I'm working with what I've got. If I patch the crawler, I don't want the search function to go down. If I patch the search function, I don't want to have to wait 5 minutes to restart the index.
It would certainly be cleaner to say break apart the Index service in terms of design, as it does several disparate things, but those things have a resource synergy which means I can't, not without buying another server and running a small 100 GbE network between them. Seems silly for a living room operation.
That's an overstatement if I've ever heard one. Staring a greenfield project with microservices is a nightmare. In a greenfield monolith you can code and deploy your dependencies together. In greenfield microservices? Oh, right. Set them all up for deployment, figure out netwroking and access between them, any change requires the fix-test-deploy dance etc.
Surely there are excesses in the realm of microservice architecture… “every library is a microservice…” is advice I’ve read in the past. Service discovery / routing isn’t easy. Naming things is hard.
Personally I prefer the term Service and try to make no prescriptions as to what size it will end up being.
Kafka producers use a partition key of your choice, so the UserLoggedInEvent and UserActivityEvent that relate to the same userId will always be written to same partition. This is how horizontal scaling of Kafka consumers works, without ordering problems. Anyone that isn't aware of this has very limited experience with Kafka and event-driven microservices.
I respect the author's attempt to give some balance, but I think some parts aren't well informed.
It's important to understand the benefits of microservices AND the costs. Small focused projects have small, fast test suites, can be wholly owned and developed by a single team amongst many, can be released and deployed without fear of breaking unrelated features across a vast platform, can be retired easily when obsolete. They also introduce the complexity of communication across many systems to complete an end-to-end journey, and require incredibly careful design and carefully chosen boundaries and responsibilities if you want to allow each one to evolve independently. This takes time and experience, and many organisations get themselves into a big mess with no governance, consistency or cohesion across a confusing sea of services and teams.
There's no free lunch, but let's get beyond fashion-driven lurching from one extreme to another. This cycle of having a few years where the costs of an approach aren't acknowledged, to a few years where the benefits aren't acknowledged, is very lame.
Well said. I do sympathize with some of the criticisms but only insofar as people who didn't have experience with microservices get sold on a lot of hype, the tradeoffs aren't made clear, and they get burned. The cycle seems to be "This is the silver bullet for programming complexity!" for a few years, followed by a few years of "This isn't a silver bullet!" before the Next Big Thing (TM) comes along and the cycle starts over.
Most of the problems people are talking about seem to be around using microservices inappropriately, without sufficient preparation, or when an application is not complex enough to benefit from it.
Microservices are not a panacea any more than any other architecture. Properly implemented, they have significant strengths vs monoliths, but require experience to get right.
Properly implemented monoliths built by experienced developers are likely to strongly resemble microservices but without some of the advantages that I enjoy.
As others have said, it’s not all or nothing, one or the other, but I don’t get the hate for microservices here.
People have different opinions on microservices because microservices is great when done right, but can easily screw things up if people don't split the bounded contexts carefully. It's also more expensive to correct these mistakes because there are APIs between them instead of just code, so many projects would live on with them painfully forever.
Choosing architecture has to be conscious not based on trends or beliefs. Even small app can be easy to mantain and develop in miroservices manner if you know WHY you decided to split it up. Don't do (or don't stop doing this) because you read some comments on hacker news or reddit :)