Exactly, I would wager that very few organizations go from monolith to microservices and back. Getting organization buy-in to do the "big rewrite" is hard. Getting buy-in to do the second "big rewrite" after the first one didn't go well is going to be even harder.
A few years ago when the topic of outsourcing/offshoring development was a hot topic for conversation, I bumped into a consultant at a bar. We got to talking about offshoring, and he said he has two folders full of notes, one about offshoring & one about bringing resources in-house. He said he whenever one approach starts to peak, he starts pitching the other one.
>VCF East XI -- Ted Nelson: Ted Nelson designed the Xanadu hypertext software and wrote the two-in-one personal computing book, Computer Lib / Dream Machines, in 1974. His work deeply influenced the personal computing revolution. Ted earned two Ph.D.s and penned several other well-regarded academic papers and books about ethical, historical, and moral issues in computing.
>Computer Lib/Dream Machines is a 1974 book by Ted Nelson, printed as a two-front-cover paperback to indicate its "intertwingled" nature. Originally self-published by Nelson, it was republished with a foreword by Stewart Brand in 1987 by Microsoft Press.
>In Steven Levy's book Hackers, Computer Lib is described as "the epic of the computer revolution, the bible of the hacker dream. [Nelson] was stubborn enough to publish it when no one else seemed to think it was a good idea."
A team that fails to understand how to write modular code, is just going to write spaghetti RPC calls, while having to deal with all the traditional failures and performance issues of distributed computing.
Naturally it is a recipe doomed to fail in the large majority of cases, but it doesn't matter because whoever drove the change is no longer at the company and a new consulting team/new hire gets the money to drive everything back to the monolith.
So goes the money around on plenty of consulting gigs.
I feel this at my current work - and the worst thing is that they have some really smart folks which can understand the whole thing and make it work for a few years more.
> A team that fails to understand how to write modular code, is just going to write spaghetti RPC calls
This is interesting. I always assumed we were talking about good developers here.
I wonder what's a more likely cause for a failed attempt at microservices. Is it developer incompetence and lack of discipline, or is it environmental factors related to the product and the organization?
For almost all works produced by more than one developer the "good developer / bad developer" dichotomy is just useless social darwinism. Talking about the team, organisation, incentives, or business is far more useful.
(My favourite example is John Romero, part of the very small team that produced Doom - but who also produced Daikatana, which keeps showing up on lists of notoriously bad games.)
That was pure hubris, foreshadowing GamerGate (and inventing the self-Pwn)! As you say, it's all about the team, not the technique. And talking about that particular team:
>One advert for the game became notorious; a 1997 poster containing the phrase "John Romero's About To Make You His Bitch[. Suck It Down.]". According to Mike Wilson, the advert was created by the same artist who designed the game's box art under order of their chosen advertising agency. Originally, both he and Romero thought it was funny and approved it. Romero had second thoughts soon after but was persuaded by Wilson to let it pass. Speaking ten years later, Romero said while wary of the slogan at the time, he went along with it as he had a reputation for similar crass phrases. In the same interview, he noted that reactions to the poster tarnished the game's image long before release, and continued to impact his public image and career. In a 2008 blog post concerning the recent activities of Wilson, Romero attributed him for the marketing tactic. This prompted a hostile exchange of public messages between the two at the time.
>I'm going to quote our very own Shamus Young here for a moment: For almost a decade, Ion Storm's Daikatana has been the example of "industry waste, arrogance, and incompetence, as well as a universal punchline for things that suck." The shooter was supposed to be an epic vision, the masterpiece of John Romero - the mastermind behind genre-defining Doom and Quake.
>Then it came out in May of 2000, and it sucked. The arrogance and hubris that crippled Daikatana have been well chronicled over the years, but none of it is quite as infamous as the ad you see here to the right: "John Romero's About to Make You His Bitch. Suck it Down." It was a pretty ballsy statement in itself, but after the game's failure simply became laughable.
John Romero Is So Sorry About Trying To Make You His Bitch:
>Game designer John Romero and John Romero's hair ruled the roost during the 1990s. With titles like Doom and Quake, he not only helped popularize the first-person shooter, he defined it. Then the unthinkable happened. He made Daikatana.
>[...] Romero, who now says he is resigned to the ad, dished on the ad back in 2008, which evoked a saucy response from the marketer that spearheaded the suck-it-down campaign.
When you have 200 developers working on the same product without well defined boundaries it will be a mess with a monolith, and a mess with microservices.
Also arbitrary rules such as "one service per team" or "one service per employee" that force engineers to jerry-rig things that don't belong together. Either allow them to make a new service, make or a new team, or admit that microservices are not single-purpose and are just a blob of multipurpose code. I've seen this way too much.
Also pure organizational inertia working against good engineering practices: sometimes a team will be severely overworked while others are overstaffed. But god forbid there's a temporary reorganization to improve the work of engineers, so people send tasks to other teams, but there's minimal communication between engineers.
Truth be told, no one stops you from having modular 'monolith'. It feels the name was invented just to sell books and conference tickets (and yummy consulting fees).
There is no substitute for a good model and responsibility separation, (micro)services or otherwise.
Writer Michael Feathers has an article where he suggests that Microservices are a replacement to encapsulation, all we have to do it use encapsulation well.
We did this, but on the trip back we made some major improvements.
We didnt stop at monolithic service. We stopped at monolithic repository+organization. There is now 1 VS2019 solution that covers our entire business. All of our services can be built out from this one solution via various configurations. We've even created additional solution "views" for more focused work (i.e. so you don't have to load a bunch of projects you don't care about for a specific task).
At this point, if we ran into scalability issues with a monorepo, I'd start looking for better source control/CI/CD technologies rather than splitting things up in hopes of arbitrarily keeping git viable. The benefits of having all of your source code in 1 repository with strongly-typed models throughout are impossible to overstate. When checkbuilds complete successfully in GitHub, we know the entire business is clean. Not just 1 little aspect of the product stack.
Could it be valid to say that even if microservices are implemented correctly, you're swimming (if not drowning) in complexity? But for some problems and teams, that's better than the alternative.
My experience of moving to a microservice architecture: the most important consideration with microservices is "who will develop, maintain and operate them?". You can split them down functional lines, architectural lines, whatever you like, but if you don't have teams with definite ownership of each microservice (and that aren't swamped by maintaining lots of them like it seems happened with Segment), it will become impossible. The "operational complexity tax" is a real thing but is manageable if your engineer:service ratio is sensible and considered.
Still waiting for Monzo's following blogpost on cutting down their outrageous number of 1500 microservices [0] and moving some back into monoliths. I'm not sure if I would be too excited over the number of microservices if there is a degree of complexity involved here. That is just too many here.
Why do you think/feel that there are "too many"? What's the threshold for an acceptable number of microservices? (Not asking this to be confrontational. Just curious, because it's a sentiment I've seen before, without the reasoning behind it being articulated.)
Even then this is risky - if that developer is hit by a bus do you throw the service away and have another developer write it again?
We recently had an interview candidate say this when we questioned the wisdom of having over a thousand microservices: some in languages that only the one developer maintaining them used! For me this is insane, but I digress
Monzo says that they have 800 people, and 1500 services. If we're generous and say 500/800 are developers, then each developer is responsible for 3 services! A team of 6 would have 18 projects in their domain.
There is a classic tradeoff here between top down organisation dictates giving consistency vs engineering independence giving flexibility.
Two organisations that I know of who favour the latter are Spotify and Netflix. It has benefits - different languages are good for different jobs and engineers like to be able to choose their tools.
It would be bad if this was taken too far, and something was written in a language only one person knows, but that problem already exists with the technical knowledge if something only has one mantainer.
I have this thing about micro-services/complexity in that it follows Conway's Law - the architecture follows the organisational structure.
If you push authority and decision making and responsibility for a service to a (2 pizza) team then guess what, microservices work really well.
If you have vast monolithic centralised production operations teams, and no way in hell is their C-Exec going to assign two of them to look after the user-login service, you might not do so well.
Like most things, the organisation needs to change to get the best out of the opportunities software offers. Those that don't will face increasing friction and eventually die off.
Well.. just consider every "microservice" a separate company, exposing its own product/service. Also think about all the overhead that comes with it - product managers, finance, recruitment etc.
Occasionally companies actually do this by fragmenting divisions into separate companies, such as outsourcing IT. It has a very broad range of outcomes, from saving to destroying the business.
So like monoliths vs microservices it's probably a balance between the two (leaning heavily in one direction).
I've never understood why it needs to be either/or. Is it really that difficult to support a microservice deployment that only represents 50% or even 20-25% of the org/project?
Conway's Laws isn't a law, it's just an interesting thought experiment. Organization and architecture bidirectionally effect each other, but not directly, and not completely. I hate how current discourse invokes these different "Laws" as if they are physical properties of the universe. I've worked at places with a strong, hierarchical organization that created a wonderful set of "micro" services, and I've worked at places with a chaotic environment that developed monoliths.
There are shitty hierarchies and shitty flat organizations, just like there are shitty monoliths and shitty microservices.
Sorry if you actually agree with this more nuanced view, it's just that I've seen Conway's "Law" invoked more than once in this discussion and it drives me bonkers. I get the same way when someone ("Medium Developers" I call them, more than green but less than seasoned who swallow everything the read on Medium as gospel and run around quoting it zealously) quoted liskov substitution principle at me as if it was one of Newton's Laws.
Conway's Law is a physical law in the same sense as Murphy's Law.
It's also obviously true. The organization builds the architecture. The architecture either helps or hinders the organization. The organization builds a new architecture. There's no indirect connection here. If you've seen hierarchical organizations implement microservices, it's because that organization's complement was a microservices architecture. And likewise for a chaotic organization.
--well, sidetrack: Aren't strongly hierarchical organizations the best suited for microservices? With all the strongly divided responsibilities and whatnot?
Thank you for everything you said. The reality is more nuanced and depends on the specifics. The "law" being zealously cited here isn't a rule. Nor is the thought an approach is wrong if a big organization failed at it.
It seems like they introduce microservices for the wrong reason. Instead of having a service per team, the focussed on services to solve a technical problem:
"Having a code repository for each service was manageable for a handful of destination workers"
Microservices should be introduced to make teams go faster, not to decouple external api endpoints....
I mean you are right of course, but at the same time I can't knock the superficial idea of having one codebase for one domain-specific application. Applications / codebases like that are usually not the problem; it's integrating them into the larger whole where things start getting fucky.
I see it like organization of large companies: you have to split it into divisions, but you can't make every single person their own division.
Don't make the divisions too large, don't make them too small.
The art is to make them the proper size for the particular company.
If you have a small company you don't need divisions. If you have a large one, you need to make divisions as you no longer can speak to every single employee.
I always felt like the biggest benefit of microservices (for the average company that just jumped on the band wagon) was simply the fact it forced them to break things up. Yes, they could achieve the same result with none of the overhead on a monolith, but it would take... discipline. It's much easier to just enforce a hard external constraint.
Realizing this and circling back is still a useful life lesson.
And if you happen to leak concerns in your services (in a monolith), it's really easy to adjust, as opposed to having to coordinate the deployment of 5+ services.
And even then, a distributed monolith is still a risk.
Micro-services add cement to your project. Be prepared to keep boundaries you write for a long time.
Another reason is that it gives you more agency to adjust your code. Want to refactor? Cool, I don't have to talk to 15 teams about how this might impact them. Same thing with changing a schema, changing scaling strategies, etc, etc. I can do things in 2 weeks that would take 2 weeks of just talking to people on a monolith. That might be more organizational than a technical limit, but I've never seen an agile monolith before.
If your code is bad in a monolith, it’ll be bad with Microservices. If you can’t build a good monolith, you can’t build a good Microservices architecture — because it introduces even more complexity and requires even more consideration.
Discipline is fundamental to good software engineering: you can’t force it with Microservices.
> Any organization that designs a system (defined broadly) will produce a design whose structure is a copy of the organization's communication structure.
I think microservices work well in organizations that are big enough to have a team per microservice. However if you've just split your monolith up and have the same team managing lots of microservices you've made a lot more work for the team without the organisational decoupling which are the real win of microservices.
In my experience it is really difficult to fight Conway's law, you have to work with it and arrange your business accordingly.
You don't tackle Conway's law. You can't. You use it on your favor by creating organizational structures that reflect the design that you want in your software.
So you are saying something in line of: let's increase our development staff X-fold and then we can finally do the same thing that way fewer people doing just fine right now?
With microservices there's no way around it, as there's additional overhead when splitting a for loop between multiple services.
Won't stop people from jumping on the bandwagon though.
They're clearly not saying that. If your team is too large to effectively work on a monolith, splitting it up can make sense, but you also need to split the team into smaller groups responsible for different parts. And if you don't end up with teams responsible for individual services, you likely split to small. And quite possibly, your staff isn't large enough to warrant it.
As with a lot of things, it comes down to communication. Between teams, and between the services they write. Which is just another expression of Conway's Law.
IIRC Fred Brooks pointed out that the # of bugs in a system correlates closely with the # of lines of communication within and between the teams. Joshua Bloch recommends in "Effective Java" that, if possible, 3 potential clients should participate in the design of an API, for the same reason. So a well-designed interface or OpenAPI spec is worth its weight in gold.
Ofc, "microservices" here means separate running instances available on a network. But monoliths can be "service"-oriented as well. OSGi was good for this in Java, but any system able to load shared objects or plugins dynamically can follow the same pattern. And the benefit is that, if your app hits the jackpot and needs to scale outwards, the service interfaces, ie the lines of communication, are already well-defined.
So, service-oriented monolith first, then microservices if needed.
> So a well-designed interface or OpenAPI spec is worth its weight in gold.
When I worked on a SOA team, I tried to begin any new effort (whether a new API or modification to existing API) solely discussing the API contract. It was (ideally) high-level enough that business analysts and project managers would understand it, and it helped to guide us away from getting mired in implementation discussions too early.
At that organization, we rarely had the opportunity to involve multiple customers at the same time during design discussions (we were typically engaged to help a specific consumer implement a specific feature), but the institutional memory in the SOA team helped us to keep in mind existing/potential other users of each particular webservice.
The last place I worked that split the devs into UI and backend teams was in a sort of slapstick comedy situation. Nothing ever shipped on time because the front end and backend could never quite talk to each other or needed elaborate conversations to do the simplest of things. This was our new flagship project, I got consolidated in from another team and ended up as a lead not long after.
We had been doing some UML modeling, sequence diagrams during planning, and still having this problem, so rather than repeating the same action and expecting a different outcome I started trying to flip the script. What ended up working was not code diagrams but data flow diagrams and sequences. To get X you need Y, and to derive Z you need A, B, and X. To publish you need all five.
After that, the APIs mostly wrote themselves, we reordered a few different forms, but most importantly variance dropped like a rock.
Mostly these were white boarded, but essentially I/we would draw a collaboration diagram (although I could have sworn these used to be called something else). They showed what data was needed to make certain decisions (eg, a conditional drop down that is populated based on another piece of data, or complex validation steps) and where to get data that already existed.
Yeah it looks like the activity diagram was substantially altered in UML 2.0. What we called an activity diagram then looks more like a collaboration diagram now.
Aside, take a look at tools like PlantUML as a way to create your diagrams. It's higher-level than, say, rolling everything with Graphviz, while easier to share and edit than a bunch of PowerPoint/Visio/etc. files.
The great thing about generated-diagrams is that you can easily store and version the original text representation along with the code it describes or applies to.
> As with a lot of things, it comes down to communication. Between teams, and between the services they write. Which is just another expression of Conway's Law.
This is so accurate. I've heard engineers give state not needing to communicate, chillingly, as a positive for microservices, like "we won't need to talk to each other if all of us are working on different services". My other favorite is using microservices as an excuse for why the product isn't working "oh, my service is working fine, but his service is doing this when it shouldn't", when we're on a small engineering team.
I think there's an element of truth to the engineers' claims. Working on different code bases means there are a lot of things you would otherwise need to talk about that now you don't. It's very much the case that you still need your interfaces to be clear (in fact, clearer!) but those discussions can be somewhat isolated, so more work can proceed asynchronously. Just how isolated depends on how exact (and correct) the specifications are, which is a question of trading up-front work against interruption.
Yeah, the problem with microservices is because the organisation structure is wrong. I’ve literally heard every excuse about microservices at this point. My architecture is better but it doesn’t have a snappy name; it’s called the smallest possible number of services that can be reasoned about and network partitions are NOT necessary to create bounded contexts in a codebase, often just a directory is FINE.
I agree. I hate the term microservice for the same reason I hate superlative infected clickbait titles. There's no need for half of the word to exist. Service. What's wrong with service?
I agree with you, but I find some value in people using that term - it signals to me that I should consider the the architecture was prematurely split-up and could suffer from the various pitfalls associated with microservices.
Microservice implies systems that are decoupled for deployment purposes. For example, Microservice A could restart to a new version while Microservice B keeps running. This is a more complicated interaction contract than services where their deployment is coordinated in concert.
I don’t think this is accurate. I’ve worked at companies that did “service-oriented architecture” long before the rise of the term “microservice” and it was clearly recognized that different “services” shouldn’t be so coupled together you can’t redeploy them separately.
This thread considered an issue: whether services and microservices are equivalent concepts. They are not. There is a quality that is held by Microservices, yet which is not universally held by Services.
You have observed that other services also have that quality. Indeed. Nowhere did I say, "all services with decoupled deployment are microservices".
Revisit. I can where brown9 is coming from. I could have avoided leaving that interpretation open by writing, "here is an example of a quality that is held by all X, yet not by all Y".
There was a period of time when Service had a different meaning than microservice. A service traditionally may exist across bounded contexts and be almost a mini-monolith whereas a strict microservice should touch very few data models and exist strictly in a bounded context.
Of course real life is messy and plenty of people realized writing small single purpose services was valuable, and plenty of people build giant "microservices" that have nothing to do with the original term and are just badly constructed monoliths.
It is probably too late to change the name. But you have a good point, the "micro" prefix is highly misleading. Furthermore, there is little guidance in the literature on how big the microservices are.
Plenty. A subroutine is a service. A library is a service. A Windows daemon is a service. The vendor I just inked a contract with provides a service. A web service is a service.
I really hate that word when used without further definition.
Yes there is. A service is a very generic concept to the point it's only relevant as a high-level concept.
The concept of a microservice makes all the sense in the world if you look back to where we came from: web services. When compared with all the work and requirements and complications of using SOAP and WSDL and UDDI and everything around, just sending small JSON payloads around, and the ability to peel off smaller services leveraging that architecture approach, was a far lighter and uncomplicated way of doing business.
I mean, the name microservices becomes obvious once you look back and all that you see is macroservices.
That's not what the person you are replying to said though, "the organizational structure is wrong". More like: It is a mistake to use microservices UNLESS you have a certain organizational structure/capacity already.
I think they were saying something more aligned with your opinion than you read it as.
We are dealing with poorly defined terms. However, services mapping ~1:1 teams was generally called service oriented architecture not micro services. Micro services involved breaking things into even smaller chunks, so backing off of that idea really just means SOA as originally defined is a bad idea.
That’s not quite right, SOA as originally defined had no mapping to team structure or deployment runtime, it was mostly about defining discrete service interfaces and ensuring your clients used that contract rather than back channels to communicate. Most often you had a dozen services running in a single app server cluster. Conway’s law was rarely discussed (with some exceptions).
Microservices tended towards a single runtime per service, ensuring the deployment lifecycle was tied to the build lifecycle and thus allowing for independent evolution.
I am not saying that’s how SOA was defined, just that it was used to refer to such team organization around architecture. EX: Amazon famously uses a Service-oriented architecture where a service often maps 1:1 with a team of 3 to 10 engineers.https://en.wikipedia.org/wiki/Microservices
At the beginning Microservice was generally viewed as more granular than SOA, though that’s been backed off of.
The general view of microservices was largely invented out of thin air ;) when you look at Martin Fowler’s wiki or Adrian Cockcroft’s presentations which were the originating popularizers of the term , it was all a reasonable refinement of SOA.
But then you’d get some that would make bizarre claims like a microservices must be under 100 lines of code. :shrug:
Noo! Building teams around software components cements your architecture and prevents most cross-cutting improvements.
I'll claim that splitting a well-structured monolith into microservices will always make it less maintanable, but it might be worth it if you need to for some reason like elasticity or failure tolerance.
But for the love of god, keep the design open. Don't tie the existence of internal software components to peoples livelihoods.
I think the ties arise because people are actively seeking areas of responsibility. Software components are an obvious grab if your eyes are on the software specifically. But there are other ways of dividing your teams; based on for instance customers, use-cases, aspects of the code (performance, security).
The problem is that the software usually keeps expanding until programmers find it hard to cope. If you split teams up so that some people are only concerned with a certain part of the codebase, chances are you are going to grow the size of the codebase by a quite large factor.
I think there should be an incentive in place to keep the codebase small and understandable by most.
It's pretty hard to keep the design open once the whole architecture is bigger than what a single programmer can keep track of. Say, the Linux kernel. The overall architecture is fixed, there's no way around it. At that point, splitting into components that are maintained separately does no harm. AFAIK the Linux kernel is maintained like that already in practice, even if it's a single repo.
I agree with you in such cases, but I'm willing to bet that most codebases don't need to be as big as they are, and that it's better to create an incentive to collaborate and keep the codebase maintanable and small.
That's generally true, and it's a big problem with microservices, because they need so much upkeep.
But if your code is living as a few hundred or thousand readable lines in the common codebase, that isn't really a problem. The code is there, readable and working, and if anyone needs to change it they can. If it falls out of fashion, it can be deleted.
What is your alternative? Tying "the existence of internal software components to people's livelihoods" across the expanse of the entire codebase is the only remotely effective approach I've seen to scaling the SDLC at scale.
Aggressively small teams, with no hands-off middle-management layer.
You can build massive capability around a small number of well-managed message-backbones and a single codebase. By keeping the number of hands small and the structure flat, you force high standards. (Skilled staff won't tolerate distractions caused by bad engineering or inadequate automation.)
Heuristic for analysing firms: who has strategic power in decision-making? Conventional answer: a group of hands-off middle-managers who run on meeting tempo, and who are valued by how many people and systems report into them. Under AST: an engineering effort running on maker tempo in cooperation with a hands-on sales effort.
Microservices tend to have multilateral contracts with other systems in the organisation. This steers all planning towards meetings. This creates middle-management bloat.
Amazon has a famous love for what they call “two-pizza teams” and you can find writeups about the philosophy by searching the term. The joke is that a team should be small enough that you only need to order two pizzas to feed them all. The philosophy is about the number of participants in the decision-making process. Keep teams small and give them total ownership of decision making so that decisions can be made by a small group of people who work with each other every day. That way no meetings (and certainly no cross-team meeting) need to happen for most decisions to be made.
OP’s post was “ Aggressively small teams, with no hands-off middle-management layer”. 6-8 swe teams + hands off people manager reporting to middle manager, who reports to director, is how Amazon organizes teams, therefore it isn’t an example of what their suggestion was...
Think of all the open source libs. Generally speaking, anyone can contribute to any part of the project.
That's not to say that some people are better than others at certain parts of the codebase, but you don't want people fighting to keep old cruft in because it's on their job title (figuratively speaking).
You can organize around customers, use-cases, platforms, concerns or other things. Some might naturally map 1-1 to software components, but the software component should not be the "reason d'aitre" for a team, rather the customer experience or something else which can transcend multiple interations of the software.
I see, you meant things in a more literal sense. I generally agree with you in that case, that the customer experience should be the thing which the team owns, which incidentally involves owning software components. But on the other hand, it's also certainly the case that at a company of a given size or in a given sector, certain kinds of software components and infrastructure are not directly customer facing and yet must be owned in house, and logistically serve as one of (if not the only) competitive advantage over competitors.
Is it wasteful to have whole teams at GOOG, FB et al owning and improving the state of the art of infrastructure? It depends. At a certain point, there are enough internal customers for teams to reach contribution margin positive on engineering initiatives that have no direct but only second order effects on customer experience.
I've seen this pendulum swing both ways, often within an organization. Cross functional teams owning code bases allows divergence to specialize and ownership of a release, teams with a single functional focus allows efficiency of work and cross cutting gains.
Both have their boatloads of suck, neither is inherently better. Interestingly, trying to mix them to get the benefits of each doesn't seem to invalidate any of their downsides; often it exacerbates them.
Mmmmyes and no. Depending on the size of your project, that may not be the case. I've had to work with two titans of monoliths, maintained by relatively small teams(anywhere between 2 and 6-7 people for several million lines of code). At some point managing a codebase this big within a single project becomes a huge burden, for both developers and even more so for those who develop and do code-reviews(first hand experience right here). At times I've spent 3 weeks straight doing code reviews with 2 notebooks filled with notes and diagrams of the different components inside the code. And at that point, the easiest and most sensible thing to do is chunk out large parts of the project and put them aside as a microservice with the adequate amounts of tests. For small projects, microservices make little absolutely no sense. But in the case of something the size of AdWords(which my two such experiences can be compared to), you are playing with a raging lion if you decide to go monolith.
My argument here is that it's not so much the size of your team but rather the size and scale of your project that needs to be taken into consideration.
Good monoliths are highly modularized. But it's a whole different thing to package up a module as a separately deployable unit for external "public" use (external to your app, that is, not your company).
I'm just curious to know, when you said "the easiest and most sensible thing to do is chunk out large parts of the project and put them aside as a microservice" ... were these chunks separately deployable units for external "public" use.
I don’t know that you need to be much more disciplined to write a large application in a module way vs writing any application in a modular way. A monolith could definitely get messy though if you write them how I see people write microservices.
I think this is actually one of the reasons that microservices became a thing to begin with: teams wouldn't actually apply engineering best practices.
Microservices actually make you encapsulate your code, at least within the microservices, because you can't call out to it directly. They don't necessarily force you to implement the single responsibility principle, but they do a good job of pushing you. Microservices implement a service-locator pattern through DNS or web routing, one form of the dependency inversion principle. Microservices make you pass data around as entities, instead of Active Record instances.
The price for this sort of thing is very steep, though; distributed systems are inherently icky, harder to trace, and more prone to failure, and besides this, you've added network overhead to each service call.
I wish more engineering teams would consider spending half the effort of microservices on simply disciplining their monoliths. They might get somewhere...
> They don't necessarily force you to implement the single responsibility principle, but they do a good job of pushing you.
In my experience, if your services are developed by the same people, and not separated by teams, engineers will often tightly couple the services with fragile and opaque dependent changes regardless.
While in monolith this is painful, at least you have a complete stack trace and the ability to run things through a step debugger you orient yourself. In a distributed system tribal knowledge tends to be your only savior.
When we design systems, we need to spend more time thinking about what is most likely to happen as opposed to what we feel should happen.
>I wish more engineering teams would consider spending half the effort of microservices on simply disciplining their monoliths
100%. This is an uphill battle, though. I've encountered so many engineers who equate "real engineering" with "building giant machines." You just can't convince them otherwise.
I've watched people build giant, real-time stream processing pipelines compromising tons of moving pieces (lambda, sqs, s3, sns, stepFunctions, etc..) to build... a reporting table, and all for... 1.3gb of data. Literally.
Ultimately, despite the "sell," I don't think microservices as a forcing function for good practices works in practice. If the team lacks the skills to build a disciplined monolith, then they 100% lack the skills to build a distributed one.
Oh, all of those were heavily modularized to begin with. But that wasn't enough to keep them manageable. So at the end what we did is figure out which are the core components between the different modules, isolate what they did and put them aside in a smaller microservices, which were easier to track, maintain and monitor. What was once the monolith is now arguably just an interface/API for all the heavy lifting which is done by microservices. Again, my point is that all this must be done depending on the scale and complexity of your application. If you are going to make an authentication microservice for an application that has 50,000 users which simply fetches a username and compares a hash in a database, obviously you are doing it wrong. I am talking about applications which in the simplest of times operated on 24 different databases located in completely different geographical locations(the case of my first such monolith). Some of those databases used different engines. And due to the nature of the infrastructure and the requirements we couldn't simply ditch everything and start over from scratch. So splitting everything into microservices was the only option. And this is something I was working on back in 2012 iirc so back when microservices were considered witchcraft by most people. And yes, I'm talking about several million lines of code and 2 developers - my inexperienced out of uni ass, and an utterly conservative dev twice my age. Took us around 6 months but the project was extremely successful.
There is this trend in technology - every few years everyone changes their minds about everything:
* 2012 - sql is the best.
* 2016 - sql sucks, nosql is the future
* 2020 - nosql suck, sql is the best.
* 2024 - {fill in the blank}.
The same thing is happening with microservices. But in addition docker, kubernetes and recently unikernels have joined the party. The concept is the same though.
What I am trying to say is that either of those can be good or bad in different scenarios. It's a question of picking the most appropriate one for the situation.
If you've got 7 people maintaining millions of lines of code, you're going to have a heavy burden no matter what you do. Extracting a service does not a priori simplify anything. It can encapsulate and enforce a more strict boundary, and optimize compile time or test suite throughput and operations for the extracted logic, but it always comes with overhead, and if the interface between the services is not well-defined and stable it can easily be a net-negative in terms of productivity as you are now giving up your in-language tools for distributed systems tools. Now if you have large swathes of code stable functionality, then it's easier, but at that point why not just isolate modules within the same codebase?
There is nothing wrong with most developers working on and communicating about the entire code base. Having teams work in silos is not a benefit. You're touting as a benefit what is one of microservices' gravest issues - teams stop communicating beyond the surface level of their respective APIs.
Have you tried coordinating entire teams to work on a shared codebase?
Honestly, I have never been in an organization so large that this became a necessity (if you solve tens of different problems, that would require almost thousands of developers). But coordinating single developers without an API is hard enough already, I can only assume for teams its nearly impossible.
Define “codebase”? You can have multiple services, user facing apps or modules inside a single repository, but if there are no boundaries coordination will be difficult of course.
The definition implied by the GGP is: shared codebase = everybody will change the same lines; separated codebase = people will work on different sides of an API.
At least, that's what I understand from his comment.
Maybe its conways law, or maybe it's just that designing a distributed service is difficult, and when you break a monolith down, you're having to deal with distributing that monolith N times, and solving those CAP issues N times, which usually is not trivial. Not to mention tuning the network.
In all honesty, I think the monolith/microservice distinction misses the point a little bit.
It's inevitable that the longer the codebase exists, the more difficult it is to maintain. It's a battle that you can't necessarily win and it's turtles all the way down as your dependencies, and their dependencies, tackle the same issues.
All it takes is one or two roughly defined APIs and you've already created the nucleation point for ever-more tech debt, and while you'll be able to tame some of it you won't manage all of it due to business requirements, or other teams depending on private APIs to save time, or whatever else you can imagine. Switch the architecture and you'll either have all your problems bunched in one codebase, or you'll have distributed your problems all over the place.
I'd go as far as saying that a perfect monolith and a perfect distributed architecture are theoretical ideals that require perfect communication to build them.
I cannot agree more: I worked at a company where we went from a monolith deployed on IaaS with a couple handful of engineers to Docker containers deployed on ECS with over 200 engineers. The main reason we did it was because Docker+ECS was cheaper than a bunch of EC2 instances and you can't effectively use 200+ engineers with a single monolith.
After 2 years we had over 450 microservices while keeping our AWS bill flat or slightly decreasing.
On the other hand, over 200 engineers on payroll is way more expensive than a couple handful!
Presumably you're getting significant value out of the additional engineering work in which case the architecture shift probably makes sense (to stay aligned with the expanded organizational structure), but there are also cases where a small and flexible team maintaining a simple monolith would be much more nimble and cost-effective.
I don't agree with your premise that development structure == deployment structure. There are plenty of good ways of splitting up development of a monolith without the huge devops headache of deploying microservices.
> I think microservices work well in organizations that are big enough to have a team per microservice.
Presumably by definition we’re talking about a few hundred lines of code, or a couple of weeks development time here at most. What does this team do all day otherwise?
I'm just curious if there is a middle ground somewhere?
On one end you have a giant monolith. All services rolled into one which includes your API, Middle ware and then Database.
One the other end you have microservices which bundle services into individual distinct units with each service being responsible for its API, middleware and database.
Are there any preexisting patterns which seek to combine these two and come up with an architecture which is midway between those two. A few months ago I had read an article about Data Oriented Architecture on HN which comes close though I'm wondering if there are others.
This is why I struggle with microservice architectures. It seems like there is a basic contradiction. On the one hand, it's vitally important that the microservices are carved into the correct modules otherwise you get a nightmare of operational complexity where simple functional changes require coordinated changes across multiple services. But defining the correct way modules requires a bird's-eye architectural view of the entire system, which seems contradictory to idea of self-organizing, independent teams. I can see how it works when the right way to divide things up is obvious or when you are dealing with IaaS or PaaS services, but in a complex business domain who decides how to carve things up?
There's what DHH calls the 'Citadel' architecture, which is basically you carve off a few small chunks from your monolith where needed and call them 'outposts':
However, in my experience, OOP made it possible to do really big stuff.
It’s all about not having a “one-size-fits-all” approach. I don’t think it’s just about scaling architectures; it’s about changing architectures to match scale.
It’s difficult as hell to make these changes, because people get invested in methodology, and insist on applying the same lens to everything we do.
It sounds like they had the right idea, but they probably had the wrong people.
"Turning everything into an object can make a small program into a big program, so it’s maybe not such a good idea for small-scale stuff."
In my experience OOP actually makes programs smaller. Assuming of course they have good programmers/architects and the program itself is larger than "Hello world".
In my experience, OOP makes programs different. Might be bigger or smaller, but the real difference is complexity. Not in that it makes things more or less complex, but in that it moves the complexity around to different places. Those places being complex (and others simpler) might make it easier or harder to maintain your program, which is what makes these decisions highly dependent on your particular systems and teams.
There is a difference. OOP is just one of many tools to help accomplish a task. Many other tools as well. It is up to the people how to use tools for a job and what tools for what job. You equating OOP with the dangerous things that should be kept away has no basis in programming.
The way people use OOP causes them to move the complexity in a particular way. That's why the distinction doesn't make a difference in this context. You're right, technically it wasn't OOP that was writing the code, it was the person. We would have never figured that out without your guidance, we all just though OOP was banging away on the keyboard.
Why? I wrote it, I know exactly what it means. Maybe you should take a closer look? If everyone who uses OOP moves the complexity in the same way because they're adhering to OOP, then it is a distinction without a difference because the complexity is moved regardless. It's a nuanced concept so don't beat yourself up if you don't understand.
I want taking a side here, just noting a tradeoff inherent to the choice in programming paradigm. When you make effective abstractions, you make your program bendy in all the right places. It's easy to add new objects where you will need them, and it's easy to extend behavior in places you need to do that.
Of course, if you guess wrong, you're totally fucked. Well, either that, or you are smart and see it coming in time to rewrite the code that put the complexity in the wrong place.
Hope that helps with the context. I'm not some anti-OOP zealot, and those do exist.
I am in total agreement with what you've just said. Basically it all comes down to programmer being either smart or stupid. OOP on its own has nothing to do with the overall complexity and where it is "moved". Shitty programmer will fuck things up no matter paradigm. And good programmer can use various paradigms to their advantage depending on particular situation. But no. From what I see we have crusaders here.
You do not need a lot of things to write software. Does not mean you can no benefit from those when applied with good reasons. OOP concepts can help while coding certain domains and yet other can benefit from a different style.
Don't get me wrong. I love OOP, and have been using it since before it was cool. It's been a standard wrench in my toolbox for decades.
In fact, I have been running into folks, these days, that don't understand it, as, apparently, OOP is becoming "uncool."
I've always been a "right tool for the right job" kind of guy. I started off with ML (Machine Language, not Machine Learning). I am quite comfortable, sitting down with a breadboard, and flashing an OS.
But I remember the old days of OOP, where "classic" structured programmers didn't "get" OOP, and designed these horrific chimeras.
I always make it a point to understand my methodology and drivers "to the bone." Just because someone at a conference said it, doesn't mean that I should use it for everything.
But I have made it a point of personal ethos not to post criticism or polemics, denigrating/excoriating the work of others.
I know that could buy me a lot of clicks (and probably some considerable HN Above The Fold time), but I think we have enough negativity and finger-pointing on the Internet.
If you read my stuff, you won't see much of that. I may, in a rather vague way, allude to something that gives me a frowny-face, but I don't want that to be part of my "personal brand," so to speak.
I do take tremendous personal pride in my work; both coding and writing, and hold myself to a high bar. I may even project that bar onto others (only in some circumstances), but I don't think it's helpful to do so in public.
I find it most gratifying to write a "This is how I do this..." post, as opposed to a "This isn't how you should do it..." post.
Lots of words. What's the conclusion? OOP is bad? Or maybe it is incompetent people who manage to f.. things up no matter what you give them or people with the agenda going on holy crusades?
No, OOP is not "Bad." I'm not feeling particularly argumentative. I apologize if what I wrote upset you. I suspect that we may actually agree on most things.
In some cases, it is not the best tool for the job, but I find that I tend to use OOP for almost everything I do; large or small.
It isn't of much use in small utility scripts, though, and some languages are just not written to natively support it. In those cases, procedural (or FP) is the way to go.
There are new-ish methodologies, like functional programming, and protocol-oriented programming, that deprecate "classic" OOP. Some folks are using these as backing for declaring OOP "dead."
I suspect that might be a bit...premature. My current fave lang is Swift, which pretty much allows you to use any methodology you want, or mix them together (Will it blend? That is the question).
I have found that it isn't helpful for me to "write off" any methodology, and most of my work is actually a hybrid approach; with elements of multiple methodologies.
"There are new-ish methodologies, like functional programming, and protocol-oriented programming, that deprecate "classic" OOP. Some folks are using these as backing for declaring OOP "dead."
Those are anything but new-ish. In programming new often means that some old concept suddenly becomes fashionable. Myself I do not restrict to any single paradigm and use what I believe is the most suitable for current task.
Note the "-ish". I know that they aren't actually new, and that many "new paradigms" are actually rebranded old stuff (I have been writing software since the early 1980s, and have seen these waves sweeping through the industry; often, with some amusement).
What does happen, though, is that a canon develops, based on these technologies, and [actual] new techniques get created, based on them.
Some of these are nightmares, and need to be strangled before they can crawl off the slab, but sometimes, a gem comes up.
I remember writing "object-oriented" software for classic C, in the late 1980s. I called it the "faux object" pattern, and was based in state being kept in a structure that was passed around functions. I refined and formalized the idea when I encountered Apple's QuickDraw GX in the early '90s (I suspect that may be the only good thing that I ever got from that sad debacle).
I used the faux object pattern in an SDK I designed in 1994, and it's still being used to this day. Back then, you couldn't pass OOP across module connections, so we had to figure out a way to do it with C.
Nowadays, I can easily pass Swift extensions and virtual implementations across SDKs; no sweat. It's cool.
In my uninformed opinion many of these "nothing new" cases happen because the newcomer actually solved some (maybe minor) pain point in usability of the old solution. Maybe combined with some good old nostalgia.
In general I agree that people should study more the past, but I do not see any point in dismissing new trends just because the do not market their full genealogy upfront.
This phenomenon can be described as "excessive factoring" and it can easily happen under any paradigm.
Perhaps it's more prevalent with OOP programmers, but perhaps it just appears that way because the boilerplate for classes is a bit larger than the boilerplate for functions and structs.
The reason OOP made it possible to do big stuff, seems tobe because it improved the average productivity of the average programmer.
With procedural code, you would need an exceptional programmer to produce a big program. With OOP, an average programmer can deconstruct a problem into its component parts and solve it, mainly because, the human brain can reason about concrete objects more easily, than say, abstract methods like functional programming.
Edit : OOP has encapsulation which, in my view, significantly reduces the cognitive load when thinking about state management in an app. I remwmber writing a small graphics library using Borland Graphics Interface in Turbo C++. It was a breeze to do because I know about 'things' I want on my screen and coded my classes to reflect those things.
>Shared libraries were created to provide behavior that was similar for all workers. However, this created a new bottleneck, where changes to the shared code could require a week of developer effort, mostly due to testing constraints.
That is a big red flag. Microservices that suffer from shared code changes are not really microservices, but a distributed monolith instead.
Because if you share something (like auth for example), you should have microservice for that.
The question is not about duplicate code, but about duplicate libraries that handle the same thing.
Decoupling the auth process into separate microservice removes the bottleneck.
Eventually you'll have a service to format phone numbers in the format that the company needs to be standard across all services.
If you don't want to do that, then you need a simple shared library for that.
The problem is that there is no easy way to draw the line between "this is obviously a trivial library function we should just link into our code" and "this is something we can't share because it would create friction or break our isolation".
Auth is obviously a "service" but phone number formatting as a service seems extreme.
One of the lines is going to be acceptable performance. Your phone number formatting microservice is going to be orders of magnitude slower than a client library.
The auth service will likely have to hit a DB anyways. Assuming the microservice call has roughly the same network latency as the DB call and the DB has 0 response time, it would double the total time to perform the auth. It only gets more favorable as DB response times go up.
More generally, I think microservices make sense in scenarios where the time to process the request is longer than the network latency incurred by making it a microservice. Things that have to hit a DB are generally okay. Pure functional things that just compute on CPU and RAM are generally not, unless they're very computationally expensive like running a simulation or something like that.
I'm reminded of the classic problem of static utility classes, where you have functions for say formatting phone numbers, or computing a commonly occurring simple mathematical function. It can be difficult to figure out how to better modularize the functionality provided by this class, the motivation typically being having a large static utility class often violates the principle of a class having a clear, single responsibility. Breaking up a large static class into other static classes that better encapsulate some functionality/concept can help but isn't always the best solution.
So lets say we have shared code for doing something like phone formatting. My question for more experienced microservice practitioners is -- does it make sense to create a microservice for preprocessing data in general? Phone number-formating-as-a-service is excessive but creating a microservice for processing data where phone number formatting is just one aspect of this service makes sense to me. All other services can throw data at the data processing service and get data back in some sort of standard and expected way conforming to whatever business logic/processing rules required.
A common practice is to introduce services that handle shared functionality.
One common example is, instead of having a shared library that reads & verifies JWTs, use a gateway service that handles this before requests reach the upstream service.
This means changes to your organization's JWT code will only require a redeployment of one service, the JWT Auth service.
But that also for input handling, formatting or simple business libraries? Sure you could implement that as services but that would probably result in up to a hundred service calls for one customer interaction. Maybe it looks clean from an architecture perspective but I can't imagine how that'll result in a good user experience.
This is really a time-of-binding argument; the difference between a "library" and a "service" is that one is in-process and accessed over function calls, and the other is out-process and accessed over RPC.
If you change code that other services are using, you can break those other services. No way round that.
While that is true, a microservices architecture can (and in my opinion should) rely on messaging and account for message schema evolution. Dependencies between services should be way less coupling than dependencies between an application and a library.
Schema evolution is just as big of a dependency hell as managing direct library dependencies. With a monolithic architecture, a lot of those concerns are contained within the context of a single repo, and can be tested much more easily than with many repos.
A library API can rely on versioning and account for schema evolution too. Even different versions can coexist if you decide that's important from the beginning (what is the same requirement as with services).
The only real difference is that services have a slow serialized network interface that fails 4 or 5 orders of magnitude more often than libraries, but can migrate over memory domains.
There are circumstances where they are equivalent, but they're very different overall. Namely, if you use a service, you update it once and see the new behavior everywhere. If you use a shared library, you have to update and redeploy every service. Libraries are strictly inferior in that scenario. This sounded, to me, like it was Segment's problem. They were updating shared libraries all over the place all the time.
I generally avoid creating shared libraries, they're a trap. They have a very narrow band of usefulness squeezed in between the more palatable solutions of creating new services or just copy & pasting code and allowing it to diverge for each different use-case.
Sharing code or reinventing the wheel repeatedly is inevitable once you have more than one concern by which you can divide services.
For example: let's say you have lots of integrations, and you need to scale compute, and parse and generate common data sent to and from the integrations.
You can either have a monolithic integration service which you scale out on load; or you can have integration-specific services that scale out on load and share your data parsing & generation library. Due to multiple concerns, there's no "best slice".
FWIW, scaling out compute is a stronger argument to me for a service boundary than responsibility segregation. Scaling out requires distribution; scaling up complexity doesn't, though it can help for other reasons, like CI/CD. I prefer FaaS architectural patterns with the freedom to share libraries in different functions (images) to services, especially if long-running state is not needed.
Having a shared library is not a bad thing on its own. Making the library a bottleneck is the anti-pattern.
If you wish to have a shared-library of microservices you should be prepared to have multiple versions of it running at the same time without any pressure to update everything at once.
If your shared library is the bottleneck, it means that your microservices are tighly coupled (hence the distributed monolith)
That just sounds like the shared libraries needed to make breaking changes less often. If you're going to make changes to core code, it's going to take time to get everything up to date no matter how your code is organized. In other words, shared code needs to be treated just like a third-party library/service (both from the developers and users points of view).
One view is that the difference between a service and a microservice is that a microservice can be sketched between being a local library or wrapped in an RPC server.
It doesn't seem like their end architecture is anywhere near what they had at the beginning... It's a lot smarter, and it sounds like it's a monolithic distribution system that manages hot-swappable services. So the whole thing seems to fall in the "micro services where we need them" kind of architecture.
They just went from naive monolith, to naive micro services, then to smart coupling of the two...
Without knowing more about their architecture it is difficult to comment beyond the conclusion Alexandra Noonan came to, stated at the beginning of the article. It looks like to me that the architectural assumptions were changing too quickly due to the demands of a fast growing business. Having all their code in a single repository means that they can control dependencies, versioning and deployment centrally, it gives them central control of their software development lifecycle. I can't see how they could not have had the same benefits of the monolith if their microservices existed in a single repo to begin with and had the appropriate tooling to enforce testing, versioning, deployment across all services in the repo. I guess this is the whole monorepo debate and tooling.
This article for me is more about the complexity of managing a large team across different sites where the architecture needs to change rapidly when modularity is absent. They did get a measurable benefit around performance, though. I wonder if Alexandra will comment on the challenges of running a team in an environment of this complexity?
If the microservices are in a single repo and tested and deployed together then they are arguably no longer microservices but a "distributed monolith"!
I'm referring to having the same testing, deployment,packaging,versioning policies etc being consistently applied across projects within the same repository not deploying, testing and releasing together.
It's the drift and inconsistencies across these concerns across projects that makes deployment and operations less predictable.
I think this article is more evidence against the credibility of multi-repo than against "microservices".
Anecdotally, my current place of work has grown to about 200 engineers, maintains a monorepo, and hundreds of deployed cron jobs, ad-hoc jobs, and "microservices". We have none of the problems discussed here. We invest maybe 20 eng weeks a year in monorepo-specific tooling, and perhaps another 30 eng weeks per year in "microservices"-tooling.
> Also, a proper solution for true fault isolation would have been one microservice per queue per customer, but that would have required over 10,000 microservices.
I'm a bit confused, they seem to imply that they need a microservice per costumer/destination, but you generically have one instance (aka process) per costumer not an entire separate codebase. The article seem to use the same term for two different concepts. Or i am missing something?
Everyone seems to have their preferred style of coding, and it is an easy defence mechanism, when presented with anyone who tries and finds it wanting, to say that "Well they didn't do it properly".
You find that with Microservices vs Monolith, Strong types vs Weak types, Exception Handling vs Results, Agile vs Waterfall.
People fragment into camps which turn into echo chambers and it's easy to dismiss anyone who doesn't commit to that cult as being unpure and not worthy of being in the cult anyway.
Especially if you consider RedwoodJS, a new full stack JS framework that's build on Prisma technology (their stuff is an alternative to Rails Active Record ORM). My takeaway is that they provide a similar monolith like experience by acting as a glue between different services.
I have always viewed using many micro services as something that adds complexity, something to be used when necessary.
I started working remotely as a consultant in the early 2000s when my wife and I moved to a remote area. I had several development jobs that used the same monolith pattern: I would embed everything in a web app using Apache Tomcat, taking advantage of work threads for background tasks. The only external services were a database and crontab settings to frequently snapshot databases. This pattern was so easy to code to, so easy to debug and deal with any runtime problems. One customer reported that a system ran without stopping for six years (ouch, no OS upgrades??) until they restarted it on a larger server.
Micro services can be great, but not always the best choice when horizontal scaling is not required.
Are there any case studies where microservices went well?
From an end user perspective, Netflix runs in “constantly degraded” mod.
From an engineering perspective, they track “number of successful stream starts”, instead of percentage of the time 100% of their services are working. That’s a huge red flag.
As a researcher, the monitoring and fault-propagation / modeling work they’ve done to get it to stay up at all is impressive, but it’s not clear all of that tooling would be necessary if they didn’t have to reason about N^2 fault tolerance scenarios, where N = 100’s of microservices. That’s on the order of one fault tolerance scenario for each atom in the universe.
I think the person you're replying to meant 2^N based on the context. They're saying you have to account for every possible combination of services being down.
> Are there any case studies where microservices went well?
You answered your own question with Netflix. While you're right that it's not clear Netflix would've needed to develop their chaos monkey tooling and the like, it's not clear at all at that an equivalent system is possible as a monolith. Even if a monolithic Netflix system were technically possible, it's not clear if a monolithic system would be organizationally feasible either (Conway's Law)
Why wouldn't it be possible? It doesn't mean you can't modularize the application. And scaling can also work well for monoliths. I don't see how Netflix' service (browsing the catalog, serving content, reencoding videos) can't be done in a monolith.
Microservices working well is entirely about the team, and my current group of teams works very well with the monolith pattern. A big part of this is because (for business reasons, not technical ones) they frequently trade ownership of parts - so if a service isn't well constrained it will be very quickly. We also have mature DevOps practices and engineers handle significant parts of DevOps themselves instead of just kicking the can.
But the reason I say this is about the team is because I've seen a well built, well groomed service be passed off to an outside team and immediately turned into a disaster. Same service, same business case, just a team without the DevOps savvy and willingness to follow the patterns.
Netflix does some.. unusual things with microservices, mostly with how they treat version rollovers. It's not bad or good, but it looks different from how many other shops handle the same problem and it means asking the question "is everything working?" is extremely difficult but asking the question "how many people are able to start watching?" is pretty easy.
By sheer number of attempts somebody probably got good results with microservices somewhere.
Netflix runs quite well in practice. I think they do redundant service calls, what is the only minimally sane way to develop a distributed system. The funny thing is that I have never seen any serious discussion of redundant calls, except for it being implicit on practical designs on the anecdotal "how it works for us" articles that pop once in a while. Most times people won't even discuss redundant servers. I imagine everybody thinks it's obvious, and well, I would agree, except for the fact that most people I see do not think so.
But well, Netflix couldn't avoid having a distributed system, so they aren't really representative for nearly anybody.
Amazon does micro services (or SOA) extremely well. In fact they practically invented the concept. It’s intricately linked with the 2 pizza team and service ownership concept (you build it, you support it)
> As a researcher, the monitoring and fault-propagation / modeling work they’ve done to get it to stay up at all is impressive, but it’s not clear all of that tooling would be necessary if they didn’t have to reason about N^2 fault tolerance scenarios, where N = 100’s of microservices. That’s on the order of one fault tolerance scenario for each atom in the universe.
That doesn't seem true. I would imagine that at Netflix scale, you probably have request tracing libraries that can give you a graph of service dependencies. Whether it's worthwhile to consume that, or easier to just let Chaos Monkey run rampant is another question.
Also, I very rarely have issues with Netflix. Typically when I do I can just exit the stream and restart it. Anecdata, but I could count on one hand the number of times I've had a title just not play, or Netflix be down entirely.
A monolith can run distributed and be scaled across data centers. It's not a mainframe application with one host. One process can crash without affecting system stability.
Where you can get into trouble is if the problem affects all your instances across the entire cluster. If all your organization's code is running in the same process, if any of that code has a severe memory leak or other serious issue, it could impact the stability of everything else.
Not to say that wouldn't happen for SPOF microservices (e.g. your auth servers), but the surface area is potentially larger for monoliths.
Why do you assume they have more fault tolerance scenarios simply because they have more deployable units?
In every service architecture I've worked with in the last few years, you could theoretically run every single "service" within the same physical process -- provided they all shared the same runtime/lang the way a monolith does. I tend to start service architectures using Ruby exclusively with the Eventide toolkit, so this is actually a viable approach for most teams I've worked with. But it's never ultimately made sense. Weighing the pros/cons, a consolidated deployment topology wouldn't add any benefit, and it would actually make it far more difficult for the operations folks, practically speaking.
I've helped put services into production that 1. carry out crucial, "core" business logic reliably and efficiently, 2. can be scaled horizontally without changing the code, 3. never raise exceptions or cause outages, and 4. don't need to be touched for years because new features are more naturally composed around them anyways (i.e. open/closed principle). Practically speaking, it's quite a bit easier for human programmers to reach this degree of precision with a smaller program than with a much larger one. And if you add up a lot of these high quality programs, you get a high quality system. Building a high quality system out of a single program is much, much harder for humans in practice.
I'll use crude terms here for a second: every good service put into production brings a net benefit to the organization relative to the same code having been entangled with an existing pile of code. That may sound like a "No True Scotsman" fallacy, but the definition of "good" in this context is precisely the net benefit being added. If you build and deploy services that have a high degree of quality, you get a corresponding benefit. If you build and deploy programs that _don't_ stand on their own, _don't_ leverage durable messaging, and take other programs down with them when they crash, then you get a rather large mess on your hands. In fact, you never had a service architecture at all; we call this failure mode a "distributed monolith."
I'll acknowledge that most attempts at microservice architectures in the wild don't seem to succeed. Anecdotally, they are particularly prone to failure when their architects don't understand the underlying principles of distributed systems all that well; they neglect important considerations like messaging idempotence, the deleterious effect of synchronous request/response messaging, and the need for deliberate, thoughtful design. In other words, they build systems out of N number of microservices, and get N^2 fault tolerance scenarios, which you adeptly called out for being foolish. They're arguably worse off than if they went with a monolith, but neither would be an architecture I'd personally want to work with.
I'm currently working with realntl on a transition to Eventide from a Rails monolith for a client project. We reached a point where the monolith was getting harder and harder to change. This is inevitable in my experience, no matter how careful you think you are.
Though the transition is still in progress, I can say that the path forward is clear and hopeful. This company had previously explored other "SOA" paths (distributed monolith) and it was clear that those were very problematic. Luckily, we were able to steer them to an actual evented architecture.
As I said, it's still early days for this particular project, but if you're feeling your monolith is getting harder and harder to develop on, you should start looking into evented architectures. Eventide (ruby)/messagedb (postgresql) are awesome technologies and would be the first I'd consider. Eventide also has a nice slack community filled with people that are learning and improving their software design skill both in the large (architecture) and the small (individual classes/etc). It's small, but they're good people.
If the question "are there any case studies where microservices went well" is a valid question, then so must be, "are there any case studies where monolithic architecture went well".
My point being that if and only if we have a track record as an entire field of making decisions based on case studies, AND if case studies have a track record of being objective rather than proffered as a result of a marketing agenda, then the question is ultimately legitimate.
There are more shops by total count that fail with monolithic architecture. That's inevitable just based on the infinitesimal number of projects executed as service architectures rather than monoliths at large in the wild. But still, we carry on with monolithic architectural style as if its outcomes were assured.
It's far easier for the vast majority of developers to build a monolith because it allows development to proceed without having to have any knowledge of or practice with the tricks and traps of distributed systems - an entire body of knowledge that a developer might never get meaningful and practical exposure to for an entire career.
The trouble starts when microservices are attempted by developers who can't imagine that there are entire bodies of software development knowledge that developers aren't presently in possession of.
Microservices is just, as Adrian Cockroft used to say, "Service-Oriented Architecture with bounded contexts".
Web development is absolutely not a preparatory course in service oriented architecture. But the vast majority of web developers who attempt to take on SOA while simultaneously presuming an omniscience in all things software development due to their experiences only with monolithic web development will often fail to build a SOA. They usually end up with something that isn't quite SOA and isn't quite a monolith. And that's where the failures largely come from.
I work exclusively in microservices and SOA, and have since 2015. Before that, I worked principally as a web app developer, and did some work off-and-on in SOA implementations. And before that I spent years becoming oriented to the architecture. I don't make the mistakes that web developers typically do when they presume that web development knowledge is a sufficient prerequisite for working in SOA.
So, it's not a question of whether an architectural style works or doesn't. The majority of failures in microservices can be attributed to ignorance and to the narcissism that is permissive of it.
So, I would ask this question instead: Are there any cases where developer over-confidence and over-simplification went well?
These qualities don't tend to serve any architectural style well.
I've never heard of a well-designed SOA not going well. Every single case of microservice project remediation that I've participated in had as the most significant contributing factor an utter disregard for the body of knowledge that the microservices architectural style is built upon.
There are a lot of things that developers can get away with when doing the kinds of tinkering and wandering that typifies typical web development work. But those things don't work once we cross the line into SOA. And unfortunately, the incessant chasing after trivial resumé candy hasn't prepared the average developer for the rigorous mindset needed for SOA work.
As "microservices" became to next fad for perennial fad chasers of the software development world, they finally encountered a kind of work that they could not get away with by faking it. And so, we see a lot of failures. But the vast majority of the failures are personal failures and character failures, rather than failures of an architectural style.
The fat part of the developer bell curve was simply overreaching when it presumed to try to get away with building service architectures with the same level of disinterest in architecture and process that we can get away with in typical web development. Like a kid w...
Like others have mentioned here, simply pointing out examples where microservices have failed doesn't imply that microservices can't succeed. I've attempted to bake bread twice and they both failed. I didn't conclude that baking bread can't be done, but that my skills to do it were insufficient.
There are lots of examples of successful companies using microservices, but I believe the real problem is in defining what constitutes a microservice. Most people call things "microservices" that are nothing of the sort. I can unequivocally say if you built a "service" that depends on other things being 100% available (like another "service") than you haven't built a microservice (ie: those things you built shouldn't be called services).
I wouldn't suggest, however, that anyone can just watch the course, pick up these tools and succeed. Like baking a good loaf of bread, it takes a lot of skill, work and experience. Whether or not you succeed at building microservices is ultimately up to you and your team.
> From an engineering perspective, they track “number of successful stream starts”, instead of percentage of the time 100% of their services are working. That’s a huge red flag.
They don’t measure how often 100% or their services are up because perfect uptime is not the goal and is too expensive (if it’s even possible). If an internally facing service being down doesn’t affect a core metric like number of stream starts by customers, then it’s foolish to treat it as needing 99.999% uptime.
Microservice is not an organizational problem. Microservices is a design problem. If you implement DDD (Domain Driven Design) first to your application and then start to design the application around your DDD concept, then, it might work.
But, it's extremely hard even to do that. Microservices simply complicates things if any of your domains need to share code with each other. Many DDD paradigms exist to address this, but none are practical. For example, authentication related code. IF one domain sets a cookie and the other one has to rely on that to keep the user (a shared model between the two domains) authenticated, then this means, you need to duplicate code bases across two domains or in the very least put them into some sort of shared helper/library, which DDD is kind of against.
That's why it totally makes sense to go Monolith first and really identify the parts of your application that are slowing you down either development wise, testing wise or performance wise and put them into separate contexts.
Phoenix actually does Microservices right. From all the way to scaffold generation to instructing best practices on keeping your domains properly separated. But even then, I've burnt my finger many a times trying to write simple CMS solutions into mutliple microservices then going back to monoliths again.
The takeaway is about trade-offs. They made a rational decision to improve fault isolation by dividing the app into smaller building blocks managed separately. After working with it for a while, they realized the higher operational overhead made the architecture a bad choice for them. So they went back to a monolith architecture and tried to do fault isolation within the boundaries of that architecture, which might have made fault isolation not as good as in the micro-services architecture but it was acceptable.
It's incredibly tough to know the full effect of a trade-off on your organization until you start going down that path.
We're early in the process of adopting a micro-service architecture. With only a handful of services so far, I can already see how a team of two is going to spend a lot more time with operational issues and debugging.
They went to 50+ services in a few months, were applying the same policies across all services. It sounds like they didn't plan well enough and jumped straight into it, without any good DevOps or infrastructure mindset. It was a disaster waiting to happen. This shouldn't be an article that people read and say "Oh Im never using Microservices". This should be an article people read and say WOW that is exactly the right way to NOT break apart a monolith.
327 comments
[ 5.1 ms ] story [ 271 ms ] threadAnd of course the Microservices section should be really thin, while the Monolith section is extremely thick.
Like PopeDotNinja pointed out, you can just flip the book over when one approach starts to peak.
https://computerlibbook.com/
https://www.youtube.com/watch?v=dcfwLhDDMz4
>VCF East XI -- Ted Nelson: Ted Nelson designed the Xanadu hypertext software and wrote the two-in-one personal computing book, Computer Lib / Dream Machines, in 1974. His work deeply influenced the personal computing revolution. Ted earned two Ph.D.s and penned several other well-regarded academic papers and books about ethical, historical, and moral issues in computing.
https://en.wikipedia.org/wiki/Computer_Lib/Dream_Machines
>Computer Lib/Dream Machines is a 1974 book by Ted Nelson, printed as a two-front-cover paperback to indicate its "intertwingled" nature. Originally self-published by Nelson, it was republished with a foreword by Stewart Brand in 1987 by Microsoft Press.
>In Steven Levy's book Hackers, Computer Lib is described as "the epic of the computer revolution, the bible of the hacker dream. [Nelson] was stubborn enough to publish it when no one else seemed to think it was a good idea."
A team that fails to understand how to write modular code, is just going to write spaghetti RPC calls, while having to deal with all the traditional failures and performance issues of distributed computing.
Naturally it is a recipe doomed to fail in the large majority of cases, but it doesn't matter because whoever drove the change is no longer at the company and a new consulting team/new hire gets the money to drive everything back to the monolith.
So goes the money around on plenty of consulting gigs.
This is interesting. I always assumed we were talking about good developers here.
I wonder what's a more likely cause for a failed attempt at microservices. Is it developer incompetence and lack of discipline, or is it environmental factors related to the product and the organization?
(My favourite example is John Romero, part of the very small team that produced Doom - but who also produced Daikatana, which keeps showing up on lists of notoriously bad games.)
https://en.wikipedia.org/wiki/Daikatana
>One advert for the game became notorious; a 1997 poster containing the phrase "John Romero's About To Make You His Bitch[. Suck It Down.]". According to Mike Wilson, the advert was created by the same artist who designed the game's box art under order of their chosen advertising agency. Originally, both he and Romero thought it was funny and approved it. Romero had second thoughts soon after but was persuaded by Wilson to let it pass. Speaking ten years later, Romero said while wary of the slogan at the time, he went along with it as he had a reputation for similar crass phrases. In the same interview, he noted that reactions to the poster tarnished the game's image long before release, and continued to impact his public image and career. In a 2008 blog post concerning the recent activities of Wilson, Romero attributed him for the marketing tactic. This prompted a hostile exchange of public messages between the two at the time.
At least he apologized, though:
https://www.youtube.com/watch?v=BF_sahvR4mw
John Romero Apologizes for Trying to Make You His Bitch:
https://v1.escapistmagazine.com/news/view/100748-John-Romero...
>I'm going to quote our very own Shamus Young here for a moment: For almost a decade, Ion Storm's Daikatana has been the example of "industry waste, arrogance, and incompetence, as well as a universal punchline for things that suck." The shooter was supposed to be an epic vision, the masterpiece of John Romero - the mastermind behind genre-defining Doom and Quake.
>Then it came out in May of 2000, and it sucked. The arrogance and hubris that crippled Daikatana have been well chronicled over the years, but none of it is quite as infamous as the ad you see here to the right: "John Romero's About to Make You His Bitch. Suck it Down." It was a pretty ballsy statement in itself, but after the game's failure simply became laughable.
John Romero Is So Sorry About Trying To Make You His Bitch:
https://kotaku.com/john-romero-is-so-sorry-about-trying-to-m...
>Game designer John Romero and John Romero's hair ruled the roost during the 1990s. With titles like Doom and Quake, he not only helped popularize the first-person shooter, he defined it. Then the unthinkable happened. He made Daikatana.
>[...] Romero, who now says he is resigned to the ad, dished on the ad back in 2008, which evoked a saucy response from the marketer that spearheaded the suck-it-down campaign.
Romero Dishes on the Ad:
https://web.archive.org/web/20081225071219/http://kotaku.com...
>[...] these are the kinds of jackass stunts he pulled [...]
Suck-It-Down Campaign Marketer's Saucy Response:
https://web.archive.org/web/20081225070532/http://kotaku.com...
>[...] and ill advised breast implants strewn across this fair nation [...]
When you have 200 developers working on the same product without well defined boundaries it will be a mess with a monolith, and a mess with microservices.
Also arbitrary rules such as "one service per team" or "one service per employee" that force engineers to jerry-rig things that don't belong together. Either allow them to make a new service, make or a new team, or admit that microservices are not single-purpose and are just a blob of multipurpose code. I've seen this way too much.
Also pure organizational inertia working against good engineering practices: sometimes a team will be severely overworked while others are overstaffed. But god forbid there's a temporary reorganization to improve the work of engineers, so people send tasks to other teams, but there's minimal communication between engineers.
There is no substitute for a good model and responsibility separation, (micro)services or otherwise.
Writer Michael Feathers has an article where he suggests that Microservices are a replacement to encapsulation, all we have to do it use encapsulation well.
https://michaelfeathers.silvrback.com/microservices-and-the-...
We didnt stop at monolithic service. We stopped at monolithic repository+organization. There is now 1 VS2019 solution that covers our entire business. All of our services can be built out from this one solution via various configurations. We've even created additional solution "views" for more focused work (i.e. so you don't have to load a bunch of projects you don't care about for a specific task).
At this point, if we ran into scalability issues with a monorepo, I'd start looking for better source control/CI/CD technologies rather than splitting things up in hopes of arbitrarily keeping git viable. The benefits of having all of your source code in 1 repository with strongly-typed models throughout are impossible to overstate. When checkbuilds complete successfully in GitHub, we know the entire business is clean. Not just 1 little aspect of the product stack.
[0] https://monzo.com/blog/we-built-network-isolation-for-1-500-...
We recently had an interview candidate say this when we questioned the wisdom of having over a thousand microservices: some in languages that only the one developer maintaining them used! For me this is insane, but I digress
Monzo says that they have 800 people, and 1500 services. If we're generous and say 500/800 are developers, then each developer is responsible for 3 services! A team of 6 would have 18 projects in their domain.
Two organisations that I know of who favour the latter are Spotify and Netflix. It has benefits - different languages are good for different jobs and engineers like to be able to choose their tools.
It would be bad if this was taken too far, and something was written in a language only one person knows, but that problem already exists with the technical knowledge if something only has one mantainer.
If you push authority and decision making and responsibility for a service to a (2 pizza) team then guess what, microservices work really well.
If you have vast monolithic centralised production operations teams, and no way in hell is their C-Exec going to assign two of them to look after the user-login service, you might not do so well.
Like most things, the organisation needs to change to get the best out of the opportunities software offers. Those that don't will face increasing friction and eventually die off.
hire("developer", 10, "10x).addToPayroll().office("openplan", "wfh").enforceHRPolicies()
Is all you need
The best thing about this is that you can keep everything in change control and just rollback whenever you need to, or spin up new companies at will.
Occasionally companies actually do this by fragmenting divisions into separate companies, such as outsourcing IT. It has a very broad range of outcomes, from saving to destroying the business.
I've never understood why it needs to be either/or. Is it really that difficult to support a microservice deployment that only represents 50% or even 20-25% of the org/project?
Well finally I might get my own microservice after all.
There are shitty hierarchies and shitty flat organizations, just like there are shitty monoliths and shitty microservices.
Sorry if you actually agree with this more nuanced view, it's just that I've seen Conway's "Law" invoked more than once in this discussion and it drives me bonkers. I get the same way when someone ("Medium Developers" I call them, more than green but less than seasoned who swallow everything the read on Medium as gospel and run around quoting it zealously) quoted liskov substitution principle at me as if it was one of Newton's Laws.
It's also obviously true. The organization builds the architecture. The architecture either helps or hinders the organization. The organization builds a new architecture. There's no indirect connection here. If you've seen hierarchical organizations implement microservices, it's because that organization's complement was a microservices architecture. And likewise for a chaotic organization.
--well, sidetrack: Aren't strongly hierarchical organizations the best suited for microservices? With all the strongly divided responsibilities and whatnot?
It's like a tautology: "In logic, a tautology is a formula or assertion that is true in every possible interpretation."
"Having a code repository for each service was manageable for a handful of destination workers"
Microservices should be introduced to make teams go faster, not to decouple external api endpoints....
Don't make the divisions too large, don't make them too small.
The art is to make them the proper size for the particular company.
If you have a small company you don't need divisions. If you have a large one, you need to make divisions as you no longer can speak to every single employee.
Realizing this and circling back is still a useful life lesson.
And if you happen to leak concerns in your services (in a monolith), it's really easy to adjust, as opposed to having to coordinate the deployment of 5+ services.
And even then, a distributed monolith is still a risk.
Micro-services add cement to your project. Be prepared to keep boundaries you write for a long time.
Discipline is fundamental to good software engineering: you can’t force it with Microservices.
> Any organization that designs a system (defined broadly) will produce a design whose structure is a copy of the organization's communication structure.
I think microservices work well in organizations that are big enough to have a team per microservice. However if you've just split your monolith up and have the same team managing lots of microservices you've made a lot more work for the team without the organisational decoupling which are the real win of microservices.
In my experience it is really difficult to fight Conway's law, you have to work with it and arrange your business accordingly.
You could still do microservices and still fail to deal with Conway’s law.
That's what the poster suggests happened. Nowhere do they suggest that microservices are overall required.
IIRC Fred Brooks pointed out that the # of bugs in a system correlates closely with the # of lines of communication within and between the teams. Joshua Bloch recommends in "Effective Java" that, if possible, 3 potential clients should participate in the design of an API, for the same reason. So a well-designed interface or OpenAPI spec is worth its weight in gold.
Ofc, "microservices" here means separate running instances available on a network. But monoliths can be "service"-oriented as well. OSGi was good for this in Java, but any system able to load shared objects or plugins dynamically can follow the same pattern. And the benefit is that, if your app hits the jackpot and needs to scale outwards, the service interfaces, ie the lines of communication, are already well-defined.
So, service-oriented monolith first, then microservices if needed.
When I worked on a SOA team, I tried to begin any new effort (whether a new API or modification to existing API) solely discussing the API contract. It was (ideally) high-level enough that business analysts and project managers would understand it, and it helped to guide us away from getting mired in implementation discussions too early.
At that organization, we rarely had the opportunity to involve multiple customers at the same time during design discussions (we were typically engaged to help a specific consumer implement a specific feature), but the institutional memory in the SOA team helped us to keep in mind existing/potential other users of each particular webservice.
We had been doing some UML modeling, sequence diagrams during planning, and still having this problem, so rather than repeating the same action and expecting a different outcome I started trying to flip the script. What ended up working was not code diagrams but data flow diagrams and sequences. To get X you need Y, and to derive Z you need A, B, and X. To publish you need all five.
After that, the APIs mostly wrote themselves, we reordered a few different forms, but most importantly variance dropped like a rock.
The great thing about generated-diagrams is that you can easily store and version the original text representation along with the code it describes or applies to.
https://plantuml.com/
This is so accurate. I've heard engineers give state not needing to communicate, chillingly, as a positive for microservices, like "we won't need to talk to each other if all of us are working on different services". My other favorite is using microservices as an excuse for why the product isn't working "oh, my service is working fine, but his service is doing this when it shouldn't", when we're on a small engineering team.
sighhhhhhhhh
API documentation is a medium of communication as much as any user interface.
If you don't keep this in mind, then using your service's application programming interface will be a bad experience.
You have observed that other services also have that quality. Indeed. Nowhere did I say, "all services with decoupled deployment are microservices".
Of course real life is messy and plenty of people realized writing small single purpose services was valuable, and plenty of people build giant "microservices" that have nothing to do with the original term and are just badly constructed monoliths.
I really hate that word when used without further definition.
Which makes the term "microservice" even weirder, given that any microservice is going to be bigger than a single subroutine.
Yes there is. A service is a very generic concept to the point it's only relevant as a high-level concept.
The concept of a microservice makes all the sense in the world if you look back to where we came from: web services. When compared with all the work and requirements and complications of using SOAP and WSDL and UDDI and everything around, just sending small JSON payloads around, and the ability to peel off smaller services leveraging that architecture approach, was a far lighter and uncomplicated way of doing business.
I mean, the name microservices becomes obvious once you look back and all that you see is macroservices.
I think they were saying something more aligned with your opinion than you read it as.
Microservices tended towards a single runtime per service, ensuring the deployment lifecycle was tied to the build lifecycle and thus allowing for independent evolution.
At the beginning Microservice was generally viewed as more granular than SOA, though that’s been backed off of.
But then you’d get some that would make bizarre claims like a microservices must be under 100 lines of code. :shrug:
Cockroft’s Rule of Thumb
Can complete a service in two weeks or less Completed = coded, tested, and in production • Fits in “one or two developers’ heads”
At that rate you quickly hit hundreds of services.
I'll claim that splitting a well-structured monolith into microservices will always make it less maintanable, but it might be worth it if you need to for some reason like elasticity or failure tolerance.
But for the love of god, keep the design open. Don't tie the existence of internal software components to peoples livelihoods.
The claim is that such ties, at the macro-structure level, are inevitable and exist regardless.
The point is then to determine the best way either to restructure the organisation, or, the code base, to cope.
The problem is that the software usually keeps expanding until programmers find it hard to cope. If you split teams up so that some people are only concerned with a certain part of the codebase, chances are you are going to grow the size of the codebase by a quite large factor.
I think there should be an incentive in place to keep the codebase small and understandable by most.
But if your code is living as a few hundred or thousand readable lines in the common codebase, that isn't really a problem. The code is there, readable and working, and if anyone needs to change it they can. If it falls out of fashion, it can be deleted.
Aggressively small teams, with no hands-off middle-management layer.
You can build massive capability around a small number of well-managed message-backbones and a single codebase. By keeping the number of hands small and the structure flat, you force high standards. (Skilled staff won't tolerate distractions caused by bad engineering or inadequate automation.)
Heuristic for analysing firms: who has strategic power in decision-making? Conventional answer: a group of hands-off middle-managers who run on meeting tempo, and who are valued by how many people and systems report into them. Under AST: an engineering effort running on maker tempo in cooperation with a hands-on sales effort.
Microservices tend to have multilateral contracts with other systems in the organisation. This steers all planning towards meetings. This creates middle-management bloat.
That's a tricky way to measure, given that I can eat a large pizza myself in a single sitting ;)
That's not to say that some people are better than others at certain parts of the codebase, but you don't want people fighting to keep old cruft in because it's on their job title (figuratively speaking).
You can organize around customers, use-cases, platforms, concerns or other things. Some might naturally map 1-1 to software components, but the software component should not be the "reason d'aitre" for a team, rather the customer experience or something else which can transcend multiple interations of the software.
Is it wasteful to have whole teams at GOOG, FB et al owning and improving the state of the art of infrastructure? It depends. At a certain point, there are enough internal customers for teams to reach contribution margin positive on engineering initiatives that have no direct but only second order effects on customer experience.
Both have their boatloads of suck, neither is inherently better. Interestingly, trying to mix them to get the benefits of each doesn't seem to invalidate any of their downsides; often it exacerbates them.
My argument here is that it's not so much the size of your team but rather the size and scale of your project that needs to be taken into consideration.
I'm just curious to know, when you said "the easiest and most sensible thing to do is chunk out large parts of the project and put them aside as a microservice" ... were these chunks separately deployable units for external "public" use.
You need to be disciplined to keep a monolith highly modularized. For microservices, in contrast, their architecture encourages modularization.
Microservices actually make you encapsulate your code, at least within the microservices, because you can't call out to it directly. They don't necessarily force you to implement the single responsibility principle, but they do a good job of pushing you. Microservices implement a service-locator pattern through DNS or web routing, one form of the dependency inversion principle. Microservices make you pass data around as entities, instead of Active Record instances.
The price for this sort of thing is very steep, though; distributed systems are inherently icky, harder to trace, and more prone to failure, and besides this, you've added network overhead to each service call.
I wish more engineering teams would consider spending half the effort of microservices on simply disciplining their monoliths. They might get somewhere...
In my experience, if your services are developed by the same people, and not separated by teams, engineers will often tightly couple the services with fragile and opaque dependent changes regardless.
While in monolith this is painful, at least you have a complete stack trace and the ability to run things through a step debugger you orient yourself. In a distributed system tribal knowledge tends to be your only savior.
When we design systems, we need to spend more time thinking about what is most likely to happen as opposed to what we feel should happen.
100%. This is an uphill battle, though. I've encountered so many engineers who equate "real engineering" with "building giant machines." You just can't convince them otherwise.
I've watched people build giant, real-time stream processing pipelines compromising tons of moving pieces (lambda, sqs, s3, sns, stepFunctions, etc..) to build... a reporting table, and all for... 1.3gb of data. Literally.
Ultimately, despite the "sell," I don't think microservices as a forcing function for good practices works in practice. If the team lacks the skills to build a disciplined monolith, then they 100% lack the skills to build a distributed one.
There is this trend in technology - every few years everyone changes their minds about everything:
* 2012 - sql is the best.
* 2016 - sql sucks, nosql is the future
* 2020 - nosql suck, sql is the best.
* 2024 - {fill in the blank}.
The same thing is happening with microservices. But in addition docker, kubernetes and recently unikernels have joined the party. The concept is the same though.
What I am trying to say is that either of those can be good or bad in different scenarios. It's a question of picking the most appropriate one for the situation.
Sun RPC, CORBA, DCE, DCOM, XML-RPC, SOAP, REST, gRPC,....
Honestly, I have never been in an organization so large that this became a necessity (if you solve tens of different problems, that would require almost thousands of developers). But coordinating single developers without an API is hard enough already, I can only assume for teams its nearly impossible.
At least, that's what I understand from his comment.
It's inevitable that the longer the codebase exists, the more difficult it is to maintain. It's a battle that you can't necessarily win and it's turtles all the way down as your dependencies, and their dependencies, tackle the same issues.
All it takes is one or two roughly defined APIs and you've already created the nucleation point for ever-more tech debt, and while you'll be able to tame some of it you won't manage all of it due to business requirements, or other teams depending on private APIs to save time, or whatever else you can imagine. Switch the architecture and you'll either have all your problems bunched in one codebase, or you'll have distributed your problems all over the place.
I'd go as far as saying that a perfect monolith and a perfect distributed architecture are theoretical ideals that require perfect communication to build them.
After 2 years we had over 450 microservices while keeping our AWS bill flat or slightly decreasing.
Presumably you're getting significant value out of the additional engineering work in which case the architecture shift probably makes sense (to stay aligned with the expanded organizational structure), but there are also cases where a small and flexible team maintaining a simple monolith would be much more nimble and cost-effective.
Presumably by definition we’re talking about a few hundred lines of code, or a couple of weeks development time here at most. What does this team do all day otherwise?
On one end you have a giant monolith. All services rolled into one which includes your API, Middle ware and then Database.
One the other end you have microservices which bundle services into individual distinct units with each service being responsible for its API, middleware and database.
Are there any preexisting patterns which seek to combine these two and come up with an architecture which is midway between those two. A few months ago I had read an article about Data Oriented Architecture on HN which comes close though I'm wondering if there are others.
Layered microservices are an antipattern. In most cases, functionality is best divided by domain.
So, SOA?
https://m.signalvnoise.com/the-majestic-monolith-can-become-...
Turning everything into an object can make a small program into a big program, so it’s maybe not such a good idea for small-scale stuff.
http://www.solipsys.co.uk/new/TheParableOfTheToaster.html
However, in my experience, OOP made it possible to do really big stuff.
It’s all about not having a “one-size-fits-all” approach. I don’t think it’s just about scaling architectures; it’s about changing architectures to match scale.
It’s difficult as hell to make these changes, because people get invested in methodology, and insist on applying the same lens to everything we do.
It sounds like they had the right idea, but they probably had the wrong people.
In my experience OOP actually makes programs smaller. Assuming of course they have good programmers/architects and the program itself is larger than "Hello world".
Why don't you try to read carefully what you've just said in the sentence above
Of course, if you guess wrong, you're totally fucked. Well, either that, or you are smart and see it coming in time to rewrite the code that put the complexity in the wrong place.
Hope that helps with the context. I'm not some anti-OOP zealot, and those do exist.
In fact, I have been running into folks, these days, that don't understand it, as, apparently, OOP is becoming "uncool."
I've always been a "right tool for the right job" kind of guy. I started off with ML (Machine Language, not Machine Learning). I am quite comfortable, sitting down with a breadboard, and flashing an OS.
But I remember the old days of OOP, where "classic" structured programmers didn't "get" OOP, and designed these horrific chimeras.
I always make it a point to understand my methodology and drivers "to the bone." Just because someone at a conference said it, doesn't mean that I should use it for everything.
But I have made it a point of personal ethos not to post criticism or polemics, denigrating/excoriating the work of others.
I know that could buy me a lot of clicks (and probably some considerable HN Above The Fold time), but I think we have enough negativity and finger-pointing on the Internet.
If you read my stuff, you won't see much of that. I may, in a rather vague way, allude to something that gives me a frowny-face, but I don't want that to be part of my "personal brand," so to speak.
I do take tremendous personal pride in my work; both coding and writing, and hold myself to a high bar. I may even project that bar onto others (only in some circumstances), but I don't think it's helpful to do so in public.
I find it most gratifying to write a "This is how I do this..." post, as opposed to a "This isn't how you should do it..." post.
In some cases, it is not the best tool for the job, but I find that I tend to use OOP for almost everything I do; large or small.
It isn't of much use in small utility scripts, though, and some languages are just not written to natively support it. In those cases, procedural (or FP) is the way to go.
There are new-ish methodologies, like functional programming, and protocol-oriented programming, that deprecate "classic" OOP. Some folks are using these as backing for declaring OOP "dead."
I suspect that might be a bit...premature. My current fave lang is Swift, which pretty much allows you to use any methodology you want, or mix them together (Will it blend? That is the question).
I have found that it isn't helpful for me to "write off" any methodology, and most of my work is actually a hybrid approach; with elements of multiple methodologies.
BTW: I'm a "lots of words" kind of guy.
Prolix, JAMES Prolix...
Those are anything but new-ish. In programming new often means that some old concept suddenly becomes fashionable. Myself I do not restrict to any single paradigm and use what I believe is the most suitable for current task.
What does happen, though, is that a canon develops, based on these technologies, and [actual] new techniques get created, based on them.
Some of these are nightmares, and need to be strangled before they can crawl off the slab, but sometimes, a gem comes up.
I write about some of my experiences around that here: https://littlegreenviper.com/miscellany/concrete-galoshes/#e...
I remember writing "object-oriented" software for classic C, in the late 1980s. I called it the "faux object" pattern, and was based in state being kept in a structure that was passed around functions. I refined and formalized the idea when I encountered Apple's QuickDraw GX in the early '90s (I suspect that may be the only good thing that I ever got from that sad debacle).
I used the faux object pattern in an SDK I designed in 1994, and it's still being used to this day. Back then, you couldn't pass OOP across module connections, so we had to figure out a way to do it with C.
Nowadays, I can easily pass Swift extensions and virtual implementations across SDKs; no sweat. It's cool.
In general I agree that people should study more the past, but I do not see any point in dismissing new trends just because the do not market their full genealogy upfront.
Perhaps it's more prevalent with OOP programmers, but perhaps it just appears that way because the boilerplate for classes is a bit larger than the boilerplate for functions and structs.
With procedural code, you would need an exceptional programmer to produce a big program. With OOP, an average programmer can deconstruct a problem into its component parts and solve it, mainly because, the human brain can reason about concrete objects more easily, than say, abstract methods like functional programming.
Edit : OOP has encapsulation which, in my view, significantly reduces the cognitive load when thinking about state management in an app. I remwmber writing a small graphics library using Borland Graphics Interface in Turbo C++. It was a breeze to do because I know about 'things' I want on my screen and coded my classes to reflect those things.
That is a big red flag. Microservices that suffer from shared code changes are not really microservices, but a distributed monolith instead.
They might well all share the same basic framework code, of course. Why not share code for recurring concerns like auth?
If you don't want to do that, then you need a simple shared library for that.
The problem is that there is no easy way to draw the line between "this is obviously a trivial library function we should just link into our code" and "this is something we can't share because it would create friction or break our isolation".
Auth is obviously a "service" but phone number formatting as a service seems extreme.
You clearly haven't finished drinking your Kool-Aid yet.
The auth service will likely have to hit a DB anyways. Assuming the microservice call has roughly the same network latency as the DB call and the DB has 0 response time, it would double the total time to perform the auth. It only gets more favorable as DB response times go up.
More generally, I think microservices make sense in scenarios where the time to process the request is longer than the network latency incurred by making it a microservice. Things that have to hit a DB are generally okay. Pure functional things that just compute on CPU and RAM are generally not, unless they're very computationally expensive like running a simulation or something like that.
So lets say we have shared code for doing something like phone formatting. My question for more experienced microservice practitioners is -- does it make sense to create a microservice for preprocessing data in general? Phone number-formating-as-a-service is excessive but creating a microservice for processing data where phone number formatting is just one aspect of this service makes sense to me. All other services can throw data at the data processing service and get data back in some sort of standard and expected way conforming to whatever business logic/processing rules required.
One common example is, instead of having a shared library that reads & verifies JWTs, use a gateway service that handles this before requests reach the upstream service.
This means changes to your organization's JWT code will only require a redeployment of one service, the JWT Auth service.
Microservices using shared libraries -> ok
Microservices "suffering" from shared libraries -> not ok.
If you change code that other services are using, you can break those other services. No way round that.
The only real difference is that services have a slow serialized network interface that fails 4 or 5 orders of magnitude more often than libraries, but can migrate over memory domains.
I generally avoid creating shared libraries, they're a trap. They have a very narrow band of usefulness squeezed in between the more palatable solutions of creating new services or just copy & pasting code and allowing it to diverge for each different use-case.
In other words, if you can share a lot of code between services, a monolith is actually an appropriate architecture.
For example: let's say you have lots of integrations, and you need to scale compute, and parse and generate common data sent to and from the integrations.
You can either have a monolithic integration service which you scale out on load; or you can have integration-specific services that scale out on load and share your data parsing & generation library. Due to multiple concerns, there's no "best slice".
FWIW, scaling out compute is a stronger argument to me for a service boundary than responsibility segregation. Scaling out requires distribution; scaling up complexity doesn't, though it can help for other reasons, like CI/CD. I prefer FaaS architectural patterns with the freedom to share libraries in different functions (images) to services, especially if long-running state is not needed.
Having a shared library is not a bad thing on its own. Making the library a bottleneck is the anti-pattern.
If you wish to have a shared-library of microservices you should be prepared to have multiple versions of it running at the same time without any pressure to update everything at once.
If your shared library is the bottleneck, it means that your microservices are tighly coupled (hence the distributed monolith)
Shared code shackles everything together, like global variables...
What does that mean?
They just went from naive monolith, to naive micro services, then to smart coupling of the two...
This article for me is more about the complexity of managing a large team across different sites where the architecture needs to change rapidly when modularity is absent. They did get a measurable benefit around performance, though. I wonder if Alexandra will comment on the challenges of running a team in an environment of this complexity?
It's the drift and inconsistencies across these concerns across projects that makes deployment and operations less predictable.
I think this article is more evidence against the credibility of multi-repo than against "microservices".
Anecdotally, my current place of work has grown to about 200 engineers, maintains a monorepo, and hundreds of deployed cron jobs, ad-hoc jobs, and "microservices". We have none of the problems discussed here. We invest maybe 20 eng weeks a year in monorepo-specific tooling, and perhaps another 30 eng weeks per year in "microservices"-tooling.
I'm a bit confused, they seem to imply that they need a microservice per costumer/destination, but you generically have one instance (aka process) per costumer not an entire separate codebase. The article seem to use the same term for two different concepts. Or i am missing something?
Everyone seems to have their preferred style of coding, and it is an easy defence mechanism, when presented with anyone who tries and finds it wanting, to say that "Well they didn't do it properly".
You find that with Microservices vs Monolith, Strong types vs Weak types, Exception Handling vs Results, Agile vs Waterfall.
People fragment into camps which turn into echo chambers and it's easy to dismiss anyone who doesn't commit to that cult as being unpure and not worthy of being in the cult anyway.
There's also a clear conflict of interest with SAAS and Cloud providers benefiting from the perception that microservices are the way to go.
Under these circumstances, letting someone else figure out all the issues is the wise thing to do. Thanks to the authors for doing just that.
Especially if you consider RedwoodJS, a new full stack JS framework that's build on Prisma technology (their stuff is an alternative to Rails Active Record ORM). My takeaway is that they provide a similar monolith like experience by acting as a glue between different services.
I started working remotely as a consultant in the early 2000s when my wife and I moved to a remote area. I had several development jobs that used the same monolith pattern: I would embed everything in a web app using Apache Tomcat, taking advantage of work threads for background tasks. The only external services were a database and crontab settings to frequently snapshot databases. This pattern was so easy to code to, so easy to debug and deal with any runtime problems. One customer reported that a system ran without stopping for six years (ouch, no OS upgrades??) until they restarted it on a larger server.
Micro services can be great, but not always the best choice when horizontal scaling is not required.
From an end user perspective, Netflix runs in “constantly degraded” mod.
From an engineering perspective, they track “number of successful stream starts”, instead of percentage of the time 100% of their services are working. That’s a huge red flag.
As a researcher, the monitoring and fault-propagation / modeling work they’ve done to get it to stay up at all is impressive, but it’s not clear all of that tooling would be necessary if they didn’t have to reason about N^2 fault tolerance scenarios, where N = 100’s of microservices. That’s on the order of one fault tolerance scenario for each atom in the universe.
You answered your own question with Netflix. While you're right that it's not clear Netflix would've needed to develop their chaos monkey tooling and the like, it's not clear at all at that an equivalent system is possible as a monolith. Even if a monolithic Netflix system were technically possible, it's not clear if a monolithic system would be organizationally feasible either (Conway's Law)
But the reason I say this is about the team is because I've seen a well built, well groomed service be passed off to an outside team and immediately turned into a disaster. Same service, same business case, just a team without the DevOps savvy and willingness to follow the patterns.
Netflix does some.. unusual things with microservices, mostly with how they treat version rollovers. It's not bad or good, but it looks different from how many other shops handle the same problem and it means asking the question "is everything working?" is extremely difficult but asking the question "how many people are able to start watching?" is pretty easy.
Netflix runs quite well in practice. I think they do redundant service calls, what is the only minimally sane way to develop a distributed system. The funny thing is that I have never seen any serious discussion of redundant calls, except for it being implicit on practical designs on the anecdotal "how it works for us" articles that pop once in a while. Most times people won't even discuss redundant servers. I imagine everybody thinks it's obvious, and well, I would agree, except for the fact that most people I see do not think so.
But well, Netflix couldn't avoid having a distributed system, so they aren't really representative for nearly anybody.
This is much bigger than the typical microservices, and by choice.
That doesn't seem true. I would imagine that at Netflix scale, you probably have request tracing libraries that can give you a graph of service dependencies. Whether it's worthwhile to consume that, or easier to just let Chaos Monkey run rampant is another question.
Also, I very rarely have issues with Netflix. Typically when I do I can just exit the stream and restart it. Anecdata, but I could count on one hand the number of times I've had a title just not play, or Netflix be down entirely.
Not to say that wouldn't happen for SPOF microservices (e.g. your auth servers), but the surface area is potentially larger for monoliths.
In every service architecture I've worked with in the last few years, you could theoretically run every single "service" within the same physical process -- provided they all shared the same runtime/lang the way a monolith does. I tend to start service architectures using Ruby exclusively with the Eventide toolkit, so this is actually a viable approach for most teams I've worked with. But it's never ultimately made sense. Weighing the pros/cons, a consolidated deployment topology wouldn't add any benefit, and it would actually make it far more difficult for the operations folks, practically speaking.
I've helped put services into production that 1. carry out crucial, "core" business logic reliably and efficiently, 2. can be scaled horizontally without changing the code, 3. never raise exceptions or cause outages, and 4. don't need to be touched for years because new features are more naturally composed around them anyways (i.e. open/closed principle). Practically speaking, it's quite a bit easier for human programmers to reach this degree of precision with a smaller program than with a much larger one. And if you add up a lot of these high quality programs, you get a high quality system. Building a high quality system out of a single program is much, much harder for humans in practice.
I'll use crude terms here for a second: every good service put into production brings a net benefit to the organization relative to the same code having been entangled with an existing pile of code. That may sound like a "No True Scotsman" fallacy, but the definition of "good" in this context is precisely the net benefit being added. If you build and deploy services that have a high degree of quality, you get a corresponding benefit. If you build and deploy programs that _don't_ stand on their own, _don't_ leverage durable messaging, and take other programs down with them when they crash, then you get a rather large mess on your hands. In fact, you never had a service architecture at all; we call this failure mode a "distributed monolith."
I'll acknowledge that most attempts at microservice architectures in the wild don't seem to succeed. Anecdotally, they are particularly prone to failure when their architects don't understand the underlying principles of distributed systems all that well; they neglect important considerations like messaging idempotence, the deleterious effect of synchronous request/response messaging, and the need for deliberate, thoughtful design. In other words, they build systems out of N number of microservices, and get N^2 fault tolerance scenarios, which you adeptly called out for being foolish. They're arguably worse off than if they went with a monolith, but neither would be an architecture I'd personally want to work with.
Though the transition is still in progress, I can say that the path forward is clear and hopeful. This company had previously explored other "SOA" paths (distributed monolith) and it was clear that those were very problematic. Luckily, we were able to steer them to an actual evented architecture.
As I said, it's still early days for this particular project, but if you're feeling your monolith is getting harder and harder to develop on, you should start looking into evented architectures. Eventide (ruby)/messagedb (postgresql) are awesome technologies and would be the first I'd consider. Eventide also has a nice slack community filled with people that are learning and improving their software design skill both in the large (architecture) and the small (individual classes/etc). It's small, but they're good people.
My point being that if and only if we have a track record as an entire field of making decisions based on case studies, AND if case studies have a track record of being objective rather than proffered as a result of a marketing agenda, then the question is ultimately legitimate.
There are more shops by total count that fail with monolithic architecture. That's inevitable just based on the infinitesimal number of projects executed as service architectures rather than monoliths at large in the wild. But still, we carry on with monolithic architectural style as if its outcomes were assured.
It's far easier for the vast majority of developers to build a monolith because it allows development to proceed without having to have any knowledge of or practice with the tricks and traps of distributed systems - an entire body of knowledge that a developer might never get meaningful and practical exposure to for an entire career.
The trouble starts when microservices are attempted by developers who can't imagine that there are entire bodies of software development knowledge that developers aren't presently in possession of.
Microservices is just, as Adrian Cockroft used to say, "Service-Oriented Architecture with bounded contexts".
Web development is absolutely not a preparatory course in service oriented architecture. But the vast majority of web developers who attempt to take on SOA while simultaneously presuming an omniscience in all things software development due to their experiences only with monolithic web development will often fail to build a SOA. They usually end up with something that isn't quite SOA and isn't quite a monolith. And that's where the failures largely come from.
I work exclusively in microservices and SOA, and have since 2015. Before that, I worked principally as a web app developer, and did some work off-and-on in SOA implementations. And before that I spent years becoming oriented to the architecture. I don't make the mistakes that web developers typically do when they presume that web development knowledge is a sufficient prerequisite for working in SOA.
So, it's not a question of whether an architectural style works or doesn't. The majority of failures in microservices can be attributed to ignorance and to the narcissism that is permissive of it.
So, I would ask this question instead: Are there any cases where developer over-confidence and over-simplification went well?
These qualities don't tend to serve any architectural style well.
I've never heard of a well-designed SOA not going well. Every single case of microservice project remediation that I've participated in had as the most significant contributing factor an utter disregard for the body of knowledge that the microservices architectural style is built upon.
There are a lot of things that developers can get away with when doing the kinds of tinkering and wandering that typifies typical web development work. But those things don't work once we cross the line into SOA. And unfortunately, the incessant chasing after trivial resumé candy hasn't prepared the average developer for the rigorous mindset needed for SOA work.
As "microservices" became to next fad for perennial fad chasers of the software development world, they finally encountered a kind of work that they could not get away with by faking it. And so, we see a lot of failures. But the vast majority of the failures are personal failures and character failures, rather than failures of an architectural style.
The fat part of the developer bell curve was simply overreaching when it presumed to try to get away with building service architectures with the same level of disinterest in architecture and process that we can get away with in typical web development. Like a kid w...
There are lots of examples of successful companies using microservices, but I believe the real problem is in defining what constitutes a microservice. Most people call things "microservices" that are nothing of the sort. I can unequivocally say if you built a "service" that depends on other things being 100% available (like another "service") than you haven't built a microservice (ie: those things you built shouldn't be called services).
By that token, autonomy is a pretty important factor. The Udi Dahan teachings (https://particular.net/adsd) (currently available for free) promote this style of architecture. A concrete example of a toolkit for building true microservices can be found in Message DB (https://github.com/message-db/message-db) and/or Eventide (http://docs.eventide-project.org/)
I wouldn't suggest, however, that anyone can just watch the course, pick up these tools and succeed. Like baking a good loaf of bread, it takes a lot of skill, work and experience. Whether or not you succeed at building microservices is ultimately up to you and your team.
They don’t measure how often 100% or their services are up because perfect uptime is not the goal and is too expensive (if it’s even possible). If an internally facing service being down doesn’t affect a core metric like number of stream starts by customers, then it’s foolish to treat it as needing 99.999% uptime.
But, it's extremely hard even to do that. Microservices simply complicates things if any of your domains need to share code with each other. Many DDD paradigms exist to address this, but none are practical. For example, authentication related code. IF one domain sets a cookie and the other one has to rely on that to keep the user (a shared model between the two domains) authenticated, then this means, you need to duplicate code bases across two domains or in the very least put them into some sort of shared helper/library, which DDD is kind of against.
That's why it totally makes sense to go Monolith first and really identify the parts of your application that are slowing you down either development wise, testing wise or performance wise and put them into separate contexts.
Phoenix actually does Microservices right. From all the way to scaffold generation to instructing best practices on keeping your domains properly separated. But even then, I've burnt my finger many a times trying to write simple CMS solutions into mutliple microservices then going back to monoliths again.
Did they go back to a monolith service or a monolith repo? It really just sounded like monorepo
It's incredibly tough to know the full effect of a trade-off on your organization until you start going down that path.
We're early in the process of adopting a micro-service architecture. With only a handful of services so far, I can already see how a team of two is going to spend a lot more time with operational issues and debugging.