I'm not sure what they've been left with is a monolith after all. I would say they just have a new service, which is the size of what they should have originally attempted before splitting.
In particular, as to their original problem, the shared library seems to be the main source of pain and that isn't technically solved by a monolith, along with not following the basic rule of services "put together first, split later".
I feel prematurely splitting services like that is bound to have issues unless they have 100 developers for 100 services.
The claim of "1 superstar" is misleading too, this service doesn't include the logic for their API, Admin, Billing, User storage etc etc, it's still a service, one of a few that make up Segment in totality.
Reading about their setup and comparing with some truly large scale services I work with, I'm left with the idea that Segment's service is roughly the size of one microservice on our end.
Perhaps the takeaway is don't go overboard with fragmenting services when they conceptually fulfill the same business role. And regardless of the architecture of the system, there are hard state problems to deal with in association with service availability.
The most telling fact is that it "took milliseconds to complete running the tests for all 140+ of our destinations". I've never worked on a single service whose tests ran that fast, given that the time spent by the overhead of the test framework and any other one-time initialization can take a few seconds just itself. It's great to have tests that run fast, but that's a bit ridiculous.
Some rules of thumb I just came up with:
Number of repos should not exceed number of developers.
Number of tests divided by number of developers should be at least 100.
Number of lines of code divided by number of repos should be at least 5000.
Your tests should not run faster than the time it takes to read this sentence.
A single person should not be able to memorize the entire contents of a single repo, unless that person is Rain Man.
> never worked on a single service whose tests ran that fast
I'd say you've never had good tests.
I have a test-suite for a bunch of my frameworks that dates to the mid 90s, with tests added regularly with new functionality.
It currently takes 4 seconds total for 6 separate frameworks and 1000 individual tests. Which is actually a bit slower than it should be, it used to take around 1-2 seconds, so might have to dig a little to see what's up.
With tests this fast, they become a fixed part of the build-process, so every build runs the tests, and a test failure is essentially treated the same as a compiler error: the project fails to build.
The difference goes beyond quantitative to qualitative, and hard to communicate. Testing becomes much less of a distinct activity but simple an inextricable part of writing code.
So I would posit:
Your tests should not run slower than the time it takes to read this sentence.
Unit tests that don’t read or write to disk and don’t try thousands of repetitions of things should be bleeding fast, but the most useful integration tests that actually help find faults (usually with your assumptions about the associated APIs) often need interaction with your disk or database or external service and tend to take a bit more than a few seconds. I find you need both.
I have tests which verify DNA analysis. The test data vectors are large -- a few hundred MB here, a couple GB there. The hundreds of tests that use these test vectors still run in a few seconds.
If you're using a tape drive or SD cards, sure. But even a 10 year old 5400RPM on an IDE connection should be able to satisfy your tests' requirements in a few seconds or less.
I suspect your tests are just as monolithic as you think microservices shouldn't be. Break them down into smaller pieces. If it's hard to do that, then redesign your software to be more easily testable. Learn when and how to provide static data with abstractions that don't let your software know that the data is static. Or, if you're too busy, then hire a dedicated test engineer. No, not the manual testing kind of engineer. The kind of engineer who actually writes tests all day, has written thousands (or hundreds of thousands) of individual tests during their career. And listen to them about any sort of design decisions.
Sounds like you have tests that need to read (probably cached) data files while the parent poster has tests that need to write to disks (probably in a database transaction). Those are different enough that run times won't ever be comparable.
I have tests that need to read. I have tests that need to write. All data written must also be read and verified. You're right, the data is probably cached.
If you need to access a database in your tests you're probably doing it wrong. Build a mock-up of your database accessor API to provide static data, or build a local database dedicated for testing.
Sure. I'd venture to say that integration tests should be fewer than unit tests, see hexagonal etc. Hopefully those external interfaces are also more stable, so they don't need to be run as often.
I tend to use my integration tests also as characterization tests that verify the simulator/test-double I use for any external systems within my unit tests.
See also: the testing pyramid[1] and "integrated tests are a scam"[2], which is a tad click-bait, but actually quite good.
> I've never worked on a single service whose tests ran that fast, given that the time spent by the overhead of the test framework and any other one-time initialization can take a few seconds just itself. It's great to have tests that run fast, but that's a bit ridiculous.
It's not ridiculous. It's good.
I work on an analysis pipeline with thousands of individual tests across a half dozen software programs. Running all of the tests takes just a few seconds. They run in under a second if I run tests in parallel.
If your tests don't run that fast then I suggest you start making them that fast.
I'd be willing to bet that if you learned (or hired someone with the knowledge of) how to optimize your code, you could get some astounding performance increases in your product.
That cure is worse than the disease. Every service works differently and 80% of them are just wrong, and there’s nothing you can do because Tim owns that bit.
But if you can explain to the team or the CTO why Tim is doing it wrong and how it is impacting X, Y and Z, then Tim will fix or be sent else where, no?
Tim accuses everyone else of being lazy or stupid.
Tom (real guy) was too busy all the time to do anything other than the 80/20 rule. He was too busy because he didn't share. So of course he was a fixture of the company...
Each developer works in their own little silo and doesn't bother to learn the code outside their silo. Each team member developers their own idiosyncratic style. If they have to work with someone else's code, it's unfamiliar and they make slow progress and get cranky.
Now all the developers are going to the CTO or CEO and undermining the other developers, trying to persuade the CTO that so-and-so's code is shit.
I work on that project. Every time some idiot starts talking about 'code coverage' my face turns red. Our code coverage is 1e-10%. Don't talk to me about this 70% bullshit.
It's not just code coverage that matters. It's the code path selection that matters. If you have a ton of branches and you've evaluated all of them once then yeah you sure might have 100% "coverage". But you have 0% path selection coverage since a single invokation of your API might choose true branch on one statement, false branch on another statement, and a second invokation might choose false branch on the first and true on the second.
While the code was 100% tested, the scenarios were not. What happens if you have true/true or false/false? That's not tested.
There's a term for this but I forgot what it is and don't care to go spelunking to find it.
How does the service architecture affect that? Tim could be as protective of a code file as he is of a service. At least with a service you could work around it.
One way is that with different services, its more likely to have both a different language, framework, and paradigms -- that perhaps only Tim is familiar with (that's been my experience). Its definitely got a different repo, perhaps with different permissions.
I felt this article is more about how to use microservices right way vs butchering the idea. It is not right to characterize this as microservices vs monolith service.
Initial version of their attempt went too far by spinning up a service for each destination. This is taking microservices to extreme which caused organizational and maintenance issue once number of destinations increased. I am surprised they did not foresee this.
The final solution is also microservice architecture with a better separation of concerns/functionalities. One service for managing in bound queue of events and other service for interacting with all destinations.
> along with not following the basic rule of services "put together first, split later".
Agreed. I treat services like an amoeba. Let your monolith grow until you see the obvious split points. The first one I typically see is authentication, but YMMV.
Notice I also do not say 'microservices'. I don't care about micro as much as functional grouping.
I can see their reasoning though; most of those services are pretty straightforward I think (common data model in -> transform -> specific outbound API data out -> convert result back to common data model). The challenge they had is that a lot of the logic in each of those services could be reused (http, data transformation, probably logging / monitoring / etc), so shared libraries and such.
It looks to me that the shared library issue got solved by the monorepo approach. They could have gone the monorepo way and still have microservices.
Managing a lot of repos and keeping them consistent with regards do dependencies is not easy. In reality you do not want everyone to use a different version of a dependency. You might allow deviations but ultimately you wand to minimize them.
Next to a monorepo they would also need a deployment strategy allowing them to deploy multiple services (e.g. every service that was affected by the library change) simultaneously, so that after deploying they can still talk to one another. For a single service this is doable enough (start up, wait for green health, route requests to new service instance), but it increases in complexity when there's >1 service. I'm sure the process can be repeated and automated etc, but it will be more complex. Doing zero-downtime deployments for a single service is hard enough.
> the basic rule of services "put together first, split later"
Is this rule mentioned or discussed somewhere? A quick google search links to a bunch of dating suggestions about splitting the bill. Searching for the basic rule of services "put together first, split later" reveals nothing useful.
I've always said if the Linux kernel can be a giant monolith, in C no less, than there's maybe 100 web applications in the world that need to be split into multiple services.
I've worked with microservices a lot. It's a never-ending nightmare. You push data consistency concerns out of the database and between service boundaries.
Fanning out one big service in parallel with a matching scalable DB is by far the most sane way to build things.
> You push data consistency concerns out of the database and between service boundaries.
Sing that from the rooftops. That is exactly my observation as well. All the vanilla "track some resource"-style webapps I've worked on were never designed to cope with a consistency boundary that spans across service boundaries. Turning a monolith into distributed services is hard for that reason - you have to redesign your data access to ensure that consistency boundaries don't span across multiple services. If you don't do that, then you have to learn to cope with eventual consistency; in my experience, most people just don't think that way. I know I have trouble with it. Surely I'm not the only one.
It's worse than that; it's my observation that most microservice architectures just ignore consistency altogether ("we don't need no stinking transactions!") and blindly follow the happy path.
I've never quite understood why people think that taking software modules and separating them by a slow, unreliable network connection with tedious hand-wired REST processing should somehow make an architecture better. I think it's one of those things that gives the illusion of productivity - "I did all this work, and now I have left-pad-as-a-service running! Look at the little green status light on the cool dashboard we spent the last couple months building!"
Programmers get excited about little happily running services, these are "real" to them. Customers couldn't care less, except that it now takes far longer to implement features that cross multiple services - which, if you've decomposed your services zealously enough, is pretty much all of them.
When I was designing a microservices architecture for a former employee, the justification for it was security. We needed separation of the components in the system because we were going to be handling money. It used ZeroMQ for communication, because I wanted something more lightweight and falut-resistant for message passing (in such a sensitive application, we didn't want to trust the network). Although it was of course fun to design, microservices weren't my first choice. It just made sense to us to use it in that particular scenario.
Sometimes, I've seen a lack of regard for data consistency within a monolith.
That is not completely on the developer, either. Pre 4.0 Mongodb, for example, does not do transactions. On the other hand, I've seen some pretty flagrant disregard for it just because there are not atomicity guarantees.
> That is not completely on the developer, either. Pre 4.0 Mongodb, for example, does not do transactions
I'd argue that's on the developer, if he was the one to choose a database that doesn't support transactions, and then didn't implement application-level transactions (which is very hard to do correctly).
Txns are just hard in general but mature RDBMS have spent a huge number of man-years on getting it right. After the time I've spent on Couchbase, Mongo, Elasticsearch, I doubt I'll ever use something non-transactional for anything OLTP or OLTP-adjacent. If Postgres or MySQL can't scale up to handle it, make them...or get a new job. Scale is a PITA.
I've seen it on monoliths too. We run a pretty large app just fine with the "nuclear option". Database transaction isolation level serial. Makes it impossible to have inconsistencies on the database level, and on some RDBMS like PostGres the performance impact is small
>It's worse than that; it's my observation that most microservice architectures just ignore consistency altogether ("we don't need no stinking transactions!") and blindly follow the happy path.
If two microservices have to share databases, they shouldn't be microservices.
One microservice should have write access to one database and preferably, all read requests run through that microservice for exactly the reason you mentioned.
>I've never quite understood why people think that taking software modules and separating them by a slow, unreliable network connection with tedious hand-wired REST processing should somehow make an architecture better.
If you're running microservices between regions and communicating with each other outside of the network it is living in, you're probably doing it wrong.
Microservices shouldn't have to incur the cost of going from SF to China and back. If one lives in SF, all should and you can co-locate the entire ecosystem (+1 for "only huge companies with big requirements should do microservices")
>ustomers couldn't care less, except that it now takes far longer to implement features that cross multiple services - which, if you've decomposed your services zealously enough, is pretty much all of them.
Again, that is an example of microservices gone wrong. You'll have the same amount of changes even in a monolith and I'd argue adding new features is safer in microservices (No worries of causing side effects, etc).
I will give you +1 on that anyway because I designed a "microservice" that ended up being 3 microservices because of dumb requirements. It probably could've been a monolith quite happily.
My problem with microservices is the word 'micro'. It should just be services.
Problem domains (along with organizational structures) inherently create natural architectural boundaries... certain bits of data, computation, transactional logic, and programming skill just naturally "clump" together. Microservices ignore this natural order. The main driving architectural principle seems to be "I'm having trouble with my event-driven dynamically-typed metaprogrammed ball-of-mud, so we need more services!".
It took me longer than it probably should have to realize that the term microservices was made by analogy to the term microkernel. I think that part of my main issue with the "microservices" is that it conflates highly technical semantics with word forms that are a bit more wishy washy in meaning (i.e., a service is something formed more of human perception, not rooted directly in operating system abstractions).
I wonder if microservices will have a similar evolution as microkernels. While some microkernels almost closed the performance gap to monolith kernels, the communication overhead killed them. It turned out that the kernel's complexity could be reduced by moving some functions to the applications ("library OS"). Linux kernel drivers are only accepted when the functionality can't be done in user space. The extreme version is running single purpose, highly specialized unikernels on a hypervisor.
> Problem domains (along with organizational structures) inherently create natural architectural boundaries
The "natural" order is very often bad for reliability, speed and efficiency. It forces the "critical path" of a requests to jump through a number of different services.
In well built SOA you often find that the problem space is segmented by criticality and failure domains, not by logical function.
Yes! But it’s worse again. Because some bright spark comes along and says “hey there’s some commonality between these n services. I’ll make a shared library that will remove that duplicated code.
Then before you know it there are a dozen more shared libraries and you have the distributed monolith.
Then either have to Stand up every micro service every time integration test or make changes and hope for the best.
I've heard of even Unicorns throwing consistency out the window. Apparently Netflix has a bunch of "cleanup jobs" that comb the database for various inconsistencies that inevitably show up.
You can't have consistent microservices without distributed transactions. If a service gets called, and inside that call, it calls 3 others, you need to have a roll back mechanism that handles any of them failing in any order.
If you write to the first service and the second two fail, you need to write a second "undo" call to keep consistent.
Worse, this "undo state" needs to be kept transactionally consistent in case it's your service that dies after the first call.
In reality, nobody does this, so they're always one service crash away from the whole system corrupting the hell out of itself. Since the state is distributed, good luck making everything right again.
Microservices are insane. Nobody that knows database concepts well should go near them
Data consistency is one of those things that sounds like it matters but often doesn't. There's not much in Netflix's platform that screams data consistency is an ultra high priority. Any application that deploys active/active multi-region is by definition going to encounter scenarios where data loss is possible. There's just no way around CAP.
I'd venture a guess that most applications have all sorts of race conditions that could cause data corruption. The fact of the matter is that almost nobody notices or even cares.
A little birdie who worked at Grab once told me that they pretty much don't use transactions anywhere. So... maybe there are problem domains where you can get away without transactions, but I'm quite sure that a marketplace that arranges transportation for fee is not one of them. The next time you're standing in a monsoon waiting for the "coming" car that never comes, remember this post :-)
Aside from that, I've found that even in non-mission-critical scenarios ("it's just porn!") it's incredibly convenient to have a limited number of states the system can be in. It makes debugging easier and reduces the number of edge cases ("why is this null??") you have to handle.
> maybe there are problem domains where you can get away without transactions, but I'm quite sure that a marketplace that arranges transportation for fee is not one of them.
I think you'd be surprised/alarmed at how little transactions actually get used in the software world. Not just on small systems where it doesn't matter but I've seen a complete absence of them in big financial ones handling billions of dollars worth transactions (the real world kind) a day. Some senior, highly paid people even defend this practice for performance reasons because they don't realize the performance cost of implicit transactions. And this is just the in process stuff where transactions are totally feasible, it get's even worse when you look at how much is moved around via csv files to FTP and excel sheets attached to emails. I've spent the last 2 weeks being paid to fix data consistency issues that should never have been issues in the first place.
Maybe when we're teaching database theory we shouldn't start at select/join but at begin transaction/commit/rollback?
Having spent a good amount of time fixing bugs related to code that fetches data with a consistency level of READ_UNCOMMITTED for "performance" reasons I can appreciate what transactions give you as an application developer.
However, I would argue that transactions are overkill. What's the worst case scenario if I book a ride for Grab and my request gets corrupted? I'm guessing I'll see an error message and I'll have to re-request my ride.
Re: I've heard of even Unicorns throwing consistency out the window. Apparently Netflix has a bunch of "cleanup jobs" [to fix "bad" transactions]
If your business model is to be cheap with high volume sales, then corrupting say 1 out of 10,000 customer transactions may be worth it. If you give customers a good price and/or they have no viable alternative, you can live with such hiccups, and the shortcuts/sacrifices may even make the total system cheaper. You are like a veterinarian instead of a doctor: you can take shortcuts and bork up an occasional spleen without getting your pants sued off. But most domains are NOT like that.
I remember using BEA Tuxedo 15 years ago and I can't help wondering if theoretically one could do microservices using it or some OLTP to achieve consistency when needed. Has it progressed over the years? Is there any free alternative to it? Or is it dead tech? When I used it it was hard and too much extra work, but it did the job successfully.
"Need to" and "sane" are among my favourite subjective terms!
(Further below, I'll go into in which contexts I'd agree with your assessment and why. But for now the other side of the coin.)
In the real world, current-day, why do many enterprises and IT departments and SME shops go for µservice designs, even though they're not multimillion-user-scale? Not for Google/Netflix/Facebook scale, not (primarily/openly) for hipness, but they do like among other reasons:
- that µs auto-forces certain level of discipline in areas that would be harder-to-enforce/easier-to-preempt by devs in other approaches --- modularity is auto-enforced, separation of concerns, separation of interfaces and implementations, or what some call (applicably-or-not) "unix philosophy"
- they can evolve the building blocks of systems less disruptively (keep interfaces, change underlyings), swap out parts, rewrites, plug in new features to the system etc
- allows for bring-your-own-language/tech-stack (thx to containers + wire-interop) which for one brings insights over time as to which techs win for which areas, but also attracts & helps retain talent, and again allows evolving the system with ongoing developments rather than letting the monolith degrade into legacy because things out there change faster than it could be rewritten
I'd prefer your approach for intimately small teams though. Should be much more productive. If you sit 3-5 equally talented, same-tech/stack and superbly-proficient-in-it devs in a garage/basement/lab for a few months, they'll probably achieve much more & more productively if they forgoe all the modern µservices / dev-ops byzantine-rabbithole-labyrinths and churn out their packages / modules together in an intimate tight fast-paced co-located self-reinforcing collab flow. No contest!
Just doesn't exist often in the wild, where either remote distributed web-dev teams or dispersed enterprise IT departments needing to "integrate", rule the roost.
(Update/edit: I'm mostly describing current beliefs and hopes "out there", not that they'll magically hold true even for the most inept of teams at-the-end-of-the-day! We all know: people easily can, and many will, 'screw up somewhat' or even fail in any architecture, any language, any methodology..)
Do they actually force a discipline? Do people actually find swapping languages easier with RPC/messaging than other ffi tooling? And do they really attract talent?!
You make some amazing claims that I have seen no evidence of, and would love to see it.
> "You make some amazing claims that I have seen no evidence of"
I'm just relaying what I hear from real teams out there, not intending to sell the architecture. So these are the beliefs I find on the ground, how honest and how based-in-reality they are are harder to tell and only slowly over time at any one individual team.
A lot of this is indeed about hiring though, I feel, at least as regards the enterprise spheres. Whether you can as a hire really in-effect "bring your own language" or not remains to be seen, but by deciding on µs architecture for in-house you can certainly more credibly make that pitch to applicants, don't you think?
Remember, there are many teams that have suffered for years-to-decades from the shortcomings and pitfalls of (their effectively own interpretation of / approach to) "monoliths" and so they're naturally eagerly "all ears". Maybe they "did it wrong" with monoliths (or waterfall), and maybe they'll again "do it wrong" (as far as outsiders/gurus/pundits/coachsultants assess) with µs (or agile) today or tomorrow. The latter possibility/danger doesn't change the former certainties/realities =)
In my experience, there's a lot of cargo culting around microservices. The benefits are conferred by having a strong team that pays attention to architecture and good engineering practices.
Regardless of whether you are a monolith or a large zoo of services, it works when the team is rigorous about separation of concerns and carefully testing both the happy path and the failure modes.
Where I've seen monoliths fail, it was developers not being rigorous/conscientious/intentional enough at the module boundaries. With microservices... same thing.
Also, having a solid architectural guideline that is followed across the company in several places (both in infrastructure and application landscapes) makes up the major bulk of insuring stability and usability.
The disadvantage is obviously that creating such's a 'perfect architecture' is hard to do because of different concerns by different parties within the company/organisation.
> The disadvantage is obviously that creating such's a 'perfect architecture' is hard to do because of different concerns by different parties within the company/organisation.
I think you get at two very good points. One is that realistically you will never have enough time to actually get it really right. The other is that once you take real-world tradeoffs into account, you'll have to make compromises that make things messier.
But I'd respond that most organizations I see leave a lot of room for improvement on the table before time/tradeoff limitations really become the limiting factor. I've seen architects unable to resolve arguments, engineers getting distracted by sexy technologies/methodologies (microservices), bad requirements gathering, business team originated feature thrashing, technical decisions with obvious anticipated problems...
Pure anecdote so I know it is meaningless but I have rewritten/refactored old services with new code, or even new languages twice with little problem because the interface was well defined. We had hundreds of devs working in other areas and we were all on different release cycles because changes were easy and decoupled. We let any team submit bug reports when we either weren't complying with our interface or we had a bug somewhere.
The only teams I had to spend time on were the ones which were on a common DB before we moved off of it.
In my experience if your developers were going to make choices that lead to tight coupling in a monolith, they’re going to make the same choices in a distributed architecture. Only now you’ve injected a bunch of network faults and latency that wouldn’t have been there otherwise.
In this case it sounds like they started with a microservice architecture, but CI/CD automation necessary for robust testing and auto-scaling was not in place. The problem of queues getting backed up might have been addressed by adding a circuit breaker, but instead they chose to introduce shared libraries (again, without necessary testing and deployment), which resulted in very tight coupling of the so-called microservices.
Are you telling me you chose micro services just to enforce coding standards and allow devs to be more comfortable?
The legacy concerns I don’t see being true, as it’s mainly a requirements/documentation problem and you can achieve the same effect with feature toggles.
Except the language argument, don't you get all that by just having modules in your code (assuming a statically typed language since the boundaries are type checked)?
Not really. For example, it's easier to mock a microservice, than a module, for testing purposes. Let's say you have component A and component B, A depends on B (dependency implemented via runtime sync or async call), B is computationally intensive or has certain requirements on resources that make it harder or impossible to test on developer's machine. You may want to test only A: with monolithic architecture you'll have to produce another build of the application, that contains mock of B (or you need something like OSGi for runtime module discovery). When both components are implemented as microservices, you can start a container with mock of B instead of real B.
Running E2E blackbox test is equally simple for all kinds of architectures, especially today, when it's so easy to create a clean test environment with multiple containers even on developer's machine. It may be harder to automate this process for a distributed system, but, frankly speaking, I don't see a big difference between a docker-compose file or a launch script for monolith - I've been writing such tests for distributed systems casually for several years and from my personal experience it's much easier to process the test output and debug the microservices than monolithic applications.
> it's much easier to process the test output and debug the microservices than monolithic applications.
You easier to debug end-to-end tests of a microservice architecture that monolith? That's not my experience. How do you manage to put side by side all the events when they are in dozen of files?
I just don't use files for anything (services should be designed with the assumption that container can be destroyed any time, so files are simply not an option here). If you are talking about the logs, there are solutions like Graylog to aggregate and analyze them.
Using only files for logging is the last thing I would do in 2018.
I use Serilog for structured logging. Depending on the log destination, your logs are either stored in an RDMS (I wouldn’t recommend it) or created as JSON with name value pairs that can be sent directly to a JSON data store like ElasticSearch or Mongo where you can do adhoc queries.
Easy until you have 100,000 of them anyway, in which case it's expensive and slow to run it for every dev. (At that point you have enough devs that microservices 100% make sense, though)
A dependency injection framework where you use flags at the composition root to determine whether the “real” implementation class or the mock class is used based on the environment.
Do you realize that this is actually an anti-pattern, that adds unnecessary complexity and potential security problems to your app? Test code must be separated from production code - something, that should know every developer.
It’s basically a feature flag. I don’t like feature flags but it is a thing.
But if you are testing an artifact, why isn’t the artifact testing part of your CI process? What you want to do is no more or less an anti pattern than swapping out mock services to test a microservice.
I’m assuming the use of a service discovery tool to determine what gets run. Either way, you could screw it up by it being misconfigured.
First of all it is the test code, no matter whether it's implemented as a feature flag or in any other way. Test code and test data shall not be mixed with the production one for many well-documented and well-known reasons: security, additional points of failure, additional memory requirements, impact on architecture etc.
>But if you are testing an artifact, why isn’t the artifact testing part of your CI process?
It is and it shall be part of the CI process. Commit gets assigned build number in tag, artifact gets the version and build number in it's name and metadata, deployment to CI environment is performed, tests are executed against specific artifact, so every time you deploy to production you have a proof, that the exact binary that is being deployed has been verified in its production configuration.
>I’m assuming the use of a service discovery tool to determine what gets run.
Service discovery is irrelevant to this problem. Substitution of mock can be done with or without it.
If you are testing a single microservice and don’t want to test the dependent microservice - if you are trying to do a unit test and not an integration test, you are going to run against mock services.
If you are testing a monolith you are going to create separate test assemblies/modules that call your subject under test with mock dependencies.
They are both going to be part of your CI process then and either way you aren’t going to publish the artifacts until the tests pass.
Your deployment pipeline either way would be some type of deployment pipeline with some combination of manual and automated approvals with the same artifacts.
The whole discussion about which is easier is moot.
Edit: I just realized why this conversation is going sideways. Your initial assumptions were incorrect.
you may want to test only A: with monolithic architecture you'll have to produce another build of the application, that contains mock of B (or you need something like OSGi for runtime module discovery).
> What exactly are you trying to accomplish?
Good test must verify the contract on the system boundaries: in case of the API, it's verification done by calling the API. We are discussing two options here: integrated application, hosting multiple APIs, and microservice architecture. Verification on the system boundaries means running the app, not running a unit test (unit tests are good, but serve different purpose). Feature flags make it only worse, because testing with them covers only non-production branches of your code.
> Your initial assumptions were incorrect.
With nearly 20 years of engineering and management experience, I know very well how modern testing is done. :)
Verification on the system boundaries means running the app, not running a unit test
What is an app at the system boundaries if not a piece of code with dependencies?
If you have a microservice - FooService that calls BarService. The "system boundary" you are trying to test is FooService using a fake BarService. I'm assuming that you're calling FooService via HTTP using a test runner like Newman and test results.
In a monolithic application you have class FooModule that depends on BarModule that implements IBarModule. In your production application you use create FooModule:
var x = FooModule(new BarModule)
y = x.Baz(5);
In your Unit tests, you create your FooModuleL
var x = FooModule(new FakeBarModule)
actual= x.Baz(5)
Assert.AreEqual(10,actual)
And run your tests with a runner like NUnit.
There is no functional difference.
Of course FooModule can be at whatever level of the stack you are trying to test - even the Controller.
You will end up with something like OSGi. That can be the right choice, but is also a quite 'heavyweight' architecture.
For a certain class of applications and organizational constraints, I also would prefer it. But it requires a much tighter alignment of implementation than microservices (e.g., you can't just release a new version of a component, you always have to release the whole application).
For a certain class of applications and organizational constraints, I also would prefer it. But it requires a much tighter alignment of implementation than microservices (e.g., you can't just release a new version of a component, you always have to release the whole application).
Why is that an issue with modern CI/CD tools? It’s easier to just press a button and have your application go to all of your servers based on a deployment group.
With a monolith, with a statically typed language, refactoring becomes a whole lot easier. You can easily tell which classes are being used, do globally guaranteed safe renames, and when your refactoring breaks something, you know st compile time or with the correct tooling even before you compile.
> It’s easier to just press a button and have your application go to all of your servers based on a deployment group.
It's not so much about the deployment process itself (I agree with you that this can be easily automated), but rather about the deployment granularity. In a large system, your features (provided by either components or by independent microservices) usually have very different SLAs. For example, credit card transactions need to work 24x7, but generating the monthly account statement for these credit cards is not time-critical. Now suppose one of the changes in a less critical component requires a database migration which will take a minute. With separate microservices and databases, you could just pause that microservice. With one application and one database, all teams need to be aware of the highest SLA requirements when doing their respective deployments, and design for it. It is certainly doable, but requires a higher level of alignment between the development teams.
I agree with your remark about refactoring. In addition, when doing a refactoring in a microservice, you always need a migration strategy, because you can't switch all your microservices to the refactored version at once.
With separate microservices and databases, you could just pause that microservice. With one application and one database, all teams need to be aware of the highest SLA requirements when doing their respective deployments, and design for it. It is certainly doable, but requires a higher level of alignment between the development teams.
That’s easily accomplished with a Blue-Green deployment. As far as the database, you’re going to usually have a replication set up anyway. So your data is going to live in multiple databases anyway.
Once you are comfortable that your “blue” environment is good, you can slowly start moving traffic over. I know you can gradually move x% of traffic every y hours with AWS. I am assuming on prem load balancers can do something similar.
If your database is a cluster, then it is still conceptually one database with one schema. You can't migrate one node of your cluster to a new schema version and then move your traffic to it.
If you have real replicas, then still all writes need to go to the same instance (cf. my example of credit card transactions). So I also don't understand how your migration strategy would look like.
blue-green is great for stateless stuff, but I fail to see how to apply it to a datastore.
These are great observations. For anyone interested in going more in depth on the topic, I highly recommend the book Building Evolutionary Architectures
I think this is probably true for larger or more distributed corporate environments, but I think a modular monolith is going to be a more productive and flexible architecture for most teams, and should be the default option for most startups (many of whom are doing microservices from day 1 it seems).
1. Is auto-enforced modularity, separation of concerns, etc actually better than enforcing these things through development practices like code review? Why are you paying people a 6 figure salary if they can't write modular software?
2. Is the flexibility you gain from this loose coupling worth the additional costs and overhead you incur? And is it really more flexible than a modular system in the first place? And how does their flexibility differ? With an API boundary breaking changes are often not an option. In a modular codebase they can easily be made in a single commit as requirements change.
3. Is bring-your-own-language actually a good idea for most businesses? Is there a net benefit for most people beyond attracting and retaining talent? What about the ability to move developers across teams and between different business functions? Having many different tech stacks is going to increase the cost of doing this.
I do see the appeal of some of these things, but IMO the pros outweigh the cons for a smaller number of businesses than you've mentioned. And the above is only a small sample of that. Most things are just more difficult with a distributed system. It's going to depend on the problem space of course, but most backend web software could easily be written in a single language in a single codebase, and beyond that modularization via libraries can solve a lot of the same problems as microservices. I'm very skeptical of the idea that microservices are somehow going to improve reliability or development speed unless you have a large team.
I was doing this with COM twenty years ago. It had the same advantages of modularity and language independence but without the unnecessary headaches of a distributed system.
I take your point, but it saddens me that there aren't better ways of achieving this modularity nowadays.
I was too, of course a distributed system could also be built with DCOM and MTC (later COM+), and the DTC (Distributed Transaction Coordinator) could be used when you needed a transaction across services (or DBs, or MQs). Obviously the DTC was developed in recognition of the fact that distributed transactional service calls were a real requirement - something that current microservice architectures over HTTP REST don't seem to support.
It depends but in general I'd disagree. If the small balls of mud have all kinds of implicit dependencies, but you have to find them by searching across codebases (and languages) -- does that sound easier than finding them all in the same codebase and language? Overall, its comparing bad design to bad design. I think the main argument I'd make here is, micro-services doesn't actually solve the big ball of mud problem, it solves a completely different problem.
You can step-through your big ball of mud in a debugger, but you can't do the same with your small balls of mud. Not easily at least. That alone makes a huge difference.
They're both bad. It feels like a reverse Sophie's Choice to have to pick one.
The real friction in the system is always in the boundaries between systems. With microservices it's all boundaries. Instead of a Ball of Mud you have Trees and No Forest. Refactoring is a bloody nightmare. Perf Analysis is a game of finger pointing that you can't defuse.
I generally agree that for the vast majority of web applications KiiS holds true and use monolith.
In terms of the common code divergence why not just use private NPM and enforce latest? Have a hard-fast rule that all services must always use the latest version of common.
Right, but the thing that makes Linux actually useful isn't really the kernel is it? I would say what makes it useful is all the various small, targeted programs (some might call them microservices) it lets you interact with to solve real world problems.
If Linux tried to be an entire computing system all in one code base, (sed, vim, grep, top, etc., etc.) what do you think that would look like code base/maintainability wise? Sounds like a nightmare to me.
> If Linux tried to be an entire computing system all in one code base, (sed, vim, grep, top, etc., etc.) what do you think that would look like code base/maintainability wise?
> Same with BSD; the userspace programs are not part of the kernel, and developed separately by the same entity.
they're part of the same repo and built at the same time than the kernel. Run a big "make universe" here : https://github.com/freebsd/freebsd and see for yourself. That they are different binaries does not matter a lot, it's just a question of user interface. See for instance busybox where all the userspace is in a single binary.
I would argue that the fact that they're in the same repo does not make it a monolith at all. All of google and facebook is in a single repo and built at the same time, but it doesn't make them monoliths.
With web apps the main concern is data consistency between relations. On the OS level you have these same concerns with memory and disk, and there's database-like systems in the kernel and drivers to handle it. Essentially all these utilities are running within the same "database" which is disk and memory management handled by the kernel. Usually microservices have their own databases, which is where consistency hell begins
Unless you mean "their own database tables", not "database servers". But that's just the same as having multiple directories and files in a Unix filesystem.
That architecture is commonly known as a distributed monolith. If you put two services on the same DB you can guarantee that someone will be joining onto a table they shouldn't have before the week is out.
If you can afford all your components sharing the same database without creating a big dependency hell, then your problem is _too small_ for microservices.
If your problem is so large that you have to split it up to manage its complexity, start considering microservices (it might still not be the right option for you).
Yes, they do fine as examples. I have worked both on decades old booking systems and banking software. Both were dependency hells. Refactoring was impossible. Everybody was super-careful even with tiny changes because the risk to break something was just too high.
If it could be avoided, these systems were not touched anymore. Instead, other applications where attached to the front and sides.
I don't think so. For example, the Linux kernel is old, but still quite well maintainable.
So I would say: it applies to systems where proper modularization was neglected. In the anecdotical cases I referred to, one major element of this deficiency was a complex database, shared across the whole system.
To be blunt, its news to a lot of people, but it also isn't wrong. Microservices really shouldn't share a database, and if they do then they aren't "microservices".
(a) a prototype
(b) a set of applications that share a database.
You would have to have an oddly disconnected schema if modifications to the program don't result in programs accessing parts of the database that other programs are already accessing. If this isn't a problem it means you're using your database as nature intended and letting it provide a language-neutral, shared repository with transactional and consistency guarantees.
so maybe not microservices, but fine nonetheless.
EDIT: two more comments:
- this is exactly what relational databases were designed for. If people can't do this with their micro-services, maybe their choice of database is the issue.
- "micro-service" as the original post suggests, is not synonymous with good. "monolith" is only synonymous with bad because it got run-over by the hype-train. If you have something that works well, be happy. Most people don't.
I think you are exactly right. Microservice architectures are definitely not automatically good, and there is nothing wrong with a well architected "monolith".
If you're going to go microservices, you want service A to use service B's public API (MQ or RPC or whatever), not to quietly depend on the schema B happened to choose. And sharing a database server instance turns overloads into cascading failures, unless the stack is very good at enforcing resource limits on noisy neighbors.
While that's true. It also means you are only one grant away from sharing database tables. Maybe with good discipline you will be ok but all it take is for one dev to take that one shortcut.
Agreed. I've only seen proper user separation happen when legal gets involved. Usually it's an all-access free-for-all, and after all, why shouldn't it be? It's a lot easier to pull data and logs from adjacent services yourself rather than taking a ticket and hoping the team still exists for a response.
If you have different database servers then you are only a "connection string" away from breaking your separation. Using a different login and schema is perfectly fine in a lot of architectures.
Nah, just use a database that can scale with your app and run a ton of instances. You'll be hard pressed to find any size app that doesn't fit in RAM these days.
But there is a big difference. These small targeted programs are invoked in user land, usually by the user. Microservices get invoked directly by the user when debugging is going on. Otherwise they are expected to automagically talk to each other and depending on the abstraction even discovery each other automatically.
Also I can pipe these tools together from the same terminal session, like
tail -f foo | grep something | awk ...
You don't have that in general with Microservices. Unix tools are Lego, Microservices aren't. They are Domino at best.
Probably one could come up with an abstraction to do Lego with Microservices but we're not there yet.
And they can be a right mess to wrangle because upstream gets into some artistic frenzy, and shouts down anyone that worries about the ensuing breakages as luddites and haters.
And still nobody would argue to pack everything together into one executable because of these issues. As everything in software engineering, it is about finding the right trade-off.
The big problems with microservices come from distributed transactions being difficult and clocks getting out of sync. Multiple services on a single machine don't have that problem.
My understanding is that OpenBSD is constructed this way, and I have also heard that their code is well-organized and easy to follow.
There is the "base" system which is the OS itself and common packages (all the ones you mentioned), then there is the "ports" repo which contains many open-source applications with patches to make them work with OpenBSD.
I think OpenBSD has reaped many of the same advantages described by Segment with their monorepo approach, such as easily being able to add pledge[1] to many ports relatively quickly.
The whole thing is released and deployed together, which is the usual distinction between microservices and not. I don't think anyone's advocating not using modules and folders to structure your code in your repository (certainly I hope not), but a single deployment artifact makes life a lot easier.
One example I'm familiar with that sounds like microservices is the Robot Operating System (ROS). At its heart it's just a framework for pub/sub over IP, it just happens to be targeted towards robotics. A ROS system comprises of 'nodes' for each logical operation e.g. image acquisition -> camera calibration -> analysis -> output.
The system is defined by a graph of these nodes, since you can pipe messages wherever they're needed; all nodes can be many-many. Each node is a self-contained application which communicates to a master node via tcp/ip (like most pub/sub systems a master node is requried to tell nodes where to send messages). So you can do cool stuff like have lots of seprate networked computers all talking to each other (fairly) easily.
It works pretty well and once you've got a particular node stable - e.g. the node that acquires images - you don't need to touch it. If you need to refactor or bugfix, you only edit that code. If you need to test new things, you can just drop them into an existing system because there's separation between the code (e.g. you just tell the system what your new node will publish/subscribe and it'll do the rest).
There is definitely a feeling of duct tape and glue, since you're often using nodes made by lots of different people, some of which are maintainted, others aren't, different naming conventions, etc. However, I think that's just because ROS is designed to be as generic as possible, rather than a side effect of it running like a microservice.
There's actually no reason you can't architect microservices like this. You can put RabbitMQ or some AMQP service as a comm layer between services. But then you have to architect your system to be event-driven. It's not a bad approach.
I've worked on a system that used Kafka in-between. It makes consistency issues a ton worse because everything is async. At least with HTTP you can do synchronous calls
This is just not true. Google uses almost entirely immediately consistent databases and gRPC internally. Eventual consistency is hardly ever required for REST type calls even at massive scale.
gRPC has no queuing and the connection is held open until the call returns. All of Google's cloud databases are immediately consistent for most operations
My understanding is that Spanner is optimistically immediately consistent, with very clever logic for deciding on whether to bounce the attempted transaction (TrueTime).
But strictly speaking, even inside a single multicore CPU, there is no such thing as immediate consistency. The universe doesn't allow you to update information in two places simultaneously. You can only propagate at the speed of light.
Oh, and the concept of "simultaneous" is suspect too.
Our hardware cousins have striven mightily and mostly successfully for decades to create the illusion that code runs in a Newtonian universe. But it is very much a relativistic one.
IMO, the main benefit of ROS is the message definitions. Having exactly one definition of "point cloud" that everybody accepts means everyone's code will be compatible. That isn't normally the case in library ecosystems. If ROS was replaced by <robotics_types.hpp> I think we'd get 90% of the benefit.
ROS is not a particularly good implementation of this concept though. And it tends to encourage extremely fine splitting of jobs, which basically just turns your robotics problem into a distributed robotics problem, making it much harder in the process (as well as less efficient).
Microservices aren't some magic bullet for scaling. If anything, they conform to Conway's Law [1]. I'd agree, though, if a single engineer is singularly responsible for 2+ microservices that are only supporting a single product...you're doing it wrong.
I think that's possibly wrong as a key advantage of micro service over monolith is scaling. We have a 'single' product built as 2 micro services which is basically an external facing orchestration service that calls the compute heavy backend and it allows the external facing service to serve all the incoming traffic out of (on average) 4 containers whilst the backend computes with 32 containers and scales independently - very small changes to the incoming traffic can have large effects on the volume of traffic going to the compute service.
Teams have a lot to do with microservice use. If you have a small team a monolith can work well. If you have a large team, microservices have some significant advantages. And in some cases, you might have a microservice in Swift when you need high performance, but other less intensive services might be in Ruby or some other language, etc.
Right took for the job should be the goal as opposed to chasing fashion. Microservices are definitely overused, but they do have many legitimate use cases. Your CRUD web application probably doesn’t need microservices, but complex build systems might.
Imagine the complexity of Amazon.com or Netflix were a monolith. But something like Basecamp is probably better as a monolith.
There is also the issue of scale. An ML processor might need more (and different) hardware than a user with system.
Yes, but "monolithic" web applications can be built in the same way. It might not be a 100% accurate usage of the term, but microservice/SOA advocates love to call modular applications monliths anyways, to the point that it's something that most people seem to do.
The key to a large code base is a hierarchical design. People are looking for a technical solution to a design problem. No matter what technology you use if you don't maintain the hierarchical layers your code will turn to mush. Micro services can provide a really strict boundary but if you draw the line in the sand at the wrong place it isn't going to matter.
The kernel can't be categorized fully as monolith: Many kernel subsystems are developed individually, by different teams and merged into the main branch by subsystem.
> 100 web applications in the world that need to be split into multiple services.
Services are not micro services. Most large scale applications can and should be split into multiple services. However, when approaching a new problem you should work within the monolith resisting the service until you absolutely can't any longer. Ideally this will make your services true services, that could capture an entire business unit. When it's all said and done you should be able to sell off the service as a business.
The other use case, which should be obvious, is compliance. If you are thinking about implementing anything that would require PCI or SOX you should do that in a service to shield the rest of the dev org from the complexities. So, any webapp that takes payment and interacts directly a payment processor.
That said, you're correct in that you should not be rolling out a new service to avoid sharding.
This is really missing the point. Microservices are not about code organisation - they are about runtime separation.
And even kernels have kernel threads, which are basically local microservices. Anything which needs to scale beyond a single system is more deserving of microservices than a kernel.
> I've always said if the Linux kernel can be a giant monolith, in C no less, than there's maybe 100 web applications in the world that need to be split into multiple services.
This is apples and oranges. The production profile of operating system kernels and web applications are so dissimilar that the analogy is not useful. It may be true that most web applications don't need to be split into multiple services, but the Linux kernel provides no evidence either way.
I've always said if the Linux kernel can be a giant monolith, in C no less, than there's maybe 100 web applications in the world that need to be split into multiple services.
Similarly it’s made with make. If anyone has a project more complex than the Linux kernel or GCC I’ll gladly listen to why they need some exotic build system... never met anyone yet...
"I've worked with microservices a lot. It's a never-ending nightmare. You push data consistency concerns out of the database and between service boundaries."
This exactly. Me too. Data consistency concerns in sufficiently large real world projects can be practically dealt with only 2 ways IMO: transactions or spec changes.
I think that with the rise in popularity of functions as a service (lambda, gcf, azure), we are heading more and more towards nanoservices.
Small services are easier to develop with several teams, in my opinion. Each team knows what to input, and output. They can do whatever in between as long as these two contracts are respected.
But the overlooking of all these moving pieces changing at different paces is tough. And the smaller the services get, the harder it will become.
>I think that with the rise in popularity of functions as a service (lambda, gcf, azure), we are heading more and more towards nanoservices.
There are some pretty big asterisks next to running "nanoservices". Mostly how expensive they actually are to run at large scale and the weird caveats that can happen due to them not always being up.
And I wouldn't advocate for "always-up nanoservices".
The basic answer to both "nanoservices" and microservices is do what you think is right but don't go too far. There are good reasons to make a nanoservice and good reasons not to, same with microservices.
So they split everything apart because their tests were failing and they didn't want to spend time fixing them, and they they merged it back together by spending time fixing and improving their tests?
It seems like the problem here was bad testing and micro repos, not microservices.
I agree, and their conclusion even features this salient bit:
>However, we weren’t set up to scale. We lacked the proper tooling for testing and deploying the microservices when bulk updates were needed. As a result, our developer productivity quickly declined.
My impression after reading this post was that microservices were symptoms of problems in how their organization wasn't set up to implement them effectively, rather than the actual cause of those problems.
It's amazing to me how people still "meh" away testing as a secondary concern, and then regret it later. Over and over again.
WRITING software is easy, anyone can do it. CHANGING software is extremely difficult. THAT is why we have tests. Also, if you are smart about it, you can get documentation out of the deal for relatively little additional cost.
My go to example is on-boarding new developers:
New dev: "OK, i'm here! How do I start?"
with tests: Clone the repo, install dependencies, and run the test suite. As you develop new features, be sure to write additional test.
They are up and going in a matter of minutes.
without tests: Clone the repo, install deps, download testing database, achieve homeostasis with your dev environment, learn the entire system, build up the state you require to write your feature, iterate on it by hand over and over again.
>without tests: Clone the repo, install deps, download testing database, achieve homeostasis with your dev environment, learn the entire system, build up the state you require to write your feature, iterate on it by hand over and over again.
My first job out of college was like that. I had been doing professional-ish (I was paid and employed but I basically worked alone with no other engineers around) work for two years but this still didn't raise any flags.
They took right approach. Don't treat it like a religion and do what works best for your team and project. There is no single perfect model so be willing to change if it makes sense.
"2020 prediction: Monolithic applications will be back in style after people discover the drawbacks of distributed monolithic applications."
-Kelsey Hightower on Twitter
Been using JS for years and still loving it. The only thing I would move towards from here is some kind of ML like Elm, Reason, Haskell, etc. Certainly wouldn't go back to Java.
Am I understanding correctly that they had 3 engineers and >140 microservices? Microservices definitely have their own costs and tradeoffs, but 140 services and 3 engineers sounds like just a terrible engineering choice.
I feel that when it comes to making any decision about separating things out, you really do have to consider the size of the team you've got before you commit to it. If you've got three devs and they all work across all of the microservices, then what on earth do they add except a hell of a lot of overhead?
If you had ten teams of seven and they managed two services each... it's easier to see how that architecture could actually help.
Same as if you have a two person team building a web-app and they go for client-server architecture rather than a basic full stack web framework. If you're both working the frontend and backend at the same time, save yourself the extra ops effort.
I don't think so, the article says there were 3 engineers working just to keep the system up and running. I'm sure there were many more engineers than that working on and with those services.
Agreed. "Micro" is a terribly defined term, and it sounds like this team went absolutely nuts in one direction. (And then in response to their problems, went as far back as possible to a single monolith.) This suggests a bit of a lack of nuance in their decision making process to me.
I'm not interested in figuring out exactly what the right marketing term for it is, but I've had good experiences with teams of 6-10 engineers owning something like 2-5 services with a larger ecosystem of dozens to hundreds of services. Of course, I've been working at very large companies with extremely high traffic for several years now, so my experience is skewed in that direction.
If I had three engineers on my team I'd be unlikely to end up with more than a small handful of services. Half the benefit of splitting up your services has to do with keeping your independent teams actually independent -- if it's just one team then that isn't a problem in the first place.
> If I had three engineers on my team I'd be unlikely to end up with more than a small handful of services.
We have a small team and have a handful of services. We also have a fully automated CI/CD pipeline. It's worked really well. I doubt any would be considered 'micro', but instead they are designed around functional areas like authentication or backend processing.
> I doubt any would be considered 'micro', but instead they are designed around functional areas like authentication or backend processing.
Yeah, who knows what micro means, but that's exactly how I like to split up services. If at some point a service gets too large, split it up. Hundreds of services out the gate is a gross premature optimization. (And like most premature optimizations, ends up costing much more time both in development and in maintenance.)
At the time we did the split, we had <30 destinations and 3-4 engineers working on this system. At the time we decided to merge, we had > 140 destinations with ~8 engineers, of which roughly 3 full-time engineers spending most of their time just keeping the system alive.
How much of the code was shared among these services? It sounds like you essentially had mostly the same code running in 140 different configurations with only some translation logic and glue varying between each. I'm not surprised you found this untenable. This is akin to running a microservice per web page.
Personally, I don’t like the terms ‘microservices’ or ‘nanoservices.’ What’s the value add in describing the relative size of the service? The _domain_ should drive what becomes its own service. Every service should handle the business logic within a particular domain. It’s definitely a goldilocks problem, though, in that there’s a too-small and too-large, and we’re looking for the just-right fit!
That's exactly it. But when you start doing 'microservices' you get the architecture astronauts who go and see into how many silly little services a monolith can be broken up. The end results are as predictable as the original monolith, both end up as an unmaintainable mess in a couple of years.
I predict the same will happen to the 'superstar', it just isn't old enough yet (and at least it was built with some badly needed domain knowledge).
Microservices typically have two goals, performance and modularity. However, porting a typical webapp to a fast, modular compiled language will typically achieve at least one (often two) orders of magnitude performance improvement over a typical interpreted language. We are seeing this to be true more often than not, and a large performance gain off the bat like that may even obviate a lot of the desire to move to microservices.
Furthermore, if one uses modules (as one should), one can arbitrarily and somewhat trivially run those modules either in-process (compiled in) or out-of-process (via REST, gRPC, Cap’n Proto, or another RPC system), e.g., in a separate service/microservice/whatever you want to call it. This gives you a best of both worlds approach where code can be arbitrarily run in a monolith or a separate service as-needed. This changes the thought dynamics from a rigid "monolith vs. microservice" decision to a more fluid process where things can be rather easily changed on a whim. When modularity is the goal, then services become something of a secondary concern.
Microservices are used as something of a sledgehammer to force modularity and performance in languages that lack proper modularity and/or are innately slow or otherwise inefficient, while suffering orchestration costs and the performance penalties of copying data across multiple processes and networks as well as making it harder to derive a single-source of truth in some cases.
Probably a good approach for a typical webappp looking to improve performance would be to first port core logic to a modern, fast, compiled language with modules, then evaluate the performance from there, and then determine if any modules should be split out into separate processes or services.
Like NoSQL, microservices can be (but not always are) a case of the cure being worse than the disease; however, they can also be useful in certain situations or architectures. Like anything in engineering, there are tradeoffs and it depends on your situation.
Not that I disagree with microservices easily going awry, the problem here seems to be traced to shared library code. Each microservice should be as standalone as possible. Your contract with that service is the service contract. Not some shared library.
As soon as you have shared library, you now have coordinated deployments. And that is just not fun and will cause problems.
The trick here is that this does mean you will duplicate things in different spots. But that duplication is there for a reason. It is literally done in two places. When you update the service, you have to do it in a backwards compatible way. And then you can follow with updates to the callers. This makes it obvious you will have a split fleet at some point, but it also means you can easily control it.
This is deployment tooling related though - the ops part of devops really.
If your build and test process doesn't actually exercise your deployment pipeline across versions, then it's not testing anything. I don't think shared libraries are a problem at all - they're probably a good idea - the problem is when they're used as an excuse to not worry about testing your upgrade and rollback scenarios.
I'd pan that criticism a bit wider too: it's not just about having a testing process either - it's about making sure your devs are able to easily use it and watch it work as part of their regular cycle.
This is in fact something I'm about to start working on at my new job for a new project - pushing the development of each service down so someone can write `make test` and not only run tests, but see if what they've done can upgrade between the currently deployed version.
Fair. But don't get to where you expect code to go out as a single deployment if it can't. Advanced tooling helps. So does a more transparent codebase.
What did each service own, if they all shared code? Make those ownership lines as crisp as you can. And unless you have x teams, consider not having too many more than x services.
Doesn’t that get excessive, making network calls to do what could be more naturally expressed as a method call?
If you have two different teams collaborating or you expand beyond what a single box can do, create services. But if you can express things reasonably as a single service, why make things more complicated and error prone?
They were already breaking things into separate services. They violated DRY by putting the same or similar code into all of them. Then instead of cleaning up the DRY violations by refactoring into a shared service or refactoring into a shared library as a static asset, they factored it out into a mushy shared library still in lots of flux.
Whether they were wanting to factor into another service with a defined and mostly static API or a common library with a defined and mostly static API, it was a failure to factor common code into an amorphous blob that gates the release of all the other services. Instead, they've de-modularized the code and called that a success.
If you're drawing hard lines between services and having them talk to one another, having one of these models for at least parts of that makes a lot of sense:
* filter system of the flow-based nature
* REST API that does a transformation and returns transformed data
* message broker with producer/consumer model where the consumer of one queue does a transform and puts the data into another queue
* a full actor model
* a full flow-based model
Too many smart folks that I've worked with, for some reason, just stop thinking critically when it comes to certain ideas.
I was a product manager on a team of really rockstar developers. They all earn at least $200k a year.
Instead of demanding more ambitious projects, you could keep about 99% of them happy by just letting them use the new framework of the week to build their next web app. Their excitement when they were green-lighted to use react was mind-boggling. Building another dumb website with the new framework, yay.
Mindlessly applying microservice architecture is the same issue at heart.
Sounds like you really don't understand development if you think getting to use React isn't a big deal.
React (and others like it, e.g. Vue) really is a huge win and solves a ton of pain points common to front end development. It still has its own pain points, but it's hard to overstate how much of an improvement it is over other older approaches like jQuery.
They're both tools used to build interactive websites.
The parent didn't understand why developers would be happy that React got the greenlight. My response was that React (and Vue) really is a much better tool for building interactive websites than many older tools like jQuery.
> Building another dumb website with the new framework, yay.
He's just astonished how excited they are although what they build is "another dumb website" in his opinion. Many developers love the "how?" and don't care about the "why?". I don't judge it, but it can explain some of the excitement. Angular, React and Vue are great for development. But Ember, Backbone and others were also good back in their times. What he essentially says is "We use specific technology so they get excited although the problems we solve are boring af"
The reality is that since the days of the social contract between employer and employee are long gone - i.e. I show a company loyalty and they will keep me at least at market rates and not lay me off to “increase shareholder value.
Developers have to use the new and shiny to keep themselves marketable and be ready to jump ship at the first opportunity or out of necessity.
How frequently are developers being laid off? And why should companies just keep people around if they aren’t adding value? The lack of that “social contract” has resulted in much higher wages. Look at developer salaries in Paris compared to New York. Since there is more job security in France, the trade off is that “market rates” are dramatically lower.
I’m not saying that the lack of a social contract for developers is bad. I’m just saying it is a thing. But if you’re wondering why developers always want to do the new and shiny even if the underlying product is boring, they are probably doing resume driven development and looking at thier next opportunity.
My salary has gone up by $45K - $50K in the last 4 years by changing jobs three times. I’m not complaining.
But on the companys’ side, it was completely illogical not to pay me market rates. They still had to pay my replacement market rates and they loss institutional knowledge when I left.
Why is it that HR will approve a req for a new developer at market rates but have strict limits on what they can pay current employees.
There is different social contract there: in Paris the "fraternite, egalite" means you cannot pay developers more than the other people around and high taxes are a second way to ensure this will not happen. This makes Paris a prime choice not to do development and in the end gives dramatically lower "market" rates.
Don't forget that "the best of the best" more to places like Silicon Valley where they earn top dollar, the ones that chose to stay in Paris are what is left behind and a few exceptions.
At least this is what it looks from an European point of view.
Programming world is in a constant flux, now you have lit-html that gives you react in 5kb without vdom overhead. And in next few years there will be other interesting projects to look at.
With lit-element you get native elements that are similar to react elements. And only parts of the dom that changed actually do mutations - like with vdom.
> Too many smart folks that I've worked with, for some reason, just stop thinking critically when it comes to certain ideas.
Because doing something new is much more fun that doing something effective. And most software developers don't really get into this job out of doing things effectively - we're get into it out of the sheer fun of it, altough we would never admit it to our bosses and often even to ourselves.
My initial reaction is that it should be the opposite: people who are enthusiasts would want to learn something new because they enjoy using and learning technology, whereas a "workdrone" would focus on being productive because they are focused on work.
I'm not saying you're wrong or that there's anything wrong with either mentality, I just don't find your statement self explanatory. Please do elaborate =]
React/Vue + Hot Reload is a game changer. It's very productive and enjoyable to develop and iterate with these tools.
Granted, it's absolutely true they're often used in places they probably shouldn't be and can easily result in JS bloat. When working on the frontend it's tempting to use them for everything because of how comparatively fast/easy development is. Ultimately the tradeoffs should be considered on a case by case basis.
I don't doubt any of that. But recently I had to choose a framework, and after evaluating all our choices, I ended up choosing plain ol' php, and as vanilla as we could get javascript. For a lot of reasons, but the most important ones were this:
- Next year, the new kid on the block will be the most amazing thing ever created (tm) and everything else will be obsolete (knockoutjs, angular just to name a few)
- If you pack everything now and leave the code untouched, in a couple of months there's a big chance half your tools will need upgrading (angular?) When you upgrade, half the things won't work and you will spend a ton of time fixing things that were perfectly working before, just to get the thing compiling again
-If, down the line, someone else has to take the work where we left, not having the exact development machines set up, they're going to have it much harder to make simple fixes.For example, in six months of development, a simple datepicker javascript library changed the way where it declares the locale config, and we needed to change every declaration of that function to make it work everywhere again. When a framework deprecates a function that you use while you're in development, you fix it. If someone has to pick the project two years later, they are f*
With "old" stuff this doesn't happen. Sure, if you pick php 3 code and try to run it in PHP 7 you'll find problems, but nothing like with webpack/yarn/whatever is doing the packing work this month
Nowadays is all Vue and react, a couple years ago it was Ruby, next year will be something else. In the end, whatever works for you will be good enough, but if you have to think in advance and try to get some future proof code, half the things that are 'what needs to be used right now', will be unmanageable legacy code a year or two down the line
(Just my two cents)
It's highly possible that you..the project manager..were the one who was ill in-formed when it came to the technology aspect of why 200k per year developers were excited by React.
This speaks to the general boring middle that hits many successful software projects. There's a point where it's simply about iteratively improving it while keeping it running 99% of the time.
It's unglamorous, monotonous and boring, but it's wildly important to the continued success of the project. Who can blame engineers for not being interested in trudging through the boring middle?
Interestingly sometimes efforts to keep engineers from leaving, like splitting an application into 100's of micro services, can actually undermine the entire business. Kind of fascinating if you think about it.
A person in charge may be so afraid to lose an employee they fail to see that the very efforts to keep them may derail the success of the business.
Micro Services solve for logical smaller teams, speed, deployment isolation & ton of other problems. Every solution comes with a trade off. There is no perfect solution. It is up to us to decide whether we need a monolith or micro service for our need & use case instead of comparing them.
Not only that, but there is a lot of room between a 'monolith' and 'micro services'. How about medium services? You break somethings up and leave other things combined.
Some people call this... microservices! Common advice is that a microservice should align with a bounded context in domain driven design, which can involve a LOT of code.
Many large companies have millions of LOC behind their microservices. Your average startup probably doesn't.
What's old is new again. The inevitable rewinding of the 2012 architecture "revolution" starts now. I look forward to people in 2026 saying "monoliths are bad, split everything out to services"
100's of problem children sounds like a step too far.
A few services > monolith
monolith > 100's of services.
The big trick with any technology is to apply it properly rather than dogmatically and if you are breaking up your monolith into a 100's(!) of microservices you are clearly not in control of your domain. That's a spaghetti of processes and connections between them rather than a spaghetti of code. Just as bad, just in a different way.
That's basically how microservices are operated on orchestrators like Kubernetes—just substitute "container" for "VM", which is a mostly-academic difference from the perspective of your application. Operations tooling—distributed tracing, monitoring, logging...—is essential.
Probably because when they suggested it they had about 10, growing by 1 per 6 months. Then a few years later they end up with 100, growing by a few a month, and need to rethink.
I though this was an honest and interesting look at a decision which in retrospect was a bad idea. Hopefully it'll stop some other people making similar mistakes (too many repos, too many services, fast changing libraries shared between many services, etc...).
It'd be better if it wasn't framed as having found that the one true way is the monolith, but there are some lessons here for most devs.
- Harder to reason about, refactor, change/add functionality
- You've (basically) turned a lot of the operations your services need to perform into RPCs, probably killing performance on top of everything else, leading to
- More complex/demanding (or just MOAR) infrastructure requirements
- Higher dev, maintenance and infrastructure costs
- Slower delivery of value to the business and customers
- Potentially crippling opportunity costs
It surprises me how often people don't see this coming. Seriously: keep your systems as simple as you possibly can. Unless you're Netflix, dozens or hundreds of microservices probably isn't as simple as you possibly can.
> if a bug is introduced in one destination that causes the service to crash, the service will crash for all destinations.
This sounds dangerous. If I was a destination provider, and returned some garbage, could that take system down?
Also, reading up on centrifuge, instead of breaking up by source, destination; would it be more spatially conservant to create queues by response type, which are finite. The entire time I was wondering why bad requests are put back into the same queue they came from. Shouldn't they be isolated? You could then treat those isolated requests as one unit, so in a destination outage, your not backlogged on your actual compute resources for good requests.
Nonetheless, seems like just another day in an engineers workday :)
Seems to me the problem is the shared libraries. Yes, without sharing it means you have to repeat a fair amount of code, but in most cases the representation that each service cares about is not necessarily the same, which reduces the value of these shared libraries. It seems that they would have solved a lot of the really critical issues by simply not sharing as much code.
I maintained the shared library for five teams trying to move data around.
The biggest challenge is making the shared library forward and backward compatible with itself for at least a few releases in either direction, because not everyone will redeploy at the exact same moment.
If you can't solve that problem everything gets painful. Doing that right was the second hardest part of that job (meetings were the hardest). The job title (securing the data interchange) came in third place.
Or use versioning and a package manager. You should be able to introduce vNext and then have each service update at its own pace, with a deprecation strategy so that service owners are responsible to get off the old version by such-and-such time. But as others mentioned, part of the problem seems to be too many services given the size of the team.
As with J2EE EJBs, microservices conflates two things:
- a strong API between components
- network calls
The former is a very good idea that should be implemented widely in most code bases, especially as they mature, using techniques like modules and interfaces.
The latter is incredibly powerful in some cases but comes at a huge cost in system complexity, performance and comprehensibility. It should be used sparingly.
I probably don't understand their architecture well enough since they don't try and explain it in detail. The blame is also put on microservices whereas it's the architecture that can be fixed to improve performance and the cost of deployment.
A shared library works up until a certain point. If every service uses a shared library then you are already getting to a world of a monorepo. A monorepo for different services should work fine if the overall architecture is feasible.
Surely this is "no true scotsman"? They've tried microservices and it didn't work for them, you're saying that if only they'd "fixed" their micro-services architecture it would have worked?
> While our systems would automatically scale in response to increased load, the sudden increase in queue depth would outpace our ability to scale up, resulting in delays for the newest events.
This strikes me as the core of their problem, and every step taken was a way to bandaid this limitation. Would the cost of moving to faster-scaling infrastructure been so high as rearchitecting the entire system?
> When we wanted to deploy a change, we had to spend time fixing the broken test even if the changes had nothing to do with the initial change.
This seems like a separate and even larger problem. Changes are breaking tests for unrelated code areas? Is the code too tightly coupled? Sounds like it. The unit tests are doing exactly what they're designed to do. Hard to feel sympathy for the person who's breaking them and then trying to figure out a way to sidestep them rather than fix the underlying issues.
The job of these services is to transform their internal event format to 140 different output formats. You can imagine that there is a lot of duplication in the functionality that these services need to do. Are you suggesting that they avoid any shared libraries and just rewrite the same code over and over hundreds of times and update them independently?
If making a change to a shared library breaks half the services, should it have been shared in the first place? It still smells odd to me there’s so much interdependence amongst the individual services break at will.
Maybe, but it’s conceivable to me that with 140 endpoints growing organically it would be very hard for an individual engineer to know for sure what should be abstracted or not. I think it’s a fundamentally hard problem even though on the surface it seems like it should be simple. Adding arbitrary new unaffiliated services is exactly the kind of thing that leads to irreducible complexity and a moving target that is very difficult to design for.
Microservices aren't a magic bullet and won't save you from a poorly designed application. You already need to have very good separation of concerns within your application to make it out of microservices, and at that point an API that communicates over HTTP isn't much different from one that communicates over the application stack.
I liked the rundown and agree there are tactical benefits for building monolithic API's, but one of the core tenants of microservice architecture coming out of the domain driven design space is that monolithic systems hide business logic. I'm pretty sure by moving everything back to a monolith, you've abandon DDD entirely. From a strategic perspective, that's bad for your business.
If you get your DevOps ducks in order, a lot of the issues you have with standing up and maintaining individual microservices should be manageable. I certainly understand the pain of dependency management, but that's also a part of good architecture.
I'm willing to hear more of these stories though. We can always learn about edge cases or even new paradigms that come out of current thinking.
It's refreshing to read an article which challenges common wisdom.
I've endured a lot of suffering at the hands of the microservices fan club. It's good to see reason finally prevail over rhetoric.
It would have been nice if people had written articles like this 2 years ago but unfortunately, people with such good reasoning abilities would probably not have been able to find work back then.
Software development rhetoric is like religion. If you're not on board you will be burned at the stake.
So many times during technical discussions, I had to keep my mouth shut in the name of self-preservation.
> So many times during technical discussions, I had to keep my mouth shut in the name of self-preservation.
This sounds like an issue with being able to articulate why something is or isn't going to net the expected benefits or being able to foresee unexpected risks. Keeping silent is better than throwing out silly hyperbole risks, but not bring up real risks because "they don't want to hear it" is completely bogus. Any solid engineer will bite at another potential risk to ensure they don't find themselves engineered into a corner 65% through a project. Your comment also makes it out like the notion the article is making, monolith over microserves, is gospel for every situation; that in no condition would it ever make sense to use microserves and that only naive zealots would espouse the wisdom (dogma) to use them. You can use any piece of technology poorly, that doesn't mean the core concept is flawed, just that your problem space is different than what that software is trying to solve. Consider using HDFS as a primary data store in place of MySQL where it doesn't make sense and you might cry the wisdom of wishing someone had told you HDFS is terrible and to just use the tried and true MySQL of olden days.
>> This sounds like an issue with being able to articulate why something is...
The issue is not articulation of ideas; the issue is that when all the books, all the articles and all people believe that something is true, there is no amount of articulation which will be able to convince them otherwise.
You have to wait for the hype to go away before even considering bringing up the argument.
781 comments
[ 2.3 ms ] story [ 339 ms ] threadIn particular, as to their original problem, the shared library seems to be the main source of pain and that isn't technically solved by a monolith, along with not following the basic rule of services "put together first, split later".
I feel prematurely splitting services like that is bound to have issues unless they have 100 developers for 100 services.
The claim of "1 superstar" is misleading too, this service doesn't include the logic for their API, Admin, Billing, User storage etc etc, it's still a service, one of a few that make up Segment in totality.
Reading about their setup and comparing with some truly large scale services I work with, I'm left with the idea that Segment's service is roughly the size of one microservice on our end.
Perhaps the takeaway is don't go overboard with fragmenting services when they conceptually fulfill the same business role. And regardless of the architecture of the system, there are hard state problems to deal with in association with service availability.
This gave me a good chuckle. Dunning–Kruger in full effect.
Some rules of thumb I just came up with:
Number of repos should not exceed number of developers.
Number of tests divided by number of developers should be at least 100.
Number of lines of code divided by number of repos should be at least 5000.
Your tests should not run faster than the time it takes to read this sentence.
A single person should not be able to memorize the entire contents of a single repo, unless that person is Rain Man.
I'd say you've never had good tests.
I have a test-suite for a bunch of my frameworks that dates to the mid 90s, with tests added regularly with new functionality.
It currently takes 4 seconds total for 6 separate frameworks and 1000 individual tests. Which is actually a bit slower than it should be, it used to take around 1-2 seconds, so might have to dig a little to see what's up.
With tests this fast, they become a fixed part of the build-process, so every build runs the tests, and a test failure is essentially treated the same as a compiler error: the project fails to build.
The difference goes beyond quantitative to qualitative, and hard to communicate. Testing becomes much less of a distinct activity but simple an inextricable part of writing code.
So I would posit:
Your tests should not run slower than the time it takes to read this sentence.
Either way, the idea of considering slow tests a feature was novel to me.
If you're using a tape drive or SD cards, sure. But even a 10 year old 5400RPM on an IDE connection should be able to satisfy your tests' requirements in a few seconds or less.
I suspect your tests are just as monolithic as you think microservices shouldn't be. Break them down into smaller pieces. If it's hard to do that, then redesign your software to be more easily testable. Learn when and how to provide static data with abstractions that don't let your software know that the data is static. Or, if you're too busy, then hire a dedicated test engineer. No, not the manual testing kind of engineer. The kind of engineer who actually writes tests all day, has written thousands (or hundreds of thousands) of individual tests during their career. And listen to them about any sort of design decisions.
If you need to access a database in your tests you're probably doing it wrong. Build a mock-up of your database accessor API to provide static data, or build a local database dedicated for testing.
I tend to use my integration tests also as characterization tests that verify the simulator/test-double I use for any external systems within my unit tests.
See also: the testing pyramid[1] and "integrated tests are a scam"[2], which is a tad click-bait, but actually quite good.
[1] https://martinfowler.com/bliki/TestPyramid.html
[2] https://www.youtube.com/watch?v=VDfX44fZoMc
It's not ridiculous. It's good.
I work on an analysis pipeline with thousands of individual tests across a half dozen software programs. Running all of the tests takes just a few seconds. They run in under a second if I run tests in parallel.
If your tests don't run that fast then I suggest you start making them that fast.
I'd be willing to bet that if you learned (or hired someone with the knowledge of) how to optimize your code, you could get some astounding performance increases in your product.
Tom (real guy) was too busy all the time to do anything other than the 80/20 rule. He was too busy because he didn't share. So of course he was a fixture of the company...
This is a business decision, reality has no influence here.
That's why they invented the term "Business reality".
Now all the developers are going to the CTO or CEO and undermining the other developers, trying to persuade the CTO that so-and-so's code is shit.
It's not just code coverage that matters. It's the code path selection that matters. If you have a ton of branches and you've evaluated all of them once then yeah you sure might have 100% "coverage". But you have 0% path selection coverage since a single invokation of your API might choose true branch on one statement, false branch on another statement, and a second invokation might choose false branch on the first and true on the second.
While the code was 100% tested, the scenarios were not. What happens if you have true/true or false/false? That's not tested.
There's a term for this but I forgot what it is and don't care to go spelunking to find it.
Happy path?
sqlite calls this "branch coverage"
https://www.sqlite.org/testing.html#statement_versus_branch_...
I felt this article is more about how to use microservices right way vs butchering the idea. It is not right to characterize this as microservices vs monolith service. Initial version of their attempt went too far by spinning up a service for each destination. This is taking microservices to extreme which caused organizational and maintenance issue once number of destinations increased. I am surprised they did not foresee this.
The final solution is also microservice architecture with a better separation of concerns/functionalities. One service for managing in bound queue of events and other service for interacting with all destinations.
Agreed. I treat services like an amoeba. Let your monolith grow until you see the obvious split points. The first one I typically see is authentication, but YMMV.
Notice I also do not say 'microservices'. I don't care about micro as much as functional grouping.
Changing one "shared library" shouldn't mean deploying 140 services immediately.
They had one service to begin with forked for each destination. Of course that was a nightmare to maintain!
They also just might have had too many repos.
Is this rule mentioned or discussed somewhere? A quick google search links to a bunch of dating suggestions about splitting the bill. Searching for the basic rule of services "put together first, split later" reveals nothing useful.
I've worked with microservices a lot. It's a never-ending nightmare. You push data consistency concerns out of the database and between service boundaries.
Fanning out one big service in parallel with a matching scalable DB is by far the most sane way to build things.
Sing that from the rooftops. That is exactly my observation as well. All the vanilla "track some resource"-style webapps I've worked on were never designed to cope with a consistency boundary that spans across service boundaries. Turning a monolith into distributed services is hard for that reason - you have to redesign your data access to ensure that consistency boundaries don't span across multiple services. If you don't do that, then you have to learn to cope with eventual consistency; in my experience, most people just don't think that way. I know I have trouble with it. Surely I'm not the only one.
I've never quite understood why people think that taking software modules and separating them by a slow, unreliable network connection with tedious hand-wired REST processing should somehow make an architecture better. I think it's one of those things that gives the illusion of productivity - "I did all this work, and now I have left-pad-as-a-service running! Look at the little green status light on the cool dashboard we spent the last couple months building!"
Programmers get excited about little happily running services, these are "real" to them. Customers couldn't care less, except that it now takes far longer to implement features that cross multiple services - which, if you've decomposed your services zealously enough, is pretty much all of them.
That is not completely on the developer, either. Pre 4.0 Mongodb, for example, does not do transactions. On the other hand, I've seen some pretty flagrant disregard for it just because there are not atomicity guarantees.
Microservices makes reasoning on that harder.
I'd argue that's on the developer, if he was the one to choose a database that doesn't support transactions, and then didn't implement application-level transactions (which is very hard to do correctly).
If two microservices have to share databases, they shouldn't be microservices.
One microservice should have write access to one database and preferably, all read requests run through that microservice for exactly the reason you mentioned.
>I've never quite understood why people think that taking software modules and separating them by a slow, unreliable network connection with tedious hand-wired REST processing should somehow make an architecture better.
If you're running microservices between regions and communicating with each other outside of the network it is living in, you're probably doing it wrong.
Microservices shouldn't have to incur the cost of going from SF to China and back. If one lives in SF, all should and you can co-locate the entire ecosystem (+1 for "only huge companies with big requirements should do microservices")
>ustomers couldn't care less, except that it now takes far longer to implement features that cross multiple services - which, if you've decomposed your services zealously enough, is pretty much all of them.
Again, that is an example of microservices gone wrong. You'll have the same amount of changes even in a monolith and I'd argue adding new features is safer in microservices (No worries of causing side effects, etc).
I will give you +1 on that anyway because I designed a "microservice" that ended up being 3 microservices because of dumb requirements. It probably could've been a monolith quite happily.
Problem domains (along with organizational structures) inherently create natural architectural boundaries... certain bits of data, computation, transactional logic, and programming skill just naturally "clump" together. Microservices ignore this natural order. The main driving architectural principle seems to be "I'm having trouble with my event-driven dynamically-typed metaprogrammed ball-of-mud, so we need more services!".
Which, strangely enough, ends up looking almost the same thing as running services on a microkernel.
The "natural" order is very often bad for reliability, speed and efficiency. It forces the "critical path" of a requests to jump through a number of different services.
In well built SOA you often find that the problem space is segmented by criticality and failure domains, not by logical function.
Then before you know it there are a dozen more shared libraries and you have the distributed monolith.
Then either have to Stand up every micro service every time integration test or make changes and hope for the best.
You can't have consistent microservices without distributed transactions. If a service gets called, and inside that call, it calls 3 others, you need to have a roll back mechanism that handles any of them failing in any order.
If you write to the first service and the second two fail, you need to write a second "undo" call to keep consistent.
Worse, this "undo state" needs to be kept transactionally consistent in case it's your service that dies after the first call.
In reality, nobody does this, so they're always one service crash away from the whole system corrupting the hell out of itself. Since the state is distributed, good luck making everything right again.
Microservices are insane. Nobody that knows database concepts well should go near them
I'd venture a guess that most applications have all sorts of race conditions that could cause data corruption. The fact of the matter is that almost nobody notices or even cares.
Aside from that, I've found that even in non-mission-critical scenarios ("it's just porn!") it's incredibly convenient to have a limited number of states the system can be in. It makes debugging easier and reduces the number of edge cases ("why is this null??") you have to handle.
I think you'd be surprised/alarmed at how little transactions actually get used in the software world. Not just on small systems where it doesn't matter but I've seen a complete absence of them in big financial ones handling billions of dollars worth transactions (the real world kind) a day. Some senior, highly paid people even defend this practice for performance reasons because they don't realize the performance cost of implicit transactions. And this is just the in process stuff where transactions are totally feasible, it get's even worse when you look at how much is moved around via csv files to FTP and excel sheets attached to emails. I've spent the last 2 weeks being paid to fix data consistency issues that should never have been issues in the first place.
Maybe when we're teaching database theory we shouldn't start at select/join but at begin transaction/commit/rollback?
However, I would argue that transactions are overkill. What's the worst case scenario if I book a ride for Grab and my request gets corrupted? I'm guessing I'll see an error message and I'll have to re-request my ride.
No, you just need loosely-coupled services, where inconsistency in this circumstance doesn't manifest as a problematic end-state.
If your business model is to be cheap with high volume sales, then corrupting say 1 out of 10,000 customer transactions may be worth it. If you give customers a good price and/or they have no viable alternative, you can live with such hiccups, and the shortcuts/sacrifices may even make the total system cheaper. You are like a veterinarian instead of a doctor: you can take shortcuts and bork up an occasional spleen without getting your pants sued off. But most domains are NOT like that.
No architecture principle will help you if you design things the wrong way.
The point of Micro services architecture is to design large working systems, from smaller working systems.
(Further below, I'll go into in which contexts I'd agree with your assessment and why. But for now the other side of the coin.)
In the real world, current-day, why do many enterprises and IT departments and SME shops go for µservice designs, even though they're not multimillion-user-scale? Not for Google/Netflix/Facebook scale, not (primarily/openly) for hipness, but they do like among other reasons:
- that µs auto-forces certain level of discipline in areas that would be harder-to-enforce/easier-to-preempt by devs in other approaches --- modularity is auto-enforced, separation of concerns, separation of interfaces and implementations, or what some call (applicably-or-not) "unix philosophy"
- they can evolve the building blocks of systems less disruptively (keep interfaces, change underlyings), swap out parts, rewrites, plug in new features to the system etc
- allows for bring-your-own-language/tech-stack (thx to containers + wire-interop) which for one brings insights over time as to which techs win for which areas, but also attracts & helps retain talent, and again allows evolving the system with ongoing developments rather than letting the monolith degrade into legacy because things out there change faster than it could be rewritten
I'd prefer your approach for intimately small teams though. Should be much more productive. If you sit 3-5 equally talented, same-tech/stack and superbly-proficient-in-it devs in a garage/basement/lab for a few months, they'll probably achieve much more & more productively if they forgoe all the modern µservices / dev-ops byzantine-rabbithole-labyrinths and churn out their packages / modules together in an intimate tight fast-paced co-located self-reinforcing collab flow. No contest!
Just doesn't exist often in the wild, where either remote distributed web-dev teams or dispersed enterprise IT departments needing to "integrate", rule the roost.
(Update/edit: I'm mostly describing current beliefs and hopes "out there", not that they'll magically hold true even for the most inept of teams at-the-end-of-the-day! We all know: people easily can, and many will, 'screw up somewhat' or even fail in any architecture, any language, any methodology..)
Do they actually force a discipline? Do people actually find swapping languages easier with RPC/messaging than other ffi tooling? And do they really attract talent?!
You make some amazing claims that I have seen no evidence of, and would love to see it.
I'm just relaying what I hear from real teams out there, not intending to sell the architecture. So these are the beliefs I find on the ground, how honest and how based-in-reality they are are harder to tell and only slowly over time at any one individual team.
A lot of this is indeed about hiring though, I feel, at least as regards the enterprise spheres. Whether you can as a hire really in-effect "bring your own language" or not remains to be seen, but by deciding on µs architecture for in-house you can certainly more credibly make that pitch to applicants, don't you think?
Remember, there are many teams that have suffered for years-to-decades from the shortcomings and pitfalls of (their effectively own interpretation of / approach to) "monoliths" and so they're naturally eagerly "all ears". Maybe they "did it wrong" with monoliths (or waterfall), and maybe they'll again "do it wrong" (as far as outsiders/gurus/pundits/coachsultants assess) with µs (or agile) today or tomorrow. The latter possibility/danger doesn't change the former certainties/realities =)
Regardless of whether you are a monolith or a large zoo of services, it works when the team is rigorous about separation of concerns and carefully testing both the happy path and the failure modes.
Where I've seen monoliths fail, it was developers not being rigorous/conscientious/intentional enough at the module boundaries. With microservices... same thing.
The disadvantage is obviously that creating such's a 'perfect architecture' is hard to do because of different concerns by different parties within the company/organisation.
I think you get at two very good points. One is that realistically you will never have enough time to actually get it really right. The other is that once you take real-world tradeoffs into account, you'll have to make compromises that make things messier.
But I'd respond that most organizations I see leave a lot of room for improvement on the table before time/tradeoff limitations really become the limiting factor. I've seen architects unable to resolve arguments, engineers getting distracted by sexy technologies/methodologies (microservices), bad requirements gathering, business team originated feature thrashing, technical decisions with obvious anticipated problems...
The only teams I had to spend time on were the ones which were on a common DB before we moved off of it.
The legacy concerns I don’t see being true, as it’s mainly a requirements/documentation problem and you can achieve the same effect with feature toggles.
You easier to debug end-to-end tests of a microservice architecture that monolith? That's not my experience. How do you manage to put side by side all the events when they are in dozen of files?
I use Serilog for structured logging. Depending on the log destination, your logs are either stored in an RDMS (I wouldn’t recommend it) or created as JSON with name value pairs that can be sent directly to a JSON data store like ElasticSearch or Mongo where you can do adhoc queries.
https://stackify.com/what-is-structured-logging-and-why-deve...
IOrderService -> OrderService in production.
IOrderService ->FakeOrderService when testing.
But if you are testing an artifact, why isn’t the artifact testing part of your CI process? What you want to do is no more or less an anti pattern than swapping out mock services to test a microservice.
I’m assuming the use of a service discovery tool to determine what gets run. Either way, you could screw it up by it being misconfigured.
>But if you are testing an artifact, why isn’t the artifact testing part of your CI process?
It is and it shall be part of the CI process. Commit gets assigned build number in tag, artifact gets the version and build number in it's name and metadata, deployment to CI environment is performed, tests are executed against specific artifact, so every time you deploy to production you have a proof, that the exact binary that is being deployed has been verified in its production configuration.
>I’m assuming the use of a service discovery tool to determine what gets run.
Service discovery is irrelevant to this problem. Substitution of mock can be done with or without it.
If you are testing a single microservice and don’t want to test the dependent microservice - if you are trying to do a unit test and not an integration test, you are going to run against mock services.
If you are testing a monolith you are going to create separate test assemblies/modules that call your subject under test with mock dependencies.
They are both going to be part of your CI process then and either way you aren’t going to publish the artifacts until the tests pass.
Your deployment pipeline either way would be some type of deployment pipeline with some combination of manual and automated approvals with the same artifacts.
The whole discussion about which is easier is moot.
Edit: I just realized why this conversation is going sideways. Your initial assumptions were incorrect.
you may want to test only A: with monolithic architecture you'll have to produce another build of the application, that contains mock of B (or you need something like OSGi for runtime module discovery).
That’s not how modern testing is done.
https://www.developerhandbook.com/unit-testing/writing-unit-...
> Your initial assumptions were incorrect. With nearly 20 years of engineering and management experience, I know very well how modern testing is done. :)
What is an app at the system boundaries if not a piece of code with dependencies?
If you have a microservice - FooService that calls BarService. The "system boundary" you are trying to test is FooService using a fake BarService. I'm assuming that you're calling FooService via HTTP using a test runner like Newman and test results.
In a monolithic application you have class FooModule that depends on BarModule that implements IBarModule. In your production application you use create FooModule:
var x = FooModule(new BarModule)
y = x.Baz(5);
In your Unit tests, you create your FooModuleL
var x = FooModule(new FakeBarModule) actual= x.Baz(5) Assert.AreEqual(10,actual)
And run your tests with a runner like NUnit.
There is no functional difference.
Of course FooModule can be at whatever level of the stack you are trying to test - even the Controller.
For a certain class of applications and organizational constraints, I also would prefer it. But it requires a much tighter alignment of implementation than microservices (e.g., you can't just release a new version of a component, you always have to release the whole application).
https://www.developerhandbook.com/unit-testing/writing-unit-...
It works similarly in almost every language.
For a certain class of applications and organizational constraints, I also would prefer it. But it requires a much tighter alignment of implementation than microservices (e.g., you can't just release a new version of a component, you always have to release the whole application).
Why is that an issue with modern CI/CD tools? It’s easier to just press a button and have your application go to all of your servers based on a deployment group.
With a monolith, with a statically typed language, refactoring becomes a whole lot easier. You can easily tell which classes are being used, do globally guaranteed safe renames, and when your refactoring breaks something, you know st compile time or with the correct tooling even before you compile.
It's not so much about the deployment process itself (I agree with you that this can be easily automated), but rather about the deployment granularity. In a large system, your features (provided by either components or by independent microservices) usually have very different SLAs. For example, credit card transactions need to work 24x7, but generating the monthly account statement for these credit cards is not time-critical. Now suppose one of the changes in a less critical component requires a database migration which will take a minute. With separate microservices and databases, you could just pause that microservice. With one application and one database, all teams need to be aware of the highest SLA requirements when doing their respective deployments, and design for it. It is certainly doable, but requires a higher level of alignment between the development teams.
I agree with your remark about refactoring. In addition, when doing a refactoring in a microservice, you always need a migration strategy, because you can't switch all your microservices to the refactored version at once.
That’s easily accomplished with a Blue-Green deployment. As far as the database, you’re going to usually have a replication set up anyway. So your data is going to live in multiple databases anyway.
Once you are comfortable that your “blue” environment is good, you can slowly start moving traffic over. I know you can gradually move x% of traffic every y hours with AWS. I am assuming on prem load balancers can do something similar.
If your database is a cluster, then it is still conceptually one database with one schema. You can't migrate one node of your cluster to a new schema version and then move your traffic to it.
If you have real replicas, then still all writes need to go to the same instance (cf. my example of credit card transactions). So I also don't understand how your migration strategy would look like.
blue-green is great for stateless stuff, but I fail to see how to apply it to a datastore.
https://www.goodreads.com/book/show/36308520-building-evolut...
1. Is auto-enforced modularity, separation of concerns, etc actually better than enforcing these things through development practices like code review? Why are you paying people a 6 figure salary if they can't write modular software?
2. Is the flexibility you gain from this loose coupling worth the additional costs and overhead you incur? And is it really more flexible than a modular system in the first place? And how does their flexibility differ? With an API boundary breaking changes are often not an option. In a modular codebase they can easily be made in a single commit as requirements change.
3. Is bring-your-own-language actually a good idea for most businesses? Is there a net benefit for most people beyond attracting and retaining talent? What about the ability to move developers across teams and between different business functions? Having many different tech stacks is going to increase the cost of doing this.
I do see the appeal of some of these things, but IMO the pros outweigh the cons for a smaller number of businesses than you've mentioned. And the above is only a small sample of that. Most things are just more difficult with a distributed system. It's going to depend on the problem space of course, but most backend web software could easily be written in a single language in a single codebase, and beyond that modularization via libraries can solve a lot of the same problems as microservices. I'm very skeptical of the idea that microservices are somehow going to improve reliability or development speed unless you have a large team.
I take your point, but it saddens me that there aren't better ways of achieving this modularity nowadays.
You really do have to modularize. In some languages, you can even use separate compilation units for separate modules to enforce the separation.
You can do all of that but get simultaneous deployment, which cuts out whole classes of integration nightmares.
The real friction in the system is always in the boundaries between systems. With microservices it's all boundaries. Instead of a Ball of Mud you have Trees and No Forest. Refactoring is a bloody nightmare. Perf Analysis is a game of finger pointing that you can't defuse.
In terms of the common code divergence why not just use private NPM and enforce latest? Have a hard-fast rule that all services must always use the latest version of common.
If Linux tried to be an entire computing system all in one code base, (sed, vim, grep, top, etc., etc.) what do you think that would look like code base/maintainability wise? Sounds like a nightmare to me.
You mean like BSD?
Busybox is a slightly better example (even though it’s also a userspace program).
they're part of the same repo and built at the same time than the kernel. Run a big "make universe" here : https://github.com/freebsd/freebsd and see for yourself. That they are different binaries does not matter a lot, it's just a question of user interface. See for instance busybox where all the userspace is in a single binary.
That's news to me, and seems insane.
Unless you mean "their own database tables", not "database servers". But that's just the same as having multiple directories and files in a Unix filesystem.
If you can afford all your components sharing the same database without creating a big dependency hell, then your problem is _too small_ for microservices.
If your problem is so large that you have to split it up to manage its complexity, start considering microservices (it might still not be the right option for you).
If it could be avoided, these systems were not touched anymore. Instead, other applications where attached to the front and sides.
So I would say: it applies to systems where proper modularization was neglected. In the anecdotical cases I referred to, one major element of this deficiency was a complex database, shared across the whole system.
You would have to have an oddly disconnected schema if modifications to the program don't result in programs accessing parts of the database that other programs are already accessing. If this isn't a problem it means you're using your database as nature intended and letting it provide a language-neutral, shared repository with transactional and consistency guarantees.
so maybe not microservices, but fine nonetheless.
EDIT: two more comments:
- this is exactly what relational databases were designed for. If people can't do this with their micro-services, maybe their choice of database is the issue.
- "micro-service" as the original post suggests, is not synonymous with good. "monolith" is only synonymous with bad because it got run-over by the hype-train. If you have something that works well, be happy. Most people don't.
This video is strangely relevant in this thread... https://www.youtube.com/watch?v=X0tjziAQfNQ
Also I can pipe these tools together from the same terminal session, like
You don't have that in general with Microservices. Unix tools are Lego, Microservices aren't. They are Domino at best.Probably one could come up with an abstraction to do Lego with Microservices but we're not there yet.
There is the "base" system which is the OS itself and common packages (all the ones you mentioned), then there is the "ports" repo which contains many open-source applications with patches to make them work with OpenBSD.
Here is a Github mirror of their repos: https://github.com/openbsd
I think OpenBSD has reaped many of the same advantages described by Segment with their monorepo approach, such as easily being able to add pledge[1] to many ports relatively quickly.
https://www.slideshare.net/GiovanniBechis/pledge-in-openbsd
The system is defined by a graph of these nodes, since you can pipe messages wherever they're needed; all nodes can be many-many. Each node is a self-contained application which communicates to a master node via tcp/ip (like most pub/sub systems a master node is requried to tell nodes where to send messages). So you can do cool stuff like have lots of seprate networked computers all talking to each other (fairly) easily.
It works pretty well and once you've got a particular node stable - e.g. the node that acquires images - you don't need to touch it. If you need to refactor or bugfix, you only edit that code. If you need to test new things, you can just drop them into an existing system because there's separation between the code (e.g. you just tell the system what your new node will publish/subscribe and it'll do the rest).
There is definitely a feeling of duct tape and glue, since you're often using nodes made by lots of different people, some of which are maintainted, others aren't, different naming conventions, etc. However, I think that's just because ROS is designed to be as generic as possible, rather than a side effect of it running like a microservice.
I get where you are coming from. But you will go async sooner or later if you need any reasonable error recovery or reliability.
The only question is how much pain you will suffer before you do so.
gRPC has no queuing and the connection is held open until the call returns. All of Google's cloud databases are immediately consistent for most operations
But strictly speaking, even inside a single multicore CPU, there is no such thing as immediate consistency. The universe doesn't allow you to update information in two places simultaneously. You can only propagate at the speed of light.
Oh, and the concept of "simultaneous" is suspect too.
Our hardware cousins have striven mightily and mostly successfully for decades to create the illusion that code runs in a Newtonian universe. But it is very much a relativistic one.
You mean GNU coreutils?
[1] https://en.wikipedia.org/wiki/Conway%27s_law
Right took for the job should be the goal as opposed to chasing fashion. Microservices are definitely overused, but they do have many legitimate use cases. Your CRUD web application probably doesn’t need microservices, but complex build systems might.
Imagine the complexity of Amazon.com or Netflix were a monolith. But something like Basecamp is probably better as a monolith.
There is also the issue of scale. An ML processor might need more (and different) hardware than a user with system.
Right tool (and architecture) for the job.
That was called Obidos and it's shortcomings were why Jeff pushed through the services mandate at Amazon.
Gotta put that CS Masters to work somehow. Can't just sit here doing plumbing all day every day.
Maybe there could be a software development corollary to the Politician's Syllogism[1], or even just a webdev one.
1. https://en.wikipedia.org/wiki/Politician%27s_syllogism
Services are not micro services. Most large scale applications can and should be split into multiple services. However, when approaching a new problem you should work within the monolith resisting the service until you absolutely can't any longer. Ideally this will make your services true services, that could capture an entire business unit. When it's all said and done you should be able to sell off the service as a business.
The other use case, which should be obvious, is compliance. If you are thinking about implementing anything that would require PCI or SOX you should do that in a service to shield the rest of the dev org from the complexities. So, any webapp that takes payment and interacts directly a payment processor.
That said, you're correct in that you should not be rolling out a new service to avoid sharding.
And even kernels have kernel threads, which are basically local microservices. Anything which needs to scale beyond a single system is more deserving of microservices than a kernel.
This is apples and oranges. The production profile of operating system kernels and web applications are so dissimilar that the analogy is not useful. It may be true that most web applications don't need to be split into multiple services, but the Linux kernel provides no evidence either way.
Agreed but I also prefer to keep any changing storage data as a separate concern (s3 or similar).
So the trinity of services would be DB, storage, application.
Similarly it’s made with make. If anyone has a project more complex than the Linux kernel or GCC I’ll gladly listen to why they need some exotic build system... never met anyone yet...
This exactly. Me too. Data consistency concerns in sufficiently large real world projects can be practically dealt with only 2 ways IMO: transactions or spec changes.
> You push data consistency concerns out of the database and between service boundaries.
Small services are easier to develop with several teams, in my opinion. Each team knows what to input, and output. They can do whatever in between as long as these two contracts are respected.
But the overlooking of all these moving pieces changing at different paces is tough. And the smaller the services get, the harder it will become.
Not a size fits all, clearly!
https://github.com/fractalide/fractalide/
There are some pretty big asterisks next to running "nanoservices". Mostly how expensive they actually are to run at large scale and the weird caveats that can happen due to them not always being up.
And I wouldn't advocate for "always-up nanoservices".
The basic answer to both "nanoservices" and microservices is do what you think is right but don't go too far. There are good reasons to make a nanoservice and good reasons not to, same with microservices.
It seems like the problem here was bad testing and micro repos, not microservices.
>However, we weren’t set up to scale. We lacked the proper tooling for testing and deploying the microservices when bulk updates were needed. As a result, our developer productivity quickly declined.
My impression after reading this post was that microservices were symptoms of problems in how their organization wasn't set up to implement them effectively, rather than the actual cause of those problems.
It's amazing to me how people still "meh" away testing as a secondary concern, and then regret it later. Over and over again.
WRITING software is easy, anyone can do it. CHANGING software is extremely difficult. THAT is why we have tests. Also, if you are smart about it, you can get documentation out of the deal for relatively little additional cost.
My go to example is on-boarding new developers:
New dev: "OK, i'm here! How do I start?"
with tests: Clone the repo, install dependencies, and run the test suite. As you develop new features, be sure to write additional test.
They are up and going in a matter of minutes.
without tests: Clone the repo, install deps, download testing database, achieve homeostasis with your dev environment, learn the entire system, build up the state you require to write your feature, iterate on it by hand over and over again.
My first job out of college was like that. I had been doing professional-ish (I was paid and employed but I basically worked alone with no other engineers around) work for two years but this still didn't raise any flags.
You basically quoted exactly the setup.
https://twitter.com/kelseyhightower/status/94025989833123840...
Bet you think PHP is sometimes the best tool for the job and that people shouldn't be so quick to dismiss it, too. ;)
If you had ten teams of seven and they managed two services each... it's easier to see how that architecture could actually help.
Same as if you have a two person team building a web-app and they go for client-server architecture rather than a basic full stack web framework. If you're both working the frontend and backend at the same time, save yourself the extra ops effort.
I'm not interested in figuring out exactly what the right marketing term for it is, but I've had good experiences with teams of 6-10 engineers owning something like 2-5 services with a larger ecosystem of dozens to hundreds of services. Of course, I've been working at very large companies with extremely high traffic for several years now, so my experience is skewed in that direction.
If I had three engineers on my team I'd be unlikely to end up with more than a small handful of services. Half the benefit of splitting up your services has to do with keeping your independent teams actually independent -- if it's just one team then that isn't a problem in the first place.
We have a small team and have a handful of services. We also have a fully automated CI/CD pipeline. It's worked really well. I doubt any would be considered 'micro', but instead they are designed around functional areas like authentication or backend processing.
Yeah, who knows what micro means, but that's exactly how I like to split up services. If at some point a service gets too large, split it up. Hundreds of services out the gate is a gross premature optimization. (And like most premature optimizations, ends up costing much more time both in development and in maintenance.)
I predict the same will happen to the 'superstar', it just isn't old enough yet (and at least it was built with some badly needed domain knowledge).
Furthermore, if one uses modules (as one should), one can arbitrarily and somewhat trivially run those modules either in-process (compiled in) or out-of-process (via REST, gRPC, Cap’n Proto, or another RPC system), e.g., in a separate service/microservice/whatever you want to call it. This gives you a best of both worlds approach where code can be arbitrarily run in a monolith or a separate service as-needed. This changes the thought dynamics from a rigid "monolith vs. microservice" decision to a more fluid process where things can be rather easily changed on a whim. When modularity is the goal, then services become something of a secondary concern.
Microservices are used as something of a sledgehammer to force modularity and performance in languages that lack proper modularity and/or are innately slow or otherwise inefficient, while suffering orchestration costs and the performance penalties of copying data across multiple processes and networks as well as making it harder to derive a single-source of truth in some cases.
Probably a good approach for a typical webappp looking to improve performance would be to first port core logic to a modern, fast, compiled language with modules, then evaluate the performance from there, and then determine if any modules should be split out into separate processes or services.
Like NoSQL, microservices can be (but not always are) a case of the cure being worse than the disease; however, they can also be useful in certain situations or architectures. Like anything in engineering, there are tradeoffs and it depends on your situation.
As soon as you have shared library, you now have coordinated deployments. And that is just not fun and will cause problems.
The trick here is that this does mean you will duplicate things in different spots. But that duplication is there for a reason. It is literally done in two places. When you update the service, you have to do it in a backwards compatible way. And then you can follow with updates to the callers. This makes it obvious you will have a split fleet at some point, but it also means you can easily control it.
If your build and test process doesn't actually exercise your deployment pipeline across versions, then it's not testing anything. I don't think shared libraries are a problem at all - they're probably a good idea - the problem is when they're used as an excuse to not worry about testing your upgrade and rollback scenarios.
I'd pan that criticism a bit wider too: it's not just about having a testing process either - it's about making sure your devs are able to easily use it and watch it work as part of their regular cycle.
This is in fact something I'm about to start working on at my new job for a new project - pushing the development of each service down so someone can write `make test` and not only run tests, but see if what they've done can upgrade between the currently deployed version.
What did each service own, if they all shared code? Make those ownership lines as crisp as you can. And unless you have x teams, consider not having too many more than x services.
The more I read about the problems people have with microservices, the more I'm convinced they've never read about flow-based programming.
If you have two different teams collaborating or you expand beyond what a single box can do, create services. But if you can express things reasonably as a single service, why make things more complicated and error prone?
Whether they were wanting to factor into another service with a defined and mostly static API or a common library with a defined and mostly static API, it was a failure to factor common code into an amorphous blob that gates the release of all the other services. Instead, they've de-modularized the code and called that a success.
If you're drawing hard lines between services and having them talk to one another, having one of these models for at least parts of that makes a lot of sense: * filter system of the flow-based nature * REST API that does a transformation and returns transformed data * message broker with producer/consumer model where the consumer of one queue does a transform and puts the data into another queue * a full actor model * a full flow-based model
I was a product manager on a team of really rockstar developers. They all earn at least $200k a year.
Instead of demanding more ambitious projects, you could keep about 99% of them happy by just letting them use the new framework of the week to build their next web app. Their excitement when they were green-lighted to use react was mind-boggling. Building another dumb website with the new framework, yay.
Mindlessly applying microservice architecture is the same issue at heart.
React (and others like it, e.g. Vue) really is a huge win and solves a ton of pain points common to front end development. It still has its own pain points, but it's hard to overstate how much of an improvement it is over other older approaches like jQuery.
The parent didn't understand why developers would be happy that React got the greenlight. My response was that React (and Vue) really is a much better tool for building interactive websites than many older tools like jQuery.
> Building another dumb website with the new framework, yay.
He's just astonished how excited they are although what they build is "another dumb website" in his opinion. Many developers love the "how?" and don't care about the "why?". I don't judge it, but it can explain some of the excitement. Angular, React and Vue are great for development. But Ember, Backbone and others were also good back in their times. What he essentially says is "We use specific technology so they get excited although the problems we solve are boring af"
Developers have to use the new and shiny to keep themselves marketable and be ready to jump ship at the first opportunity or out of necessity.
My salary has gone up by $45K - $50K in the last 4 years by changing jobs three times. I’m not complaining.
But on the companys’ side, it was completely illogical not to pay me market rates. They still had to pay my replacement market rates and they loss institutional knowledge when I left.
Why is it that HR will approve a req for a new developer at market rates but have strict limits on what they can pay current employees.
On other hand you still have things broken in react not to mention few years with uncertain patent situation.
https://twitter.com/sveltejs/status/999704064937156611 sums this up pretty well.
Programming world is in a constant flux, now you have lit-html that gives you react in 5kb without vdom overhead. And in next few years there will be other interesting projects to look at.
Uh, lit-html seems like just another templating library. Marginally better than something like Soy. Calling it "React" is pretty reductive.
https://github.com/Polymer/lit-element#minimal-example
Because doing something new is much more fun that doing something effective. And most software developers don't really get into this job out of doing things effectively - we're get into it out of the sheer fun of it, altough we would never admit it to our bosses and often even to ourselves.
Trouble is, "fun" for the enthusiast / self-perceived-non-workdrone does align with "productive". =)
My initial reaction is that it should be the opposite: people who are enthusiasts would want to learn something new because they enjoy using and learning technology, whereas a "workdrone" would focus on being productive because they are focused on work.
I'm not saying you're wrong or that there's anything wrong with either mentality, I just don't find your statement self explanatory. Please do elaborate =]
Granted, it's absolutely true they're often used in places they probably shouldn't be and can easily result in JS bloat. When working on the frontend it's tempting to use them for everything because of how comparatively fast/easy development is. Ultimately the tradeoffs should be considered on a case by case basis.
- Next year, the new kid on the block will be the most amazing thing ever created (tm) and everything else will be obsolete (knockoutjs, angular just to name a few)
- If you pack everything now and leave the code untouched, in a couple of months there's a big chance half your tools will need upgrading (angular?) When you upgrade, half the things won't work and you will spend a ton of time fixing things that were perfectly working before, just to get the thing compiling again
-If, down the line, someone else has to take the work where we left, not having the exact development machines set up, they're going to have it much harder to make simple fixes.For example, in six months of development, a simple datepicker javascript library changed the way where it declares the locale config, and we needed to change every declaration of that function to make it work everywhere again. When a framework deprecates a function that you use while you're in development, you fix it. If someone has to pick the project two years later, they are f*
With "old" stuff this doesn't happen. Sure, if you pick php 3 code and try to run it in PHP 7 you'll find problems, but nothing like with webpack/yarn/whatever is doing the packing work this month
Nowadays is all Vue and react, a couple years ago it was Ruby, next year will be something else. In the end, whatever works for you will be good enough, but if you have to think in advance and try to get some future proof code, half the things that are 'what needs to be used right now', will be unmanageable legacy code a year or two down the line (Just my two cents)
It's unglamorous, monotonous and boring, but it's wildly important to the continued success of the project. Who can blame engineers for not being interested in trudging through the boring middle?
Interestingly sometimes efforts to keep engineers from leaving, like splitting an application into 100's of micro services, can actually undermine the entire business. Kind of fascinating if you think about it.
A person in charge may be so afraid to lose an employee they fail to see that the very efforts to keep them may derail the success of the business.
Many large companies have millions of LOC behind their microservices. Your average startup probably doesn't.
A few services > monolith
monolith > 100's of services.
The big trick with any technology is to apply it properly rather than dogmatically and if you are breaking up your monolith into a 100's(!) of microservices you are clearly not in control of your domain. That's a spaghetti of processes and connections between them rather than a spaghetti of code. Just as bad, just in a different way.
Engineer 1: "We'll have one repo per downstream service."
Engineer 2: "But we have hundreds of those, so now we have to manage hundreds of github repos???"
Anyone sane: "That doesn't sound right, we should rethink this..."
A couple of hundred VMs is nothing in a scenario like that. Good luck trying to debug anything.
I though this was an honest and interesting look at a decision which in retrospect was a bad idea. Hopefully it'll stop some other people making similar mistakes (too many repos, too many services, fast changing libraries shared between many services, etc...).
It'd be better if it wasn't framed as having found that the one true way is the monolith, but there are some lessons here for most devs.
Actually worse in many ways:
- Harder to test
- Harder to debug
- Harder to deploy
- Harder to monitor
- Harder to reason about, refactor, change/add functionality
- You've (basically) turned a lot of the operations your services need to perform into RPCs, probably killing performance on top of everything else, leading to
- More complex/demanding (or just MOAR) infrastructure requirements
- Higher dev, maintenance and infrastructure costs
- Slower delivery of value to the business and customers
- Potentially crippling opportunity costs
It surprises me how often people don't see this coming. Seriously: keep your systems as simple as you possibly can. Unless you're Netflix, dozens or hundreds of microservices probably isn't as simple as you possibly can.
> if a bug is introduced in one destination that causes the service to crash, the service will crash for all destinations.
This sounds dangerous. If I was a destination provider, and returned some garbage, could that take system down?
Also, reading up on centrifuge, instead of breaking up by source, destination; would it be more spatially conservant to create queues by response type, which are finite. The entire time I was wondering why bad requests are put back into the same queue they came from. Shouldn't they be isolated? You could then treat those isolated requests as one unit, so in a destination outage, your not backlogged on your actual compute resources for good requests.
Nonetheless, seems like just another day in an engineers workday :)
The biggest challenge is making the shared library forward and backward compatible with itself for at least a few releases in either direction, because not everyone will redeploy at the exact same moment.
If you can't solve that problem everything gets painful. Doing that right was the second hardest part of that job (meetings were the hardest). The job title (securing the data interchange) came in third place.
- a strong API between components
- network calls
The former is a very good idea that should be implemented widely in most code bases, especially as they mature, using techniques like modules and interfaces.
The latter is incredibly powerful in some cases but comes at a huge cost in system complexity, performance and comprehensibility. It should be used sparingly.
A shared library works up until a certain point. If every service uses a shared library then you are already getting to a world of a monorepo. A monorepo for different services should work fine if the overall architecture is feasible.
This strikes me as the core of their problem, and every step taken was a way to bandaid this limitation. Would the cost of moving to faster-scaling infrastructure been so high as rearchitecting the entire system?
> When we wanted to deploy a change, we had to spend time fixing the broken test even if the changes had nothing to do with the initial change.
This seems like a separate and even larger problem. Changes are breaking tests for unrelated code areas? Is the code too tightly coupled? Sounds like it. The unit tests are doing exactly what they're designed to do. Hard to feel sympathy for the person who's breaking them and then trying to figure out a way to sidestep them rather than fix the underlying issues.
If you get your DevOps ducks in order, a lot of the issues you have with standing up and maintaining individual microservices should be manageable. I certainly understand the pain of dependency management, but that's also a part of good architecture.
I'm willing to hear more of these stories though. We can always learn about edge cases or even new paradigms that come out of current thinking.
I've endured a lot of suffering at the hands of the microservices fan club. It's good to see reason finally prevail over rhetoric.
It would have been nice if people had written articles like this 2 years ago but unfortunately, people with such good reasoning abilities would probably not have been able to find work back then.
Software development rhetoric is like religion. If you're not on board you will be burned at the stake.
So many times during technical discussions, I had to keep my mouth shut in the name of self-preservation.
This sounds like an issue with being able to articulate why something is or isn't going to net the expected benefits or being able to foresee unexpected risks. Keeping silent is better than throwing out silly hyperbole risks, but not bring up real risks because "they don't want to hear it" is completely bogus. Any solid engineer will bite at another potential risk to ensure they don't find themselves engineered into a corner 65% through a project. Your comment also makes it out like the notion the article is making, monolith over microserves, is gospel for every situation; that in no condition would it ever make sense to use microserves and that only naive zealots would espouse the wisdom (dogma) to use them. You can use any piece of technology poorly, that doesn't mean the core concept is flawed, just that your problem space is different than what that software is trying to solve. Consider using HDFS as a primary data store in place of MySQL where it doesn't make sense and you might cry the wisdom of wishing someone had told you HDFS is terrible and to just use the tried and true MySQL of olden days.
The issue is not articulation of ideas; the issue is that when all the books, all the articles and all people believe that something is true, there is no amount of articulation which will be able to convince them otherwise.
You have to wait for the hype to go away before even considering bringing up the argument.
This didn't challenge any "common wisdom". Common wisdom didn't tell them to do this. Cargo cult engineering is not common wisdom.