Nice article! Although I think you are overdramatizing microservices complexity a little.
- Kubernetes is rather a harder way to build microservices.
- DB is not an obligatory part of microservices.
- Kafka isn't as well. It's a specific solution for specific cases when you need part of your system to be based on an events stream.
- Jenkins is not necessary, you can still deploy stuff with a local bash script and you need containerization whether you are on Microservices or Monolyth architecture
- Kibana, Prometheus, Zipkin are not required. But I think you need both logs aggregation and monitoring even if you have just a Monolith with horizontal scalability.
Also, all this is assuming you are not using out of the box Cloud solutions.
> - DB is not an obligatory part of microservices.
If the microservices don't have their own store, but are all mucking around in a shared data store, everything will be much harder. I wouldn't even call that a microservice, it's a distributed something. It can work, sure.
DB isn't needed. Our microservices pipeline either uses MQ, EFS or S3 for the destination for another pipeline to pick up. Unless you count those 3 as DBs ;)
Yeah I would say those are key value document based DB. Just silicon valley hipster coming up with cool names to call something different so it is a bit easier for developer to use. Anything does permeant storage are DB.
I'd see a "distributed something" that takes an input, processes it in multiple ways, possibly passing it around to some APIs or queuing it somewhere without ever needing to store it in its own dedicated area to be a good idea.
That could probably the best instance of something that can be built outside of the monolith and can be manipulated separately.
Back when I was studying CS in the early 90s, it wasn't obvious at all that I am going to work with a DB anytime in my career. I loved the subject, I passed with A*. But I thought I am not going to see it later, because I didn't plan to work for a bank or some large enterprise.
Then, in about two years, everything changed. Suddenly, every new web project (and web was also novel) included a MySQL DB. That's when the idea about the three tier architecture was born. And since then, a few generations of engineers have been raised that can't think of a computer system without a central DB.
I'm telling this because in microservices I see the opportunity to rethink that concept. I've built and run some microservices based systems and the biggest benefit wasn't technical, but organizational. Once, the system was split into small services, each with its own permanent storage (when needed) of any kind, that freed the teams to develop and publish code on their own. As long as they respected communication interfaces between teams, everything worked.
Of course, you have to drop, or at least weaken, some of ACID requirements. Sometimes, that means modifying a business rule. For example, you can rely on eventual consistency instead of hard consistency, or replenishing the data from external sources instead of durability.
Otherwise, I agree with the author that if you are starting alone or in a small team, it's best to start with a monolith. With time, as the team gets bigger and the system becomes more complex, your initial monolith will become just another microservice.
>"Jenkins is not necessary, you can still deploy stuff with a local bash script and you need containerization whether you are on Microservices or Monolyth architecture"
This is what I do. Single bash script when ran on bare OS can install and configure all dependencies, create database from backup, build and start said monolith. All steps are optional and depend on command line parameters.
Since I deploy on dedicated servers I have no real need for containers. So my maintenance tasks are - ssh to dedicated server and run that script when needed. Every once in a while run the same thing on fresh local VM to make sure everything installs, configures, builds and works from the scratch.
I originally had my search engine running on a kubernetes-style setup off mk8s.
The code is a microservice-esque architecture. Some of the services are a bit chonky, but overall it's roughly along those lines, besides the search engine I've got a lot of random small services doing a lot of things for personal use, scraping weather forecasts and aggregating podcasts and running a reddit frontend I built.
I'd gone for kubernetes mostly because I wanted to dick around with the technology. I'm exposed to it at work and couldn't get along with it, so I figured we may get on better terms if I got to set it up myself. Turns out, no, I still don't get along with it.
Long story short, it's such a resource hog I ended up getting rid of it. Now I run everything on bare metal debian, no containers no nothing. Systemd for service management, logrotate+grep instead of kibana, I do run prometheus but I've gotten rid of grafana which was just eating resources and not doing anything useful. Git hooks instead of jenkins.
I think I got something like 30 Gb of additional free RAM doing this. Not that any of these things use a lot of resources, but all of them combined do. Everything works a lot more reliably. No more mysterious container-restarts, nothing ever stuck in weird docker sync limbo, no waiting 2 minutes for an idle kubernetes to decide to create a container. It's great. It's 100 times easier to figure out what goes wrong, when things go wrong.
I do think monoliths are underrated in a lot of cases, but sometimes it's nice to be able to restart parts of your application. A search engine is a great example of this. If I restart the index, it takes some 5 minutes to boot up because it needs to chew through hundreds of gigabytes of data to do so. But the way it's built, I can for example just restart the query parser, that takes just a few seconds. If my entire application was like the query parser, it would probably make much more sense as a monolith.
This is why PHP wound up dominating the world. You get all the benefits of a monolith, and if designed well, you get a highly modular system that can be deployed without restarting every single server.
the main issue with micro-services these days is that people write small applications and think they have a micro service. whereas in reality, they have SOA. a true micro service is a mere worker to process a queue, like send emails. create thumbnails for uploaded images. or delete obsolete data from database. so the truth is in the middle - have a monolith and if one machine can no longer handle your workers' load, move them out into separate MICRO services.
I can't understand why a blog has cookies and has facebook hooks at this point in time but then teaching architecture undermining whatever the main argument was at the moment of landing.
There is a lot of talk about monoliths vs microservices lately.. I just want to throw into the ring that you can do both at the same time. easily. And noone is going to kill you for it either.
maybe we are getting caught up in sematics because its christmas, but "monorepo/monolith/microservices/etc" is -just- the way you organize your code.
Developing a montolith for years but now you have written a 15 line golang http api that converts pdfs to stardust and put it into on a dedicted server in your office? welp thats a microservice.
Did you write a 150 repo application that can not be deployed seperatly anyway? welp thats a monolith.
You can also build a microservice ecosystem without kubernetes on your local network. We have done it for years with virtual machines. Software defined networking just makes things more elegant.
So dont stop using microservices because its "hard" or start writing monoliths because its "easy", because none of that is true in the long run.
What is true is that you have a group of people trying to code for a common goal. The way you reach that goal together defines how you organize your code.
>monorepo/monolith/microservices/etc" is -just- the way you organize your code.
It’s also about how you deploy your code. If you have 1000 micro services do you have 1000 deployment pipelines? If so how do you manage those pipelines? If not, you sacrifice independent deployment of each micro service.
I don't even think the deployment pipelines are the problem. If you have 1000 microservices you have 2^1000 possible states of your application based on any of those services being up or down (reality is much more complex). It is genuinely hard to keep that number of services up and running so you then need to be extraordinarily diligent about how you handle failures.
yes and no. i totally get what you are saying but this problem has been solved lately in my opinion.
Also deployment is part of code organistaion no? i like to point out the fact that i explicitly mentioned you do not need to decide, you can use both at the same time but i would like to try to answer your point anyway:
Maybe I am just spoiled with go and github, but those deployment pipelines can be directly integrated into your repository. The same way I can counter argue that your build ci and deployment phase will take significantly more time too and if you change a variable from uppercase to lowercase you will wait for minutes too.
I come from a c# background a long time ago and this has been true with it for eons:
https://xkcd.com/303/
Another thing that I have noticed is that its easily scriptable too. What I end up doing is to provide a makefile into my repository too. This way i can define the build process amongst other things eaisly.
In the end: We have a predefined spec that will create the desired state on a host of our chosing.
Ansible really does not care if you deploy 1 or 1000 "services" for instance!
Tools Like ArgoCD will also deploy stuff when you commit to your master.
There is tooling for everyone available, but what happnes quite often is that the people in charge of the company expected the developers to be all knowing entities that need to define everything end to end. The Kubernetes Space is vast because we are still defining and communication its future.
But recently? I am trying to think of something that would not be completly automateable in 2021
If you have that many services you could define a set of generic pipeline versions that you maintain. Then you can tag each service with which type of pipeline to use. The rest is solved with automation.
Maybe it is just me, but I always understood that properly designed microservices have their own specific datastore, which is not shared with other microservices even if these all collaborate to the same process.
If this is actually (still) true, that means that "the way you organize your code" is a bit simplistic. Your example of an "http api that converts pdfs to ..." is surely a valid example of a microservice, but most business products have to handle much more "state" than those, and this will create further complications which go far beyond "how to organize your code" (and make monoliths more appealing).
Well, it's all about the data responsibility: who is the owner of the data, how others can access the data. Once you have defined these, you see that you can "share the access" with other microservices (for example read only mode on a view), as long as the ownership and the access rules are preserved.
Yes, a view would be exactly how I would address the problem I described in my other answer above ("What happens if I need to change zip code representation in a data source that is read by multiple microservices?").
But this also means that we are now back into "YesSQL" territory, and specifically that we have to use a RDBMS which allows us to create Views.
Goodbye NoSQL, goodbye Key+Valus datastore. (Or maybe you will just create an extra "newZipCode" and mantain it in parallel with "ZipCode" allowing every other consumer to adapt at their leisure...?).
So it is another step back to "more traditional ways" to design a system... or a recipe for a disaster as soon as you start dealing with significant quantities of "state".
IMHO Microservices done well should actually cut a whole vertical through your applications feature space. So not only should it be responsible completely for its own storage of data but it should be responsible for how that data is shown on the front end (or as close to that as you can realistically achieve). A microservice should ideally be reviews or left navigation not customer authentication or order processing.
Well I dont believe that i need a seperate datastore, but yes i need to communicate my state much more.
The PDF Example for instance:
I have to provide all assets that will be converted to startdust or at least links for them
I have to define where the finished stardust would be send to (e.g http post)
but its function is nonetheless independent. the epepheral disk on your container is a datastore too. If you need what is stored actually for longer than the request.... that is another story.
Single writer multiple readers, ideally with permissions to enforce, is a useful hybrid. Enforce going through the front door if you want side-effects. Reporting and ad-hoc data enrichment can be painful to materialize otherwise.
When you have multiple bits of code responsible for writing the same columns, maintaining global invariants becomes much harder.
I can still see rationale for exceptions to the rule, e.g. status fields for rows which logically represent some ongoing process or job, where a UI controller may set something to "processing" and the background job sets it to "processed" when it's complete. But there are ways to close the loop here. For example, implement the background job by having the job processing system invoke the same system which owns the field and sets it in response to the UI, like this:
UI start ---> Back End ---> status := processing
Back End ---> add job to job queue
Job Queue ---> Back End does work
Back End ---> status := processed
Thank you for this as this is exactly how its implemented in a recent project of mine.
I didn't want to put unneeded information in the first post but here i have some space ;)
What I ended up doing was:
- two sqs queues, one for processing one for failures after 3 tries
- the go backend reads from its queue and the message contains a json object with all the information needed for pdf generation
- the backend posts the file back to the "internal api" of the application, which saves it in s3 and also sets processing to true which means the user can go and download it.
This results in:
- my pdf generated the way i want it (sorry i dont actually make stardust)
- the queues are only reachable through the assosiated iam roles
- if a job failes it will be added to the failed queue, which sends me an email
- if i recieve an email by the failed queue, 90% of the time i have found another edge case
- since the message already contains everything that i need to build the pdf, my edge case has just defined itself
- for the postback i supply an api token that is only valid for that one file
All this means my client has total flexibility in deployment.
Caveat: I am really not qualified to discuss the nuances (because I have never used microservices so the little I know is based on reading a bit on those here and on other online forums).
"Single writer multiple readers", yes, this is what I would probably use, but yet again, wasn't the "promise" of Microservices being able to work in total isolation?
If I have one table (e.g. "Customer") which is written by one specific microservice and read by a dozen or more... what happens when I decide that I have to change the schema because the current representation of "Zip Code" is not adequate anymore because, I dunno, we started dealing with UK customers now?
Lo and behold, I have to change code in 13 Microservices - the one actually writing to it, and the 12 more that only need to get the data to show or print or convert to JSON or whatever... ¯\_(ツ)_/¯
actually you would only have to change the one reader and the one writer.
The services themselves do not differ, but are fanned out. E.G. so 100 workers watch your pdf queue to generate pdfs.
If your zip code is really used by many services, the question is how you want to communicate those objects and there are really a lot of choices out there. GRPC / Soap / Shared model package / and the list goes one.
On the other hand i have obsverved that people push full blown models around just to end up reading one field of it in their monolith. I believe the PHP frameworks of the past where the biggest offenders like that.
When i was working with sugarcrm for instance, it was not uncommon to pass huuuuge objects between functions because you needed 1 field. It's PHP so by default, function arguments are passed by value, so you can already see where the fun begins.
Database schemas are generally isomorphic with RPC data structures - usually a subset one way or the other. If you had a service which handed out Zip Codes and the structure of Zip Codes changed (maybe you used numbers and now you realize you need to use strings, which I infer from your UK example) you'd have an isomorphic problem: updating 12 consumers and 1 producer for the new type in the RPC message.
Having an RPC to migrate means you can put code on the read path to support backward compatibility while migrating. But you can do the same thing with the database - support both fields until all readers have ported to the new field - and since big database migrations are expensive, in practice large systems do migrations that way anyhow. Introduce the new field, start writing to it, back-fill it, port readers across individually (there may be many read locations even within a single service), drop the old field.
If a system evolved from a monolith, one way to partition services is to provide views to limit set of visible columns. That's another point at which you can put shims for compatibility while porting schemas.
First you publish the next version of your service with the new data, then the disparate teams in charge of 12 clients (not you) update the rest of the application, then the old version is retired and every team has coordinated properly. Microservices allow lean coordination, basically just asking every team when they think they'll be ready without anyone messing with someone else's code.
My point is precisely that: if you have to handle large quantity of state (e.g.: travel agency handling diverse item bookings to sell as a complete holiday packages - note that this includes having conflicts on inventory, like "cruise cabin categories" or "hotel rooms") microservices add latency by "replacing function calls with RPC", and gain you... an unspecified advantage in terms of... deployment? The possibility to have hundreds of developers working on the system in parallel?
I have always worked on medium-size monoliths during most of my career, and "ah, if we had 137 developers all working on this everything would be magically solved, but alas, we have a monolith" was a sentence I uttered (or heard) exactly 0 times so far.
Clarify the microservice architecture concept with "how you are going to deploy your system", as per your example, is exactly what I'm trying to explain to my teams since the microservice architecture inception. There are too many concepts conflating into the "microservice" term: code architecture(separation of concerns), source code organization, deployment, etc. This is very confusing, which is the reason why it's now common to say that microservices are "hard".
> Developing a montolith for years but now you have written a 15 line golang http api that converts pdfs to stardust and put it into on a dedicted server in your office? welp thats a microservice.
But the 15 lines of Golang are not just 15 lines of Golang in production. You need:
- auth? Who can talk to your service? Perhaps ip whitelisting?
- monitoring? How do you know if you service is up and running? If it's down, you need alerts as well. What if there is a memory problem (because code is not optimal)?
- how do you deploy the service? Plain ansible or perhaps k8s? Just scp? Depending on your solution, how do you implement rollbacks?
- what about security regarding outdated packages the Go app is using? You need to monitor it as well.
And so on. The moment you need to store data that somehow needs to be in sycn with the monolith's data, everything gets more complicated.
The task of the Microservice is to convert the pdf to stardust and to return it to its sender. so no auth.
Furthermore its most likely only reachable through the local network, or at least should be if you want some stranger not to be able to also make stardust from pdfs.
Monitoring: are you trying to say that its a lot esaier to pick up one logfile thant lets say 15? because they should be aggregated somewhere anyway no?
Deployment: Depending on anything you listed how do i do anything? Of course if have to define it but if you want a fancy example: k8s argocd canary deployments done. I literally set it up once.
security? Really?
Please dont get this wrong but this feels to me like whataboutism but well here i go:
i implement security just the same way as i would in the monorepo. The thing/person/entity just has to look into more repositories ;)
It comes down do one sentence i think:
State is not shared, state is communicated.
For monitoring I’d also say the alerting side of things can be done via the older monolith. That is, catch exceptions and log/re-raise them as “PdfServiceException”.
A microservice runs as some (somewhat) privileged user, you may want some auth. Can everyone internally create sales tickets? Or can everyone just query them? If a team provides a library to run, and you run it, you still only run as whatever user you have access to.
Monitoring: it's easier to look at a stack trace, including some other team's external library, than a HTTP error code 500.
Deployment is certainly easier when you're just shipping code and a build. You don't have to faff around with the previous instance running, maybe having some active connections/transactions/whatever, needing to launch a new one. Maybe it's not hard overall, but less fun.
It's easy to be snarky and dismissive about these things, but i've stood in the offices of a governmental org and have looked at people who are unable to receive their healthcare services queueing up because some similarly neglected system refused to work. One that also had a lot of the basics left out.
Figuring out what was wrong with it was hard, because the logging was inconsistent, all over the place in where it logged and also insufficient.
The deployments and environments were inconsistent, the sysadmins on clients's side manually changed stuff in the .war archives, all the way up to library versions, which was horrendous from a reproducibility perspective.
The project was also severely out of date, not only package wise, but also because the new versions that had been developed actually weren't in prod.
After ripping out the system's guts and replacing it with something that worked, i became way more careful when running into amateur hour projects like that and about managing the risks and eventual breakdown of them. I suggest that you do the same.
Don't put yourself at risk, especially if some sort of a liability about the state of the system could land on you. What are you going to do when the system gets breached and a whole bunch of personal data gets leaked?
> ...then don't spend hundreds of thousands on infrastructure.
I find it curious how we went from doing the basics of software development that would minimize risks and be helpful to almost any project out there to this.
To clarify, i agree with the point that you'll need to prioritize different components based on what matters the most, but i don't think that you can't have a common standard set of tools and practices for all of them. Let me address all of the points with examples.
> Auth: nobody knows IP address of our server anyway, don't bother with that. And for extra security we have secret port number.
Port scanning means that none of your ports are secret.
JWT is trivial to implement in most languages. Even basic auth is better than nothing with HTTPS, you don't always need mTLS or the more complicated solutions, but you need something.
This should take a few days to a week to implement. Edit: probably a day or less if you have an easily reusable library for this.
> Monitoring? Well, we have our clients for that. They'll call us if something happens.
This is not viable if you have SLAs or just enjoy sleeping and not getting paged.
There are free monitoring solutions out there, such as Zabbix, Nagios, Prometheus & Grafana and others. Preconfigured OS templates also mean that you just need the monitoring appliance and an agent on the node you want to monitor in most cases.
This should take close to a week to implement. Edit: probably an hour or less if you already have a server up and just need to add a node.
This is an error prone way to do things, as the experience of Knight Capital showed: https://dougseven.com/2014/04/17/knightmare-a-devops-caution.... In addition, manual configuration changes lead to configuration drift and after a few years you'll have little idea about who changed what and when.
In contrast, setting up Ansible and versioning your config, as well as using containers for the actual software releases alongside fully automated CI cycles addresses all of those problems. In regards to rollbacks, if you have automated DB migrations, you might have to spend some time writing reverse migrations for all of the DDL changes.
This should take between one to two weeks to implement. Edit: probably a day or less once per project with small fixes here and there.
> Outdated packages? We call those stable packages.
Log4j might be stable, but it also leads to RCEs. This is not a good argument, at least for as long as the software packages that we use are beyond our ability to control or comprehend.
This should be a regular automated process, that alerts you about outdated and/or insecure packages, at least use npm audit or something, or proactive scanning like OpenVAS. This should take close to a week to implement.
All of the mentioned software can easily run on a single node with 2 CPU cores and about 8 GB of RAM. I know this, because i did all of the above in a project of mine (mentioned technologies might have been changed, though). Of course, doing that over N projects will probably increase the total time, especially if they have been badly written.
In my eyes that's a worthwhile investment, since when you finally have that across all of your apps, you can develop with better certainty that you won't have to stay late and deliver 2 or 3 hotfixes after your release, which goes hand in hand with test coverage and automated testing.
That is hardly hundreds of thousands of dollars/euros/whatever on infrastructure. And if the personnel costs are like that, i'd like to work over there, then.
There are plenty of solutions to authentication. But really, don't implement a user system if it is not needed. There are plenty of other ways to secure on applications, which are way out of scope for this discussion.
The main point is, that one should never spend a "a few days to a week" to implement a feature that at best i useless and at worst is detrimental to the service stood up.
Implement auth, if it is needed, implement monitoring, CI, CD, dependency monitoring, testing, everything, if it is needed.
But don't implement it as dogmatic consequences of doing software development.
And regarding the spend: one week worth of work could be USD 8k. So just the initial implementation of your JWT based authentication system is 4% into the "hundreds of thousands of dollars". Then you need to factor in the extra complexity on maintenance and before you know it we do not talk about hundreds of thousands of dollars but millions...
I feel like "spaghetti garbage now, we'll fix it later" is a big part of why startups fail to execute well. Yeah you saved $8000 by launching your thing in a completely unmaintainable way, but now it's both harder to maintain and more likely to need it. Literally the first time it breaks you will probably lose that cost advantage just because it takes so long to debug.
The point you should have made is that dogmatic approaches usually produce a lot of waste, but the example you gave us exactly why teams end up that way. Otherwise people come up with bullshit hacks like you describe and the entire team pays for it.
> But really, don't implement a user system if it is not needed.
Sure, i'm not necessarily advocating for a full blown RBAC implementation or something like that, merely something so that when your API is accidentally exposed to the rest of the world, it's not used for no good (at least immediately).
> Implement auth, if it is needed, implement monitoring, CI, CD, dependency monitoring, testing, everything, if it is needed.
> But don't implement it as dogmatic consequences of doing software development.
Now this is a bit harder to talk about, since the views here will probably be polarized. I'd argue that if you're developing software for someone else, software that will be paid for (and in many cases even in pro-bono development), most of that is needed, unless you just don't care about the risks that you (or someone else) might have to deal with otherwise.
If there's an API, i want it to at the very least have basicauth in front of it, because of the reasons mentioned above.
If there is a server running somewhere, i want to be alerted when something goes wrong with it, see its current and historic resource usage and get all of the other benefits running a few apt/yum commands and editing a config file would get me, as opposed to discovering that some bottleneck in the system is slowing down everything else because the memory utilization is routinely hitting the limits because someone left a bad JVM GC config in there somewhere.
If something's being built, i want it to be done by a server in a reasonably reproducible and automated manner, with the tasks are described in code that's versioned, so i'm not stuck in some hellscape where i'm told: "Okay, person X built this app around 2017 on their laptop, so you should be able to do that too. What do you mean, some-random-lib.jar is not in the classpath? I don't know, just get this working, okay? Instructions? why would you need those?"
Furthermore, if there is code, i want to be sure that it will work after i change it, rather than introducing a new feature and seeing years of legacy cruft crumble before my eyes, and to take blame for all of it. Manual testing will never be sufficient and integration tests aren't exactly easy in many circumstances, such as when the app doesn't even have an API but just a server side rendered web interface, which would mean that you need something like Selenium for the tests, the technical complexity of which would just make them even more half baked than unit tests would be.
Plus, if i ever stumble upon a codebase that lacks decent comments or even design docs/issue management, i will want to know what i'm looking at and just reading the code will never be enough to understand the context behind everything but if there are at least tests in place, then things will be slightly less miserable.
I'm tired of hating the work that i have to do because of the neglect of others, so i want to do better. If not for those who will come after me, then at least for myself in a year or so.
Do i do all of that for every single personal project of mine? Not necessarily, i cherrypick whatever i feel is appropriate (e.g. server monitoring and web monitoring for everything, tests for things with "business logic", CI/CD for everything that runs on a server not a local script etc.), but the beauty is that once you have at least the basics going in one of your projects, it's pretty easy to carry them over to others, oftentimes even to different stacks.
Of course, one can also talk about enterprise projects vs startups, not just personal projects, but a lot of it all depends on the environment you're in.
As for the money, i think that's pretty nice that you're paid decently over there! Here in Latvia i got about 1700 euros last month (net). So that's about 425 euros a week, or more like 850 if you take taxes and other expenses for the employer into account. That's a far cry fro...
Offtopic, but if you really are paid 425 euros per week then you are seriously underpaid even for Eastern European standards. There are (relatively rare, but still) jobs that pay this much per day.
Yep, i've heard that before. Currently thinking of finishing the modernization of the projects that i'm doing at my current place of work and then maybe looking at other opportunities.
However, i just had a look at https://www.algas.lv/en, one site that aggregates the local salary information, based on a variety of stats. As someone with a Master's degree and ~5 years of experience and whose job description states that i'm supposed to primarily work in Java (even though i do full stack and DevOps now), i input those stats and looked at what information they have so far.
The average net salary monthly figures, at least according to the site, are:
So there's definitely a bit of variance, but it's still in the same ballpark. Of course, there are better companies out there, but the reality for many is that they're not rewarded for their work with all that much money.
USD 8k/w is not the pay a employee will receive, but the cost of the operation of that one employee, and it is ballpark numbers -- I work in Europe also, I am a Danish citizen.
Again, I advocate developing in a timely manner, and not do over engineering (neither under engineering).
> USD 8k/w is not the pay a employee will receive, but the cost of the operation of that one employee
I made the above response with that in mind.
> And regarding the spend: one week worth of work could be USD 8k.
The original claim was that one week's worth could be 8000 USD, or let's say roughly 7094 EUR. That comes out to 28376 EUR per month.
Last month i made around 1700 EUR, so it's possible to calculate approximately how much my work cost to my employer. Let's do that with a calculator here: https://kalkulatori.lv/en/algas-kalkulators
After inputting the data that's relevant to me, i got the following:
Gross salary 2654.51 EUR
Social tax 278.72 EUR
Personal income tax from income till 1667 EUR 277.66 EUR
Personal income tax (987.51 EUR), from part ... 227.13 EUR
Social tax, employer's part 626.20 EUR
Business risk fee 0.36 EUR
Total employer's expenses 3281.07 EUR
It should be apparent that 28376 EUR is a far cry from 3281 EUR, which is how much my work cost to my employer.
Thus, per week, 7094 EUR is also a far cry from 820 EUR, which is how much my work cost to my employer.
Also, 820 is actually pretty close to my initial guess of 850 EUR.
Of course, it's possible to argue that either i'm underpaid individually, or that many of my countrymen in Latvia are underpaid in general (on which i elaborated in an adjacent comment https://news.ycombinator.com/item?id=29595158), but then the question becomes... so what?
Does that mean that if you're in a well paid country like US, then you cannot afford proper development practices due to all of the payroll expenses that would cause? While that may well be, to me that sounds weird and plain backwards - if that were really true, then US would outsource even more to countries like mine and these outsourced systems would work amazingly well, since you can supposedly afford a team of developers here for what would buy you a single developer over there. And yet, most systems are still insecure, buggy and slow.
Maybe someone else is pocketing a lot of the money they receive in these countries, and is simply charging high consulting rates? The prevalence of WITCH companies here is telling, but that's a discussion for another time.
I really can’t tell how serious you are. I’m too aware that what you describe often is exactly how it works in practice. It’s just that very few admit it in public. :)
1. auth? probably an internal service, so don't expose it to the outside network.
2. monitoring? if the service is being used anywhere at all, the client will throw some sort of exception if its unreachable.
memory problem? it should take <1 day to ensure the code for such a small service does not leak memory. if it does have memory leaks anyways, just basic cpu/mem usage monitoring on your hosts will expose it. then ssh in, run `top` voila now you know which service is responsible.
3. deployment? if its a go service, literally a bash script to scp over the binary and an upstart daemon to monitor/restart the binary.
rollback? ok, checkout previous version on git, recompile, redeploy. maybe the whole process is wrapped in a bash script or assisted by a CI/CD build job.
4. security? well ok, PDFs can be vulnerable to parser attacks. so lock down the permissions and network rules on the service.
Overall this setup would work perfectly fine in a small/medium company and take 5-10x less time than doing everything the FAANG way. i don't think we should jump to calling these best practices without understanding the context in which the service lives.
I agree more or less with 1 and 4 mostly. But for monitoring either you would have to monitor the service calling this microservice or need to have a way to detect error.
> if it does have memory leaks anyways, just basic cpu/mem usage monitoring on your hosts
Who keeps on monitoring like this? How frequently would you do it? In a startup there are somewhere in the range of 5 microservice of that scale per programmer and daily monitoring of each service by doing top is not feasible.
> 3. deployment? if its a go service, literally a bash script to scp over the binary and an upstart daemon to monitor/restart the binary.
Your solution literally is more complex than simple jenkins or ansible script for build then kubectl rollout restart yet is lot more fragile. Anyways the point stands that you need to have a way for deployment
My larger point is basically just against dogma and “best practices”. Every decision has tradeoffs and is highly dependent on the larger organizational context.
For example, kubectl rollout assumes that your service is already packaged as a container, you are already running a k8s cluster and the team knows how to use it. In that context, maybe your method is a lot better. But in another context where k8s is not adopted and the ops team is skilled at linux admin but not at k8s, my way might be better. There’s no one true way and there never will be. Technical decisions cannot be made in a vacuum.
> Overall this setup would work perfectly fine in a small/medium company and take 5-10x less time than doing everything the FAANG way.
The point was never comparing it to the FAANG way. The point is: it's easier (at the beginning) to maintain ONE monolith (and all the production stuff related to it) than N microservices.
Yup. This is why I think that microservices require a stronger operational plattform, but then it enables new and more effective ways of developing new tunctionality.
Our internal software plattform is getting to a point so it can answer most of these things - auth via the central OIDC providers, basic monitoring via annotations of the job's services, deployments via the orchestration and some infrastructure around it, including optional checks and automated rollbacks and automated vulnerability scanning on build-servers and for the running systems. It wouldn't be 15 lines of go, more like 15 lines, plus about 100-200 lines of terraform and/or yaml to get everything configured, and a ticket do register the service in the platform. It's pretty nice and our solution consultants like it very much.
The thing is - this took a team about a year to build and it'll take another half a year to get everything we currently want to do right. And it takes a non-trivial time to maintain and support all of this. This kind of infrastructure only makes business sense, because we have enough developers and consultants moving a lot faster with this.
Back when we were a lot smaller, it made a lot more sense to just push a single java monolith on VMs with chef or ansible, because that was a lot easier and quicker to get working correctly (for one thing).
Many of these points you're mentioning is exactly why k8s was developed. Yes it makes deploying simple applications unnecessary hard, but it make deploying more complicated applications WAY more manageable.
So in the k8s world:
- auth: service meshes, network policies, ...
- monitoring: tons of tooling there to streamline that
- deploy: this at scale is trickier than you'd think, many seem to assume k8s on it's own here is the magic dust they need. But GitOps with ArgoCD + helm has worked pretty well at scale in my experience.
- Security is a CI problem, and you have that with every single language, not just Go. See Log4j.
Kubernetes is my bread & butter, but I do realise this has way too much overhead for small applications. However, once you reach a certain scale, it solves many of the really really hard problems by streamlining how you look at applications from an infrastructure and deployment side of things. But yes - you need dedicated people who understand k8s and know what the hell they're doing - and that's in my experience a challenge on it's own.
Let's also dispel a myth that k8s is only suitable for microservices. I have clients that are running completely separate monolith applications on k8s, but enough of those that managing them 'the old way' became very challenging, and moving these to k8s in the end simplified thing. But getting there was a very painful process.
Well, it quickly can become a mess because of people having different ideas about what microservice is, and also decrying things as "for microservices only" when for example I just want to offload auth and monitoring to a specialized service.
It's also a common trope when I'm dealing with k8s decriers - yes, you might have one application that you can easily deploy, but suddenly there are 15 other medium-weight applications that solve different important problems and you want them all ;)
P.S. Recently a common thing in my own architectures is separate keycloak deployment that all services either know how to use, or have it handled at separate request router (service mesh or ingress or loadbalancer)
> "monorepo/monolith/microservices/etc" is -just- the way you organize your code
I don't think this is true.
I think — at least as far as I've observed — microservices in practice means replacing various functions calls with slow and error-prone network requests.
Exactly, it doesn't have to be one or the other. So far, i've been using a monolith for core functionality and microservices for independent processing tasks.
That way, i can attach as much functionality as i want without bloating the main app and processing scales with demand.
> There is a lot of talk about monoliths vs microservices lately.
Actually it's been going on for years, and it's always the same argument. People think they're thought leaders for saying "Start with monoliths and only move to microservices if you absolutely need to!"
It's a pretty obvious conclusion to anyone who has worked in both environments, or have had to migrate from one to the other, so it's not particularly insightful. And yet here were are, 5+ years later saying the same thing over and over again.
> It's a pretty obvious conclusion to anyone who has worked in both environments, or have had to migrate from one to the other, so it's not particularly insightful.
It should be obvious, but apparently it's not. So many architecture astronauts drinking the kool-aid and making a mess. Premature microservices can easily kill a product.
In what way do Microservices even help? It seems to me you still have to synchronize to be sure that the Microservice from team B does exactly the things that are specified in the new version?
Is it not easier to have a pull request that says "this will do thing x", you merge it into your monolith, and then you can see in the git log that this version will indeed to x?
How do Microservice organizations even manage that? Is it the reason that Atlassian has a billion dollar evaluation, because people need it to keep track?
The only thing you need to synchronise is API, no?
Which is where version numbers comes in.
To me, if youre working with a lot of modules/micro services, lots of modules should be able to sit on old versions and develop independently (which is the crucial part for 100 developer scenario)
Well, not git, but modularity was invented for that.
You can have a modular monolith that works just as well with 100 people as something service-oriented would. The difference lies in the level of discipline needed. It's much easier to "just go in and make that field public because it makes my implementation easier" when you have a modular monolith. With microservices, you are more explicitly changing an external API by doing that.
Yes, it's the same thing. But somehow, psychologically, people feel worse about changing a networked API than making an identifier public instead of private.
Edit: I forgot, there's one more thing: with service orientation, you can deploy in finer grains. You shouldn't have heavily stateful services, but if you do (and you always do!), it can be cumbersome to redeploy them. At that point, it's nice to be able to deploy only the parts that changed, and avoid touching the stateful stuff.
This is the right take on this. All tech people here that rave on that microservices really make their life easier even though they are working in a small team for an entire product are looking through rose colored glasses while chucking down the koolaid and also not the intended audience. The tech complexity is hardly ever worth it unless you are a large Corp.
Org complexity is a valid point. Sure you can solve it using microservices. But in this particular case (solving org complexity) such "microservice" is akin to a library with some RPC. You might as well have each team developing their "microservice" as a shared / statically linkable lib. Same thing in this context.
I feel like microservices are a solution for magpie developers. It's hard to keep a hundred engineers excited about working in an aging Java stack when there's all these shiny new tools out there. But maybe that's just my perspective, coming from a consultancy firm whose devs wanted to stay on the cutting edge.
I think this leads to silos.
MicroServices written in different lang's mean Java Dev X can't maintain Python service Y.. Not in an efficient way.
What's worse, Java Dev can't move to Team Y without upskilling not only on the service, but also the lang, so they get pidgin holed. She also can't move because she's the only Java dev left.
Except that you do not need microservices to solve organisational problems. You need, as has always been done, to have well-defined modules with well-defined interfaces.
If there’s no organizational barrier (e.g. microservices architecture, separate repos with strict permissions) that will prevent devs from leaking abstractions across technical boundaries, those well-defined modules and interfaces will devolve into a big ball of mud.
I say this with the assumption that the team is large and members regularly come and go.
Except when the team division in the company does not map to any natural service API boundaries, yet it is insisted that each "team" own their services.
Then microservices increase organizational complexity too.
Suddenly product owners and middle management are chief architects without even knowing it.
Microservices is an organizational optimization. It can allow one team to manage and deploy their own subsystem with minimal coordination with other teams. This is a useful thing, but be aware what it's useful for.
If each of your developers manages a microservice, that probably reflects that you do no actual teamwork.
Monoliths and microservices are two bad ways to develop software. Monoliths are rife with hidden dependencies and microservices tend to collapse from even simple faults
Need to rightsize the modules. More than one, but just a few with boundaries chosen around recovering from faults
Maybe I'm too old, but I don't even want to have to worry about all that. I think in terms of functions and I don't care if they are being called remotely or local.
That was the promise back in the day of J2EE, and it seems to me Microservices are just a rehash of that same promise.
Which never really worked out with J2EE - it was mostly invented to sell big servers and expensive consultants, which is how Sun made money?
These days I sometimes don't even bother to make extra interface classes for everything. If I need it, I can still insert that, but no need to do it up front.
And "DevOps" is just a ploy to make developers do more work, that was formerly done by admins.
Writing every Microservice in a new language also seems like a huge headache. The only bonus is that you can attract developers by being able to promise them that they can work with some shiny new technology. Maybe that is actually the main reason for doing that?
Otherwise, again, perhaps it is my age, but I prefer to minimize the dependencies. Do I really want to have to learn a whole new programming language just so that I can fix a bug in some Microservice. I personally don't want to.
It was a bit cheeky and off-hand. I am certainly interested in DevOps. If things can be automated, it is usually a good thing.
Still I think having extra admins for server maintenance, or nowadays DevOps, also makes sense in my opinion. In theory I should be interested, but in practice, somehow I can not really get passionate about the details of how to host things. So it would be better to have specialists who really care.
Ultimately there are also a lot of aspects that are different from the programming problems. It's difficult to keep up to date on both fronts.
I am 100% not interested in ops. Deploying is handled by other way more qualified than I.
I never expect a JS expert to build elixir, I don't expect an elixir expert to write bash, and I don't expect a bash expert to know about switches and cabling.
I don't understand where you draw the line..
Should the designer who also crafts the css do ops too?
I think high quality comes from specialists. Sharp knives in the hands of pro's
Just like the preceding question, it depends on context.
In a huge organisation, it can work well to be very specialized.
In, for example, a smaller growing business with an engineering team of < 10, having a proper understanding of the context in which your code will run is a game-changer.
The goal is to sustainably build software, which only can be done if ops and maintenance is regarded a goal, a pillar to build upon.
Like TDD the design of software changes when you have experience with ops/maintenance. Take logging/tracing: what to log or trace is first and foremost a matter of experience with maintenance/ops!
Building in logging/tracing/monitoring from the get-go because you know that you want to know what goes wrong, when it goes wrong etc...
can one regard oneself as expert if you know only a part of the domain- I think not!
And this person is a DevOps person, in every org I've ever worked in.
Maybe "before my time" it was a webmaster or other admin. I think of "dev ops" as an umbrella term subsuming many specific titles and roles from the past, much like "data science".
I suspect you don't mean it like that, but developers have to care just a little about operations. There's the classic stuff about developers who build stuff, because they didn't realize that ops could do the same with a few lines in a web server config, so they waste weeks on trivial stuff. There's also the issue that if you expect databases, queues, disks and so on to just be available, while not thinking about how you use them, then you can get bad performance or cause downtime and crashes in the worst case.
The initial idea of DevOps also seems to have been twisted to having developers do operations, rather than having the two teams work in tandem. If we absolutely must combine the two roles, I'd be more comfortable having operations do development work, but that hardly ideal either.
These days DevOps is used as a bludgeon to fire sysadmins and make devs pretend to manage infrastructure. The idea from the corporate viewpoint is to save money by not having anyone to handle systems stuff, just shove into "The CloudTM" as serverless microservices, then just restart it when it falls over instead of troubleshooting.
Yes, I'm cynical. Having to use javalike stuff to do what should be a simple shell script will do that to you.
I don't know. I totally agree on the premise that developers should be involved in ops as well.
But I'm a developer by heart and my heart aches whenever I see what we devs have wrought in the ops space. (Insert joke about the CNCF technology landscape map.)
There's just so many tech/tools/etc. involved that just reasonably "doing the ops stuff" on the side seems way unrealistic.
Sometimes I feel that all we've accomplished was job security for legions of consultants.
> But I'm a developer by heart and my heart aches whenever I see what we devs have wrought in the ops space.
At the same time, I am conflicted. I don't care for the toss-it-over-the-wall approach that used to be the norm, but I also don't like having dev's have to take more on than they are capable.
In an ideal environment, I would like to see a crashing together of developers and operations people on teams. What I mean is that for a given team you have several developers and one or two ops folks. This way there is less of an us vs them sentiment and teams can be held accountable for their solutions from ideation to running in production. Finally while it's not the sole responsibility of the dev's to manage their DevOps stuff, they will have more knowledge of how it works, and get to put input and ideas to it.
This was the original intent of DevOps - a cultural shift that put sysadmins/ops people into the dev team, thus have a better feedback loop between dev and prod, and tear down the wall between dev and ops. But now it's being used as a way to eliminate ops people and push it all into dev.
DevOps is a sad story both for devs and for ops. It was supposed to treat operations as a SW problem thus take away the toll and draw devs in. In reality for most places it either means that devs also do pipelines and operations or that operations were rebranded and are using "DevOps" tooling to do operations.
To me, DevOps means SDEs can troubleshoot, diagnose, and fix app-related prod issues. SREs don’t have the app-level knowledge for that, so it’s wasteful for them to be involved. SREs should be paving paths for SDEs and responding to infrastructure ops issues.
I also think silos are unhealthy. SDEs need to have some overlapping responsibilities with SREs. Otherwise, you’ll likely have a “throw it over the fence” culture
I guess the idea is fine but the implementation tends to be about using 1 employee in multiple roles and working 60-hour work weeks while paying just one salary. You got DevOps, DevTest, then DevSecOps, etc. It´s insane, the next step is probably being a DevTestSecOps sleeping under the company desk.
> "DevOps" is just a ploy to make developers do more work, that was formerly done by admins.
Developers cost a lot more than admins, so this seems silly.
Moreover “devops” people are supposed to be developers, just ones with a whole bunch of networking and admin skills as well. But few places outside Google have real SREs.
Unless you have a strong technical or organizational reason to use microservices, using microservices is just more work to achieve the same results.
Organizational reason would be multiple people/teams who don't want or can't talk much to each other, so they develop pieces of a larger system as relatively independent projects, with clear API and responsibility boundaries. Frontend/backend style web development is an example of such approach, even though we don't typically call these parts "microservices".
A technical reason I can see is some component of a system actually having to be written in a different stack, or to be run in a separate location for business reasons (separate physical computer, separate VMs or containers don't count). Like a firmware running on an IoT system. Or most of the system uses python, but there's a really good library in java for solving some very specific problem, so let's use it.
If neither of these reasons stands, you don't have a microservice architecture, you have a distributed monolith. You just replaced some function calls with RPC. RPC call which takes a much a longer time than a local one, and can randomly fail. Most of your microservices are written in a single stack, so you refactor common parts into a library, but then different services are stuck to use different versions of this library. You end up with a much slower and a much more fragile system which is harder to work on for no good reason.
How about deployment speed? If I’ve got a microservice collecting events off a queue and writing a csv out to S3 on a schedule, it’s really nice to be able to add a column and deploy in minutes without having to rebuild and deploy a giant monolith. It also allows for fine grained permissions: that service can only read from that specific queue and write to that specific bucket.
People throw around “distributed monolith” like it’s a dirty phrase but I’ve found it actually a very pleasant environment to work in.
That's a fallicy. You've optimized for one use case, but you've made everything else more complicated as a consequence.
Deploying a single monolith is faster than deploying 10 microservices, especially if you find yourself in the model where your microservices share code, you've ended up with a distributed monolith instead of microservices.
> You've optimized for one use case, but you've made everything else more complicated as a consequence.
Yes, but that use case happens to be something that I need to do 5 times a day - make a small (<200 line) code change to one small part of the distributed monolith, and deploy it immediately.
This also means that if something goes wrong, I can rollback those changes immediately without rolling back the work of any of the other 50 engineers in the company.
Very little signoff required, very small blast radius.
Say your little service just changed how it parsed backticks. Now that innocuous change may affect none of the immediately connected microservices but another services three hops away relied on the old behavior of your parser through some complex business rules driven logic. Now go test and later troubleshoot that vs standing up a single monolithic jar on your laptop and seeing the exception stack trace tell you exactly what you broke.
I don't know, is it really harder to deploy 10 services? Isn't it all automated? The organization overhead is lower because you can leave most of the app untouched and only deploy what you need to.
You could screw it up and make it harder for yourself but its not guaranteed either way.
You would only deploy 10 microservices if all 10 changed at once. Why are all 10 changing at once?
> especially if you find yourself in the model where your microservices share code
Good architecture decisions trump the monolith vs microservices argument. I'm not saying cross-service shared code is inherently bad, but it does have a bad smell.
I worked at a place that had a few dozen services and one DB, worked on by dozens of developers. It was amazing. Oh and one repo, that made a huge difference.
> implement change on that branch (in this case adding a column), test
> deploy to prod
I realize the build and deployment process may be more complex than that making it hard... but it doesn't have to be.
I agree that a microservice OR even another system (a collection of services) is a good solution if you need to make quick iterative changes, and you can't do so with your current system.
That workflow is exactly what I do, but it's on a small codebase rather than on a big one.
The benefits of working on a small repo include:
* Faster compilation (only depend on exactly the libraries you need)
* Faster testing (only need to run the tests which are actually relevant)
* Easier to understand for new joiners because there's just less code to sift through
* Faster startup (this is probably java specific - the classloader is slooow)
* No (okay, fewer) rebase wars
* Easy to see what's currently being worked on because the Pull Requests tab isn't 200 long
* Very fast rollbacks without needing to recompile a git revert (just hotswap the container for that particular service). You can't do this in a monolith without risking rolling back someone else's important changes.
> People throw around “distributed monolith” like it’s a dirty phrase but I’ve found it actually a very pleasant environment to work in.
For ones I saw, the answer to the question "how do I run our project on a single laptop?" is "haha, you don't". That makes features which could take hours to implement take weeks. But deployment is 5 minutes instead of… 15? Not too worthy of a trade-off.
Your CSV writer probably has both technical and organizational reasons being an independent unit of development. Or, in other words, something which appeared organically, rather than someone deciding months prior before a project even started that authentication and user profiles should live in two parallel universes.
Often with monoliths, once they go into production service at scale, the deployment can be "3 days, arranged 4 weeks in advance, requiring 9 sign offs including one VP"
I would call that a megalith, and not to be cute but to give a sense of scale which is useful. I think there is a point in which a monolith grows so big that is painfully obvious that size became an obstacle greater than the benefits.
Pain tolerance differs so the label gets applied at different sized monoliths.
You bring up an excellent point. As of now, it is impossible for me to run my company's backend ecosystem on my machine locally. What we do could easily be done by one simple monolith, but our eng. lead is obsessed with overengineered microservice architecture, so nobody can actually develop locally. Everything on the backend is done by a bunch of lambdas behind an API gateway.
I got so burnt out developing in that environment that I asked to become a pure frontend dev so that I wouldn't have to deal with it anymore.
one strong operational reason I have seen recently is resource management.
The monolith where most API endpoints are instant and use constant memory, but some use much more memory and can be slower... is tough.Like if you just give a bunch of memory to each process now you're overprovisioning and if you try to be strict you run into quality of service issues.
If you split out homogenous API endpoints into various groups you now have well behaved processes that are each acting similarly. One process could be very small, another could be much larger (but handle only one kind of request), etc...
of course the problem with standard microservice-y stuff is now you gotta have N different applications be able to speak your stack. The idea of a monolith with feature sets is tempting... but also can negate chunks of microservice advantages.
Ultimately the microservice-y "everything is an API" can work well even as a monolith, and you would then have the flexibility to improve things operationally later.
At the same time, you have to be at a pretty huge scale before resource over-provisioning really hurts the bottom line. You can buy a lot of compute for the price of a single engineer's salary, and it usually takes more than one engineer to support a microservice architecture. Most applications hit problems scaling the database vertically long before they exhaust resources at the application level.
Hmm I get what you’re saying of course but in certain domains (think B2B SaaS) you might be running some compute-intensive stuff enough to where the differential is an issue.
Imagine 95% of your workload can run in 100 megs but 5% requires 1000 megs. You can overprovision of course but in a world of containers if you can isolate the 5% and route it you’re going to have a lot less in terms of operational headaches
I mean, this is kind of how microservices should be done. Start with a MVP monolith then carve off microservices if needed (performance or large team size).
The problem is when the lead dev has been huffing the architecture paint too hard and starts prematurely spinning up microservices because it feels good.
What has worked well for us, something that IMO combines the best out of both worlds:
* break down the problem into sensible components. for a reporting system I'm working on atm. we're using one component per type of source data (postgres, XLSX files, XML), one component for transformations (based on pandas) and one component for the document exporter.
* let those components talk through http requests with each other, including an OpenAPI specification that we use to generate a simple swagger GUI for each endpoint as well as to validate and test result schemas.
* deploy as AWS lambda - let amazon worry about scaling this instead of having our own Kubernetes etc.
* BUT we have a very thin shim in front that locally emulates API gateway and runs all the lambda code in a single process.
- (big) advantage: we can easily debug & test business code changes locally.
only once that is correct we start worrying about deployment, which is all done as IaC and doesn't give us trouble that often.
- (minor) disadvantage: the various dependencies of lambdas can conflict with each other locally, so gotta be careful with library versions etc.
doing so the scaling works quite well, there is no up-front infrastructure cost
and code tends to stay nice and clean because devs CANNOT just import something from another component - common functionality needs to first be refactored into
a commons module that we add in each lambda, which puts a nice point for sanity
checking what goes in there.
It has the disadvantages of lock-in, but this is why I love ECS. 80% of the advantages of K8S for 20% of the complexity. Even hosted K8S requires a lot of config and concepts that are just baked into ECS.
Monolith and Micro-services at different times during the progression of a business can have their places. I have experienced the pains of having to work with both within the same company over 9+ years and here is what I think are forces that can pull you in either direction.
A monolith makes developing initially a lot easier. Over 15 years though, you are bound to have developers of various calibre leave their mark on it. Many don't even understand good modelling and inevitably drive an otherwise healthy monolith with well defined boundaries into a soup of couplings between domain concepts that should not know about each other. In theory though it is totally possible to have micro systems within the same monolith code if you model things that way.
Eventually, your team will flip the table and start thinking how to avoid the problems they are having with the monolith and decide to do down the micro-services way. In most situations developers are likely to face a lack of commitment from the business to spend time/money on a technical problem they do not understand but have to support the development of for a lengthy period of time. Most developers will compromise by building pseudo micro-services without their own databases which send requests to the monolith where the core business logic had to stay.
The benefit of micro-services IMO is driven from being able to separate business domains in a way that keeps each responsibility simple to understand for a new comer and compact enough to not hide a great deal of complexity. It's worth saying this is a very hard thing to achieve and the average developer shop won't have the experience to be able to pull it off.
This is all to say, regardless of Monolith or Micro-services architecture, the key is experience and discipline and without a good amount of both in your team the outcome is unlikely to be a success over a long enough period of time. This is the curse of a business that lives long enough.
In some languages, you can enforce boundaries within a monolith nicely using the build system. The key is to break the build up into a hierarchically structured set of libraries somehow where each library only gets to use those libraries it is allowed to depend on architecturally. Independent bits of business logic would need to go into libraries that cannot "see" each other when they are compiled. Everything would still be part of a single build process and still be linked together into a single big program at the end.
The exact method depends on the language. In C/C++, you'd limit the include paths for each library. In C#, you'd have to compile different assemblies. And so on.
I think you didn't quite get the point of engineers of different levels of quality, talent and opinions working on the same monoliths.
Eventually they tear down any boundary, even those in the build system.
Developer discipline is something that eludes many companies for lack of enough high quality engineers and awareness for the problem in upper management. It's easier to quibble over formatting guidelines.
Code review is usually a bad place to catch design flaws, unless it's done early. Usually a bad design means re-doing a lot of the work. That means either getting management buy in or not hitting expected velocity. If not communicated well it can also lead to bad feelings between the writer and reviewer.
Where I work the code review process has mostly broken down. There is just no bandwidth to get anything done beyond the most basic sanity checks. To actually make someone improve their code I'd need the time, energy, authority, good will and probably other things to explain and teach the other programmer what and how to do it differently. But shit needs to get done and if the code works it's hard to convince management why I should invest so much of our time essentially redoing work that doesn't need redoing.
I suspect that this bandwidth problem exists elsewhere also.
But the problem with this is it's a technical solution to a social problem.
If your developers are writing crap code in a monolith they're going to continue writing crap code in microservices but now you have new problems of deployment, observability, performance, debugging, etc etc.
As an aside I have a sneaking probably ahistorical suspicion microservices hype happened because people realised Ruby (or similar trendy dynamic languages) often ended up being a write only framework and rather than try and recover some codebase sanity people would rather abandon the code entirely and chase the new codebase high.
> If your developers are writing crap code in a monolith they're going to continue writing crap code in microservices but now you have new problems of deployment, observability, performance, debugging, etc etc.
Anecdotally I witnessed this once. There was this huge ball of mud we had that worked okay-ish. Then the architects decided "hey microservices could solve this", so we started building out microservices that became a distributed ball of mud. Every microservice shared and passed a singular data model across ~30 microservices, which made things interesting when we needed to change that model. Also, we took mediocre developers and asked them to apply rigor they didn't have in developing these services to that they were "prepared" for the failures that happen with distributed systems.
The big upside to management though was that we could farm out parts of the system to different teams on different sides of the planet and have each of them build out the microservices, with each team having different standards as to what is acceptable (what response messages look like, what coding standards should be, ect). All of this was less of a technical problem and more of a managment one, but we felt the pain of it as it was made into a technical problem.
I know. Which is why I have architectural tests that scream if someone uses the wrong module from the wrong place (adds a dependency that isn't what we want).
Of course, any dev could just remove the test, and by that tear down that boundary too. But that test being removed is much more obvious to the code reviewer who would not have noticed the dependency being snuck in. The architectural tests can also contain comments that form a high level documentation of the architecture. Unlike "architectural documents" in the form of some up front word document, this type tends to stay up to date.
In .NET I do it om the binaries, I reflect over the assemblies and list the dependencies.
In the same run one can also validate that there are no incompatible transitive dependencies.
Within assemblies (between namespaces) is much harder unfortunately. That means assemblies have to be made to make module boundaries obvious, even though it’s a bit of an antipattern. There are tools such as nsdepcop that watch namespace dependencies, but it’s brittle and a lot of work.
So your argument is that bad developers will mess things up. I am sure most would agree with that argument. What does that have to do with monolith vs. micro-services? Bad developers will make a mess of micro-services as well.
You most definitely can enforce boundaries in libraries. Simple make sure that each library can compile and work with only the dependencies available it is allowed to have.
If a team is not disciplined/skilled enough to build a well structured monolith the chances they can build and support a microservices solution which is a distributed system with orders of magnitude more failure modes and requires an order of magnitude more tooling and testing is pretty much 0.
> Eventually, your team will flip the table and start thinking how to avoid the problems they are having with the monolith and decide to do down the micro-services way.
So, yet another comment saying that modularity requires a distributed system.
That's not what I was saying. I'm saying your team is likely to want to do something drastic and is likely to go down the micro-services path. This is not a comment on correctness. This is a behavioral prediction.
I always do this, but how do when to re-architect or at what level of granularity to look for optimizations? Do we burn time looking at async processing, do we just scale the cluster, etc
I remember reading sam newman and if I am not wrong, he says something along the lines of the team size, problem at hand and few other facts determine which one to choose. Nothing is a silver bullet. Trite, but it's having knowledge of multiple architectures and using the wisdom to choose when to use one.
It is why the Reverse Conway is preached. Define your desired architecture, and fit your organization to fit it. At that point your code base will reflect the organization of people working on it, and the output will reflect the desired architecture.
I hate the idea of replacing function calls with network requests for organizational reasons, even for a fast network in the same datacenter you add several orders of magnitude of latency to the calls. If the problem is that groups can't work independently of each other in the monolith, it should be solvable with modularization and defining APIs for those modules.
I am just using services with my medium sized application.
They are not "micro", but they separate the domains and different concerns pretty well.
I have the deployment setup more or less like a monolith, but still having separation of concerns with my services. And stateless service runners.. Fair enough I have the state in a single (mirrored) database. But this works perfectly fine for the medium sized app.
Not sure why everything must be black or white.
I've felt SOA is the easiest to grow because it encourages you to swap out concrete implementations as requirements change. For example, IUserService can start off with a "local" UserService implementation that makes direct calls to a database. Once you signup with an IdP this might become UserServiceAzureAD/Okta/Auth0. Unlike microservices, I keep my compile-time guarantees that IUser continues to have the properties I require without any tooling.
Given the rhetoric here I worry that I'm the only person who's genuinely swapped out their implementation. The ol' "N-tier is stupid - you're never going to change the database" comment couldn't be more wrong.
More and more I think of OOP and services as the same thing at different scales. Objects are services, dependency injection is your service discovery/orchestration layer. Your monolith is already a microservice architecture.
In the end, extracting a microservice from a monolith built this way is just a matter of moving the implementation of the object to a separate application, and making the object a frontend for talking to that application.
The single biggest reason OOP gets a bad reputation is because lots of languages insist on treating OOP as the be-all end-all of code structure (Java and Ruby are particularly bad examples) and insist on trying to shoehorn this sort of logic into tiny dumb pieces of pure data.
recently I am getting more and more thoughtful about "accidental" complexity we add to our solutions in form of dependencies on external libs/module, frameworks such as IoC, log services anyone ;) and on the architectural side microservices etc.
> One or more (relational) databases, depending on whether you’re gonna go with single database per service or not
This imho is where serious complications can come in. A single database for all services is a good trade off if you want the nice parts of microservice decoupling but not the headaches of a distributed system. Just perhaps don’t call it “microservices” to avoid having to deal with arguments from purists who want to explain why this is not true microservices, etc.
All ideas that are good in principle, become absurd the moment they are elevated to a kind of dogma, applied to every problem, no matter if it makes sense to do so or no.
Microservices are no exception from that rule, and often repeat the same mistake as OOP did with its promise of "reusable code".
Does it sometimes make sense to break some larger services up in smaller ones? Yes.
Does it make sense to "factor out" every minor thing of the implementation into something that can individually be manhandled into a docker container because at some point in the far future, someone may save a few minutes of typing by talking to that service? No.
Why not? Because on the 1:1000 chance that what the service does is actually exactly what that other thing requires, it will probably take more time to implement an interface than it would to simply implement a clone of the functionality.
I've seen organizations that have hundreds of developers organized in 5-10 man teams, each managing their microservice. I think it tends to happen when a large organization decides to get down with the kids and start to do microservices.
Conway's law enters into it in a lot of ways. Because of the way the people are organized into tiny isolated teams, the code shares that shape too. There is an event horizon one team/service away, beyond which nobody knows what happens or who wrote the code.
What you get is that the services actually don't do that much, except take a request from one service, translate it to an internal format, perform some trivial operation, then translate it to another external format and pass it on to another service. It's a lot of code, but it's not a lot of logic. Add to that maintaining the test and prod environments as code, and suddenly it looks like this is a lot of work, but you've essentially gotten a hundred people to do work that three people could probably accomplish if it wasn't for this pathological misapplication of an architectural pattern.
Going for microservices without a central director role is indeed madness and leads to inefficiency.
My employer has a landscape like that, hundreds of microservices each managed by a different team (some teams manage multiple). However, we have an enterprise architecture group whose job it is to keep an overview and make sure every microservice is meaningful and fulfills a clear role for the organization. Every project presents their architecture to this group as well as a group of peers and this often results in changes that increase cohesion and avoid redundant work. We had a few semi-interconnected monoliths before, and from what I’m told (I joined after the microservice transition) the new way is better.
However, I still wouldn’t recommend microservices to a new team / org starting from scratch. IMHO microservices only make sense when the system grows so vast it cannot be understood in its entirety by a single person.
> However, I still wouldn’t recommend microservices to a new team / org starting from scratch. IMHO microservices only make sense when the system grows so vast it cannot be understood in its entirety by a single person.
I wouldn't go that far. The problem is prescribing a stock design solution to every problem without even considering the problem domain or what benefits it will bring.
There are domains where this style of programming is an absolute benefit, even at smaller scales, and it's really nothing new either. A lot of the patterns in microservice design rhyme rather well with what Erlang has done for decades.
> The exact reason why I compare microservices to OOP ;-)
That's just because some guru -- Uncle Bob, or even more probably Martin Fowler, I think -- some (rather long by now) time ago wrote a lot of examples along the lines of "methods should be at most five, preferably three, lines long" in Java.
If you look for later examples of the same kind of recommendation you'll probably find they're mostly written in funtional languages nowadays, so you could just as well say "that's why I compare microservices to FP".
I agree, but I think this kind of "ad absurdum" endpoint is principally an organisational problem.
Companies are hard. They have histories. Different people with opposing ideas. Dissonance between how the company operates and how it operates on paper. Conflicting personal interests, politics. Stories they tell...
They need some sort of rough-cut ideology, system or whatnot. A way of adjudicating right or wrong. That way arguments can be settled. People can be onboarded more easily. The company's decisions become more legible.
A simple requirement to report on important decisions, the choices available and reasoning behind the choice... stuff like this has consequences. An ethos is more flexible than a ruleset, and provides a way of navigating organisational issues.
People working on their own or in small autonomous groups don't tend to go overboard as easily.
I find with microservices, they are a fake separation of concerns. When I have to work on something, and it covers two or three services, I'm actually working on a mono app.
I've found smaller "service" classes that do one job meet the same need. One or two public methods "perform" and "valid" seem to work perfectly.
> What if the product (department) doesn’t give a damn about the underlying system architecture? I mean shall they?
They should not. Either it works correctly, or it doesn't. Even buildings need to be useful, even though people might admire the architecture. No non-technical person will admire your microservice or whatever architecture.
Microservices is kind of a superpower that allows you to choose where you want your complexity to be. You are supposed to start with something simple, so the microservices decision needs to come in later.
Not starting full in with microservices is a good pattern.
467 comments
[ 3.3 ms ] story [ 477 ms ] thread- Kubernetes is rather a harder way to build microservices.
- DB is not an obligatory part of microservices.
- Kafka isn't as well. It's a specific solution for specific cases when you need part of your system to be based on an events stream.
- Jenkins is not necessary, you can still deploy stuff with a local bash script and you need containerization whether you are on Microservices or Monolyth architecture
- Kibana, Prometheus, Zipkin are not required. But I think you need both logs aggregation and monitoring even if you have just a Monolith with horizontal scalability.
Also, all this is assuming you are not using out of the box Cloud solutions.
If the microservices don't have their own store, but are all mucking around in a shared data store, everything will be much harder. I wouldn't even call that a microservice, it's a distributed something. It can work, sure.
I certainly don't agree that 'anything that does permanent storage is a DB'. Not many people would call their hard drive a 'database'
I personally think that they all have a place and are very useful tools for certain situations, depending on the problem.
That could probably the best instance of something that can be built outside of the monolith and can be manipulated separately.
Then, in about two years, everything changed. Suddenly, every new web project (and web was also novel) included a MySQL DB. That's when the idea about the three tier architecture was born. And since then, a few generations of engineers have been raised that can't think of a computer system without a central DB.
I'm telling this because in microservices I see the opportunity to rethink that concept. I've built and run some microservices based systems and the biggest benefit wasn't technical, but organizational. Once, the system was split into small services, each with its own permanent storage (when needed) of any kind, that freed the teams to develop and publish code on their own. As long as they respected communication interfaces between teams, everything worked.
Of course, you have to drop, or at least weaken, some of ACID requirements. Sometimes, that means modifying a business rule. For example, you can rely on eventual consistency instead of hard consistency, or replenishing the data from external sources instead of durability.
Otherwise, I agree with the author that if you are starting alone or in a small team, it's best to start with a monolith. With time, as the team gets bigger and the system becomes more complex, your initial monolith will become just another microservice.
This is what I do. Single bash script when ran on bare OS can install and configure all dependencies, create database from backup, build and start said monolith. All steps are optional and depend on command line parameters.
Since I deploy on dedicated servers I have no real need for containers. So my maintenance tasks are - ssh to dedicated server and run that script when needed. Every once in a while run the same thing on fresh local VM to make sure everything installs, configures, builds and works from the scratch.
The code is a microservice-esque architecture. Some of the services are a bit chonky, but overall it's roughly along those lines, besides the search engine I've got a lot of random small services doing a lot of things for personal use, scraping weather forecasts and aggregating podcasts and running a reddit frontend I built.
I'd gone for kubernetes mostly because I wanted to dick around with the technology. I'm exposed to it at work and couldn't get along with it, so I figured we may get on better terms if I got to set it up myself. Turns out, no, I still don't get along with it.
Long story short, it's such a resource hog I ended up getting rid of it. Now I run everything on bare metal debian, no containers no nothing. Systemd for service management, logrotate+grep instead of kibana, I do run prometheus but I've gotten rid of grafana which was just eating resources and not doing anything useful. Git hooks instead of jenkins.
I think I got something like 30 Gb of additional free RAM doing this. Not that any of these things use a lot of resources, but all of them combined do. Everything works a lot more reliably. No more mysterious container-restarts, nothing ever stuck in weird docker sync limbo, no waiting 2 minutes for an idle kubernetes to decide to create a container. It's great. It's 100 times easier to figure out what goes wrong, when things go wrong.
I do think monoliths are underrated in a lot of cases, but sometimes it's nice to be able to restart parts of your application. A search engine is a great example of this. If I restart the index, it takes some 5 minutes to boot up because it needs to chew through hundreds of gigabytes of data to do so. But the way it's built, I can for example just restart the query parser, that takes just a few seconds. If my entire application was like the query parser, it would probably make much more sense as a monolith.
maybe we are getting caught up in sematics because its christmas, but "monorepo/monolith/microservices/etc" is -just- the way you organize your code.
Developing a montolith for years but now you have written a 15 line golang http api that converts pdfs to stardust and put it into on a dedicted server in your office? welp thats a microservice.
Did you write a 150 repo application that can not be deployed seperatly anyway? welp thats a monolith.
You can also build a microservice ecosystem without kubernetes on your local network. We have done it for years with virtual machines. Software defined networking just makes things more elegant.
So dont stop using microservices because its "hard" or start writing monoliths because its "easy", because none of that is true in the long run.
What is true is that you have a group of people trying to code for a common goal. The way you reach that goal together defines how you organize your code.
It’s also about how you deploy your code. If you have 1000 micro services do you have 1000 deployment pipelines? If so how do you manage those pipelines? If not, you sacrifice independent deployment of each micro service.
Also deployment is part of code organistaion no? i like to point out the fact that i explicitly mentioned you do not need to decide, you can use both at the same time but i would like to try to answer your point anyway:
Maybe I am just spoiled with go and github, but those deployment pipelines can be directly integrated into your repository. The same way I can counter argue that your build ci and deployment phase will take significantly more time too and if you change a variable from uppercase to lowercase you will wait for minutes too.
I come from a c# background a long time ago and this has been true with it for eons: https://xkcd.com/303/
Another thing that I have noticed is that its easily scriptable too. What I end up doing is to provide a makefile into my repository too. This way i can define the build process amongst other things eaisly.
In the end: We have a predefined spec that will create the desired state on a host of our chosing.
Ansible really does not care if you deploy 1 or 1000 "services" for instance! Tools Like ArgoCD will also deploy stuff when you commit to your master.
There is tooling for everyone available, but what happnes quite often is that the people in charge of the company expected the developers to be all knowing entities that need to define everything end to end. The Kubernetes Space is vast because we are still defining and communication its future.
But recently? I am trying to think of something that would not be completly automateable in 2021
How? Can you provide any examples?
If this is actually (still) true, that means that "the way you organize your code" is a bit simplistic. Your example of an "http api that converts pdfs to ..." is surely a valid example of a microservice, but most business products have to handle much more "state" than those, and this will create further complications which go far beyond "how to organize your code" (and make monoliths more appealing).
But this also means that we are now back into "YesSQL" territory, and specifically that we have to use a RDBMS which allows us to create Views. Goodbye NoSQL, goodbye Key+Valus datastore. (Or maybe you will just create an extra "newZipCode" and mantain it in parallel with "ZipCode" allowing every other consumer to adapt at their leisure...?).
So it is another step back to "more traditional ways" to design a system... or a recipe for a disaster as soon as you start dealing with significant quantities of "state".
The PDF Example for instance: I have to provide all assets that will be converted to startdust or at least links for them I have to define where the finished stardust would be send to (e.g http post)
but its function is nonetheless independent. the epepheral disk on your container is a datastore too. If you need what is stored actually for longer than the request.... that is another story.
When you have multiple bits of code responsible for writing the same columns, maintaining global invariants becomes much harder.
I can still see rationale for exceptions to the rule, e.g. status fields for rows which logically represent some ongoing process or job, where a UI controller may set something to "processing" and the background job sets it to "processed" when it's complete. But there are ways to close the loop here. For example, implement the background job by having the job processing system invoke the same system which owns the field and sets it in response to the UI, like this:
I didn't want to put unneeded information in the first post but here i have some space ;)
What I ended up doing was:
This results in: All this means my client has total flexibility in deployment."Single writer multiple readers", yes, this is what I would probably use, but yet again, wasn't the "promise" of Microservices being able to work in total isolation?
If I have one table (e.g. "Customer") which is written by one specific microservice and read by a dozen or more... what happens when I decide that I have to change the schema because the current representation of "Zip Code" is not adequate anymore because, I dunno, we started dealing with UK customers now?
Lo and behold, I have to change code in 13 Microservices - the one actually writing to it, and the 12 more that only need to get the data to show or print or convert to JSON or whatever... ¯\_(ツ)_/¯
The services themselves do not differ, but are fanned out. E.G. so 100 workers watch your pdf queue to generate pdfs.
If your zip code is really used by many services, the question is how you want to communicate those objects and there are really a lot of choices out there. GRPC / Soap / Shared model package / and the list goes one.
On the other hand i have obsverved that people push full blown models around just to end up reading one field of it in their monolith. I believe the PHP frameworks of the past where the biggest offenders like that.
When i was working with sugarcrm for instance, it was not uncommon to pass huuuuge objects between functions because you needed 1 field. It's PHP so by default, function arguments are passed by value, so you can already see where the fun begins.
Having an RPC to migrate means you can put code on the read path to support backward compatibility while migrating. But you can do the same thing with the database - support both fields until all readers have ported to the new field - and since big database migrations are expensive, in practice large systems do migrations that way anyhow. Introduce the new field, start writing to it, back-fill it, port readers across individually (there may be many read locations even within a single service), drop the old field.
If a system evolved from a monolith, one way to partition services is to provide views to limit set of visible columns. That's another point at which you can put shims for compatibility while porting schemas.
It changes the situation regarding deployment, but it does not miraculously absolve you from changing code when data formats change. How could it?
My point is precisely that: if you have to handle large quantity of state (e.g.: travel agency handling diverse item bookings to sell as a complete holiday packages - note that this includes having conflicts on inventory, like "cruise cabin categories" or "hotel rooms") microservices add latency by "replacing function calls with RPC", and gain you... an unspecified advantage in terms of... deployment? The possibility to have hundreds of developers working on the system in parallel?
I have always worked on medium-size monoliths during most of my career, and "ah, if we had 137 developers all working on this everything would be magically solved, but alas, we have a monolith" was a sentence I uttered (or heard) exactly 0 times so far.
But the 15 lines of Golang are not just 15 lines of Golang in production. You need:
- auth? Who can talk to your service? Perhaps ip whitelisting?
- monitoring? How do you know if you service is up and running? If it's down, you need alerts as well. What if there is a memory problem (because code is not optimal)?
- how do you deploy the service? Plain ansible or perhaps k8s? Just scp? Depending on your solution, how do you implement rollbacks?
- what about security regarding outdated packages the Go app is using? You need to monitor it as well.
And so on. The moment you need to store data that somehow needs to be in sycn with the monolith's data, everything gets more complicated.
Production stuff is not just about lines of code.
The task of the Microservice is to convert the pdf to stardust and to return it to its sender. so no auth. Furthermore its most likely only reachable through the local network, or at least should be if you want some stranger not to be able to also make stardust from pdfs.
Monitoring: are you trying to say that its a lot esaier to pick up one logfile thant lets say 15? because they should be aggregated somewhere anyway no?
Deployment: Depending on anything you listed how do i do anything? Of course if have to define it but if you want a fancy example: k8s argocd canary deployments done. I literally set it up once.
security? Really? Please dont get this wrong but this feels to me like whataboutism but well here i go:
i implement security just the same way as i would in the monorepo. The thing/person/entity just has to look into more repositories ;)
It comes down do one sentence i think: State is not shared, state is communicated.
Monitoring: it's easier to look at a stack trace, including some other team's external library, than a HTTP error code 500.
Deployment is certainly easier when you're just shipping code and a build. You don't have to faff around with the previous instance running, maybe having some active connections/transactions/whatever, needing to launch a new one. Maybe it's not hard overall, but less fun.
Auth can be a very broad term in the case of services, especially in the cloud. IAM could only allow access from certain other machines, etc...
Auth: nobody knows IP address of our server anyway, don't bother with that. And for extra security we have secret port number.
Monitoring? Well, we have our clients for that. They'll call us if something happens.
Deploy? Copy .war file and restart tomcat. Rollback? What's rollback?
Outdated packages? We call those stable packages.
I wish I would work in a miracle world you describe.
Figuring out what was wrong with it was hard, because the logging was inconsistent, all over the place in where it logged and also insufficient.
The deployments and environments were inconsistent, the sysadmins on clients's side manually changed stuff in the .war archives, all the way up to library versions, which was horrendous from a reproducibility perspective.
The project was also severely out of date, not only package wise, but also because the new versions that had been developed actually weren't in prod.
After ripping out the system's guts and replacing it with something that worked, i became way more careful when running into amateur hour projects like that and about managing the risks and eventual breakdown of them. I suggest that you do the same.
Don't put yourself at risk, especially if some sort of a liability about the state of the system could land on you. What are you going to do when the system gets breached and a whole bunch of personal data gets leaked?
Yes, if there is mission critical software that affects millions of people, do all that stuff.
If your pdf -> stardust service is more a convenience then don't spend hundreds of thousands on infrastructure.
I find it curious how we went from doing the basics of software development that would minimize risks and be helpful to almost any project out there to this.
To clarify, i agree with the point that you'll need to prioritize different components based on what matters the most, but i don't think that you can't have a common standard set of tools and practices for all of them. Let me address all of the points with examples.
> Auth: nobody knows IP address of our server anyway, don't bother with that. And for extra security we have secret port number.
Port scanning means that none of your ports are secret.
JWT is trivial to implement in most languages. Even basic auth is better than nothing with HTTPS, you don't always need mTLS or the more complicated solutions, but you need something.
This should take a few days to a week to implement. Edit: probably a day or less if you have an easily reusable library for this.
> Monitoring? Well, we have our clients for that. They'll call us if something happens.
This is not viable if you have SLAs or just enjoy sleeping and not getting paged.
There are free monitoring solutions out there, such as Zabbix, Nagios, Prometheus & Grafana and others. Preconfigured OS templates also mean that you just need the monitoring appliance and an agent on the node you want to monitor in most cases.
This should take close to a week to implement. Edit: probably an hour or less if you already have a server up and just need to add a node.
> Deploy? Copy .war file and restart tomcat. Rollback? What's rollback?
This is an error prone way to do things, as the experience of Knight Capital showed: https://dougseven.com/2014/04/17/knightmare-a-devops-caution.... In addition, manual configuration changes lead to configuration drift and after a few years you'll have little idea about who changed what and when.
In contrast, setting up Ansible and versioning your config, as well as using containers for the actual software releases alongside fully automated CI cycles addresses all of those problems. In regards to rollbacks, if you have automated DB migrations, you might have to spend some time writing reverse migrations for all of the DDL changes.
This should take between one to two weeks to implement. Edit: probably a day or less once per project with small fixes here and there.
> Outdated packages? We call those stable packages.
Log4j might be stable, but it also leads to RCEs. This is not a good argument, at least for as long as the software packages that we use are beyond our ability to control or comprehend.
This should be a regular automated process, that alerts you about outdated and/or insecure packages, at least use npm audit or something, or proactive scanning like OpenVAS. This should take close to a week to implement.
All of the mentioned software can easily run on a single node with 2 CPU cores and about 8 GB of RAM. I know this, because i did all of the above in a project of mine (mentioned technologies might have been changed, though). Of course, doing that over N projects will probably increase the total time, especially if they have been badly written.
In my eyes that's a worthwhile investment, since when you finally have that across all of your apps, you can develop with better certainty that you won't have to stay late and deliver 2 or 3 hotfixes after your release, which goes hand in hand with test coverage and automated testing.
That is hardly hundreds of thousands of dollars/euros/whatever on infrastructure. And if the personnel costs are like that, i'd like to work over there, then.
The main point is, that one should never spend a "a few days to a week" to implement a feature that at best i useless and at worst is detrimental to the service stood up.
Implement auth, if it is needed, implement monitoring, CI, CD, dependency monitoring, testing, everything, if it is needed.
But don't implement it as dogmatic consequences of doing software development.
And regarding the spend: one week worth of work could be USD 8k. So just the initial implementation of your JWT based authentication system is 4% into the "hundreds of thousands of dollars". Then you need to factor in the extra complexity on maintenance and before you know it we do not talk about hundreds of thousands of dollars but millions...
The point you should have made is that dogmatic approaches usually produce a lot of waste, but the example you gave us exactly why teams end up that way. Otherwise people come up with bullshit hacks like you describe and the entire team pays for it.
Though these were not the cases I talked about.
I talked about implementing features that are not necessary for the product because of dogmatic reasons.
Sure, i'm not necessarily advocating for a full blown RBAC implementation or something like that, merely something so that when your API is accidentally exposed to the rest of the world, it's not used for no good (at least immediately).
> Implement auth, if it is needed, implement monitoring, CI, CD, dependency monitoring, testing, everything, if it is needed.
> But don't implement it as dogmatic consequences of doing software development.
Now this is a bit harder to talk about, since the views here will probably be polarized. I'd argue that if you're developing software for someone else, software that will be paid for (and in many cases even in pro-bono development), most of that is needed, unless you just don't care about the risks that you (or someone else) might have to deal with otherwise.
If there's an API, i want it to at the very least have basicauth in front of it, because of the reasons mentioned above.
If there is a server running somewhere, i want to be alerted when something goes wrong with it, see its current and historic resource usage and get all of the other benefits running a few apt/yum commands and editing a config file would get me, as opposed to discovering that some bottleneck in the system is slowing down everything else because the memory utilization is routinely hitting the limits because someone left a bad JVM GC config in there somewhere.
If something's being built, i want it to be done by a server in a reasonably reproducible and automated manner, with the tasks are described in code that's versioned, so i'm not stuck in some hellscape where i'm told: "Okay, person X built this app around 2017 on their laptop, so you should be able to do that too. What do you mean, some-random-lib.jar is not in the classpath? I don't know, just get this working, okay? Instructions? why would you need those?"
Furthermore, if there is code, i want to be sure that it will work after i change it, rather than introducing a new feature and seeing years of legacy cruft crumble before my eyes, and to take blame for all of it. Manual testing will never be sufficient and integration tests aren't exactly easy in many circumstances, such as when the app doesn't even have an API but just a server side rendered web interface, which would mean that you need something like Selenium for the tests, the technical complexity of which would just make them even more half baked than unit tests would be.
Plus, if i ever stumble upon a codebase that lacks decent comments or even design docs/issue management, i will want to know what i'm looking at and just reading the code will never be enough to understand the context behind everything but if there are at least tests in place, then things will be slightly less miserable.
I'm tired of hating the work that i have to do because of the neglect of others, so i want to do better. If not for those who will come after me, then at least for myself in a year or so.
Do i do all of that for every single personal project of mine? Not necessarily, i cherrypick whatever i feel is appropriate (e.g. server monitoring and web monitoring for everything, tests for things with "business logic", CI/CD for everything that runs on a server not a local script etc.), but the beauty is that once you have at least the basics going in one of your projects, it's pretty easy to carry them over to others, oftentimes even to different stacks.
Of course, one can also talk about enterprise projects vs startups, not just personal projects, but a lot of it all depends on the environment you're in.
As for the money, i think that's pretty nice that you're paid decently over there! Here in Latvia i got about 1700 euros last month (net). So that's about 425 euros a week, or more like 850 if you take taxes and other expenses for the employer into account. That's a far cry fro...
However, i just had a look at https://www.algas.lv/en, one site that aggregates the local salary information, based on a variety of stats. As someone with a Master's degree and ~5 years of experience and whose job description states that i'm supposed to primarily work in Java (even though i do full stack and DevOps now), i input those stats and looked at what information they have so far.
The average net salary monthly figures, at least according to the site, are:
A brief litmus test makes me think that the values are vaguely believable.For example, if we have a look at some other sources for Lithuania, such as https://www.glassdoor.com/Salaries/vilnius-software-develope..., we get a figure of 2 458 EUR/month --> 614 EUR/week.
Or, if we look at other sources for Estonia, such as https://www.payscale.com/research/EE/Job=Software_Developer/..., we get a figure of 28 000 EUR/year --> 2 333 EUR/month --> 583 EUR/week.
So there's definitely a bit of variance, but it's still in the same ballpark. Of course, there are better companies out there, but the reality for many is that they're not rewarded for their work with all that much money.
I actually wrote about how that has impacted my ability to make savings on my blog: https://blog.kronis.dev/articles/on-finances-and-savings
Again, I advocate developing in a timely manner, and not do over engineering (neither under engineering).
I made the above response with that in mind.
> And regarding the spend: one week worth of work could be USD 8k.
The original claim was that one week's worth could be 8000 USD, or let's say roughly 7094 EUR. That comes out to 28376 EUR per month.
Last month i made around 1700 EUR, so it's possible to calculate approximately how much my work cost to my employer. Let's do that with a calculator here: https://kalkulatori.lv/en/algas-kalkulators
After inputting the data that's relevant to me, i got the following:
It should be apparent that 28376 EUR is a far cry from 3281 EUR, which is how much my work cost to my employer.Thus, per week, 7094 EUR is also a far cry from 820 EUR, which is how much my work cost to my employer.
Also, 820 is actually pretty close to my initial guess of 850 EUR.
Of course, it's possible to argue that either i'm underpaid individually, or that many of my countrymen in Latvia are underpaid in general (on which i elaborated in an adjacent comment https://news.ycombinator.com/item?id=29595158), but then the question becomes... so what?
Does that mean that if you're in a well paid country like US, then you cannot afford proper development practices due to all of the payroll expenses that would cause? While that may well be, to me that sounds weird and plain backwards - if that were really true, then US would outsource even more to countries like mine and these outsourced systems would work amazingly well, since you can supposedly afford a team of developers here for what would buy you a single developer over there. And yet, most systems are still insecure, buggy and slow.
Maybe someone else is pocketing a lot of the money they receive in these countries, and is simply charging high consulting rates? The prevalence of WITCH companies here is telling, but that's a discussion for another time.
1. auth? probably an internal service, so don't expose it to the outside network.
2. monitoring? if the service is being used anywhere at all, the client will throw some sort of exception if its unreachable.
memory problem? it should take <1 day to ensure the code for such a small service does not leak memory. if it does have memory leaks anyways, just basic cpu/mem usage monitoring on your hosts will expose it. then ssh in, run `top` voila now you know which service is responsible.
3. deployment? if its a go service, literally a bash script to scp over the binary and an upstart daemon to monitor/restart the binary.
rollback? ok, checkout previous version on git, recompile, redeploy. maybe the whole process is wrapped in a bash script or assisted by a CI/CD build job.
4. security? well ok, PDFs can be vulnerable to parser attacks. so lock down the permissions and network rules on the service.
Overall this setup would work perfectly fine in a small/medium company and take 5-10x less time than doing everything the FAANG way. i don't think we should jump to calling these best practices without understanding the context in which the service lives.
Haha. You're a bit of an optimist, eh?
> if it does have memory leaks anyways, just basic cpu/mem usage monitoring on your hosts
Who keeps on monitoring like this? How frequently would you do it? In a startup there are somewhere in the range of 5 microservice of that scale per programmer and daily monitoring of each service by doing top is not feasible.
> 3. deployment? if its a go service, literally a bash script to scp over the binary and an upstart daemon to monitor/restart the binary.
Your solution literally is more complex than simple jenkins or ansible script for build then kubectl rollout restart yet is lot more fragile. Anyways the point stands that you need to have a way for deployment
For example, kubectl rollout assumes that your service is already packaged as a container, you are already running a k8s cluster and the team knows how to use it. In that context, maybe your method is a lot better. But in another context where k8s is not adopted and the ops team is skilled at linux admin but not at k8s, my way might be better. There’s no one true way and there never will be. Technical decisions cannot be made in a vacuum.
The point was never comparing it to the FAANG way. The point is: it's easier (at the beginning) to maintain ONE monolith (and all the production stuff related to it) than N microservices.
Our internal software plattform is getting to a point so it can answer most of these things - auth via the central OIDC providers, basic monitoring via annotations of the job's services, deployments via the orchestration and some infrastructure around it, including optional checks and automated rollbacks and automated vulnerability scanning on build-servers and for the running systems. It wouldn't be 15 lines of go, more like 15 lines, plus about 100-200 lines of terraform and/or yaml to get everything configured, and a ticket do register the service in the platform. It's pretty nice and our solution consultants like it very much.
The thing is - this took a team about a year to build and it'll take another half a year to get everything we currently want to do right. And it takes a non-trivial time to maintain and support all of this. This kind of infrastructure only makes business sense, because we have enough developers and consultants moving a lot faster with this.
Back when we were a lot smaller, it made a lot more sense to just push a single java monolith on VMs with chef or ansible, because that was a lot easier and quicker to get working correctly (for one thing).
So in the k8s world:
- auth: service meshes, network policies, ...
- monitoring: tons of tooling there to streamline that
- deploy: this at scale is trickier than you'd think, many seem to assume k8s on it's own here is the magic dust they need. But GitOps with ArgoCD + helm has worked pretty well at scale in my experience.
- Security is a CI problem, and you have that with every single language, not just Go. See Log4j.
Kubernetes is my bread & butter, but I do realise this has way too much overhead for small applications. However, once you reach a certain scale, it solves many of the really really hard problems by streamlining how you look at applications from an infrastructure and deployment side of things. But yes - you need dedicated people who understand k8s and know what the hell they're doing - and that's in my experience a challenge on it's own.
Let's also dispel a myth that k8s is only suitable for microservices. I have clients that are running completely separate monolith applications on k8s, but enough of those that managing them 'the old way' became very challenging, and moving these to k8s in the end simplified thing. But getting there was a very painful process.
(And yes, if you need extra argument to sell k8s, remind them about bin-packing more applications onto less hw)
It's also a common trope when I'm dealing with k8s decriers - yes, you might have one application that you can easily deploy, but suddenly there are 15 other medium-weight applications that solve different important problems and you want them all ;)
P.S. Recently a common thing in my own architectures is separate keycloak deployment that all services either know how to use, or have it handled at separate request router (service mesh or ingress or loadbalancer)
I don't think this is true.
I think — at least as far as I've observed — microservices in practice means replacing various functions calls with slow and error-prone network requests.
This is a big and meaningful difference.
The application dev does not need acces to production whatever and the devops person does not need to know how the application works.
But then again these to people also need to communicate ;)
That way, i can attach as much functionality as i want without bloating the main app and processing scales with demand.
Actually it's been going on for years, and it's always the same argument. People think they're thought leaders for saying "Start with monoliths and only move to microservices if you absolutely need to!"
It's a pretty obvious conclusion to anyone who has worked in both environments, or have had to migrate from one to the other, so it's not particularly insightful. And yet here were are, 5+ years later saying the same thing over and over again.
It should be obvious, but apparently it's not. So many architecture astronauts drinking the kool-aid and making a mess. Premature microservices can easily kill a product.
rather they are used to solve organizational problems. Having 10 developers working on a single monolith? Probably fine. 100? Good luck managing that.
Yes they add technical complexity. But they reduce organizational complexity.
In what way do Microservices even help? It seems to me you still have to synchronize to be sure that the Microservice from team B does exactly the things that are specified in the new version?
Is it not easier to have a pull request that says "this will do thing x", you merge it into your monolith, and then you can see in the git log that this version will indeed to x?
How do Microservice organizations even manage that? Is it the reason that Atlassian has a billion dollar evaluation, because people need it to keep track?
To me, if youre working with a lot of modules/micro services, lots of modules should be able to sit on old versions and develop independently (which is the crucial part for 100 developer scenario)
You can have a modular monolith that works just as well with 100 people as something service-oriented would. The difference lies in the level of discipline needed. It's much easier to "just go in and make that field public because it makes my implementation easier" when you have a modular monolith. With microservices, you are more explicitly changing an external API by doing that.
Yes, it's the same thing. But somehow, psychologically, people feel worse about changing a networked API than making an identifier public instead of private.
Edit: I forgot, there's one more thing: with service orientation, you can deploy in finer grains. You shouldn't have heavily stateful services, but if you do (and you always do!), it can be cumbersome to redeploy them. At that point, it's nice to be able to deploy only the parts that changed, and avoid touching the stateful stuff.
I think you can remove Java in that sentence.
I say this with the assumption that the team is large and members regularly come and go.
Then microservices increase organizational complexity too.
Suddenly product owners and middle management are chief architects without even knowing it.
Microservices is an organizational optimization. It can allow one team to manage and deploy their own subsystem with minimal coordination with other teams. This is a useful thing, but be aware what it's useful for.
If each of your developers manages a microservice, that probably reflects that you do no actual teamwork.
Need to rightsize the modules. More than one, but just a few with boundaries chosen around recovering from faults
That was the promise back in the day of J2EE, and it seems to me Microservices are just a rehash of that same promise.
Which never really worked out with J2EE - it was mostly invented to sell big servers and expensive consultants, which is how Sun made money?
These days I sometimes don't even bother to make extra interface classes for everything. If I need it, I can still insert that, but no need to do it up front.
And "DevOps" is just a ploy to make developers do more work, that was formerly done by admins.
Writing every Microservice in a new language also seems like a huge headache. The only bonus is that you can attract developers by being able to promise them that they can work with some shiny new technology. Maybe that is actually the main reason for doing that?
Otherwise, again, perhaps it is my age, but I prefer to minimize the dependencies. Do I really want to have to learn a whole new programming language just so that I can fix a bug in some Microservice. I personally don't want to.
Devs only focusing on developing/writing source code is certainly not the way to produce sustainable, high quality software- at least in my opinion...
Still I think having extra admins for server maintenance, or nowadays DevOps, also makes sense in my opinion. In theory I should be interested, but in practice, somehow I can not really get passionate about the details of how to host things. So it would be better to have specialists who really care.
Ultimately there are also a lot of aspects that are different from the programming problems. It's difficult to keep up to date on both fronts.
Maintenance and the automation of repetitive tasks are in my view part of DevOps, but not every dev in a team has to participate at the same degree...
I think high quality comes from specialists. Sharp knives in the hands of pro's
In a huge organisation, it can work well to be very specialized.
In, for example, a smaller growing business with an engineering team of < 10, having a proper understanding of the context in which your code will run is a game-changer.
Like TDD the design of software changes when you have experience with ops/maintenance. Take logging/tracing: what to log or trace is first and foremost a matter of experience with maintenance/ops! Building in logging/tracing/monitoring from the get-go because you know that you want to know what goes wrong, when it goes wrong etc...
can one regard oneself as expert if you know only a part of the domain- I think not!
Maybe "before my time" it was a webmaster or other admin. I think of "dev ops" as an umbrella term subsuming many specific titles and roles from the past, much like "data science".
I suspect you don't mean it like that, but developers have to care just a little about operations. There's the classic stuff about developers who build stuff, because they didn't realize that ops could do the same with a few lines in a web server config, so they waste weeks on trivial stuff. There's also the issue that if you expect databases, queues, disks and so on to just be available, while not thinking about how you use them, then you can get bad performance or cause downtime and crashes in the worst case.
The initial idea of DevOps also seems to have been twisted to having developers do operations, rather than having the two teams work in tandem. If we absolutely must combine the two roles, I'd be more comfortable having operations do development work, but that hardly ideal either.
Yes, I'm cynical. Having to use javalike stuff to do what should be a simple shell script will do that to you.
But I'm a developer by heart and my heart aches whenever I see what we devs have wrought in the ops space. (Insert joke about the CNCF technology landscape map.)
There's just so many tech/tools/etc. involved that just reasonably "doing the ops stuff" on the side seems way unrealistic. Sometimes I feel that all we've accomplished was job security for legions of consultants.
> But I'm a developer by heart and my heart aches whenever I see what we devs have wrought in the ops space.
At the same time, I am conflicted. I don't care for the toss-it-over-the-wall approach that used to be the norm, but I also don't like having dev's have to take more on than they are capable.
In an ideal environment, I would like to see a crashing together of developers and operations people on teams. What I mean is that for a given team you have several developers and one or two ops folks. This way there is less of an us vs them sentiment and teams can be held accountable for their solutions from ideation to running in production. Finally while it's not the sole responsibility of the dev's to manage their DevOps stuff, they will have more knowledge of how it works, and get to put input and ideas to it.
I also think silos are unhealthy. SDEs need to have some overlapping responsibilities with SREs. Otherwise, you’ll likely have a “throw it over the fence” culture
Developers cost a lot more than admins, so this seems silly.
Moreover “devops” people are supposed to be developers, just ones with a whole bunch of networking and admin skills as well. But few places outside Google have real SREs.
Organizational reason would be multiple people/teams who don't want or can't talk much to each other, so they develop pieces of a larger system as relatively independent projects, with clear API and responsibility boundaries. Frontend/backend style web development is an example of such approach, even though we don't typically call these parts "microservices".
A technical reason I can see is some component of a system actually having to be written in a different stack, or to be run in a separate location for business reasons (separate physical computer, separate VMs or containers don't count). Like a firmware running on an IoT system. Or most of the system uses python, but there's a really good library in java for solving some very specific problem, so let's use it.
If neither of these reasons stands, you don't have a microservice architecture, you have a distributed monolith. You just replaced some function calls with RPC. RPC call which takes a much a longer time than a local one, and can randomly fail. Most of your microservices are written in a single stack, so you refactor common parts into a library, but then different services are stuck to use different versions of this library. You end up with a much slower and a much more fragile system which is harder to work on for no good reason.
How about deployment speed? If I’ve got a microservice collecting events off a queue and writing a csv out to S3 on a schedule, it’s really nice to be able to add a column and deploy in minutes without having to rebuild and deploy a giant monolith. It also allows for fine grained permissions: that service can only read from that specific queue and write to that specific bucket.
People throw around “distributed monolith” like it’s a dirty phrase but I’ve found it actually a very pleasant environment to work in.
Deploying a single monolith is faster than deploying 10 microservices, especially if you find yourself in the model where your microservices share code, you've ended up with a distributed monolith instead of microservices.
Yes, but that use case happens to be something that I need to do 5 times a day - make a small (<200 line) code change to one small part of the distributed monolith, and deploy it immediately.
This also means that if something goes wrong, I can rollback those changes immediately without rolling back the work of any of the other 50 engineers in the company.
Very little signoff required, very small blast radius.
Say your little service just changed how it parsed backticks. Now that innocuous change may affect none of the immediately connected microservices but another services three hops away relied on the old behavior of your parser through some complex business rules driven logic. Now go test and later troubleshoot that vs standing up a single monolithic jar on your laptop and seeing the exception stack trace tell you exactly what you broke.
You could screw it up and make it harder for yourself but its not guaranteed either way.
> especially if you find yourself in the model where your microservices share code
Good architecture decisions trump the monolith vs microservices argument. I'm not saying cross-service shared code is inherently bad, but it does have a bad smell.
I call this a worker.
that's a tooling issue not a monolith vs microservice issue.
tooling can allow the exact same speed of deployment for tiny background processing without the need to completely segment your code base.
> branch from whatever branch is in prod
> implement change on that branch (in this case adding a column), test
> deploy to prod
I realize the build and deployment process may be more complex than that making it hard... but it doesn't have to be.
I agree that a microservice OR even another system (a collection of services) is a good solution if you need to make quick iterative changes, and you can't do so with your current system.
The benefits of working on a small repo include:
* Faster compilation (only depend on exactly the libraries you need)
* Faster testing (only need to run the tests which are actually relevant)
* Easier to understand for new joiners because there's just less code to sift through
* Faster startup (this is probably java specific - the classloader is slooow)
* No (okay, fewer) rebase wars
* Easy to see what's currently being worked on because the Pull Requests tab isn't 200 long
* Very fast rollbacks without needing to recompile a git revert (just hotswap the container for that particular service). You can't do this in a monolith without risking rolling back someone else's important changes.
For ones I saw, the answer to the question "how do I run our project on a single laptop?" is "haha, you don't". That makes features which could take hours to implement take weeks. But deployment is 5 minutes instead of… 15? Not too worthy of a trade-off.
Your CSV writer probably has both technical and organizational reasons being an independent unit of development. Or, in other words, something which appeared organically, rather than someone deciding months prior before a project even started that authentication and user profiles should live in two parallel universes.
I got so burnt out developing in that environment that I asked to become a pure frontend dev so that I wouldn't have to deal with it anymore.
This is orthogonal to monolith vs microservices. I've worked on monoliths that could easily be (and were) deployed very frequently.
The monolith where most API endpoints are instant and use constant memory, but some use much more memory and can be slower... is tough.Like if you just give a bunch of memory to each process now you're overprovisioning and if you try to be strict you run into quality of service issues.
If you split out homogenous API endpoints into various groups you now have well behaved processes that are each acting similarly. One process could be very small, another could be much larger (but handle only one kind of request), etc...
of course the problem with standard microservice-y stuff is now you gotta have N different applications be able to speak your stack. The idea of a monolith with feature sets is tempting... but also can negate chunks of microservice advantages.
Ultimately the microservice-y "everything is an API" can work well even as a monolith, and you would then have the flexibility to improve things operationally later.
Imagine 95% of your workload can run in 100 megs but 5% requires 1000 megs. You can overprovision of course but in a world of containers if you can isolate the 5% and route it you’re going to have a lot less in terms of operational headaches
The problem is when the lead dev has been huffing the architecture paint too hard and starts prematurely spinning up microservices because it feels good.
* break down the problem into sensible components. for a reporting system I'm working on atm. we're using one component per type of source data (postgres, XLSX files, XML), one component for transformations (based on pandas) and one component for the document exporter.
* let those components talk through http requests with each other, including an OpenAPI specification that we use to generate a simple swagger GUI for each endpoint as well as to validate and test result schemas.
* deploy as AWS lambda - let amazon worry about scaling this instead of having our own Kubernetes etc.
* BUT we have a very thin shim in front that locally emulates API gateway and runs all the lambda code in a single process.
doing so the scaling works quite well, there is no up-front infrastructure cost and code tends to stay nice and clean because devs CANNOT just import something from another component - common functionality needs to first be refactored into a commons module that we add in each lambda, which puts a nice point for sanity checking what goes in there.I’m sure has its place for bigger applications though
I do like a modular approach though (think docker rather than k8s) which I suppose is partial micro services
A monolith makes developing initially a lot easier. Over 15 years though, you are bound to have developers of various calibre leave their mark on it. Many don't even understand good modelling and inevitably drive an otherwise healthy monolith with well defined boundaries into a soup of couplings between domain concepts that should not know about each other. In theory though it is totally possible to have micro systems within the same monolith code if you model things that way.
Eventually, your team will flip the table and start thinking how to avoid the problems they are having with the monolith and decide to do down the micro-services way. In most situations developers are likely to face a lack of commitment from the business to spend time/money on a technical problem they do not understand but have to support the development of for a lengthy period of time. Most developers will compromise by building pseudo micro-services without their own databases which send requests to the monolith where the core business logic had to stay.
The benefit of micro-services IMO is driven from being able to separate business domains in a way that keeps each responsibility simple to understand for a new comer and compact enough to not hide a great deal of complexity. It's worth saying this is a very hard thing to achieve and the average developer shop won't have the experience to be able to pull it off.
This is all to say, regardless of Monolith or Micro-services architecture, the key is experience and discipline and without a good amount of both in your team the outcome is unlikely to be a success over a long enough period of time. This is the curse of a business that lives long enough.
The exact method depends on the language. In C/C++, you'd limit the include paths for each library. In C#, you'd have to compile different assemblies. And so on.
Eventually they tear down any boundary, even those in the build system.
Developer discipline is something that eludes many companies for lack of enough high quality engineers and awareness for the problem in upper management. It's easier to quibble over formatting guidelines.
I suspect that this bandwidth problem exists elsewhere also.
If your developers are writing crap code in a monolith they're going to continue writing crap code in microservices but now you have new problems of deployment, observability, performance, debugging, etc etc.
As an aside I have a sneaking probably ahistorical suspicion microservices hype happened because people realised Ruby (or similar trendy dynamic languages) often ended up being a write only framework and rather than try and recover some codebase sanity people would rather abandon the code entirely and chase the new codebase high.
Anecdotally I witnessed this once. There was this huge ball of mud we had that worked okay-ish. Then the architects decided "hey microservices could solve this", so we started building out microservices that became a distributed ball of mud. Every microservice shared and passed a singular data model across ~30 microservices, which made things interesting when we needed to change that model. Also, we took mediocre developers and asked them to apply rigor they didn't have in developing these services to that they were "prepared" for the failures that happen with distributed systems.
The big upside to management though was that we could farm out parts of the system to different teams on different sides of the planet and have each of them build out the microservices, with each team having different standards as to what is acceptable (what response messages look like, what coding standards should be, ect). All of this was less of a technical problem and more of a managment one, but we felt the pain of it as it was made into a technical problem.
Of course, any dev could just remove the test, and by that tear down that boundary too. But that test being removed is much more obvious to the code reviewer who would not have noticed the dependency being snuck in. The architectural tests can also contain comments that form a high level documentation of the architecture. Unlike "architectural documents" in the form of some up front word document, this type tends to stay up to date.
In the same run one can also validate that there are no incompatible transitive dependencies.
Within assemblies (between namespaces) is much harder unfortunately. That means assemblies have to be made to make module boundaries obvious, even though it’s a bit of an antipattern. There are tools such as nsdepcop that watch namespace dependencies, but it’s brittle and a lot of work.
Correct. That is why the advice not to start with Microservices. Perhaps later may make sense; but not in the beginning.
So, yet another comment saying that modularity requires a distributed system.
If something screams to be moved into a separate project, do it.
Virtually all technical merits are outweighed by that.
It is why the Reverse Conway is preached. Define your desired architecture, and fit your organization to fit it. At that point your code base will reflect the organization of people working on it, and the output will reflect the desired architecture.
Given the rhetoric here I worry that I'm the only person who's genuinely swapped out their implementation. The ol' "N-tier is stupid - you're never going to change the database" comment couldn't be more wrong.
In the end, extracting a microservice from a monolith built this way is just a matter of moving the implementation of the object to a separate application, and making the object a frontend for talking to that application.
The single biggest reason OOP gets a bad reputation is because lots of languages insist on treating OOP as the be-all end-all of code structure (Java and Ruby are particularly bad examples) and insist on trying to shoehorn this sort of logic into tiny dumb pieces of pure data.
recently I am getting more and more thoughtful about "accidental" complexity we add to our solutions in form of dependencies on external libs/module, frameworks such as IoC, log services anyone ;) and on the architectural side microservices etc.
This imho is where serious complications can come in. A single database for all services is a good trade off if you want the nice parts of microservice decoupling but not the headaches of a distributed system. Just perhaps don’t call it “microservices” to avoid having to deal with arguments from purists who want to explain why this is not true microservices, etc.
If you require microservices to enforce decoupling you're "doing it rong"
Microservices are no exception from that rule, and often repeat the same mistake as OOP did with its promise of "reusable code".
Does it sometimes make sense to break some larger services up in smaller ones? Yes.
Does it make sense to "factor out" every minor thing of the implementation into something that can individually be manhandled into a docker container because at some point in the far future, someone may save a few minutes of typing by talking to that service? No.
Why not? Because on the 1:1000 chance that what the service does is actually exactly what that other thing requires, it will probably take more time to implement an interface than it would to simply implement a clone of the functionality.
Conway's law enters into it in a lot of ways. Because of the way the people are organized into tiny isolated teams, the code shares that shape too. There is an event horizon one team/service away, beyond which nobody knows what happens or who wrote the code.
What you get is that the services actually don't do that much, except take a request from one service, translate it to an internal format, perform some trivial operation, then translate it to another external format and pass it on to another service. It's a lot of code, but it's not a lot of logic. Add to that maintaining the test and prod environments as code, and suddenly it looks like this is a lot of work, but you've essentially gotten a hundred people to do work that three people could probably accomplish if it wasn't for this pathological misapplication of an architectural pattern.
My employer has a landscape like that, hundreds of microservices each managed by a different team (some teams manage multiple). However, we have an enterprise architecture group whose job it is to keep an overview and make sure every microservice is meaningful and fulfills a clear role for the organization. Every project presents their architecture to this group as well as a group of peers and this often results in changes that increase cohesion and avoid redundant work. We had a few semi-interconnected monoliths before, and from what I’m told (I joined after the microservice transition) the new way is better.
However, I still wouldn’t recommend microservices to a new team / org starting from scratch. IMHO microservices only make sense when the system grows so vast it cannot be understood in its entirety by a single person.
I wouldn't go that far. The problem is prescribing a stock design solution to every problem without even considering the problem domain or what benefits it will bring.
There are domains where this style of programming is an absolute benefit, even at smaller scales, and it's really nothing new either. A lot of the patterns in microservice design rhyme rather well with what Erlang has done for decades.
The exact reason why I compare microservices to OOP ;-)
That's just because some guru -- Uncle Bob, or even more probably Martin Fowler, I think -- some (rather long by now) time ago wrote a lot of examples along the lines of "methods should be at most five, preferably three, lines long" in Java.
If you look for later examples of the same kind of recommendation you'll probably find they're mostly written in funtional languages nowadays, so you could just as well say "that's why I compare microservices to FP".
Companies are hard. They have histories. Different people with opposing ideas. Dissonance between how the company operates and how it operates on paper. Conflicting personal interests, politics. Stories they tell...
They need some sort of rough-cut ideology, system or whatnot. A way of adjudicating right or wrong. That way arguments can be settled. People can be onboarded more easily. The company's decisions become more legible.
A simple requirement to report on important decisions, the choices available and reasoning behind the choice... stuff like this has consequences. An ethos is more flexible than a ruleset, and provides a way of navigating organisational issues.
People working on their own or in small autonomous groups don't tend to go overboard as easily.
They should not. Either it works correctly, or it doesn't. Even buildings need to be useful, even though people might admire the architecture. No non-technical person will admire your microservice or whatever architecture.
Not starting full in with microservices is a good pattern.