I'm living the Microservices nightmare every day at work! Every single function is a service and you have to run 40 docker containers to get something show up in your browser!
If there is an issue in one those services it takes a long time to figure out which service was it and why it wasn't throwing a 500 error. Even if they throw 500 status codes you still don't get the nice line number and stack trace in a large application gives you.
I think the whole microservices thing is a mistake, I as a developer should not care how this application is running in production. I want to work on a large app and leave the deployment business to someone else. I get it that it's hard to scale a big application but it shouldn't be my concern.
I miss the old .NET days where I could run the app on my machine...
Those days are not gone. Outside of the HN bubble there's still a lot of successful monolithic application development going on. Not every project needs to scale to the size of Twitter.
I believe that a huge amount of SOA in production wouldn't qualify as microservices - especially enterprise SOA.
Heck, I'd bet a lot of enterprise SOA is just a giant crufty monolith with a bunch of its functionality exposed via service endpoints. I suppose that a pretty decent approach - add endpoints to a monolith. As long as the service interfaces remain the same, you can bust it up into as many medium or micro sized services as are necessary on the back end and the users of your services won't notice the difference.
You can also mix'n'match. Have a monolithic entity that does most of the work, and split out the heavy stuff into it's own microservice which can be independently scaled.
(Sub in any other reason to separate a service for 'heavy stuff' - written in a more appropriate language for the function, etc)
I've lived the Monolithic nightmare as well. Dozens and dozens of "utility" functions that got reused between unrelated modules, worrying amounts of copy/paste due to leaky abstractions and a build time of an hour even if you're just working on a small part. Not to mention having to run automated tests across the entire codebase just in case some developer has unexpectedly relied on your class.
And in your case I don't know what you can't get the line number and stack trace from a microservice. Likewise you can get tracing between microservices using something like Zipkin.
Personally I think the "truth" lies in between. Bigger services that cover key functional areas.
> Personally I think the "truth" lies in between. Bigger services that cover key functional areas.
This. What the GP was talking about is something like nano-services. Who really writes a whole service stack for every function?
In addition to functional areas, I also like a service when a single job needs to get done. For example, read message from queue, do some processing, put result somewhere. The beauty of the service here is I can write it once, test it, deploy and it just works. I don't have to worry about it randomly getting broken from code changes in a larger monolith.
I've worked in monoliths, and I've worked with good logically thought out microservices. I'll take the open source microservices model with a good dev team any day of the week.
I can't imagine being in a place where I don't care about how my code runs in production or scales.
You should be able to work on one microservice at a time, run tests on them, and then wrap things up and work on another one.
If you need to work on all of them at the same time that means that you managed to couple them together. In that case they're no longer true microservices, just spaghetti.
Then, with .NET it is much easier to end up with gigantic spaghetti solutions with projects that are all coupled to each other.
Coupling is the problem, learn to control coupling. If everything needs to know about everything, how do you expect to have a team that is capable of maintaining that?
If you have to run them all together to get useful, testable functionality, then all you (they) have done is written a -badly designed- monolith (an overly complex, distributed one). Not a bunch of microservices.
Great way to put it. The very definition of a microservice is that it does ONE thing independently of others. The number of dependencies should be kept as low and to as few points as possible.
I believe that in order to be effective an engineer should be able to run the whole application stack on his/her laptop. Microservices seem attractive because each microservice is easier to setup and hack on than on a huge monolith application, but you don't know that it is working until it is integrated with all the other microservices. Thus I end up having to deploy a dozen applications to ensure that my feature is working.
In theory you could avoid the integration hassle if each service is so well defined and "loosely coupled" that I can be assured it will work without having to test with all the other services, but in practice the architecture is never that well defined.
As the author mentions, it's not that microservices are necessarily bad - just that they're too often used when they're not the appropriate solution for the problem at hand.
I feel as though about 80% of the programming blog posts that breathlessly pontificate about why everything you're doing is wrong and what you should be doing instead would all be better if replaced by a phrase along the lines of "understand the problem you're trying to solve, understand the possible solutions at hand, and pick the one most appropriate for your problem and your circumstances."
Instead, we often see dev teams chasing off after the newest and shiniest options available. And sometimes, the newest and shiniest option is the best one, which is great! But often, it's a far from optimal. However, in an industry where so many companies hire largely on the basis of buzzword compliance, I really don't blame developers at all for choosing the new, hot, and shiny solutions over proven solutions that are better solutions for a wide class of problems.
Same as most approaches. OOP can also be bad, and even something as basic as generic programming (e.g: templates in C++) or metaprogramming can be harmful if you abuse it.
If there was ever an uBlock for the brain, the first thing I'd block would be programming blog posts. Sure, I'll miss a ton of good ones (which I can whitelist!), but they're really not worth the investment.
1. X that is a good idea for solving problem P appears on the tech scene
2. People who have problem P pick up X and say good things about X
3. People, usually junior devs, read blog post about X. They start treating X as solution to every problem they encounter and ofter take it to the extreme.
4. After realizing it's not helping, they blame X and write posts like this.
You can replace X by many things, starting from methodologies like microservices or TDD, ending on things like nosql databases.
A guy with 30 years of experience that has only exposure to imperative programming, and never really learned OOP, networking, functional programming (it's not new: lisp, scheme, haskell, etc. have been around for decades), never really learned about operating systems, never really cared about automation, testing... unless you work on implementing software at a low level, which is very rare these days... guess what, that guy has a lot to learn from a fresher engineer, or engineers that actually cared about learning.
Reasonable judgment is fine. But reasonable judgment can be applied to everything, including who is better suited for making decisions about technological adoption.
Someone who is biased towards "getting job done" and seeing technology adoption as a mere distraction, is simply too biased to be of any help during decision making. When evaluating technology adoption, you need to see the pros and cons (including adoption costs), and evaluate things seriously.
I think it's junior ones who lack understanding of how is OS and networks are operating, as they don't needed most of the time in more popular industries (mobile development, web, etc).
source: by the chance I conducted few interviews for software engineers and devops in web industry - and instead of fizzbuzz I started with asking what's the difference between TCP and UDP. Results will surprise you (also too many people want to exploit current hiring system by just cramming algorithms).
The "intermediate beginner" who latches on whatever is "new" and trendy is my biggest annoyance with the industry. Unfortunately functional programming seems to be what at lot of these people are claiming is the solution to everything now, even if they have little experience building systems with it and it's older than OOP.
Anything that behaves like an object, would be the obvious answer. GUI elements, things in games or simulations. More abstractly, an object's a namespace with persistent properties, so for something that has a persistent, possibly mutable, state, an object is the metaphor you want.
Whether data structures should be objects rather than primitives depends very much on the data.
I think what frustrates me is that I see these people as, for lack of a better word, "posers". I'm no functional expert and I'm just in the beginning of my career, but I'm very skeptical of devs who had little problem with imperative/OOP code suddenly writing Medium articles saying OOP is dead and "functional" is the "next" step. Just makes me uncomfortable, but I'm learning to ignore it.
It's pretty strange to think that OOP cannot be implemented using a functional language. Various Lisps had several OO systems, and there's OCaml, where the "O" stands for "objective".
Objects are merely collections of partially-applied functions (or often procedures): methods are bound to `this` aka `self`, the instance. When you have a language where partial function application (currying) is a norm, you have little trouble implementing this approach.
What made OOP popular, things like good modularity and hiding of implementation details, has little to do with the "inheritance + encapsulation + polymorphism" trio proper. They can be and often are achieved without these.
What makes FP powerful is higher-order functions / function composition, immutable data / pure functions / referential transparency, and isolation of effects. Technically, all of these are easily available in most popular OO languages, and you can reap many benefits of FP writing in C#, Python, or even Java 8. But the standard libraries of most of them actively resist such usage, and the translators do not rely on the functional features heavily for optimization.
Where pure FP has trouble is e.g. large mutable structures with constant small mutations, like editing a bitmap in a paint program. Real-time stuff can also be harder, because the shape of the machine code is much less obvious than when using a lowest-possible-level language like C.
Since the real world is inherently effectful, there will always be a mix of approaches. Pick the right tool for the job; a job of any serious complexity will likely require multiple different tools.
mostly don't make architectural decisions. It's a bit easy to keep on blaming "junior devs". Who keeps on hyping up and marketing useless stuffs ? that's not "junior devs", that's big consultancies that want to sell their crappy services, that's PAAS that want to sell more servers, that's vendors that want to sell more useless stuff. Enough with blaming "junior devs". Junior devs don't lead teams and have very little say when it comes to architectural choices, they don't contract this or that consultancy, they don't decide whether a business will run on Amazon or Azure. Junior devs didn't invent NoSQL, Microservices, "Serveless" and all that bullshit. Well established players sell these services to others, and the people who drink the kool-aid are managers, not junior devs.
Do you really think "junior devs" want to maintain 50 micro servers just because it's trendy? when they could get away with a single one?
Heh, you describe me exactly. I'm "junior"-ish and the second to last place I worked at was microservice, a ton of Docker, and MongoDB. Definitely wished we had a simple monolith with Postgres instead.
Thank you for defending me/us :) Whenever there's bullshit talking at my company I always make a "meh" face and try to bring to the table the negative sides of a solution.
3. People, usually junior devs, read blog post about X. They start treating X as solution to every problem they encounter and ofter take it to the extreme.
Another possibility: senior dev / architect wants to try it on a relatively small project, as a pilot to get a better feel for how it works and where it might be useful. Sure it might cause headaches and extra cost on that one project, but the knowledge gained might be worth it. The resulting blog posts are either public dissemination of that knowledge, or griping by people who either aren't interested in that big picture or who think the cost/benefit call was made incorrectly.
Martin Fowler, as usual, wrote something foundational, deeper et better balanced, years before this article and many other. martinfowler.com/articles/microservices.html
Microservices probably aren't appropriate for companies with fewer than 100 engineers due to the maintenance overhead. For larger companies, however, they're an incredible, team-oriented organizational tool. Microservices let you deploy more frequently, make changes with higher confidence, and scale with less heft and burden. You'll almost invariably need an entire team (or teams) dedicated to the tooling and platform aspect of the SOA that supports the rest of the engineering. Unless you can afford this, do not attempt a microservice architecture.
Microservice architecture is more about business modelling rather than the way how your coding functions.
I think if someone would dislike microservice if they don't understand the following points very well:
1: The business model of the system you are working on.
2: how to design/develop business models – write a bunch of CURD services on top of database doesn't count.
3: define business boundary is far more important than you think, especially you are working on some projects for computerizing real world business operations.
I think the good question would be: say you are Google and accidentally lost your code base (I heard it's a java monolith with around 5000 major intricate dependencies, something unsplittable). How would the Google eng team rewrite Google if they had to? I'm kind of 100% sure they would not go monolithic. They more likely would have services built upon a platform, maybe not nano services, but still services that can be handled by a small team with well defined responsibilities.
The same with Facebook and php or stackoverflow and .NET: a successful start-up doesn't prove it's tech stack is the best, it just proves it can work. Someone who left to build a new ambitious competitor is more likely to choose a better tech stack, e.g. Python for Quora.
I don't work at google but have many friends that do. (so take this with a grain of salt)
But just because they may put all their code and commits in a giant common repository, does not mean that its hosted, scaled, and deployed as a monolith in current day.
From the grapevine, I know that some of their stack is c++, and some is java, as an example.
Also, google losing all their code, made me laugh. Maybe that is a possibility, but seems like they have to have so many backups and failsafes. But I get you said that to illustrate a point :)
Google front end services are no where near monoliths. Why do you think Google invented protobuf? Practically everything we do is RPCs from front ends to backend smaller services.
Yes, it's easy to get burned. Yes, it's often the absolute wrong tool for the job as a young startup. Further, its all too easy to just light cash on fire once you get some real serious traffic hitting all these things that may autoscale individually. We have seen it all before, and we will see it again.
But don't throw the baby out with the bath water. In fact, I think the healthiest approach is to drop the "micro" and think of classic Service Oriented Architecture (SOA).
When you hire your 50th engineer, and you don't have this yet, its not the end of the world, but slowly moving from a monolith to a SOA, is pretty much a very well travelled path, and at a certain scale is how you maintain sanity.
It's like the old saying, the first code you write is never the best. Try the 2nd or 3rd rewrite, and you finally know what you want. So start with a monolith, but don't back yourself into a corner, and move one obvious service out at time.
The main benefits are not even what the article gripes about. It is compartmentalization between teams, and honoring stable internal functionality between teams that it makes the biggest difference. Sure scaling CAN be easier, but it can also be harder. Things like managing connection pooling becomes easier with an SOA. But not to the tune of doing it concurrently across 40 services, that is irresponsible to design without knowing exactly how they are all going to get hit when you finally get insane traffic.
One thing the author said that I agree should be more obvious to people is that somewhere in-between hiring engineer 12 and 50, you will end up running your monolith a bit like an SOA, where even if its the same codebase, you can run in different clusters/pools and service a specific part of the application. This is a wonderful way to get the benefits of SOA, with out the nightmare, furthermore, you will quickly learn which of the parts of code, when your rewriting it the 2nd time, you actually want to divorce from the monolith, and you already have patterns, load balancers, and metrics for what that code should do.
When you hire your 200th engineer, it's easy to get on HN and say SOA is the only way to get the job done. But people need to realize the importance of the journey of getting there. Otherwise your just taking a shotgun to the skateboard your cruising down the road with.
Sanity is the key takeaway. Always optimize for sanity.
Doesn't depend on number of rewrites or number of engineers.
It's an approach to organizing the responsibilities of a code module, and abstracting implementation details such as what technology stack is implemented in, what operating system is running on, etc.
By making services depend on each other through a protocol like REST (or Thrift, gRPC or whatever), you can then replace that module by another one that respects the same interface... and you can reimplement it in whatever technology you want. I can have a module implemented on Elixir running on FreeBSD talking to another one running on OpenBSD implemented on C, talking to another one running on Linux implemented in node.js...
Similar to SOA, but SOA was much broader in scope. You had service composability, discoverability, etc. That's in most cases overkill for most companies.
> By making services depend on each other through a protocol like REST (or Thrift, gRPC or whatever), you can then replace that module by another one that respects the same interface... and you can reimplement it in whatever technology you want. I can have a module implemented on Elixir running on FreeBSD talking to another one running on OpenBSD implemented on C, talking to another one running on Linux implemented in node.js...
I said:
> It is compartmentalization between teams, and honoring stable internal functionality between teams that it makes the biggest difference.
Maybe we agree more than disagree?
You also say:
> Similar to SOA, but SOA was much broader in scope. You had service composability, discoverability, etc. That's in most cases overkill for most companies.
Not sure if the normal way of thinking of micro-services does not include this. If so, that is a new definition to me. To me micro-services mean services which perform less functionality, therefore you need more of them. If you have a platform that has 40 functions-as-a-service running in docker containers, you will need discoverability, for example.
You not necessarily need to roll out a discovery service. You can put services behind a DNS, load balancer... and load balancers can be smart enough to mark a node as down.
Now if you need more granularity than that, there's for instance Zookeeper ephemeral nodes, that get deleted when the client stops emitting heartbeats.
It drives me nuts when people nonchalantly talk about hiring the 12th or 50th with no apparent acknowledgement about how insanely difficult it is to get to that point.
"Given a choice of solutions, pick the least powerful solution capable of solving your problem"
Be it microservices, OOP or whatever... keep this principle in mind.
I would like to propose a new variant on an old maxim: Premature microservice-izing is the root of all evil.
I've worked with an app that had been factored into half a dozen microservices. Each of these needed two EC2 instances for redundancy. They weren't using docker or any other container infrastructure and were treating EC2 instances as pets instead of cattle, which meant that a LOT of effort was spent maintaining the dozen or so instances. Each of these instances was running at < 1% load average! So in this case the microservice architecture was also costing a significant amount of money.
My takeaway: don't build an app from scratch using microservices, especially if all you're splitting apart is web handling which is almost never going to be where your application spends a significant amount of time. Instead, build apps in the straightforward monolithic way, and only split things out into microservices when there is a very clear reason that has been identified from performance monitoring or some other metric gathering.
My takeaway from your example is not to not use microservices, but to get proper operational processes in place. At most I'd stretch to conceding that they should have done that first.
Reliability and speed are features, and often the features that make users stay or go.
If your devops is inadequate, so that you have low visibility into your product's performance, can't degrade gracefully, and have trouble deploying things without significant downtime, investing in proper devops infrastructure and processes may directly help your bottom line.
That stuff is table stakes and relatively easy to deliver in 2016. Compared to achieving product/market fit and getting people to pay for your service, comparatively simple.
Comparatively simple, but without getting at least the basics right you'll be moving comparatively slowly as you end up with time-suck after time-suck to compensate for lacking processes.
As somebody who's actually paid to implement devops practices for many clients at scale, this is nuts. The only performance argument for a service-oriented architecture (and let's be really real, the overhead of HTTP means that microservices rarely have a performance argument at all) is when and only when you have specific requirements that cannot be solved by horizontally scaling a monolithic system. Until you have an actual, metrically verified demonstration of the failure of horizontally scaling your monolithic architecture, sticking with it will in almost every instance outperform your "microservice architecture". Reliability, too, is a joke argument; building a single application (whether or not you internally segment the code with an eye towards later service segmentation) means that you have N instances of every component, rather than N of some, M of others, and so on. And nothing related to deploying your very bloggable microservice should differ from a reasonably developed monolithic system in the first place.
The set of companies that have any sort of reliability or performance-oriented need a "microservice architecture" is three, maybe four orders of magnitude smaller than the set of companies building web applications. They--and you--have been sold on a trend that is actively antithetical to the goals of delivering a product.
Build your application. Use an auto-scaling group and an ELB. You have more important things to do until you reach the scale where you can pay someone (hi) to do them.
If you don't have proper devops processes in place, you'll be spending a progressively larger proportion of your time handling manually what should be automated, and firefighting problems that shouldn't even occur, as your system grows more complex.
A pre-requisite to "spend as much energy as possible on end user features" is to have some sort of foundation to build on that can grow with your platform, or you might as well build on quicksand.
This approach makes a lot of sense. The only other scenario where I think it makes to go microservice is when development progress is impacted by organizational structure. That seems to be a huge reason why corps like amazon broke things down into microservice teams; so productivity could stay high in extremely complex and high headcount environments.
While the architecture of the app you had to deal with is absolutely insane I'm not sure I'd start with monolithic either. Typically I've seen people go monolithic and when they start realizing they need to break things up everything is glued together so tightly they basically need to completely rewrite to get to microservices.
I think there is a middle ground. Especially if you can start writing common libraries for the other components to use.
Tight vs loose coupling is not a new concept. You can write a monolith with bounded-context components which are loosely coupled to each other with an eye towards possibly putting those bounded contexts in different containers in the future.
Unfortunately, this requires an actual architect to think about the design of the monolith ahead of time, and most startups either have trouble recruiting anyone above the junior level, confuse the responsibilities of the project manager (who, in the early days, is typically the CEO/CTO) with the responsibilities of the architect, or simply cannot afford to put an architect on payroll.
What you are referring to is the Monolith First approach: http://martinfowler.com/bliki/MonolithFirst.html. The key is to identify service boundaries early on and keep them relatively clean so that separating them into microservices at a later point is easier.
That is exactly my company's situation. And at least 1/2 energy spent on "infrastructure" instead of end-user features. And the whole thing could easily run monolithic on one small instance.
I'm amazed that your takeaway from that case was about microservices. The problem was obviously outdated operations approaches (not using containers, hand-tuning each server, etc.).
Newer operations approaches may have helped make the state of affairs tractable, but it does not at all argue for that microservice architecture. There was literally no value gained from running a dozen web servers instead of two. And that being the case, the straightforward but "outdated" way of managing one or two servers manually would almost definitely be cheaper than learning and implementing the more modern approach.
This is not a very good example. These people were just doing it wrong. If they had been running something like AppEngine, Heroku, GKE, ECS, etc, they wouldn't have had to worry about maintaining VMs, replicating app instances, proper VM sizing, etc. It's not the microservice architecture that was costing them money, it was their poor technical choices.
Microservices are useful when there's inherent isolation between the data used by each service. If they talk to totally different databases, and you don't have to do atomic updates across service boundaries, why not?
Our DARPA Grand Challenge vehicle in 2005 was all microservices. There were tasks for each sensor, a location task which integrated AHRS, GPS, and map data into "where are we now", a steering task, and a planning task which took in the sensor data, built a local point cloud map, and issued steering commands. There was also an emergency stop task monitoring the radar and able to stop the vehicle if it detected an incipient collision, independent of the mapping systems. The non-hardware tasks could be run separately, or in a simulator, and we could tap in at a boundary and log.
This was all QNX, hard real time. This part of the system worked quite well.
Synchronization was unusual, in that the sensor data all came in at different times depending on the sensor. So we timestamped everything at the millisecond level, and interpolated between sensor frames to get values for all sensors at the same instant. The overall update rate for planning was 100ms, rather slow by modern standards.
Everyone gets so caught up in the terminology "micro" "macro" . The real value in what the microservice movement brings to the table is the push for well defined contracts between components before writing any code. But any good software engineer/architect knows the value of documenting design, so this should not come as as surprise.
When considering only production systems, anybody that has a nightmare microservice deployment that is unruly to upgrade, version, or maintain always had some story about getting to market in record time before hand. Shipping services as quickly as possible to get the feature out the door without considering the downstream impact. (my absolute favorite... "we can solve auth later with a microservice in a container")
I don't care if you have micro, macro, monolithic, or (pick a buzz word) architecture. If its not designed properly, it will always be a nightmare.
In the right circumstances, some microservices make a lot of sense. I was recently working on a .NET project where the need for compiling handlebars templates on the sever came up. This was the perfect use case for a little service. Its purpose was clear and independent from the rest of the application. In this case, we deployed the microservice on AWS Lambda, saving the need to install Node or spin up another server with all the associated care & feeding. Compiling large handlebars is fairly CPU intensive, and we don't have to worry about the load on our app servers. To me, it was an ideal scenario to develop a microservice. It solved a specific problem and didn't bring new ones to the table.
Microservices make sense to me when the task is clearly abstract from the application using it (which allows the service to be used more widely), the nature of the service is such that it has significantly different resource requirements than the rest of the application (allowing it to be scaled independently), and the problem it solves is not trivial.
I would be happy indeed as a developer to walk into a project and have a handful of tools at my disposal for doing things like sending emails, event notifications, logging, encryption key rotation, etc. I just consume the API. It's the reason I appreciate AWS; they provide a load of features (i.e. microservices) that let me focus on domain problems and not things that have been solved a thousand times already. So as a thoughtful developer, deploying something as a microservice can be a prudent contribution that brings a great deal of business value down the road.
But it starts to get out of control. After a bit of success with a legitimate microservice or two you become a hammer and every function starts to look like a nail.
If you can't visualize the microservice you're building ever becoming an AWS product, popular open source project or service offered by some SaaS company, likely it's too domain-centric and should stay in the monolith.
"Better learn balance. Balance is key. Balance good, everything good. Balance bad, better pack up, go home. Understand?" - Mr. Miyagi
Microservices are an application of Conway's Law at scale. If you don't have the scale, don't do microservices, because you can't afford the trade-off.
The original post is completely on point and this thread is full of so full of "No True Scotsman" syndrome.
There's a fundamental mathematical truth that if you have N moving parts in a system, you're looking at possible O(N^2) ways the system can break. Now that doesn't mean that you build everything as one gigantic binary but it certainly means that system architects should be always be looking at reducing the number of moving parts not increasing them. The whole microservices craze is exactly the opposite of that.
I've been using microservices for a particular app since 2011 and it's working quite well. So in my instance, yes, please do. Maybe not for you, but for me. So, change this to: "Microservices - not for me".
I cannot help but wonder if the over-enthusiasm for microservices these past few years is a direct result of the over-enthusiasm for "MVC" these past 10 years or so (since Rails).
MVC outlines very clearly the areas of responsibility for the framework. But many of the popular MVC frameworks take it a step further, declaring that MVC should be the primary dividing lines for your app code and business logic, with little thought given to separation along functional/domain lines. (Or in DHH's words, "The framework is the application!")
So is it any wonder that 10 years later, the industry is freaking out because our nice, clean MVC apps became business logic casseroles? Microservices can help with that, just like MVC can help with the PHP spaghetti code that came before. But none of those things can substitute for basic design principles.
I wonder how things would be different if 10 years ago, instead of app/models, app/views, and app/controllers, Rails had promoted app/auth, app/billing, app/spline_reticulation, etc. Might such a simple thing have pushed us in a better direction, leaving microservices to the 10% of apps that really need them? Or would we all have followed that pattern to illogical extremes as well?
Geez - Am I supposed to really believe all the hyperbole in here? It makes it sound like a move to Microservices is the main reason for a company's failure.
To me, Microservices are a pragmatic solution to build software in an age of complete chaos. Clearly it's not a silver bullet - but what ever has been?
As someone who has to manage a complicated application architecture with lots of pieces, very little time to think, a flat organization of full stack developers, and a bunch of executives who pivot every other week, it's the only scheme that works.
Would I like to go back to the old days when a team of 20 and six months would do what a team of 3 and two weeks does today? Hell yes. But, that's not reality - the world has moved on, and it's not slowing down anytime soon.
Startups these days are often about taking on as much risk as can be tolerated in a race toward dominant market share and exponentially positive revenue. And, Microservices help get us to that goal. It's often not pretty, no doubt. But, it's way easier to manage risk in that kind of environment. With proper monitoring, alerting, and automation - the risks go down a lot.
Some might say that I'm crazy to work in such an environment. Maybe they're right. Generally, this is all completely experimental. But, it's helped me scale to 25 developers and $25M revenue in 8 months. Is it sustainable? Who knows - but, I know that the alternatives certainly wouldn't have worked. In another 8 months, we will either be at $200M in revenue or will have died a horrible death - but we will have died because of the business model, not because of the software.
94 comments
[ 0.24 ms ] story [ 148 ms ] threadIf there is an issue in one those services it takes a long time to figure out which service was it and why it wasn't throwing a 500 error. Even if they throw 500 status codes you still don't get the nice line number and stack trace in a large application gives you.
I think the whole microservices thing is a mistake, I as a developer should not care how this application is running in production. I want to work on a large app and leave the deployment business to someone else. I get it that it's hard to scale a big application but it shouldn't be my concern.
I miss the old .NET days where I could run the app on my machine...
People seem to forget that microservices isn't new. It's just SOA all over again.
Heck, I'd bet a lot of enterprise SOA is just a giant crufty monolith with a bunch of its functionality exposed via service endpoints. I suppose that a pretty decent approach - add endpoints to a monolith. As long as the service interfaces remain the same, you can bust it up into as many medium or micro sized services as are necessary on the back end and the users of your services won't notice the difference.
(Sub in any other reason to separate a service for 'heavy stuff' - written in a more appropriate language for the function, etc)
And in your case I don't know what you can't get the line number and stack trace from a microservice. Likewise you can get tracing between microservices using something like Zipkin.
Personally I think the "truth" lies in between. Bigger services that cover key functional areas.
This. What the GP was talking about is something like nano-services. Who really writes a whole service stack for every function?
In addition to functional areas, I also like a service when a single job needs to get done. For example, read message from queue, do some processing, put result somewhere. The beauty of the service here is I can write it once, test it, deploy and it just works. I don't have to worry about it randomly getting broken from code changes in a larger monolith.
I can't imagine being in a place where I don't care about how my code runs in production or scales.
Whether it's monolith or whatnot, what you really want is the "good dev team" part.
Then, with .NET it is much easier to end up with gigantic spaghetti solutions with projects that are all coupled to each other.
Coupling is the problem, learn to control coupling. If everything needs to know about everything, how do you expect to have a team that is capable of maintaining that?
- distributed transactions are hard to do well especially in write scenarios
- microservices often solve team issues better than technical issues
In theory you could avoid the integration hassle if each service is so well defined and "loosely coupled" that I can be assured it will work without having to test with all the other services, but in practice the architecture is never that well defined.
I feel as though about 80% of the programming blog posts that breathlessly pontificate about why everything you're doing is wrong and what you should be doing instead would all be better if replaced by a phrase along the lines of "understand the problem you're trying to solve, understand the possible solutions at hand, and pick the one most appropriate for your problem and your circumstances."
Instead, we often see dev teams chasing off after the newest and shiniest options available. And sometimes, the newest and shiniest option is the best one, which is great! But often, it's a far from optimal. However, in an industry where so many companies hire largely on the basis of buzzword compliance, I really don't blame developers at all for choosing the new, hot, and shiny solutions over proven solutions that are better solutions for a wide class of problems.
2. People who have problem P pick up X and say good things about X
3. People, usually junior devs, read blog post about X. They start treating X as solution to every problem they encounter and ofter take it to the extreme.
4. After realizing it's not helping, they blame X and write posts like this.
You can replace X by many things, starting from methodologies like microservices or TDD, ending on things like nosql databases.
A guy with 30 years of experience that has only exposure to imperative programming, and never really learned OOP, networking, functional programming (it's not new: lisp, scheme, haskell, etc. have been around for decades), never really learned about operating systems, never really cared about automation, testing... unless you work on implementing software at a low level, which is very rare these days... guess what, that guy has a lot to learn from a fresher engineer, or engineers that actually cared about learning.
Since you brought up functional programming, it actually fits the same framework.
In that case P includes problems with a "pure" domain (e.g. compilers) or problems that require lots of multi-threading.
It does not work well for problems that are fundamentally state-full or sequential. It also does not work in "messy" domains - e.g. data mungling.
Someone who is biased towards "getting job done" and seeing technology adoption as a mere distraction, is simply too biased to be of any help during decision making. When evaluating technology adoption, you need to see the pros and cons (including adoption costs), and evaluate things seriously.
I think it's junior ones who lack understanding of how is OS and networks are operating, as they don't needed most of the time in more popular industries (mobile development, web, etc).
source: by the chance I conducted few interviews for software engineers and devops in web industry - and instead of fizzbuzz I started with asking what's the difference between TCP and UDP. Results will surprise you (also too many people want to exploit current hiring system by just cramming algorithms).
I can only think of a couple examples: data structures and some APIs.
Whether data structures should be objects rather than primitives depends very much on the data.
I love both FP and OOP and especially love using them together. I honestly don't get these arguments.
Objects are merely collections of partially-applied functions (or often procedures): methods are bound to `this` aka `self`, the instance. When you have a language where partial function application (currying) is a norm, you have little trouble implementing this approach.
What made OOP popular, things like good modularity and hiding of implementation details, has little to do with the "inheritance + encapsulation + polymorphism" trio proper. They can be and often are achieved without these.
What makes FP powerful is higher-order functions / function composition, immutable data / pure functions / referential transparency, and isolation of effects. Technically, all of these are easily available in most popular OO languages, and you can reap many benefits of FP writing in C#, Python, or even Java 8. But the standard libraries of most of them actively resist such usage, and the translators do not rely on the functional features heavily for optimization.
Where pure FP has trouble is e.g. large mutable structures with constant small mutations, like editing a bitmap in a paint program. Real-time stuff can also be harder, because the shape of the machine code is much less obvious than when using a lowest-possible-level language like C.
Since the real world is inherently effectful, there will always be a mix of approaches. Pick the right tool for the job; a job of any serious complexity will likely require multiple different tools.
> junior devs
mostly don't make architectural decisions. It's a bit easy to keep on blaming "junior devs". Who keeps on hyping up and marketing useless stuffs ? that's not "junior devs", that's big consultancies that want to sell their crappy services, that's PAAS that want to sell more servers, that's vendors that want to sell more useless stuff. Enough with blaming "junior devs". Junior devs don't lead teams and have very little say when it comes to architectural choices, they don't contract this or that consultancy, they don't decide whether a business will run on Amazon or Azure. Junior devs didn't invent NoSQL, Microservices, "Serveless" and all that bullshit. Well established players sell these services to others, and the people who drink the kool-aid are managers, not junior devs.
Do you really think "junior devs" want to maintain 50 micro servers just because it's trendy? when they could get away with a single one?
Another possibility: senior dev / architect wants to try it on a relatively small project, as a pilot to get a better feel for how it works and where it might be useful. Sure it might cause headaches and extra cost on that one project, but the knowledge gained might be worth it. The resulting blog posts are either public dissemination of that knowledge, or griping by people who either aren't interested in that big picture or who think the cost/benefit call was made incorrectly.
I think if someone would dislike microservice if they don't understand the following points very well: 1: The business model of the system you are working on. 2: how to design/develop business models – write a bunch of CURD services on top of database doesn't count. 3: define business boundary is far more important than you think, especially you are working on some projects for computerizing real world business operations.
Strongly recommend reading Domain-Driven Design book.
The same with Facebook and php or stackoverflow and .NET: a successful start-up doesn't prove it's tech stack is the best, it just proves it can work. Someone who left to build a new ambitious competitor is more likely to choose a better tech stack, e.g. Python for Quora.
But just because they may put all their code and commits in a giant common repository, does not mean that its hosted, scaled, and deployed as a monolith in current day.
From the grapevine, I know that some of their stack is c++, and some is java, as an example.
Also, google losing all their code, made me laugh. Maybe that is a possibility, but seems like they have to have so many backups and failsafes. But I get you said that to illustrate a point :)
Yes, it's easy to get burned. Yes, it's often the absolute wrong tool for the job as a young startup. Further, its all too easy to just light cash on fire once you get some real serious traffic hitting all these things that may autoscale individually. We have seen it all before, and we will see it again.
But don't throw the baby out with the bath water. In fact, I think the healthiest approach is to drop the "micro" and think of classic Service Oriented Architecture (SOA).
When you hire your 50th engineer, and you don't have this yet, its not the end of the world, but slowly moving from a monolith to a SOA, is pretty much a very well travelled path, and at a certain scale is how you maintain sanity.
It's like the old saying, the first code you write is never the best. Try the 2nd or 3rd rewrite, and you finally know what you want. So start with a monolith, but don't back yourself into a corner, and move one obvious service out at time.
The main benefits are not even what the article gripes about. It is compartmentalization between teams, and honoring stable internal functionality between teams that it makes the biggest difference. Sure scaling CAN be easier, but it can also be harder. Things like managing connection pooling becomes easier with an SOA. But not to the tune of doing it concurrently across 40 services, that is irresponsible to design without knowing exactly how they are all going to get hit when you finally get insane traffic.
One thing the author said that I agree should be more obvious to people is that somewhere in-between hiring engineer 12 and 50, you will end up running your monolith a bit like an SOA, where even if its the same codebase, you can run in different clusters/pools and service a specific part of the application. This is a wonderful way to get the benefits of SOA, with out the nightmare, furthermore, you will quickly learn which of the parts of code, when your rewriting it the 2nd time, you actually want to divorce from the monolith, and you already have patterns, load balancers, and metrics for what that code should do.
When you hire your 200th engineer, it's easy to get on HN and say SOA is the only way to get the job done. But people need to realize the importance of the journey of getting there. Otherwise your just taking a shotgun to the skateboard your cruising down the road with.
Sanity is the key takeaway. Always optimize for sanity.
It's an approach to organizing the responsibilities of a code module, and abstracting implementation details such as what technology stack is implemented in, what operating system is running on, etc.
By making services depend on each other through a protocol like REST (or Thrift, gRPC or whatever), you can then replace that module by another one that respects the same interface... and you can reimplement it in whatever technology you want. I can have a module implemented on Elixir running on FreeBSD talking to another one running on OpenBSD implemented on C, talking to another one running on Linux implemented in node.js...
Similar to SOA, but SOA was much broader in scope. You had service composability, discoverability, etc. That's in most cases overkill for most companies.
> By making services depend on each other through a protocol like REST (or Thrift, gRPC or whatever), you can then replace that module by another one that respects the same interface... and you can reimplement it in whatever technology you want. I can have a module implemented on Elixir running on FreeBSD talking to another one running on OpenBSD implemented on C, talking to another one running on Linux implemented in node.js...
I said:
> It is compartmentalization between teams, and honoring stable internal functionality between teams that it makes the biggest difference.
Maybe we agree more than disagree?
You also say: > Similar to SOA, but SOA was much broader in scope. You had service composability, discoverability, etc. That's in most cases overkill for most companies.
Not sure if the normal way of thinking of micro-services does not include this. If so, that is a new definition to me. To me micro-services mean services which perform less functionality, therefore you need more of them. If you have a platform that has 40 functions-as-a-service running in docker containers, you will need discoverability, for example.
Now if you need more granularity than that, there's for instance Zookeeper ephemeral nodes, that get deleted when the client stops emitting heartbeats.
"Given a choice of solutions, pick the least powerful solution capable of solving your problem" Be it microservices, OOP or whatever... keep this principle in mind.
I've worked with an app that had been factored into half a dozen microservices. Each of these needed two EC2 instances for redundancy. They weren't using docker or any other container infrastructure and were treating EC2 instances as pets instead of cattle, which meant that a LOT of effort was spent maintaining the dozen or so instances. Each of these instances was running at < 1% load average! So in this case the microservice architecture was also costing a significant amount of money.
My takeaway: don't build an app from scratch using microservices, especially if all you're splitting apart is web handling which is almost never going to be where your application spends a significant amount of time. Instead, build apps in the straightforward monolithic way, and only split things out into microservices when there is a very clear reason that has been identified from performance monitoring or some other metric gathering.
If your devops is inadequate, so that you have low visibility into your product's performance, can't degrade gracefully, and have trouble deploying things without significant downtime, investing in proper devops infrastructure and processes may directly help your bottom line.
The set of companies that have any sort of reliability or performance-oriented need a "microservice architecture" is three, maybe four orders of magnitude smaller than the set of companies building web applications. They--and you--have been sold on a trend that is actively antithetical to the goals of delivering a product.
Build your application. Use an auto-scaling group and an ELB. You have more important things to do until you reach the scale where you can pay someone (hi) to do them.
I think most of your points are valid (wherever my experience is enough to judge).
A pre-requisite to "spend as much energy as possible on end user features" is to have some sort of foundation to build on that can grow with your platform, or you might as well build on quicksand.
I think there is a middle ground. Especially if you can start writing common libraries for the other components to use.
Unfortunately, this requires an actual architect to think about the design of the monolith ahead of time, and most startups either have trouble recruiting anyone above the junior level, confuse the responsibilities of the project manager (who, in the early days, is typically the CEO/CTO) with the responsibilities of the architect, or simply cannot afford to put an architect on payroll.
There is sooo much truth to this statement. Never be afraid to put down an instance, and if so your doing it wrong!
Doesn't EC2 already have redundancy built in?
Our DARPA Grand Challenge vehicle in 2005 was all microservices. There were tasks for each sensor, a location task which integrated AHRS, GPS, and map data into "where are we now", a steering task, and a planning task which took in the sensor data, built a local point cloud map, and issued steering commands. There was also an emergency stop task monitoring the radar and able to stop the vehicle if it detected an incipient collision, independent of the mapping systems. The non-hardware tasks could be run separately, or in a simulator, and we could tap in at a boundary and log.
This was all QNX, hard real time. This part of the system worked quite well.
Synchronization was unusual, in that the sensor data all came in at different times depending on the sensor. So we timestamped everything at the millisecond level, and interpolated between sensor frames to get values for all sensors at the same instant. The overall update rate for planning was 100ms, rather slow by modern standards.
When considering only production systems, anybody that has a nightmare microservice deployment that is unruly to upgrade, version, or maintain always had some story about getting to market in record time before hand. Shipping services as quickly as possible to get the feature out the door without considering the downstream impact. (my absolute favorite... "we can solve auth later with a microservice in a container")
I don't care if you have micro, macro, monolithic, or (pick a buzz word) architecture. If its not designed properly, it will always be a nightmare.
Microservices make sense to me when the task is clearly abstract from the application using it (which allows the service to be used more widely), the nature of the service is such that it has significantly different resource requirements than the rest of the application (allowing it to be scaled independently), and the problem it solves is not trivial.
I would be happy indeed as a developer to walk into a project and have a handful of tools at my disposal for doing things like sending emails, event notifications, logging, encryption key rotation, etc. I just consume the API. It's the reason I appreciate AWS; they provide a load of features (i.e. microservices) that let me focus on domain problems and not things that have been solved a thousand times already. So as a thoughtful developer, deploying something as a microservice can be a prudent contribution that brings a great deal of business value down the road.
But it starts to get out of control. After a bit of success with a legitimate microservice or two you become a hammer and every function starts to look like a nail.
If you can't visualize the microservice you're building ever becoming an AWS product, popular open source project or service offered by some SaaS company, likely it's too domain-centric and should stay in the monolith.
"Better learn balance. Balance is key. Balance good, everything good. Balance bad, better pack up, go home. Understand?" - Mr. Miyagi
There's a fundamental mathematical truth that if you have N moving parts in a system, you're looking at possible O(N^2) ways the system can break. Now that doesn't mean that you build everything as one gigantic binary but it certainly means that system architects should be always be looking at reducing the number of moving parts not increasing them. The whole microservices craze is exactly the opposite of that.
MVC outlines very clearly the areas of responsibility for the framework. But many of the popular MVC frameworks take it a step further, declaring that MVC should be the primary dividing lines for your app code and business logic, with little thought given to separation along functional/domain lines. (Or in DHH's words, "The framework is the application!")
So is it any wonder that 10 years later, the industry is freaking out because our nice, clean MVC apps became business logic casseroles? Microservices can help with that, just like MVC can help with the PHP spaghetti code that came before. But none of those things can substitute for basic design principles.
I wonder how things would be different if 10 years ago, instead of app/models, app/views, and app/controllers, Rails had promoted app/auth, app/billing, app/spline_reticulation, etc. Might such a simple thing have pushed us in a better direction, leaving microservices to the 10% of apps that really need them? Or would we all have followed that pattern to illogical extremes as well?
To me, Microservices are a pragmatic solution to build software in an age of complete chaos. Clearly it's not a silver bullet - but what ever has been?
As someone who has to manage a complicated application architecture with lots of pieces, very little time to think, a flat organization of full stack developers, and a bunch of executives who pivot every other week, it's the only scheme that works.
Would I like to go back to the old days when a team of 20 and six months would do what a team of 3 and two weeks does today? Hell yes. But, that's not reality - the world has moved on, and it's not slowing down anytime soon.
Startups these days are often about taking on as much risk as can be tolerated in a race toward dominant market share and exponentially positive revenue. And, Microservices help get us to that goal. It's often not pretty, no doubt. But, it's way easier to manage risk in that kind of environment. With proper monitoring, alerting, and automation - the risks go down a lot.
Some might say that I'm crazy to work in such an environment. Maybe they're right. Generally, this is all completely experimental. But, it's helped me scale to 25 developers and $25M revenue in 8 months. Is it sustainable? Who knows - but, I know that the alternatives certainly wouldn't have worked. In another 8 months, we will either be at $200M in revenue or will have died a horrible death - but we will have died because of the business model, not because of the software.
This is yet another race to the bottom, which as programmers , we should advocate against whenever we can.