I understand the need for such a thing, but the wild amount of indirection that is needed in microservices world is astounding.
Hmm, that request didn’t make it through! I wonder if it was dropped by the database, the business logic, the spring http-handling framework, the retry annotation, nginx reverse proxy, the authentication sidecar, the “ambassador service”, the SQS queue, the internal VPN, or cloudflare?
Infrastructure like this is overkill for the vast majority of cases. When you do need something like this, you're probably using a service mesh that also provides introspection and observability into the pipeline to be able to help debug and diagnose issues.
This type of infra only makes sense when you have Scaling People Problems to solve using technical solutions. Say you're MSFT and you need to let thousands of people of various skillset and skill levels, ship locally useful software at decent velocity, but while preserving important cross-cutting stuff like TLS, telemetry, wtv. Or when dealing with legacy software.
Rule of thumb: if you cannot staff a team (even a small one) for a service - it cannot be a standalone service. This rule is sometimes a bit too aggressive, but only a bit - for example it doesn't apply to an off-the-shelf database.
i don't think you can really say this is ever necessary. or at least, not that microservices make it necessary.
this is a business thing. you need this when you're hiring a contractor to add some functionality, and you don't want them to touch the service behind the ambassador. probably because you've hired a different contractor to build the original service, nobody at your company knows how it works anymore, and making the new contractor figure it out is out of budget.
I don't see much difference with the sidecar pattern. You have something standard besides your application code that handles, requests, retries, circuit breakers and security.
> I understand the need for such a thing, but the wild amount of indirection that is needed in microservices world is astounding.
What does this have to do with microservices?
I'm trying to debug my monolithic service:
Was it the database? My own business logic? The HTTP framework? The reverse proxy we run to terminate TLS? Was it an authentication or authorization issue? The network appliance in front of all of this? Did the queue fail to persist it at a layer above this? Was it the hosting provider?
I can't help but think that, when past teams programmed mostly in assembly, a "Function" could be explained as a pattern.
Having such "design patterns" seem like a side effect of our current tools' limitations. Maybe new technologies like Wing[0] (no affiliation) will help pushing us to the next generation of tools.
Absolutely. Take something like the Builder pattern in Java for handling large numbers of possibly-optional arguments - in Python rarely-if-ever used due to named keyword arguments.
i like this observation - it's not uncommon for patterns to turn into language primitives when they end up becoming a very common thing in your language.
this is reflected both in wing as a language but also in the fact that wing's preflight phase in a sense is a programming model for turning infrastructure patterns into composable components.
> An ambassador service can be thought of as an out-of-process proxy that is co-located with the client.
> This pattern can be useful for offloading common client connectivity tasks such as monitoring, logging, routing, security (such as TLS), and resiliency patterns in a language agnostic way. It is often used with legacy applications, or other applications that are difficult to modify, in order to extend their networking capabilities. It can also enable a specialized team to implement those features.
Not surprised this is a Microsoft page, given their legacy of long lifetime support for their software products.
It’s not for microservices, but rather for software maintenance of systems that other vendors would consider past EOL.
If anything, an obfuscated microservice-based application is easier to understand than a monolithic version: network data transfer is easier for observers to understand than memory modification.
This could be argued, but obfuscated apps are a land of their own.
(You could also argue that obfuscated monolithic programs are be easier to reverse engineer, breakpoint, replay, emulate, time-travel-debug, trace, etc because you can completely control them in your test bench and aren't then working against a hostile distributed system)
It's similar to a thick client, what Google does by eschewing language agnosticism. It's a reasonable approach, really: thin clients of course work, and you have to provide them across popular languages, making them thicker adds value.
Speaking of this has anyone had much experience with Dapr (https://dapr.io/) before?
I always thought this was a particularly interesting approach from Microsoft where they use this pattern to essentially take the complexity of micro services and instead try and keep it as simple as a normal .NET application but (and I think this is the clever part) in both a vendor and language neutral way.
But all of a sudden it means you can start removing all kinds of cruft and random SDKs from your codebase and push almost all of your interactions with the outside world into something like this .
I saw that Dapr was a side-car instead of an SDK, so I've avoided it for now. For comparison, most other microservices frameworks such as Service Fabric or Azure Functions generally ship as an in-process SDK.
The problem with having Dapr being a separate thing that my code has to communicate with over the network is that this basically triples the number of hops in an environment that already has too many network hops.
As a former game developer used to replacing divisions by multiplications with a reciprocal to save precious clocks, the design of Dapr makes me... itchy.
The difference between Game (engine) dev and modern microservice development is pretty staggering. It actually cuts both ways. As a webdev when I dabbled with some OpenGL programming it actually felt amazing how much stuff you can cram into 1/60th of a second when it's all in-process or at worse has to be shuffled between CPU and GPU. When you are used to anything useful taking 10-100 seconds due to network hops and/or having to access a RDBMS it's very reassuring to know that actually, computers are really fast.
Microservices are surprisingly common in game server platforms.
Also, I would argue that latency almost always matters.
Those microseconds add up faster than you think. You have to factor in things like TCP slow start, congestion issues, buffering, the extra CPU spent encoding/decoding, etc…
Not to mention that it rapidly becomes impossible or impractical to do things like process requests in a streaming fashion, so for large requests you can’t overlap the stages of a pipeline. You can easily bloat out a “1 second task” into minutes without ever knowing where you went wrong.
“We’re following best practices!” you say.
“Oh, is that why I’ve been waiting ten seconds for the little spinner while the site is processing a 1 KB transaction using 5 GHz CPUs?”
> Microservices are surprisingly common in game server platforms.
Yes, but you wouldn't use dapr for game dev.
> Also, I would argue that latency almost always matters.
Strongly disagree there. For every interesting business out there, there's 99 boring ones for whom sub-second added latency absolutely does not matter (not that dapr would add that much latency).
I understand what this pattern does, but it makes me wonder how useful it is in practice. I am focusing specifically on the retry and circuit breaker functionality.
If the call is done in a synchronous context (for example original client executes a REST call), you are adding latency by inserting an extra service in between. When the remote service is not immediately available or responsive or fails, and you do a retry, in the end your call might take too long and the caller might cancel the request. I have seen this behavior in practice and it makes me wonder how useful this implementation is in a synchronous context. You add complexity (on the infra level) and you add latency in the happy path. When a retry is needed, most of the time
(in my experience) the call to the remote service does not succeed in time, and the original call still fails.
If the call is done in an asynchronous context (for example the original client picks up a message from a queue and processes it), you are adding latency and complexity by inserting an extra service and extra logic. However, when the remote service is not immediately available, you can just let the processing of the message fail. The queue or bus should contain retry logic that could be finetuned based on the type of error you get. So, in that case, you should already have a retry mechanism out of the box, then is the added latency and complexity really worth it?
I understand about the circuit breaker, and I understand that might be useful, because it could prevent the remote service being overloaded with requests (well, at least if every caller to the remote service implements circuit breaker... but then, the ambassador service would better be placed on the side of the remote service and every client should be forced to pass through it, and then it might just be implemented inside the remote service instead of in a separate service adding extra latency/complexity/... basically, the remote service should protect itself against this).
It's pretty useful when you have some distribution layer (i.e. some pubsub system)
Consider 10-15 applications running on a host, and all of them are listening to data being distributed by another service. Instead of all of them opening a connection to that service, instead they would all be connected to this sidecar, and the sidecar would merge the distribution of data (and subscriptions) to the pubsub system
You seem to be quite focused on latency. Latency is not necessarily something you care about. For coarse grained logistics information, for example, it doesn't really matter if the push notification arrives a few tens or hundreds of milliseconds (or even seconds) later.
So if you don't like the latency aspects I am inclined to ask: "What do you need the low latency for?".
The explanation on the page itself states the following:
Use this pattern when you:
* Need to build a common set of client connectivity features for multiple languages or frameworks.
* Need to offload cross-cutting client connectivity concerns to infrastructure developers or other more specialized teams.
* Need to support cloud or cluster connectivity requirements in a legacy application or an application that is difficult to modify.
This pattern may not be suitable:
* When network request latency is critical.
* When client connectivity features are consumed by a single language.
* When connectivity features cannot be generalized and require deeper integration with the client application.
If you want to question the usefulness of the pattern, it is best to argue against the use-cases in which it is recommended to be used instead of using a scenario the pattern _explicitly_ states it is not meant for (i.e.: low latency).
Now you also had some thoughts on complexity. Regarding what you said: (retries, resiliency, extra logic) it will have to live somewhere. You don't add meaningless complexity for the sake of it after all.
How are you going to add another retry policy to a blob you have no control over otherwise?
Shifting the burden of networking is also an explicit option listed in the "suitable for" section. You can decide where complexity lives after all. If accidental complexity is your issue I am inclined to ask where you see it here in general. Both the ambassador pattern and your proposed alternatives add overhead in terms of complexity somewhere. I'm struggling to see a clear favorite here.
Explicitly regarding your last statement that "[...] the remote service should protect itself against this).":
Retries and Monitoring are things the remote service can't do by itself by definition. Even load balancing/shedding might not be solvable by it depending on the situation. Notice that circuit breaking is not the only thing the ambassador is used for. Network related configuration updates (see section "Context and problem") are something that might not be done by the remote service either.
What I had in mind, is a webapp that does backend api calls. That is a synchronous call, in the sense that often the user is waiting for it since it is often the result of an interaction with the website. I did not consider that a low latency requirement. Still, when a backend service (accessed either directly or indirectly) is not available or has problems, and there is a retry mechanism, this quickly runs into seconds.
> You don't add meaningless complexity for the sake of it after all.
> You can decide where complexity lives after all.
Good points and something I should think about when designing systems.
You have good and interesting points, and it is true that I am very wary about introducing extra latency in the context of an http api that is used by a webapp. With the infra that is available today, it is possible to build snappy webapps, but my feeling is that you have to be wary about introducing extra "hops" in the execution of one http call, even though that is not strictly a "low latency" requirement.
When I build some kind of API, I would usually also provide a library for the clients. This library would to an extent perform the function of "ambassador". It captures my intended logic to handle certain situations that are not described by purely functional part of API contract.
And yes, this library usually also covers things like retries and handling errors and outages.
Maintaining an entire service to achieve the same seems like really expensive way to achieve this...
So when I "sideload" HAProxy with my application, handling (incoming!) and outgoing http(s) and tcp connections, balanving, failover, logging and what not, that is cool now and I get to call it Ambassador Sandwich Pattern.
I want to try and avoid snark etc. here but, good god like 99% of microservices this should be a library. Honestly I feel like microservices has just lost the plot entirely.
Honestly, I only kow this stuff to be a library. I can see some advantage in offloading this to a separate process, so you can update both processes separately, don't trample over the other's heap, etc. (Especially if the original executable can't be modified, as seems to be their rationale)
But it should obviously run in the same container/pod/etc.
How would you build and keep in sync the same library with the same functionality in 10 different languages? Every new feature would have to be implemented 10 times. Sounds quite foolish
I think what happens outside of microservices shops is that "the Java people" write their own library, "the Python people" write their own, etc. Is that a little bonkers? No more bonkers than wrapping services with other services to get them to behave reasonably.
This is a big thing that happens w/ microservices: your system is implemented as a Tower of Babel (it's oddly touted as a feature). In that scenario, you no longer have good choices, you're constantly reimplementing things with slight differences in multiple languages, the maintenance overhead and defect rate is sky high, etc. The big bad here is having the "10 different languages" problem in the 1st place.
I don't think this is really a truism, but maybe! But like, even Google says, "you can choose any language as long as it's C++, Go, Java, or mayyyyyybe Python". That's pretty manageable, and probably the worst problem you'd ever reasonably have. I think most people should probably be restricting themselves to Python/Go/Node, which again is very manageable--certainly better than 10 languages.
> Resilient cloud-based applications require features such as circuit breaking, routing, metering and monitoring, and the ability to make network-related configuration updates.
I understand that at different points, you'll frequently run into situations where you need one or several of those, but the way this sentence is phrased, it sounds more like chastizing people for not following the latest fad (the components of which they are conveniently selling too).
"what, your deployment doesn't have advaned metering and circuitbreaking abilities and I have to restart the service to make config changes? get back to me when you have a real deployment..."
50 comments
[ 5.2 ms ] story [ 69.5 ms ] threadHmm, that request didn’t make it through! I wonder if it was dropped by the database, the business logic, the spring http-handling framework, the retry annotation, nginx reverse proxy, the authentication sidecar, the “ambassador service”, the SQS queue, the internal VPN, or cloudflare?
Just writing it out makes my brain hurt.
But otherwise it's not required.
Or maybe scale doesn't matter. Even if you replace thousands with tens you still often have a huge variety in skillset and focus.
this is a business thing. you need this when you're hiring a contractor to add some functionality, and you don't want them to touch the service behind the ambassador. probably because you've hired a different contractor to build the original service, nobody at your company knows how it works anymore, and making the new contractor figure it out is out of budget.
sidecars bad
What does this have to do with microservices?
I'm trying to debug my monolithic service:
Was it the database? My own business logic? The HTTP framework? The reverse proxy we run to terminate TLS? Was it an authentication or authorization issue? The network appliance in front of all of this? Did the queue fail to persist it at a layer above this? Was it the hosting provider?
You outlined any distributed service.
Having such "design patterns" seem like a side effect of our current tools' limitations. Maybe new technologies like Wing[0] (no affiliation) will help pushing us to the next generation of tools.
[0] https://www.winglang.io/
Actually it was an entire new paradigm.
Procedural programming!
i like this observation - it's not uncommon for patterns to turn into language primitives when they end up becoming a very common thing in your language.
this is reflected both in wing as a language but also in the fact that wing's preflight phase in a sense is a programming model for turning infrastructure patterns into composable components.
> This pattern can be useful for offloading common client connectivity tasks such as monitoring, logging, routing, security (such as TLS), and resiliency patterns in a language agnostic way. It is often used with legacy applications, or other applications that are difficult to modify, in order to extend their networking capabilities. It can also enable a specialized team to implement those features.
Not surprised this is a Microsoft page, given their legacy of long lifetime support for their software products.
It’s not for microservices, but rather for software maintenance of systems that other vendors would consider past EOL.
(You could also argue that obfuscated monolithic programs are be easier to reverse engineer, breakpoint, replay, emulate, time-travel-debug, trace, etc because you can completely control them in your test bench and aren't then working against a hostile distributed system)
https://www.getambassador.io/learn/envoy-proxy
20 something years ago "mobile objects" and "agent systems" were a thing and one of the practical incarnations was Jini: https://jan.newmarch.name/java/jini/tutorial/Jini.html
I always thought this was a particularly interesting approach from Microsoft where they use this pattern to essentially take the complexity of micro services and instead try and keep it as simple as a normal .NET application but (and I think this is the clever part) in both a vendor and language neutral way.
But all of a sudden it means you can start removing all kinds of cruft and random SDKs from your codebase and push almost all of your interactions with the outside world into something like this .
The problem with having Dapr being a separate thing that my code has to communicate with over the network is that this basically triples the number of hops in an environment that already has too many network hops.
As a former game developer used to replacing divisions by multiplications with a reciprocal to save precious clocks, the design of Dapr makes me... itchy.
Also, I would argue that latency almost always matters.
Those microseconds add up faster than you think. You have to factor in things like TCP slow start, congestion issues, buffering, the extra CPU spent encoding/decoding, etc…
Not to mention that it rapidly becomes impossible or impractical to do things like process requests in a streaming fashion, so for large requests you can’t overlap the stages of a pipeline. You can easily bloat out a “1 second task” into minutes without ever knowing where you went wrong.
“We’re following best practices!” you say.
“Oh, is that why I’ve been waiting ten seconds for the little spinner while the site is processing a 1 KB transaction using 5 GHz CPUs?”
Yes, but you wouldn't use dapr for game dev.
> Also, I would argue that latency almost always matters.
Strongly disagree there. For every interesting business out there, there's 99 boring ones for whom sub-second added latency absolutely does not matter (not that dapr would add that much latency).
If the call is done in a synchronous context (for example original client executes a REST call), you are adding latency by inserting an extra service in between. When the remote service is not immediately available or responsive or fails, and you do a retry, in the end your call might take too long and the caller might cancel the request. I have seen this behavior in practice and it makes me wonder how useful this implementation is in a synchronous context. You add complexity (on the infra level) and you add latency in the happy path. When a retry is needed, most of the time (in my experience) the call to the remote service does not succeed in time, and the original call still fails.
If the call is done in an asynchronous context (for example the original client picks up a message from a queue and processes it), you are adding latency and complexity by inserting an extra service and extra logic. However, when the remote service is not immediately available, you can just let the processing of the message fail. The queue or bus should contain retry logic that could be finetuned based on the type of error you get. So, in that case, you should already have a retry mechanism out of the box, then is the added latency and complexity really worth it?
I understand about the circuit breaker, and I understand that might be useful, because it could prevent the remote service being overloaded with requests (well, at least if every caller to the remote service implements circuit breaker... but then, the ambassador service would better be placed on the side of the remote service and every client should be forced to pass through it, and then it might just be implemented inside the remote service instead of in a separate service adding extra latency/complexity/... basically, the remote service should protect itself against this).
Thoughts? Does it make sense what I am thinking?
Consider 10-15 applications running on a host, and all of them are listening to data being distributed by another service. Instead of all of them opening a connection to that service, instead they would all be connected to this sidecar, and the sidecar would merge the distribution of data (and subscriptions) to the pubsub system
Now you also had some thoughts on complexity. Regarding what you said: (retries, resiliency, extra logic) it will have to live somewhere. You don't add meaningless complexity for the sake of it after all. How are you going to add another retry policy to a blob you have no control over otherwise? Shifting the burden of networking is also an explicit option listed in the "suitable for" section. You can decide where complexity lives after all. If accidental complexity is your issue I am inclined to ask where you see it here in general. Both the ambassador pattern and your proposed alternatives add overhead in terms of complexity somewhere. I'm struggling to see a clear favorite here.
Explicitly regarding your last statement that "[...] the remote service should protect itself against this).": Retries and Monitoring are things the remote service can't do by itself by definition. Even load balancing/shedding might not be solvable by it depending on the situation. Notice that circuit breaking is not the only thing the ambassador is used for. Network related configuration updates (see section "Context and problem") are something that might not be done by the remote service either.
> You don't add meaningless complexity for the sake of it after all. > You can decide where complexity lives after all.
Good points and something I should think about when designing systems.
You have good and interesting points, and it is true that I am very wary about introducing extra latency in the context of an http api that is used by a webapp. With the infra that is available today, it is possible to build snappy webapps, but my feeling is that you have to be wary about introducing extra "hops" in the execution of one http call, even though that is not strictly a "low latency" requirement.
And yes, this library usually also covers things like retries and handling errors and outages.
Maintaining an entire service to achieve the same seems like really expensive way to achieve this...
But it should obviously run in the same container/pod/etc.
I understand that at different points, you'll frequently run into situations where you need one or several of those, but the way this sentence is phrased, it sounds more like chastizing people for not following the latest fad (the components of which they are conveniently selling too).
"what, your deployment doesn't have advaned metering and circuitbreaking abilities and I have to restart the service to make config changes? get back to me when you have a real deployment..."