Locking down this network of services is a massive security improvement and they've used some very neat ways of achieving it. Overall, I really appreciate them writing this up.
However, 1500 services? That really feels like they're separating things at too granular a level. Does every one of those things really need to sit behind a network call? Couldn't some of that re-use be via code libraries? I wonder what the service to developer ratio is?
Code libraries in microservices can lead to a distributed monolith. each service needs to be be able to evolve independently. if you end up with a library where an update will require more than one service to be updated at the same time, then you are losing the advantage of microservices.
nevertheless 1500 is a lot, but if you think it's a lot, wait till someone builds out their entire business with lambda/serveless. :-)
Yeah, our (mostly) ban on shared code is a major reason we end up with a lot of services; there's even service.business-day which tells you if a day is a working day
I’m interested in the term “distributed monolith” and why shared libraries are bad, can you elaborate? (I’m an experienced dev who’s never worked on micro services). Say a service that consumed 6 other services became a service that consumed 3 other services and 3 libraries. The interfaces were the same, the libraries are consumed via package management and have good test coverage. Why is that worse?
I’m actually not surprised you have a service for handling working days. The list of UK bank holidays should ideally be stored in one place, with everything else consuming that. But not all of the 1500 services have their own data stores, do they?
The common idea is that if you have to regularly roll out all your services, then you still basically have a monolith. So if you have a lot of shared code in libaries, there are no benefits to microservices. In our case, maybe we do have core libaries like to read from Cassandra, but you don't necessarily have to roll out everything if you improve them. Some pods could be even a year old
All the services have their own keyspace in Cassandra, there is near-zero cross service data access
That's not what makes a distributed monolith, IMO. Or to put it differently, it's not what makes a "monolith" be equal to "bad" in people minds. You want to avoid the badness, don't want to avoid the label.
So, what is bad about a monolith? The complexity. Coupling of unrelated things; because it's simple to do so, because it's right there next to you and you know how it's implemented and what it performs, so you do thing A in component X knowing/relying on the fact that component Y will do the thing B, because that's how it's implemented (not because it's inherent unavoidable business logic).
How do you end up with a distributed monolith? E.g. assume that microservice B will call microservice C only after you (i.e. microservice A) had a chance to call C first. Yes your code is distributed, but you don't get the benefit of logic segregation between the modules - you just get all the downsides of distributed systems.
Do note that my scenario requires no shared code or library. Not sharing code doesn't save you from the distributed monolith - careful thought does.
Amen. Lots of micro service engineering is at the level of cargo cults and label avoidance. There are companies that do it well but relatively few of those. Whether TFA is a good or a bad example can not be told from the article contents, it would need a lot more information.
It seems crazy to me that you'd have to do a HTTP(S) request to find out whether a day was a working day. A HTTP(S) call which then probably does a DB lookup over the network. Just seems like an extra layer of overhead, when you could just have a library instead.
I've never worked on microservices, so I may just have a fundamental misunderstanding of how granular you're supposed to be & also don't know if this is the norm.
Would be nice to hear some other opinions on this.
I must admit I'm struggling to see what problem they are trying to solve with this. I'm not aware of a single platform where you can't modularise your code.
Though I’m against micro services for lots of reasons, there is truth to the “library update” problem
If you update the business day library, then you also need to bump the version of this lib used in your 100+ “real services”, or risk some services seeing differences on this front.
Imagine a bug from different logic due to version differences between libs in a distributed system. Would be a mess to debug.
If you say “oh well services should just install latest” now you have the distributed monolith problem. A new version of a library could get pushed out but have a regression bringing down half your services
You just roll back that one service and every other service picks up the change automatically.
Doing the same thing with shared libraries requires redeploying everything that is dependent on the bad library (i.e. hundreds of containers instead of two or three).
It's not necessarily that one model is better than the other just that using a hybrid approach often means having the issues of both with only a few (if any!) of the benefits.
You don't make incompatible updates, via adhering to service contracts. You can use protobuf or Avro as the schema for these service contracts and get varying levels of forward and backwards compatibility guarantees for schema changes.
Just like when developping an API, you build changes in backwards-compatible ways for the most part.
If you have to make a backwards-incompatible change, you probably need to make your service support two different API versions for a while, and hunt down services that are calling old stuff for a while.
This is basically how you can move forward on improvements without coupling "improve this microservice" to "update every single other microservice that is calling to it".
Though I believe part of the theory with microservices is that since you are working on such tiny surfaces, most services will never need to roll out a backwards incompatible change
The network in an AWS VPC is insanely fast. It's not a big deal. If they add a new national holiday, now you have to deploy all your services. I guess microservices self perpetuate themselves in that sense
Figuring out whether today is a working day is actually a dynamic problem. It is different for every locale, and it changes more frequently than one might expect.
It's actually a great example of something that changes rather frequently, and it's a concern that cuts across dozens (or in Monzo's case, maybe hundreds) of services.
Causing a roll-out to hundreds of critical business functions (whether hosted in a monolith or microservices) because Uganda added a holiday seems quite excessive.
The rule I and my company follows is that the shape of a module should follow its deployment model and scope. Some services are global in scope (literally targeted at the earth) others are scoped to a small subset of our inner network. They should have a different code repo because they have a different rollout schedule.
In the case of the working-days services. Only 1 rollout needs to occur, and it seems like a fairly safe rollout.
Let's consider another service, the CriticalTransactionService, or CTS. It executes critical business transactions in a very stateful way. When deploying a new version, in order to avoid any loss of availability, often a special rollout dance must be executed. E.g. switching the master transaction writer to another region, which means changing databases from passive replication to active master, and vice-versa.
One might consider this rollout "risky" and therefore only limits it to happen at 3am on a Saturday. It seems like a good idea to limit the scope of this 3am Saturday rollout only to the CriticalTransactionService, and most other services are free to deploy to prod whenever they wish.
This particular case could be a couple tables in a database. To follow proper encapsulation guidelines, it would have to be front-ended by some stored procedures, so that the underlying representation is free to change.
How would you roll out this change to production? If you just swapped out tables and stored procedures, you'd have downtime. You can't just install a parallel table and stored procedures, because there is no way to tell all of the consumers to use the v2 of your functions. So you'd have to temporarily remove availability of the WorkingDays functionality.
If it were deployed as a microservice, you are free to deploy a v2 of WorkingDays in parallel. When it's live, the old one goes away, and there is no loss of availability.
There might be db schema changes, migrations, who knows what else. Your service might even have multiple databases it interacts with. Your service might need to precache some stuff, that will add startup time. You can't simply shut down your service, you need to finish answering the requests already started.
A database table with proactively-invalidatable caches in front is already a little bit complicated.
Some of the things that can go wrong are:
- One or more caches failing to be invalidated by your "update business-day database" function.
- Temporary loss of connectivity to a cache from the updater, resulting in failure to proactively invalidate, so incorrect business-day results used by some services depending on which cache they read.
- Additional caches you didn't realise someone had added to the application or library code, that don't get invalidate.
- Behavioural inconsistencies as different functions in the distributed system read either the database or different caches, and get inconsistent results shortly after an update (after DB write, before proactive invalidation)
... In other words, the usual problems with distributed systems.
Also some logic issues, which a microservice is more likely to detect and log:
- No entry in the database for some far-future or far-past date, that application code assumed was ok to query, but nobody filled out in the DB.
Anyway, a database table can be thought of as a kind of microservice, in the sense that it's also remote network call, and it's also something you need to treat as an API contract with careful updates. If you aren't using a consistently distributed database, it suffers from multi-zone latency just like microservices calls.
Another reason why something like "business day" might be a function of stored data rather than just database data is that's actually a bit of a complicated and evolving logic as the product evolves:
Maybe the first application logic assumed a business day to be "Monday to Friday except for national holidays in the DB", did a DB lookup for the holiday flag, and combined with Mon-Fri. Later, we expanded to new countries and find it's different in other countries. Later, we found country was insufficient and it needed further details about administrative regions.
After each step, the previous database query and logic would still work but be incorrect due to missing a query field. So at each step, new logic needed to be rolled out globally across all services. For behaviour consistency across the system, all at the same time (similar to a DB schema change).
With a microservice, the logic update would be global and immediately consistent. Each part of the application could be updated separately, on their own development schedules, to provide the additional query field, long before the microservice was switched over to requiring the field. After that the microservice would fail any requests without the additional query field, making certain all functions and services acted consistently with the new logic, or failed.
...Having said all that, my favourite isn't microservices at all. In principle, logic updates can be rolled out with ACID-like properties and transactions, all the way to the edge of a system, just like data updates and cached data. There is no inherent reason why application and library code needs to suffer from complicated updates across a large business system. Business logic can be dynamic, replicated, and guaranteed consistent, while at the same time being available as fast, direct function calls. But I've rarely seen this implemented.
How do you deal with passing common data between services? For example, does every service checking the holiday endpoint need to write its own access/deserialisation/mapping starting from http or some other RPC layer? If yes, how do you keep that in sync/compatible? Do you have an explosion of versions on each endpoint?
This is likely trivial for the holidays endpoint, but I'm interested in the more complex ones.
This is what I want to know too -- seems like you have to have at least one common library, that is, the library that lets you write a service call in one line instead of twenty lines. IME Go can be quite verbose, especially with >3 lines of error checking after every function call.
Why (mostly) ban shared code? If you have that many microservices the deployment has to be well automated; so rolling out updated services on library update can't be difficult, and the interface to a compiled library is about as controllable as the interface to a network service.
Is there a separate reason to those? Do you worry about the additional latency of additional service calls (for some trivial functions that doesn't feel terribly worthwhile)?
I can imagine these shared libraries being in an inconsistent state at one time or the risks of updating thousands of services at once, would be two factors.
I'm curious why the investment in so many services instead of investing in improved CI/CD/build tools. For example, automatically deploying services when the libraries they depend on are updated. Monitoring to automatically check if a deployment is having issues. I mean, due to security concerns you still will need to re-deploy services every so often so having great CI/CD seems a more worthwhile investment.
edit: Also, for the same reason chaos engineering, short lived certificates and periodic backup tests make sense; constantly testing out the deployment of all services makes sense to me. If you do something like deploying a service rarely then the chance of something going wrong (forgot a step, radical external changes, etc.) is decently higher.
> wait till someone builds out their entire business with lambda/serveless.
Perhaps you could built in a "regular" fashion and just compile it like that? You could write normal code and let the compiler do the massive work to handle all that and compile everything into somethign that distributes over the network.
It's
1) theoretically possible in generalized fashion (although the language could make things easier)
2) it does exist in loose sense if you consider actor models (like someone else was mentioning)
If I'm not mistaken that's how most of the actor model frameworks are supposed to work. An example is Akka. The developer writes code communicating between actors using an interface that abstracts how they are actually communicating.
Great post. I really appreciate engineering blogs written in this storytime format. I don't have time to dive into the implementation of Calico or <insert one of the 1,261 kubernetes projects here>, but I learn a lot from reading the process a team goes through in figuring out and iterating on a solution.
Impressive achievement. It still sounds like callee's have more knowledge of callers than is justified. Is it a security property or a component functionality property? How do those interact?
A centralised graph representation of the security/functionality properties would be a better way to represent this information, so it can catch adding interfaces which should be forbidden. Also able to be configuration managed as sets of microservices.
If you have a connectivity graph it would be good to do taint analysis to see how far bad information can propagate.
I think we do effectively have a graph, but its not stored in a centralised way. If it was, then we'd have to somehow gatekeep that state, which when you consider that we do hundreds of deployments a day, becomes tricky
Then the state at present is changing hundreds of times a day, without being centrally visible, via runtime query or configuration service. Seems like it would be hard to retrospectively analyze what could connect to what. Maybe consider a permissive system, that allows a service code change to change the connectivity graph without a review step, but centrally logs the change; another service can supervise what's happening, and alert if strange things happen. It would help to have a system for categorising service types; the graph properties of the system might be useful for understanding that. Looking at service changes in terms of graph properties, like edit distance, can also be useful.
- with a proper XSD schema you can leverage validation already during parsing, which means you don't even have to boot your application at all - simply load your configuration in any XML parser and you'll automatically catch errors.
What are the alternatives? The minute you get into using code for these things, cleverness begins to turn configuration repos into something you have to debug before editing.
Instead you have YAML you have to debug because all of a sudden you have decided to support Norway as a country and something about Norway's country code and the way YAML handles Boolean values bites you in the ass.
I was curious enough to Google and found [1]. It turns out YAML supports figuring out the type of a value from the plain value itself. Norway's cc is "NO", which parses as a Bool instead of a string.
1. We have a lot of code paths, there isn't an integration test for everything. And for runtime, just because something is rarely called doesn't mean it never called; a bank can have yearly processes
2. We'll do service mesh too but this was probably an easier first step. We have rolled our own mesh with envoy sidecars and moving it towards istio style behaviour isn't trivial
> We have a lot of code paths, there isn't an integration test for everything.
but you will have the service names that you're contacting configured somewhere (or in the worst case hardcoded). I would think this is much easier to analyze than the traffic flows.
It's slightly more complicated than that because you can import protos to use a constant or to consume an event from another service. But that is the gist
> We have a lot of code paths, there isn't an integration test for everything.
This is just saying that writing "tests is hard". I agree, it is.
I always wonder about the value of static analysis. I've used it myself and have found bugs with it, but they are mostly easy bugs to find, and there's a lot of noise. The hard bugs are often only found with serious integration tests. I'll feel much more confident with a codebase that's 90% covered by tests than one that's statically analyzed.
You should think of static analysis as a form of linting. It's there to catch stupid errors and brainfarts that should not need to even reach a reviewer.
Dynamic analysis, instrumentation and intelligent logging are invaluable in catching a good chunk of the remaining non-trivial errors.
(I have a soft spot in my heart for fuzzing, which really should be considered as just another form of exploratory/destructive testing.)
> You should think of static analysis as a form of linting. It's there to catch stupid errors and brainfarts that should not need to even reach a reviewer.
I actually agree. But the implication is that static analysis should really live in the IDE, not the compiler or test suite. It's best when catching the silly whoopsy bugs as you're typing them. But I've yet to see a static analyzer that can catch race conditions, network or I/O errors, or something that can verify what you expect to happen actually does happen.
Thinking about static analysis on a higher level, arguably they shouldn't be necessary at all. If a system can statically analyze your code and find faults, then it implies that there is a deficiency in the language itself.
They say that they do both, though. First static analysis, then also log whatever requests that analysis missed, so that they can add it later on.
And they also add the static analysis to their CI pipeline, so that it can be checked then and there, not by obscure "happens once every half year during runtime" integration with a very uncommon service that they don't build very often.
It does not have to be. But that is the least of your worries, even so there might be valid reasons for setting things up this way but it would be a relatively rare case. They do have 150 devs and that alone is a difference that makes this approach untenable for most companies.
And in the evening, your wife will say, oh, dear, you are damn good architecture with many.. many services but my ex could deploy me with one several times a day much better.
Applying network filtering, while being a nice extra layer, it should not be the only layer. Services should need authorization like if it was an open api.
oAuth generally won't give you JWT's. Its recommended it gives you an opaque token which you introspect. Whats nice is you delegate the Auth part to your AS. And then your resource servers (micro-services) can make the determinations based on the client_id/claims from introspection.
Nice write-up! Thats the beauty of scale, explain a part in detail, then go with the 30,000 foot view.
IMM, the security orchestration may actually become the "app" as speeds continue to increase, compute costs go even lower and losses incurred from compromised data/networks increase.
A true zero trust platform that keeps all of the doors closed or "instances/vm" offline until (the milliseconds) they're needed is the security symphony we might see on the horizon.
Data silos and walled gardens may never go out of style, they'll just take on new acronyms.
Nice work. If you define your policies based on a tagging taxonomy you could centrally manage these inbound/outbound service relationships. Every new instance or container would assume same network policies based on tag.
87 comments
[ 3.1 ms ] story [ 201 ms ] threadHowever, 1500 services? That really feels like they're separating things at too granular a level. Does every one of those things really need to sit behind a network call? Couldn't some of that re-use be via code libraries? I wonder what the service to developer ratio is?
nevertheless 1500 is a lot, but if you think it's a lot, wait till someone builds out their entire business with lambda/serveless. :-)
I’m actually not surprised you have a service for handling working days. The list of UK bank holidays should ideally be stored in one place, with everything else consuming that. But not all of the 1500 services have their own data stores, do they?
All the services have their own keyspace in Cassandra, there is near-zero cross service data access
So, what is bad about a monolith? The complexity. Coupling of unrelated things; because it's simple to do so, because it's right there next to you and you know how it's implemented and what it performs, so you do thing A in component X knowing/relying on the fact that component Y will do the thing B, because that's how it's implemented (not because it's inherent unavoidable business logic).
How do you end up with a distributed monolith? E.g. assume that microservice B will call microservice C only after you (i.e. microservice A) had a chance to call C first. Yes your code is distributed, but you don't get the benefit of logic segregation between the modules - you just get all the downsides of distributed systems.
Do note that my scenario requires no shared code or library. Not sharing code doesn't save you from the distributed monolith - careful thought does.
I've never worked on microservices, so I may just have a fundamental misunderstanding of how granular you're supposed to be & also don't know if this is the norm.
Would be nice to hear some other opinions on this.
If you update the business day library, then you also need to bump the version of this lib used in your 100+ “real services”, or risk some services seeing differences on this front.
Imagine a bug from different logic due to version differences between libs in a distributed system. Would be a mess to debug.
If you say “oh well services should just install latest” now you have the distributed monolith problem. A new version of a library could get pushed out but have a regression bringing down half your services
I fail to see how they won't have the exact same problems.
Doing the same thing with shared libraries requires redeploying everything that is dependent on the bad library (i.e. hundreds of containers instead of two or three).
It's not necessarily that one model is better than the other just that using a hybrid approach often means having the issues of both with only a few (if any!) of the benefits.
If you have to make a backwards-incompatible change, you probably need to make your service support two different API versions for a while, and hunt down services that are calling old stuff for a while.
This is basically how you can move forward on improvements without coupling "improve this microservice" to "update every single other microservice that is calling to it".
Though I believe part of the theory with microservices is that since you are working on such tiny surfaces, most services will never need to roll out a backwards incompatible change
It's actually a great example of something that changes rather frequently, and it's a concern that cuts across dozens (or in Monzo's case, maybe hundreds) of services.
Causing a roll-out to hundreds of critical business functions (whether hosted in a monolith or microservices) because Uganda added a holiday seems quite excessive.
The rule I and my company follows is that the shape of a module should follow its deployment model and scope. Some services are global in scope (literally targeted at the earth) others are scoped to a small subset of our inner network. They should have a different code repo because they have a different rollout schedule.
In the case of the working-days services. Only 1 rollout needs to occur, and it seems like a fairly safe rollout.
Let's consider another service, the CriticalTransactionService, or CTS. It executes critical business transactions in a very stateful way. When deploying a new version, in order to avoid any loss of availability, often a special rollout dance must be executed. E.g. switching the master transaction writer to another region, which means changing databases from passive replication to active master, and vice-versa.
One might consider this rollout "risky" and therefore only limits it to happen at 3am on a Saturday. It seems like a good idea to limit the scope of this 3am Saturday rollout only to the CriticalTransactionService, and most other services are free to deploy to prod whenever they wish.
Doesn't the data just live in a database & there's a cache going on that can be invalidated?
Or is everything just so overcomplicated these days?
How would you roll out this change to production? If you just swapped out tables and stored procedures, you'd have downtime. You can't just install a parallel table and stored procedures, because there is no way to tell all of the consumers to use the v2 of your functions. So you'd have to temporarily remove availability of the WorkingDays functionality.
If it were deployed as a microservice, you are free to deploy a v2 of WorkingDays in parallel. When it's live, the old one goes away, and there is no loss of availability.
Some of the things that can go wrong are:
- One or more caches failing to be invalidated by your "update business-day database" function.
- Temporary loss of connectivity to a cache from the updater, resulting in failure to proactively invalidate, so incorrect business-day results used by some services depending on which cache they read.
- Additional caches you didn't realise someone had added to the application or library code, that don't get invalidate.
- Behavioural inconsistencies as different functions in the distributed system read either the database or different caches, and get inconsistent results shortly after an update (after DB write, before proactive invalidation)
... In other words, the usual problems with distributed systems.
Also some logic issues, which a microservice is more likely to detect and log:
- No entry in the database for some far-future or far-past date, that application code assumed was ok to query, but nobody filled out in the DB.
Anyway, a database table can be thought of as a kind of microservice, in the sense that it's also remote network call, and it's also something you need to treat as an API contract with careful updates. If you aren't using a consistently distributed database, it suffers from multi-zone latency just like microservices calls.
Another reason why something like "business day" might be a function of stored data rather than just database data is that's actually a bit of a complicated and evolving logic as the product evolves:
Maybe the first application logic assumed a business day to be "Monday to Friday except for national holidays in the DB", did a DB lookup for the holiday flag, and combined with Mon-Fri. Later, we expanded to new countries and find it's different in other countries. Later, we found country was insufficient and it needed further details about administrative regions.
After each step, the previous database query and logic would still work but be incorrect due to missing a query field. So at each step, new logic needed to be rolled out globally across all services. For behaviour consistency across the system, all at the same time (similar to a DB schema change).
With a microservice, the logic update would be global and immediately consistent. Each part of the application could be updated separately, on their own development schedules, to provide the additional query field, long before the microservice was switched over to requiring the field. After that the microservice would fail any requests without the additional query field, making certain all functions and services acted consistently with the new logic, or failed.
...Having said all that, my favourite isn't microservices at all. In principle, logic updates can be rolled out with ACID-like properties and transactions, all the way to the edge of a system, just like data updates and cached data. There is no inherent reason why application and library code needs to suffer from complicated updates across a large business system. Business logic can be dynamic, replicated, and guaranteed consistent, while at the same time being available as fast, direct function calls. But I've rarely seen this implemented.
This is likely trivial for the holidays endpoint, but I'm interested in the more complex ones.
Is there a separate reason to those? Do you worry about the additional latency of additional service calls (for some trivial functions that doesn't feel terribly worthwhile)?
edit: Also, for the same reason chaos engineering, short lived certificates and periodic backup tests make sense; constantly testing out the deployment of all services makes sense to me. If you do something like deploying a service rarely then the chance of something going wrong (forgot a step, radical external changes, etc.) is decently higher.
Perhaps you could built in a "regular" fashion and just compile it like that? You could write normal code and let the compiler do the massive work to handle all that and compile everything into somethign that distributes over the network.
[1] https://twitter.com/JackKleeman/status/1190354757308862468
https://landscape.cncf.io/
A centralised graph representation of the security/functionality properties would be a better way to represent this information, so it can catch adding interfaces which should be forbidden. Also able to be configuration managed as sets of microservices.
If you have a connectivity graph it would be good to do taint analysis to see how far bad information can propagate.
https://jsonnet.org/
https://json5.org/
See https://arp242.net/yaml-config.html
- it's unambiguous
- with a proper XSD schema you can leverage validation already during parsing, which means you don't even have to boot your application at all - simply load your configuration in any XML parser and you'll automatically catch errors.
The solution is to always quote strings.
[1] https://medium.com/@lefloh/lessons-learned-about-yaml-and-no...
Is there a link about how much Go does Monzo they use?
E.g. https://medium.com/@ahelwer/checking-firewall-equivalence-wi...
This is much more scalable in the long run.
1. Why was static analysis of the code chosen over observing the system during runtime and integration testing?
2. What was the reason rhe CNI layer was chosen for the implementation of this over the service mesh layer?
Something that really interests me about bazel/buck/pants/please is it automates #1 entirely with dep queries.
but you will have the service names that you're contacting configured somewhere (or in the worst case hardcoded). I would think this is much easier to analyze than the traffic flows.
This is just saying that writing "tests is hard". I agree, it is.
I always wonder about the value of static analysis. I've used it myself and have found bugs with it, but they are mostly easy bugs to find, and there's a lot of noise. The hard bugs are often only found with serious integration tests. I'll feel much more confident with a codebase that's 90% covered by tests than one that's statically analyzed.
Dynamic analysis, instrumentation and intelligent logging are invaluable in catching a good chunk of the remaining non-trivial errors.
(I have a soft spot in my heart for fuzzing, which really should be considered as just another form of exploratory/destructive testing.)
I actually agree. But the implication is that static analysis should really live in the IDE, not the compiler or test suite. It's best when catching the silly whoopsy bugs as you're typing them. But I've yet to see a static analyzer that can catch race conditions, network or I/O errors, or something that can verify what you expect to happen actually does happen.
Thinking about static analysis on a higher level, arguably they shouldn't be necessary at all. If a system can statically analyze your code and find faults, then it implies that there is a deficiency in the language itself.
And they also add the static analysis to their CI pipeline, so that it can be checked then and there, not by obscure "happens once every half year during runtime" integration with a very uncommon service that they don't build very often.
E.g. the monitoring service should only be able to access the metrics part of each service.
And in the evening, your wife will say, oh, dear, you are damn good architecture with many.. many services but my ex could deploy me with one several times a day much better.
Also didn't see any mention of prior art like https://cloud.google.com/beyondcorp/.
Thanks for the great writeup!
IMM, the security orchestration may actually become the "app" as speeds continue to increase, compute costs go even lower and losses incurred from compromised data/networks increase.
A true zero trust platform that keeps all of the doors closed or "instances/vm" offline until (the milliseconds) they're needed is the security symphony we might see on the horizon.
Data silos and walled gardens may never go out of style, they'll just take on new acronyms.
> We generally fixed those cases by adding a special comment in the code that told rpcmap about the link
Why not enforce all endpoints/urls be defined in a config file and sidestep this? - scanning code for URLs/constructed URL is overkill and brittle.