Semi-related: when people talk about monorepos, is it implied that all the project has only one version number? Why not just version subprojects of the monorepo, that way you have a small vetting process when cutting a release of a specific subproject. The rest of the subprojects that depend on it can read the release notes for breaking changes, etc.
Good question! I always assume it is the same version, mostly because a common pattern is to use SCM tags to track versions, and I haven't seen that work fine on any monorepo.
If you have a specific version per subproject, how do you track that in the repo? Different tag schemes for different subprojects? I have used that in a small-ish monorepo and I didn't like it specially.
When I'm doing it, I version subprojects independently using Git tags, but that's primarily because we mix and match versions in production, which creates need to make version numbering semantic.
If we were doing continuous delivery, too, I could see there not being much value in messing with independent versioning, semver, whatever. Just make today's date the universal version numbering system for all modules and move along.
No. Not at all. A monorepo with multiple back end projects might push all at different times. So what this means is that when designing new features across multiple services you need to design them with push safety in mind with a roll out plan to accomplish that.
For example, you are updating service B to call new endpoint on service A. First you need to make service A endpoint available, and then make service B call service A.
Just because everything exists in the same repo does not mean it all gets shoved out at once. The downside is that you can't just read the code and assume the running service is doing that, unless it's embedded in your build. Processes like automated updates and a forced update cadence (no running binaries over X days old) with proper canary/vetting before a full release allow a large org to still manage this complexity.
To me the whole point of having a monorepo is to avoid versioning. But that doesnt mean you always need to deploy everything, you can take the newest commit hash from each project directory and only deploy if that has changed.
I'd be interested to better understand the timeline around this statement:
"Metaserver was stuck on a deprecated legacy framework that unsurprisingly had poor performance and caused maintenance headaches due to esoteric bugs. For example, the legacy framework only supports HTTP/1.0 while modern libraries have moved to HTTP/1.1 as the minimum version."
Dropbox has been around for a lot of years, and raised a lot of cash; was it only recently that they could pay down this technical debt? Were they really so busy in other areas that this was allowed to fester?
The legacy framework was Pylons, which eventually evolved into Pyramid.
The tldr is there were hundreds of unowned endpoints that, yes, were allowed to fester. They eventually got ownership on all endpoints, so you had somebody to exert pressure on to make things happen.
I'm most curious about how the experience was, both for the atlas team and the product team, around this quote, "Atlas is “managed,” which means that developers writing code in Atlas only need to write the interface and implementation of their endpoints. Atlas then takes care of creating a production cluster to serve these endpoints. The Atlas team owns pushing to and monitoring these clusters."
Does this imply that the atlas team gets into the weeds of understanding the business and business logic behind these endpoints to know the scalability and throughput needs? Is the autoscaler really good enough to handle this? If it's transparent to the product team, are they aware of their usage (potentially unexpected)? I imagine the atlas team would have to be very large with these sorts of responsibilities.
From a product team perspective I imagine they are still responsible for database configuration and tuning? Has the daily auto-deployment led to unexpected breaks? Who is responsible for rollbacks? And is the product team responsible and capable of hotfixes?
Maybe a more broad question which all of my questions above speak to: how are the roles and responsibilities set up between the atlas team and the product engineering team that owns the code, and how has the transition to that system been?
So what's the end game here, is Dropbox going to keep building out an internal Kubernetes-like platform with Atlas, or do they plan to eventually just move to k8s? I noticed this line in particular:
"We evaluated using off-the-shelf solutions to run the platform. But in order to de-risk our migration and ensure low engineering costs, it made sense for us to continue hosting services on the same deployment orchestration platform used by the rest of Dropbox."
It sounds like they acknowledge they're reinventing a lot of stuff but for now are sticking to their internal platform. Perhaps Atlas is a half-step then to get teams used to owning and running their code as isolated services. But everything I read that they built in Atlas--isolated orchestrated services, gRPC load balancing, canary deployments, horizontal scaling, etc.--are bog standard features of Kubernetes today. I'd be very leery of maintaining a bespoke Kubernetes-like platform in 2021 and beyond--in some ways it seems like it's just shifting the monolith technical debt into an internal Atlas platform team's technical debt. What's the plan to get rid of that debt for good I wonder?
This hurdle shows there's already some cracks in the idea of long-term Atlas too:
"While splitting up Metaserver had wins in production, it was infeasible to spin up 200+ Python processes in our integration testing framework. We decided to merge the processes back into a monolith for local development and testing purposes. We also built heavy integration with our Bazel rules, so that the merging happens behind the scene and developers can reference Atlasservlets as regular services."
If I read that right does it really mean the first time a developer's code is run like it will run in production is when it goes out to canary deployment? I.e. integration tests are done in a local monolith instead of setting up a mini-prod cluster. That seems a bit nerve-racking as a dev to have no way to really test the service until bits are hitting user requests. In the k8s world a ton of work has been put into tooling and processes to make setting up local clusters easily. It's a shame to not have something similar for Atlas.
Counter considerations would be: What is the delta between out of box solutions and current solutions? What is the cost of the migration? For what period will two services be supported simultaneously? Will development effort continue on the previous service while the new one is created? Will the new service successfully solve problems the old one didn't? What happens when Kubernetes is insufficient for a task or has a critical bug that only appears at scale? How will people be onboarded into the new system? Will the team handling how services run perform the migration or will the teams who own services perform the migration? How much time should be spent experimenting with new things compared to fixing bugs/adding requested features?
> in some ways it seems like it's just shifting the monolith technical debt into an internal Atlas platform team's technical debt.
This is a key insight into the monolith problem. How does a monolith become poor and unmaintainable? A monolith becomes poor quality and unmaintainable when there is no entity enforcing architectural simplicity. It becomes unmaintainable when there is no team focusing solely on how the monolith functions. It becomes unmaintainable if there is no entity capable of saying "no" to a product engineer. A monolith in a company with weak leadership is a tragedy of the commons where everyone takes from the commons by adding complexity and there is no governing entity to ensure that the commons remains viable.
The exact statement you made is the key strength of this approach. Where there was a vacuum of responsibility before (monolith technical debt), a team has been created with direct responsibility and authority creating a governing force over that technical debt/overall complexity and therefore an entity directly responsible for improving it. This is a key first step. Atlas appears to be a compromise solution rather than an ideal end state.
yes, this is why you need a dedicated team who owns the quality and scalability of the monolith once you reach a certain size. many companies call this team "product infrastructure".
The greater problem is that the ways to fail monolith specific work vastly outnumbers the way to succeed. Furthermore, the primary work involved is months to years worth of horrible unattractive refactoring. Then this is compounded by it being very hard to measure success. So you run into a situation that makes principal engineers run or look for the door. Rather than working on an actually hard problem, they would rather pad their resumes with increasingly complex novel work, further detracting from the commons. So not only is it an un-owned common, but senior engineer alignment is directly against work involving the root problem because no one in their right mind would be responsible or accountable for something with so few avenues to success.
Not only is senior engineering alignment individually against being directly responsible for the monolith, but managerial leadership is as well. You have infrastructure with infrastructure concerns, and product worried about shipping product, and then you have to ask where this monolith responsible team is. Pushing code, monitoring, capacity management, oncall, etc. are all traditionally under the infra umbrella. Business logic is traditionally under the product umbrella. So who takes responsibility for the monolith, the chief repository of business logic? A monolith is a marriage between the two entities. Each individual product team doesn't want to do monolith work, because it makes it harder to ship the features they promised. Infra doesn't want to do monolith work because it is business logic code that needs to be refactored. So no infra team and no product team will directly feel responsible. Senior managerial leadership is at the mercy of senior engineering leadership, and therefore also cannot claim responsibility for the monolith. If you travel up the management hierarchy, that leaves roughly 4 people capable of doing anything about this most critical problem. The CEO, who has ultimate responsibility for all problems, the VP of infra, who could choose to claim responsibility for this area, the VP of product, who could choose to claim responsibility for this area, or the CTO who is the most directly responsible person for solving this problem as the de facto mediator of product and infra concerns and responsibilities. The answer is probably inventing a 3rd VP responsible for the area between product and infra and giving them proper authority.
I believe that one way to deal with communal ownership is to ensure that transgression against the communal contract are captured and addressed, just as should be in social commune. Of course the challenge is to ensure that is done in a positive and constructive manner.
Fundamentally, enforcing a communal contract is work, and it's a companies least favorite kind of work: "unplanned". Doing it manually and iteratively is what's called "toil". Toil is anything that is sisyphusian in nature. A key example of toil is pushing code every day. It is necessary and useful, but it doesn't make the company any money, and it's a constant cost that is paid every time code updates. Toil is pushing code, productive work is making continuous deployment a reality.
This enforcement of communal contract is toil. The only way to enforce it without toil is via spending engineering time creating systems. Maybe they are dependency injection frameworks, maybe they are servlets, maybe it's refactoring, maybe it's dependency elimination, maybe it's migration, but it is all fundamentally planned engineering work that will come at the cost of other potential work and therefore be at the mercy of management's priorities. Furthermore this kind of work must be opinionated and have the backing of internal authority, because people will absolutely violate the communal contract to get work out faster, especially senior engineers with lots of clout.
From a managerial perspective, you plan work on some cadence and expect work to get done. The engineers who are most effective at doing what they said they would do are the ones who are most going to please you, and therefore the ones who are going to get promoted. All toil comes directly at the cost of this. In order for this work to be internally incentivized it must be done as part of the companies overall planning and reward system.
Remember that this communal ownership is over a commons (overall complexity), and just like cows over grazing a field and running out of food, engineers over produce complexity and run out of time (time gets spent pushing code, dealing with bugs, running tests, refactoring code, etc.). When individual incentives are sour, you must have a governing body regulate the system. Each cow is individually incentivized to eat their fill, and each engineer is individually incentivized to ship product asap, regardless of code quality.
Most damning of all to communal enforcement is that as the number of engineers scales up, any non-explicit contract will be unenforceable as the number of communal "enforcers" with reasonable/relevant scope disappears entirely.
I just want to know when they'll switch to native battery efficient clients, especially given the daemon is always running and monitoring file system events.
Then I don’t understand the delay to shipping an Apple Silicon build. Right now we still have to used Rosetta... it’s the only such piece of software I have that does.
> Every line of code they wrote was, whether they wanted or not, shared code—they didn’t get to choose what was smart to share, and what was best to keep isolated to a single endpoint.
I am know very little about Python, but does this mean that Python has no way to encapsulate code at a level larger than a class? Something like a package or a module. It does not seem like it should be necessary to break a system into separate services just to get encapsulation at a module of subsystem level.
> I am know very little about Python, but does this mean that Python has no way to encapsulate code at a level larger than a class?
Python has packages, which contain modules. This doesn't seem to be a reference to the lack of encapsulation, in any case.
Though it seems to be projecting a social problem (absence of a policy involving active decisions on when particular code can be used by endpoints) onto architecture.
Python doesn't have true encapsulation unless you drop down to the C level to enforce it separately from the actual runtime. Unofficially, there are conventions on how to mark functions as part of a public interface, but practically speaking, if someone can import your library, they can do whatever they want with it, including adding and changing behavior at runtime. Classes, modules, and even class and function definitions are just hash tables under the hood, and you can change what a given symbol refers to in whatever arbitrary way you want to.
Granted, you shouldn't do very much of that except in some extremely specialized and careful circumstances, but in a language that makes it really easy, you have to be extremely disciplined and often delivery velocity takes precedence over careful planning of what really constitutes a public API.
Practically speaking, putting separate services in literally separate process spaces where the only way they can communicate is via message passing is the only way to really enforce encapsulation.
For an example of what I mean, check out the https://pypi.org/project/aioify/ library. This intercepts all of the function and method definitions in a module as they are being imported and rewrites them on the fly to turn all normal functions into async functions.
The language designers definitely took the path of we're just going to assume you know what you're doing and give you absolutely every last foot of rope you could possibly want to hang yourself with.
It does make things fast and easy. See all of the complaints here lately about having to rewrite significant parts of an application to become async as soon as any part of it becomes async? Not a problem in Python. There's a library to just automatically rewrite all code as it is being imported, at the cost that the code you're actually running is not the code you see in your repos.
You can kinda sorta make private variables in Python with some namespace abuse:
def make_get_private(var):
. . . .private_var = var
. . . .def get_private():
. . . . . . . .return
private_var
. . . .return get_private
get_private = make_get_private("spam")
You can't readily modify private_var after it is set in this case, you could also make and return a set_private function as well. I imagine you could do a whole module like this and then have it create it's functions at runtime to produce this "private" namespace. It would be a mess though.
Headline of article: "Our journey from a Python monolith to a managed platform"
So... about this headline. I read this aloud to a friend at a cafe. We laughed. It makes perfect sense to us. We know what Python is. We know what a monolith means in this context.
To my other friends it was the funniest / silliest / nonsensical thing they'd heard for awhile.
IT is weird.
(ps I know no one will see this comment but I'll leave it here. Because.)
30 comments
[ 5.2 ms ] story [ 76.1 ms ] threadIf you have a specific version per subproject, how do you track that in the repo? Different tag schemes for different subprojects? I have used that in a small-ish monorepo and I didn't like it specially.
If we were doing continuous delivery, too, I could see there not being much value in messing with independent versioning, semver, whatever. Just make today's date the universal version numbering system for all modules and move along.
For example, you are updating service B to call new endpoint on service A. First you need to make service A endpoint available, and then make service B call service A.
Just because everything exists in the same repo does not mean it all gets shoved out at once. The downside is that you can't just read the code and assume the running service is doing that, unless it's embedded in your build. Processes like automated updates and a forced update cadence (no running binaries over X days old) with proper canary/vetting before a full release allow a large org to still manage this complexity.
"Metaserver was stuck on a deprecated legacy framework that unsurprisingly had poor performance and caused maintenance headaches due to esoteric bugs. For example, the legacy framework only supports HTTP/1.0 while modern libraries have moved to HTTP/1.1 as the minimum version."
Dropbox has been around for a lot of years, and raised a lot of cash; was it only recently that they could pay down this technical debt? Were they really so busy in other areas that this was allowed to fester?
The legacy framework was Pylons, which eventually evolved into Pyramid.
The tldr is there were hundreds of unowned endpoints that, yes, were allowed to fester. They eventually got ownership on all endpoints, so you had somebody to exert pressure on to make things happen.
Does this imply that the atlas team gets into the weeds of understanding the business and business logic behind these endpoints to know the scalability and throughput needs? Is the autoscaler really good enough to handle this? If it's transparent to the product team, are they aware of their usage (potentially unexpected)? I imagine the atlas team would have to be very large with these sorts of responsibilities.
From a product team perspective I imagine they are still responsible for database configuration and tuning? Has the daily auto-deployment led to unexpected breaks? Who is responsible for rollbacks? And is the product team responsible and capable of hotfixes?
Maybe a more broad question which all of my questions above speak to: how are the roles and responsibilities set up between the atlas team and the product engineering team that owns the code, and how has the transition to that system been?
"We evaluated using off-the-shelf solutions to run the platform. But in order to de-risk our migration and ensure low engineering costs, it made sense for us to continue hosting services on the same deployment orchestration platform used by the rest of Dropbox."
It sounds like they acknowledge they're reinventing a lot of stuff but for now are sticking to their internal platform. Perhaps Atlas is a half-step then to get teams used to owning and running their code as isolated services. But everything I read that they built in Atlas--isolated orchestrated services, gRPC load balancing, canary deployments, horizontal scaling, etc.--are bog standard features of Kubernetes today. I'd be very leery of maintaining a bespoke Kubernetes-like platform in 2021 and beyond--in some ways it seems like it's just shifting the monolith technical debt into an internal Atlas platform team's technical debt. What's the plan to get rid of that debt for good I wonder?
This hurdle shows there's already some cracks in the idea of long-term Atlas too:
"While splitting up Metaserver had wins in production, it was infeasible to spin up 200+ Python processes in our integration testing framework. We decided to merge the processes back into a monolith for local development and testing purposes. We also built heavy integration with our Bazel rules, so that the merging happens behind the scene and developers can reference Atlasservlets as regular services."
If I read that right does it really mean the first time a developer's code is run like it will run in production is when it goes out to canary deployment? I.e. integration tests are done in a local monolith instead of setting up a mini-prod cluster. That seems a bit nerve-racking as a dev to have no way to really test the service until bits are hitting user requests. In the k8s world a ton of work has been put into tooling and processes to make setting up local clusters easily. It's a shame to not have something similar for Atlas.
> in some ways it seems like it's just shifting the monolith technical debt into an internal Atlas platform team's technical debt.
This is a key insight into the monolith problem. How does a monolith become poor and unmaintainable? A monolith becomes poor quality and unmaintainable when there is no entity enforcing architectural simplicity. It becomes unmaintainable when there is no team focusing solely on how the monolith functions. It becomes unmaintainable if there is no entity capable of saying "no" to a product engineer. A monolith in a company with weak leadership is a tragedy of the commons where everyone takes from the commons by adding complexity and there is no governing entity to ensure that the commons remains viable.
The exact statement you made is the key strength of this approach. Where there was a vacuum of responsibility before (monolith technical debt), a team has been created with direct responsibility and authority creating a governing force over that technical debt/overall complexity and therefore an entity directly responsible for improving it. This is a key first step. Atlas appears to be a compromise solution rather than an ideal end state.
Having worked in a company where no single team "owned" the monolith, the term "communally owned" tended to come up.
It was generally understood within the platform teams that if everyone owns it then in reality, no one owns it :)
The greater problem is that the ways to fail monolith specific work vastly outnumbers the way to succeed. Furthermore, the primary work involved is months to years worth of horrible unattractive refactoring. Then this is compounded by it being very hard to measure success. So you run into a situation that makes principal engineers run or look for the door. Rather than working on an actually hard problem, they would rather pad their resumes with increasingly complex novel work, further detracting from the commons. So not only is it an un-owned common, but senior engineer alignment is directly against work involving the root problem because no one in their right mind would be responsible or accountable for something with so few avenues to success.
Not only is senior engineering alignment individually against being directly responsible for the monolith, but managerial leadership is as well. You have infrastructure with infrastructure concerns, and product worried about shipping product, and then you have to ask where this monolith responsible team is. Pushing code, monitoring, capacity management, oncall, etc. are all traditionally under the infra umbrella. Business logic is traditionally under the product umbrella. So who takes responsibility for the monolith, the chief repository of business logic? A monolith is a marriage between the two entities. Each individual product team doesn't want to do monolith work, because it makes it harder to ship the features they promised. Infra doesn't want to do monolith work because it is business logic code that needs to be refactored. So no infra team and no product team will directly feel responsible. Senior managerial leadership is at the mercy of senior engineering leadership, and therefore also cannot claim responsibility for the monolith. If you travel up the management hierarchy, that leaves roughly 4 people capable of doing anything about this most critical problem. The CEO, who has ultimate responsibility for all problems, the VP of infra, who could choose to claim responsibility for this area, the VP of product, who could choose to claim responsibility for this area, or the CTO who is the most directly responsible person for solving this problem as the de facto mediator of product and infra concerns and responsibilities. The answer is probably inventing a 3rd VP responsible for the area between product and infra and giving them proper authority.
This enforcement of communal contract is toil. The only way to enforce it without toil is via spending engineering time creating systems. Maybe they are dependency injection frameworks, maybe they are servlets, maybe it's refactoring, maybe it's dependency elimination, maybe it's migration, but it is all fundamentally planned engineering work that will come at the cost of other potential work and therefore be at the mercy of management's priorities. Furthermore this kind of work must be opinionated and have the backing of internal authority, because people will absolutely violate the communal contract to get work out faster, especially senior engineers with lots of clout.
From a managerial perspective, you plan work on some cadence and expect work to get done. The engineers who are most effective at doing what they said they would do are the ones who are most going to please you, and therefore the ones who are going to get promoted. All toil comes directly at the cost of this. In order for this work to be internally incentivized it must be done as part of the companies overall planning and reward system.
Remember that this communal ownership is over a commons (overall complexity), and just like cows over grazing a field and running out of food, engineers over produce complexity and run out of time (time gets spent pushing code, dealing with bugs, running tests, refactoring code, etc.). When individual incentives are sour, you must have a governing body regulate the system. Each cow is individually incentivized to eat their fill, and each engineer is individually incentivized to ship product asap, regardless of code quality.
Most damning of all to communal enforcement is that as the number of engineers scales up, any non-explicit contract will be unenforceable as the number of communal "enforcers" with reasonable/relevant scope disappears entirely.
https://dropbox.tech/infrastructure/rewriting-the-heart-of-o...
Then I don’t understand the delay to shipping an Apple Silicon build. Right now we still have to used Rosetta... it’s the only such piece of software I have that does.
I am know very little about Python, but does this mean that Python has no way to encapsulate code at a level larger than a class? Something like a package or a module. It does not seem like it should be necessary to break a system into separate services just to get encapsulation at a module of subsystem level.
import foo
foo.auth("admin", "badpass") returns False
def new_auth(user, pass):
. . . .return True
foo.auth = new_auth
foo.auth("admin", "badpass") now returns True
Python has packages, which contain modules. This doesn't seem to be a reference to the lack of encapsulation, in any case.
Though it seems to be projecting a social problem (absence of a policy involving active decisions on when particular code can be used by endpoints) onto architecture.
Granted, you shouldn't do very much of that except in some extremely specialized and careful circumstances, but in a language that makes it really easy, you have to be extremely disciplined and often delivery velocity takes precedence over careful planning of what really constitutes a public API.
Practically speaking, putting separate services in literally separate process spaces where the only way they can communicate is via message passing is the only way to really enforce encapsulation.
The language designers definitely took the path of we're just going to assume you know what you're doing and give you absolutely every last foot of rope you could possibly want to hang yourself with.
It does make things fast and easy. See all of the complaints here lately about having to rewrite significant parts of an application to become async as soon as any part of it becomes async? Not a problem in Python. There's a library to just automatically rewrite all code as it is being imported, at the cost that the code you're actually running is not the code you see in your repos.
def make_get_private(var):
. . . .private_var = var
. . . .def get_private():
. . . . . . . .return private_var
. . . .return get_private
get_private = make_get_private("spam")
You can't readily modify private_var after it is set in this case, you could also make and return a set_private function as well. I imagine you could do a whole module like this and then have it create it's functions at runtime to produce this "private" namespace. It would be a mess though.
So... about this headline. I read this aloud to a friend at a cafe. We laughed. It makes perfect sense to us. We know what Python is. We know what a monolith means in this context.
To my other friends it was the funniest / silliest / nonsensical thing they'd heard for awhile.
IT is weird.
(ps I know no one will see this comment but I'll leave it here. Because.)