> In an ideal world, a version control system should be strictly internal facing and be able to fall over without impacting live traffic; however the Piper team kept discovering that this wasn’t the case.
This is something that surprised me while I was at Google too. Of course a source code version control system shouldn't be a dependency of production, but it turns out it is just so convenient to go from (1) use your source code system to store configuration so that they are versioned and tracked; (2) have tools to read configuration and apply them automatically, Terraform style; (3) have tools to modify said configuration automatically instead of manually. Voila your source version control system suddenly becomes a NoSQL database.
In addition to the benefits above, by storing storing config in version control is 2 party control, presubmit linting, and easy rollbacks. I suppose the trade off is slower rollouts of config and some delay time in pushing out a config but hey, there's no free lunch :-)
Nothing preventing you from building a versioned configuration system using a database or similar, especially if you build tools to actually change the configuration rather than hand editing text files.
The key to this is machine AND human readable formats.
That is why the text format of protobufs is such a good approach:
- structured formats
- declared schema
- the parser to read the file is also a validity checker (at least for data types)
There are tools that do similar with JSON but none are as simple and unobtrusive as protobuf tool chain.
When SOX compliance became a thing (remember Sarbannes-Oxley?), many moons ago, I was able to send EY consultants away during their audit for Unix controls, just by showing them that all our (Autodesk Unix team) configs were in SVN, logged, with authors and full change control.
The Windows team had to spend months figuring out something that was not even remotely close to control or auditability.
Frankly, I just assumed that it was the std practice today and I am surprised people are still not doing it.
Google's source code system isn't only used to store config: some teams also use it to store data like ML models. There's a limit on the max file size, causing all sorts of trouble in the LLM era. So it's a bit like Git LFS, but worse (why have the limit at all?).
> Better question, why no use a more appropriate, instead of convenient, tool for the job?
Because ML engineers also want version control and easy history and rollbacks and pull requests with code reviews for model changes.
There is no tool that does that except for the version control systems that programmers use. Other disciplines doesn't seem to care as much about these things.
Edit: And in general the extra storage/compute cost of storing it there is less than the saved work in programmer hours from having version control. It is much less worth adding support for larger files than trying to build an entire separate version control solution. Google already has support for images in the version control system, with image diffs etc, there is no reason not to also add support for these large ML models similarly.
> Because ML engineers also want version control and easy history and rollbacks and pull requests with code reviews for model changes.
That's not an answer so much as an excuse.
> There is no tool that does that except for the version control systems that programmers use.
It wouldn't be much effort at all, especially for Google, to write a simple wrapper that relied on a database or zfs snapshots or anything actually more suitable.
There are a number of really good internal tools for ML model management. It'd be really weird to rely on Piper to manage production ML models since this does not provide critical functionalities (monitoring, deployment, dependency management etc etc)
That team has some uncommon testing and deployment requirements, and they don't really run a service, so their use of Piper makes some sense to me. Although I agree it's still very weird. They are trying to migrate right now.
When I was there models weights were pretty much universally stored in colossus, either directly or through some bare bones tooling. I’m surprised the team you were working with had requirements that lead them to using piper directly for that.
We did abuse piper for all sorts of things though, most notably “golden” screenshots for diffing ui code changes across multiple search results pages.
Data in Piper (actually, even in SrcFS…?) is stored forever. So the cost of a “small” amount of data like few gigabytes can add up over time (cost for storage, replication, transmission, etc.)
Almost everything anyone ever wants to do with production is put files on a server and start a process. The rest comes down to your tolerance for baroque scaffolding and someone else’s janky shell scripts
"Jeff Dean, now Alphabet's chief scientist, personally stopped by the room that day to check in on the progress, helping boost morale, and further emphasizing the critical nature of the project."
How do you think outsiders get access to insiders to write articles about what happened behind closed doors at large corporations?
Affiliated with, no. Unless the author was there, just made everything up or sourced the details via 3rd party reporting (or as is the case in the opening line - via google published papers/press releases) then yes google PR personel would have been involved.
What makes it fearful? I read that as they were happy that they would likely to get promoted if the project went well since it had attention from higher ups, how is that something to fear? Google never fired people at that point in time, nobody had anything to fear.
The more interesting story here IMO is the seemingly extreme prevalence of NIH syndrome at Google, and whether or not that was a smart investment for Google compared to, say, contributing to GitLab or something.
I would love to hear the reasoning behind going custom, and if that ended up truly being worth the investment.
Or maybe I'm misremembering the timeline on when certain things became available. shrug
I think the takeaway from your comment is that you really do not understand the sheer scale of volume of changes done by +100k engineers, their heavy reliance on tools that do cluster-based testing (not local dev machine) and the tens of thousands of tools that perform auto-commits.
It’s a lot easier to shit on a big corp. Intellectually lazy, too.
The point is that either product should theoretically have the ability to handle Google's load. They were both built for scale. If I have the timeline correct, they existed at that point. I would expect those organizations to have positively salivated at the possibility of having Google as a customer; they would have bent over backwards to make it support Google's workflows.
The ROI of such a choice vs going custom is the interesting question.
Of course, if I have the timeline wrong, then the discussion is moot.
There is a big difference between one VCS handling 1,000 QPS and 1,000,000 VCS instances each handling 0.001 QPS. A system built for one is not necessarily suitable for the other.
It's not moving the goalpost because you didn't understand where the goalpost was in the first place.
Lots of tiny repos like GitHub? Easy sharding problem. They don't even need Paxos. One large monorepo? Everything changes. Let us not forget that GitHub cannot even handle the Homebrew repo be cloned and updated by Mac users and asked them to switch to a CDN.
> The more interesting story here IMO is the seemingly extreme prevalence of NIH syndrome at Google, and whether or not that was a smart investment for Google compared to, say, contributing to GitLab or something.
this is the perfect HN comment - so confident but with so little thought.
essentially all google code - something like a billion LOC - are in one repository.
there's something like 100 000 people who use source control, plus jillions of systems doing automatic commits all the time.
it's atomic on each commit.
so that's several QPS globally synchronising a huge data store, with the entire dev tooling system hanging off it.
git is ... very very much not suitable for such systems.
gitlab is just a simple web app reading from a bunch of small and rarely changing git repositories.
Piper is also the successor to Perforce, which is a completely different model to git.
It's interesting how people treat the monorepo as some kind of religion that MUST be kept in this discussion.
How repos are organized is a choice. Google could just as easily (financially speaking) have chosen to break up that repo and use more standard tooling. They did not choose that. Did they have (possibly perfectly valid) reasons for that? Sure.
That does not render it as their only possible path.
Rather than switch their source control strategy to something that would work well with existing products, they chose to build a brand new product to support their existing strategy. This is, IMO, not an obviously sane decision from an outside perspective. Why?
Because source control has nothing to do with their core business, and in addition to the cost of developing and deploying that solution, they now have to bear the cost of maintaining it, which is likely not insignificant.
That's how it looks from an outside perspective.
The question I'm asking is not "why didn't they use git?!". The question I'm asking is, "what is the ROI of building and maintaining their own solution versus biting the bullet and pivoting so they can use pre-existing products?"
Haven't seen an answer to that yet. People are too busy implying or outright stating that I'm stupid for asking the question in the first place.
> It's interesting how people treat the monorepo as some kind of religion that MUST be kept in this discussion.
not at all, it's just what google decided to do on balance. not only did they decide it was the best choice for them, the cost of switching would be ... astronomical. the amount of tooling that assumes it can find all code in the one place is very very large.
A single senior developer full time @ Google is probably a half million dollar a year investment, give or take. How many do they have maintaining it? How long before that yearly cost exceeds both license fees for an alternate solution and the cost of development and migration?
What makes you think that a multirepo system would be cheaper to maintain than a monorepo? It's just moving complexity from one place (source control software) to another (everything that interacts with the source control software). TANSTAAFL
I never argued that I thought multirepo would be cheaper than monorepo. Or vice versa, for that matter.
I wondered if it would have been cheaper overall to use pre-existing tools and adjust things than to build a whole new product that has nothing to do with your core business, and then have to maintain that product. This is a wholly different argument.
TANSTAAFL right back at you; development and maintenance time is not free either, and there's a potential argument that pivoting their other tooling would also open them up to other off-the-shelf opportunities, though that is much harder to evaluate.
I've worked at scale. I am well aware that sometimes custom is the only way. I've been there and done that, and walked it back when someone finally did something off the shelf that was cheaper to maintain. Custom comes with a cost, and that cost can often be hefty.
Source control... is not usually where you hear the "it should be custom" argument, and even less often is it at all supportable when you look at the big picture.
Google is certainly large enough to have unique requirements, but that alone doesn't justify the choice.
I thought about the larger organizations, and of them I could identify only Microsoft and IBM as other companies who have their own SCM products (though that's probably ignorance; there are probably more ;). The difference with them, though, is in the fact that they sell those products as products, making them part of the core business. Profit center instead of cost center.
Google doesn't do that AFAICT, so all that investment in development and maintenance is nothing more than cost at the bottom line. And not just monetary cost: what else could those developers be doing with their time?
Hence, it really makes me wonder if there's a bunch of developer religion that's costing Google a lot of unnecessary cash. If I were by some strange miracle brought in as a CTO at Google? I'd be wanting some pretty hard facts on that (or a plan to productize it) to justify its continued existence.
If someone tells me that monorepo (or multirepo) is better, I'll respect their opinion even if I disagree with it. If, however, someone tells me that one or the other is the only way, and they must have a custom source control system to prevent things from collapsing like a house of cards?
I think you are looking at this from a hindsight perfectionist bias.
Maybe that is the right decision. But Google has organically grown over the decades and their tooling evolved with that.
It's hard to say which requirements they were working with and what they were trying to optimize for - sometimes higher costs are just okay if they allow or enable other things.
There is validity to that, it just seems to be a very odd decision to me. If I was going to do custom infrastructure, Source control is not where I would be looking. I'd rather put up with a few warts there and focus the time and resources that saves elsewhere.
For me it's always "what are we trading?". In an organization the size of Google, it's far easier to amortize away the cost of such a thing. That doesn't mean it's not a cost, though. And there's a lot of tooling around modern SCM that they won't get because they're on a bespoke system, so there's a cost there too.
As you say, at the time the variables may have been very different.
It's just an area where if I had to make that decision (even back then), someone would have had to be exceptionally persuasive to convince me to authorize what is effectively a multi-million dollar project... to recreate source control. =)
You're comparing "expensive custom solution" with "free off-the-shelf solution" when in reality it's "expensive custom solution" vs "expensive off-the-shelf solution". Source control at Google scale is difficult and you can't just wave a magic wand and make that complexity go away.
> what is the ROI of building and maintaining their own solution versus biting the bullet and pivoting so they can use pre-existing products?
Using made-up numbers, if you have 50000 employees using source control at your company and a custom solution would improve their productivity by 0.1%, having a team of 10 building such a solution is positive ROI. In contrast, if you have 1000 employees, it would be a negative ROI. This is why it's worth doing if you're Google or Facebook but not most other companies.
So I was at FB when they attempted to add support to Git for really large monorepos. They offered cash and people to work on it, but the git team was completely uninterested and basically said that you're holding it wrong.
Perhaps Google tried to do it this way and hit the same issues.
Git also just doesn't play nice with monorepos of the scale of Google's/Facebooks. From a different article explaining why FB moved away from Git:
"The engineers tried running a simulation, creating a dummy repo that matched the expected scale of Facebook’s codebase in a few years. The result was horrifying - basic Git commands took over 45 minutes to complete."[1]
Yeah, I remember this. The team involved did try really hard to get the Git team to listen to them, before working with Mercurial (and later building their own thing, although that was after I left).
I don't think you understand how much early Google values engineer productivity.
> Because source control has nothing to do with their core business, and in addition to the cost of developing and deploying that solution, they now have to bear the cost of maintaining it, which is likely not insignificant.
God I hate this outsourcing mindset. It's so refreshing to see a company value their developer productivity that they make a custom tool for just their needs. The ROI is readily measurable as the increased productivity of hundreds of thousands of people writing code. It's also measurable in the simplified tooling of every single tool that needs to interact with the source control. Same thing with the decision to switch to multi-repo from a monorepo: lots of added overhead to every engineer and every tool.
Git sucks so much compared to Piper that it boggles the mind to even ask such a question.
I haven't used piper, so I can't comment on that. But people seem to have focused on git. Remind me not to actually use product names as examples; it inevitably triggers people into holy wars and prevents them from actually seeing the point.
I don't care if the competing solution would have been git, Perforce, hg, or whatever. The point was not "git". The point was "off the shelf". I also do not dispute that Piper may well be a great system; I've never even seen it, so I wouldn't know.
The issue is not about "outsourcing" (which is a rather loaded term that should not be substituted for "using off the shelf products"). It's not about "engineer productivity", per se. The question is about overall business efficiency.
I'm sure it could make engineers that much more productive to have the latest and greatest laptop every year. That doesn't mean we replace all our hardware every year. We also don't give every developer a terabyte of RAM. It's the same basic issue.
It's always a trade; those dollars can potentially do more good somewhere else.
You'll note that I have at no point stated outright that I thought they made the wrong decision; only that I'm skeptical. I asked the question about the ROI, and people decided to get all heated about it. The only arguably inflammatory point in my comment was about Google's case of NIH.
It's really hard to deny that Google has acute NIH.
Sometimes that's a good thing; they've done some amazing things and given us some amazing technologies. Other times, I question their decisions.
I don't think you are wrong; it's just that a lot of people here on this thread are current or former Googlers. Compared to them, you don't know Piper so your argument is at a disadvantage.
My perspective is simple: inventing its own source control system at Google is absolutely worthwhile and the ROI is highly positive. The tradeoff is worthwhile. You might say that before Piper existed it might be a gamble whether it would be worth it. But in hindsight, absolutely 100% yes it's worthwhile. It does not need to be argued, for anyone who has used it.
The only thing that is really subject to debate is whether Google's engineering leads had the foresight of what Piper would accomplish or whether they simply wanted to build something custom due to NIH and it only fortuitously turned out to be good. That's a question no one here can answer.
> I don't think you are wrong; it's just that a lot of people here on this thread are current or former Googlers. Compared to them, you don't know Piper so your argument is at a disadvantage.
Bah. Text boxes in browsers suck, and mine's not cooperating at the moment, so instead of writing another book, I'll just say: thank you for the reasonable response. =)
GitHub uses Git. Git itself is ill suited to Google's size. Years ago Google has tried something called git5 which aimed to bring git to the Google monorepo. It failed miserably because Git itself isn't scalable. Lest you think they didn't try hard enough, Facebook tried the same thing and came to the exact same conclusion that Git isn't scalable. Nowadays both companies use mercurial which is much better. They couldn't even switch to Git let alone a forge based on Git.
Mercurial wasn't scalable either but the Mercurial community was much more receptive to changes to improve scalability. So Meta pushed Mercurial as far as possible with a single server solution while developing -> https://github.com/facebook/sapling.
I have been reading about Google's monorepo setup and source control system, and my answer to the question in the title is simple: they had money and people. It is hard to think any other company that is not of Google's scale would be able to create their in-house source control system and complete a migration. This especially feels true after my company failed migrate from Perforce to git. You don't just build your source control, you build an entire ecosystem around it. And you need (very well-paid, smart) people to do that.
> It is hard to think any other company that is not of Google's scale would be able to create their in-house source control system and complete a migration
But it is easy to think of a company that is of Google's scale failing to complete such a migration, wasting years with many projects not being able to migrate and then ultimately having two systems that everyone has to sync between. That they actually succeeded at doing it instead of failing is the impressive part.
64 comments
[ 3.1 ms ] story [ 76.0 ms ] threadThis is something that surprised me while I was at Google too. Of course a source code version control system shouldn't be a dependency of production, but it turns out it is just so convenient to go from (1) use your source code system to store configuration so that they are versioned and tracked; (2) have tools to read configuration and apply them automatically, Terraform style; (3) have tools to modify said configuration automatically instead of manually. Voila your source version control system suddenly becomes a NoSQL database.
There are tools that do similar with JSON but none are as simple and unobtrusive as protobuf tool chain.
Because it's probably buggy and not tested with commits past a certain file size.
Better question, why no use a more appropriate, instead of convenient, tool for the job?
Because ML engineers also want version control and easy history and rollbacks and pull requests with code reviews for model changes.
There is no tool that does that except for the version control systems that programmers use. Other disciplines doesn't seem to care as much about these things.
Edit: And in general the extra storage/compute cost of storing it there is less than the saved work in programmer hours from having version control. It is much less worth adding support for larger files than trying to build an entire separate version control solution. Google already has support for images in the version control system, with image diffs etc, there is no reason not to also add support for these large ML models similarly.
That's not an answer so much as an excuse.
> There is no tool that does that except for the version control systems that programmers use.
It wouldn't be much effort at all, especially for Google, to write a simple wrapper that relied on a database or zfs snapshots or anything actually more suitable.
https://www.microsoft.com/en-us/research/uploads/prodnew/202...
We did abuse piper for all sorts of things though, most notably “golden” screenshots for diffing ui code changes across multiple search results pages.
That is such a cringy, fearful, statement...
https://en.wikipedia.org/wiki/Seagull_management
Though perhaps I lack perspective, having just toured Normandy where the D Day landings took place.
Affiliated with, no. Unless the author was there, just made everything up or sourced the details via 3rd party reporting (or as is the case in the opening line - via google published papers/press releases) then yes google PR personel would have been involved.
I would love to hear the reasoning behind going custom, and if that ended up truly being worth the investment.
Or maybe I'm misremembering the timeline on when certain things became available. shrug
It’s a lot easier to shit on a big corp. Intellectually lazy, too.
(edit: the point being that at least one of those tools existed when they started Piper if I have the timeline correct).
On a single project? Verry much doubt either of them do
The point is that either product should theoretically have the ability to handle Google's load. They were both built for scale. If I have the timeline correct, they existed at that point. I would expect those organizations to have positively salivated at the possibility of having Google as a customer; they would have bent over backwards to make it support Google's workflows.
The ROI of such a choice vs going custom is the interesting question.
Of course, if I have the timeline wrong, then the discussion is moot.
Lots of tiny repos like GitHub? Easy sharding problem. They don't even need Paxos. One large monorepo? Everything changes. Let us not forget that GitHub cannot even handle the Homebrew repo be cloned and updated by Mac users and asked them to switch to a CDN.
this is the perfect HN comment - so confident but with so little thought.
essentially all google code - something like a billion LOC - are in one repository.
there's something like 100 000 people who use source control, plus jillions of systems doing automatic commits all the time.
it's atomic on each commit.
so that's several QPS globally synchronising a huge data store, with the entire dev tooling system hanging off it.
git is ... very very much not suitable for such systems.
gitlab is just a simple web app reading from a bunch of small and rarely changing git repositories.
Piper is also the successor to Perforce, which is a completely different model to git.
How repos are organized is a choice. Google could just as easily (financially speaking) have chosen to break up that repo and use more standard tooling. They did not choose that. Did they have (possibly perfectly valid) reasons for that? Sure.
That does not render it as their only possible path.
Rather than switch their source control strategy to something that would work well with existing products, they chose to build a brand new product to support their existing strategy. This is, IMO, not an obviously sane decision from an outside perspective. Why?
Because source control has nothing to do with their core business, and in addition to the cost of developing and deploying that solution, they now have to bear the cost of maintaining it, which is likely not insignificant.
That's how it looks from an outside perspective.
The question I'm asking is not "why didn't they use git?!". The question I'm asking is, "what is the ROI of building and maintaining their own solution versus biting the bullet and pivoting so they can use pre-existing products?"
Haven't seen an answer to that yet. People are too busy implying or outright stating that I'm stupid for asking the question in the first place.
not at all, it's just what google decided to do on balance. not only did they decide it was the best choice for them, the cost of switching would be ... astronomical. the amount of tooling that assumes it can find all code in the one place is very very large.
A single senior developer full time @ Google is probably a half million dollar a year investment, give or take. How many do they have maintaining it? How long before that yearly cost exceeds both license fees for an alternate solution and the cost of development and migration?
The answer to that is not obvious.
(edit: response specifically regarding "astronomical" costs)
I wondered if it would have been cheaper overall to use pre-existing tools and adjust things than to build a whole new product that has nothing to do with your core business, and then have to maintain that product. This is a wholly different argument.
TANSTAAFL right back at you; development and maintenance time is not free either, and there's a potential argument that pivoting their other tooling would also open them up to other off-the-shelf opportunities, though that is much harder to evaluate.
I've worked at scale. I am well aware that sometimes custom is the only way. I've been there and done that, and walked it back when someone finally did something off the shelf that was cheaper to maintain. Custom comes with a cost, and that cost can often be hefty.
Source control... is not usually where you hear the "it should be custom" argument, and even less often is it at all supportable when you look at the big picture.
Google is certainly large enough to have unique requirements, but that alone doesn't justify the choice.
I thought about the larger organizations, and of them I could identify only Microsoft and IBM as other companies who have their own SCM products (though that's probably ignorance; there are probably more ;). The difference with them, though, is in the fact that they sell those products as products, making them part of the core business. Profit center instead of cost center.
Google doesn't do that AFAICT, so all that investment in development and maintenance is nothing more than cost at the bottom line. And not just monetary cost: what else could those developers be doing with their time?
Hence, it really makes me wonder if there's a bunch of developer religion that's costing Google a lot of unnecessary cash. If I were by some strange miracle brought in as a CTO at Google? I'd be wanting some pretty hard facts on that (or a plan to productize it) to justify its continued existence.
If someone tells me that monorepo (or multirepo) is better, I'll respect their opinion even if I disagree with it. If, however, someone tells me that one or the other is the only way, and they must have a custom source control system to prevent things from collapsing like a house of cards?
Yeah, I'm going to call bullshit.
Maybe that is the right decision. But Google has organically grown over the decades and their tooling evolved with that.
It's hard to say which requirements they were working with and what they were trying to optimize for - sometimes higher costs are just okay if they allow or enable other things.
For me it's always "what are we trading?". In an organization the size of Google, it's far easier to amortize away the cost of such a thing. That doesn't mean it's not a cost, though. And there's a lot of tooling around modern SCM that they won't get because they're on a bespoke system, so there's a cost there too.
As you say, at the time the variables may have been very different.
It's just an area where if I had to make that decision (even back then), someone would have had to be exceptionally persuasive to convince me to authorize what is effectively a multi-million dollar project... to recreate source control. =)
Using made-up numbers, if you have 50000 employees using source control at your company and a custom solution would improve their productivity by 0.1%, having a team of 10 building such a solution is positive ROI. In contrast, if you have 1000 employees, it would be a negative ROI. This is why it's worth doing if you're Google or Facebook but not most other companies.
Perhaps Google tried to do it this way and hit the same issues.
"The engineers tried running a simulation, creating a dummy repo that matched the expected scale of Facebook’s codebase in a few years. The result was horrifying - basic Git commands took over 45 minutes to complete."[1]
[1] https://graphite.dev/blog/why-facebook-doesnt-use-git
> Because source control has nothing to do with their core business, and in addition to the cost of developing and deploying that solution, they now have to bear the cost of maintaining it, which is likely not insignificant.
God I hate this outsourcing mindset. It's so refreshing to see a company value their developer productivity that they make a custom tool for just their needs. The ROI is readily measurable as the increased productivity of hundreds of thousands of people writing code. It's also measurable in the simplified tooling of every single tool that needs to interact with the source control. Same thing with the decision to switch to multi-repo from a monorepo: lots of added overhead to every engineer and every tool.
Git sucks so much compared to Piper that it boggles the mind to even ask such a question.
I don't care if the competing solution would have been git, Perforce, hg, or whatever. The point was not "git". The point was "off the shelf". I also do not dispute that Piper may well be a great system; I've never even seen it, so I wouldn't know.
The issue is not about "outsourcing" (which is a rather loaded term that should not be substituted for "using off the shelf products"). It's not about "engineer productivity", per se. The question is about overall business efficiency.
I'm sure it could make engineers that much more productive to have the latest and greatest laptop every year. That doesn't mean we replace all our hardware every year. We also don't give every developer a terabyte of RAM. It's the same basic issue.
It's always a trade; those dollars can potentially do more good somewhere else.
You'll note that I have at no point stated outright that I thought they made the wrong decision; only that I'm skeptical. I asked the question about the ROI, and people decided to get all heated about it. The only arguably inflammatory point in my comment was about Google's case of NIH.
It's really hard to deny that Google has acute NIH.
Sometimes that's a good thing; they've done some amazing things and given us some amazing technologies. Other times, I question their decisions.
I don't think there's anything wrong with that.
My perspective is simple: inventing its own source control system at Google is absolutely worthwhile and the ROI is highly positive. The tradeoff is worthwhile. You might say that before Piper existed it might be a gamble whether it would be worth it. But in hindsight, absolutely 100% yes it's worthwhile. It does not need to be argued, for anyone who has used it.
The only thing that is really subject to debate is whether Google's engineering leads had the foresight of what Piper would accomplish or whether they simply wanted to build something custom due to NIH and it only fortuitously turned out to be good. That's a question no one here can answer.
Bah. Text boxes in browsers suck, and mine's not cooperating at the moment, so instead of writing another book, I'll just say: thank you for the reasonable response. =)
But it is easy to think of a company that is of Google's scale failing to complete such a migration, wasting years with many projects not being able to migrate and then ultimately having two systems that everyone has to sync between. That they actually succeeded at doing it instead of failing is the impressive part.