"There are limitations this system. Potvin says certain highly sensitive code—stuff akin to the Google’s PageRank search algorithm—resides in separate repositories only available to specific employees."
Did you read the article? "There are limitations this system. Potvin says certain highly sensitive code—stuff akin to the Google’s PageRank search algorithm—resides in separate repositories only available to specific employees. "
> Potvin says certain highly sensitive code—stuff akin to the Google’s PageRank search algorithm—resides in separate repositories only available to specific employees.
> There are limitations this system. Potvin says certain highly sensitive code—stuff akin to the Google’s PageRank search algorithm—resides in separate repositories only available to specific employees.
> (...) all 2 billion lines sit in a single code repository available to all 25,000 Google engineers.
FWIW, apart from the previously mentioned sensitive stuff, we give engineering interns the same level of access we give full-time engineers. We keep things open because it makes things faster; we have an excellent code search tool that's great for navigating through the Piper repo (e.g. finding subclasses, finding uses of an API) which really speeds up dev time.
When we're not talking about the sensitive stuff, there's not much magic to what many engineers write every day, it's the same "glue technology X to technology Y" stuff you see everywhere, so I don't think there's any value to hiding that in the name of secrecy.
I though Google's search advantage really speeds up software development. I wonder if they use the same principle in alphabet across other engineering disciplines - creating some unique knowledge tools, and how do they look.
How are changes that affect sensitive code handled? Are the owners of that code on the hook for making any API updates that the person pushing the change can't make?
Having never worked on the secret sauce, I honestly don't know. There is a small team of people who tend to do many of the global refactors, I might expect that they are given special permission.
Why wouldn't almost every engineer have access to almost the entire repo? Most of Google's code is only relevant to another company working at Google scale, such as Facebook, Microsoft, Apple, etc. These are the companies with deep pockets that are willing to spend lots of money to acquire technology that will help them compete with Google. But none of these companies will buy code that's been stolen from Google.
The most famous corporate trade secret, the Coke formula, was stolen by two employees who attempted to sell it to Pepsi. Pepsi alerted Coke, the companies worked together to bring in the FBI, and both employees went to prison: http://www.cnn.com/2007/LAW/05/23/coca.cola.sentencing/
Comparing "Google" to Windows isn't really a fair comparison. I'm sure all of the code that represents products that Microsoft has in the wild far exceeds 2B lines.
Note that this is just the monolithic repository.
Google also has other non-piper repositories containing hundreds of millions of lines too :P
For example, android and chrome are git based.
Note also that when codesearch used to crawl and index the world's code, it was not actually that large. It used to download and index tarballs, svn and cvs repositories, etc.
All told, the amount of code in the world that it could find on the internet a few years ago was < 10b lines, after deduplication/etc.
So while you may be right or wrong, i don't think it's as obvious you are right as you do.
I'm still trying to figure out why having everything dumped in one big pile is something worth bragging about. I'd far rather have code sorted well into proper repositories.
So much is shared, though, right? Which is why Android is sorted into proper repositories but still has the 'repo' front-end wrapper to make sure you're getting the right versions of everything you need.
If I wanted to change something fundamental, like I found a 10% speedup in Protobuf wire decode by changing the message slightly, there are likely very many services that all need it.
Everyone at Google operates on HEAD. You're not allowed to break HEAD, and pre-submit/post-submit bots ensure you don't and will block your submit.
From my perspective, at least, this design seems to explain why Google websites are so frequently broken in the ways different services integrate. Because Googlers edit shared resources that affect products they don't personally work on, and they just trust automated tests, which almost certainly miss a lot of the edge cases I encounter.
I admit that I'm not an expert at large software development, but this seems to nearly fully explain Google's declining code quality.
I also agree especially as this includes the whole Google software ecosystem and Microsoft has their own ecosystem. Microsoft's whole ecosystem of products that work together and run on Windows is much larger than Windows itself. Might not be as monolithically developed, though.
"The two internet giants (Google and Facebook) are working on an open source version control system that anyone can use to juggle code on a massive scale. It’s based on an existing system called Mercurial. “We’re attempting to see if we can scale Mercurial to the size of the Google repository,” Potvin says, indicating that Google is working hand-in-hand with programming guru Bryan O’Sullivan and others who help oversee coding work at Facebook."
Well, Piper conforms to the Perforce API-ish, and Android and Chrome are both on Git.
Mercurial was pushed internally as being the "better" (for some dimension of better) between it and Git back in 2010, but I think even the most hardline Mercurial fans have realized that in order to meet developers in the middle in 2015, we need to use Git for our open-source releases. We have a large investment in Gerrit [1] and Github [2] now.
So the Mercurial comment is probably entirely based on scaling and replacement for the Piper Perforce API, rather than anything externally facing.
Though both Android and Chromium still build in a way more similar to that for a monolithic repo. They use repo and depot_tools, respectively, as abstractions on top of Git in order to clone and manage all their Git repositories together as if they were a single large repository.
I don't think I'm leaking anything by saying that that article about Google is incorrect. Mercurial isn't used at Google, not anywhere I've seen. Probably that article has something to do with the now-defunct Google Code site, which did in fact use Mercurial.
Google devs are working (with some help from Facebook) on getting hg into Google, but they're not there yet. Look at all the work that Google has been putting into hg. It's not purely out of the goodness of their hearts:
I can't comment on what Google might be doing in the future with RCS. Well, I can't comment but also don't know. I personally like hg.
But if I were to take a poll of the informed and opinionated engineers sitting around me, they would almost all likely take git over hg.
In any case, git is in actual widespread use _now_ at Google, for Android and Chrome, and other open source stuff (some of which was moved recently from code.google.com to github).
I was at Google for a long time and it was almost entirely Perforce (some Googlers used git) until it was Piper (which is very similar to Perforce). L&S used CVS at Stanford for the original Backrub/Google code and for some time after the establishment of Google, Inc.
It was built from scratch because it was designed as a replacement for Perforce because Perforce is expensive and the G used wrappers around it, anyways.
The commands are still basically the same as Google's wrappers around Perforce and the learning curve was non-existent for someone who was used to using Perforce at Google.
There are also a massive number of git repos internal to googl e as well.
One thing mentioned in the paper, but not mentioned here, is that there are teams that live in GIT, such as android and chrome, but they are not monolithic.
There may be Mercurial plans for the future, but to date I've never encountered it at Google. Lots of people use git, though, including folks outside of Chrome and Android who are working against Piper.
It looks to me like mercurial is easier to hack on than Git (written in clean Python vs written in C), and in their initial stage both mercurial and git weren't up to the task.
Why not IBM? Why not Microsoft? Is that what the HN crowd has turned into? You now have to justify the non mainstream solution like a scared mega-corp career drone who will only go with the safest bet?
Git isn't so awesome that it's inconceivable that people would be willing to use something else.
Personally; I wondered the same thing as the parent, why not Git? For me, this thought was not in a, "How could they possibly not choose Git!?", but more along the lines of, "There are a lot of smart people working at Google; I wonder what technical aspects of Mercurial might make it more suited to such a large code repository than Git?"
I'm at Google now, but would share my opinion of Perforce while I was at Treyarch - Perforce is simply very easy for anyone to use - from software engineer, to game designer, scripter, artist, producer, technical writer, linguist, translator, etc. You can announce your changes by simply saying - it's submitted in CL: 12345678 - grab it and you are done. Eases the communication.
Git and HG are not. Not that they can't be learned, but the learning curve is much higher.
Piper and CitC (mentioned in Rachel Potvik's videO) are even more advanced, I can work from my desktop machine, then go and continue from my laptop, or open and edit files directly from critique in an internal web-based editor.
Key to all this - make it accessible through the web! Anyone can work then from almost any machine. But at the same time the sources is accessible in the file system - even more awesome - so i can use emacs, and whatever other tools are there.
It's a legitimate question, damn it. Especially if the chosen solution is not mainstream.
You don't pick non-mainstream solutions "just because". You do pick mainstream solutions "just because". As in, "just because" they are tried and true, "just because" it's familiar to devs, etc.
So let me ask the reverse question: Is that what the HN crowd [sic] has turned into? You can no longer ask the reasons behind your tech choices?
> Git isn't so awesome that it's inconceivable that people would be willing to use something else.
Inconceivable? No, I just asked why. Git to me is a great solution for source control, so I want to understand it's deficiency for handling large amounts of code, especially considering it is used for managing Linux source and handles everything in GitHub I've used quite well.
Git is fine at scaling horizontally (many repositories), so things like GitHub work well. Neither Git nor Mercurial do well at scaling to a single massive repository; getting Mercurial to do so was easier because we can implement it as out-of-core extensions (see http://bitbucket.org/Google/narrowhg, for example).
There's also internal-only things to deal with, such as the CitC integration. Because of some of the design decisions behind CitC, storing a typical .git or a .hg directory in CitC is essentially impossible. Mercurial's .hg directory is intentionally a black-box - you interact with the repo using the hg tool. The .git directory can be seen as an API - there are at least three implementations that matter for our purposes, and if we change something in the .git directory, we lose the editor/IDE integration powered by the ones we didn't fix (or we did fix, but they haven't been released yet; some of those are linked into products that have a commercial release cycle).
I've used git on repos far less than 2 billion lines of code and it was kind of a nightmare to clone and do basic local checkouts and normal everyday stuff. Out of the box, git's design is not well suited to large repos.
I know there are various workarounds for dealing with large repos but having had some experience using git on big-ish projects I can certainly understand some possible reasons why it wasn't their first choice given the size of their codebase.
I don't work for FB, but word through the grapevine is that FB wanted to use Git, but the Git folks refused to talk to FB about making it scale. By contrast, hg was happy to have them as users and contributors.
IIRC, Piper is a reimplementation of the perforce backend, in order to handle the code size and the sheer number of "changelists" submitted per second. Nothing to do with code.google.com.
I worked on code.google.com, i can tell you the are 100% unrelated.
piper grew out of a need to scale the source control system the initial internal repositories were using
code.google.com was a completely separate thing supporting completely different version control models, and a very different scale
(very large number of small repositories, vs very small number of very large repositories)
Still, this is not a very forward-thinking solution. Building and combining microservices – effectively UNIX philosophy applied to the web – is the most effective way to make progress.
EDIT: Seems like I misunderstood the article – from the way I read it, it sounded like Google has a monolithic codebase, with heavily dependent products, deployed monolithically. As zaphar mentioned, it turns out this is just bad phrasing in the article and me misunderstanding that phrasing.
I take everything back I said and claim the opposite.
It explains how even in a small company modularization can help extremely. Now look at Google, where some issues (like the google.com april 1st XSS issue) were only fixed after outsiders mentioned it.
Usually internally the team responsible for that part should have cought that.
The issue was that they wanted to load the page – with the user logged in, etc – on com.google. For this they implemented an explicit URL parameter that would allow this.
> Attackers could have seized on the omission of the X-Frame-Options header to change a user's search settings, including turning off SafeSearch filters
Yes, I'm sure Google has a lot to learn from SoundCloud about how to deploy software at scale, like that time NASA got advice from Estes Industries on how to launch rockets.
Google runs practically everything internally as services. Nothing about the code repository makes it impossible to run microservices. Where did you get the idea that google runs a single monolithic app for everything?
You should think of Piper as a single filesystem which permits atomic multi-file edits. And that's about it; there's nothing in that which forces any particular release structure on you.
The talk did mention (briefly) that monolithic codebase and monolithic binaries/software aren't strictly related. It's likely that monolithic software is easier in a monolithic codebase, but I don't think that microservices are harder in a monolithic codebase. Yes, we tend to statically link, but that's for library dependencies (i.e. things like the protocol buffers libraries/definitions). I don't work at all in this area, so this is a guess, but I imagine that it's extremely rare for unrelated teams to have application logic linked into the same binary - just making a release would involve coordination from so many teams :)
I will say that I saw and experienced many things that changed my definition of 'large' at Google, but the most amazing was the source code control / code review / build system that kept it all together.
The bad news was that it allowed people to say "I've just changed the API to <x> to support the <y> initiative, code released after this commit will need to be updated." and have that effect hundreds of projects, but at the same time, the project teams could do the adaptation very quickly and adapt. With the orb on their desk telling them at that their integration and unit tests were passing.
I thought to myself, if there is ever a distributed world wide operating system / environment, it is going to look something like that.
We do, mostly. Because Piper is a global repository, we have systems to do global safe refactors, and do so often. If the API changes drastically, there's usually a lengthy deprecation period before the API is switched over.
I guess that works for the Linux kernel, but I would presume that for a large distributed operation like Google it would be much better to simply deprecate/version APIs and have the project teams update to a deadline.
I mean, it's presumably impossible to have a single computer running a single OS build all of the Google software and run the testing.
Google is rumored to have an extremely powerful distributed compilation farm. It wouldn't surprise me if a single developer could make a change that affects the entire codebase and test it themselves.
> The solution to the excessive API change problem is to force
> whoever changes the API to fix all the consumers himself
> before the change is accepted.
This doesn't seem scalable. Let's consider the case of one api endpoint being changed by one developer, to add a new param to a function call. Further assume that this impacts hundreds of projects.
Does it really make sense to make one developer update those hundred projects? Not only will it take forever for it to get finished (possibly never if there are new consumers of this api coming online frequently), but the developer of the core api may not have any experience in the impacted consumers of this codebase. I think the end result of this policy would be nothing once written ever would get updated, and new apis would just be added all the time (api explosion).
Google's product explosions / surprise deprecations possibly hint that this is what happens? Changing the API becomes cumbersome, so you just make a new product with a new API to do an end-run around the requirement...
It seems bad but the alternative (wait forever for groups that don't understand the change and may not even exist anymore to update their code) seems worse.
It maybe isn't scalable, but that's part of the benefit. If you want to make a change to a widely-used API, it's going to be a lot of work, and it's not going to be a lot of work for the users of the API, it's going to be a lot of work for _you_ because _you_ are required to do it yourself. This prevents a lot of API churn unless the benefit is clear and sufficiently large.
If it was any other way you'd rapidly reach a useless equilibrium where random engineers were demanding that thousands of other engineers fulfill unfunded mandates for what might turn out to be negligible benefits.
That's one extreme. Another extreme is that you have the API versioning from hell, where you can never get rid of technical debt because any and all API changes will break someone, somewhere, who has no reason to migrate, so you're left keeping ancient code on life support indefinitely.
When that's the cultural norm, people adjust accordingly. Tools include liberal use of assertions, defensive tests, and most important, code reviews that catch and remove quirkiness.
It's a nice environment to work in. In addition to hastening Noogler onboarding, it also increases employee retention. If you are an expert in your project's codebase but get burned out, you can easily transfer to another project and be almost immediately productive.
Obviously, there's domain-specific knowledge that doesn't transfer easily or quickly from project to project. But that's quite different from self-inflicted code fragility; one's an asset and the other's a liability.
Wow, $150? Shocked there isn't a cheaper USB version that is more easily hackable. I mean, is it really more than color-changing LEDs in a plastic sphere?
blink(1) is a small USB light to give you glanceable notice of anything on your computer or the internet. blink(1) makes it easy to connect data sources in the cloud or on your computer to a full-color RGB LED so you can know what’s happening
Thanks for the link--looks awesome, wish the form factor were a bit larger and not so directly attached to the computer. I'll bet I could create an extension cable for it and rip off the housing to make the LEDs allow more light through, then put it in a translucent plastic sphere.
Started with a project that used the Ambient Orb to show test status, they were expensive which led to a contest to make the cheapest one, basically an ATTiny with a software USB hid stack tied to an RGB LED. Henner Zeller won that contest and I think maybe 5000 gOrbs were built.
I built something similar as a hack project... Closet dome light fixture turned upside down. Rip out the innards. Install RGB LED. Connect LED up to Arduino and pull information from an API. Adjust LED based on your conditions. I spent around $75, but that was years ago, probably far cheaper now.
Can you (or somebody) explain how that works in practice?
Here I am ready to deploy some new feature to gmail. In the meantime, I'm getting a steady stream of API changes. Can I build and release gmail to a specific revision number, and only incorporate the changes when I am ready, or are all release essentially off of the tip of the tree.
I don't need specifics, just the general idea. Where we work basically every project lives in its own branch, which makes it essentially impossible to synchonize changes. Things have to get merged to trunk, then pulled into the appropriate branch, and I don't like it at all.
Do you have version numbers for libraries/components: projext X uses version 1.5.4 of Y, 32.4.18 of Z, and so on, do you pull by revision number, are you all on tip?
It'd be very interesting to hear how you manage this.
Every feature has a flag that you can turn on/off to your heart content. Releases are cut at head. No need to use the version control hammer to screw all the zippers.
No, features are configurable at runtime. This allows A/B testing or slowly rolling out a new feature. Depending on the type of change, you might test it on 1% of users, 1% of servers, etc.
Every feature is in HEAD, behind a flag. Precisely because managing conflicts cannot be automated, relying on branching is unwise. With HEAD based development, you manually solve conflicts exactly once, when submitting to HEAD, then you never have to think about them again.
Local branches for development, a single HEAD for merging code.
In theory:
Everyone works from head, so there is only one version of everything.
When you write a feature, you follow these steps.
1. Write code and submit to the main (and only) branch. Hide your feature behind a flag.
2. Releases happen from head (with no regard to your feature)
3. When you want to enable your feature, you flip a flag to turn it on.
Yes, you sync you workspace to a particular revision and work within that. Syncing is a manually triggered process. And generally speaking, people changing APIs are required to find and fix all users (which isn't that hard with the infrastructure available). There is still some churn created by the constant stream of changes, but it's usually not on the changing APIs level.
That's actually a good question. What would a world wide environment look like. Currently we have the web, but that isn't really a world wide environment in this sense. I think that this could be a way it would look, but more of a combination of this and a way to split everything down into the parts again without breaking the system. However, I've never worked at G, so I can't know if they have a method for this
How do the monolithic repository companies handle dependencies on external source code?
Are libraries and large projects e.g. RDBMS generally vendored/forked into the monolithic repositories, regardless of whether the initial intent is to make significant changes?
There's typically a subdirectory called third_party, with subdirectories for each vendor, product and version. If the team is smart, they will also enact a rule saying "only one version". If you're really, really smart, local changes are kept as a set of patches, keeping them separate from the imported tar file.
So, for source deliveries:
third_party/apache/httpd/2.4/release.tgz
/patch.tgz
/Makefile (or other config)
third_party/apache/httpd/2.2/release.tgz
/patch.tgz
...
In the spirit of "You didn't build that," I wonder how many lines of code comprise the binaries that Google binaries run on? Windows, Linux, network stacks, Mercurial, etc, etc.
I also wonder if there's a circular relationship anywhere in there.
I imagine that there's a lot of Java and C++. I do like Go but it makes you wonder if a more expressive language that requires a fraction of the code would be helpful. Maybe Steve Yegge will see Lisp at Google after all.
So a monolithic codebase makes it easier to make an organization wide change. Microservices make it easier to have people work and ship in independent teams. The interesting thing is that your can have have microservices with a monolithic codebase (as Google and Facebook are comprised of many services). But you can also have a monolithic service with many codebases (like our GitLab that uses 800+ gems that live in separate codebases). And of course you can have a monolithic codebase with a monolithic service (a simple php app). And you can have microservices with diverse codebases (like all the hipsters are doing).
I'm wondering if microservices force you to coordinate via the codebase just like using many codebases force you to coordinate via the monolithic service. Does the coordination has to happen somewhere? I wonder if early adopters of microservices in many codebases (SoundCloud) are experiencing coordination problems trying to change services.
I'd be interested on an analysis of the meta-differences between those, but I think one of the main ones is a monolithic codebase makes it massively harder to open source components. I'm sure that, working on GitLab, you can agree with that - if GitLab were a massive, monolithic codebase and you wanted to open source specific parts of it, it'd be a huge pain.
Indeed, if you have a monolithic codebase releasing your code for other people use is much harder. In git it is possible to create a separate repository for it (using subtree and filter-branch). But the harder part is versioning and release management. At Google they avoid the cost of versioning their services and the release management around that, but this makes it really hard to have the outside world use their code. Maybe this is hard anyway since their code probably depends on a lot of services common to Google (GFS, BigTable). This is probably why they can't release Borg/Omega but have to make Kubernetes instead.
I think that advantages mentioned in the presentation about the monolithic codebase can be achieved if you have one source code server that is open to everyone (for example GitLab with most projects set to internal). Some of the tools will be easier to write for one repository than iterating over many, but that seems solvable.
The biggest advantage seems to be that when you are an author of a dependency you can propose upgrades to all services that use your application. It is not clear to me but it seems that for small changes you can just force that change on the code owners. This ensures that the dependency author incurs the cost of a change (as is done for API changes in the Linux kernel) and that you do not need to version the API of the dependency.
Interestingly Google recently started marking API's private by default. So they are moving in the direction of explicit API management.
As soon as you work with people that are outside your control (as is common in open source) you would need to version the API as well in my opinion.
Google has tons of services internally that talk via RPC. The monolithic repo means that it's much easier to hunt down and find people that are (say) using an outdated RPC method and help them fix their code.
(Just one example of how it's useful even when things are mostly services.)
This discourages you from ever making breaking changes to an API. On the face of it that sounds good but sometimes you do have to make breaking changes. My guess is that there are many duplicate projects in the Google code base for when a breaking change is needed. This is a way to sidestep the problem.
I think sometimes it's actually the opposite. I was able to make a breaking change to an API and update all of the callers to use the new one in a single commit.
Tests would run on all the clients and, since in my workspace the server was updated simultaneously, I could be more sure it would work.
That's fine for small breaking changes like an API being renamed, but some times breaking changes require actually refactoring code which is hard/impossible to do without intimate knowledge of a codebase.
Think python 2->3 or Angular 1->2. These types of changes do happen, and I bet they happen at Google. I don't think anyone is rewriting a downstream app when they make these changes. Most likely they are doing something like forking the library and renaming it, which is just another form of versioning.
python 2->3 and angular 1->2 indeed need a different strategy and the strategy varies per case. For Angular both versions will have to be supported (basically just with different directory name) and switch most probably won't happen in terms of single applications, but by writing completely new applications on new stack. Lifespan of web application is rather short, so we can afford this.
As someone who's responsible for migrating the users of a large breaking change (~300 projects), it does discourage you from making a breaking change. I'm not convinced that this is a bad thing though. Breaking changes can be both expensive and tempting. By making the author of such a change feel the cost of it you prevent the migration work from becoming an externality.
It also encourages writing tools for automating this stuff. A big part of the motivation for polyup (https://github.com/PolymerLabs/polyup) was the fact that we'd be responsible for getting people to upgrade.
To add to your point: let's differentiate between a monolithic REPOSITORY vs a monolithic CODE BASE. Having all the code in a monolithic REPOSITORY allows developers to more efficiently explore and transform the code with a better understanding of the breadth of their impact.
Whether the CODE BASE is monolithic or not is orthogonal to the repository's nature. I was G for a couple years and I'd say they've done an ok job of breaking things up into libraries and services. Certainly there are interfaces that have done a better or worse job of setting the code up for open sourcing but because of the nature of the repository, large scale refactoring is more efficiently accomplished.
So a monolithic codebase makes it easier to make an organization wide change. Microservices make it easier to have people work and ship in independent teams. The interesting thing is that your can have have microservices with a monolithic codebase (as Google and Facebook are comprised of many services). But you can also have a monolithic service with many codebases (like our GitLab that uses 800+ gems that live in separate codebases). And of course you can have a monolithic codebase with a monolithic service (a simple php app). And you can have microservices with diverse codebases (like all the hipsters are doing).
I'm wondering if microservices force you to coordinate via the codebase just like using many codebases force you to coordinate via the monolithic service. Does the coordination has to happen somewhere? I wonder if early adopters of microservices in many codebases (SoundCloud) are experiencing coordination problems trying to change services.
The CitC filesystem is very interesting. This is local changes overlaid on top of the full Piper repository. Commits are similar to snapshots of the filesystem. Sounds similar to https://github.com/presslabs/gitfs
What are the best practices to follow in a single-repo-multiple-projecrs world?
Some people recommend git submodule, others recommend subtree.
How do you guys manage alerts and messages - does every developer get a commit notification,or is there a way to filter out messages based upon submodule.
How does branching and merging work?
I'm wondering what processes are used by non-Google/FB teams to help them be more productive in a monolithic repo world.
FWIW: git submodules are not a single repo by definition. It's just a way to automate the checkout of specifically-versioned external projects without requiring hackery like packing tarballs into the project source. It has its uses, but it's definitely not what they're talking about here.
Agree 100%. Git submodules are for tracking other stuff, not for doing dev on that other stuff.
If you would like to see how things would work with submodules that behaved just like files behave (full distributed workflow) we've got a (unfortunately commercial) solution here:
Generally branching isn't really a thing at Google. Work is done at the code review level per change list ("CL"). Most changes happen through incremental submission of reviewed CLs, not by merging in feature branches. Every CL must run the gauntlet of code review, as well as can not usually be submitted without passing tests. There are rare cases where branching is used, but not commonly.
As for notifications, the CL has a list of reviewers and subscribers. If you want to see code changing, you watch those CLs. Most projects have a list where all submitted CLs go.
Can you explain this a little more - what is a CL vs a changeset...and what do you mean by watching changelists.
It sounds like you're subscribing to specific commits...but I'm talking about more at a project/directory level within the monolithic repo.
Changelist is Perforce-speak for changeset. Because each CL gets an individual review before applying to the codebase, there is no merge process -- there is no branching/merging.
There is a solution for project/directory-level CC / review requirements. I didn't see it discussed in the talk, though.
Really? This article sounds very over simplified, but I haven't worked at google so I wouldn't know. I'm assuming if you want to change some much depended on library, there's a way to up the version number so you don't hose all your downstream users. That's the way it worked at Amazon at least. Also, I wonder why the people in the story think Google's codebase is larger than that of other tech giants, not that it really matters.
It's incumbent upon the person updating the library to get all users migrated to the new one. There are a few strategies for doing this though, including temporarily having two versions of the library.
There are also tools for making large scale changes safely and quickly.
Last I heard Google is still on Java 7 precisely because of this, although that might have changed. It's fun seeing the different theologies at Amazon and Google - I remember Yegge's famous platform rant, and he highlighted the Amazon versioned-library system as something which it did better than Google.
Google mostly works at HEAD. Very little is versioned, and branches are almost unheard of.
In general you change the much depended on library and all of its consumers (probably over time in multiple changes, but you can do it in one go if it really needs to be a single giant change).
One humorous side-effect of having all that code viewable (and searchable!) by everyone was that the codebase will contain whatever typo, error, or mistake you can think of (and convert into a regular expression).
I remember seeing an internal page with dozens of links for humorous searches like "interger", "funciton", or "([A-Z][a-z]+){7,} lang:java"...
It searches for CamelCase identifiers that are made of seven or more "terms", where each term is a capital letter followed by one or more lowercase letters.
I'm a google software engineer and it's nice to see this public article about our software control system. I think it has plusses and minuses, but one thing I'll say is that when you're in the coding flow, working on a single code base with thousands of engineers can be an intensely awesome experience.
Part of my job- although it's not listed as a responsibility- is updating a few key scientific python packages. When I do this, I get immediate feedback on which tests get broken and I fix those problems for other teams along side my upgrades. This sort of continuous integration has completely changed how I view modern software development and testing.
Being able to make sweeping changes to a shared piece of code, and ensure that everyone's up to date (Hi, Rosie!) and not broken by your change (yay TAP train!) is phenomenal as well.
Technically, you can't see every use of a function because dynamic dispatch mechanisms aren't available at source code time. That's why you would normally run tests of all your library's dependents (having this dependency graph is one of the most important parts of the piper/blaze system).
Yep, this is the key factor, you get to shorten your deprecation cycle to little or none. In contrast, with multiple repos, even for small projects, you will get a few days up to a month of lag time.
We've been really crappy at marketing this but we have a multiple repos
answer that is semantically close to identical to a monolithic repo.
Any work flow or command line works identically in both (there are a
few corner cases that don't work but almost 100% of the stuff works
like you'd expect).
It's sort of like a partial clone, you can clone any subset you want,
work on it, add in other parts, the tool takes care of making sure the
stuff you add in is lined up (if you cloned a week ago and you add in
another repo, it's rolled back so it matches time/commit wise).
If you want to search for something it's the same command in a collection
of repos as it is in a monolithic repo:
bk -U grep refactor_me
but truth in advertising, that searches only what you have populated.
The google answer is "ship grep to the data" so they'll search everything.
Google wins if you have their datacenter. We win on UI, it's the same
everywhere.
Our design is much more tightly coupled than Git's submodules. We manage
the subrepos so that they are in sync just like files are in sync.
What I mean by that is if you have two files modified in the same commit,
you can't pull that commit and only get one of the files updated,
they are both updated because that's what happened in that commit.
We've provided the same semantics for collections of repositories.
Git doesn't, getting those semantics is an exercise left to the user.
We get better performance because you can clone as little as one
repository, what we call the product. It would be easy and fun to put a
Unix distro in this system and have the top repo just have a makefile and
you type make X11 and it populates the compiler, libc, the X11 sources,
and builds them.
It's commercial so maybe that's uninteresting but if you want submodules
that work and scale in performance, we might be worth a look (sorry for
the marketing, if that's against the rules here then I'll learn how to
delete a post).
All dependencies between directories are specified as part of the bazel[0] BUILD files. Since dependencies between files/packages are all laid out in BUILD files, you can have a graph of all dependencies between all packages. So if you change something in 1 package it can figure out all packages that should be effected.
Bazel test[1] doesn't provide dependency testing as far as I know, but it creates the framework to support doing it.
It's a form of efficiency due to vertical integration. I agree as others mentioned that it would be possible to do this without a single code base but it would be very unwieldly.
I just submitted a change today that affected 266,259 different build targets within google3. I was slightly terrified to make a change of this magnitude, having been at Google for less than three months, but I ran all the affected tests multiple times and it doesn't seem to have broken anything. (Fingers crossed!)
Working on the google codebase is pretty awesome. This week I've made changes/improvements to libraries owned by three different teams (not counting my own), in C++ and python, when my main project is all in Java. It's super fun. The code search tool is great - it's ridiculously fast, and makes navigating through the codebase very easy.
It sounds like a really awesome workflow, although parts of it really only work on very large teams. I bet the version control system would be very cool though, hopefully it gets a github equivalent and has better branching than mercurial
Could you tell something about the level of documentation? For instance, do you have to write a paragraph of documentation for every function that you add to the system? How about adding preconditions and postconditions, and other invariants?
Also, is the code that you add to the repository always inspected by other people? Is that done systematically?
It is mandatory that each code change is inspected for correctness, language style and approved by a code owner (all 3 may be the same person, or require multiple, depending on the situation).
> Could you tell something about the level of documentation? For instance,
> do you have to write a paragraph of documentation for every function that
> you add to the system? How about adding preconditions and postconditions,
> and other invariants?
It varies a lot by project and by how widely used the code is.
There are few (if any) strictly enforced rules for documentation or invariant checking. You basically have to convince at least one other engineer that what you have is sufficient.
The documentation is _generally_ pretty decent for core libraries, but sometimes you just have to read the code.
If I recoup with latest news: The ranking algorithm is in one repo, Chrome and Android in another, and all the rest is in Piper? What is all the rest? Gmail, Calendar, the defunct Reader, the car, Maps, and OnHub... How does that fit with Alphabet? Is Piper a cross-functional initiative of Alphabet now?
Look at the companies that are non-Google entities (the ones under Alphabet that aren't under Google):
* Google X - moonshot projects that aren't software centric.
* Google Fiber - mainly a infrstructure setup. May share some stuff in piper.
* Google Ventures - investment arm, not code related.
* Google Capital - more investment stuff.
* Calico - R&D for biotech.
Anything that would be software centric will probably still live under Google inc. As well, you are forgetting so many products that Google has. There are various lists of them out there [0][1].
Don't forget Google Life Sciences -- we have lots of software. (The team I'm on is 100% software, and we're around 25 people split among 3 sub-teams.) All our code is in piper, and depends heavily on the rest of the google infrastructure.
Is Google going to split this out of their infrastructure, or is this whole separation between "Alphabet" companies and "Google" really just a marketing ploy for no change at all?
At least as far as Google Life Sciences is concerned, there are multiple reasons that are known internally (at least within GLS) that I cannot comment on at this time.
An astute observer may be able to come up with some plausible reasons (at least for GLS) as to why such an arrangement would be desirable.
As far as splitting up the infrastructure, we're still going to depend on Google infrastructure and use their services where it makes sense.
The ranking algorithm (and other restricted stuff) is in Piper, but locked down as to who can browse it, check it out, etc. There's some magic in the build system that lets you compile/link against it without needing to see it.
There is considerable magic of that kind, actually, because the build system has to be able to prevent insufficiently-entitled engineers from being able to back out the protected source code, while still allowing the program depending on the protected code to be built, linked, and debugged.
I am unable to believe that Google has 2B lines of original code written from scratch at Google.
Maybe they are counting everything they use. Somewhere among those 2B lines is all the source code for Emacs, Bash, the Linux kernel, every single third-party lib used for any purpose, whether patched with Google modifications or not, every utility, and so on.
Maybe this is a "Google Search two billion" rather than a conventional, arithmetic two billion. You know, like when the Google engine tells you "there about 10,500,000 results (0.135 seconds)", but when you go through the entire list, it's confirmed to be just a few hundred.
Yes, that is counting everything. It's "the software needed to run all of Google’s Internet services" (so probably not Emacs, but the other stuff). But it's all in the repo, and it all has to be maintained.
Google has been around for 17 years and employees roughly 10,000+ software developers. I think it's reasonable to assume that the 2B LOC metric is accurate...
Windows has been around for 35 years and Microsoft had 61,000+ employees (ok, that’s not only software developers and they don’t work only on Windows) in 2005; and it’s only ~50M LOC. I don’t think the number of years + developpers really show something; you don’t write new code everyday.
You pointed it out yourself, but I think you underestimated its importance: Microsoft works on many other things. Office, XBox, Windows Phone, Exchange, SQL Server, .Net, etc. I suspect Microsoft's total line count is similar to Google's. The difference, however, is that it's not one codebase.
> Google engineers modify 15 million lines of code across 250,000 files each week
> Google’s 25,000 engineers
So employees modify 120 lines / day
if we imagine a linear growth in employees in 17 years to 25K coders, with 250 work day a year they employ around 6 more coders each work day, so about 55M man day, so around 6,3G LOC modified. But modified != added, so I wound't believe this is all their own lines.
Its interesting that they compare LoC with Windows. I suppose that this article wants us to be amazed at those numbers. However, my experience with Google's products indicates a gradual decline in performance and a simultaneous gradual increase in memory bloat (Maps, Gmail, Chrome, Android). Which ironically, FWIW, hasn't been the case with Windows. I have noticed zero difference in performance going from Windows 7 to 8 to 10.
I'd have to disagree with this. First the baseline: windows is very slow. Second I found later versions slower. Third (and most maddening) every version of windows I've ever used has gotten slower over time (including not installing new s/w and defragmenting).
8.1 and 10 run incredibly well even on very old hardware. I will agree a given Windows install may feel slower over time, and it makes sense to rebuild the PC occasionally, though that may, again, be less so with 8.1 and 10.
I think Microsoft has tried to address this with 8/10, which have a "refresh" feature, which tries to clean out everything besides Modern apps and your personal data.
I think it's gotten better over time, but non-technical users have an incredible willingness to push buttons they shouldn't. Unfortunately, your options are to either cripple your product's functionality (Chromebook) or accept that users have the power to do things that they shouldn't. I vastly prefer the latter, and I think the latter is better for consumers, in fact, as well. But to each their own.
I know that people have had experiences similar to yours. It's fine to disagree, but AFAIK pretty much all benchmarks show that there is no noticeable difference in performance from 7 to 8 to 10 and this matches with my own experience. I refuse to upgrade unless I get similar or better performance. But then again, I'm not really interested in researching every single benchmark. Windows is fast, stays fast, and that's pretty much all I care about.
Windows is slow? Compared to what? In what task? Running a game? Boot time? Opening Firefox?
I have problems with Windows, but it's the fastest desktop os I think, mostly because it's graphics stack is way the best of all. Running a number crunching C code is exactly the same on Windows or Linux. (See all the benchmarks on the Internet.)
It's really not that fast. The filesystem is a total dog (MFT contention) to the point that manipulating lots of small file is up to two orders of magnitude slower than ext4. This is made bearable thanks to SSDs being on the market. Also the amount of friction getting stuff built and running and maintaining it is detrimental to general productivity meaning you piss execution time out of the window regularly just fixing stuff.
Note: windows programmer for 19 years now. Only because of the cash.
I can't say I'm surprised to see people eager to point out how Windows sucks. And sure, maybe it does. However, the fundamental point you're missing is that I don't think that Windows was ever positioned as this OS that was designed for every single type of workload out there (not withstanding marketing noise). Windows is a very general purpose OS meant for general purpose 'mainstream' things. Things that hundreds of millions of people might want to do. Specialty workloads are simply not what Microsoft is ever going to invest any significant amount of time in - unless they see some money there. In that sense, Windows would probably be a far better OS if users could modify it to suit their needs, but thems the breaks. Linux seems to fill that void for some.
The disadvantage of NTFS which you point out, isn't because of a fuckup. It's not designed for your use case. You might even find Microsoft telling you that themselves here :- https://technet.microsoft.com/en-us/library/Cc938932.aspx
As to your point about productivity, I can't comment without knowing specifics. As a primarily C++ programmer, I haven't run into any Windows showstoppers that prevented me from shipping. I have run into showstoppers with their dev tools, but I see them as separate from the OS.
Can you solve all the problems in ext4 that NTFS claims to solve? No you cant. I am not saying either of the systems is perfect nor either of them is horrible. They are perfect for the use cases they are designed for. If somebody had a file system structure that was unusual(say lots of small files) to NTFS, I think it was his mistake in treating it as a black box.
The problem is that a large number of small files is a very common use case. Even Windows itself consists of lots of small files i.e. the source code and WinSxS.
It should handle general scenarios consistently. We've had a few minor versions of NTFS and now ReFS. ReFS should solve this but it doesn't as it's a copy and paste of the NTFS code initially rather than a complete reengineering effort.
You don't need to defragment NTFS either, defragmenter runs in the background. Defragmentation is more an issue of driver and not fs itself, I suspect ext4 does some form of defragmentation itself otherwise I can imagine edge cases that could ruin its day.
I disagree. It isn't that it works well but merely that it is there. In fact the majority of Windows networks both corporate and small business I can safely say that it barely works and is usually a mismanaged unpatched mess or filled with crapware. Occasionally there's a nice tight network (our operations guys run a tight ship which is cool) but the general case is a pit of incompetence.
The mantra among the consults I've met in the UK is if you're charging by the hour, do it in .Net on Windows. If you're charging a fixed rate, use Linux and Python.
I'm not suggesting there is anything better for an end user but I'm pointing out that it doesn't work well enough.
I still use it however and have a fondness. The accumulated knowledge of fixes is incredibly valuable.
In fact the majority of Windows networks both corporate and small business I can safely say that it barely works and is usually a mismanaged unpatched mess or filled with crapware.
And yet, no vendor can hold a candle to Active Directory, which is the single best thing about running Windows in an enterprise.
You literally cannot manage SSO, patching, and config management on a non-Windows environment for more than a few hundred machine without the right tools. Shell scripts and Chef aren't going to cut it when you have 20,000 laptops to take care of.
You're 100% right. Sort of. I've built many an AD forest in my years and it is certainly powerful. However when your laptops drop off the domain after a month or so, things get hairy. That and numerous other problems such as an entire network of 5000 machines with TrustedInstaller.exe hanging at 100% CPU etc etc and it's not all smooth. I've got my fair share of horror stories there.
One of our enterprise clients has just bought 500 Chromebooks and we integrated OpenID in our application and no one has to deal with AD, SSO is sorted and zero management overhead. I really like this solution. If someone could build a standalone product with equal quality it would destroy Microsoft overnight. Their ops team is 4 people and two of them are network people to keep the pipes (and APs) working.
And of course there is FreeIPA and PolicyKit as a contender but that's not really there yet.
Well it's been two months and I haven't heard any complaining other than one ActiveX based drum scanning thing doesn't work. Turns out the scanner's direct descendant has a web API so we're integrating that.
To be fair they are financial point of sale machines so it's all process driven and they're not general purpose computers.
I'm just saying I'd like to see how it goes long-term. Particularly in terms of TCO. Google, AFAIK, doesn't have any vendors making Toughbook-style Chrome OS machines yet.
Haven't tested ext4 and ntfs drivers directly against each other, but an useful trick if you ever need to copy millions of small files from NTFS is to mount it on Linux because Linux driver can work with it way way faster than Windows one.
> Windows is slow? Compared to what? In what task?
I dual booted a laptop for a while with Vista. (I can't speak to anything later, because I use Linux now, and haven't looked back, so take the appropriate grain of salt.) So with Vista / Gentoo on exactly the same hardware (a Lenovo T61):
- boot time on Linux was orders of magnitude faster
- WiFi AP connect was significantly faster[1], esp. on resuming from suspend-to-RAM
- Windows had a tendency to swap things out if they weren't in use, and had to swap like crazy if you paged back to a program you hadn't used in a while; Linux, by comparison, will only swap if required to due to memory pressure.
[1] i.e., WiFi was reconnected before I could unlock the screen. No other OS I've had has been able to do this, and it's bliss.
> mostly because it's graphics stack is way the best of all.
Riiiight. The T61 had an nvidia in it, and it was fairly decent; drivers were decent between the two OSs, and performance on each was about on par with the other. (I used the proprietary drivers; nouveau performed unacceptably bad — bear in mind this was 7 years ago.)
> Running a number crunching C code is exactly the same on Windows or Linux. (See all the benchmarks on the Internet.)
This I will agree with; but what do you do after the number crunching? It's the scaffolding around the program that mattered to me: Linux has a real shell, with real tools. I can accomplish the odd task here or there. But yes, running a "number crunching C code" will perform about equally: you're really only testing the processor, maybe the memory — crucially, the hardware, not the OS.
Well, in all fairness, isn't this true of Linux too? At login, only lightdm (or whatever) is ready; we've not yet loaded the WM, mate-panel, other random stuff, etc.
A giant repo works for Google, and works for Facebook, and Microsoft, but it's bad for the development community at large.
If you start centralizing your development you’re killing any type of collaboration with the outside world and discouraging such collaboration between your own teams.
346 comments
[ 2.9 ms ] story [ 254 ms ] threadAlthough I kind of doubt that "almost every" engineer has access to the entire repo, especially when it comes to the search ranking stuff.
"There are limitations this system. Potvin says certain highly sensitive code—stuff akin to the Google’s PageRank search algorithm—resides in separate repositories only available to specific employees."
> (...) all 2 billion lines sit in a single code repository available to all 25,000 Google engineers.
When we're not talking about the sensitive stuff, there's not much magic to what many engineers write every day, it's the same "glue technology X to technology Y" stuff you see everywhere, so I don't think there's any value to hiding that in the name of secrecy.
The most famous corporate trade secret, the Coke formula, was stolen by two employees who attempted to sell it to Pepsi. Pepsi alerted Coke, the companies worked together to bring in the FBI, and both employees went to prison: http://www.cnn.com/2007/LAW/05/23/coca.cola.sentencing/
For example, android and chrome are git based.
Note also that when codesearch used to crawl and index the world's code, it was not actually that large. It used to download and index tarballs, svn and cvs repositories, etc.
All told, the amount of code in the world that it could find on the internet a few years ago was < 10b lines, after deduplication/etc.
So while you may be right or wrong, i don't think it's as obvious you are right as you do.
If I wanted to change something fundamental, like I found a 10% speedup in Protobuf wire decode by changing the message slightly, there are likely very many services that all need it.
Everyone at Google operates on HEAD. You're not allowed to break HEAD, and pre-submit/post-submit bots ensure you don't and will block your submit.
I admit that I'm not an expert at large software development, but this seems to nearly fully explain Google's declining code quality.
Assume the code is sorted well (it is). Why does breaking them into "proper" repositories help anything?
What does "proper" even mean when tons of stuff is shared?
You act as if there is some obvious split that if they had "done it the right way" they would have done. That is 100% not obvious to me.
How do you demarcate these lines, maintain things across these boundaries, etc.
Why Mercurial instead of Git?
Edit: And for those that are just shocked that git isn't the answer.
Facebook: https://code.facebook.com/posts/218678814984400/scaling-merc...
Google: http://www.primordia.com/blog/2010/01/23/why-google-uses-mer...
Mercurial was pushed internally as being the "better" (for some dimension of better) between it and Git back in 2010, but I think even the most hardline Mercurial fans have realized that in order to meet developers in the middle in 2015, we need to use Git for our open-source releases. We have a large investment in Gerrit [1] and Github [2] now.
So the Mercurial comment is probably entirely based on scaling and replacement for the Piper Perforce API, rather than anything externally facing.
[1] https://www.gerritcodereview.com/ [2] https://github.com/google
https://selenic.com/hg/log?rev=@google.com&revcount=200
I always see a lot of Google and Facebook at the Mercurial sprints.
But if I were to take a poll of the informed and opinionated engineers sitting around me, they would almost all likely take git over hg.
In any case, git is in actual widespread use _now_ at Google, for Android and Chrome, and other open source stuff (some of which was moved recently from code.google.com to github).
The commands are still basically the same as Google's wrappers around Perforce and the learning curve was non-existent for someone who was used to using Perforce at Google.
One thing mentioned in the paper, but not mentioned here, is that there are teams that live in GIT, such as android and chrome, but they are not monolithic.
Git isn't so awesome that it's inconceivable that people would be willing to use something else.
Git and HG are not. Not that they can't be learned, but the learning curve is much higher.
Piper and CitC (mentioned in Rachel Potvik's videO) are even more advanced, I can work from my desktop machine, then go and continue from my laptop, or open and edit files directly from critique in an internal web-based editor.
Key to all this - make it accessible through the web! Anyone can work then from almost any machine. But at the same time the sources is accessible in the file system - even more awesome - so i can use emacs, and whatever other tools are there.
It's a legitimate question, damn it. Especially if the chosen solution is not mainstream.
You don't pick non-mainstream solutions "just because". You do pick mainstream solutions "just because". As in, "just because" they are tried and true, "just because" it's familiar to devs, etc.
So let me ask the reverse question: Is that what the HN crowd [sic] has turned into? You can no longer ask the reasons behind your tech choices?
Inconceivable? No, I just asked why. Git to me is a great solution for source control, so I want to understand it's deficiency for handling large amounts of code, especially considering it is used for managing Linux source and handles everything in GitHub I've used quite well.
There's also internal-only things to deal with, such as the CitC integration. Because of some of the design decisions behind CitC, storing a typical .git or a .hg directory in CitC is essentially impossible. Mercurial's .hg directory is intentionally a black-box - you interact with the repo using the hg tool. The .git directory can be seen as an API - there are at least three implementations that matter for our purposes, and if we change something in the .git directory, we lose the editor/IDE integration powered by the ones we didn't fix (or we did fix, but they haven't been released yet; some of those are linked into products that have a commercial release cycle).
I know there are various workarounds for dealing with large repos but having had some experience using git on big-ish projects I can certainly understand some possible reasons why it wasn't their first choice given the size of their codebase.
piper grew out of a need to scale the source control system the initial internal repositories were using
code.google.com was a completely separate thing supporting completely different version control models, and a very different scale (very large number of small repositories, vs very small number of very large repositories)
Still, this is not a very forward-thinking solution. Building and combining microservices – effectively UNIX philosophy applied to the web – is the most effective way to make progress.
EDIT: Seems like I misunderstood the article – from the way I read it, it sounded like Google has a monolithic codebase, with heavily dependent products, deployed monolithically. As zaphar mentioned, it turns out this is just bad phrasing in the article and me misunderstanding that phrasing.
I take everything back I said and claim the opposite.
Or think about April 1st, when they set a Access-Control-Location: * header on google.com because someone wrote the com.google easteregg.
Read the post from the SoundCloud dude from yesterday to find out how to do software management properly (hint: modularization is everything)
Neither HN search nor Google search show anything to "modularization is everything".
http://philcalcado.com/2015/09/08/how_we_ended_up_with_micro...?
(That question mark is part of the URL)
It explains how even in a small company modularization can help extremely. Now look at Google, where some issues (like the google.com april 1st XSS issue) were only fixed after outsiders mentioned it.
Usually internally the team responsible for that part should have cought that.
The issue was that they wanted to load the page – with the user logged in, etc – on com.google. For this they implemented an explicit URL parameter that would allow this.
The horror!
If that’s not the case, I apologize for misunderstanding it.
But if it was the case, I wanted to state that it might not be wise, for the same reasons as this thread mentioned https://news.ycombinator.com/item?id=10195423
EDIT: Thanks for telling me, though! Always nice to be proven wrong, as at least I learnt something today :D
You should think of Piper as a single filesystem which permits atomic multi-file edits. And that's about it; there's nothing in that which forces any particular release structure on you.
I wonder how much time it takes to clone the repo, provided they use git.
The bad news was that it allowed people to say "I've just changed the API to <x> to support the <y> initiative, code released after this commit will need to be updated." and have that effect hundreds of projects, but at the same time, the project teams could do the adaptation very quickly and adapt. With the orb on their desk telling them at that their integration and unit tests were passing.
I thought to myself, if there is ever a distributed world wide operating system / environment, it is going to look something like that.
The Linux kernel generally uses this policy for internal APIs for example.
I mean, it's presumably impossible to have a single computer running a single OS build all of the Google software and run the testing.
Does it really make sense to make one developer update those hundred projects? Not only will it take forever for it to get finished (possibly never if there are new consumers of this api coming online frequently), but the developer of the core api may not have any experience in the impacted consumers of this codebase. I think the end result of this policy would be nothing once written ever would get updated, and new apis would just be added all the time (api explosion).
Not that it's a silver bullet, but it can make a lot of these cases non-issues.
If it was any other way you'd rapidly reach a useless equilibrium where random engineers were demanding that thousands of other engineers fulfill unfunded mandates for what might turn out to be negligible benefits.
Having people unaware of a project's purpose making changes to its code sounds like a nightmare to me.
It's a nice environment to work in. In addition to hastening Noogler onboarding, it also increases employee retention. If you are an expert in your project's codebase but get burned out, you can easily transfer to another project and be almost immediately productive.
Obviously, there's domain-specific knowledge that doesn't transfer easily or quickly from project to project. But that's quite different from self-inflicted code fragility; one's an asset and the other's a liability.
Anymore info on this? Is it in house hardware?
http://ambientdevices.myshopify.com/products/stock-orb
https://blink1.thingm.com/
Here I am ready to deploy some new feature to gmail. In the meantime, I'm getting a steady stream of API changes. Can I build and release gmail to a specific revision number, and only incorporate the changes when I am ready, or are all release essentially off of the tip of the tree.
I don't need specifics, just the general idea. Where we work basically every project lives in its own branch, which makes it essentially impossible to synchonize changes. Things have to get merged to trunk, then pulled into the appropriate branch, and I don't like it at all.
Do you have version numbers for libraries/components: projext X uses version 1.5.4 of Y, 32.4.18 of Z, and so on, do you pull by revision number, are you all on tip?
It'd be very interesting to hear how you manage this.
How are conflicts managed?
Local branches for development, a single HEAD for merging code.
When you write a feature, you follow these steps.
1. Write code and submit to the main (and only) branch. Hide your feature behind a flag. 2. Releases happen from head (with no regard to your feature) 3. When you want to enable your feature, you flip a flag to turn it on.
In practice: Things are messier.
https://www.youtube.com/watch?v=W71BTkUbdqE
Are libraries and large projects e.g. RDBMS generally vendored/forked into the monolithic repositories, regardless of whether the initial intent is to make significant changes?
So, for source deliveries:
https://chromium.googlesource.com/chromium/src.git/+/master/...
I also wonder if there's a circular relationship anywhere in there.
https://sites.google.com/site/steveyegge2/ten-predictions
I'm wondering if microservices force you to coordinate via the codebase just like using many codebases force you to coordinate via the monolithic service. Does the coordination has to happen somewhere? I wonder if early adopters of microservices in many codebases (SoundCloud) are experiencing coordination problems trying to change services.
The biggest advantage seems to be that when you are an author of a dependency you can propose upgrades to all services that use your application. It is not clear to me but it seems that for small changes you can just force that change on the code owners. This ensures that the dependency author incurs the cost of a change (as is done for API changes in the Linux kernel) and that you do not need to version the API of the dependency.
Interestingly Google recently started marking API's private by default. So they are moving in the direction of explicit API management.
As soon as you work with people that are outside your control (as is common in open source) you would need to version the API as well in my opinion.
(Just one example of how it's useful even when things are mostly services.)
Tests would run on all the clients and, since in my workspace the server was updated simultaneously, I could be more sure it would work.
Think python 2->3 or Angular 1->2. These types of changes do happen, and I bet they happen at Google. I don't think anyone is rewriting a downstream app when they make these changes. Most likely they are doing something like forking the library and renaming it, which is just another form of versioning.
https://isocpp.org/blog/2015/05/cppcon-2014-large-scale-refa...
http://research.google.com/pubs/pub41342.html
It also encourages writing tools for automating this stuff. A big part of the motivation for polyup (https://github.com/PolymerLabs/polyup) was the fact that we'd be responsible for getting people to upgrade.
Whether the CODE BASE is monolithic or not is orthogonal to the repository's nature. I was G for a couple years and I'd say they've done an ok job of breaking things up into libraries and services. Certainly there are interfaces that have done a better or worse job of setting the code up for open sourcing but because of the nature of the repository, large scale refactoring is more efficiently accomplished.
I'm wondering if microservices force you to coordinate via the codebase just like using many codebases force you to coordinate via the monolithic service. Does the coordination has to happen somewhere? I wonder if early adopters of microservices in many codebases (SoundCloud) are experiencing coordination problems trying to change services.
How do you guys manage alerts and messages - does every developer get a commit notification,or is there a way to filter out messages based upon submodule.
How does branching and merging work?
I'm wondering what processes are used by non-Google/FB teams to help them be more productive in a monolithic repo world.
If you would like to see how things would work with submodules that behaved just like files behave (full distributed workflow) we've got a (unfortunately commercial) solution here:
http://www.bitkeeper.com/nested
As for notifications, the CL has a list of reviewers and subscribers. If you want to see code changing, you watch those CLs. Most projects have a list where all submitted CLs go.
There is a solution for project/directory-level CC / review requirements. I didn't see it discussed in the talk, though.
There are also tools for making large scale changes safely and quickly.
In general you change the much depended on library and all of its consumers (probably over time in multiple changes, but you can do it in one go if it really needs to be a single giant change).
I remember seeing an internal page with dozens of links for humorous searches like "interger", "funciton", or "([A-Z][a-z]+){7,} lang:java"...
Yeah this one was my favorite of the code search examples, there are some really good ones in there.
E.g., ProjectPotatoLoginPageBuilderFactoryObserver.
(Disclaimer: I just made it up. Not an actual Google project name.)
"lang: java" is not a part of regexp; just a Google code search extension that searches for Java.
(But as great as Google Code Search was, my grudge is because of Reader.)
Part of my job- although it's not listed as a responsibility- is updating a few key scientific python packages. When I do this, I get immediate feedback on which tests get broken and I fix those problems for other teams along side my upgrades. This sort of continuous integration has completely changed how I view modern software development and testing.
* I can see literally _every use_ of the old function.
* I can run the tests for everyone who uses that function.
* * this is automated; the build/test tooling can figure out the transitive set of build/test targets that are affected by such a change.
* I can (relatively) easily update _every use_ of the deprecated call with the new hotness
* I can do that all within the same commit (or set of commits, realistically)
---
None of this is impossible with multiple repos, it's just a lot more difficult to coordinate
It's sort of like a partial clone, you can clone any subset you want, work on it, add in other parts, the tool takes care of making sure the stuff you add in is lined up (if you cloned a week ago and you add in another repo, it's rolled back so it matches time/commit wise).
If you want to search for something it's the same command in a collection of repos as it is in a monolithic repo:
but truth in advertising, that searches only what you have populated. The google answer is "ship grep to the data" so they'll search everything. Google wins if you have their datacenter. We win on UI, it's the same everywhere.Our design is much more tightly coupled than Git's submodules. We manage the subrepos so that they are in sync just like files are in sync. What I mean by that is if you have two files modified in the same commit, you can't pull that commit and only get one of the files updated, they are both updated because that's what happened in that commit. We've provided the same semantics for collections of repositories. Git doesn't, getting those semantics is an exercise left to the user.
We get better performance because you can clone as little as one repository, what we call the product. It would be easy and fun to put a Unix distro in this system and have the top repo just have a makefile and you type make X11 and it populates the compiler, libc, the X11 sources, and builds them.
It's commercial so maybe that's uninteresting but if you want submodules that work and scale in performance, we might be worth a look (sorry for the marketing, if that's against the rules here then I'll learn how to delete a post).
Read more here, comments welcome, it's beta copy:
http://www.bitkeeper.com/nested
Bazel test[1] doesn't provide dependency testing as far as I know, but it creates the framework to support doing it.
[0] http://bazel.io/docs/build-encyclopedia.html
[1] http://bazel.io/docs/test-encyclopedia.html
Working on the google codebase is pretty awesome. This week I've made changes/improvements to libraries owned by three different teams (not counting my own), in C++ and python, when my main project is all in Java. It's super fun. The code search tool is great - it's ridiculously fast, and makes navigating through the codebase very easy.
Also, is the code that you add to the repository always inspected by other people? Is that done systematically?
OWNER and readability approvals may require additional reviewers but not always.
It varies a lot by project and by how widely used the code is.
There are few (if any) strictly enforced rules for documentation or invariant checking. You basically have to convince at least one other engineer that what you have is sufficient.
The documentation is _generally_ pretty decent for core libraries, but sometimes you just have to read the code.
* Google X - moonshot projects that aren't software centric.
* Google Fiber - mainly a infrstructure setup. May share some stuff in piper.
* Google Ventures - investment arm, not code related.
* Google Capital - more investment stuff.
* Calico - R&D for biotech.
Anything that would be software centric will probably still live under Google inc. As well, you are forgetting so many products that Google has. There are various lists of them out there [0][1].
[0] http://www.google.com/about/products/
[1] https://en.wikipedia.org/wiki/List_of_Google_products
An astute observer may be able to come up with some plausible reasons (at least for GLS) as to why such an arrangement would be desirable.
As far as splitting up the infrastructure, we're still going to depend on Google infrastructure and use their services where it makes sense.
Maybe they are counting everything they use. Somewhere among those 2B lines is all the source code for Emacs, Bash, the Linux kernel, every single third-party lib used for any purpose, whether patched with Google modifications or not, every utility, and so on.
Maybe this is a "Google Search two billion" rather than a conventional, arithmetic two billion. You know, like when the Google engine tells you "there about 10,500,000 results (0.135 seconds)", but when you go through the entire list, it's confirmed to be just a few hundred.
So employees modify 120 lines / day if we imagine a linear growth in employees in 17 years to 25K coders, with 250 work day a year they employ around 6 more coders each work day, so about 55M man day, so around 6,3G LOC modified. But modified != added, so I wound't believe this is all their own lines.
are you in CO?
Hence why his post is being downvoted.
I have problems with Windows, but it's the fastest desktop os I think, mostly because it's graphics stack is way the best of all. Running a number crunching C code is exactly the same on Windows or Linux. (See all the benchmarks on the Internet.)
Note: windows programmer for 19 years now. Only because of the cash.
The disadvantage of NTFS which you point out, isn't because of a fuckup. It's not designed for your use case. You might even find Microsoft telling you that themselves here :- https://technet.microsoft.com/en-us/library/Cc938932.aspx
As to your point about productivity, I can't comment without knowing specifics. As a primarily C++ programmer, I haven't run into any Windows showstoppers that prevented me from shipping. I have run into showstoppers with their dev tools, but I see them as separate from the OS.
It should handle general scenarios consistently. We've had a few minor versions of NTFS and now ReFS. ReFS should solve this but it doesn't as it's a copy and paste of the NTFS code initially rather than a complete reengineering effort.
The mantra among the consults I've met in the UK is if you're charging by the hour, do it in .Net on Windows. If you're charging a fixed rate, use Linux and Python.
I'm not suggesting there is anything better for an end user but I'm pointing out that it doesn't work well enough.
I still use it however and have a fondness. The accumulated knowledge of fixes is incredibly valuable.
You literally cannot manage SSO, patching, and config management on a non-Windows environment for more than a few hundred machine without the right tools. Shell scripts and Chef aren't going to cut it when you have 20,000 laptops to take care of.
One of our enterprise clients has just bought 500 Chromebooks and we integrated OpenID in our application and no one has to deal with AD, SSO is sorted and zero management overhead. I really like this solution. If someone could build a standalone product with equal quality it would destroy Microsoft overnight. Their ops team is 4 people and two of them are network people to keep the pipes (and APs) working.
And of course there is FreeIPA and PolicyKit as a contender but that's not really there yet.
To be fair they are financial point of sale machines so it's all process driven and they're not general purpose computers.
Haven't tested ext4 and ntfs drivers directly against each other, but an useful trick if you ever need to copy millions of small files from NTFS is to mount it on Linux because Linux driver can work with it way way faster than Windows one.
I dual booted a laptop for a while with Vista. (I can't speak to anything later, because I use Linux now, and haven't looked back, so take the appropriate grain of salt.) So with Vista / Gentoo on exactly the same hardware (a Lenovo T61):
- boot time on Linux was orders of magnitude faster
- WiFi AP connect was significantly faster[1], esp. on resuming from suspend-to-RAM
- Windows had a tendency to swap things out if they weren't in use, and had to swap like crazy if you paged back to a program you hadn't used in a while; Linux, by comparison, will only swap if required to due to memory pressure.
[1] i.e., WiFi was reconnected before I could unlock the screen. No other OS I've had has been able to do this, and it's bliss.
> mostly because it's graphics stack is way the best of all.
Riiiight. The T61 had an nvidia in it, and it was fairly decent; drivers were decent between the two OSs, and performance on each was about on par with the other. (I used the proprietary drivers; nouveau performed unacceptably bad — bear in mind this was 7 years ago.)
> Running a number crunching C code is exactly the same on Windows or Linux. (See all the benchmarks on the Internet.)
This I will agree with; but what do you do after the number crunching? It's the scaffolding around the program that mattered to me: Linux has a real shell, with real tools. I can accomplish the odd task here or there. But yes, running a "number crunching C code" will perform about equally: you're really only testing the processor, maybe the memory — crucially, the hardware, not the OS.
To be fair Vista was the slowest NT 6+ OS, especially booting is way faster on Windows 8+.
If you start centralizing your development you’re killing any type of collaboration with the outside world and discouraging such collaboration between your own teams.
http://code.dblock.org/2014/04/28/why-one-giant-source-contr...