That would still be 20-30k additional LOC per person year. Which is difficult to achieve especially in a corporate setting with a mature code base, without code generation, lots of blank lines, or including static data. Even in startup environments it's hard to get 20k-30k a year for several years from the average unless many of the engineers don't overlap.
I don't know about that. 100 lines a day, 280-or-so days a year, is pretty doable for the average L4. You could do that in migrations alone, and people do.
L5 or above there's normally a lot more communication and planning, which pulls down the mean significantly. In migrations you typically need to subtract some old code in the end, and it's not uncommon to result in net negative LOC (if the migration is ever complete).
For L4 maybe there are more days where one can do several hundred LOC in an afternoon than L5, but there are also many days that result in 100 or less over an entire week, because there's testing, debugging, profiling and figuring out what to code.
But, maybe I'd be wrong about Google. Afterall, it might be easier to dump lots of code into the codebase if there are 10 independent chat apps in development at any one time.
Yes but the point of contention was whether this can be done without code generation and static data. And probably both you and I agree it's quite difficult for a large organization to get 20k-30k net LOC additions per engineer without "conifg or autogen".
> Do they do an immense amount of code generation?
Blaze (aka Bazel [0]) has provisions that make it easy to generate code but this happens as a compile step rather than something that is checked into a git repo.
I believe the YouTube video linked from the wired article goes into this, but there's more machine generated code checked in than human generated code at this point. A lot of this is boilerplate or config data that is stored in source control.
I feel like Google definitely believes that config is code, so we have a ton of configuration of our systems checked into Piper.
If you commit all your codegen’d stuff you don’t need to worry about how to run codegen in most places. You can enforce a lot of security properties, you can run large scale analysis on the code in production because it’s all “statically” available, etc.
That said, I think Google’s source control ecosystem is really quite different to what’s publicly available with systems like Git/Hub/Lab etc, even to Phabricator. These approaches work well at Google but don’t necessarily translate to other tools in the same way.
Don’t take these sorts of things as gospel, or even as google advice about good engineering. Understand what they achieve, and then figure out a way to achieve that same result with the tools available to you.
Pure speculation on my part, but I've noticed Go has generate and build as separate steps and it's standard to check in generated code. This split no doubt makes compilation a lot simpler/faster, but perhaps it's a reflection of standard practices the language authors were used to.
Theory: it makes secure builds that much more straightforward. You can just point goma at a repo and whatever it inhales is everything needed. The simplicity cascades downwards.
For example, "everything comes from a repo" would make object (.o) caching that much more straightforward.
And you can reason both about the logic behind a service and how it was configured from a single commit hash/CL.
Sorry, I never checked the replies to this comment, but I just want to point out that I was just joking with my 10B statement and have no clue as to how much code is actually in google's repos. it may be 1T for all I know.
leaks of what exactly? It's billions of lines of code spread amongst millions of files which make up thousands of services... It's not like there 10 files with all the magic sauce.
To add to this, it's not all on every engineer's machine. It's not like they can just walk out of the office with a computer with all the source code. It's on a virtual file system that requests files as needed, and you can bet there are systems set up to detect if someone is just downloading the whole thing for no reason.
it's not even that. if you assume that the entire codebase was leaked verbatim, it would still take entrenched knowledge of many google engineers to replicate the same services and infrastructure to support those services.
If any state actor who stole that code could deploy such engineering capabilities, they don't really need the source code tbh.
What really happened is that China hacked Google, and it pissed Google off when they finally discovered it, many months later. This wasn’t some small intrusion or data breach. It was a systemic, coordinated, widespread, very deep hack, which among other things, gave China all of Google’s source code. It was very similar in scope and ambition to the Solar Winds hack recently out of Russia. It forced Google to completely rethink their security, which at the time was an Igloo model (hard on the outside, soft on the inside, a Gary Larson reference I think), and they had to migrate to where internal access was also limited, which took years.
I’ll share with you, confidentially wink wink, that most companies way overvalue their source code. It’s actually their engineers who are their biggest asset, because the engineers can reproduce the source code if it’s lost (quite quickly at that), whereas the source code can’t do fuck-all on its own. Moreover, most source code bases are so ugly that you couldn’t give them away. But Google had what they now call “HIP” (High-value Intellectual Property) scattered through their source code, which are the tuning parameters and constants for various algorithms and AI models which are the true Secret Sauce to Google’s Search and Ads dominance.
There are even advantages from a clean slate methodology; if there’s sufficient buy in from management.
Reason being: if you have done things before and bore the weight of architectural misgivings, you can clearly reproduce the software.
The biggest barrier to doing this normally is:
A) time pressure.
B) pressure to use existing log available tech
C) pressure to be cheap (which is manifested most often in A&B but is it’s own thing too).
This causes “large rewrites” to be rushed, subpar and lesser quality as the emphasis often isn’t on getting to feature parity.
Regardless, if google lost their code- entirely, we would probably have search by next week, auth in a fortnight, mail by next month and something approximating google cloud by the end of the year.
Because a lot of the really hard lessons have been learned, a reimplementation is just time.
At both small companies I worked at, the real value was in the business relationships built up (suppliers etc). The code could be rewritten. It wouldn't be bug-compatible, but all the ideas and design were backed up in engineers' minds.
I do. If, and only if, you build up the exact same thing. Once engineers band together for a rewrite that changes project architectural, conceptual, and engineering principles, then you're looking at a runaway deadline.
Most of snippets of Google's code wouldn't be usable without the entire ecosystem around them. Most things are doable because of libraries, internal infrastructure, etc.
It is still baffling to me how bad both cs.github and sourcegraph are compared to this. I mean when your entire business is browsing code why don't you have decent crossrefs? Bizarre, especially when you consider that the whole underlying infrastructure of google codesearch has been open-sourced.
I'm guessing that blaze (bazel.build/ outside) is the big difference.
You'd need to do a lot of educated guesses to properly build the cross-references, and within Google you can just build the code or whatever analyzer you need.
I believe that both cs.github and sourcegraph do not use proper semantic linking by running the compiler. They do it mostly off of text-based analysis, which by definition cannot do really good linking for a lot of programming langauges.
The cross references underneath Google codesearch (both internal and external) are created by, roughly speaking, partially compiling everything and groveling around in the AST.
> Bizarre, especially when you consider that the whole underlying infrastructure of google codesearch has been open-sourced.
Some of the code is also open sourced at kythe.io, though it lacks a considerable amount of the internal logic necessary to scale it out to all of google3. So it's not exactly plug & play. But, it should be enough for a company the size of Microsoft/Github to get started using it.
Sourcegraph actually does do proper semantic linking, integrating with compilers (and type inferencers for dynamic languages): https://docs.sourcegraph.com/code_intelligence/explanations/.... This is one area of significant differentiation between Sourcegraph and GitHub code navigation (which is based on GitHub's semantic library, and which ironically isn't actually semantic in the sense of understanding types as compilers do).
Admittedly, we need to do a better job of articulating this difference, but if anyone would like help getting spun up with precise code intelligence on Sourcegraph, please DM me: https://twitter.com/beyang.
Reading the sibling comments, I realized that GitHub and SG have built tooling to index text, while Google have built tooling to index C++.
cs.chromium et al experiences instantaneous dropoff from "wow this is kind of amazing" the moment you start wading through JavaScript or Mojo glue code. Here's the JS source of the dino game; notice how you can't click anything: https://source.chromium.org/chromium/chromium/src/+/main:com...
A quick Google for "most popular languages on github" just found https://madnight.github.io/githut/, which (if it's correct) reveals that GitHub's most popular language (by commit activity) is Python (17%), very closely followed by JavaScript (14%). Then you have Java (12%), TypeScript (8%), Go (8%), C++/Ruby (both 6%), and PHP (5%). So on GitHub C++ apparently represents 6% of PR activity, while the top two languages (Python/JS) which represent 31% of PR activity are not only interpreted but also dynamically typed.
Which tells a very interesting story about the benefits of being able to generate an AST with type information: you can scale insight that much more broadly and deeply. :(
Clearly they're in the business of something else because the product isn't very compelling when it comes to browsing and searching.
You need to be able to build the code to even have a chance at that level of understanding of the source. And really you need to be able to build the code with a specialized toolchain. This is tractable with a huge investment if you get to choose or dictate your tools. If folks are bringing their own tools it is basically impossible.
Pushing for standardization of fact output from toolchains could be a path to getting somewhere when you can't dictate toolchains. Or really just having some kind of fact output from default toolchains would get you a long way.
Because different languages have different build systems, so inferring the right build commands, dependencies etc. is not so straightforward; these are necessary pre-requisites for compiler-accurate cross references. We're working on fixing this with auto-indexing: https://docs.sourcegraph.com/code_intelligence/explanations/...
For C and C++ specifically, auto-indexing is challenging because of the large variety in build systems, informal specification of dependencies (such as in a README instead of a machine-readable format), and platform-specific code.
Outside of auto-indexing, we do have an indexer for C and C++ right now (https://github.com/sourcegraph/lsif-clang) which can be run in CI; that way one can generate an index and upload it to Sourcegraph on a regular basis. It is 'Partially available' (https://docs.sourcegraph.com/code_intelligence/references/in...) right now. We're keenly aware of the interest in C++, and are working our way through different languages based on usage.
The coolest part is it's regex search. So you can do really neat things like answer "Does this codebase prefer ' or ` for surrounding code in comments?"
Hmm, no we should not. The default is already that any stated opinion is your own, disclosing ties is only necessary when you're expressing an opinion about a company you are actually working for.
This case, when an employee speaks about their employer in a public forum, is pretty rare, so rare in fact that people are surprised when they see disclosures -like in this very thread.
Yes, I suppose people do that through the official channels blessed by the employer? Are you likely to write any of your employer's opinions as 'vntok'? Why would you do that?
Comment in relation to the comments in this thread:
What is it with all the Googlers systematically prefacing their comments with "Opinions are my own"? Is that some kind of policy at Google? It's so ubiquitous now that it feels almost like some kind of humble brag.
If you identify yourself as a Googler without disclosing that you are not speaking on behalf of the company, you leave the window open for people to think you _are_. Which is a no-no of course.
The standard cop out is to say that line even when it’s completely unnecessary. It’s just easier to add it than consider how your words may be missunderstood.
At a high level there’s some dissonance between having such a strong bond with one’s employer that one would identify as an employee of theirs, socially, outside of work, but also claim to still be independent enough of the employer to have independent thoughts on technical matters directly related to the employer.
If you wore an I’m with her! tee, you would not claim to have a neutral opinion on politics.
If you work for Google, comment in public on Google stuff, with an account that says “I work for Google!”, I would say it is not possible for you to remain independent, no matter what kind of spell-of-disclaiming is attached to your post.
When it comes to Google matters, one either comments as a Google employee, or not at all. It comes across as distasteful to try to do wade in while waving off claims of bias with “opinions are my own” with the implication that they are unbiased.
Maybe what these people mean is: I work at Google so this is biased towards them, but they didn’t tell me to write this, and perhaps I should be reading it that way?
> I would say it is not possible for you to remain independent
Good thing nobody said this, then. They're just making it clear that they're not explicitly relaying company opinion.
> with the implication that they are unbiased.
Nobody's trying to imply this. Obviously if you work at company X, you're gonna have at least some subconscious bias. The point is to make your potential bias transparent, so others can judge for themselves.
> I work at Google so this is biased towards them, but they didn’t tell me to write this
Yes, though of course the amount of bias can vary. I used to work at Amazon, and my opinion of Amazon didn't seem to significantly change when I started working there, or when I left.
(Of course some people develop a more negative opinion of a company after working there, rather than a positive one)
I think you're misunderstanding. It's about making it very clear that you aren't speaking on behalf of the company. It might seem like anyone reasonable would understand that, but that would just mean there are a lot of unreasonable people in the world.
I'd say that a situation where you feel the need to add that sort of disclaimer generally means you should keep your mouth shut, but stopping people from talking about things they know about will never happen.
In other industries, you'll find similar boilerplate disclaimers like "this is not investment advice." Particularly amusing when it is attached to even character-limited tweets.
If someone spoke of Google -without- disclosing that Google is their employer, and that was later found out, do you think that might change your view of their statement?
To borrow your broken analogy and fix it, if someone were to post "Hey, here are all the reasons I'm voting for (candidate) and I think you should too", and didn't disclose that they work for that candidate's campaign, don't you think any credibility they might have had is gone once it's found out?
Who is claiming there opinions are unbiased? We are explicitly saying we probably are biased. It’s hard to be unbiased toward a company that is paying you over a quarter million dollars a year - I’m not claiming or not claiming that’s what I make (I’m not a “software engineer” by title), it’s just the typical compensation of even a mid level software engineer in tech in the US at one of the BigTech companies.
I assume you mean it seems biased because it implies that the poster has different views to Google even if they actually have the exact same views as Google. I kind of get that, it implies a level of distance which may not actually be true. However I do think that's at least a bit uncharitable. But also there is a good point there.
The point of the "my thoughts" disclaimer isn't to claim you're neutral. The point is to say the company hasn't approved your statement, so it shouldn't be taken as an official company position.
The point of the "I work at X" disclaimer is to explicitly point out that you do have a conflict of interest that the reader needs to be aware of.
Disclaimer: I work at Google and my thoughts are my own. ;)
Why would anyone presume a statement made by an individual to have been made on behalf of a company? (Unless it's on the company's blog or social media, or unless explicitly mentioned that the individual is speaking on behalf of the company.)
There are examples from nindalf and freedomben above.
Why do people think it? If they see me say "I work at Google" in the comment, and then say some sort of opinion or recommendation or prediction, they might think that that's an opinion that's widely held or officially promoted within Google, or that the recommendation is something that Google recommends or that the prediction is something Google predicts. The sentence "I work at Google" could be thought of as something I'm invoking to say "I'm an authority believe me", or "I work at Google, so I can provide you info from Google", rather than a disclaimer about a conflict of interest. That's why "I work at Google" is often followed by "but my thoughts are my own".
This is interesting. I think it's a healthy way of looking at things, and a good development. I'm always a bit curious about people complaining that "everyone defines themselves by their job", because I come from a blue collar background and I never wanted to do that - just the opposite. My job is what I do, not who I am. I write code; I'm good at it, but I don't view myself as any different from a car mechanic who's good at his job. And as I rose in rank, I never had the temptation to introduce myself as "CTO of X". But that wasn't because I wasn't proud of it. It's because I was suspicious of people. I've always wanted to see what other people made of me and how they acted towards me based on how they encountered me, without knowing my job or title. Partly this was to avoid people who are only interested in associating themselves with people they think can help them; partly it was to gauge whether I was still a decent and smart enough guy to be worth their time at face value. Occasionally it was to create simpatico with plumbers and mechanics. But I still refer to myself in most introductions as a former waiter, bartender and taxi driver who picked up some code in my free time, and unless someone knows me for a few months and I trust them, they wouldn't know what I actually did for a living. [edited for clarity]
It has nothing to do with “delusions of grandeur” or bragging about the company you work for. Most large tech companies tell you during your onboarding to disclose where you work and disclaim opinions as your own anytime you speak publicly about them.
Especially in my case, I do speak for the company in a limited scope when I am working with one of our customers or when I do approved communications in the form of blog posts, presentations, etc.
I’ve worked for 8 companies over 25+ years. The current company I work at (AWS) is another means for me to exchange labor for money.
Part of our employee agreement says we have to state it whenever we talk about Google in a public forum. I feel like it's meant to provide Google a level of protection in case we say something stupid.
Previous companies I worked at (like Cisco) explicitly said in my employment contract that I couldn't post things online about the company without prior approval from PR or whoever.
Part of why I do it is also so people know I may be biased as I'm financially motivated to have Google seen in a positive light and be successful.
An internal page on "posting about work on your personal accounts" has this as the first suggestion among 20 or so points about "dos and don'ts". Also includes sections such as "suggestions for content to post", "how to respond to questions from candidates" etc.
We are specifically told to disclaim that we don’t speak for AWS when we speak on social media about anything related to the company.
Even if not, if I say something positive or get in a discussion about AWS on Hacker News that is opinion based, I disclaim so people can consider my biases (“It is difficult to get a man to understand something when his salary depends on his not understanding it.”)
Also when I post on r/aws, I sometimes disclaim. It depends on the question.
If someone ask “how can I stop API Gateway from timing out after 30 seconds?” - ie something that can be answered factually (you can’t) and post documentation, I don’t bother about saying where I work.
On the other hand, if someone asks, “what are the steps I need to take to improve security on AWS?”. Then my usual reply starts off with, “I am a consultant in Professional Services at AWS, I’m not your consultant. All opinions are my own”. The last thing I need is to be in the middle of a controversy because a company got breached based on taking my very generic advice and saying they were listening to me.
Why did you feel compelled to disclose on HR where you work at all? I work at a large tech firm with a household name. I’ve never disclosed that here (and many other places). And I say what I want without any need for disclosure.
It’s more honest. When I got into discussions about “cloud” and specifically AWS before I started working at AWS two years ago, I was just telling my opinions. But it’s different now, I have a vested interest in AWS succeeding.
Wouldn’t you think it was kind of dishonest if I said “I think everyone should move to the cloud” (I wouldn’t say that) and not disclose my biases? Most of my arguments are much more subtle. But I would rather err on the side of disclosing than people think I’m astroturfing.
There are lots of people on HN that work at companies that get discussed all of the time. It would be dishonest if we didn’t say where we worked when talking about them.
So do you think that yev also shouldn’t say he works for Backblaze when he posts here frequently answering questions or discussing something company related?
It’s considered the honest thing to do as you’ve seen many of us do when we talk about the company we work for.
We all have biases. I was biased toward cloud computing when I worked for a 60 person startup. But if AWS had gone belly up then, I would shrugged and start working with Azure. Now I have a financial interest. Of course I’m going to be more biased.
Since you've never disclosed where, that makes it possible for me to ask: what's the tech firm's policy - or, to reduce the number of bits of entropy in your answer, does it at least stipulate that you should disclose that your opinions are your own, or specify that you need to clear things with HR?
If there's no actual policy for these sorts of things, that flips the coin and makes it very unfair that I don't know where you work, sounds like an interesting place :P
The company and employee both want it to be clear that the employee doesn’t speak for the whole company.
This is because people attribute the actions of one person to the whole group. An example from the HN page yesterday - Microsoft insults dev, then takes credit for his idea (https://news.ycombinator.com/item?id=31284419). The title was changed later by HN mods to something less incendiary.
But the point stands. 1 person working at Microsoft out of 182k said something sarcastic. That sarcasm was interpreted as an insult. Then the entire organisation was blamed (Microsoft insults dev). This happens because it’s easier for us to think reductively. Nuance is too much effort.
That’s why employees go to great lengths to add these disclaimers every comment.
Yep. I worked at Red hat for a while, and anytime I would comment on any post that had anything to do with something that Red hat had to do with, I would have people accuse me of being deceptive by not putting a disclaimer up, Even though in many of those cases it felt stupid to do so.
The point at which I started doing it religiously though, was after I made a comment about one of Red hat's competitors. Somebody wrote another post where they talked about Red hat employee Said certain things, so that's the red hat perspective or that's what red hat thinks or something like that. I want to make absolutely clear that everybody reading anything I write knows that I do not speak officially for Red hat. My opinions are truly my own.
I disclose. But I never ever say anything negative about a competitive division of another company. Meaning I might say something negative about Google, but not GCP. It just feels crass.
Yes I agree and I should clarify, it wasn't a negative comment that people started saying was "Red Hat's" opinion. I likewise don't say anything negative about a competitor (though I may state uncontroversial facts that could seem negative, like "xyz's quarterly earnings were a lot lower than wall street expectations and that sent their stock down." The comment I was referring to above was this (which was several years ago for time reference)):
"VMware seems to dominate the bare metal landscape. Even though Red Hat has a competing product, there's still a ton of customers that want to run OpenShift on top of VMware so we end up working with it a fair amount. They made some design decisions that I wouldn't have made, but overall it's solid technology if you don't mind proprietary."
I didn't (and still don't) think that was a negative comment. It felt very factual and fair to me. But "overall it's solid technology" is my opinion, not Red Hat's, and it seems ridiculous to me that people would say it was Red Hat's opinion, but ¯\_ (ツ)_/¯
I think its just a legal thing. You don't want "Google says ___" headlines based on internet comments from employees. In any case, if I was a Google employee, I'd want to hide the fact.
considering the rapid decline of quality of both the text and image searches, I don't think there are any people left at google that actually know how any of that works
Consider that your experience may not be representative. You might be part of the very low percent of people whom Google Search does not target and does not want to spend money on acquiring/retaining.
They know how it works, why do you think it's not working as designed. They do things to make a profit, not to make a good search engine [any more].
This sort of opinion appears rife somehow, people who make it clear their raison d'être is make money above all else are considered incompetent because, whilst making massive amounts of money, their actions are highly deleterious in other areas.
Do you just not believe that those in control only care about the money and couldn't care less about search quality other than to the extent it's necessary to make money?
> Do you just not believe that those in control only care about the money and couldn't care less about search quality other than to the extent it's necessary to make money?
I definitely believe this, but I also think companies fall victim to the classic quote from Upton Sinclair:
"It is difficult to get a man to understand something when his salary depends upon his not understanding it."
It would not surprise me if Googlers had a rationalization for why these "worse" search results are somehow a better user experience.
Other examples: Apple's tight control over the iPhone software experience is controversial. Facebook employees often say they're benefiting the world, while outsiders think they're harming it.
I don't think these are black-and-white - someone can credibly argue a case for each. But I think the Upton Sinclair quote tends to tip the balance for what insiders believe.
103 comments
[ 0.17 ms ] story [ 155 ms ] threadFor L4 maybe there are more days where one can do several hundred LOC in an afternoon than L5, but there are also many days that result in 100 or less over an entire week, because there's testing, debugging, profiling and figuring out what to code.
But, maybe I'd be wrong about Google. Afterall, it might be easier to dump lots of code into the codebase if there are 10 independent chat apps in development at any one time.
A 200 line change typically doesn't grow the code base by 200 lines.
I believe in my 2.5 years at Google, I've actually deleted more lines of code than I've added.
So working some week-ends and no vacations at all?
There are about 260 week days in a year. Add bank holidays and vacations and you should ve closer to 230 working days in a year.
> Do they do an immense amount of code generation?
Blaze (aka Bazel [0]) has provisions that make it easy to generate code but this happens as a compile step rather than something that is checked into a git repo.
[0] - https://bazel.build/
I believe the YouTube video linked from the wired article goes into this, but there's more machine generated code checked in than human generated code at this point. A lot of this is boilerplate or config data that is stored in source control.
I feel like Google definitely believes that config is code, so we have a ton of configuration of our systems checked into Piper.
But you know, there are tons of configs, all kinds of.
If you commit all your codegen’d stuff you don’t need to worry about how to run codegen in most places. You can enforce a lot of security properties, you can run large scale analysis on the code in production because it’s all “statically” available, etc.
That said, I think Google’s source control ecosystem is really quite different to what’s publicly available with systems like Git/Hub/Lab etc, even to Phabricator. These approaches work well at Google but don’t necessarily translate to other tools in the same way.
Don’t take these sorts of things as gospel, or even as google advice about good engineering. Understand what they achieve, and then figure out a way to achieve that same result with the tools available to you.
For example, "everything comes from a repo" would make object (.o) caching that much more straightforward.
And you can reason both about the logic behind a service and how it was configured from a single commit hash/CL.
Can someone explain why there have never been any leaks?
If any state actor who stole that code could deploy such engineering capabilities, they don't really need the source code tbh.
What really happened is that China hacked Google, and it pissed Google off when they finally discovered it, many months later. This wasn’t some small intrusion or data breach. It was a systemic, coordinated, widespread, very deep hack, which among other things, gave China all of Google’s source code. It was very similar in scope and ambition to the Solar Winds hack recently out of Russia. It forced Google to completely rethink their security, which at the time was an Igloo model (hard on the outside, soft on the inside, a Gary Larson reference I think), and they had to migrate to where internal access was also limited, which took years.
I’ll share with you, confidentially wink wink, that most companies way overvalue their source code. It’s actually their engineers who are their biggest asset, because the engineers can reproduce the source code if it’s lost (quite quickly at that), whereas the source code can’t do fuck-all on its own. Moreover, most source code bases are so ugly that you couldn’t give them away. But Google had what they now call “HIP” (High-value Intellectual Property) scattered through their source code, which are the tuning parameters and constants for various algorithms and AI models which are the true Secret Sauce to Google’s Search and Ads dominance.
Does anyone really believe this?
There are even advantages from a clean slate methodology; if there’s sufficient buy in from management.
Reason being: if you have done things before and bore the weight of architectural misgivings, you can clearly reproduce the software.
The biggest barrier to doing this normally is:
A) time pressure.
B) pressure to use existing log available tech
C) pressure to be cheap (which is manifested most often in A&B but is it’s own thing too).
This causes “large rewrites” to be rushed, subpar and lesser quality as the emphasis often isn’t on getting to feature parity.
Regardless, if google lost their code- entirely, we would probably have search by next week, auth in a fortnight, mail by next month and something approximating google cloud by the end of the year.
Because a lot of the really hard lessons have been learned, a reimplementation is just time.
At both small companies I worked at, the real value was in the business relationships built up (suppliers etc). The code could be rewritten. It wouldn't be bug-compatible, but all the ideas and design were backed up in engineers' minds.
1. All of Google's Source, All New Engineers
Vs.
2. All of Google's Engineers, No source
I'd probably bet on #2 getting up and running quicker.
Most of snippets of Google's code wouldn't be usable without the entire ecosystem around them. Most things are doable because of libraries, internal infrastructure, etc.
Try the powerful search, including free text search, operators like f: or symbol: (the full list of operators can be found at https://developers.google.com/code-search/reference)
Then try the references feature by clicking on symbols.
You'd need to do a lot of educated guesses to properly build the cross-references, and within Google you can just build the code or whatever analyzer you need.
But you're half right here, using blaze makes it way easier.
The cross references underneath Google codesearch (both internal and external) are created by, roughly speaking, partially compiling everything and groveling around in the AST.
One of our teammates Luke Zarko has a decent overview of this: https://www.youtube.com/watch?v=VYI3ji8aSM0
> Bizarre, especially when you consider that the whole underlying infrastructure of google codesearch has been open-sourced.
Some of the code is also open sourced at kythe.io, though it lacks a considerable amount of the internal logic necessary to scale it out to all of google3. So it's not exactly plug & play. But, it should be enough for a company the size of Microsoft/Github to get started using it.
Admittedly, we need to do a better job of articulating this difference, but if anyone would like help getting spun up with precise code intelligence on Sourcegraph, please DM me: https://twitter.com/beyang.
cs.chromium et al experiences instantaneous dropoff from "wow this is kind of amazing" the moment you start wading through JavaScript or Mojo glue code. Here's the JS source of the dino game; notice how you can't click anything: https://source.chromium.org/chromium/chromium/src/+/main:com...
A quick Google for "most popular languages on github" just found https://madnight.github.io/githut/, which (if it's correct) reveals that GitHub's most popular language (by commit activity) is Python (17%), very closely followed by JavaScript (14%). Then you have Java (12%), TypeScript (8%), Go (8%), C++/Ruby (both 6%), and PHP (5%). So on GitHub C++ apparently represents 6% of PR activity, while the top two languages (Python/JS) which represent 31% of PR activity are not only interpreted but also dynamically typed.
Which tells a very interesting story about the benefits of being able to generate an AST with type information: you can scale insight that much more broadly and deeply. :(
You need to be able to build the code to even have a chance at that level of understanding of the source. And really you need to be able to build the code with a specialized toolchain. This is tractable with a huge investment if you get to choose or dictate your tools. If folks are bringing their own tools it is basically impossible.
Pushing for standardization of fact output from toolchains could be a path to getting somewhere when you can't dictate toolchains. Or really just having some kind of fact output from default toolchains would get you a long way.
> why don't you have decent crossrefs
We do have compiler-accurate cross references for many repos. Some examples:
- TypeScript: https://sourcegraph.com/github.com/sindresorhus/got/-/blob/s...
- C: https://sourcegraph.com/github.com/neovim/neovim/-/blob/src/...
- C++: https://sourcegraph.com/github.com/KhronosGroup/Vulkan-Sampl...
Why are not all repos covered?
Because different languages have different build systems, so inferring the right build commands, dependencies etc. is not so straightforward; these are necessary pre-requisites for compiler-accurate cross references. We're working on fixing this with auto-indexing: https://docs.sourcegraph.com/code_intelligence/explanations/...
For C and C++ specifically, auto-indexing is challenging because of the large variety in build systems, informal specification of dependencies (such as in a README instead of a machine-readable format), and platform-specific code.
Outside of auto-indexing, we do have an indexer for C and C++ right now (https://github.com/sourcegraph/lsif-clang) which can be run in CI; that way one can generate an index and upload it to Sourcegraph on a regular basis. It is 'Partially available' (https://docs.sourcegraph.com/code_intelligence/references/in...) right now. We're keenly aware of the interest in C++, and are working our way through different languages based on usage.
https://source.chromium.org/search?q=comment:%27%5B%5E%5C%5C...
https://source.chromium.org/search?q=comment:%60%5B%5E%5C%5C...
In this case... about even unfortunately lol.
Disclosure: I work at Google.
I also highly recommend this ACM article talking about our source control from 2016.
https://m-cacm.acm.org/magazines/2016/7/204032-why-google-st...
There's also a YouTube video which is linked from the wired article that is good.
https://youtu.be/W71BTkUbdqE
This case, when an employee speaks about their employer in a public forum, is pretty rare, so rare in fact that people are surprised when they see disclosures -like in this very thread.
What is it with all the Googlers systematically prefacing their comments with "Opinions are my own"? Is that some kind of policy at Google? It's so ubiquitous now that it feels almost like some kind of humble brag.
The standard cop out is to say that line even when it’s completely unnecessary. It’s just easier to add it than consider how your words may be missunderstood.
If you wore an I’m with her! tee, you would not claim to have a neutral opinion on politics.
Someone literally explained to you that it's a matter of transparency, and you're jumping to, "well obviously they're just bootlickers".
If you work for Google, comment in public on Google stuff, with an account that says “I work for Google!”, I would say it is not possible for you to remain independent, no matter what kind of spell-of-disclaiming is attached to your post.
When it comes to Google matters, one either comments as a Google employee, or not at all. It comes across as distasteful to try to do wade in while waving off claims of bias with “opinions are my own” with the implication that they are unbiased.
Maybe what these people mean is: I work at Google so this is biased towards them, but they didn’t tell me to write this, and perhaps I should be reading it that way?
Good thing nobody said this, then. They're just making it clear that they're not explicitly relaying company opinion.
> with the implication that they are unbiased.
Nobody's trying to imply this. Obviously if you work at company X, you're gonna have at least some subconscious bias. The point is to make your potential bias transparent, so others can judge for themselves.
> I work at Google so this is biased towards them, but they didn’t tell me to write this
Yes, though of course the amount of bias can vary. I used to work at Amazon, and my opinion of Amazon didn't seem to significantly change when I started working there, or when I left.
(Of course some people develop a more negative opinion of a company after working there, rather than a positive one)
I'd say that a situation where you feel the need to add that sort of disclaimer generally means you should keep your mouth shut, but stopping people from talking about things they know about will never happen.
In other industries, you'll find similar boilerplate disclaimers like "this is not investment advice." Particularly amusing when it is attached to even character-limited tweets.
To borrow your broken analogy and fix it, if someone were to post "Hey, here are all the reasons I'm voting for (candidate) and I think you should too", and didn't disclose that they work for that candidate's campaign, don't you think any credibility they might have had is gone once it's found out?
What I’m getting at is that opining, disclosing, but then claiming the opinions are unbiased just does not compute.
If it's positive, then leaving off the disclaimer is basically astroturfing.
If you're wearing an I'm with her! tee, you also aren't her spokesperson.
It's not neutrality, it's non-representative
The point of the "I work at X" disclaimer is to explicitly point out that you do have a conflict of interest that the reader needs to be aware of.
Disclaimer: I work at Google and my thoughts are my own. ;)
Disclaimer: All my opinions are Google's ;)
Why do people think it? If they see me say "I work at Google" in the comment, and then say some sort of opinion or recommendation or prediction, they might think that that's an opinion that's widely held or officially promoted within Google, or that the recommendation is something that Google recommends or that the prediction is something Google predicts. The sentence "I work at Google" could be thought of as something I'm invoking to say "I'm an authority believe me", or "I work at Google, so I can provide you info from Google", rather than a disclaimer about a conflict of interest. That's why "I work at Google" is often followed by "but my thoughts are my own".
The phrase "identify as" refers to considering that a nontrivial part of one's identity.
These are separate concepts that happen to use similar wording.
I remember a time when I was younger and I too would present much of my persona as being based in the company for which I worked.
As time passes this fades significantly. I feel much more like my own person now, from whom my employer benefits, and not the other way around.
Delusions of grandeur? Moi? Or the true wisdom of age? Why not both!
Especially in my case, I do speak for the company in a limited scope when I am working with one of our customers or when I do approved communications in the form of blog posts, presentations, etc.
I’ve worked for 8 companies over 25+ years. The current company I work at (AWS) is another means for me to exchange labor for money.
Previous companies I worked at (like Cisco) explicitly said in my employment contract that I couldn't post things online about the company without prior approval from PR or whoever.
Part of why I do it is also so people know I may be biased as I'm financially motivated to have Google seen in a positive light and be successful.
https://news.ycombinator.com/item?id=31279881
We are specifically told to disclaim that we don’t speak for AWS when we speak on social media about anything related to the company.
Even if not, if I say something positive or get in a discussion about AWS on Hacker News that is opinion based, I disclaim so people can consider my biases (“It is difficult to get a man to understand something when his salary depends on his not understanding it.”)
Also when I post on r/aws, I sometimes disclaim. It depends on the question.
If someone ask “how can I stop API Gateway from timing out after 30 seconds?” - ie something that can be answered factually (you can’t) and post documentation, I don’t bother about saying where I work.
On the other hand, if someone asks, “what are the steps I need to take to improve security on AWS?”. Then my usual reply starts off with, “I am a consultant in Professional Services at AWS, I’m not your consultant. All opinions are my own”. The last thing I need is to be in the middle of a controversy because a company got breached based on taking my very generic advice and saying they were listening to me.
Wouldn’t you think it was kind of dishonest if I said “I think everyone should move to the cloud” (I wouldn’t say that) and not disclose my biases? Most of my arguments are much more subtle. But I would rather err on the side of disclosing than people think I’m astroturfing.
There are lots of people on HN that work at companies that get discussed all of the time. It would be dishonest if we didn’t say where we worked when talking about them.
It’s considered the honest thing to do as you’ve seen many of us do when we talk about the company we work for.
We all have biases. I was biased toward cloud computing when I worked for a 60 person startup. But if AWS had gone belly up then, I would shrugged and start working with Azure. Now I have a financial interest. Of course I’m going to be more biased.
If there's no actual policy for these sorts of things, that flips the coin and makes it very unfair that I don't know where you work, sounds like an interesting place :P
This is because people attribute the actions of one person to the whole group. An example from the HN page yesterday - Microsoft insults dev, then takes credit for his idea (https://news.ycombinator.com/item?id=31284419). The title was changed later by HN mods to something less incendiary.
But the point stands. 1 person working at Microsoft out of 182k said something sarcastic. That sarcasm was interpreted as an insult. Then the entire organisation was blamed (Microsoft insults dev). This happens because it’s easier for us to think reductively. Nuance is too much effort.
That’s why employees go to great lengths to add these disclaimers every comment.
The point at which I started doing it religiously though, was after I made a comment about one of Red hat's competitors. Somebody wrote another post where they talked about Red hat employee Said certain things, so that's the red hat perspective or that's what red hat thinks or something like that. I want to make absolutely clear that everybody reading anything I write knows that I do not speak officially for Red hat. My opinions are truly my own.
"VMware seems to dominate the bare metal landscape. Even though Red Hat has a competing product, there's still a ton of customers that want to run OpenShift on top of VMware so we end up working with it a fair amount. They made some design decisions that I wouldn't have made, but overall it's solid technology if you don't mind proprietary."
I didn't (and still don't) think that was a negative comment. It felt very factual and fair to me. But "overall it's solid technology" is my opinion, not Red Hat's, and it seems ridiculous to me that people would say it was Red Hat's opinion, but ¯\_ (ツ)_/¯
Google Is 2B Lines of Code, All in One Place - https://news.ycombinator.com/item?id=10227000 - Sept 2015 (325 comments)
This sort of opinion appears rife somehow, people who make it clear their raison d'être is make money above all else are considered incompetent because, whilst making massive amounts of money, their actions are highly deleterious in other areas.
Do you just not believe that those in control only care about the money and couldn't care less about search quality other than to the extent it's necessary to make money?
I definitely believe this, but I also think companies fall victim to the classic quote from Upton Sinclair:
"It is difficult to get a man to understand something when his salary depends upon his not understanding it."
It would not surprise me if Googlers had a rationalization for why these "worse" search results are somehow a better user experience.
Other examples: Apple's tight control over the iPhone software experience is controversial. Facebook employees often say they're benefiting the world, while outsiders think they're harming it.
I don't think these are black-and-white - someone can credibly argue a case for each. But I think the Upton Sinclair quote tends to tip the balance for what insiders believe.