GCP has been able to host your repo since before the turndown of Google Code. In fact, the original GCP Source Repo team and the Google Code team were the same team (though the current product has a different backend and team).
This looks interesting but impractical for me to mirror hundreds of repos here just to get better search.
Has anyone played with this or similar tools that can provide search by indexing at source. It’s a little surprising that Google doesn’t just index everything public in Github, gitlab, etc.
I would like a good code search solution especially because I have projects across lots of different repo servers but want developers to be able to find code regardless of repo home. A colleague indexed it with solr and that was ok, but had limited semantic ability and definitely nothing like what Google describes where your recent search history and activity is displayed to each user.
And I am glad they do, because when I search on Google, the above mentioned kinds of results are exactly what I am looking for, not individual source files etc.
Thanks for taking a look. I'm glad you find Cloud Source Repositories interesting. We'd love to Cloud Source Repositories more practical to use. Can you explain what is impractical for you? Is the initial process of setting up multiple mirrors? Or the routine of having to go to another site for search?
(Disclaimer: I work at Google and am the PM on this product)
Thanks for responding. I work in an organization (happy to PM) with a few Github orgs and about 300 repos. Dozens of project teams work in them and use various GitHub features.
I have no easy insight or overview into code across these repos without scripting. The impractical part is in cloning mirroring all of these repos to get insight from your new function because user behavior is baked into all the workflow around the Github repos.
The go to another site isn’t so bad as this is a new function and not available in Github, so programmers will likely check it out when they need.
But manually forming everything and syncing. It would be neat if I can import an org in one fell swoop.
That being said, I do have scripting forking everything over on my list of things to do when I have cycles on the weekend.
One nice thing is how trivial it is to mirror a repository from Bitbucket or GitHub. I wish it was as easy to mirror a repository from Bitbucket to GitHub or GitHub to Bitbucket.
Very nice. Reading that makes me miss Google’s internal dev environment. I may not much use anymore Google Search and gmail, but Google’s paid products like GCP (and Google Play Music, Movies, etc.) are awesome. Also, amazing permanent free tier levels. I need to use github for the repos for my books because they are already linked, but for my personal private repos I think I may switch.
Could you share why as a former Google dev you no longer use Google search and gmail? I'm curious why you mentioned that in response to a post about cloud source repositories. Thanks.
I'm not OP, but I'm also a former Googler. I switched to DDG and Firefox after I left because I think it's important that alternatives exist. Google may be a better actor in monopoly situations than some other companies, but they're still a company trying to make profit for shareholders.
(As a bonus, I find ddg+Firefox snappier than Google+Chrome)
I'm a former Googler and I mostly don't use Gmail for the mundane reason that its filtering isn't as powerful as what I do with dovecot and sieve.
I may transition from my mail server after I've deployed automation for this built atop GCP and the Gmail API.
Even then I'll still make plenty of use of mutt for better supporting OpenPGP, bottom-posting, and true threading - all of these things are still useful in Debian, and I am a Debian developer. But I'll be more easily able to mix usage of Gmail and mutt, without some hacks I currently use to achieve that.
> For Java, JavaScript, Go, C++, Python, TypeScript and Proto files, you’ll see result suggestions indicating whether the match is an entity such as a class, method, enum, or field.
We've generally heard that developers move from Bitbucket to Cloud Source Repositories because they want to use other Google Cloud services and it's simpler for them to manage their source in the same place where they debug, build and deploy that code. Developers often mention that they appreciate the unified identity, IAM permissions, and integrations that Cloud Source Repositories has with Cloud Shell, Cloud Build, Cloud Debugger and others Google Cloud services.
However, developers don't need to move from Bitbucket to take advantage of the utility provided by Cloud Source Repositories. You can mirror any number of your repositories into Cloud Source Repositories and take advantage of the code browser, code search, and various integrations with Google Cloud Platform.
(Disclaimer: I work at Google and am the PM on this product)
I know we deprecated the prediction API with a year's warning while bringing out Cloud ML Engine. And Cloud SQL changed a bit when the second generation came in and supported MySQL, PostgreSQL and more.
Can't think of others off the top of my head, so that is two if we want to consider major upgrades "abandoned." Perception often doesn't match objective fact.
Nice. Is the code search the same as what Google has internally? I do miss that.
(Though honestly, the value from the internal code search was that everything was in there. So if you got an error message from a service you were a client of, you could just search for the error message in their code and see what it actually meant. "Aha, this 'optional' field that's marked 'deprecated' isn't actually optional or deprecated," most of the time ;)
It’s great that they opened Kythe but as someone who deployed it from scratch I say it’s like ordering a DIY aircraft carrier kit. Many steps before you get something useful.
Having glanced at it a bit I don’t see how this can be based on internal code search / kythe. Kythe works by building your codes. That’s how it can find callers and specializations and whatnot. This doesn’t seem to.
The code search functionality is the same as what Google engineers use internally. Cloud Source Repositories uses the same indexing and retrieval technologies on the same type of infrastructure.
As you mention, the more code you have, the more benefit you get from having fast search tools across your entire code base that can perform complex semantic and regular expression queries. Even with smaller codebases, I find it the fastest way to find the code I need.
(Disclaimer: I work at Google and am the PM on this product)
Quickly glanced through the blog post and the docs but didn’t find any mention of Git LFS. Does Cloud Source Repositories support mirroring LFS content?
Cloud Source Repositories currently supports very large repositories. The same backend scales to the needs of the Android open source project, which regularly checks in massive binary files. There are many APKs and VM images in the Git repositories we host. However, Cloud Source Repositories does not support LFS or the mirroring of LFS content.
LFS is not deeply integrated into Git, creating usability problems. Because the content is not part of the object graph of the repository, you have to decide in advance to use Git LFS. You don't get the benefit in existing repositories with large files. You also can't back out --- once you're using Git LFS on a medium-sized file, you can't change your mind and instruct Git to send it inline in fetches, without rewriting history, which breaks existing clients that have cloned the repository.
For the same reason, history mining commands like "git blame" and "git log -S" don't have access to the object.
In addition, it complicates migrating to another host. Usually in Git, you can take out your content by running "git clone --mirror one-url and then "git -C directory.git push --mirror another-url". With Git LFS, this copies over the pointer files but not the underlying large file content and you must remember to take extra steps to instruct the new host about where the blobs are stored.
Google's Cloud Source Repositories team believes that Git itself needs to deal better with large files. The first step of this work within the Git project has been partial clone, which is supported in Cloud Source Repositories and in the public Git 2.17 release (for best results, please use Git 2.19, released September, 2018). If you run "git clone --filter=blob:limit=512M <url>", files larger than 512M will be omitted from the initial clone and fetched automatically on demand when needed (for example during checkout operations). See https://crbug.com/git/2 for more details about this feature. We are continuing to work with the community on adding other features related to large file support into Git.
(Disclaimer: I work at Google and am the PM on this product)
I was going to make this same joke. It's hard to put faith into a product when Google keeps killing them off. Wave and Google Code are two top of mind products that I missed when they were gone.
Conversely, I think it's important for Google engineers to be able to try new things and publish products without having to worry about supporting them forever. Is there some sort of happy medium? Like a Google labs of products that are in various states of whimsical testing?
Cloud source looks nice, and for GCP projects I'm sure it will have a nice tie in, but for now I'm 100% more likely to push code to Gitlab until it makes sense to do otherwise.
I know that is the meme, but are there any examples of Google dropping enterprise (i.e. paid) offerings? I can think of a few free services that they have dropped, but that's a different story than dropping a commercial service that you're selling to enterprise users.
> Like a Google labs of products that are in various states of whimsical testing?
They actually had precisely that, but then they cancelled it.
> Google described Google Labs as "a playground where our more adventurous users can play around with prototypes of some of our wild and crazy ideas and offer feedback directly to the engineers who developed them."
I'm glad you like it. Which languages would you like to see support added for?
Note that search works today across all languages but the semantic understanding of source which enhances search is limited to Java, JavaScript, Go, C++, Python, TypeScript and Proto files.
(Disclaimer: I work at Google and am the PM on this product)
Thanks for the post Russell. Our SaaS application is running on Rails / Typpescript frontend. We're patiently waiting for App Engine to add Standard support for Ruby before we go all in on GCP. I would be most excited to see Cloud Source support Ruby. Hope your team will consider it.
Being able to search code so efficiently across the whole company’s code base is a huge productivity gain. But I found people who hasn’t worked in Google or other companies that have this capability generally don’t believe it’s a big deal.
Well, you need to have a fairly large codebase with a fair amount of in-house infrastructure and interdependencies before you have both the need to find code that you're not working on and have a dense enough source tree where search makes more sense than just learning the organization and navigating to where you need to go.
Honestly, it's not even the monorepo and tooling. Simply the ability to navigate our code IDE-style in the browser, and move back and forth between different revisions, makes a huge difference.
I've been mucking around with some code in javac recently and was trying to show someone something I'd found, and browsing the openjdk mercurial repo was just so painful compared to hunting around google3.
For those that haven't used Google's code search before, I believe this will be very similar to Chromium's code search[0] (which is based on Google's internal one).
I believe the difference is this is part of GCP and their terms for GA products are that they will support for at least one year on deprecated versions / products and not just instantly remove something.
55 comments
[ 6.2 ms ] story [ 200 ms ] threadHas anyone played with this or similar tools that can provide search by indexing at source. It’s a little surprising that Google doesn’t just index everything public in Github, gitlab, etc.
I would like a good code search solution especially because I have projects across lots of different repo servers but want developers to be able to find code regardless of repo home. A colleague indexed it with solr and that was ok, but had limited semantic ability and definitely nothing like what Google describes where your recent search history and activity is displayed to each user.
GitHub excludes a few patterns in robots.txt that means a lot of the public data is never seen by crawlers that respect robots.txt, for example:
and https://github.com/robots.txtThey probably want people landing mainly on
- user profiles
- repo roots
- individual issues
- individual pull requests
And I am glad they do, because when I search on Google, the above mentioned kinds of results are exactly what I am looking for, not individual source files etc.
Thanks for taking a look. I'm glad you find Cloud Source Repositories interesting. We'd love to Cloud Source Repositories more practical to use. Can you explain what is impractical for you? Is the initial process of setting up multiple mirrors? Or the routine of having to go to another site for search?
(Disclaimer: I work at Google and am the PM on this product)
I have no easy insight or overview into code across these repos without scripting. The impractical part is in cloning mirroring all of these repos to get insight from your new function because user behavior is baked into all the workflow around the Github repos.
The go to another site isn’t so bad as this is a new function and not available in Github, so programmers will likely check it out when they need.
But manually forming everything and syncing. It would be neat if I can import an org in one fell swoop.
That being said, I do have scripting forking everything over on my list of things to do when I have cycles on the weekend.
(As a bonus, I find ddg+Firefox snappier than Google+Chrome)
That made me think, so thank you and yes I agree.
I may transition from my mail server after I've deployed automation for this built atop GCP and the Gmail API.
Even then I'll still make plenty of use of mutt for better supporting OpenPGP, bottom-posting, and true threading - all of these things are still useful in Debian, and I am a Debian developer. But I'll be more easily able to mix usage of Gmail and mutt, without some hacks I currently use to achieve that.
Well, permanent until they kill the entire service 5 years from now.
Apparently they forgot about Flutter users.
We've generally heard that developers move from Bitbucket to Cloud Source Repositories because they want to use other Google Cloud services and it's simpler for them to manage their source in the same place where they debug, build and deploy that code. Developers often mention that they appreciate the unified identity, IAM permissions, and integrations that Cloud Source Repositories has with Cloud Shell, Cloud Build, Cloud Debugger and others Google Cloud services.
However, developers don't need to move from Bitbucket to take advantage of the utility provided by Cloud Source Repositories. You can mirror any number of your repositories into Cloud Source Repositories and take advantage of the code browser, code search, and various integrations with Google Cloud Platform.
(Disclaimer: I work at Google and am the PM on this product)
Can't think of others off the top of my head, so that is two if we want to consider major upgrades "abandoned." Perception often doesn't match objective fact.
(Disclosure: I work at GCP)
(Though honestly, the value from the internal code search was that everything was in there. So if you got an error message from a service you were a client of, you could just search for the error message in their code and see what it actually meant. "Aha, this 'optional' field that's marked 'deprecated' isn't actually optional or deprecated," most of the time ;)
Correct. Looks like it just parses and indexes the source.
The code search functionality is the same as what Google engineers use internally. Cloud Source Repositories uses the same indexing and retrieval technologies on the same type of infrastructure.
As you mention, the more code you have, the more benefit you get from having fast search tools across your entire code base that can perform complex semantic and regular expression queries. Even with smaller codebases, I find it the fastest way to find the code I need.
(Disclaimer: I work at Google and am the PM on this product)
Cloud Source Repositories currently supports very large repositories. The same backend scales to the needs of the Android open source project, which regularly checks in massive binary files. There are many APKs and VM images in the Git repositories we host. However, Cloud Source Repositories does not support LFS or the mirroring of LFS content.
LFS is not deeply integrated into Git, creating usability problems. Because the content is not part of the object graph of the repository, you have to decide in advance to use Git LFS. You don't get the benefit in existing repositories with large files. You also can't back out --- once you're using Git LFS on a medium-sized file, you can't change your mind and instruct Git to send it inline in fetches, without rewriting history, which breaks existing clients that have cloned the repository.
For the same reason, history mining commands like "git blame" and "git log -S" don't have access to the object.
In addition, it complicates migrating to another host. Usually in Git, you can take out your content by running "git clone --mirror one-url and then "git -C directory.git push --mirror another-url". With Git LFS, this copies over the pointer files but not the underlying large file content and you must remember to take extra steps to instruct the new host about where the blobs are stored.
Google's Cloud Source Repositories team believes that Git itself needs to deal better with large files. The first step of this work within the Git project has been partial clone, which is supported in Cloud Source Repositories and in the public Git 2.17 release (for best results, please use Git 2.19, released September, 2018). If you run "git clone --filter=blob:limit=512M <url>", files larger than 512M will be omitted from the initial clone and fetched automatically on demand when needed (for example during checkout operations). See https://crbug.com/git/2 for more details about this feature. We are continuing to work with the community on adding other features related to large file support into Git.
(Disclaimer: I work at Google and am the PM on this product)
Conversely, I think it's important for Google engineers to be able to try new things and publish products without having to worry about supporting them forever. Is there some sort of happy medium? Like a Google labs of products that are in various states of whimsical testing?
Cloud source looks nice, and for GCP projects I'm sure it will have a nice tie in, but for now I'm 100% more likely to push code to Gitlab until it makes sense to do otherwise.
[1] https://en.wikipedia.org/wiki/List_of_Google_products#Discon...
https://news.ycombinator.com/item?id=16980166
They actually had precisely that, but then they cancelled it.
> Google described Google Labs as "a playground where our more adventurous users can play around with prototypes of some of our wild and crazy ideas and offer feedback directly to the engineers who developed them."
https://en.wikipedia.org/wiki/Google_Labs
I'm glad you like it. Which languages would you like to see support added for?
Note that search works today across all languages but the semantic understanding of source which enhances search is limited to Java, JavaScript, Go, C++, Python, TypeScript and Proto files.
(Disclaimer: I work at Google and am the PM on this product)
I've been mucking around with some code in javac recently and was trying to show someone something I'd found, and browsing the openjdk mercurial repo was just so painful compared to hunting around google3.
[0] https://cs.chromium.org/