Honestly, I have caught myself looking for an advanced search option on my github page more than a couple of times. Just to remember that it does not exist. Long overdue!
I used this recently when I was searching for examples of CPython extensions implementing a particular feature, and it actually did help me find what I was looking for! I also used it when looking for examples of Github action workflows that did particular things.
It's nice that you can use the path: option to specify the file extension and/or the path to the file.
Hello from the GitHub code search team! I'm really proud of this release. It's the result of a year of hard work from multiple teams to bring the custom code search engine we built for cs.github.com to github.com. In the background, we've also spent a lot of time since the launch rebuilding the index format to let us scale up the system.
All characters are indexed (UTF-8 only). Files larger than 350 KiB are currently excluded, though we welcome feedback on use cases that require larger files to be indexed.
A few weeks earlier and I could have not be forced to hack together a tool to search for specific repos in specific languages that contain certain code-snippets by scraping the existing search API for a week. :P
Joking aside though, looks really great, thank you so much! :D
Last time when I heard this, the GitHub search was the shittiest search ever I came across. It didn't had even basic dedup. You search for something and there will be 10 pages of results with exact same code from thousand forks. The intern guy who worked on it claimed he had used "AI" however.
I’ve been in the beta (I guess the alpha) for a little over a year now. It’s absolutely fantastic and has been great to understand, discover, and learn new code.
This looks very similar to the internal code search product at Google (called CodeSearch). If the quality of this is going to be anywhere near that of CodeSearch, I would use it every day
Does it still only index the primary/default branch of a repository, or can it search within all branches? If it can now, can we filter the search for stuff within the branches of forks for a specific repository?
So many interesting work can only be found manually, laboriously (but is so rewarding) within the explore network graph where people work on their own branches in their own forks. Their code is never found from the search function.
Currently, we index the head of the default branch. However, the index format will support branch indexing, and we use that functionality to index forks, which was not possible in prior incarnations of GitHub code search. Indexing specific commits (e.g., associated with a PR or feature branch) is on our roadmap.
One feature for code search that I've wanted for some time that could unlock some really interesting results would be to allow search to reach into the git file history.
I've found myself sometimes thinking that something existed in a particular codebase and I go to look for only to discover that I'm unable find it. Then, I realize that the code I was looking for has been removed. I can only find it by browsing back into the history figuring out the events that led to the deletion or change. Being able to search the history would be killer IMO.
This is something that I'm working on and I believe a couple of others as well. My solution won't be available until early next year though, as indexing public repos at scale is far from trivial and I'm expecting this will consume over 40TB of fast storage to be performant.
We (Sourcegraph) have other features to stay competitive. One thing we have been working on indexing other code hosts like Eclipse Foundation, Crates.io, etc.
I have been using cs.github.com religiously since it originally went into beta and it has increased productivity for me to insane levels. There are many features I still want but its usually *good enough
Our ultimate goal is to surpass the current code search index. As of today, there are about 25 million repositories in the new index, but we have more work to do before we reach full scale.
Heck, it'd be nice if the current system right now at least did grep with users being able to toggle POSIX options. It doesn't even do that.
Instead, I have to clone the entire repo to do a trivial lookup. To be fair, supporting this would require GitHub to do the compute themselves, but I have no idea what they could possibly be doing right now that's more effective.
My only problem with the new code search is I've noticed some examples where exact string searches fail to locate some matches. I was looking for callers of a function and assumed no results meant no callers. But a simple 'search all' in VS Code found results missing from GitHub code search.
This will also improve the github.com code search, right? I always found it frustrating that I couldn't really search for exact strings, even using quotes. I hope this fixes that too.
This will become the new github.com code search. Last year, we launched a Technology Preview of our new code search as a standalone site: cs.github.com. Today, we're bringing that powerful code search, plus a redesigned code navigation UX, to github.com.
Yes, Stack Graphs power code navigation for supported languages (currently, Python). We also use tree-sitter parsers for CTags-style symbol navigation.
48 comments
[ 2.7 ms ] story [ 104 ms ] threadMy colleague Tim Clem will be speaking at GitHub Universe today about some of technical details: https://githubuniverse.com/events/detail/virtual-schedule/49...
Joking aside though, looks really great, thank you so much! :D
Great work!
So many interesting work can only be found manually, laboriously (but is so rewarding) within the explore network graph where people work on their own branches in their own forks. Their code is never found from the search function.
I've found myself sometimes thinking that something existed in a particular codebase and I go to look for only to discover that I'm unable find it. Then, I realize that the code I was looking for has been removed. I can only find it by browsing back into the history figuring out the events that led to the deletion or change. Being able to search the history would be killer IMO.
[0] https://sourcegraph.com/search?q=context:global+repo:%5Egith...
[1] https://sourcegraph.com/search?q=context:global+repo:%5Egith...
(I work there, just commenting on my own though. we're all pretty happy to have competition, more awareness of code search, etc.)
For example: git log -pS 'deletedvar'
We put this together anticipating these questions: https://docs.sourcegraph.com/getting-started/github-vs-sourc...
We also just launched Search Ranking and Relevance: https://about.sourcegraph.com/blog/new-search-ranking
More to come! And BTW we are happy to have competition, GitHub is a great brand and it will only make code search better for all developers.
[0]: https://github.blog/changelog/2020-12-17-changes-to-code-sea...
Heck, it'd be nice if the current system right now at least did grep with users being able to toggle POSIX options. It doesn't even do that.
Instead, I have to clone the entire repo to do a trivial lookup. To be fair, supporting this would require GitHub to do the compute themselves, but I have no idea what they could possibly be doing right now that's more effective.
[0] https://github.com/tree-sitter/tree-sitter-graph