I just want to say about time. A lot of the time when using libraries with inadequate documentation, being able to find usages of a method or class gives really good insight into the library. But the current code search's stemming removes all the context needed to find that and then gives alternate spellings too.
> Search for an exact string, with support for substring matches and special characters, or use regular expressions (enclosed in / separators).
Finally!
Search-for-literal is so important when you have technical users working on non-prose text.
They say this is going in a dedicated search page 'to start with', if "<literally any text>" doesn't work in the top bar eventually this is still going to be miserable.
I'm from the team that developed this at GitHub - if you are in the technology preview, then you can jump into cs.github.com from searches done at the top bar.
I use Github's UI for exploring and searching codebases more often than my own environment, since I do a lot of curious browsing.
No offense, but the search is so bad for anything worse than a single word, that I've developed a sort of intuition for how to phrase things -- and then still spend a lot of time crawling pages of results haha.
Hi Abdallah. If you haven't signed up yet, you can do so here: https://github.com/features/code-search/signup. People on the waitlist are getting access as quickly as the team can support it.
What's your take on developing a new code search instead of partnering with an existing global code graph like Sourcegraph? What are the advantages of GitHub Code Search over Sourcegraph?
Well, in the past I've tried Sourcegraph several times, but it never give me experiences that match the was-dead-many-years-ago Google Code Search. I wish the new github code search does that.
Hey Edwin, if you're open to providing feedback, I'd love to understand which types of searches worked well for you in Google Code Search but not in Sourcegraph. We've invested a lot of thought into our query syntax, supporting literal matches, regex, and the Comby pattern matching syntax with a rich set of keywords and filters—but we know the syntax isn't always intuitive for every user. We're always trying to improve the experience for all our users (I'm the CTO at Sourcegraph), so if you have any recollections you're open to sharing, would love to hear them!
For me for one thing, although I do use Sourcegraph for searching public code (thanks for creating and maintaining it!), I find the website too slow and heavy-feeling. Not in a way that makes it impossible to use, for sure, but somehow just slow enough to clearly not feel "snappy" but rather "bulky" and lagging. In particular this (and maybe also some other UX choices, like not 100% always fully supporting right-clicking and "open in new tab"? Not sure now if that's indeed so, but kinda feeling I'm always afraid to do this - or maybe because I fear slowing down my browser?) makes me do less searching with it than I'd need/want to, and makes me hesitate internally every single time I'm thinking whether to open a (new) tab with Sourcegraph. Although I know the results will be very good; but I know also I'll feel tired by the lagginess.
It's not quite what you're asking, but it you want to limit your searches to a particular repo, you can just type that into the URL bar (and then bookmark that): https://cs.github.com/$OWNER/$REPO, just like how the repo's primary site on GitHub is https://github.com/$OWNER/$REPO
I use github search a lot and this would be an insane productivity boost. I signed up for the waitlist. can you please give me a nudge in the queue? This is my profile https://github.com/abdallahmansour6
The addition of exact match search is so exciting that I haven’t internalized any of the other new features. I’ve abandoned an ungodly number of semi-common-word searches after getting 30 pages of results in a monorepo
I didn't even see this in the feature list before doing the signup. One of the signup questions is "how do you usually search?" or so, I wrote in the blank "I want to search for symbols, not substrings, so if I'm searching for `bar` I don't want `foo_bar` to show up as a match". I usually do this with word boundaries in regexes, but I pretty much have to have the repo downloaded, so it's useless for searching on github.com this way.
Ah, great. GitHub throwing out special characters in searches was infuriating for languages with sigils and patterns, like $somevar or %sql% and so on.
Slight tangent: The video has a guy describing the tool and he includes the fact that it’s written in rust when introducing it. I’ve always found this sort of name dropping in rust projects/devs baffling. Is there anything that I’m expected to infer from it? Is it that it’s backend is memory safe? I can’t think of anything else. Now it may very well be very memory safe but why include that highly specific detail when talking about a very high level thing that is the UX of search. What if it was written in Haskell or C#? Would it still be brought up? It’s almost as if being written in rust is a feature in itself these days. As a technical guy I can’t help but take the person less seriously, especially when it’s as unwarranted as this.
Go had this issue for a while, too, it's finally started to calm down as Go hits a mainstream that is (imo) much farther than Rust is currently. I think much is just people trying to add validity to Rust for large-scale production workloads, in the same way that Kubernetes was "a compute scheduler written in Go" or Terraform was "infrastructure as code written in Go" (maybe those are bad examples, but I know I've seen the "X written in Go" thing going on).
This is exactly how I see it as well. Rust used to be an obscure language with a compiler written in OCAML. If something was written in D or zig, it’s noteworthy so you mention it. I think rust has come into the mainstream enough that we can drop the “written in rust” line imo.
I think depending on where the audience is coming from—for example people who primarily work in scripting/interpreted languages—Rust can also be a positive signal for performance.
Not ashamed to be a Rust evangelist! The reason I mentioned Rust is because we spent a lot of time making the experience really fast - which is super important for a product like this. I really think getting the performance we have would have been enormously more difficult in any other language.
Fellow Rustacean here. Is the search engine secret sauce or something that could perhaps be open sourced? I'd like better tooling for searching private code bases. Also, would you consider writing about optimization techniques you used?
We are looking into open sourcing some libraries that we've developed for search. And we're going to write a blog post with way more technical details soon!
I agree with you, but I just wanted to point out the following:
In general, Rust, C, and C++ are going to be faster than languages like Ruby*. He brought up Rust while discussing the performance of the new tool. Although performance is more complex than language choice, etc., saying it's written in Rust gives the viewer an approximate lower bound as to how fast the tool should be.
*: (GH started as a Ruby shop, so I wouldn't be surprised if that's what the original tool was written in).
He’s talking about text search and the post thanks @BurntSushi. That means they’re using the fastest text search tool out there - ripgrep. I won’t mention what it’s written in, because that clearly upsets you.
I always though Github's bad search functionality was a business decision. It was so bad for so long. Even if basic improvements are significantly harder at their scale, I just can't comprehend how Microsoft left something so potentially useful be so bad for so long.
It really looks like they took a lot of inspiration from https://sourcegraph.com/search with this. Not a bad thing at all. I hope SourceGraph doesn't get obsoleted by this though, they're great people.
Sourcegraph CEO here. I'm really sorry about that. We work really hard on making our interviews good for everyone, including documenting it publicly at https://handbook.sourcegraph.com/talent/interview_process. Could you please email me at sqs@sourcegraph.com so I could find out what happened?
I've met two of their devs randomly in different Discord servers. Both were great people (Noah, Olaf) and are very active in OSS communities. Perhaps not coincidentally, both worked on Language Server related stuff.
Ólafur is responsible for a lot of Scala tooling and some pretty neat original ideas.
Sourcegraph also came up with LSIF, which is useful format for building tooling for language servers:
I interviewed for Sourcegraph and it was one of the best. Super transparent process, open source handbook, fun coding tasks -- really nothing to complaint about. Would be curious to know what made you have such a different experience.
Sourcegraph CEO here. Imitation is the sincerest form of flattery. We are very transparent, have a ton of users, and are open-core, so it's easy to get inspiration from us. :) We want way more devs to be using code search since it's so valuable 10x+/day, and if this helps, then we are very happy for that. Devs get to choose the code search tool they use, so the best tool will win (you wouldn't use Bing if your boss made you...likewise, code search isn't like team chat or team docs).
There is a way to search for comments using the "global search", but no way to search for text over issues and their comments. In particular, no way to search from the issue tab, no way to search over comments only in issues (or only in merge requests), no way to combine a text search with label/milestone/status filters, etc.
So it's a workaround, but a bad one.
Here's the ticket (2015): https://gitlab.com/gitlab-org/gitlab/-/issues/13891. The fact that it has so many duplicates in your own project's issue tracker is a good indicator of how bad your issue search is.
I love how the Microsoft acquisition continues to result in increased investment in github with microsoft's resources, and real vision; not always how an acquisition goes.
Win32 API isn't nice, but Microsoft was always relatively good with documentation etc. and don't forget all the developer support within Excel, VBA, Visual basic etc. Bill Gates early on understood the premise of building a platform and not breaking it. Even if that meant win32 API became ugly over time. Old windows programs still work on newer releases.
What makes you say that? I love to hate on Microsoft, wouldn’t touch Windows with a 100ft pole, but I always give credit to Microsoft for VS, VS Code (and my favorite languages, TS and C#).
I admittedly haven’t used Visual Studio in a while (though I use VS Code daily) but I remember it as my favorite IDE. Certainly better than XCode, and I even preferred it to IntelliJ.
There was a moment where I thought GitLab was poised to start hitting critical mass and GitHub's best days were behind it, that was probably around right before or after the acquisition.
Microsoft has done a great job of actually improving the product and investing in it, something that seems to not happen most of the time with giant acquisitions.
GitHub today, at least for me, is definitely improved over the GitHub five or so years ago. It does feel like some things are bloated due to a push for feature after feature, but the core features have gotten better to the point I don't care. I just wish I didn't have to spend 2 minutes turning off all the annoying features I don't care about for small projects on every new repo I create.
(From a GitHub product manager) Thank you for this feedback about the pain of having to repeatedly change repo settings. It makes great sense that you'd want to repeat certain settings. Would it meet your needs if repository templates allowed you to have settings that got copied when you created a repo from the template (https://docs.github.com/en/repositories/creating-and-managin...)? I also wonder if you'd be interested in repository settings being settable in a text file. If you want to continue chatting about this it would be great if you could post it in GitHub's feedback discussions here: https://github.com/github/feedback/discussions/categories/ge.... Thanks again!
This is great, specified search on GitHub has previously been very hit or miss. Generally I use the search feature for learning / trying to see if something I'm trying to do already exists. I personally think vsCode has the best code search implementation, in terms of "exact", "partial" and "regex" matching. The UI is clear, non-technical team members can navigate their way around it and it's relatively fast assuming you don't have too many extraneous plugins installed.
Today I wanted to search for "strstr[a-z]+?_r" but got the error message "This is a partial result set. The search was stopped early because it would take too long to check every file for this regular expression.". However, I got results for the less restrictive regex "strstr.+?_r" which is weird since I'd expect that it would be easier to return results for more restrictive regular expressions. Not sure if there is a perfect solution for this, but in many cases, you could probably search for the less restrictive version and filter the results with the more restrictive one after that.
Also it would be great if more repositories were indexed. How do things work behind the scenes? Maybe it is possible to build a more memory-efficient index just for exact string search, which probably make up most searches.
Anyway, this website is amazing and I use it quite often. Thank you a lot for working on this!
* Copy & Paste does not work. When trying to select text from code snippets, the code is dragged like an image instead of being selected.
* The GitHub icon during search is not clickable. Clicking to the right of the GitHub icon does not go to GitHub but instead shows an embedded view of that repository. There, the GitHub icon somewhere on the right is a bit hard to find (maybe write "View on GitHub" next to it so it is discoverable with Ctrl + F?), but at least it is clickable.
De-duping exact matches is a game changed -- search has been miserable to use because of the dupes for so long. I can live with near-similar documents. Very excited to test this out.
Another GitHub Code Search developer here - to add more to this, we rank all the search results, and try to bring the most relevant results to the top. Ideally, if you have 10 pages of results, you shouldn't have to leave page 1 to find what you're looking for :D
That would be a tough problem. As de-dup you probably want to show/point towards the 'original' tree. But which one is the source? Or even worse someone abandons a project but someone else forked it and kept going should it show that one instead? Or should it show the one it was forked from depending on the version number. Which one is the 'true' repo now? Most certainly an interesting problem.
I get that. Just remove the 'extra'. That is a good first pass. I was thinking the longer term you want to show the 'original' higher in the list? Wouldnt you? What sort of criteria would you use to make it so it shows one copy vs another? Probably in many cases it probably would not mater much. But if you wanted to figure out linage of imports it could be? Some projects could have thousands of forks. Yet only maybe a dozen of those actually have anything going on. Those would be more useful to show?
Now can they fix doing a language search for “Visual Basic”? If you filter a users repos or stars on that language it just shows all their repos or stars. Code search for language “Visual Basic” returns all repositories and does not limit by language like it should.
Are there any open source powerful code search engines out there? As a Googler the internal code search we have here is one of the most incredible things I've ever seen, it's so fast and powerful I'm amazed by it daily. Is there anything near that quality out there?
We built Sourcegraph taking inspiration from Google Code Search (https://about.sourcegraph.com/blog/ex-googler-guide-dev-tool...) to bring the power of code search—and precise code intelligence that just works—to every dev. Try it out here: https://sourcegraph.com. A super common thing we see is people leaving Google, missing code search, and then bringing Sourcegraph into their new org. We'd love to hear your feedback!
Sourcegraph is open-core, with a dual licensing approach. You can run the open-source version here: https://github.com/sourcegraph/sourcegraph#sourcegraph-oss, and we have an enterprise offering for companies that want to adopt for their teams. Similar to GitLab, both our enterprise and OSS code is publicly available.
The best thing about the Sourcegraph instance hosted on sourcegraph.com is that you can edit the URL in your browser from https://github.com/foo/bar to https://sourcegraph.com/github.com/foo/bar to be dropped down into a Sourcegraph search for that GH repo. I've been using it for a long time because of this convenience.
(Though it would be even better if the two options for case-sensitivity and regex search were enabled by default instead of needing me to toggle them on every time.)
You should be able to do that over in your User Settings (Click your picture in the top right and then Settings.) Adding these two things should change that default for you:
DXR has largely been replaced with mozsearch (https://github.com/mozsearch/mozsearch), and a quick glance through the really early history does show that it adopted a fair amount of stuff from DXR. The downside is that it's not as easy to set up a local mozsearch instance as old-school DXR was.
I helped write DXR for indexing Mozilla's source code based on an instrumented compiler run; this has eventually been developed into mozsearch (https://github.com/mozsearch/mozsearch), whose indexing for mozilla-central is visible here: https://searchfox.org.
I thought it was abandoned! This is great to hear it just moved. Is there anyone at Mozilla that can update the old DXR repo [0] to direct people to MozSearch?
I work on a very large c++ monolith at work and DXR has been a real game changer for helping me just figure out how so much of the codebase works. Thanks!!
My job uses https://oracle.github.io/opengrok/ and I'm generally happy with it. It has some problems with special character searches at times but generally does what I want. It's certainly better than code search in our on-prem github instance.
We're using https://searchcodeserver.com/ internally and it works quite good, it was miles ahead GH search. After today, I'll have to test GH again to reassert this statement.
For the grepping aspect, https://github.com/google/zoekt is a powerful one-stop-shop. For the navigating, I don't know. SourceGraph maybe, but the linking is somewhat heuristic I assume, not compilation-graph powered. But maybe that changes or depends per language.
This is great! As a project manager I am using github search everyday when I am searching for specific methods or part of the code in order to find logical issues or bugs in a code.
grep.app doesn't index all repositories on GitHub though. I was doing some research a few months ago and couldn't find anything that would search all of github quickly.
I use github search a lot and this would be an insane productivity boost. I signed up for the waitlist. Does anyone working at Github want to bump me in the queue? This is my profile https://github.com/adamnemecek/
Still waiting for the ability to search in other branches. It's a pain when some codebases have stable releases on the next/dev branch but keep their main branch to the previous release.
Absolutely. I get they don't want to index every branch but at least set some heuristics like it it has a certain amount of activity or something per repo. Or even allow repo to opt into 1 to 2 other branches besides main. Especially for bigger projects
I can't use source graph to search a public repo I work on without granting them read access to every private repo I have. Attempting to deny just that permission causes it to error out.
So I'll summarize that no, you can not do that on source graph, given that doing so would require everybody violating their ndas.
Appreciate the feedback here (Sourcegraph CTO). I agree that's annoying. Kicking off a conversation on our end to figure out how to fix that. In the meantime, the workaround would be to create a separate GH login, which can be used to add any public repo to our index. Or what's the URL of the repo(s) you'd like to index and we'll get those added to our index.
Great. I'm using grep.app[1] usually as for me the GitHub search is mostly useless. Your mileage may vary though.
That being said there are many other great search interfaces that I am using often when I'm trying to find solutions to common problems or specific design patterns. Chromium search[2] comes to mind, Mozilla's Firefox[3], Android[4] or of course Google[5]
Of all the tools I use on a daily basis Github is probably the worst. I mean the "Find a repository..." input field on the start page can not even filter out named repositories I have access to in all my organizations. It works for some repos but not all.
Search improvements? It is impossible to create a worse search experience than Github. Just clone and use git grep instead in most cases.
221 comments
[ 0.20 ms ] story [ 255 ms ] threadFinally!
Search-for-literal is so important when you have technical users working on non-prose text.
They say this is going in a dedicated search page 'to start with', if "<literally any text>" doesn't work in the top bar eventually this is still going to be miserable.
I use Github's UI for exploring and searching codebases more often than my own environment, since I do a lot of curious browsing.
No offense, but the search is so bad for anything worse than a single word, that I've developed a sort of intuition for how to phrase things -- and then still spend a lot of time crawling pages of results haha.
This was sorely needed
Not ashamed to be a Rust evangelist! The reason I mentioned Rust is because we spent a lot of time making the experience really fast - which is super important for a product like this. I really think getting the performance we have would have been enormously more difficult in any other language.
In general, Rust, C, and C++ are going to be faster than languages like Ruby*. He brought up Rust while discussing the performance of the new tool. Although performance is more complex than language choice, etc., saying it's written in Rust gives the viewer an approximate lower bound as to how fast the tool should be.
*: (GH started as a Ruby shop, so I wouldn't be surprised if that's what the original tool was written in).
Benchmark - ripgrep is faster than {grep, ag, git grep, ucg, pt, sift} (2016) - https://blog.burntsushi.net/ripgrep/
Ólafur is responsible for a lot of Scala tooling and some pretty neat original ideas.
Sourcegraph also came up with LSIF, which is useful format for building tooling for language servers:
https://lsif.dev
If you want to build this sort of stuff, the work Sourcegraph has done with LSIF + SemanticDB is probably your easiest bet.
N=2 isn't great, but there's my experiences if we're tossing them out there.
Comment (and code) search is available for projects in all GitLab tiers: https://docs.gitlab.com/ee/user/search/#basic-search
Premium and Ultimate users have access to Advanced Search: https://docs.gitlab.com/ee/user/search/advanced_search.html
So it's a workaround, but a bad one.
Here's the ticket (2015): https://gitlab.com/gitlab-org/gitlab/-/issues/13891. The fact that it has so many duplicates in your own project's issue tracker is a good indicator of how bad your issue search is.
> no way to combine a text search with label/milestone/status filters, etc.
You can combine text search with field search (like label/milestone/status. Here’s an example: https://gitlab.com/gitlab-org/gitlab/-/issues?search=Visuali...
I admittedly haven’t used Visual Studio in a while (though I use VS Code daily) but I remember it as my favorite IDE. Certainly better than XCode, and I even preferred it to IntelliJ.
Microsoft has done a great job of actually improving the product and investing in it, something that seems to not happen most of the time with giant acquisitions.
GitHub today, at least for me, is definitely improved over the GitHub five or so years ago. It does feel like some things are bloated due to a push for feature after feature, but the core features have gotten better to the point I don't care. I just wish I didn't have to spend 2 minutes turning off all the annoying features I don't care about for small projects on every new repo I create.
Give it a shot and let us know what you think! Where can we improve it?
Also it would be great if more repositories were indexed. How do things work behind the scenes? Maybe it is possible to build a more memory-efficient index just for exact string search, which probably make up most searches.
Anyway, this website is amazing and I use it quite often. Thank you a lot for working on this!
We're also working hard to increase the number of repositories indexed. :)
Some feedback:
* Copy & Paste does not work. When trying to select text from code snippets, the code is dragged like an image instead of being selected.
* The GitHub icon during search is not clickable. Clicking to the right of the GitHub icon does not go to GitHub but instead shows an embedded view of that repository. There, the GitHub icon somewhere on the right is a bit hard to find (maybe write "View on GitHub" next to it so it is discoverable with Ctrl + F?), but at least it is clickable.
It is not a friendly site. Open source projects would do better to use an open source code forge like <https://sr.ht/>.
This is on our radar! We de-duplicate exact matches now, but we'd like to do the same for near-similar documents.
https://github.com/google/codesearch
Source: https://github.com/ndmitchell/hoogle
(Though it would be even better if the two options for case-sensitivity and regex search were enabled by default instead of needing me to toggle them on every time.)
[0]: https://github.com/mozilla/dxr
Time to go secrets and url hunting.
That'd cover 95% of repo I've seen.
So I'll summarize that no, you can not do that on source graph, given that doing so would require everybody violating their ndas.
[1] https://grep.app/
[2] https://cs.chromium.org/
[3] https://dxr.mozilla.org/mozilla-central/source/
[4] https://cs.android.com/
[5] https://cs.opensource.google/
[6] https://about.sourcegraph.com
Search improvements? It is impossible to create a worse search experience than Github. Just clone and use git grep instead in most cases.
Edit: ...and the 425% price increase for SSO..
(Granted, this is largely due to a culture of titles like "Check out this thing" that provide zero searchable metadata + no tag system.)