I agree, that should be the case. Otherwise I have to open the file myself to get the context. And when I do that then I can also directly do the grep on the console.
I'm the author of this. I think I can index a lot more source code and still keep the speed. The important part of the index, the ngrams table, is only 400M or so of space and compresses pretty well. I had a test index at one point that was also indexing all of the C code in these projects:
You can actually run essentially the same thing over your own code with http://code.google.com/p/codesearch (written by the author of the article codesear.ch is based off).
It's not that fast. With a hot cache "grep -rn mmap /usr/src/linux/" runs in 0.3s on my desktop machine.
EDIT: Don't get me wrong, the code behind this is awesome. I just can't help but think there could be something bigger to search, that is infeasible to do on my desktop. What about all of the code from the Debian repositories or something?
I thought I would mention it here as nothing I submit ever hits the front page that searchco.de has just surpassed koders.com in number of lines of code indexed, or rather will have in about 2 hours when the indexing is finished and indices swapped over. EDIT - Now live.
I will have an official announcement once I get the exact count, but its something like 3,338,449,289 lines to 3,800,000,000 or so.
At which point do we get a search that syntactically understands what its searching? Thats such an obvious improvement when you know that your entire data set conforms to a language that an 80s C compiler can understand.
A simple grep I can do myself (with the added bonus of knowing what version/commit I'm searching against).
You could weigh search results. Instead of a random comment in a random device driver as to why we use mmap here, you could directly show me the core mmap implementation when I search for it. You can directly link me to functions, you can show me the definition for a struct object inline, you can show me dependency graphs.
Would you have an example of a tool which you could use to inspect source code to give this output? I imagine a general lexer or some description would be great for this as you could identify functions, comments, classes etc... and use that to add weights.
There's the problem of regular expressions not dealing with matching parentheses, for one. That makes a lot of questions about code hard to express in a regexp, even something as simple as 'find foo(anything) + bar(anything)' is busted. False positives within strings are also an issue.
I did a code search thing, (which I don't claim is anything more than a hack) described over in this HN thread: http://news.ycombinator.com/item?id=4113348 but yes, the syntax wasn't quite 'a string', you'd type the code you're looking for with parts wildcarded, but parentheses in what you typed had to match.
That's a good idea. Another use case occurred to me after I wrote my comment: to ignore keywords in comments. Sometimes that can be desirable.
I keep going back lately to Rob Pike's idea that breaking files into lines is not the best approach (http://news.ycombinator.com/item?id=4111667). In that paper he proposes a regular expression for breaking up lines into structured content. But maybe there's another way. Like an ometa grammar? (http://tinlizzie.org/ometa)
Regular expressions are the wrong tool for this job.
For Krugle, we actually parsed the various languages so it actually knew the differences between e.g. definitions, invocations, comments, strings, etc. It affords the capabilities to do all sorts of cool queries and analyses.
FWIW I am the author of this tool, and I would like to add something that understands syntax (I think I could start with indexing the code using ctags or cscope, which is hackish but easy to implement). There's obviously a ton of stuff you could do to improve ranking if you could rank results by whether they're a variable, function, etc.
I'm the author of this and didn't really intend for it to get submitted to hacker news at this point, which is why it's missing this feature among others. I literally just got things working on Sunday afternoon. I am definitely going to make it possible to link to search results.
See also my project, http://livegrep.com, which is very similar but does support regular expressions (and feels much snappier on the frontend to me, although his does claim comparable backend times).
41 comments
[ 2.9 ms ] story [ 114 ms ] threadI wish that the `filename` column was a link to the raw file (or a link to the file in git.kernel.org).
[0] http://codesear.ch/about
clang-3.1.src gcc-4.7.1 ghc-7.4.1 linux-3.4.4 llvm-3.1.src lua-5.2.0 openjdk perl-5.16.0 php-5.4.4 Python-2.7.3 Python-3.2.3 ruby-1.9.3-p194
And it only increased the size of the index by about 50%.
EDIT: Don't get me wrong, the code behind this is awesome. I just can't help but think there could be something bigger to search, that is infeasible to do on my desktop. What about all of the code from the Debian repositories or something?
[1] - http://www.vidarholen.net/contents/wordcount/
I had so much fun demo'ing that ability at a PHP conference one year.
http://codesear.ch/about
There is also http://searchco.de.
I will have an official announcement once I get the exact count, but its something like 3,338,449,289 lines to 3,800,000,000 or so.
A simple grep I can do myself (with the added bonus of knowing what version/commit I'm searching against).
I did a code search thing, (which I don't claim is anything more than a hack) described over in this HN thread: http://news.ycombinator.com/item?id=4113348 but yes, the syntax wasn't quite 'a string', you'd type the code you're looking for with parts wildcarded, but parentheses in what you typed had to match.
I keep going back lately to Rob Pike's idea that breaking files into lines is not the best approach (http://news.ycombinator.com/item?id=4111667). In that paper he proposes a regular expression for breaking up lines into structured content. But maybe there's another way. Like an ometa grammar? (http://tinlizzie.org/ometa)
Anyways, just a few random thoughts.
For Krugle, we actually parsed the various languages so it actually knew the differences between e.g. definitions, invocations, comments, strings, etc. It affords the capabilities to do all sorts of cool queries and analyses.
This is weak.
P.S. Chrome source search does the same thing, makes me angry and sad with so many stupid people.
I'm the author of this and didn't really intend for it to get submitted to hacker news at this point, which is why it's missing this feature among others. I literally just got things working on Sunday afternoon. I am definitely going to make it possible to link to search results.
So heres a bookmark to add links to the page, run it after completing the search query: http://pastebin.com/G53SjNWy
Also, its my first little foray into JS, so I'd really appreciate any criticism of the code and/or style.
edit: new link preserves the <td> tags