HN Explorer

Show HN: A Search Engine for Developers, Marketers, Web Designers, and Hackers

63 points by nerdydata ↗ HN
Watup Hackernews,

We have created a new type of search engine that searches on the full source code of webpages, not just the on-page text. It's called NerdyData (nerdydata.com) and we're launching today!

Our search engine is different from search engines you've used before. Traditional search engines are geared towards providing answers, whereas our goal is to give you the best list of results for a query. Our crawler has visited over 140 million homepages and collected terabytes of HTML, Javascript, and CSS code. We've also designed several search interfaces that allow anybody to query against the source code of webpages, or download a list of sites containing a specific term.

We even offer a search interface specifically for SEO's and marketers that allow you to search within specific HTML tags like meta descriptions and meta keywords.

About us: We're two 23 year olds who recently graduated from Stony Brook University in New York, and we are ridiculously passionate about making search better for our fellow nerds out there.

If you'd like to know more, you can email me at dave@nerdydata.com

Thank you so much for you time!

40 comments

[ 3.8 ms ] story [ 95.4 ms ] thread
aram
Very interesting service. At first it didn't occur to me why would I want to search the source code, but your examples were good.

What is the logic behind the search criteria? I entered `<div id="main">`[1] and the first result is LinkedIn with `<div id="main-wrapper">` highlighted, which shouldn't match my query directly as I entered a closed tag. Some of the other results also didn't contain exactly what I was searching for - e.g. `<div id="main" class="main">`, `<div id="main" class="clear">` etc. A technical explanation on this would be great.

Also, one minor note - David's photo in the front page footer is broken [1]

[1] https://search.nerdydata.com/search/#!/searchTerm=<div id="main">/searchPage=1/sort=pop

[2] http://www.arc.losrios.edu/Images/Images-arc/Foreign_Lang_De...

jayzalowitz
First off Stony Brook! Second, open crawl? Third, what's your stack?
nerdydata
1. Go Seawolves!

2. We built our own distributed crawler with a couple of dope servers.

3. Not ready to share our stack setup yet (work in progress) but we will have a write up shortly!

ErikAugust
Do results sort based on mozRank, Alexa, PageRank? Or something proprietary?
ScottWhigham
Love it - great idea and great presentation. I only tried a few cursory searches but it was definitely bookmark-worthy. Kudos :)

I'd love to hear you guys talk about the risks of searching for "dicey"/risky characters/terms and how you protect your data/server from malicious searches.

nerdydata
When the launch craziness winds down, we plan to start a blog and nerd the fuck out. We had so much fun making this and want to help others get past the insane obstacles we faced while building a search engine.
013
Is there any content filtering? When I went on to Image Locator, then "See an example" it will search for imgur.com. The first page of results shows 'thebiglistofporn.com', 'entensity.com'(Not sure if it's NSFW, but it looks it from the thumbnail.)

I know it's not too much of a big deal, but porn thumbnails wasn't something I was expecting to see on a websites example page.

nerdydata
We do not filter content, but thanks for letting us know about that thumbnail on the example page.
michaelmior
Looks pretty cool. I really dig the design. I was a little annoyed that I seem to have 10 credits without noticing and used them up just messing around. (I can't really tell because clicking on the credits doesn't do anything). It would be nice to make this more obvious up front. Anyway, I can see this being a pretty useful tool in some scenarios.

Also note that "Locater" is spelled "Locator" (sidebar on the left)

[deleted]
(comment deleted)
ilikehunny
your site has a really bad user rating? http://www.mywot.com/en/scorecard/nerdydata.com?utm_source=a...
sadkingbilly
I've never trusted wot. They are abused by groups of "wot cops" that organize in the forums and target web sites to bash (deserved or not).

As far as the site goes, it looks like they used to be a more generic seo search engine, as recently as a few months ago.

skinnymuch
That site is awful. Only thing worse is using BBB for credibility.
kevrone
This is awesome. Especially for finding use cases and implementation examples of front-end libraries.
nirvanatikku
That was my first thought, too. I checked out where my various projects are being used on the web, something I've been wanting to see for some time, and was pleased to learn about various properties using it.
tehaaron
Exactly what I would use it for but then I realized it wasn't free and to me its not worth the money.
snake_plissken
Good god, my attention span just reached terminal velocity. I like. Very cool and original.
[dead] mkessy
This is really cool. As a wanna be web dev I would love to hear about your guys' design process and the technologies you're using.

I'm curious about how credits will work. It seems that with traditional search engines like google, people often make many queries and subsequently refine their searches based on the results, so I could search for "best new car" look at the results then decide I might get better results with "highest rated new car" etc. If I'm paying by credits that doesn't seem like the best option since I'll waste credits, so it seems very important to know exactly what I'm looking for ahead of time. It looks like you guys understand this because there are some tools to refine searches.

I'm also wondering if you guys plan on adding more tools to analyze the search results, or increase details of queries. I could imagine searching by css selector syntax or regexp, or you could even design your own basic language to query by, especially if your userbase is pretty technical.

Looks great so far I look forward to following you alls progress.

jwarkentin
I thought it would be really cool until I realized the whole credit thing. It often takes many permutations of a search to find the results you're looking for. If each attempt costs money and the whole thing might be a waste in the end then it's hardly worthwhile.

I've always been annoyed that Google won't search for symbols. As a programmer that makes searching for certain things anywhere from difficult to impossible. I was hopeful that this would be a search engine I could use to search for code snippets or queries that involve symbols.

Let's say, for example, that I was browsing through some PHP code and saw a double dollar sign ($$) and wanted to know what it was used for (yes, I already know, it's just an example). I might try searching for something like 'PHP $$'. This search engine is useless for that. That's the real problem I need a search engine to solve.

ultimatedelman
agree about the credits. i know you're trying to monetize, but forcing me to pay for searches (and view more results) is a bit on the ridiculous side. there are better ways to monetize than that.

from a UX perspective, any time you _prevent_ a user from doing something, ie, you present them with the option to view more things and when they click on it, it says you need to pay to view the things, that is BAD UX. better UX would be NOT presenting the user with the option to view more, but where they would look for this feature, have a link that says something like, "want to view more results? upgrade to our premium plan!" Much better UX because as a user I'm not tricked into thinking I'm getting something and not having it be delivered.

but yeah, unless you're more explicit about the credits thing, it's going to be your downfall. i like the design and the results that i got from my one search, though :)

jwarkentin
I think charging for search in general is going to be their downfall. There are so many search engines these days that without bringing something truly revolutionary to the table, you'll be hard pressed to get much notice even offering it for free. Charging for it? Not a chance. There are other ways to monetize search.
ollerac
I don't get how to use the seo search. How do I search for all websites that have a certain name in their title for example?

    <title>name</title>
Just returns the exact match.

And how do I search meta descriptions and meta keywords?

kyle_martin1
This would be a very good tool to find XSS vulnerable websites.
cwings
Or find all sites with a vulnerable installation of wordpress.
brbcoding
Whoa... Searches must be based on IP. Shared with a co-worker and they can see all of my searches, and I can see theirs.
[deleted]
(comment deleted)
jwcrux
Just a small request - it would be way cool to have a "site:" filter like in Google searches. A lot of the results found are irrelevant to me as it is.

Cool site - great work!

nerdydata
We have a Refine Tool that allows for AND,OR,NOT filters. It's only available for paid users, but really allows you to push the limits and get the exact query you want. Thanks for the feedback!
nerdydata
Did we mention our "Basic" account is free and gives you 200 credits to use? https://search.nerdydata.com/pricing?hackerNews
bapi
No , because yesterday it was 49$ or 99$ Idon't remember exactly :). Good move I think to let people like me use it and find value in it as the promise is great.
sachin0235
while it is good, i am wondering what it can do which google can not? For example i searched for android accountmanager and results i got were far less productive than what google returns me back. If you want to build it, build it the way it perceives user perspective well