Turns out removing the top million results from a search for Google... still returns google. Or google.com.au to be precise.
It's a cool idea, but I'm not sure it's working. I tried "american history" but it wouldn't return anything at all if I changed the "Remove the Top" dropdown.
I think remove results with my search term in the domain name and this would be perfect!
For example I searched for how to start a garden and I can guarantee that startagarden.com is junk. But indie see some useful advice from small blogs etc
And I just know any domain name optimized for a certain search term is going to be garbage.
I guess a simple version of this feature is you'd have a setting: "Exclude domains that contain my search term".
When the user clicks that you'd compare the domain name (removing all special characters) with my search term (also removing all special characters and white space). Maybe compare via edit distance and exclude if it passes a threshold?
Although edit distance might not work too well, perhaps looking at the longest common substring and if it's > say 90% of the length of my query exclude it?
I guess it would take some playing around. But there should be a good algorithm to exclude domain names very similar to my query.
We're simply removing the top million (or what you select) sites. The results can be the same - it just means that that site isn't in the list of top one million sites on the web.
Yeah except now you get the content farms that don't know how to rank. A competitive spammy term like "pay day loans" still shows plenty of low quality sites because of the crazy number of sites looking to cash in on the term.
Great idea, although I think if you could explain it a bit better you could avoid the confusion like several of these comments are showing. I like how my Hacker Newsletter project shows up #2 when searching for Hacker News. :)
search engines can follow links (sometimes) through flash. and search engines know what the most popular sites are even without counting number of links to sites, so hiding links through flash (or js, whatever) wouldn't help
Only a little thing, could do with maintaining query strings between pages. It lost my query string and returned no results when I changed the drop down without me noticing.
I'm not sure this is a great idea. Predictability is a staple of a good user experience. Getting different results for the same query between users or sessions is bound to lead to broken expectations and frustration.
Great point. You might have a hard time competing with google on relevant results but you might be able to beat them on discovery, like a stumbleupon search engine.
I like the idea of offering it after the first search. I see the value of this kind of search engine as introducing some novelty/entropy into the system. I could imagine using it as a backup to primary search engines, in which case I'd definitely want to get some randomness going.
I wouldn't just want the "million-and-eleventh" site (so to speak) when clicking next.
Usually don't search unless I'm looking for something in particular, but just played with this for a good 15 minutes running random queries. The results are really good and at the same time I'm discovering sites I'd never otherwise see with Google.
I think search APIs like Yahoo BOSS allow you pass arguments that contain a black list of domains. I think it's the 'sites' argument that may be used like this: &sites=-google.com
What is the ranking used for the top million sites? A search result for "Australia" returns as the top result http://australia.gov.au, which Alexa ranks as 20,615 globally. Actually, a lot of the queries I tried returned Australian sites.
Not sure if I found an anomaly or what, but a simple search of "Privacy" returns results from thesaurus.com, merriam-webster.com, truste.com, kelloggcompany.com, and many more that are all in the top few thousand according to QuantCast and Compete.
Great idea though, will definitely try this out some more.
merriam-webster.com redirects to m-w.com - looking into the others.. Sometimes if a site has header redirects it gets lost in the filters. Thx for pointing this out.
ahh that explains it. I'm actually working on a personal project right now and this could help out quite a bit, so I am excited to see where this goes. Best of luck!
This is a great idea and I see myself coming back to this. It's a shame that a little blog on tumblr or blogspot gets taken out because it's under a big name domain - but this has spam related benefits too.
This is a breath of fresh air - I'm loving the unpredictability of the top results! It's like flicking through a new set of 1000 tv channels in a different country.
After a few test searches, this is surprisingly effective for things which I had resigned "un-findable" because of poor Google results. This is most apparent on non-technical things, in this case specific Jazz chord fingerings for a guitar class I am taking.
I am very interested as to what comes of this, or rather what is influenced by its implications.
Well that's one way to break out of the filter-bubble/echo-chamber I suppose. If only our best search technology was based on something better than a popularity contest :(
Why? If I search, say, for a game review, I don't care whether it comes from a popular website or a blog no one reads. In fact, the topmost websites are more likely to be biased, since they try to appease everyone and they also have strong relationships with publishers. The blog no one reads is nearly guaranteed to be honest (if not well-written).
This holds true for most topics I can think of. Moreover, if I ever need to read Wikipedia and such, I already know about those websites, and I can go there directly - no need to search. Shouldn't web search engines act like discovery tools?
> Shouldn't web search engines act like discovery tools?
If your business model is based on advertising that depends on masses of page views to generate value, then no. You want to be as generally useful as possible so that e.g. people use you as an (extremely inefficient) DNS service.
(Google Labs has a single optional feature available for search. Perhaps their arch doesn't make 20% or Labs projects a good fit for plugging in extra fancy search features?)
Wow that is pretty awesome. I reached some results I want that I could not find via popular search engines with hours of searching. Believe it or not, this engine is changing my life.
203 comments
[ 2.0 ms ] story [ 341 ms ] threadIt's a cool idea, but I'm not sure it's working. I tried "american history" but it wouldn't return anything at all if I changed the "Remove the Top" dropdown.
For example I searched for how to start a garden and I can guarantee that startagarden.com is junk. But indie see some useful advice from small blogs etc
http://howtogrowavegetablegarden.net/ http://www.grow-your-own-vegetable-garden.com/
And I just know any domain name optimized for a certain search term is going to be garbage.
I guess a simple version of this feature is you'd have a setting: "Exclude domains that contain my search term".
When the user clicks that you'd compare the domain name (removing all special characters) with my search term (also removing all special characters and white space). Maybe compare via edit distance and exclude if it passes a threshold?
Although edit distance might not work too well, perhaps looking at the longest common substring and if it's > say 90% of the length of my query exclude it?
I guess it would take some playing around. But there should be a good algorithm to exclude domain names very similar to my query.
http://www.apolis.org/index.php?option=com_content&view=...
http://www.alexa.com/topsites
"Bad news-- we're a top 100 hit for several of our main keywords. We'll have to change our URL scheme again."
Maybe we'd see a resurgence in Flash.
search engines can follow links (sometimes) through flash. and search engines know what the most popular sites are even without counting number of links to sites, so hiding links through flash (or js, whatever) wouldn't help
I wouldn't just want the "million-and-eleventh" site (so to speak) when clicking next.
You could use this on modern smartphone browsers: https://github.com/alexgibson/shake.js
But I like the idea of being able for users to, via a setting perhaps, add their own list of deny/include sites.
Thx for the comment.
http://millionshort.com/search.php?q=australia&remove=10...
http://millionshort.com/search.php?q=somalia&remove=1000... -- another Australian site.
Great idea though, will definitely try this out some more.
What's your source for the top million sites; where do you get your site list from for the other results?
What the hell is their problem?
Great work!
I am very interested as to what comes of this, or rather what is influenced by its implications.
This holds true for most topics I can think of. Moreover, if I ever need to read Wikipedia and such, I already know about those websites, and I can go there directly - no need to search. Shouldn't web search engines act like discovery tools?
If your business model is based on advertising that depends on masses of page views to generate value, then no. You want to be as generally useful as possible so that e.g. people use you as an (extremely inefficient) DNS service.
(Google Labs has a single optional feature available for search. Perhaps their arch doesn't make 20% or Labs projects a good fit for plugging in extra fancy search features?)