Ask HN: Building a Better Google

10 points by agencies 5y ago ↗ HN

If you had the 4 billion pages Google had on its index in 2004 in elasticsearch today, what would it take to replicate 2004-era web search quality using that static index?

What relevant research or projects are trying to make these sorts of algorithms and data accessible as a future commodity people can build on top of?

4 comments

[ 3.3 ms ] story [ 30.2 ms ] thread

mikewarot 5y ago ↗

I'd use word2vec to embed all the words into a vector space to try to get more inclusive search results.

I'd also try to separate out multiple meanings of phrases if possible... for example "hypertext markup" could mean the HTML language, or it could mean actually marking up hypertext (annotation). I'd let the user have some way to disambiguate the meanings.

ipaddr 5y ago ↗

In google cared about: - title, description meta tag - PageRank was just starting - Single word matching domains to search terms - Anchor Text

The quality of the search algo wasn't better the sites indexed were better.

immortal3 5y ago ↗

I have been thinking about these a lot lately. There is lot more scope on improving current search results.

I believe simple two stage system might be enough to produce decent system. Stage - 1. query expansion and reverse index based retrieval Stage - 2. re-ranking based on few combination of heuristics. (page rank + word embedding + query analysis)

Would you like to talk more about this ? I have email address in my profile.

blackcats 5y ago ↗

First I would ignore bad backlinks as they’re not a good indicator of quality. The better your page, the more likely a competitor will start creating thousands of bad links.