HN Explorer

Ask HN: How difficult is a search engine MVP that just works?

9 points by bluu00 ↗ HN
what it is, i mean, what's stopping you from making the next google search?

Tf-Idf? No. Just kidding. Infromation retrieval, text mining, ML/DL ?! what is going on with this field ! Every other resource seems outdated? What is the state of art ?

Reading some of these posts : https://boyter.org/2010/08/build-vector-space-search-engine-python/

https://www.dr-josiah.com/2010/07/building-search-engine-using-redis-and.html

https://stevenloria.com/tf-idf/

https://stories.algolia.com/a-search-engine-in-css-b5ec4e902e97

8 comments

[ 2.4 ms ] story [ 35.2 ms ] thread
bluu00
And to add, i am not looking for a "how to learn"?

Neither it is some site-wide search que.

And not a business model as well.

Not a "privacy first but no results" search.

Something that works.

a3camero
I’ve done this over the last year on a tiny scale for my own needs: gorillafind.com. It’s from scratch and just for government sites to sidestep some of the challenges (but so far only has 50 sites). The cost per site is around $1/mo for crawling, indexing, converting file formats and then serving up results. It’s difficult but not impossible and very educational. If you’d like to hear more about doing it yourself and some of the challenges feel free to email me with the contact info on the site. My system isn’t open source but I’m more than happy to chat about the research I’ve done and how you can make one.

I’d start off with not doing state of the art because it’s overkill for an “MVP”. And if you don’t need proper browser rendering of pages, there’s open source crawlers out there like Nutch that might work. If you’re making one yourself, the outdated academic papers and presentations by search companies are a good resource as the basic ideas of crawling and indexing haven’t changed too much (even if ranking and other components have changed a lot). A search engine is really a set of related components and there are many examples out there to use as inspiration for your MVP.

bluu00
> gorillafind.com

Interesting. (ﾟヮﾟ) You seem to know well that search is beyond ranking and retrieval. [1]

> I’d start off with not doing state of the art because it’s overkill for an “MVP”.

Can a search engine "MVP" sustain without using sota methods/practices?

What should be the initial scale for a search MVP for which it just works? In the modern realm.

[1] : https://medium.com/startup-grind/what-every-software-enginee...

just a note (you can ignore) : I am actually a believer of iterations in development and incremental optimisations. But even a minimum viable product should be able to work.

techdragon
I built a “search engine” but unfortunately my approach lacked any implicit ranking mechanism.

TLDR: my regex search engine needs result ranking to be more useful before I consider showing it to other humans.

ioli
I don't know how feasible it is, but couldn't you use ElasticSearch ???
bluu00
what is elastisearch? IDK ¯\_(ツ)_/¯

EDIT #1: you're cool. but i really dont know what elastisearch is. ¯\_(ツ)_/¯ And a google "search" hurts at the moment.

EDIT #2 :

> multiple people respond and say that Lucene basically does the same thing Google does and that Lucene is poised to surpass Google's capabilities in the next few years. It's been long enough since then that we can look back and say that Lucene hasn't improved so much that Google is in danger from a startup that puts together a Lucene cluster. [1]

[1] : https://danluu.com/sounds-easy/