Ask HN: I want to dive into how to make search engines

Let's start right away!

    urls_old = []
    urls_new = [ 'news.ycombinator.com' ]

    while urls_new.length>0:
        spider(urls_new[0])

It's pretty simple: You crawl a website and search for all links, then you crawl all the links from these linked websites and so on.

You can still see some of my code here: https://github.com/Wronnay/search-lib

Smartness (NLP): https://web.stanford.edu/~jurafsky/slp3/

There are also newer techniques, like "deep learning" for search. Not sure what's a good resource for that (using ML to learn the scoring functions is more than a decade old field (learning to rank), it just surfaced again because of the new ML techniques).

Check the table of contents of those books. Courses which mention chapters or sections from them are probably a good choice.

You can also check Apache Lucene and do a deep dive to see state-of-the-art implementation (Lucene in Action is a good introduction).

You can decompose a "search engine" into multiple big components and figure out what you want to look at first:

(1) web crawler/spiders

(2) database cache of web content -- aka building the "search index"

(3) algorithm of scoring/weighing/ranking of pages -- e.g. "PageRank"

(4) query engine -- translating user inputs into returning the most "relevant" pages

Each technical topic is a sub-specialty and can be staffed by dedicated engineers. There are also more topics such as lexical analysis, distributed computing (for all 4 areas), etc.

If you're mainly focused on experimenting with programming another ranking algorithm, you can skip part (1) by leveraging the dataset from Common Crawl: https://index.commoncrawl.org/

Here are some videos about PageRank:

https://www.youtube.com/watch?v=JGQe4kiPnrU , https://www.youtube.com/watch?v=qxEkY8OScYY

... but keep in mind that the scope of those videos omits all of (1), (2), and (4).

Truth is, this track you'll set up a search engine in about 20 minutes:

Document Store/NoSQL: https://elastic.co https://solr.apache.org

Classical AI: https://youtube.com/playlist?list=PLUl4u3cNGP63gFHB6xb-kVBiQ...

Stat ML: https://www.tensorflow.org/tutorials https://youtube.com/playlist?list=PLoROMvodv4rMiGQp3WXShtMGg...

Frontiers: https://www.ycombinator.com/companies/?query=Search

https://blogs.cornell.edu/info2040/2011/09/20/pagerank-backb...

Google's algorithm is indecipherably complex today but in the early days the way search engines more or less worked was they crawled the web and the way they decided to rank the pages was by how many other pages had a URL reference pointing to it.

You can apply this idea today in private (or I suppose public) search engines in the same way to interesting results.

For example a search engine for for scientific papers might use page rank to sort papers that are cited by the most other papers.

Or if you were going to make a search engine for open source projects you could create a page rank algorithm based on what projects had dependencies to other projects.

Part of why Google's algorithm today is more complex than this is that people try to game whatever algorithm search engines commonly use. You may remember back in the 90s and 2000s people would do stuff like put back links to other websites in the source to try to game page rank. Today that kind of behavior has expanded into a whole cottage industry (unfortunately).

What's interesting though is that for a lot of more limited data sets is you have less of that SEO type problem.

Whichever kind you're building good luck! Search engines usually wind up being petty cool (and profitable).

You can get a taste of the kinds of things covered in this field by looking at this class site (among many others)

https://web.stanford.edu/class/cs276/

That said, to build a modern competitive search engine, you'll need to look into software engineering, distributed systems, artificial intelligence, machine learning, linguistics, computer vision, signal processing, graph theory, databases, scheduling, machine translation, and FSM-only-knows what else.

[1]: https://en.wikipedia.org/wiki/Information_retrieval

https://www.amazon.com/Managing-Gigabytes-Compressing-Multim...

- Tries (patricia, radix, etc...)

- Trees (b-trees, b+trees, merkle trees, log-structured merge-tree, etc..)

- Consensus (raft, paxos, etc..)

- Block storage (disk block size optimizations, mmap files, delta storage, etc..)

- Probabilistic filters (hyperloloog, bloom filters, etc...)

- Binary Search (sstables, sorted inverted indexes, roaring bitmaps)

- Ranking (pagerank, tf/idf, bm25, etc...)

- NLP (stemming, POS tagging, subject identification, sentiment analysis etc...)

- HTML (document parsing/lexing)

- Images (exif extraction, removal, resizing / proxying, etc...)

- Queues (SQS, NATS, Apollo, etc...)

- Clustering (k-means, density, hierarchical, gaussian distributions, etc...)

- Rate limiting (leaky bucket, windowed, etc...)

- Compression

- Applied linear algebra

- Text processing (unicode-normalization, slugify, sanitation, lossless and lossy hashing like metaphone and document fingerprinting)

- etc...

I'm sure there is plenty more I've missed. There are lots of generic structures involved like hashes, linked-lists, skip-lists, heaps and priority queues and this is just to get 2000's level basic tech.

If you are comfortable with Go or Rust you should look at the latest projects in this space:

- https://github.com/quickwit-oss/tantivy

- https://github.com/valeriansaliou/sonic

- https://github.com/mosuka/phalanx

- https://github.com/meilisearch/MeiliSearch

- https://github.com/blevesearch/bleve

- https://github.com/thomasjungblut/go-sstables

A lot of people new to this space mistakenly think you can just throw elastic search or postgres fulltext search in front of terabytes of records and have something decent. The problem is that search with good rankings often requires custom storage so calculations can be sharded among multiple nodes and you can do layered ranking without passing huge blobs of results between systems.

Source: I'm currently working on the third version of my search engine and I've been studying this for 10 years.

[0] https://www.tbray.org/ongoing/When/200x/2003/07/30/OnSearchT...

I'm working at an open-source project that builds an AI-powered search framework [0], and I've built some examples in very few lines of code (for searching fashion products via image or text [1], PDF text/images/tables search [2]) and one of our community members built a protein search engine [3].

A good place to start might be with a no-code solution like (shameless self-plug time) Jina NOW [4], which lets you build a search engine and GUI with just one CLI command.

[0] https://github.com/jina-ai/jina/

[1] https://examples.jina.ai/fashion/

[2] https://colab.research.google.com/github/jina-ai/workshops/b...

[3] https://github.com/georgeamccarthy/protein_search

[4] https://now.jina.ai

I think that Facebook got around this back in the day by having the user's device do the initial scraping. A side effect of this is that sometimes I'd post an article I found, but the preview image was blank because I submitted it too quickly or something.

Until we have a free publicly downloadable cache of all websites, similar to CoralCDN (is this defunct?), then building our own search engines is probably a nonstarter.

I asked a question about my ideal search engine a few days ago: https://news.ycombinator.com/item?id=32452318

Basically, a conversational semantic search engine like the ones in Star Trek. I also listed some technologies that can possibly help implement it. You might be interested in that list.

I wrote a small but somewhat complete search engine some time ago.

The steps are basically:

Have a queue with urls.

Download a page from the queue

Use a html parser to remove markup and get the links of the page. Add the links to a queue without the already visited ones.

Use a stemmer to clean up the text (porter stemmer or whatever).

Calculate an inverted index: https://en.wikipedia.org/wiki/Inverted_index

Save the stuff in an appropriate data structure (Hash or Tree).

Write a query engine for AND, OR and (all the words).

Calculate a simple page rank by counting the links to a page.

For just learning purposes it is not that hard but if you want to get all the crazy corner cases of the "real" web you will go insane.

The easy alternative:

Use a lib to crawl a page.

Plonk all the documents text into postgres with full text search or elastic search.