HN Explorer

Show HN: Index and search *all* your documents (github.com)

21 points by spapas82 ↗ HN
Hey HN! I've build a simple tool to index and search your documents. This uses two great open source libraries: apache tika (for extracting content from docs) and apache lucene (for searching). It's been built with kotlin ktor as a web framework.

You can index all kind of files (i.e doc, docx, xls, ppt, pdf, txt, html even ORC pdfs) and then search them using very advanced queries like "always contain X", "never contain X", "X near Y", wildcard search, proper stemming support etc.

We're using it on my work where we have hundreds of thousands of doc/docx/pdf files and it works flawlessly!

6 comments

[ 2.8 ms ] story [ 20.4 ms ] thread
namanyayg
nice work on this! i've been looking for something like this to manage my own docs.

one thing that caught my eye was the mention of 'proper stemming support' - can you elaborate on how you're handling stemming? are you using a specific library or rolling your own implementation? also, have you considered adding any sort of faceting/search filtering to the results?

spapas82
Hello thank you for your kind words! Stemming is handled by the corresponding Lucene analyzer which you can configure depending on your language, I'm explaining this a bit here: https://github.com/spapas/doc-parser-searcher/?tab=readme-ov...

i.e I'm using org.apache.lucene.analysis.el for our documents (they are in greek) and for english you should can org.apache.lucene.analysis.en.EnglishAnalyzer instead.

There already are some filters that are indexed for each file and can be applied when searching:

- created/modified/accessed date

- full path

- filename

- extension

unstatusthequo
Yes this is great. I looked into stitching these together but always figured it would be a huge undertaking. Consider looking at TensorFlow for OCR which should be much better and maybe faster.
spapas82
Thank you for your kind words! That's a good suggestion for the OCR I'll take a look at it.
compressedgas
Do the search results have document page numbers?
spapas82
You mean where each result is found inside the document? Right now no but it should be possible and also results are highlighted.