HN Explorer

Show HN: I built haystack – Google for workplace knowledge (haystack.it)

46 points by _vxw6 ↗ HN
tl;dr: haystack is a workplace search engine for devs that runs in the browser.

Iv'e started working on haystack recently because I feel modern workplaces are in dire need of a good workplace search product that is free to use just like google.

Information is scattered between too many communication channels and platforms, we communicate with our peers through slack and email, share docs and specs on confluence, work with tickets on jira, commit code and have discussions on github, not to mention all the .docx, .ppt and .pdf that fly around the organization.

Existing search featuers plain suck, if you tried using confluence search you know what I mean, keyword search is terrible. Even when you find relevant looking results, they require you to commit to entering the page, and scroll through to get to the relevant paragraph.

What does haystack do?

  - Enable you to search all your workplace applications from one place. (slack, confluence, notion, jira, github, outlook, gmail, etc...).  
  - Natural language queries ("How to do X”, "Do we support Y", "How do I connect to Z").  
  - No download, all the magic happens in the browser.  
  - Local browser storage option.  
  - Code references embedded in search results.
Example, "How to connect to integ2 machine" on haystack could give you: "I've finished setting up the second integration server ... ssh connection details"

  ssh -i private.pem ubuntu@ec2-integration2.eu-west-1.compute.amazonwes.com
aggregated from a slack communication you had a while ago.

Next Steps

Fine-tuning haystack for lower-end laptops with no dedicated GPU, public release date - Feb/March 23.

If you would like to get early access + you have dedicated graphics, there's a button in our landing page, and my email address.

I'll be here in the comment section!

12 comments

[ 3.0 ms ] story [ 29.1 ms ] thread
_vxw6
Regarding under the hood details,

getting haystack up and running in the browser was quite a challenge, but here's what I ended up using:

  - Storage: IndexDB browser API for local browser storage, it stores read permission tokens to apps, the AI models and the document index.
  - Indexing: fine-tuned TinyBERT-based bi-encoder for indexing documents, messages and emails.
  - Searching: cosine similarity between query embedding and the built index, and then rerank using tuned TinyBERT cross-encoder.
  - Building useful search resuts: search result building involved fine-tuning a t5-small model for summarization and text regression.
  - Performance nodejs->browser js adaptations, wasm rewrites in rust for performance.
mritchie712
Nice! Do you only stored documents I visit in the browser (e.g. a specific slack thread / JIRA issue) or are you querying all the API's for each supported app?
_vxw6
So I'm specifically querying all the APIs of the applications you connected, and indexing all documents and threads in local browser storage.
luckman212
Upvoting for the concept alone!

This is a problem that 100% of the companies I work with suffer with. I wish every manager who showed up didn't force their own stack of tools downstream... When poor Paul wants to find the company's WFH policy, he needs to check Slack, then Google Drive, then Dropbox, then some file server somewhere, and finally ask Jennifer from HR who gives him a link to some TiddlyWiki page to find it.

Good luck with the project and happy new year!

_vxw6
This is heart warming to hear! Happy new year!
sandkoan
Looks pretty cool! I've been working on something similar, but was a bit discouraged by market saturation in the universal search space.... How does this compare to something like, say, Glean?
_vxw6
What was the name of your product hmu hey@haystack.it, would love to learn from your experience.

It's self-hostable, doesn't store your data, more focused on developers and bottom up motions, I want you to be up and running within 10 seconds.