Launch HN: Danswer (YC W24) – Open-source AI search and chat over private data
Quick Demo: https://youtu.be/hqSouur2FXw
Originally Danswer was a side project motivated by a challenge we experienced at work. We noticed that as teams scale, finding the right information becomes more and more challenging. I recall being on call and helping a customer recover from a mission critical failure but the error was related to some obscure legacy feature I had never used. For most projects, a simple question to ChatGPT would have solved it; but in this moment, ChatGPT was completely clueless without additional context (which I also couldn’t find).
We believe that within a few years, every org will be using team-specific knowledge assistants. We also understand that teams don’t want to tell us their secrets and not every team has the budget for yet another SaaS solution, so we open-sourced the project. It is just a set of containers that can be deployed on any cloud or on-premise. All of the data is processed and persisted on that same instance. Some teams have even opted to self-host open-source LLMs to truly airgap the system.
I also want to share a bit about the actual design of the system (https://docs.danswer.dev/system_overview). If you have questions about any parts of the flow such as the model choice, hyperparameters, prompting, etc. we’re happy to go into more depth in the comments.
The system revolves around a custom Retrieval Augmented Generation (RAG) pipeline we’ve built. During indexing time (we pull documents from connected sources every 10 minutes), documents are chunked and indexed into hybrid keyword+vector indices (https://github.com/danswer-ai/danswer/blob/main/backend/dans...).
For the vector index (which gives the system the flexibility to understand natural language queries), we use state of the art prefix-aware embedding models trained with contrastive loss. Optionally the system can be configured to go over each doc with multiple passes of different granularity to capture wide context vs fine details. We also supplement the vector search with a keyword based BM25 index + N-Grams so that the system performs well even in low data domains. Additionally we’ve added in learning from feedback and time based decay—see our custom ranking function (https://github.com/danswer-ai/danswer/blob/main/backend/dans... – this flexibility is why we love Vespa as a Vector DB).
At query time, we preprocess the query with query-augmentation, contextual-rephrasing, as well as standard techniques like removing stopwords and lemmatization. Once the top documents are retrieved, we ask a smaller LLM to decide which of the chunks are “useful for answering the query” (this is something we haven’t seen much of elsewhere, but our tests have shown to be one of the biggest drivers for both precision and recall). Finally the most relevant passages are passed to the LLM along with the user...
133 comments
[ 1.9 ms ] story [ 223 ms ] threadBasically, (as I understand it) OpenWebUI is more like a ChatGPT frontend and we're more like a unified search with an emphasis on LLMs.
We hear this a lot - we take it to mean we've found something that people like and need!
I’m curious about the busines model. I see more and more YC companies that are FOSS, which is nice.
Why did you choose a FOSS license instead of proprietary with a license fee?
What are your plans for securing funding for further development?
Would be very interesting to hear your thoughts on this.
Congrats on the launch!
Especially with small teams that have no budget, they can get value and never need to talk to us. We hope to grow with them.
In terms of funding and the economics of it, we have a cloud currently which is paid and we're developing additional features that are not free. For example, identifying experts on the team when the system is unable to find the information to answer the question directly. This is an example of a feature that large teams are likely willing to pay for but small teams won't need (as they know each other intimately).
Two things to note though.
The APIs are intended for serving the Danswer frontend. The functionality is generally complete for similar use cases but it's not documented so you have to look at the code.
If you're overusing the API on the cloud without providing your own OpenAI key, we will likely have to shut down the instance to prevent losing too much on inference fees.
I guess the main problem is the "private" aspect, if I've understood your goals correctly. Since most SaaS products lock down the private data unless you pay enterprise fees for compliance tooling.
For instance, if you want to ingest data from private Slack channels or Notion groups, you have to get the users in those groups to add your bot to them, otherwise there's no way of your service getting access to the data. It's possible, just a bad UX for users.
That said, built-in search for most SaaS products built after 2015 is generally quite good (e.g. Slack has an internal Learning to Rank service for a while now, which makes their search excellent: https://slack.engineering/search-at-slack/), so you'd be solving for companies like Webex and Confluence where their internal search is not great. At companies like Google they have internal search across products, which is the ideal end state, but have the benefit that they own the source code for most of their internal products.
So regarding getting access to read knowledge from the different tools, it depends tool by tool but a lot of them have API keys or options for app integrations available in the free tier (GitHub, Google Drive, Confluence come to mind). Other tools don't have a free tier and you just get access to the API keys as a part of paying for the service. I think there are probably tools that require a premium fee to get integration access but I'm not aware of any personally.
For the SlackBot, it can add itself to public channels but for private channels someone needs to add it. It is what it is sadly.
About search being available for most SaaS products: SaaS tools are definitely improving their own searches. But I still think a single place to search and aggregate data has significant value. For example, as an engineer by training, often getting the full picture for some customer escalation includes reading Slack threads, Confluence Design docs, old Pull Requests on GitHub. Would be nice to get it all in one place.
This is what I mean -- previously I built a similar search engine on top of slack, notion, etc., but didn't launch the product because I thought that requiring users to constantly add bots to private channels would be a subpar experience. I thought this would be a blocker for good UX, so didn't go further, but maybe you'll find a nice solution!
Searching over public internal data is addressed by a few existing tools, but it's the private aspect which is pretty difficult to handle and disastrous to get wrong when managed ad-hoc - e.g. someone accidentally adds the bot to a private slack group called #layoffs :) so you'd want this handled properly and centrally.
I guess you'll also need to handle privacy well, ~maybe it's OK when run as a SaaS for db admins to have access to ingested data, but if it's OSS then the people that run it probably shouldn't be able to read the private data that's ingested, so now you need to handle search over encrypted data, which is a fun problem :D
I don't understand the distinction here. If Danswer runs a SaaS version then yes I agree they can have a license agreement that lets their DB Admins see data in some cases which is fine. That seems an orthogonal issue to if a company is running the OSS version internally, in which case presumably their administrator can see all docs (but software administrators usually can do this anyway).
For self-hosted deployments, usually a select few admins who have setup the plumbing on AWS do have access (but as nl has mentioned, these people usually have access to superuser access on the tools we connect to anyways so this is a noop).
For Slack it would look something like: get the users in the Slack channel, map those Slack users to users in Danswer. Then only those users in Danswer will be able to get results from that channel.
(Disclaimer: we started with enterprise search too, and now we think a custom model is a better way to get to those goals) Also, the output needs to be integrated into their own workflows. Eg: Seeing a conversation on slack and using a bot fetch all the supplemental information to understand the context from say mail, docs etc. is mighty useful. Search using RAG is the easy part, the hard part is contextualizing it in a way it is immediately useful. That depends on understanding the company/domain lingo, understanding users, etc.
Private aspect is a bad UX but not a bottleneck. Remember, this is targeted towards power users looking to use it on a daily basis. Connecting the bot once to get started is fine as long as it adds value. If anyone offers data governance along with it (updating access on a daily basis and answers only from access), it could be a huge hit.
How do you control this deterministically? It sounds like the "hard part" is variation in prompting & selectively choosing the right data to include, both of which I could see being good enough right now but hard to deliver definitively.
The challenge is with the non-deterministic portions of the flow as you pointed out. Ensuring retrieval quality in out-of-domain datasets, guardrailing the LLM generation, working with conflicting or deprecated information are some of the interesting areas we're addressing. Happy to dive deeper on any aspect you're curious about, and I'm sure we can learn from the discussion as well.
Regarding contextualizing: we're currently working on organizational understanding and we're very excited about this one! We're embedding users based on the documents they authored or interacted with, the questions they have asked, description of projects they worked on, and the org chart. The thinking is that, there will always be questions that can't fully be answered via documentation alone. But in those cases we'll be able to recommend someone who might know. It also has the benefit of contextualizing the user asking so that we can surface more relevant results for them.
Adding the vectorisation locally is superb, I've played around with sbert models before and ability to run without GPU is going to simplify the process a lot.
Also a fun tidbit on the connectors, more than half of them now are built by open source contributors! We just have an interface that needs to be implemented and people have been able to figure it out generally.
Currently with large tables, it's not handled very well either. The more complete approach would be to pass the headers to the LLM and ask it to generate a formula to parse the data rather than feeding the whole table(s) to the LLM directly.
Some of the bigger items we want to target that require special flows are code search, SQL tables, and Excel/CSVs/TSVs
it looks like the best way to do that is understand the ooxml format in xlsx. it's all fairly easy to understand.
By asking the LLM to generate a formula though, it doesn't actually need to do any number crunching of its own which makes solving the challenge a bit more reliable when it comes to LLMs.
Would love to hear your perspective on the space!
However, what seems to be the business proposition is giving managers shallow access to documents but won't lead to rigorous information.
There's a few middle grounds where it can yield insights. like regulatory scenarios where you want to understand how public orgs satisfy permits with written plans.
however, what I don't believe will yield is the context size. when I want to explore my knowledge base, I need far more than 128k and there's sever orders of structure that language itself is not going to bridge.
Regarding the context size, the research community is doing some stunning work. If you're interested, you should check out the new Mamba (Transformers replacement) architecture: https://arxiv.org/abs/2312.00752
We couldn't use the more involved pipelines because we needed significant custom logic to enforce permissions, filters (like time filter, source filter, document-set filters), and other complexities. At that point, it's easier to write from scratch rather than conform to expectations of these third party libraries.
Depending on how custom the pipelines need to be, you'll either find that you've saved a huge amount of time using these libraries, or you'll find that you have no option but to switch off and build from scratch.
Is there any work or planned work around enterprise authentication and access? For instance, indexing Sharepoint in such a way where a user of Danswer isn't exposed to sharepoint information they wouldn't otherwise have access to?
Yes, there are several options for user authentication (Basic Auth, Google OAuth, OIDC, SAML).
Currently the RBAC is managed via the Danswer and this controls who has access to which documents (it's done at the connector level as it would be untenable to assign access to documents individually).
We're also working on automatically sync-ing permissions from the sources. Basically seeing which emails have access to each doc and mapping it to the Danswer users.
Is this something that you would find a lot of value in or is simple text processing of PDFs sufficient?
I know that might be a niche case tho.
Absolutely incredible work you’re doing tho wow, I’m very impressed by what you’re doing and the way you’re doing it. Even if you stopped now this is a masterpiece, so while yes I would definitely find a lot of value from being able to process images and graphs/tables, simply being able to process the text and cite it is already a superpower. Thank you for your amazing work!!!
For a personal use case, I'm thinking things like receipts. For work, I'm thinking OCR on architecture diagrams/etc.
We also do OIDC and SAML to integration with Identity Providers (IdPs) like Okta but that's part of the paid features. Ahhh please don't hate us!
For example, for us we have `VALID_EMAIL_DOMAINS=danswer.ai`.
I believe there are a variety of projects focused on that problem but would be good to latch onto one that handles and integrates externalities.
Speaking of which I did not see sentry in the list of connections or mentioned in an issue. Any plans there?
Without dropping any names, a big french company with 5000 people is actually doing this with Danswer and have found great success.
There is some info here: https://docs.danswer.dev/configuration_guide You'll also want to change the embedding model in the admin UI
Just the last 2-3 weeks have I had talks with enterprise companies regarding this topic. It seems to be on every CEO's agenda. I have talked to a couple of startups who wanted to do a similar thing to you. But they all feared Microsoft Copilot is not beatable. So they don't even try.
The hope is that by working with the community, we'll be able to incorporate the best ideas and contributions from a large pool of like-minded people to build something everyone can benefit from!
The OSS space has absolutely taken off in the NLP space and the excitement has bled over to developer platforms resulting in some outstanding projects, hopefully the same will happen with LLM applications.
https://huggingface.co/mistralai/Mistral-7B-Instruct-v0.2 - Mixture of Experts model like GPT-4 but this one is open source doesn't cost an arm and leg to run.
https://huggingface.co/mixedbread-ai/mxbai-rerank-xsmall-v1 - This is a very recent reranker, we will probably try it out. It's really nice that it's open source and small enough to run on CPU. I think an overall trend of the space in the next years will be to maintain/improve performance while shrinking the cost and hardware requirements.
The thing is, you don't need to beat copilot. Copilot may well be the worse system and Microsoft can still win because they offer the most enterprise-y solution. I wouldn't be worried about competing with them on functionality. But I also would never even try to outdo them on a business level.
There's a simple Document interface that needs to be implemented to provide stuff like title, content, link, etc. From there the rest of the Danswer code takes it and handles the indexing and making it available for querying.
There's a contributing guide here: https://github.com/danswer-ai/danswer/blob/main/CONTRIBUTING...
And a connector contributing guide here: https://github.com/danswer-ai/danswer/blob/main/backend/dans...
So hopefully teams will self-adopt Danswer and as they grow, they will keep using us!
For larger teams, the transparency and peace of mind of self-hosting an open source solution is also a major benefit. We've often heard from large teams that have adopted Danswer, that the customizability of it has been a driving factor in their adoption. They want to own the solution and they want to customize it specifically for their needs. At the very basic level, a lot of teams have swapped in domain specific embedding models and prompts but we've seem some significantly more involved customizations as well.
I'm particularly surprised by the speed considering all of the pre and post processing. I am doing some similar things and that is one bottlenecks. I'll dig in, but I'm curious what models you are using for each of these steps.
We're using E5 base by default but there's an embedding model admin page to choose alternatives. There's also an API for it if you know what you're doing, you can even set one of the billion+ parameter LLM bi-encoders if you want (but you'd need a GPU for sure).
When I was in consulting, being able to search our internal documents was absurdly painful.
We had all this data from client teams spanning decades and continents, but I could never find what I needed. It was all locked up in these silly PowerPoint files -- not even necessarily PDFs. I'd literally spend all night sometimes clicking through pages by hand.
Corporate knowledge management is an absolutely immense problem, and I'm thrilled to see you tackling it.
A great use case for sure!
Did you use a library to implement the integrations?
Alternatively, if you're talking about real time indexing to make it available to everyone immediately, there's an Ingestion API, where users can send documents in the expected format directly to the system, is this what you're thinking?
The integrations are built in house, some using client libraries of the particular tool (like the Atlassian python client library for example). We considered using Airbyte, LlamaHub etc. but we found that they don't support the full flexibility that we need, including pulling incremental updates and access permissions.
It's actually still in the code, just none of the connectors implement it atm: https://github.com/danswer-ai/danswer/blob/main/backend/dans...
If Danswer could surface the "official" answer, such as it is, but then also note how old the authoritative doc is, and which teams have been messaging about it and/or scheduling calls with related names more recently, and then tell me who's responsible on that team for knowing the actual procedure, I'd never need another tool!
As far as how outdated information is handled - we pass the most relevant documents along with metadata to the LLM. So in the case you mentioned, the LLM will be provided the procedure doc and the time it was updated, the relevant Slack messages and the times they were sent, and the call transcripts along with when the call happened. The LLM tends to handle this pretty well.
Additionally, during the retrieval phase, there's a time based decay applied based on the last time the document was updated. Also there is learning from feedback so users can upvote documents that are useful and downvote documents that are deprecated.
If you're just looking to upload a few personal docs into a chat assistant for your own use, probably Danswer is overkill and more complex than the effort is worth. If you're thinking of a team wide use case, then using Danswer makes sense.