Show HN: R2R V2 – A open source RAG engine with prod features (github.com)

Sure, happy to.

R2R is built around RESTful API and is dockerized, so devs can get started on app development immediately.

The system was designed so that devs can typically scale data ingestion up to provider bottlenecks w/out extra work.

We have implemented user-level permissions and high level document management alongside the vector db, which most devs need to build in a production setting, along with the API and data ingestion scaling.

Lastly, we also log every search and RAG completion that flows through the system. This is really important to find weaknesses and tune the system over time. Most devs end up needing an observability solution for their RAG.

All of these connect to an open source developer dashboard that allows you to see uploaded files, test different configs, etc.

These basic features mean that devs can spend more time on iterating / customizing their application specific features like custom data ingestion, hybrid search and advanced RAG.

I'm just seeing this now.

The key advantages can be extracted from the response above to Kluless -

R2R is built around RESTful API and is dockerized, so devs can get started on app development immediately.

The system was designed so that devs can typically scale data ingestion up to provider bottlenecks w/out extra work.

We have implemented user-level permissions and high level document management alongside the vector db, which most devs need to build in a production setting, along with the API and data ingestion scaling.

Lastly, we also log every search and RAG completion that flows through the system. This is really important to find weaknesses and tune the system over time. Most devs end up needing an observability solution for their RAG.

All of these connect to an open source developer dashboard that allows you to see uploaded files, test different configs, etc.

These basic features mean that devs can spend more time on iterating / customizing their application specific features like custom data ingestion, hybrid search and advanced RAG.

What do you want to do with the data after ingesting?

LlamaIndex can ingest directories if you want to do bulk.

I'd like to know this too. A quick: "take these docs as input, ingest and save, now sit there providing an API to get results" service guide.

All the pipelines are async, so for ingestion we have typically seen that R2R can saturate the vector db or embedding provider. We don't yet have backpressure so it is up to the client to rate limit.

Ingestion is pretty straightforward, you can call R2R directly or use the client-server interface to pass the html files in directly to the ingest_files endpoint (https://r2r-docs.sciphi.ai/api-reference/endpoint/ingest_fil...).

The data parsers are all fairly simple and easy to customize. Right now we use bs4 for handling HTML but have been considering other approaches.

What specific features around ingestion have you found lacking?

Currently R2R has out of the box logic for the following:

csv, docx, html, json, md, pdf, pptx, txt, xlsx, gif, jpg, png, svg, mp3, mp4.

There are a lot of good questions around ingestion today, so we will likely figure out how to intelligently expand this.

For mp3s we use whisper to transcribe, for videos we transcribe with whisper and sample frames to "describe" with a multimodal model. For images we again transcribe to a thorough text description - https://r2r-docs.sciphi.ai/cookbooks/multimodal

We have been testing multi-modal embedding models and open source models to do the description generation. If anyone has suggestions on SOTA techniques that work well at scale we would love to chat and work to implement these. Long run we'd like the system to be able to handle multi-modal data locally.

You are right that the quickstart is pretty large, we will think about how we can trim that and show only the essentials.

To do what you are requesting is pretty easy, you can just launch the server and use the client directly. The code would look like this:

```python

from r2r import R2RClient

base_url = "http://localhost:8000" # or other

client = R2RClient(base_url)

# load my_file_paths

...

response = client.ingest_files(file_paths=my_file_paths)

# optionally set metadata, document ids, etc.. [https://r2r-docs.sciphi.ai/api-reference/endpoint/ingest_fil...]

```

As someone without no experience with RAG in production, I'm curious how effective you've found HyDE to be in practice.

We agree that ingestion and extraction are a big part of the problem for building high quality RAG.

We've talked to a lot of different developers about these problems and haven't found a general consensus on what features are needed, so we are still evaluating advanced approaches.

For now our implementation is more general and designed to work across a variety of documents. R2R was designed to be very easy to override with your own custom parsing logic for these reasons.

Lastly, we have been focusing a lot of our effort on knowledge graphs since they provide an alternative way to enhance RAG systems. We are training our own model for triples extraction that will combine with the automatic knowledge graph construction in R2R. We are planning to release this in the coming weeks and are currently looking for beta testers [we made a signup form, here - https://forms.gle/g9x3pLpqx2kCPmcg6 for anyone interested]

Try sonnet 3.5 image understanding.

I want to second this. It seems like document chunking is the most difficult part of the pipeline at this point.

You gave the example of unstructured PDF, but there are challenges with structured docs as well. We’ve run into docs that are hard to chunk because of this deeply nested and repeated structure. For example, there might be a long experimental protocol with multiple steps; at the end of each step, there’s a table “Debugging” for troubleshooting anything that might have gone wrong in that step. The debugging table is a natural chunk, except that once chunked there are a dozen such tables that are semantically similar when decoupled from their original context and position in the tree structure of the document.

This is one example, but there are many other cases where key context for a chunk is nearby in a structured sense, but far away in the flattened document, and therefore completely lost when chunking.

I run into the same issue with an internal company RAG, all unstructured data in PDFs but even once converted to markdown, they still need fine-tuning and a lot of manual intervention.

It feels like we are inching closer to automating this type of thing, or at the very least brute-forcing it in like the LLM race is trying to do with bigger models and larger contexts.

Will have to play with this over a weekend and see what it might help me with :)

Any one here exploring to solve extraction/parsing problem for RAG, do try LLMWhisperer[1].

Try it with complex layout documents -> https://pg.llmwhisperer.unstract.com/

If anyone wants to solve for RAG right from loading from source, extraction, and sending processed data to destination/API, try Unstract [2] (it is open-source)

[1] https://unstract.com/llmwhisperer/

[2] https://github.com/Zipstack/unstract

Danswer supports pdf natively, I’ve been trialing it and it works pretty well

PaddleOCR seemed to be a good library for locating and translating text. I've been puzzling over how to translate something like a simple letter form into a LLM translatable format.

I think the serious problem is most of these LLMs are already built on-top of garbage so you're already the GI and just trying to match that as best you can.

I'm really interested in learning more about this (multimodal LLM + OCR approach for PDFs), do you have a writeup anywhere or something open source?

No not yet, I've had difficulty getting these different providers to work together on integrations. If you have any suggestions we are all ears.

In the meantime we've built our own dashboard which shows ingested documents, and has a customizeable chat interface - https://github.com/SciPhi-AI/R2R-Dashboard.

It's still a bit rough though.

See the guide here - https://r2r-docs.sciphi.ai/cookbooks/local-rag

we have instructions for getting setup and running w/ ollama. It should be pretty smooth.

Yes, there is a guide to running R2R with local models here - https://r2r-docs.sciphi.ai/cookbooks/local-rag

great question, I can talk about how we do the more challenging "List all YC founders that worked at Google and now have an AI startup."

For this we have a target dataset (the YC co directory) that we have around 100 questions over. We have found that when feeding an entire company listing in along with a single question we can get an accurate single answer (needle in haystack problem).

So to build our evaluation dataset we feed each question with each sample into the cheapest LLM we can find that reliably handles the job. We then aggregate the results.

This is not perfect but it allows us to have a way to benchmark our knowledge graph construction and querying strategy so that we can tune the system ourselves.

No we don't have any explicit code graph tools. Sourcegraph might be a good starting point for you, their SCIP indices are pretty nice

its a optional dep used for kgs

yes, I think so.

There are many exciting products that enable users to perform RAG on their own data, the growing number of use cases highlights the need for developer-friendly tools to build such applications.

While building our own RAG system with existing tools, we encountered numerous challenges in experimentation, deployment, and analysis. This led us to create our own solution that is truly developer-friendly.

You can check our docs for more details: https://r2r-docs.sciphi.ai/introduction

thanks, this is really solid feedback - we will make a more inclusive docker image to make the setup easier/faster.

Think of R2R as an SDK with an out of the box admin dashboard / playground that you can plug into.

You can have a look at Langroid[1], a multi-agent LLM framework from ex-CMU/UW-Madison researchers, in production-use at companies (some have publicly endorsed us). RAG is just one of the features, and we have a clean, transparent implementation in a single file, intended for clarity and extensibility. It has some state of the art retrieval techniques, and can be easily extended to add others. In the DocChatAgent the top level method for RAG is answer_from_docs , here's the rough pseudocode:

  answer_from_docs(query):
    extracts = get_relevant_extracts(query):
        passages = get_relevant_chunks(query):
            p1 = get_semantic_search_results(query)   # semantic/dense retrieval + learned sparse
            p2 = get_similar_chunks_bm25(query)       # lexical/sparse
            p3 = get_fuzzy_matches(query)             # lexical/sparse
            p = rerank(p1 + p2 + p3)        # rerank for lost-in-middle, diversity, relevance
            return p
        # use LLM to get verbatim relevant portions of passages if any
        extracts = get_verbatim_extracts(passages)          
        return extracts
    # use LLM to get final answer from query augmented with extracts
    return get_summary_answer(query, extracts)

[1] Langroid https://github.com/langroid/langroid

Here

https://r2r-docs.sciphi.ai/cookbooks/local-rag