RAGCache: Efficient Knowledge Caching for Retrieval-Augmented Generation (arxiv.org) 33 points by PaulHoule 2y ago ↗ HN
[–] joshcho 2y ago ↗ Worth a comparison with RAPTOR, another tiered RAG system.https://arxiv.org/abs/2401.18059
[–] unraveller 2y ago ↗ cool but deep integration requires the LLM provider to run it and get privacy right, that'll be $premium plus tip.>RAGCache reduces the time to first token (TTFT) by up to 4x and improves the throughput by up to 2.1x compared to vLLM integrated with Faiss.Are RAG systems really so heavy they add 4x to the TTFT? [–] ComputerGuru 2y ago ↗ > Are RAG systems really so heavy they add 4x to the TTFT?Only the pure python + unoptimized orm ones. So only most of them.
[–] ComputerGuru 2y ago ↗ > Are RAG systems really so heavy they add 4x to the TTFT?Only the pure python + unoptimized orm ones. So only most of them.
4 comments
[ 0.22 ms ] story [ 29.4 ms ] threadhttps://arxiv.org/abs/2401.18059
>RAGCache reduces the time to first token (TTFT) by up to 4x and improves the throughput by up to 2.1x compared to vLLM integrated with Faiss.
Are RAG systems really so heavy they add 4x to the TTFT?
Only the pure python + unoptimized orm ones. So only most of them.