We benchmarked two ways to improve the latency in RAG workflows with a multi-vector setup. Hybrid search using Postgres native capabilities and a relatively new method of token pooling. Token pooling unlocked up to 70% faster latency with <1% performance cost.
1 comment
[ 3.4 ms ] story [ 9.8 ms ] threadWe benchmarked two ways to improve the latency in RAG workflows with a multi-vector setup. Hybrid search using Postgres native capabilities and a relatively new method of token pooling. Token pooling unlocked up to 70% faster latency with <1% performance cost.