No worries at all. For the teams which are happy on CSV/JSON, i'd admit that lance is not ideal alternative for them. > It seems you build a few indexes, such that I'm guessing those are the main drivers on the…
That's a good idea. The format itself is self-contained. We need a little bit work to expose our file-level read/write API to public.
The use cases for such random access was from our experiences to maintain large-scale training data, with the needs to debugging model performance against to data, which requires fast slicing and dicing over the…
We have not done benchmarks against deeplake yet. Deeplake has some interesting concepts in their design, I'd be very interested to do a benchmark soon.
Hey, co-author of Lance here. Lance is faster in random access because the layout / encodings were designed to be fast in both scan and random access case. We borrowed many ideas from Google's Procella paper, and…
No worries at all. For the teams which are happy on CSV/JSON, i'd admit that lance is not ideal alternative for them. > It seems you build a few indexes, such that I'm guessing those are the main drivers on the…
That's a good idea. The format itself is self-contained. We need a little bit work to expose our file-level read/write API to public.
The use cases for such random access was from our experiences to maintain large-scale training data, with the needs to debugging model performance against to data, which requires fast slicing and dicing over the…
We have not done benchmarks against deeplake yet. Deeplake has some interesting concepts in their design, I'd be very interested to do a benchmark soon.
Hey, co-author of Lance here. Lance is faster in random access because the layout / encodings were designed to be fast in both scan and random access case. We borrowed many ideas from Google's Procella paper, and…