Our architecture is built around columnar design, and we use Apache Arrow for in-memory columnar processing and Apache Parquet for durable columnar storage on S3-compatible object storage. In Parseable, every labels stay as columns in the data instead of becoming a large long-lived per-series index like many TSDBs.
This looks super interesting. Question about the scale, I thought Thanos and some other Prometheus variants can handle about 100 million active time series. I would have expected your solution to scale to billions. Have you not pushed it past 100 million or am I maybe missing something.
I currently manage 50m active series and it’s brain dead easy using Mimir (which can also easily do 100-200m).
That being said, perhaps this Parseable project is much more efficient (RAM) in storing that 1b samples compared to Mimir (RAM). Once we add in object stores which cheapen the storage cost by orders of magnitude (by going from memory to disk), the comparison is even weaker. Mimir and Cortex/Thanos are quite happy to pull cold data from S3.
So I too expected to see “billions” as well.. hmm.
How does the 100M active series deployment looks like? How many ingestors are there? What's each instance size? How big is the querier so that it can query across a metric with millions of active series?
Your pricing page calculator is a bit strange. The minimum daily is set to 1 TB which is too high. Are you not interested in working with companies that have lesser ingestion ?
Oh I see, what you say! ...but that’s intentional... we start the calculator at 1 TB/day because beyond this scale it's usually where observability pricing starts to hurt and ROI discussions kicks in. So, the point we’re trying to show is that, even at 1 TB/day, Parseable will still cost you lesser than others, and easier to operate too. Also the best part is that you will have the complete ownership of your data.
Low level how does this compare to Victoriametrics, you can get the high cardinality with clickhouse and other columnar dbs but the tradeoff the time there is more ram usage, slower queries, io, etc
the tradeoff with regards to slower queries and io only comes into play if the TSDB is performing a narrow lookup on a handful of series. In that case it’ll be faster. But when you scale up to 100s of millions, columnar dbs like Parseable win because-
a) there's no per-series inverted index and labels are parquet columns so memory is not bounded by cardinality
b) data lives on much cheaper object storage (parseable gives an option to cache data locally to remove io bound latency)
c) columnar store helps with faster data scanning by aggressively pruning and filtering data out
Most time series databases are columnar in some way. It is just because sotring and compressing timestamps and values as separate columns is way more efficient. So it is not a trait on its own, unless you compare yourself with Postgres.
> a) there's no per-series inverted index and labels are parquet columns so memory is not bounded by cardinality
Lack of inverted index is trade-off, in the same way as having that index. Default per-label inverted index in Prometheus, Mimir, VictoriaMetrics is convenient for users - requires no configuration from user side, all read queries, despite which time series selector was used, are as fast as they could be. Although, VictoriaMetrics doesn't store its whole index in-memory. It only stores what's "hot" now.
If you don't index all the labels, then you either asking user to choose what matters most, or intelligently decides what should be indexed. Either of these choice can take you to a corner case when query speed will be negatively affected despite having enough resources to burn.
> c) columnar store helps with faster data scanning by aggressively pruning and filtering data out
Columnar layout is not enough. Faster reads are achieved by reducing random seeks. A sequential read of a big chunk of data will always outperform multiple reads that pull the same data but from many places.
The same data can be stored in a columnar database with and without preserving data-locality (a trait that helps doing sequential reads). Let's talk time series. If in columnar database I'd decide to sort the columns by some common label and not by "timestamps" - data locality will be lost for typical queries that request "last 30min" of data.
I’m curious too. We need to see 1b+ numbers or else it just seems like we’re leveraging compression here, which is great but it’s hard to understand why it’s novel and why it’s the best solution no contest.
I can archive my data just fine by using an access based storage tiering like S3 - have your cake and eat it too. Gets around the headache of actually querying all of the 1B+ records because realistically there’s very few use cases for that. If you ever really need it, spin up Kafka and run through the entire archive.
Parquet is also 13 years old and not the GOAT at compression either, there's been a lot of developments in research since that like Fastlanes and Btrblocks which Vortex implements to put it at the Pareto frontier
Iceberg is a table format. It adds snapshots and a catalog layer on top of Parquet files. Parseable is a full-stack observability tool. Its storage layer could theoretically use Iceberg as the table format on top of Parquet file format. Iceberg is not a direct alternative to Parseable.
23 comments
[ 0.19 ms ] story [ 30.8 ms ] threadThis is Yash, founding team at Parseable (https://github.com/parseablehq).
We've built an open source observability data lake using Rust, that handles high-cardinality data at around 100M active time series in production (https://www.parseable.com/blog/how-parseable-handles-100-mil...)
Our architecture is built around columnar design, and we use Apache Arrow for in-memory columnar processing and Apache Parquet for durable columnar storage on S3-compatible object storage. In Parseable, every labels stay as columns in the data instead of becoming a large long-lived per-series index like many TSDBs.
That being said, perhaps this Parseable project is much more efficient (RAM) in storing that 1b samples compared to Mimir (RAM). Once we add in object stores which cheapen the storage cost by orders of magnitude (by going from memory to disk), the comparison is even weaker. Mimir and Cortex/Thanos are quite happy to pull cold data from S3.
So I too expected to see “billions” as well.. hmm.
5x Queriers, each with 64 vcpu 192 GB
The current utilization sits comfortably at 10-15 vcpu and 20-30 GB memory for the ingestors 20-40 vcpu and 40-60 GB memory for the queriers
Ample of headroom for transient spikes and planned near-future growth
Azure doesn't exist.
a) there's no per-series inverted index and labels are parquet columns so memory is not bounded by cardinality
b) data lives on much cheaper object storage (parseable gives an option to cache data locally to remove io bound latency)
c) columnar store helps with faster data scanning by aggressively pruning and filtering data out
> a) there's no per-series inverted index and labels are parquet columns so memory is not bounded by cardinality
Lack of inverted index is trade-off, in the same way as having that index. Default per-label inverted index in Prometheus, Mimir, VictoriaMetrics is convenient for users - requires no configuration from user side, all read queries, despite which time series selector was used, are as fast as they could be. Although, VictoriaMetrics doesn't store its whole index in-memory. It only stores what's "hot" now.
If you don't index all the labels, then you either asking user to choose what matters most, or intelligently decides what should be indexed. Either of these choice can take you to a corner case when query speed will be negatively affected despite having enough resources to burn.
> c) columnar store helps with faster data scanning by aggressively pruning and filtering data out
Columnar layout is not enough. Faster reads are achieved by reducing random seeks. A sequential read of a big chunk of data will always outperform multiple reads that pull the same data but from many places.
The same data can be stored in a columnar database with and without preserving data-locality (a trait that helps doing sequential reads). Let's talk time series. If in columnar database I'd decide to sort the columns by some common label and not by "timestamps" - data locality will be lost for typical queries that request "last 30min" of data.
What could I do with this that I couldn't achieve with iceberg-rs + DataFusion + parquet/vortex
They repeatedly talk about "80% size reduction with compression. Isn't that essentially just the default parquet compression ratio?
What exactly is unique here
I found this out because I set Codex the task of running this locally agains another of my apps and it worked around the limitation by running this proxy: https://gist.github.com/simonw/b0e61a0aa8e3f7d30f27ce2f747c9...