Show HN: Arc – high-throughput time-series warehouse with DuckDB analytics (github.com)
Over the past months I’ve been building Arc, a time-series data platform designed to combine very fast ingestion with strong analytical queries.
What Arc does? Ingest via a binary MessagePack API (fast path), Compatible with Line Protocol for existing tools (Like InfluxDB, I'm ex Influxer), Store data as Parquet with hourly partitions, Query via DuckDB engine using SQL
Why I built it:
Many systems force you to trade retention, throughput, or complexity. I wanted something where ingestion performance doesn’t kill your analytics.
Performance & benchmarks that I have so far.
Write throughput: ~1.88M records/sec (MessagePack, untuned) in my M3 Pro Max (14 cores, 36gb RAM) ClickBench on AWS c6a.4xlarge: 35.18 s cold, ~0.81 s hot (43/43 queries succeeded) In those runs, caching was disabled to match benchmark rules; enabling cache in production gives ~20% faster repeated queries
I’ve open-sourced the Arc repo so you can dive into implementation, benchmarks, and code. Would love your thoughts, critiques, and use-case ideas.
Thanks!
13 comments
[ 3.1 ms ] story [ 37.8 ms ] threadI am afraid “Arc” became too fashionable this decade and using it might decrease brand visibility
[1] https://en.wikipedia.org/wiki/Arc_(programming_language)
Noticing that all the benchmarking is being done with MinIO which I presume is also running alongside/locally so there is no latency and it will be roughly as fast as whatever underlying disk its operating from.
Are there any benchmarks for using actual S3 as the storage layer?
How does Arc decide what to keep hot and local? TTL based? Frequency of access based?
We're going to be evaluating Clickhouse with this sort of hot (local), cold (S3) configuration soon (https://clickhouse.com/docs/guides/separation-storage-comput...) but would like to evaluate other platforms if they are relevant.
- is the schema inferred from the data? - can/does the schema evolve? - are custom partitions supported? - is there a roadmap for future features?
Schema inference: yes, Arc infers the schema automatically from incoming data (both for MessagePack and Line Protocol). Each measurement becomes a table, and fields/tags map to columns.
Schema evolution: supported. New fields can appear at any time, they’re added to the Parquet schema automatically without migration or downtime.
Custom partitions: currently partitioning is time-based (hour-level by default), but custom partitioning by tag or host or whatever is planned. The idea is to allow you to group by any tag (e.g. device, region) in the storage path for large-scale IoT data.
Roadmap: absolutely. Grafana data source, Prometheus remote write, retention policies, gRPC streaming, and distributed query execution are all in the works.
We are going to start to blogging about it, so, stay tune.
Would love any feedback on what you’d prioritize or what would make adoption easier for your use case.
Is this something I’d use instead of timescale, or, am I understanding that the intention here is to be a data warehouse, where we could potentially offload older data to Arc for longer term storage or trend analysis?
I’d say both roles are possible, though the original intent of Arc was indeed to act as an offload / long-term store for systems like TimescaleDB, InfluxDB, Kafka, etc. The idea: you send data into Arc to reduce storage and query load on your primary database for ML, deep analysis, etc.
But as we built it, we discovered that Arc is really good not just at storage but at actively answering queries, so it’s kind of hybrid: somewhat “warehouse-like,” but still retaining database qualities in performance. I feel that saying a database its too much, but we are going on that direction.
IoT is absolutely one of the core use cases. You’re often ingesting tens or hundreds of thousands of events per second from edge devices, and you need a system that doesn’t choke. Our binary MessagePack ingestion helps shrink the payload size and reduce parsing overhead, that allows higher throughput for writes, which is crucial in IoT scenarios.
Let me know if you want to explore this a little more, not for selling you anything, at least not yet, I would love to understand your use case. Let me know if you are open: ignacio[at]basekick[dot]net
this doesn't sound like much, unless records are very large..
The benchmark measures fully written time-series records, not bytes. Each record typically includes 1–4 fields, tags, and timestamps, similar to InfluxDB’s Line Protocol structure.
For comparison, the same hardware (AWS c6a.4xlarge) handles around 240K RPS using Line Protocol, while Arc reaches 1.88M RPS with MessagePack, about 7.8× faster on ingestion throughput.
You can see the full ClickBench and ingestion benchmarks are in the repo.
TL;DR: Arc’s strength isn’t massive single records, it’s sustained high-throughput ingestion of structured time-series data while still staying analytical-query friendly.