I'm looking for this too. Advice welcome. AWS managed time series is not a thing.
Timestream has been in preview for too long. Does that mean they're having trouble with productizing it?
You can host anything in OP article in EC2 but you don't get the managed features. PLUS most of those (eg timescale and influxdb) are seriously priced for serious features like clustering/scaling/HA.
Plain Old Posgres is not awful for TS [1] but I'm also looking for something better.
At $work, we're considering Plain Old Elastic, which is a managed service. It scales well and if you don't have mountains of data, there shouldn't be surprises. If you do have mountains of data, and you can tolerate older stuff being slower in archive, you can throw it in S3 and query it with Athena, while keeping the faster stuff in ES or Dynamo.
Our users have told us that AWS Timestream is quite expensive, about 10x more expensive than TimescaleDB. It doesn't seem like it was designed with operational workloads in mind. A more detailed comparison is in the works. (TimescaleDB Co-founder)
> InfluxDB is a completely open-source time series database
Well, InfluxDB is not completely open-source. They have a free-tier, which is open source, but it is not clustered. So your data will not be high-available and you can't scale beyond a single node.
I have worked extensively with systems addressing the Australian National Energy Market, where data ticks either every five minutes (dispatch of generation) or thirty minutes (settlement). I’ve often wondered whether a proper TSDB would make my life easier when storing/querying meter and market data, but I always ended up with SQL.
My guess is that at 5min resolution (288 data points per day), you're probably just better off going stock SQL. My understanding is that time series DBs start getting useful at higher write loads that standard SQL DBs aren't necessarily optimized for, and where you might want things like data to coalesce into larger timeframes to save storage, etc. I'm not an expert though, just what I've seen from my somewhat limited experience.
(TimescaleDB founder) One of the advantages of TimescaleDB is the additional SQL functions that make time-series manipulation easier (eg interpolation, locf, aggregating by arbitrary time intervals). So if you need any of that functionality I'd recommend taking a look, even if you don't have massive scale.
Umm no. I'm not an expert of the Australian energy market, but do data analysis for a large energy market and we have petabytes of data and need to run queries against multiple years where a single day of the market generates many GB of data. Still SQL is usually my go-to tool, but we have a Time series database for other tasks that is helpful when just looking at a few data points.
If observations are spaced every five minutes and are high-precision (8 bytes each) that’s only 2KB per day of data, which can be greatly reduced for most real data by delta-encoding adjacent observations. If you have many GB per day then you must have millions of such series, if not then what you meant was you’ve wasted a tremendous amount of disk storing a negligible quantity of data in sparse btrees.
Neither. The original poster doesn't point out that they likely store a lot lot more kinds of data. I think your math probably makes sense off of just two data points, but the below should help you get a better idea for the size of the data. Keep in mind that all large grid operators and utilities are probably in the same boat. This holds for America to China.
A single generator has well over a hundred properties you would care about and although we're talking about 5 minute data, there are many many overlapping simulations being run around the clock and many of those simulations look into the future for many days. Then you have load data, topology data, equipment outages, load and wind forecast, thousands of pricing points that can be further decomposed into constituent prices, changing bids and offers, energy transactions...I could type for the next hour and only cover a small amount of what I know which is still a small percentage of what we actually store.
From a time-series perspective, a single generator can show well over a hundred line-charts and you can do that off of multiple study types...and you might have well over a thousand generators in your market, not to mention all the other logged objects such as transmission lines which also have lots of properties logged. You're right that time-series data can be compressed, but the SQL data is somewhat different I think.
In short, energy markets are hugely complex creatures. There is just so much going on that I have yet to get bored and have a non-busy day in nearly a decade of employment. It's an extremely exciting field for sure as there is just so much going on and so much change occurring. In the not so distant future, many experts think the lines between transmission and distribution will become blurred and we'll all need even more data to run the grid at peak efficiency and reliability.
I work at a power plant that uses Wonderware - a SCADA system that stores its time-series data in its own proprietary database that is accessed as a linked server via SQL Server. That means any query I want to fetch can't be optimized by the SQL Engine and makes any kind of analysis very expensive.
From my understanding, something like an integrated extension to the server (not a linked/foreign server) would be great because the query optimizer would be able to plan its queries based on that information.
Has anyone been in a situation like this and taken steps to mitigate the inefficiencies of a system like this?
This is exactly what Timescale does with PostgreSQL. Timescale is an extension to the database, and all the existing database query planning systems just work.
I am not aware if MS SQLServer has anything equivalent.
I’m not aware of any out-of-the-box support for smart partitioning by time (Timescale’s main feature). You could set that kind of thing up manually but it would be a fair bit of work.
You definitely don't want to use that columnstore for ingesting time series data. If you need real time reporting, you are better served with row-store.
Columnstore engines tend to be optimized for read, rather than write. I know for certain that Microsoft's columnstore technology is not a good fit for true real time applications.
I haven’t used the columnstore in production and can’t vouch for it either. It makes sense that k-ordered write performance on column-oriented data would suffer, do you know of a good benchmark measuring this?
Also I don’t like the phrase “true real time applications” at all. It is less useful than even Microsoft’s OLAP-Cube-ETL-OLTP word salad, and embedded developers in particular are going to be cringing if they read that.
Fair. I use it as a shorthand for reporting scenarios with required data latency measured less than a second from transaction to reporting tier. Yes, there's an entire world of performance below 1 second, but it's a useful threshold.
When I've been benchmarking SQL Server for such use cases, we've always found its columnar indices to be too much overhead.
You can build a columnstore index in two ways in SQL Server. The easiest is as a non-clustered index on a regular rowstore table. Your inserts are going into the rowstore directly. This physically stores the rows as a rowstore with a columnstore index on top.
The second is as a clustered columnstore index. This stores the rows directly as a columnstore. That's the way I'd store time series data. In this case, all the inserts go into a "delta rowstore" and are moved by a background service into the clustered columnstore structure. Your inserts proceed as fast as any insert into a rowstore. There is overhead to move the data since it's doing a fair bit of compression. But that's going to be true of anything that does compression.
If you want really fast ingest, you can use an InMemory OLTP (aka Hekaton) structure and layer a column store index on top of that. And that InMemory structure can be persisted to disk. That gives you much better performance than a plain rowstore. You'd probably want to write something to move the data to a longer term structure but that would be the fastest way to ingest it I can think of. Using SQL Server :)
It's a good overview you've shared of the storage technologies in SQL Server.
I agree on memory-optimized being ideal for a real time service. The overhead of column indexing has ruled it out whenever I've benchmarked it for real time use cases. Last time was on SQL 2016, so latest optimizations may have brought it to par.
Microsoft's column store implementation uses a large delta rowgroups and functions a little like a log structured merge tree. Incoming inserts go into deltas, and are later aggregated into updates to columnar blocks.
This gives insert performance on par with row storage (it is using row storage!). This can be combined with their in memory database functionality as well.
Druid.io didn't make the list, but it's worked really well for me in the past. A bit of a chore to get running though. Great for real time aggregation and and is fast even with custom JavaScript filter functions. Also supports limited SQL, mostly not able to do joins.
Would anyone mind sharing what time series DB they use in production, and what for? I'm assuming most are used for metrics in addition to a standard SQL database, but interested to find out if that's accurate. Thanks!
The killer for us is that we allow essentially ad-hoc querying over long time intervals, but require the results to be returned quickly. And the dataset, while not Google proportions, isn't small.
Our customers successfully use VictoriaMetrics [1] in production as a long-term remote storage for big amounts of time series data from Prometheus. We are going to open source VictoriaMetrics soon.
I have used TimescaleDB for several purposes. As it is built on top of Postgres, all the existing tools, libraries and processes work out of the box. This is a huge advantage if you are operating Postgres anyway: Your existing backup tools will work, as does your user Managment.
Scaling out is a more complex story: Read Scaleout works seamless using read replicas (another existing Postgres mechanism). Replicas can also help you for high availability, Write scaleout is difficult with TimescaleDB (same for Postgres): Sharding can be an option, or buying bigger machines.
Being a small team, we try to have managed hosting as much as possible. One less thing to think/worry about.
Then our stack is primarily in AWS. So when it comes to DBs, I'd prefer to keep it all within our VPC for security / latency reasons - for sensitive data at least.
From what you're writing it seems TimescaleDB doesn't really solve large timeseries problems. So it looks like more a good transitory solution than a definitive solution.
Hi! Engineer at Timescale who’s just been working on this feature, it’s coming in our next release and going to continue to be developed over the next few releases! Would love to have you try it out and get your feedback :)
You can get 6-8x compression by running on TimescaleDB on ZFS, but compression in general is also an area where we are investing a fair amount of R&D.
Also worth noting that TimescaleDB is far more memory-efficient than other time-series databases (eg InfluxDB [1]), and for some folks the memory cost savings is significant (b/c memory cost $$$ >> disk cost $).
We ingest a lot of time series (IoT) data and use Postgres for other data so Timescale works quite well for us.
One thing Timescale treats as second class citizen though is updates to existing data points which are 1-2 areas of magnitude slower than inserts. Granted this is also the case for all other TSDB solutions out there which are for obvious reasons optimized for inserts and reads aggregated along the time dimension.
Still would be amazing if you could add to the already existing differentiation of allowing fast updates for cases like ours where we dont store events relating to a singular point in time but rather time-spans so new incoming data points might be "merged" into existing time-spans.
Same domain. I've come to the conclusion that these systems are dualistic in nature and the architecture needs to reflect that. I think the more general expression of this would be a time-series -> document-store pipeline. The concern then, if the time-series processing step requires domain data, is serving up fresh views of the evolving state in toto. That epsilon of temporal lag of 'current state' in the premier stage is a product issue. For some use cases it can be a very tight constraint, but I think quite a lot of others will be perfectly fine with snapshot isolation with generous requirement on freshness.
We're planning to deploy ClickHouse from Yandex[0]. Would like to hear from anyone who has it in production already, and what is your experience with it.
I've evaluated Timescale, Clickhouse, Druid, and Pinot for our own use case.
Druid and Pinot have a lot of moving parts. If I remember correctly, Druid was something like 6-8 different nodes for different parts of the ingestion / querying processes. So it's going to be a lot of upfront and ongoing dev ops work.
Clickhouse is interesting because it seems to just "work". One thing to deploy and you just increase the number of nodes as you scale.
In our case, the simple deployment and management of Clickhouse is a key feature. It is masterless [1] with no namenodes or coordinators, so each machine looks the same, and there is only one process to manage.
If you rely on its replication mechanism for sharding, Zookeeper becomes necessary, but writing directly to nodes in an orderly fashion is also an option (as we are).
[1] This means nodes are not aware of what each other contain, so queries hit all nodes with some maybe having no work to do. Depending on your workload this may or may not be a concern.
This is interesting. In effect, you're doing your sharding client-side?
What do you plan to do if you need to add new nodes and/or you need to rebalance nodes due to concentrated data access patterns? How do you handle cross-node queries like joins?
> In effect, you're doing your sharding client-side?
Correct. Some ETL inserts to specific nodes when sharding is necessary, in other cases Kafka engine tables on a group of nodes subscribe to common topics, and we simply let the whole cluster participate in queries. This works just fine when table scans are acceptable.
Rebalancing is a missing option here, short of moving partitions manually. But in my specific use cases, I have not yet needed to rebalance across nodes.
Note using native Clickhouse replication is still an option if we need it. One cost to it is the extra work needed in the database cluster, so addressing it in an eariler layer works for us.
> How do you handle cross-node queries like joins?
If I understand your question, since we are using the Distributed view type across the cluster definition, a query on any node will receive data from the others as part of a join-less SELECT, and federate on the node with the client connection.
We are not doing any database-side JOINs currently. Plans are to augment data in ETL, or join data post-query (potentially Spark). Clickhouse dictionaries handle simple cases.
There is a chproxy [1] - a proxy that is able to balance inserts and selects across ClickHouse nodes / replicas. Then client applications shouldn't know anything about ClickHouse cluster topology - they just talk to chproxy.
I'll second Simplicity and add in that ClickHouse is also faster on less hardware and more stable compared to our current Druid configuration.
My company currently uses Druid and has for a few years now, but I have been evaluating ClickHouse on the side as a possible replacement, and as a testament to its simplicity, I was able to stand it up and get in going as a PoC pretty quick/easily.
So far I have only found good things about ClickHouse, maybe the only ClickHouse downside has been the management of the cluster and data, but I haven't gotten too far into operationalizing ClickHouse to know how much those kind of items will cost.
Perhaps the other thing is the documentation, while reasonably good imo, still doesn't explain everything as well as I'd like. It was good enough, but I definitely had to experiment on a couple items to get them working as needed.
I would love to use Clickhouse, if only it has some kind of Cassandra-esque node-to-node replication mechanism. I'm loathe to bring in something heavy like Zookeeper for that.
It would also be ideal if they could accelerate the data ingestion process somehow without the need to buffer chunks with yet another moving piece like Kafka.
I guess ideally, I'm looking for some kind of standalone HTAP system minus transactional guarantees.
Clickhouse cluster can run without Zookeeper and without replication. Data durability may be provided by using durable storage such as Google Compute Engine disks [1].
Data ingestion process may be accelerated without resorting to Kafka - just insert data into Buffer table [2].
Not quite timeseries related but what are some good query processing solutions that let you make uni-temporal queries without too much application code? For example: assuming a simple application that employs event-stream based storage.
* New users filling up their profile - {'id': ,'name': ,'city': , ...}
* Older users making updates to their profiles or even adding new fields - {'id': ,'city': }, {'id': , 'new_field': }
Which database solutions make it easy to write queries like - "give me the profile information for id: X as of May 3rd 2018"? A document DB like Mongo definitely supports this but joining data across different streams doesn't seem quite straightforward as joining SQL tables.
Usually, time series databases use a variety of tricks to store data using as fewer bits per sample as possible. An OLAP database sounds like a much more generic system that will cover a larger number of use cases, but won't be finely tuned resource-wise for this one aspect.
Can I read about how the time series nature of the data allows the storage engines to optimize different parts of the system compared to generic databases? Some links? Papers? Interested to know.
I’m really interested how these compare to the Splunk metric store. Which I’ve had very positive experiences with. Anyone with experience in that area?
Have used RRDtool in the distant past. It's a fascinating project from a time when things were designed with hardware limitations in mind. https://oss.oetiker.ch/rrdtool/
RRDtool stores its data values in a circular buffer (or many buffers, at different resolutions) so performance is constant - it doesn't degrade over time. Also resource usage is constant. The price you pay is that older data is stored at coarser resolution. I assume it also made the data points fixed in time for performance reasons, which is fine in my opinion.
I don't know if there are modern integrations or reporting tools built on the database.
As the product manager for Crux I must point out that we are not currently optimised for time series queries, as there is no columnar compression in the core bitemporal indexes. It is certainly something we have thought about however. For instance, we have already translated a couple of TimescaleDB examples into our own test suite [0][1] and we have created a sample aggregation "decorator" that sits on top of the Datalog queries [2].
Whilst there are no immediate plans from the core team to add columnar compression or otherwise improve our support for time series use cases, I think that a lot can be readily achieved by building on top of what already exists in the core. I very much look forward to seeing others experiment with the possibilities here.
Interesting that nobody mentions GPU powered DBs that can actually compete with KDB+. There are also other "normal" DBs that have impressive benchmarks like VictoriaMetrics and Dremio.
94 comments
[ 2.7 ms ] story [ 149 ms ] threadThank you!
Timestream has been in preview for too long. Does that mean they're having trouble with productizing it?
You can host anything in OP article in EC2 but you don't get the managed features. PLUS most of those (eg timescale and influxdb) are seriously priced for serious features like clustering/scaling/HA.
Plain Old Posgres is not awful for TS [1] but I'm also looking for something better.
At $work, we're considering Plain Old Elastic, which is a managed service. It scales well and if you don't have mountains of data, there shouldn't be surprises. If you do have mountains of data, and you can tolerate older stuff being slower in archive, you can throw it in S3 and query it with Athena, while keeping the faster stuff in ES or Dynamo.
https://grisha.org/blog/2015/09/23/storing-time-series-in-po...
I've been on the timestream preview list for months...
Well, InfluxDB is not completely open-source. They have a free-tier, which is open source, but it is not clustered. So your data will not be high-available and you can't scale beyond a single node.
Thanks for the clarification.
An example with Influx is available here : https://www.terminalbytes.com/temperature-using-raspberry-pi...
Literally any database or disk hierarchy.
I'd probably use SQLite, or if I had any relational database up already, I'd use that.
seriously... simple n easy.
CSV and rsync could do it.
A single generator has well over a hundred properties you would care about and although we're talking about 5 minute data, there are many many overlapping simulations being run around the clock and many of those simulations look into the future for many days. Then you have load data, topology data, equipment outages, load and wind forecast, thousands of pricing points that can be further decomposed into constituent prices, changing bids and offers, energy transactions...I could type for the next hour and only cover a small amount of what I know which is still a small percentage of what we actually store.
From a time-series perspective, a single generator can show well over a hundred line-charts and you can do that off of multiple study types...and you might have well over a thousand generators in your market, not to mention all the other logged objects such as transmission lines which also have lots of properties logged. You're right that time-series data can be compressed, but the SQL data is somewhat different I think.
In short, energy markets are hugely complex creatures. There is just so much going on that I have yet to get bored and have a non-busy day in nearly a decade of employment. It's an extremely exciting field for sure as there is just so much going on and so much change occurring. In the not so distant future, many experts think the lines between transmission and distribution will become blurred and we'll all need even more data to run the grid at peak efficiency and reliability.
From my understanding, something like an integrated extension to the server (not a linked/foreign server) would be great because the query optimizer would be able to plan its queries based on that information.
Has anyone been in a situation like this and taken steps to mitigate the inefficiencies of a system like this?
I am not aware if MS SQLServer has anything equivalent.
I’m not aware of any out-of-the-box support for smart partitioning by time (Timescale’s main feature). You could set that kind of thing up manually but it would be a fair bit of work.
Columnstore engines tend to be optimized for read, rather than write. I know for certain that Microsoft's columnstore technology is not a good fit for true real time applications.
Also I don’t like the phrase “true real time applications” at all. It is less useful than even Microsoft’s OLAP-Cube-ETL-OLTP word salad, and embedded developers in particular are going to be cringing if they read that.
When I've been benchmarking SQL Server for such use cases, we've always found its columnar indices to be too much overhead.
The second is as a clustered columnstore index. This stores the rows directly as a columnstore. That's the way I'd store time series data. In this case, all the inserts go into a "delta rowstore" and are moved by a background service into the clustered columnstore structure. Your inserts proceed as fast as any insert into a rowstore. There is overhead to move the data since it's doing a fair bit of compression. But that's going to be true of anything that does compression.
If you want really fast ingest, you can use an InMemory OLTP (aka Hekaton) structure and layer a column store index on top of that. And that InMemory structure can be persisted to disk. That gives you much better performance than a plain rowstore. You'd probably want to write something to move the data to a longer term structure but that would be the fastest way to ingest it I can think of. Using SQL Server :)
I agree on memory-optimized being ideal for a real time service. The overhead of column indexing has ruled it out whenever I've benchmarked it for real time use cases. Last time was on SQL 2016, so latest optimizations may have brought it to par.
This gives insert performance on par with row storage (it is using row storage!). This can be combined with their in memory database functionality as well.
The killer for us is that we allow essentially ad-hoc querying over long time intervals, but require the results to be returned quickly. And the dataset, while not Google proportions, isn't small.
So, there's two use cases for us:
1. We use it to ingest tick-by-tick FX rates
2. We use it to query historical rates e.g. whenever a users opens our homepage we query it directly from TimescaleDB
We've battled tested it pretty hard now & haven't run into any scaling issues - been very much plug & play.
[1] https://github.com/VictoriaMetrics/VictoriaMetrics/wiki/Sing...
Scaling out is a more complex story: Read Scaleout works seamless using read replicas (another existing Postgres mechanism). Replicas can also help you for high availability, Write scaleout is difficult with TimescaleDB (same for Postgres): Sharding can be an option, or buying bigger machines.
The theory is they are not supporting it in favor of promoting their new time series DB.
Then our stack is primarily in AWS. So when it comes to DBs, I'd prefer to keep it all within our VPC for security / latency reasons - for sensitive data at least.
Is that correct?
It doesn't pre-aggregate your data so if you're trying to get aggregates for a large data set, it's not fast.
We also have more in the works for large datasets: eg scaling out storage across multiple machines as well as data tiering for lower storage costs.
Feel free to email me if you'd like more info on either: ajay (at) timescale (dot com)
How is TimescaleDB doing with compression by the way? Meaning if you compare your raw data and the disk usage, what's the ratio?
Also worth noting that TimescaleDB is far more memory-efficient than other time-series databases (eg InfluxDB [1]), and for some folks the memory cost savings is significant (b/c memory cost $$$ >> disk cost $).
[1] https://blog.timescale.com/timescaledb-vs-influxdb-for-time-...
Write scale-out is something we have been working on for nearly a year now.
Here are some preliminary benchmarks we shared at Postgres Conf earlier this year (benchmarks have continued to improve since then): https://twitter.com/acoustik/status/1108813512321708032
We'll have more to share regarding write scale-out later this month :)
[0] https://clickhouse.yandex/
[0]https://blog.cloudflare.com/http-analytics-for-6m-requests-p...
Druid and Pinot have a lot of moving parts. If I remember correctly, Druid was something like 6-8 different nodes for different parts of the ingestion / querying processes. So it's going to be a lot of upfront and ongoing dev ops work.
Clickhouse is interesting because it seems to just "work". One thing to deploy and you just increase the number of nodes as you scale.
http://druid.io/docs/latest/design/processes.html
In our case, the simple deployment and management of Clickhouse is a key feature. It is masterless [1] with no namenodes or coordinators, so each machine looks the same, and there is only one process to manage.
If you rely on its replication mechanism for sharding, Zookeeper becomes necessary, but writing directly to nodes in an orderly fashion is also an option (as we are).
[1] This means nodes are not aware of what each other contain, so queries hit all nodes with some maybe having no work to do. Depending on your workload this may or may not be a concern.
What do you plan to do if you need to add new nodes and/or you need to rebalance nodes due to concentrated data access patterns? How do you handle cross-node queries like joins?
Correct. Some ETL inserts to specific nodes when sharding is necessary, in other cases Kafka engine tables on a group of nodes subscribe to common topics, and we simply let the whole cluster participate in queries. This works just fine when table scans are acceptable.
Rebalancing is a missing option here, short of moving partitions manually. But in my specific use cases, I have not yet needed to rebalance across nodes.
Note using native Clickhouse replication is still an option if we need it. One cost to it is the extra work needed in the database cluster, so addressing it in an eariler layer works for us.
> How do you handle cross-node queries like joins?
If I understand your question, since we are using the Distributed view type across the cluster definition, a query on any node will receive data from the others as part of a join-less SELECT, and federate on the node with the client connection.
We are not doing any database-side JOINs currently. Plans are to augment data in ETL, or join data post-query (potentially Spark). Clickhouse dictionaries handle simple cases.
[1] https://github.com/Vertamedia/chproxy
My company currently uses Druid and has for a few years now, but I have been evaluating ClickHouse on the side as a possible replacement, and as a testament to its simplicity, I was able to stand it up and get in going as a PoC pretty quick/easily.
So far I have only found good things about ClickHouse, maybe the only ClickHouse downside has been the management of the cluster and data, but I haven't gotten too far into operationalizing ClickHouse to know how much those kind of items will cost.
Perhaps the other thing is the documentation, while reasonably good imo, still doesn't explain everything as well as I'd like. It was good enough, but I definitely had to experiment on a couple items to get them working as needed.
It would also be ideal if they could accelerate the data ingestion process somehow without the need to buffer chunks with yet another moving piece like Kafka.
I guess ideally, I'm looking for some kind of standalone HTAP system minus transactional guarantees.
Data ingestion process may be accelerated without resorting to Kafka - just insert data into Buffer table [2].
[1] https://cloud.google.com/compute/docs/disks/#pdspecs
[2] https://clickhouse.yandex/docs/en/operations/table_engines/b...
Disclaimer: I'm an organizer.
https://db.cs.cmu.edu/seminar2017/
6 tsdb vendors talk about their DB.
* New users filling up their profile - {'id': ,'name': ,'city': , ...}
* Older users making updates to their profiles or even adding new fields - {'id': ,'city': }, {'id': , 'new_field': }
Which database solutions make it easy to write queries like - "give me the profile information for id: X as of May 3rd 2018"? A document DB like Mongo definitely supports this but joining data across different streams doesn't seem quite straightforward as joining SQL tables.
Thanks!
http://www.vldb.org/pvldb/vol8/p1816-teller.pdf
https://azure.microsoft.com/en-us/services/data-explorer/
We're migrating from a 50-node Elasticsearch to ADX, imho: - amazing query language (KQL) - less work to maintain cluster - lower cost
(it appears to be similar to Clickhouse, but more feature rich)
RRDtool stores its data values in a circular buffer (or many buffers, at different resolutions) so performance is constant - it doesn't degrade over time. Also resource usage is constant. The price you pay is that older data is stored at coarser resolution. I assume it also made the data points fixed in time for performance reasons, which is fine in my opinion.
I don't know if there are modern integrations or reporting tools built on the database.
Prometheus with the Cortex backend gives you distributed Timeseries storage with Prometheus.
Whilst there are no immediate plans from the core team to add columnar compression or otherwise improve our support for time series use cases, I think that a lot can be readily achieved by building on top of what already exists in the core. I very much look forward to seeing others experiment with the possibilities here.
[0] https://github.com/juxt/crux/blob/master/test/crux/ts_device...
[1] https://github.com/juxt/crux/blob/master/test/crux/ts_weathe...
[2] https://github.com/juxt/crux/blob/master/test/crux/decorator...
times series databases would be actual datasets, I was looking forward to read about some very interesting time series datasets...