Sorry to hear, though I'd like to mention that there is a fair amount of scale you can get out of a single TimescaleDB node. We scale with disk space so you can throw more disks at it; we have production users who are in the hundreds of billions of rows and 10s of TB of data on a single node. Also with support for streaming replication you can spread out your read load and get HA support. But yes, we will have more to say in the coming months about clustering though.
are the upcoming clustering efforts developed with the intent to be mainlined in postgresql ?
Indeed some way to automatically failover/failback (ie automate pg_rewind) without all the keepalived/pacemaker/haproxy/patroni stuff is really needed.
How fast is it when it has a TB of data? I realize that this is affected by the machine it is running on, but just curious how long those sample queries in the docs would take to run.
I spent about 8 months writing data to TSDB. I was sending about 200 vehicle locations per 7-10 seconds for most of the day. I guess around 1 million events per day.
I played around with the data in the first week of setting it up, but then I ignored the feed for months.
When I came back on the 8th month, I couldn't get many/most queries to run in decent time. I didn't diagnose the issue, but I stopped the data stream and archived the database. My data was around 100GB, and I was reading it from a server-grade HDD. (I'll update my answer if I can find more accurate details when I get home).
I don't think it was a TSDB issue, might have been on my side. If I can restore my old data into a fresh DB, I'm going to try TSDB again with the data. I'm doing a Stats degree part time, so it'd be interesting to apply some forecasting knowledge to my dataset.
Can this be used side by side on normal Postgres cluster? As in could I have one DB for app data, and one for metrics data? Considering switching from MySQL (ndb cluster) to running a Postgres cluster and this could be a good motivator.
Yep, absolutely. Regular PostgreSQL tables coexist alongside TimescaleDB (hyper)tables in the same database. We believe that's actually a pretty big plus since you can keep metadata that you may need to join on your metrics data without doing it in an application layer.
Not sure I follow exactly what you're asking. You can do read replicas for HA/failover/read sharding which you can do with regular PostgreSQL databases as well. So at least on the axis TimescaleDB presents no limitations.
It does not support sharding writes across multiple nodes, but we do work with streaming replication so you can set up read replicas (and for failover).
19 comments
[ 3.4 ms ] story [ 66.3 ms ] threadexciting times!
are the upcoming clustering efforts developed with the intent to be mainlined in postgresql ?
Indeed some way to automatically failover/failback (ie automate pg_rewind) without all the keepalived/pacemaker/haproxy/patroni stuff is really needed.
* Influx: https://blog.timescale.com/timescaledb-vs-influxdb-for-time-...
* Cassandra: https://blog.timescale.com/time-series-data-cassandra-vs-tim...
* Mongo: https://blog.timescale.com/how-to-store-time-series-data-mon...
We also released a tool called Time Series Benchmark Suite (TSBS) here that someone just submitted a PR for Clickhouse: https://github.com/timescale/tsbs/pull/26
There is also this spreadsheet that compares a bunch of different time series databases, including TimescaleDB: https://docs.google.com/spreadsheets/d/1sMQe9oOKhMhIVw9WmuCE...
Hopefully some of that is useful :)
Edit: did a quick search and the discussion for the first link is here: https://news.ycombinator.com/item?id=17766566
I played around with the data in the first week of setting it up, but then I ignored the feed for months.
When I came back on the 8th month, I couldn't get many/most queries to run in decent time. I didn't diagnose the issue, but I stopped the data stream and archived the database. My data was around 100GB, and I was reading it from a server-grade HDD. (I'll update my answer if I can find more accurate details when I get home).
I don't think it was a TSDB issue, might have been on my side. If I can restore my old data into a fresh DB, I'm going to try TSDB again with the data. I'm doing a Stats degree part time, so it'd be interesting to apply some forecasting knowledge to my dataset.
Unfortunately it sometimes looses in storage cost-effectiveness comparing to competing TSDBs - https://medium.com/@valyala/when-size-matters-benchmarking-v...