Reminds me of the recent Terminal Bench controversy [1][2][3]
If theres a benchmark, people will cheat, lie and optimize for that benchmark. Honest depends on the compliance enforced on teams. But if, compliance itself is weak, it is going to be taken advantage of. Like growing up india, you would optimize for the exam and not what you learn from it.
The database wars of the late 1990s were full of this kind of stuff. Oracle, Sybase, IBM etc invested heavily in tuning specifically for benchmarks like TPC-C just so they could post ads in the Wall St Journal saying theirs was faster.
I do sympathize with OP, though, their objection to measuring cold-start queries is incomplete without also describing how often cold start needs to happen. If you restart once every five years then it doesnt matter as much if it takes 20 minutes to be warm. Every hour, that would be a real problem.
Anyone here using QuestDB in production? What is your use case? What is your experience?
We want to migrate away from InfluxDB eventually (because of their 180 on OSS, and their tendency to reinvent the product every major release), and QuestDB seems like an interesting option.
Been using it for half a year now in prod to collect sensor data from IoT devices.
My only complaints are:
1) Memory usage is a bit high. We went with the AWS instance they recommended in the docs and even that went over our provisioned memory. It's not much but I think it could be improved
2) You need to buy their enterprise plan if what you're storing is remotely sensitive like health data, PII, etc. Any row level security or credential features are locked behind that license. Our use case isn't that sensitive so we can get away with putting it in a VPN and password protecting it, but if you need DB-level security the FOSS license is severely behind Postgres in terms of features.
Other than that, it's never gone down, it's very, very fast and comes with it's own webui for querying your data. We migrated from AWS Timestream and couldn't be happier with the switch.
ClickBench is of very limited utility already because it doesn't have a single join in it. Which is maybe less weird in the context of ClickHouse not being great at joins.
9 comments
[ 2.5 ms ] story [ 32.0 ms ] threadIf theres a benchmark, people will cheat, lie and optimize for that benchmark. Honest depends on the compliance enforced on teams. But if, compliance itself is weak, it is going to be taken advantage of. Like growing up india, you would optimize for the exam and not what you learn from it.
[1] https://news.ycombinator.com/item?id=47920787
[2] https://www.tbench.ai/news/leaderboard-integrity-update
[3] https://debugml.github.io/cheating-agents/
I do sympathize with OP, though, their objection to measuring cold-start queries is incomplete without also describing how often cold start needs to happen. If you restart once every five years then it doesnt matter as much if it takes 20 minutes to be warm. Every hour, that would be a real problem.
We want to migrate away from InfluxDB eventually (because of their 180 on OSS, and their tendency to reinvent the product every major release), and QuestDB seems like an interesting option.
My only complaints are:
1) Memory usage is a bit high. We went with the AWS instance they recommended in the docs and even that went over our provisioned memory. It's not much but I think it could be improved
2) You need to buy their enterprise plan if what you're storing is remotely sensitive like health data, PII, etc. Any row level security or credential features are locked behind that license. Our use case isn't that sensitive so we can get away with putting it in a VPN and password protecting it, but if you need DB-level security the FOSS license is severely behind Postgres in terms of features.
Other than that, it's never gone down, it's very, very fast and comes with it's own webui for querying your data. We migrated from AWS Timestream and couldn't be happier with the switch.
https://www.youtube.com/watch?v=C7gJ_UxVnSk