They complement each other. Druid is "only" a real-time data storage for events and fast aggregation. This is a complex event processing engine, which would store its stuff in druid.
Could anyone comment on whether it would be possible to use a system like this for producing on-demand aggregations with filters over 100M rows with sub-second response time?
Another way may be voltdb. Since it's partitioned+single-thread/partition, and cross-partition queries are slow, you'll probably have to make a query for each partition and group results on the app.
I specialize in an in-memory analytics (business intelligence) platform called "QlikView" or "Qlik". It operates on top of RDBMSs/operational databases, so it batch processes data on a daily or hourly basis usually.
For the sake of my curiosity, I am assuming structured data with aggregations like: min, max, average, sorting by another field, count, count unique, etc. I do a lot of work with structured data and predefined aggregation functions, but dynamic filtering.
I would go to with elasticsearch, or if you reach terabytes/petabytes of data then look into Google BigQuery there you would get a constant 5-10 seconds response time for those high volume data. BigQuery is a WORM technology, so an append-only system might help you, especially in event-driven systems.
BigQuery is interesting. Thank you for the link. I am trying to determine if large quantities of data like this can be analyzed with sub-second aggregation time, targeting a 2-3 second total response time to the user, including network latency. The hypothetical data set would not necessarily be event driven, but of course it is useful to have lower data latency. The aggregation speed is the biggest performance concern for me.
The post says Pulsar can use Druid as a metrics store, so that workload should be doable. Druid is meant for exactly that sort of thing (fast aggregates with ad-hoc filters).
14 comments
[ 3.0 ms ] story [ 44.1 ms ] threadWhat is your insert/update/delete rate? If low, maybe look at postgresql in-memory column-store https://github.com/knizhnik/imcs/
Another way may be voltdb. Since it's partitioned+single-thread/partition, and cross-partition queries are slow, you'll probably have to make a query for each partition and group results on the app.
Also curious.