Happy to answer any questions about Brytlyt GPU Database and this benchmark.
Palvi Gulati Verma,
Director of Marketing,
Brytlyt
palvi.gulativerma@brytlyt.com
This is not a good benchmark. There are two problems:
1. It’s a simple GROUP BY on a single table. You’re basically just measuring the scan speed. Real queries are dominated by shuffles and the probe side of joins; these aren’t even present in this benchmark.
2. He runs the query repeatedly and takes the fastest time. This is far too cache-friendly. In this example, the intermediate stages of the query or even the result are probably just sitting in memory on the nodes after the first couple runs.
If you want to measure the performance of a data warehouse, you need to use more complex queries and not run the exact same query repeatedly.
edit: Coincidentally, I am giving a talk about data warehouse benchmarking TONIGHT in NYC. If you’re in NY and interested in this subject, please come! https://www.meetup.com/mysqlnyc/
The benchmarks are aimed at OLAP not OLTP workloads.
If I had a large query set to run on each vendor I'd be more likely to hit compatibility issues which could mean fewer benchmarks going out. As it is I spend a lot of time getting hardware and software vendors together for these benchmarks.
If caches were being hit I'd expect a lot more DBs hitting single millisecond times in my benchmarks but as far as I can see, there is a clear delta between the various setups I've tested:
This is true, but there are steps between the two. It would certainly be possible to, for each query, record both the first time and the fastest time. You could also add a couple queries that are a little harder- a query with a self-join and a window function would certainly be interesting- without getting too far down the road of building a query set that will be hard to run on different systems.
Even without those though, this is a good read, thanks for it!
My criticisms are specifically about OLAP use cases. Real OLAP queries have joins and subqueries.
I hear you about compatibility. I think this benchmark would be greatly improved by rewriting the query to include a subquery and a fact-to-fact join using the same table. For example, you could do a subquery where you find the longest and shortest ride for each driver each day, and then join that back to the full table and calculate something. That would do a much better job of exercising the key features of the query planner, while still being simple and compatible.
This was true 10 years ago. Many people building data warehouses today just use a normalized schema and write views with lots of joins. It’s a much simpler workflow and data warehouses have gotten so fast that this actually works.
I don't know any technology that makes the JOINs efficient enough for low latency queries. If you're joining two tables that have billions of rows, you need too much memory and CPU resources in order to be able to map them. Do you know any DBMS that focuses on OLAP systems and suggest normalization?
The other thing that is misleading about this benchmark is that the comparison page shows results for many different DBs on vastly different hardware without any mention of price. It would be much more valuable if it attempted to normalize performance results by hardware cost.
I've avoided going down this rabbit hole. Staff costs can outweigh software and hardware in some cases. Hiring someone that knows kdb+/q could cost a fair bit more than someone who only has some PostgreSQL experience.
Hi George, these are valid points. I can confirm we don't employ result caching in Brytlyt, if we did;
1) all run times would be identical regardless of query
2) the run times would be an order of magnitude faster
I agree there is quite a bit more to a data warehouse performance than table scan speed. There are queries where scan speed is important and in the retail industry, COUNT DISTINCT is one of them.
I agree the queries in this benchmark is simple, that doesn't mean they're not significant.
I hate to be the dismissive one, but I've seen a similar demo on a developer's laptop on CPU. On 32GB of RAM and 4/8 (core/thread) I have demos with billions of rows of data that I can report on similarly.
Ninja edit: I missed the data load times. I have a current customer loading a (much larger than my laptop demo) 1.5B row dataset with large dimensionality into a single node SSAS instance. They can process the whole model (dump and reload data) in 1.5 hours.
If I hear that 20 GPUs are necessary, I would expect multiple more orders of magnitude of data.
24 comments
[ 4.7 ms ] story [ 55.4 ms ] threadNice article anyway!
https://www.nextplatform.com/2016/09/08/refreshed-ibm-power-...
1. It’s a simple GROUP BY on a single table. You’re basically just measuring the scan speed. Real queries are dominated by shuffles and the probe side of joins; these aren’t even present in this benchmark.
2. He runs the query repeatedly and takes the fastest time. This is far too cache-friendly. In this example, the intermediate stages of the query or even the result are probably just sitting in memory on the nodes after the first couple runs.
If you want to measure the performance of a data warehouse, you need to use more complex queries and not run the exact same query repeatedly.
edit: Coincidentally, I am giving a talk about data warehouse benchmarking TONIGHT in NYC. If you’re in NY and interested in this subject, please come! https://www.meetup.com/mysqlnyc/
If I had a large query set to run on each vendor I'd be more likely to hit compatibility issues which could mean fewer benchmarks going out. As it is I spend a lot of time getting hardware and software vendors together for these benchmarks.
If caches were being hit I'd expect a lot more DBs hitting single millisecond times in my benchmarks but as far as I can see, there is a clear delta between the various setups I've tested:
http://tech.marksblogg.com/benchmarks.html
Even without those though, this is a good read, thanks for it!
I hear you about compatibility. I think this benchmark would be greatly improved by rewriting the query to include a subquery and a fact-to-fact join using the same table. For example, you could do a subquery where you find the longest and shortest ride for each driver each day, and then join that back to the full table and calculate something. That would do a much better job of exercising the key features of the query planner, while still being simple and compatible.
I also wonder how much of the speed up is due to the GPU parallelization vs the DB size seem to fit within the size of DDR4.
The core i5 system only has 16GB of RAM. I love to see what that number would looks like if it also has 512GB or more of DDR4 with NVME drive.Also one can also try increase to data size from 1.1 B -> 1000B and see how it scale on Minsky cluster.
http://tech.marksblogg.com/billion-nyc-taxi-vertica.html
I agree there is quite a bit more to a data warehouse performance than table scan speed. There are queries where scan speed is important and in the retail industry, COUNT DISTINCT is one of them.
I agree the queries in this benchmark is simple, that doesn't mean they're not significant.
Here's a demo of Azure Analysis Services looking at similar Taxi data. Demo starts around 11:00 https://azure.microsoft.com/en-us/blog/christian-wade-shows-...
Ninja edit: I missed the data load times. I have a current customer loading a (much larger than my laptop demo) 1.5B row dataset with large dimensionality into a single node SSAS instance. They can process the whole model (dump and reload data) in 1.5 hours.
If I hear that 20 GPUs are necessary, I would expect multiple more orders of magnitude of data.