Ah, thanks! I have unfortunately been too busy to watch the streams this week, and assumed it was Connect because I think Confluent is/was doing a talk with Kafka Connect and Spark at the summit.
Code generation is enabled by SPARK-8641, but not sure exactly what it entails. I think it is related to some of the RDD transformation/action merging they do to optimize runtime operations in 2.0.
Not overly familiar with Spark but usually the bottleneck for CPUs is access to DRAM. Unless you're doing completely linear reads or prefetching appropriately(which almost no one does right) you'll be cache-miss bound and your CPUs will be idle waiting for data.
This is the thesis of those who have been watching the explosion in solid state disks. The claim is that bulk I/O is becoming so damn fast that the pendulum is swinging toward computing power being the new limiting factor.
No expert here on SSDs but it still seems like they're off by an order of magnitude. PCIe SSDs appear to top out around 1.8GB/sec while DDR4 is in the 12-19 GB/sec range.
Either way CPU processing power has never really been a constraining factor(since most people don't want to write data aware code), it's always been bus-bound be it DRAM, IO or other.
Spark is built around keeping things in-memory. If there's any workload that can max out a CPU (and surely there must be, otherwise no-one would bother building faster CPUs) then there's probably someone doing it on Spark (linear algebra? monte carlo simulation?). Spark might be a good fit with RDMA (looks like there's some initial work in that direction).
It entails fusing "narrow" operations on data frames into a single method of generated code to avoid virtual method invocations, help the JIT, and improve hardware branch prediction among other things.
Basically you take the general, composable API functions and generate specific equivalent code that avoids the overhead of dealing with abstract interfaces like Iterators.
Yeah, since spark 1.6 (some optimizations in spark 1.5) the data being operated on is managed "off heap" which means it doesn't add to garbage collection times and is stored in a more contiguous and cache friendly layout. It's definitely using Unsafe but can't speak to the exact implementation.
> Basically you take the general, composable API functions and generate specific equivalent code that avoids the overhead of dealing with abstract interfaces like Iterators.
That's pretty cool. Similar to what Rust already does when the compiler elides allocations for chained iterators. I wonder if value type support in the JVM would avoid the need to do this type of code gen.
Has it become easier to run ad hoc queries with spark?
I remember a year ago that the only available solution was the job server by ooyala.
Which seems to be a missing feature of core Spark, and isn't something I was willing to bet my product on.
Datastax evangelized people to use Spark to run queries over Cassandra but it looks so awkward and time consuming to copy jars around to the master, basically you need a dev ops team to this and even more scriptology for production.
Infinitely. Yes. The addition of Dataframes in 1.3 and numerous enhancements to SparkSQL have made it as easy as:
val myDF = sqlContext.read.format("com.databricks.spark.csv") //allows you to read a csv file (for simplicity)
.option("header", "true")
.option("delimiter", "\\t")
.option("mode", "PERMISSIVE")
.option("inferSchema", "true")
.load(filename)
.where(filter_query)
.cache()
myDF.registerTempTable("myDF")
sqlContext.sql("SELECT COUNT(*) FROM myDF").show()
sqlContext.sql("SELECT COUNT(*) FROM myDF where filter_query").show()
Presto seems like a good product, haven't really test it too much tho.
When we started to look at Cassandra 1.5 years ago, Spark integration was all the rage, and the premise it was the missing link for doing analytic on Cassandra.
We tested it thoroughly and came to the conclusion that is wasn't a mature enough solution.
Are Spark streams ever going to reach a point where you can just have a table sitting in memory aggregating data and then you run queries on the whole thing without having to worry about windowing or anything?
How advanced is the Structured Streaming functionality? Looking at the JIRA [1] I cannot find even design prototype there, which is kind of strange if they want to have it ready by end of April. But as there was a presentation on the topic at the summit [2], I hope it's just developing it without discussion on JIRA.
28 comments
[ 2.5 ms ] story [ 66.7 ms ] threadBut you can always download and build the source!
Matei was actually talking about the existing Spark Kafka direct stream implementation, which has been available since Spark 1.3
The video of the talk is available here: http://livestream.com/fourstream/sparksummiteast2016-tracka/...
> CPU speeds have not kept up with I/O in the past 5 years.
I presume he means the other way around?
Also, what does he mean by native memory management? Does he mean off-heap allocation?
And what's he referring to regarding code generation?
Native refers to (I think) the following issues: https://issues.apache.org/jira/browse/SPARK-12785 https://issues.apache.org/jira/browse/SPARK-8641
Code generation is enabled by SPARK-8641, but not sure exactly what it entails. I think it is related to some of the RDD transformation/action merging they do to optimize runtime operations in 2.0.
Your thoughts?
This is the thesis of those who have been watching the explosion in solid state disks. The claim is that bulk I/O is becoming so damn fast that the pendulum is swinging toward computing power being the new limiting factor.
Either way CPU processing power has never really been a constraining factor(since most people don't want to write data aware code), it's always been bus-bound be it DRAM, IO or other.
Basically you take the general, composable API functions and generate specific equivalent code that avoids the overhead of dealing with abstract interfaces like Iterators.
How does Spark guarantee data memory contiguity(or do they at all)? Do they use misc.sun.unsafe or some form of memory management?
That's pretty cool. Similar to what Rust already does when the compiler elides allocations for chained iterators. I wonder if value type support in the JVM would avoid the need to do this type of code gen.
https://news.ycombinator.com/item?id=11126179
https://spark-summit.org/east-2016/events/graphframes-graph-...
Datastax evangelized people to use Spark to run queries over Cassandra but it looks so awkward and time consuming to copy jars around to the master, basically you need a dev ops team to this and even more scriptology for production.
Why not just use Presto? It gives you basically full SQL capability for Cassandra with minimal effort.
We tested it thoroughly and came to the conclusion that is wasn't a mature enough solution.
[1] https://issues.apache.org/jira/browse/SPARK-8360 [2] https://spark-summit.org/east-2016/events/keynote-day-3/