HN Explorer

AWS Redshift vs. Parquets and Apache Spark

1 points by dancrystalbeach ↗ HN
I've been mulling this over, and I can't make up my mind. What's better to use for a large data warehouse, AWS Redshift vs Apache Spark on top of Parquet files stored in s3? There really doesn't seem to be much different to me besides you can probably do "more" transforms with Spark. But the complexity is probably reduced with Redshift. I need thoughts and input.

6 comments

[ 3.4 ms ] story [ 23.4 ms ] thread
rahulj51
Cost will definitely be a factor but if it's not then I'd suggest Redshift because of the simplicity. Another aspect is how the data will be consumed - Analysts are typically more comfortable using SQL + Redshift. I'd avoid Spark except for very specific use cases as the operational toil is higher with Spark.
dancrystalbeach
the consumption is very SQLish based. "operational toil is higher with Spark" <- I agree with this, this is why I'm leaning towards Redshift. Throwing away half the codebase in favor of some SQL statements is tempting.
cmollis
how about the difference in cost? Redshift is pretty expensive. We use Spark to convert our data to Parquet and Presto to query it (although you can just use Spark to query it also)
dancrystalbeach
I've thought about that, i should probably do more detailed analysis of what the actual cost difference would be. I've looked at Presto as well... but again that is just another tech stack to manage.
PeterZaitsev
If you want cost efficient high performance Clickhouse is hard to beat