Spark on Your Oracle Data Warehouse

I have been developing 'Spark on Oracle', a project to provide better integration of Spark into an Oracle Data Warehouse. You can read about it at https://hbutani.github.io/spark-on-oracle/blog/Spark_on_Oracle_Blog.html

The key features are Catalog Integration, translation and pushdown of Spark SQL to Oracle SQL/PL-SQL, Language Integration and Runtime Integration.

These are provided as Spark extensions via a Catalog Plugin, v2 DataSource, Logical and Physical Planner Rules, Parser Extension, automatic Function Registration and Spark SQL Macros(a generic Spark capability we have developed).

The vision is to enable Oracle customers to deploy Spark Applications that take full advantage of the data and capabilities of their Oracle Data Warehouse; and also make Spark cluster operations simpler and unified with their existing Oracle warehouse operations.