GoodReads Data Pipeline

An end-to-end data pipeline capturing source data from GoodReads API using API calls. ETL jobs are written in Spark to transform the data and loads them into Data Warehouse on Redshift. The pipeline is managed using Airflow and scheduled to run every 10 minutes. At a minimum configuration of 3 node EMR cluster and a 2 node Redshift Cluster, the pipeline can handle around 2TB of data every day.

https://github.com/san089/goodreads_etl_pipeline