Ask HN: Custom data pipelines with minimal engineering effort
I am a one-man-shop in BI (Series B - enterprise SaaS) and our current stack includes Fivetran, Snowflake, DBT (+Cloud), and Looker. I am not an engineer by hard and our goal is to minimize engineering resources required (buy over build). What is a good approach to hosting and scheduling custom ETL pipelines for data sources not supported by Fivetran/Stitch?
We already considered the following options:
1. Run airflow on an EC2 instance. Cons: we didn't manage to install Airflow on EC2; others warned it's hard to maintain 2. Use Fivetran to schedule Lambda functions. Cons: Lambda is limited to a 15 minutes execution time and it wouldn't be the easiest to set up either 3. Cron jobs: this is limited in its ability to sequence different jobs and it doesn't really have any alerting in place
I don't think this is an uncommon problem - curious to know how you solve the issue!
0 comments
[ 3.0 ms ] story [ 11.6 ms ] threadNo comments yet.