Ask HN: Is it a good idea to use AWS Lambda for ETL Pipelines?

4 points by 91dylwyn 4y ago ↗ HN

We are working on an OSS for ETL (Typhoon data orchestrator) and we are thinking about using AWS Lambda as the backend. The benefit is that it has a built in scheduler and we don't need to build our own. We can also spawn multiple Lambdas to scale concurrency.

Do you think this would be a sensible option? Would the 15 minutes running time be a deal breaker?

3 comments

[ 1.8 ms ] story [ 13.1 ms ] thread

ignorantguy 4y ago ↗

We use lambda in our production environment for ETL. It works really well. In our case when we were using it, we modeled the code to account for 15 minute window. Hope that helps.

91dylwyn 4y ago ↗

Are you finding any pain points? How did you manage the 15 minute window in case of large amounts of data running over time?

How do you manage the DAGs on it? For example creating dependencies between tasks and monitoring and rerunning tasks?

vlahmot 4y ago ↗

You can always use step functions or some janky recursive calls to work around the 15 min timeout.

I would recommend using docker containers and writing it in a way that you can swap out lambda/ecs/k8s/etc as the backend infrastructure. That way you you can punt on doing any fancy work to try and break the jobs up across lambdas under that 15 min constraint.