Airflow but use KubernetesPodOperator so you can run your ETLs in docker containers or pods within k8s (any language you want). You may need to write a few lines of dumb python code to build a DAG of KubernetesPodOperators but the actual work is done within containers.
Pretty cool, thanks for sharing. I have no personal experience with k8s, so from what I gather seems we could achieve something similar with serverless functions instead of containers?
Not that I can vouch for this as a “good” choice, but you can always use the old school
original ETL of any operating system + small tools (awk/sed/jq/grep/… etc) + shell scripts (sh/bash/zsh/etc…) and optionally make files to help if the way they work is a good fit for how you process the data.
3 comments
[ 15.8 ms ] story [ 129 ms ] threadNot being snarky this is genuinely a potential good solution in some situations https://adamdrake.com/command-line-tools-can-be-235x-faster-...