HN Explorer

Ask HN: AWS is telling us using Python for ETL is an antipattern, true?

6 points by kfk ↗ HN
Me and my team happily use Python + Dask for various things, one of them is ETL jobs. Most of what we do with Python is reading data via ODBC drivers and bulk uploading it into AWS S3 and Redshift. Now AWS is telling our IT that this is an antipattern. It seems Glue or Talend would be the best practice. I am confused as to how is having version controlled Python code to do ETL an antipattern? I can read and manage code much better than UI's. I always thought of tools like Talend as good to haves, but not as necessary if you have a solid team that can do Python. What is your experience?

6 comments

[ 0.22 ms ] story [ 22.7 ms ] thread
ldng
Glue as in AWS Glue ? Of course they'll want you to buy in their solution. The guy is just trying to upsell you their product.

BTW, Talend uses an embedded Python for scripting purpose.

UI are good if you want to delegate the monitoring of the process to less qualified people. That is usually the case in big companies.

mk89
Talend is also an AWS partner. Interesting how expert/clever these solution architects are nowadays. And of course, it's AWS certified, much better than your low skilled data engineers, so it must be true.
gshdg
It’s quite common and not an anti pattern at all. Wtf, Amazon?
brodouevencode
Are you sure it’s python specifically they are talking about? Glue does simplify a lot of that data moving for you, and probably what they are recommending.
nunez
That sounds...incorrect. From my last exposure to Talend, what it was being used for could absolutely be done with Python