Data transforms on a data pipeline without ETL hell
So I'm building a data pipeline / analytics solutions. The functionality is pretty much what one would expect. Events are received, then stored, then some transformations are performed.
We have plenty of internal users who might do transformations with code (e.g. Python) or more standard ETL methods. Problem is we end up with ETL hell when we have a bunch of these transformations we're running, don't know who created them, which ones are no longer being used, which ones do basically the same time and therefore waste computer cycles, etc. It's just a mess.
Any suggestions on best practices around the transformation/ETL stage of a data pipeline?
0 comments
[ 0.24 ms ] story [ 7.7 ms ] threadNo comments yet.