HN Explorer

Ask HN: Explain “DMX ETLs” – Inherited Old + Obscure Tech

1 points by WhompingWindows ↗ HN
Can anyone point me to useful resources, or just explain, what DM Express ETLs might be for? Most of the results I see are about the rapper, not the tech tool. What would be analogous in "modern" data pipelines? I think this DMX is from Syncsort from like 2010 or so, I found some YT tutorials from 2012 but that's about it.

We have a database that's being cloned, the clone is helpful to make faster queries and as a protective backup, I suppose? I am still a beginner, would love any resources you may have on this older technology.

2 comments

[ 1.4 ms ] story [ 10.7 ms ] thread
ahazred8ta
It's for companies that have large data warehouse analytics systems. -- highlights: "Syncsort cites its main data integration competitors as being, no surprise, Ab Initio (their favorite choice to compare themselves to), Informatica (with PowerCenter), and IBM (with DataStage). Syncsort DMExpress runs on dedicated boxes. Unlike most of its competitors, this is not a real-time product it is a batch product that you schedule. Syncsort also told a story of an unnamed customer for whom Oracle utterly choked on joining 5 tables of 1 terabyte each. (27 days to run with clever workarounds.) DMExpress did the join in 6 hours." -- So, it's apparently so large and enterprisey that getting it to work right would take a large amount of effort, a sizable budget, and paid tech support.

https://www.dbms2.com/2011/04/07/introduction-to-syncsort-an...

WhompingWindows
Thanks for your comment! I gather we're using it for scheduled jobs and for its speed relative to R or Python. It's funny that so many of the sources of info on it are from around 2010-2012 range.

What would you say is the "modern" approach for joining large data quickly, and for creating clones and running various jobs repeatedly?