HN Explorer

Ask HN: How to build data pipelines to continuously ingest customer files?

6 points by SpeakerFrThDead ↗ HN
I'm working on onboarding a customer who wants to deliver their data as a weekly dump of CSVs. We’re supposed to ingest that data and get it into our system to provide analytics for their team.

I was initially thinking of just building a one-off ETL script, but I was warned the files may randomly break spec (new/renamed fields, etc) due to errors in the process that generates them. Is there a standard way to handle this type of thing?

2 comments

[ 30.3 ms ] story [ 669 ms ] thread
Someone
- Automate schema detection and reporting of discrepancies (to both the client and you), and make that 100% robust.

- get a good SLA in place (you can’t promise to load their data within an hour if they essentially can send random crap, for example)

- Add a zero to what you charge them.

Alternatively, walk away from that customer.

brudgers
The standard way is manually until there's enough experience to automate some of it away. Understanding the actual problem first is a direct step toward a robust solution to the actual problem. Building a non-trivial process is a process. Good luck.