ryan_green
No user record in our sample, but ryan_green has activity below (stories or comments). Likely we have partial data — the full bulk-load will fill profiles in.
No user record in our sample, but ryan_green has activity below (stories or comments). Likely we have partial data — the full bulk-load will fill profiles in.
exactly!!!
apologies for not getting into more detail--wanted to start by covering things at a high level. There are a few key concepts that might be helpful. * data state - this is contents of both your data and metadata at a…
sort of. the main issues we've found with each developer having their own DB on a complex data pipeline are 1) if that DB contains petabytes of data, creating it one for each developer is non-trivial from a time and…
saltcured, find these comments super insightful! > Yeah, there's a lot of hidden magic/assumptions in having a "writable snapshot of a specific version" of production data. That's absolutely a huge assumption. This…
NBJack, your point about the difficult of managing external dependencies is well taken. That said, our data pipeline uses cloud storage and multiple external services and the scenarios you're describing haven't…
totally agree. and creating ephemeral environments for data pipelines is quite a bit more challenging than systems with a less complicated data state. nonetheless, this has paid off for us already many times over.