Ask HN: How do you like to version datasets for production ML systems?

4 points by wskish 3y ago ↗ HN

At my previous company (industrial ai vision platform) we had an ad-hoc mechanism for continuously versioning datasets for user applications that we built internally several years ago.

I am curious how folks are versioning datasets now in practice. Platforms like Hugging Face and Weights and Biases seem to provide good abstractions for dataset versioning. Any feedback on the suitability of these for production systems, or other systems, patterns, or best practices that you have found to work well?

2 comments

[ 2.1 ms ] story [ 17.5 ms ] thread

kristenkehrer 3y ago ↗

I use CometML (because I work there), but creating data artifacts and versioning my data have made life so much easier.. especially if I put a project down for a bit and try to pick it back up.

wskish 3y ago ↗

Any talking points on how CometlML compares to HF and wandb for this purpose?