He predicts this kind of model factory will become central to organizational learning and operations. Updating and upgrading the model stack becomes the core staff function.
I've been thinking if this could be done in Nix and toyed around with the idea just a little bit using nano-gpt.
Hermeticity always seems to mean isolation, but depending on who you ask it does not always mean computing some sort of hash over all build input as the 'identity' of a particular step in the pipeline, like Nix does.
If you do that hash-based identity part, looking up intermediary results and resuming from there happens using this sort of hash.
Does savanah do that, or will it resume where it left off based on a less strict notion of identity?
The blog post nails the problem well: with distributed training handled by many teams the problem becomes organizational rather then tech-based. Having a technology solution to more easily persist pipeline and artifact configs is a good idea, but it only yields an incremental improvement. IMO this can only solved with a shared culture around experimentation which has to be maintained and reinvented as the company (and hence the teams responsible for training) grows.
Basically, if every layer of the stack from tooling to training to applied are aiming for repeatable training with predictable capability improvements that end users can see, then it's more likely to happen.
I speak from ignorance, having not worked on this type of thing: are people NOT doing this? Are folks… what, passing CSVs and clicking GUIs to kick off one-off expensive, long running training runs? That’s absolutely crazy to me.
10 comments
[ 2.0 ms ] story [ 21.9 ms ] threadSo they’ve built Savanah - a workflow engine because the existing zoo of hundreds of workflow engines didn’t cut it :)
https://youtu.be/20p5-kQXF_Q?is=72ImTNxkOEKmOXQ9
He predicts this kind of model factory will become central to organizational learning and operations. Updating and upgrading the model stack becomes the core staff function.
Hermeticity always seems to mean isolation, but depending on who you ask it does not always mean computing some sort of hash over all build input as the 'identity' of a particular step in the pipeline, like Nix does.
If you do that hash-based identity part, looking up intermediary results and resuming from there happens using this sort of hash.
Does savanah do that, or will it resume where it left off based on a less strict notion of identity?
I could see arguments for either approach.