A while back, I realized that my machine learning projects each contained a subproject dedicated to recording what the heck I was doing, scattered throughout the actual training code. Sometimes this manifested as awful json files full of hyperparameters being passed around, other times long as saving weights to long paths like "timestamp/regularizer/learning_rate/weights.trch." Sometimes this subproject didn't mature in time, and I accidentally overwrote hard-earned weight files or figures because they were just at fixed paths.
Eventually, I settled on a standard for recording hyperparameters: the best thing to do is just commit my hyperparameters to the git repo, and save git hashes alongside experiment artifacts. Of course, I am imperfect, and I found that I would sometimes start training without committing my changes. An easy fix- save the output of git diff too. Eventually everything else got packratted too- package versions, which computer we're running on, which virtual environment, command line args, etc. For the last couple years, whenever I found myself saying "Dang, I could reproduce this result if only I knew X", I threw X into footsteps.
The final piece of the puzzle is naming the folder where we shove all this stuff, along with the weights / figures / etc. Timestamp is cryptic, choosing which hyperaprameter is important enough to go in the name is GAI- the only answer is to ask the user to name their experiment. Much like git messages, this seems like a burden- the only real burden footsteps imposes, I'd argue- but it's worth it for finding the experiment you're looking for. I added tab completion to make it easier to just input a small change to the name of the last experiment, and disambiguation (tacking -1, -2, -3 on the end) to prevent overwriting old experiments if you get especially uncreative.
Now, it's packaged up and easy for you to use too! Heavily tested on windows and linux; conda, venv, and raw python, seems to work on my mac, available on pypi.
1 comment
[ 5.0 ms ] story [ 8.3 ms ] threadEventually, I settled on a standard for recording hyperparameters: the best thing to do is just commit my hyperparameters to the git repo, and save git hashes alongside experiment artifacts. Of course, I am imperfect, and I found that I would sometimes start training without committing my changes. An easy fix- save the output of git diff too. Eventually everything else got packratted too- package versions, which computer we're running on, which virtual environment, command line args, etc. For the last couple years, whenever I found myself saying "Dang, I could reproduce this result if only I knew X", I threw X into footsteps.
The final piece of the puzzle is naming the folder where we shove all this stuff, along with the weights / figures / etc. Timestamp is cryptic, choosing which hyperaprameter is important enough to go in the name is GAI- the only answer is to ask the user to name their experiment. Much like git messages, this seems like a burden- the only real burden footsteps imposes, I'd argue- but it's worth it for finding the experiment you're looking for. I added tab completion to make it easier to just input a small change to the name of the last experiment, and disambiguation (tacking -1, -2, -3 on the end) to prevent overwriting old experiments if you get especially uncreative.
Now, it's packaged up and easy for you to use too! Heavily tested on windows and linux; conda, venv, and raw python, seems to work on my mac, available on pypi.