Of course interesting things happen outside of mainstream deep learning. The problem is that sorting by citation count is a terrible way to look broadly at research, because mainstream deep learning is so hyped up that…
Moving the momentum update before the parameter update changes the algorithm from LION to SignSGD. The only novel part of their algorithm is the fact that the momentum changes after the update is computed. They are not…
The second line appears unnecessary because the formula in that comment is missing one line from the definition of LION. See my above comment.
Your formula is almost right but there is a slight difference. Namely the third line of Program 1 in their paper is absent from your statement, which will change the value of the momentum variable for subsequent steps.…
Of course interesting things happen outside of mainstream deep learning. The problem is that sorting by citation count is a terrible way to look broadly at research, because mainstream deep learning is so hyped up that…
Moving the momentum update before the parameter update changes the algorithm from LION to SignSGD. The only novel part of their algorithm is the fact that the momentum changes after the update is computed. They are not…
The second line appears unnecessary because the formula in that comment is missing one line from the definition of LION. See my above comment.
Your formula is almost right but there is a slight difference. Namely the third line of Program 1 in their paper is absent from your statement, which will change the value of the momentum variable for subsequent steps.…