The diagrams here really helped to explain neural ODEs in an intuitive fashion. Does anyone know the best library that implements them so I can play around in an iPyNB? :-)
Seems like it will be hard to get it to work in Tensorflow, because it needs to compute the gradient in an unusual way, which afaik don't play nice with the existing architecture.
My guess is it will need some deep wizardry of the same kind as OpenAI gradient check-pointing.
Neural ODE, is a nice trick to reduce the memory usage to O(1) instead of O(nb timesteps). But the implementation cost and complexity cost probably mean we are better using gradient check-pointing on a forward dynamic and pay the memory cost.
It will also probably won't play well with noise.
Are there any implementation of it in tensorflow yet?
Cool. This mean that hopefully there will be some tf open source implementations in the future. I guess there is something I don't see. I'm intrigued, does your code run inside a single sess.run() so that it can be composed nicely? If so did you use a "special trick"?
This is the first time I seriously read the paper and the code. Does this mean for ODE net, we are sharing weights for f? (even though we evaluate f at different y and t points)?
If that is the case, seems you can implement a full static version of ODE net with tf.While.
18 comments
[ 4.5 ms ] story [ 36.7 ms ] threadIf this is supposed to be the ODE definition, shouldn't it be y'(x) = f(x,y)? Otherwise I don't quite understand the definition of 'f'.
My guess is it will need some deep wizardry of the same kind as OpenAI gradient check-pointing.
Neural ODE, is a nice trick to reduce the memory usage to O(1) instead of O(nb timesteps). But the implementation cost and complexity cost probably mean we are better using gradient check-pointing on a forward dynamic and pay the memory cost.
It will also probably won't play well with noise.
Are there any implementation of it in tensorflow yet?
If that is the case, seems you can implement a full static version of ODE net with tf.While.