The name of the LLVM AD tool is actually Enzyme [http://enzyme.mit.edu/] (Zygote is a Julia tool)
Oh for sure, any ML framework worth its salt should do some amount of graph rewriting / transformations. I was (perhaps poorly) trying to explain how while yes AD (regardless of implementation in Enzyme, PyTorch, etc)…
Regarding differentiating python via CPython, theoretically yes, though practically it is likely more wise to use something like Numba which takes Python to LLVM directly to avoid a bunch of abstraction overhead that…
Whoops added one too many zero’s there, agreed that would be really nice :P
Enzyme does indeed handle mutable arrays (both in Enzyme.jl and any other frontend)! If you want to try it out forewarned that we're currently upgrading Enzyme.jl for better JIT integration (dynamic re-entry, custom…
Yeah my best guess at that is that they were trying to say you'd only need to store one value: the sum, rather than the two individual values -- but I'm not completely sure.
Reverse mode AD can always get into situations where it needs to store original values (i.e. network state). One advantage, however, of doing a more whole-program approach to AD rather than individual operators is that…
Enzyme needs to be able to access the IR of any potentially active functions (calls that it deduced could impact the gradient) to be able to differentiate them. If all of the code you care about is in one compilation…
You don't always need the input to compute the gradient. For example the gradient of a sum function doesn't require the original input, it just sets all of the derivative(input)'s to 1.
I think in essence what PartiallyTyped is trying to say is that one potential optimization opportunity in whole-program AD is that you can avoid having to cache the original inputs of the program if you know that…
Oh man that was a fun hack to write. Basically we demonstrated an easy-to-setup AD on rust by leveraging link-time optimization (LTO) as a way to make sure Enzyme's generate derivatives "optimization pass" was run.…
For GPU's, there's a couple of different things that you might want to do. You can use existing tools within LLVM to automatically generate GPU code out of existing code, and this works perfectly fine, even running…
Adding onto this, numerical derivatives have two potential problems which is why they tend not to be used in big scientific/ML frameworks. First of all they suffer from accuracy decay. For example if you were to do the…
The name of the LLVM AD tool is actually Enzyme [http://enzyme.mit.edu/] (Zygote is a Julia tool)
Oh for sure, any ML framework worth its salt should do some amount of graph rewriting / transformations. I was (perhaps poorly) trying to explain how while yes AD (regardless of implementation in Enzyme, PyTorch, etc)…
Regarding differentiating python via CPython, theoretically yes, though practically it is likely more wise to use something like Numba which takes Python to LLVM directly to avoid a bunch of abstraction overhead that…
Whoops added one too many zero’s there, agreed that would be really nice :P
Enzyme does indeed handle mutable arrays (both in Enzyme.jl and any other frontend)! If you want to try it out forewarned that we're currently upgrading Enzyme.jl for better JIT integration (dynamic re-entry, custom…
Yeah my best guess at that is that they were trying to say you'd only need to store one value: the sum, rather than the two individual values -- but I'm not completely sure.
Reverse mode AD can always get into situations where it needs to store original values (i.e. network state). One advantage, however, of doing a more whole-program approach to AD rather than individual operators is that…
Enzyme needs to be able to access the IR of any potentially active functions (calls that it deduced could impact the gradient) to be able to differentiate them. If all of the code you care about is in one compilation…
You don't always need the input to compute the gradient. For example the gradient of a sum function doesn't require the original input, it just sets all of the derivative(input)'s to 1.
I think in essence what PartiallyTyped is trying to say is that one potential optimization opportunity in whole-program AD is that you can avoid having to cache the original inputs of the program if you know that…
Oh man that was a fun hack to write. Basically we demonstrated an easy-to-setup AD on rust by leveraging link-time optimization (LTO) as a way to make sure Enzyme's generate derivatives "optimization pass" was run.…
For GPU's, there's a couple of different things that you might want to do. You can use existing tools within LLVM to automatically generate GPU code out of existing code, and this works perfectly fine, even running…
Adding onto this, numerical derivatives have two potential problems which is why they tend not to be used in big scientific/ML frameworks. First of all they suffer from accuracy decay. For example if you were to do the…