The name of the LLVM AD tool is actually Enzyme [http://enzyme.mit.edu/] (Zygote is a Julia tool)
Oh for sure, any ML framework worth its salt should do some amount of graph rewriting / transformations. I was (perhaps poorly) trying to explain how while yes AD (regardless of implementation in Enzyme, PyTorch, etc)…
Regarding differentiating python via CPython, theoretically yes, though practically it is likely more wise to use something like Numba which takes Python to LLVM directly to avoid a bunch of abstraction overhead that…
We go into more details in the Limitations section of the paper, but in short Enzyme requires the following properties: * IR of active functions must be accessible when Enzyme is called (e.g. cannot differentiate…
Whoops added one too many zero’s there, agreed that would be really nice :P
Enzyme does indeed handle mutable arrays (both in Enzyme.jl and any other frontend)! If you want to try it out forewarned that we're currently upgrading Enzyme.jl for better JIT integration (dynamic re-entry, custom…
Say you have some existing virus simulation codebase that you want to use ML on to derive an effective policy on. Without an AD tool like Enzyme, you'd have to spend significant time and effort understanding and…
Reverse mode AD can always get into situations where it needs to store original values (i.e. network state). One advantage, however, of doing a more whole-program approach to AD rather than individual operators is that…
Enzyme needs to be able to access the IR of any potentially active functions (calls that it deduced could impact the gradient) to be able to differentiate them. If all of the code you care about is in one compilation…
You don't always need the input to compute the gradient. For example the gradient of a sum function doesn't require the original input, it just sets all of the derivative(input)'s to 1.
I think in essence what PartiallyTyped is trying to say is that one potential optimization opportunity in whole-program AD is that you can avoid having to cache the original inputs of the program if you know that…
Oh man that was a fun hack to write. Basically we demonstrated an easy-to-setup AD on rust by leveraging link-time optimization (LTO) as a way to make sure Enzyme's generate derivatives "optimization pass" was run.…
For GPU's, there's a couple of different things that you might want to do. You can use existing tools within LLVM to automatically generate GPU code out of existing code, and this works perfectly fine, even running…
Adding onto this, numerical derivatives have two potential problems which is why they tend not to be used in big scientific/ML frameworks. First of all they suffer from accuracy decay. For example if you were to do the…
Enzyme is named such as it's a tool that "synthesizes derivatives" and also as a pun referencing Zygote (another AD tool) since Enzyme operates at a lower level (LLVM rather than Julia).
Hi all, another author here and happy to answer any questions! Some more relevant links for the curious Github: https://github.com/wsmoses/Enzyme Paper: https://proceedings.neurips.cc/paper/2020/file/9332c513ef44b...…
Hi all, author here. A couple of relevant links for the curious Github: https://github.com/wsmoses/Enzyme Paper: https://proceedings.neurips.cc/paper/2020/file/9332c513ef44b... Project: enzyme.mit.edu Basically the long…
The name of the LLVM AD tool is actually Enzyme [http://enzyme.mit.edu/] (Zygote is a Julia tool)
Oh for sure, any ML framework worth its salt should do some amount of graph rewriting / transformations. I was (perhaps poorly) trying to explain how while yes AD (regardless of implementation in Enzyme, PyTorch, etc)…
Regarding differentiating python via CPython, theoretically yes, though practically it is likely more wise to use something like Numba which takes Python to LLVM directly to avoid a bunch of abstraction overhead that…
We go into more details in the Limitations section of the paper, but in short Enzyme requires the following properties: * IR of active functions must be accessible when Enzyme is called (e.g. cannot differentiate…
Whoops added one too many zero’s there, agreed that would be really nice :P
Enzyme does indeed handle mutable arrays (both in Enzyme.jl and any other frontend)! If you want to try it out forewarned that we're currently upgrading Enzyme.jl for better JIT integration (dynamic re-entry, custom…
Say you have some existing virus simulation codebase that you want to use ML on to derive an effective policy on. Without an AD tool like Enzyme, you'd have to spend significant time and effort understanding and…
Reverse mode AD can always get into situations where it needs to store original values (i.e. network state). One advantage, however, of doing a more whole-program approach to AD rather than individual operators is that…
Enzyme needs to be able to access the IR of any potentially active functions (calls that it deduced could impact the gradient) to be able to differentiate them. If all of the code you care about is in one compilation…
You don't always need the input to compute the gradient. For example the gradient of a sum function doesn't require the original input, it just sets all of the derivative(input)'s to 1.
I think in essence what PartiallyTyped is trying to say is that one potential optimization opportunity in whole-program AD is that you can avoid having to cache the original inputs of the program if you know that…
Oh man that was a fun hack to write. Basically we demonstrated an easy-to-setup AD on rust by leveraging link-time optimization (LTO) as a way to make sure Enzyme's generate derivatives "optimization pass" was run.…
For GPU's, there's a couple of different things that you might want to do. You can use existing tools within LLVM to automatically generate GPU code out of existing code, and this works perfectly fine, even running…
Adding onto this, numerical derivatives have two potential problems which is why they tend not to be used in big scientific/ML frameworks. First of all they suffer from accuracy decay. For example if you were to do the…
Enzyme is named such as it's a tool that "synthesizes derivatives" and also as a pun referencing Zygote (another AD tool) since Enzyme operates at a lower level (LLVM rather than Julia).
Hi all, another author here and happy to answer any questions! Some more relevant links for the curious Github: https://github.com/wsmoses/Enzyme Paper: https://proceedings.neurips.cc/paper/2020/file/9332c513ef44b...…
Hi all, author here. A couple of relevant links for the curious Github: https://github.com/wsmoses/Enzyme Paper: https://proceedings.neurips.cc/paper/2020/file/9332c513ef44b... Project: enzyme.mit.edu Basically the long…