You can use Pytorch for more than just Neural Networks - its autograd is super powerful for any problem where you need gradients (and are too lazy to calculate them yourself...)!
Author here. Yep, obviously this is more commonly done with dedicated optimization libraries, but the fun part was doing it in Pytorch to use autograd and as a way to visualize optimizers etc :)
Nice visualizations! Brings back fond memories of playing with Maplesoft in the 1990s to build up intuition about physical systems. In a nonlinear optimization class I was taking, I wanted to find the optimal truss configuration of a stack of hexapods [1] just as you're doing, but I got stuck on the Lagrange multipliers and gave up. You've shown it's much easier today using autodiff.
It looks like the method you used has a lot in common with a finite elements analysis. At least the simple version of it that structural engineers use in software like sap2000 and RISA. Do you think it would work if the matrix in the equation were a lot bigger?
I remember several years ago when differentiable programming was an object of interest to the programming community and Lattner was trying to make Swift for Tensorflow happen[1].
I'm of the opinion that it was ahead of its time: Swift hadn't (and still hasn't) made enough progress on Linux support for it to be taken seriously as a language for writing anything that isn't associated with Apple. However, as a result, Swift now has language-level differentiability in its compiler. I'd love to see Swift get used for projects like this, but I suppose the reality of the matter is that there are so many performant runtimes for 2D/3D physics that there just isn't much of a need for automatic differentiation (and its overhead) to solve these problems. The tooling nerd in me thinks this stuff is fascinating.
He moved on to mojo right? Pretty sure the wants to implement what he had in mind using mojo without being hindered by what apple wanted to do with swift
I really wish Linux swift was a thing. I’m a relatively new cs person, I’m going to grad school and doing interdisciplinary work where I am making all sorts of neural networks. So by default my language of choice right now is Python. But I’d love to get sharp at Swift! I like the syntax, I like how I could use it to write apps for my phone, I like how it’s fast.
But my daily driver is a system76 Linux box running PopOS… that’s not a swift friendly environment. I guess the best compromise I’ve found for stuff I want to run in both places is Pythonista? Which is fantastic to be clear, but iOS is really constrained…
I want the ML infrastructure of Python, the speed of C and the ability to write fast cross platform apps in a syntax that isn’t confusing violence… Save me programming language nerds.
As user rdedev mentions, check out Mojo. It feels very much like it's trying to address a lot of your needs. The main compromise is that it appears a large portion of the toolchain is closed-source at the moment, and it appears that they're trying to get developers onto their enterprise offering, whatever that entails. Here's the link to Mojo: https://www.modular.com/max/mojo
Ehh, there’s a lot that goes into it. Just because a physics engine is differentiable doesn’t mean that its gradients going to be useful. For example, if you look at Brax/Mujoco in JAX, the gradients generated by Mujoco are absolute garbage if you’re trying to train a robotics controller, but the more video-game like engines give pretty good results (see https://github.com/jiawei-ren/diffmimic).
> You can use Pytorch for more than just Neural Networks
Facts. Pytorch is such a fun too for applied calculus. Just write down a program, compute its derivative, and do any of the fun things you can do with derivatives, like optimization or linear approximation.
Yeah exactly! I looked around, but couldn't find many other people using it for things besides neural networks though. Any favorite other examples you've seen?
Mostly I use pytorch for statistical modeling https://pyro.ai . Under the hood that package uses a lot of Monte Carlo integration and variational methods (i.e. integration by optimization). It does support neural nets, but probably >80% of pyro users stick to simpler hierarchical Bayesian models.
I want someone to do this for a fluid dynamics problem.
I suspect lots of things in everyday life could be made substantially better/cheaper/more efficient if entire system optimization like this could be done to their design.
There are neural networks which learn to produce an approximation, but much faster. Useful for games, maybe exploratory analysis before deploying using slow classic CFD.
I think there could be a combined approach where you use the fast neural net and the slow CFD as you state, but then every result you get from the CFD you add to the training data of the neural net and do a few more steps of training.
That way, the neural net gets more accurate specifically in the area of your design.
You could imagine a developers workstation where a GUI tool is letting them design stuff, and the neural net is giving them instant answers (and/or showing gradients to show how to improve stuff), and slow CFD runs are running in the background so whenever they go to lunch and another cfd run completes the results all get more accurate and detailled.
Kinda similar to the way many graphics packages have 'draft' quality 3d rendering, but then when idle for a bit will improve the render quality.
I forgot the name of the software but when I took statics way back in the day we used an app that let us design the bridge and then it animated a truck going across. It was cute but it also highlighted (literally) all the areas that were in stress during crossing. I enjoyed that class - it was interesting even if it had nothing to do with programming. :)
30 comments
[ 550 ms ] story [ 1050 ms ] thread[1] https://www.aerotech.com/products/hexapods
I'm of the opinion that it was ahead of its time: Swift hadn't (and still hasn't) made enough progress on Linux support for it to be taken seriously as a language for writing anything that isn't associated with Apple. However, as a result, Swift now has language-level differentiability in its compiler. I'd love to see Swift get used for projects like this, but I suppose the reality of the matter is that there are so many performant runtimes for 2D/3D physics that there just isn't much of a need for automatic differentiation (and its overhead) to solve these problems. The tooling nerd in me thinks this stuff is fascinating.
https://github.com/tensorflow/swift
But my daily driver is a system76 Linux box running PopOS… that’s not a swift friendly environment. I guess the best compromise I’ve found for stuff I want to run in both places is Pythonista? Which is fantastic to be clear, but iOS is really constrained…
I want the ML infrastructure of Python, the speed of C and the ability to write fast cross platform apps in a syntax that isn’t confusing violence… Save me programming language nerds.
Facts. Pytorch is such a fun too for applied calculus. Just write down a program, compute its derivative, and do any of the fun things you can do with derivatives, like optimization or linear approximation.
I suspect lots of things in everyday life could be made substantially better/cheaper/more efficient if entire system optimization like this could be done to their design.
Optimizing the bridge only works because my method for solving the forces in each beam (least squares) happens to be differentiable.
Then the network is differentiable.
There are neural networks which learn to produce an approximation, but much faster. Useful for games, maybe exploratory analysis before deploying using slow classic CFD.
That way, the neural net gets more accurate specifically in the area of your design.
You could imagine a developers workstation where a GUI tool is letting them design stuff, and the neural net is giving them instant answers (and/or showing gradients to show how to improve stuff), and slow CFD runs are running in the background so whenever they go to lunch and another cfd run completes the results all get more accurate and detailled.
Kinda similar to the way many graphics packages have 'draft' quality 3d rendering, but then when idle for a bit will improve the render quality.
It solves hyperbolic PDEs like the Euler equations, is differentiable with forward-mode AD, and MPI parallelized.
[0]https://www.chroniclogic.com/pontifex.htm