I think this actually might be one of the best ways to train people to use AI. I can see this honing people's prompting abilities and expressiveness, along with constraints and desired outcome.
Really liked your video presentation, thanks for sharing, especially the part of image generators locking us into a certain context immediately, reducing us to spectators instead of creatives.
Played a lot with p5js some years ago, might pick it up again and try some of your ideas. The reinforcement learning part sounds about above my skill level though. :)
I am reminded of a paper I was inspired by a long time ago [0], (okay it's 2018 so I guess just 8 years ago, but it feels like longer, from the before-times), that demonstrated learning brush strokes. At the time there was already a lot of work on GANs, but these are pixel-based methods, and I was really interested in the idea of how to derive descriptive methods of scene generation/understanding. I found this work really interesting because it combined RL and GAN techniques in a creative way. I miss that kind of research.
Now of course VLMs have shown that you can mix modalities in generalized sequence-to-sequence problems and it doesn't surprise me that this kind of thing is possible, but it's so nice to see it done well using modern techniques.
Really awesome! Been thinking about how to get LLMs to do generative art (yes, the pre-AI definition of generative art). Love to see this approach and results!
For anyone looking to do this on their own, Datapoint (trydatapoint.com) lets you collect thousands of pairwise human preferences within minutes, and also has a pretty generous data grant for academics.
I did that on SVG mostly to teach it to draw pelicans but also to generalize it. most of the behavior is from SFT on the base model tho. RL is very ineficient at style or at least at generating novelty out of distrib.
tldr: Instead of using stable diffusion -> image -> fine-tuning towards image they like (making LORA adapter), they LLM -> code -> image -> fine-tuned LLM towards set of code snippets producing images they like.
Did it really worked ? I guess that if you do this for lets say four orthogonal LORAs the way Diffusion community does this, then load them all to LLM it should coherently mix concepts and produce coherent result, is someone done that yet ? I have never seen someone do this with LORA and LLMs, but it is walk in the park with Diffusion models, to load up to 10 LORA, just see CivitAI if you are not familiar with this.
because image models give us
photographs of a finished dish
opinionated tools give us
the recipe that made it
Disagree, get familiar with tooling around diffusion models, loras, comfyui it's all exactly these opinionated tools.
A fun diversion and general capabilities test that I like to do is to ask the models to "programmatically assign color and alpha values to a png to generate an image. it should be a painting or whatever subject interests you in the moment. 800x800" or something along those lines, and they'll make a "generative" art piece for you. For example first time I tried this Claude Fable used Python to generate a transparent PNG of a quartz crystal with inclusions that were variable depending on the seed number. So anyway yeah it's surprisingly easy to get the frontier models nowadays to make cool art!
36 comments
[ 0.22 ms ] story [ 20.6 ms ] threadWild the possibilities
People have done plenty with SVGs but it's rare to see human-in-the-loop approaches
Played a lot with p5js some years ago, might pick it up again and try some of your ideas. The reinforcement learning part sounds about above my skill level though. :)
Are they trained roughly like this? Or is it an LLM conditioned on image? Or on diffusion latents from a model trained to emit SVG-compatible imagery?
I am reminded of a paper I was inspired by a long time ago [0], (okay it's 2018 so I guess just 8 years ago, but it feels like longer, from the before-times), that demonstrated learning brush strokes. At the time there was already a lot of work on GANs, but these are pixel-based methods, and I was really interested in the idea of how to derive descriptive methods of scene generation/understanding. I found this work really interesting because it combined RL and GAN techniques in a creative way. I miss that kind of research.
Now of course VLMs have shown that you can mix modalities in generalized sequence-to-sequence problems and it doesn't surprise me that this kind of thing is possible, but it's so nice to see it done well using modern techniques.
[0] https://proceedings.mlr.press/v80/ganin18a.html
Did it really worked ? I guess that if you do this for lets say four orthogonal LORAs the way Diffusion community does this, then load them all to LLM it should coherently mix concepts and produce coherent result, is someone done that yet ? I have never seen someone do this with LORA and LLMs, but it is walk in the park with Diffusion models, to load up to 10 LORA, just see CivitAI if you are not familiar with this.
Disagree, get familiar with tooling around diffusion models, loras, comfyui it's all exactly these opinionated tools.It seems to me that teaching a model to directly produce SVG is more helpful than teaching to write JavaScript that produce pixels.
By the way, someone here is benchmarking LLMs on how they draw an SVG of a pelican on a bicycle.