How would I go about using Mistral 7b as a starting point for making a model like SDXL? Obviously somebody's gonna do it, so I'd like to try myself :) I'd like to use a quantized version, you know?
You mean as a prompt generator? There are already finetunes for this, though most people use smaller models.
If you are talking about transformers llms that output pixels, that is an area of active research... I hope you have a few million dollars lying around, though.
I thought that you can put the prompt into Mistral but instead of going all the way to output, you just like take the state of Mistral when it's like halfway done and use that as the "embedding". Then that gives you the prompt embedding or something, then you feed that into the image generator. Is that not how SDXL works?
I don't have a few million lying around, but it shouldn't be hard to get if I have a good prototype that generates 64x64 images you know. Do you know how to make one from scratch?
I'm willing to be proven wrong, but I don't think that idea would work. They're two totally different models (and with different modalities too), so any embeddings you might produce with one model will be nonsensical to another unless translated in some way.
3 comments
[ 0.23 ms ] story [ 14.7 ms ] threadIf you are talking about transformers llms that output pixels, that is an area of active research... I hope you have a few million dollars lying around, though.
I thought that you can put the prompt into Mistral but instead of going all the way to output, you just like take the state of Mistral when it's like halfway done and use that as the "embedding". Then that gives you the prompt embedding or something, then you feed that into the image generator. Is that not how SDXL works?
I don't have a few million lying around, but it shouldn't be hard to get if I have a good prototype that generates 64x64 images you know. Do you know how to make one from scratch?