I don't understand this nightshade tool. it says the tool works for many (if not all) models without needing to accessing the weights, yet when creating a poisoned image, it requires an existing model and it tries to let the poisoned image to generate a similar feature map of the unpoisoned image.
how can it be sure that the feature map will be the same for all models?
From what I understand the attack requires access to at least a model to attack.
The result has the highest success on that model you specifically attacked, but it apparently transfers albeit with a lower success rate.
> how can it be sure that the feature map will be the same for all models?
From what I can tell it has a lot to do with what text encoder they used, although curiously it still transferred well from SDXL/SD2 which both CLIP to deep floyd which uses a completely different architecture
> The result has the highest success on that model you specifically attacked, but it apparently transfers albeit with a lower success rate.
Which is already great. Especially given that most users tend to use the same product. It's not like it has to poison all models; it just has to poison the more expensive ones.
And hopefully they will find a way to improve it such that it generalizes better.
> It's not like it has to poison all models; it just has to poison the more expensive ones.
Did you mean common ones? I'm not 100% sure what you mean by expensive, compute, api provider costs etc. But unless I missed it they didn't attempt anything against Dalle or GPT5.
The most important part is the text classifier/embedder and I'm unsure of how well this would transfer to those. I find it bizarre that the SD attack transferred to DF because DF essentially uses a frozen LLM vs the CLIP model and are nothing alike.
Yeah, probably. I guess I am mostly happy to see that there is some hope for technical solutions for artists.
> I find it bizarre that the SD attack transferred to DF because DF essentially uses a frozen LLM vs the CLIP model and are nothing alike.
Do you think that the idea of AI poisoning generalizes, though (even if it would require very different algorithms), or do you think that some models may be "safe"?
> Do you think that the idea of AI poisoning generalizes, though (even if it would require very different algorithms)
Yes, these are all different classes of adversial attacks and I don't think any model could resist everything, assuming you have direct access to the weights (although that's generally not a hard requirement)
> or do you think that some models may be "safe"?
I don't think any single set of model weights can be "safe". I suspect adding more adversaial data to the training, particularly of the text encoders might make them more resistent.
I unfortunately don't have high hopes that artists have a great chance of defending their work with these methods, nearly all of the existing ones are easily evadable, and I suspect more adversarial resistant methods will be implemented.
On top this, I think resisting adversial inputs will be a focus for many, not because they want to maliciously copy data, but because of the implications of multimodal models being attacked. Imagine a billboard that will be recognized as instructions to do XYZ (has been demoed with GPT4V)
I don't think I see it mentioned in the paper but what would happen if you used the poisoned images to train the text encoder? Would that make the text encoder itself resistant to at least the previously poisoned images?
The images will still be titled and whatnot correctly, so unless someone is also attempting to trick scrapers, the actual captions will be correct. If this is the case can't you just compare the image classifier's and the actual caption to tell if it was poisoned.
7 comments
[ 3.0 ms ] story [ 35.8 ms ] threadhow can it be sure that the feature map will be the same for all models?
The result has the highest success on that model you specifically attacked, but it apparently transfers albeit with a lower success rate.
> how can it be sure that the feature map will be the same for all models?
From what I can tell it has a lot to do with what text encoder they used, although curiously it still transferred well from SDXL/SD2 which both CLIP to deep floyd which uses a completely different architecture
Which is already great. Especially given that most users tend to use the same product. It's not like it has to poison all models; it just has to poison the more expensive ones.
And hopefully they will find a way to improve it such that it generalizes better.
Did you mean common ones? I'm not 100% sure what you mean by expensive, compute, api provider costs etc. But unless I missed it they didn't attempt anything against Dalle or GPT5.
The most important part is the text classifier/embedder and I'm unsure of how well this would transfer to those. I find it bizarre that the SD attack transferred to DF because DF essentially uses a frozen LLM vs the CLIP model and are nothing alike.
Yeah, probably. I guess I am mostly happy to see that there is some hope for technical solutions for artists.
> I find it bizarre that the SD attack transferred to DF because DF essentially uses a frozen LLM vs the CLIP model and are nothing alike.
Do you think that the idea of AI poisoning generalizes, though (even if it would require very different algorithms), or do you think that some models may be "safe"?
Yes, these are all different classes of adversial attacks and I don't think any model could resist everything, assuming you have direct access to the weights (although that's generally not a hard requirement)
> or do you think that some models may be "safe"?
I don't think any single set of model weights can be "safe". I suspect adding more adversaial data to the training, particularly of the text encoders might make them more resistent.
I unfortunately don't have high hopes that artists have a great chance of defending their work with these methods, nearly all of the existing ones are easily evadable, and I suspect more adversarial resistant methods will be implemented.
On top this, I think resisting adversial inputs will be a focus for many, not because they want to maliciously copy data, but because of the implications of multimodal models being attacked. Imagine a billboard that will be recognized as instructions to do XYZ (has been demoed with GPT4V)
The images will still be titled and whatnot correctly, so unless someone is also attempting to trick scrapers, the actual captions will be correct. If this is the case can't you just compare the image classifier's and the actual caption to tell if it was poisoned.