Prompt-Specific Poisoning Attacks on Text-to-Image Generative Models (arxiv.org)

Basically, by putting a relatively small number of adversarial examples into the training data of a text-to-image model (that don't necessarily look suspicious to a human observer), they can make it completely mislearn a concept.