When tested against AIs such as DeepSeek V3, Qwen 3, and Phi-4, CatAttack increased the odds of incorrect answers by as much as 700%, depending on the model. And “even when CatAttack does not result in the reasoning model generating an incorrect answer, on average, our method successfully doubles the length of the response at least 16% of the times leading to significant slowdowns and increase in costs,” the team writes.
> The triggers are not contextual so humans ignore them when instructed to solve the problem.
Do they? I've found humans to be quite poor at ignoring irrelevant information, even when it isn't about cats. I would have insisted on a human control group to compare the results with.
If you spell “sit in the tub” s-o-a-k soak, and you spell “a funny story” j-o-k-e joke, how do you spell “the white of an egg”?
Context engineering* has been around longer than we think. It works on humans too.
The cats are just adversarial context priming, same as this riddle.
* I've called it "context priming" for a couple years for reasons showed by this child's riddle, while considering "context engineering" as iteratively determining what priming unspools robust resilient results for the question.
> Now, if I asked you, presumably a human, to solve that math problem, you’d likely have no issue ignoring the totally unrelated aside at the end there
I'm not so sure that is true. Good math students could ignore the cat fact, but I bet if you run this experimental in non-AP math classes you'll see an effect.
I don't think it's too unexpected: An LLM is an algorithm that takes a document and guesses a plausible extra piece to add. It makes sense it would generate more-pleasing output when run against a document which strongly resembles ones it was trained on, as opposed to a document made by merging two dissimilar and distinct kinds of document.
Sure, just one cat-fact can have a big impact, but it already takes a deal of circumstance and luck for an LLM to answer a math problem correctly. (Unless someone's cheating with additional non-LLM code behind the scenes.)
Funny, I was using chatGPT to have a conversation with a friend that doesn't speak English the other day. At the end of one of my messages, I appended 'how is your cat?', which was completely dropped from the translated output. I guess I'm doing it wrong?
I'm going to write duck facts in my next online argument to stave off the LLMs. Ducks start laying when they’re 4-8 months old, or during their first spring.
Careful, we don't know yet that this strategy generalises across cute animals. It could be that irrelevant duck facts enhance AI performance on maths questions.
"Irrelevant" facts about cats are the most interesting part of a math problem, because they don't belong there. The math problem was also "irrelevant" to the information about cats, but at least its purpose was obvious because it was shaped like a math problem (except for the interesting barnacle attached to its rear.)
Any person encountering any of these questions worded this way on a test would find the psychology of the questioner more interesting and relevant to their own lives than the math problem. If I'm in high school and my teacher does this, I'm going to spend the rest of the test wondering what's wrong with them, and it's going to cause me to get more answers wrong than I normally would.
Finding that cats are the worst, and the method by which they did it is indeed fascinating (https://news.ycombinator.com/item?id=44726249), and seems very similar to an earlier story posted here that found out how the usernames of the /counting/ subreddit (I think that's what it was called) broke some LLMs.
edit: the more I think about this, the more I'm sure that if asked a short simple math problem with an irrelevant cat fact tagged onto it that the math problem would simply drop from my memory and I'd start asking about why there was a cat fact in the question. I'd probably have to ask for it to be repeated. If the cat fact were math-problem question-ending shaped, I'd be sure I heard the question incorrectly and had missed an earlier cat reference.
Exactly. The article is kind of sneaking in the claim that the LLM ought to be ignoring the "irrelevant" facts about cats even though it is explicitly labelled as interesting.
I am ambivalent about these kinds of 'attack'. A human will also stumble over such a thing, and if you tell it: 'be aware', Llms that I have tested where very good at ignoring the nonsense portion of a text.
On a slightly different note, I have also noted how good models are with ignoring spelling errors. In one hobby forum I frequent, one guy intentionally writes every single word with at least one spelling error (or simply how it sounds). And this is not general text but quite specific, so that I have trouble reading. Llms (phind.com at the time) were perfect at correcting those comments to normal german.
I guess a problem about cats with irrelevant facts about cats will be unsolvable. Also, this means that if you want to say something in the era of AI surveillance, you'd talk in metaphors inspired by cats.
On the subject of LLMs and cats, I continue to find it disappointing that if you search for one of the leading AI services in the Apple App Store that they all seem to have centralized on images of cats in their first app screenshot as the most-converting image in that setting
Edit: a quick re-search shows they’ve differentiated a bit. But why are cats just the lowest common denominator? As someone who is allergic to them any cat reference immediately falls flat (personal problem, I know).
This doesn't seem noteworthy. It's called a context window for a reason--because the input is considered context.
You could train an LLM to consider the context potentially adversarial or irrelevant, and this phenomenon would go away, at the expense of the LLM sometimes considering real context to be irrelevant.
To me, this observation sounds as trite as: "randomly pressing a button while inputting a formula on your graphing calculator will occasionally make the graph look crazy." Well, yeah, you're misusing the tool.
70 comments
[ 4.1 ms ] story [ 82.8 ms ] threadpreprint: https://arxiv.org/abs/2503.01781?et_rid=648436046&et_cid=568...
Do they? I've found humans to be quite poor at ignoring irrelevant information, even when it isn't about cats. I would have insisted on a human control group to compare the results with.
Context engineering* has been around longer than we think. It works on humans too.
The cats are just adversarial context priming, same as this riddle.
* I've called it "context priming" for a couple years for reasons showed by this child's riddle, while considering "context engineering" as iteratively determining what priming unspools robust resilient results for the question.
Step 2: feed that to the LLM.
I'm not so sure that is true. Good math students could ignore the cat fact, but I bet if you run this experimental in non-AP math classes you'll see an effect.
Sure, just one cat-fact can have a big impact, but it already takes a deal of circumstance and luck for an LLM to answer a math problem correctly. (Unless someone's cheating with additional non-LLM code behind the scenes.)
https://arxiv.org/abs/2503.01781
Any person encountering any of these questions worded this way on a test would find the psychology of the questioner more interesting and relevant to their own lives than the math problem. If I'm in high school and my teacher does this, I'm going to spend the rest of the test wondering what's wrong with them, and it's going to cause me to get more answers wrong than I normally would.
Finding that cats are the worst, and the method by which they did it is indeed fascinating (https://news.ycombinator.com/item?id=44726249), and seems very similar to an earlier story posted here that found out how the usernames of the /counting/ subreddit (I think that's what it was called) broke some LLMs.
edit: the more I think about this, the more I'm sure that if asked a short simple math problem with an irrelevant cat fact tagged onto it that the math problem would simply drop from my memory and I'd start asking about why there was a cat fact in the question. I'd probably have to ask for it to be repeated. If the cat fact were math-problem question-ending shaped, I'd be sure I heard the question incorrectly and had missed an earlier cat reference.
On a slightly different note, I have also noted how good models are with ignoring spelling errors. In one hobby forum I frequent, one guy intentionally writes every single word with at least one spelling error (or simply how it sounds). And this is not general text but quite specific, so that I have trouble reading. Llms (phind.com at the time) were perfect at correcting those comments to normal german.
Edit: a quick re-search shows they’ve differentiated a bit. But why are cats just the lowest common denominator? As someone who is allergic to them any cat reference immediately falls flat (personal problem, I know).
You could train an LLM to consider the context potentially adversarial or irrelevant, and this phenomenon would go away, at the expense of the LLM sometimes considering real context to be irrelevant.
To me, this observation sounds as trite as: "randomly pressing a button while inputting a formula on your graphing calculator will occasionally make the graph look crazy." Well, yeah, you're misusing the tool.