What I am unclear about is—, what did the participants know going in about the study? Are they aware you are dealing with email phishing or was it just the broad topic of AI?
TLDR: They generated some phishing emails using LLMs, they sent the emails to 108 elderly people who agreed to be in a study, 11% of the recipients clicked a link.
Generating a phishing email isn't very difficult to do, with or without an LLM, and claiming that because someone clicked on a link, they were "compromised" seems disingenuous.
More interesting to me, is using LLMs in multi-turn phishing correspondence with victims, the paper mentions this in the discussion, but it isn't something that they appear to have actually tested.
I’m confused - Whats the point of a study for this, for Pete's sake, this could have been a blog post - why does the AI world need a paper for everything?
We all know how easy it is to generate emails etc at a minimum level to fool the elderly and doing that via ai is not really “jailbreaking” anyway..
Doing a study is only to get PR value and its frustrating to me why the AI world has this cargo cult world of studies and fake benchmarks with every advancement a new academic paper instead of doing normal engineering…
My elderly mother is aware of AI yet she falls victim to it every time. I point out the obvious, the AI voice, the mouth not matching the words, etc and she just stares at me with disbelief that she was fooled (yet again) by AI. An hour later, she’s fooled again.
She also buys random stuff online because it’s on sale. Only because it’s on sale. She has no use for a home trampoline, a portable air conditioner, a men’s suit rack, or the dozens of Asian plates still in bubble wrap from a decade ago.
I work in marketing and I beg my father not to buy anything he sees in digital ads, especially not the garbage promoted in the word game app he likes. I tell him if he sees a product he wants in an ad to track down something similar himself from a store he already knows and trusts.
Hey, HN people - I’d pay good money for the opposite. My dad has fallen for two separate pig butcher scams. I’d love to install something on his phone to protect him, and an LLM could actually do that as long as it could integrate with whatsapp/messages/his email.
The latest one is ongoing and he doesn’t believe me, but if you could cut them off earlier before they’re hooked it would probably prevent it.
I can't promise how effective it is but Google Pixel phones have a built in scam detection feature in their dialer app. It uses a local LLM that runs on device and analyzes the phone conversation. Here's their support doc: https://support.google.com/phoneapp/answer/15654065
It likely only works for US English phones, and of course if they get him onto other platforms like whatsapp or signal then it's no help. Sorry to hear you're dealing with that it's a big fear of mine as my parents age.
Interesting measurements for something I've been tracking since 2018 (Google Duplex). When there is next to no cost for misuse of voice AI, it's profitable to fool people, the calls can be made from international jurisdictions, and there is no enforceability, you'll get these scams.
15 comments
[ 2.7 ms ] story [ 39.5 ms ] threadWe passed the majority/other shit with our guardrails but yeah
Generating a phishing email isn't very difficult to do, with or without an LLM, and claiming that because someone clicked on a link, they were "compromised" seems disingenuous.
More interesting to me, is using LLMs in multi-turn phishing correspondence with victims, the paper mentions this in the discussion, but it isn't something that they appear to have actually tested.
We all know how easy it is to generate emails etc at a minimum level to fool the elderly and doing that via ai is not really “jailbreaking” anyway..
Doing a study is only to get PR value and its frustrating to me why the AI world has this cargo cult world of studies and fake benchmarks with every advancement a new academic paper instead of doing normal engineering…
She also buys random stuff online because it’s on sale. Only because it’s on sale. She has no use for a home trampoline, a portable air conditioner, a men’s suit rack, or the dozens of Asian plates still in bubble wrap from a decade ago.
The latest one is ongoing and he doesn’t believe me, but if you could cut them off earlier before they’re hooked it would probably prevent it.
It likely only works for US English phones, and of course if they get him onto other platforms like whatsapp or signal then it's no help. Sorry to hear you're dealing with that it's a big fear of mine as my parents age.