HN Explorer

Tell HN: Google's new summarize SMS just read another person's messages to me

83 points by nogridbag ↗ HN
While I was driving, I received a text message from my father with 3-4 pictures and some of his typical dad jokes. Android Auto prompted me with a new feature "Summarize Messages". Usually I'm strongly against these features but curiosity got the better of me so I clicked "Turn on". Then I clicked "Summarize" on my father's message. It started giving me a detailed summary of names that are not in any of my contacts or my actual social circle, e.g. "Zach and Melanie are planning to visit NYC this summer". I'm not sure if it was truly another person's summary of some canned dev message. But now I'm a bit worried Google is giving other users my SMS history.

34 comments

[ 3.2 ms ] story [ 80.4 ms ] thread
[dead] lopkeny12ko
[flagged]
splatzone
Why do you think OP made this up?
nogridbag
Check my HN history. I've never posted anything of the sort. Or my follow-up comment with the embarrassing message I tried to have it summarize.
teaearlgraycold
It seems much more likely that this is a hallucination than a data breach.
xbar
Assuming the story is factually written, I think it is pretty short on evidence to make a conclusion either way.
nogridbag
I'm on mobile so I kept my message short. But it gave me a detailed itinerary of 4 different names I've never heard of including dinner plans with dates and times. I tried to replay the summary but there didn't seem to be any way to do so. Just now I've asked assistant to "summarize my messages" and it told me "that feature is not enabled on this device".
cjbprime
That doesn't provide evidence for or against it being another human's data. It could be e.g. synthetic data from its training set that it is repeating, LLMs like to do that.
SkyPuncher
To be fair, I think Google just rolled back a bunch of AI stuff.
[deleted]
(comment deleted)
HL33tibCe7
HL33t’s razor: never assume a security vulnerability for that which can adequately be explained by LLMs being shit and hallucinating
rmbyrro
Always consider with seriousness the possibility of a security vulnerability when it comes to a decadent, big-co, feeling threatened and adopting a "move desperately, break things" approach to swe.
aaomidi
Considering the layoffs, Google has lost a ton of very talented engineers and generational knowledge.

This failure mode of data leak is entirely possible with how Google operates

rmbyrro
We're also seeing increasing numbers of talented engineers getting tired of the culture of bureaucracy & ideology over shipping quality, leading many to quit Google.
dmsayer
Assistant doesn’t log messages or summaries, and the interactions aren’t used to train the Large Language Model (LLM).
nogridbag
In case anyone is skeptical of my post, the actual sms my father sent was a few pictures from a UFO festival in upstate NY (a stormtrooper, an alien, etc) with the following message: "Wish I could say that the UFO Festival was out of this world, but I would say it might have been closer to Uranus.". I was initially embarrassed to post that here :) But the additional context may be helpful.
cooperadymas
I too have been let down by that UFO Festival.

There is, however, also a Sasquatch Fest a little further north if your father is looking for some new material.

scop
That made me smile. As somebody who lost my dad over a decade ago, I can still really appreciate messages like that. A genre unto itself.
bravetraveler
+1, warmed my cold heart a little. I never had this relationship with my dad. Wish we did.

Thanks for sharing

hydrolox
could it be some sort of hallucination?
dataflow
Could it be from another app? Like Facebook or WhatsApp (especially chat groups)?
nogridbag
Nope I have one parent group on WhatsApp none of whom are named Zach or Melanie. I haven't logged in Facebook or messenger in a long time, but I checked the groups there and none have any messages resembling what the assistant read.
advael
Whether this is a breach or, as others have suggested, the LLM bullshitting you - for which I guess the politically correct term is still "hallucination" - it's bizarre to me that these tech companies who really should know better* continue the rushed and forced rollout of these "features" for use cases that are exactly the worst places to use them - where the information matters but a plausibly fluid prose kinda doesn't, and you're focused on something else meaning that factual errors and omissions aren't something you can check in the moment, but perhaps won't even think to check later because you have been led to believe you got the gist of the message while not able to check it

As someone with ADHD who could at times really benefit from a "personal assistant", the way tech companies are pushing these things sounds more like nightmare fuel than a godsend. And that's as someone who really does think there exist good use cases for these things, even as an increasingly vocal crowd not in the tech bubble is naysaying even that - also a predictable backlash to this godawful rollout

*Especially google. Like I watched them spend basically the entire time I was in grad school and many years after luring prominent professors in AI/ML with crazy salaries and their own labs

DragonMaus
A rushed product pushed out before it's ready is a very small price to pay in the endless race for ever increasing profits. The tech companies who "really should know better" do know better. It just isn't something they care about or prioritize at all.
advael
There is no issue with a company that someone's not gonna show up and say "Well of course they don't care about anything but profits" which in an environment where governments are systematically willing to backstop risky bets made with financial leverage, mostly means make whatever decisions align with the confidence or whims of the majority of an increasingly top-heavy pool of investment capital

I'm well aware and I agree. Systematically, this fundamentally means that no human being should extend any trust to any corporation in this economic environment, that no corporation can be trusted to build anything remotely resembling infrastructure (including any tool anyone will depend on for a period of time longer than one fiscal quarter). That no corporation can be trusted to keep any promise to anyone who is not a top-level investor

You are telling me, and I agree, that these large firms are dangerous animals that I can't reason with, that could decide to maul me at any time, and that my best bet is to limit my exposure to them as much as I can manage, moreso the larger they are. This means that it is in the interest of everyone who isn't themselves an oligarch to use every meager lever of power available to them to try to change this situation, up to and including killing these things when possible, because it is deeply difficult to avoid being under the power of these behemoths, and they do not and will never have your best interests in mind in any respect someone doesn't force them to

That's the logical conclusion of this thing people keep telling me is obvious as though that also means it's inevitable. But for some reason a lot of people who wanted to tell me how this all works seem either surprised or sometimes even hostile when I start talking about stuff like that

That said, the part they should "know better" about is that this use case makes no sense whatsoever for the technology as it actually works. I'd be really surprised if this kind of thing even positively impacted their stock valuation because it reeks of a flailing and incompetent vision for how to apply the technology. The most charitable I can be for a thing like this is to say "well they got caught up in the stupid hype and are just trying to throw an LLM into anything they can", which works for a lot of companies but not for one that's spent over a decade building world-class expertise and could be argued to have actually invented the specific underlying technology powering this hype wave. I get that the underlying motivation for any decision made by a company is that they think they will profit from it. This is tautological. I am saying that even from that perspective, this is conspicuously stupid in a way that only seems like it can inspire investor confidence if investors continue to buy hype that seems decreasingly tethered to reality

shreezus
A few months ago I built a custom GPT to summarize commits from a repository. It glitched out and started spewing details for unrelated private repos on the "OpenAI" account by OpenAI employees, and I assume it was just hallucinating.
WithinReason
Or, it was recalling the details of the OpenAI repos it was trained on.
schmitter
Did you happen to get a spam text at the same time?
earthtograndma
"I'm not sure if it was truly another person's summary" seems to be at odds with the headline.
nogridbag
Fair. I haven't used AI tools much. My understanding of hallucinations were the context typically matches the input, but details are wrong and the AI claims it's correct.

In this case, the input "pictures from a UFO festival and a Dad joke" have absolutely nothing to do with the summary: dinner plans for 8pm, Zach and Melanie's summer vacation trip to NYC, etc. so it sure felt like it was accessing another user's text messages. But as a developer, I also know it's possible it's mock data. Or as people here pointed out it may be simple hallucination and my understanding of that concept is not correct.

liampulles
Semi-related story: when I got my Pixel, I tested out the assistant by asking it to open YouTube every morning when I wake up. Every morning it has given me my wake-up message, then said "Opening YouTube", and done nothing.

A few months ago it changed. Now it gives me a wake up message, followed by "I don't understand" and does nothing. Every morning.

I've kept it as a reminder of what not to do with this technology.