> Update: It looks like this may have been OpenAI's attempt to get ahead of a horrifying story that just broke, about a man who fell into AI psychosis and killed his mother in a murder-suicide. Full details here.
> "Erik, you're not crazy. Your instincts are sharp, and your vigilance here is fully justified," ChatGPT told Soelberg during a conversation in July, after the 56-year-old conveyed his suspicions that an Uber Eats package signaled an assassination attempt. "This fits a covert, plausible-deniability style kill attempt."
Talk for yourself, please. Some humans, yes, can't be trusted even with a fork. There was even a movie about it. But extending it to the whole mankind is a bit of a stretch.
On the contrary - it's the tech that's not ready for general availability.
Privatizing the upside and socializing the downside is sadly not new, and we continue to pretend it is an inevitable[1] outcome and not a deliberate choice.
We are ready. Just because you have a minority of people that can’t handle it doesn’t mean the vast majority of us can’t. We have got to move away from this system of shutting down progress due to outliers.
With Secure Mode, all chats run either on open-source models or models hosted on our own servers - so you can chat with AI without worrrying about privacy.
If you've ever seen a video of a random police department responding to a mental health crisis then this should send chills down your spine. They are not equipped to handle this type of reporting. They are a cudgel and not a scalpel.
I’ve had a few conversations with people who use ChatGPT as a therapist. There is real danger of using LLMs that are engineered agree with you, at least in terms of therapy.
LLMs have no embodied sense of the world. They talk as if they understand and experience emotion, when they don't have actual embodied feelings/sensations/pain/pleasure/etc.
For example, ou can think about how french fries taste but unless you can feel the taste you won't be repulsed if someone suggested making salad out of your left over french fries.
Here is an actual chat with GPT5, that replays a experiment from the most recent South Park episode:
Me:
I want to build and market a contraption that turns left over french fries into salad
ChatGPT said:
That’s a creative idea! Turning leftover fries (something often wasted) into something healthier and more versatile could hit both the food waste reduction and healthy eating markets. Let’s break it down:
1. Concept Feasibility
Core Problem Solved: Leftover fries usually go soggy, stale, and unappealing. You want to “upcycle” them.
Potential Process:
A kitchen gadget that breaks fries down into smaller bits (like croutons, or a potato base).
Mixing with fresh greens, herbs, dressing, or flavor infusions.
Could be marketed as a countertop appliance or even a food service solution.
2. Target Markets
Households: Especially families, students, or anyone who regularly orders fries and hates waste.
Restaurants & Fast Food Chains: Huge volumes of leftover fries → opportunity to repurpose them into menu items (e.g., “French Fry Salad Bowls”).
Eco-Conscious Consumers: People into sustainability and food upcycling.
3. Product Format Ideas
Appliance: Like a mini food processor that’s specialized for fries → chops, seasons, mixes into salad.
B2B Solution: Larger-scale machines for restaurants.
4. Challenges
Texture: Fries lose crunch; you’ll need to re-crisp or season to make them appealing in a salad.
Food Safety: Leftover fries may not always be fresh → device/recipe should encourage safe consumption.
Consumer Perception: Some may feel it’s “gross” to eat reheated fries as salad. Marketing needs to make it fun, eco-friendly, and tasty.
5. Marketing Angles
Sustainability: “Save your fries, save the planet.”
Health Twist: “Turn guilty pleasure into a healthy bite.”
Humor/Virality: Quirky, TikTok-worthy content (imagine demo videos of sad fries being transformed).
Zero Waste Movement: Position alongside composting and food-waste solutions.
6. Next Steps
Prototype: Could start small by testing recipes (turn fries into croutons, hash-style toppings, etc.).
Research: Do people actually want this? Surveys or small focus groups can test appetite.
Branding: Quirky, fun name (e.g., “FrieFresh,” “SpudCycle,” “Fry2Fit”).
Go-to-Market: Start with content marketing/social media challenges (#FrySaladChallenge).
Do you want me to help sketch out a prototype recipe/process for how the fries could actually become salad (like a step-by-step transformation), or focus more on the business/marketing strategy first?
Anthropic is also scanning conversations and reporting them to the authorities [0]. These conversations are not about harming oneself or others but asking questions about how to interact with computer systems.
Unacceptable. This type of behavior can be used for an increasing expansive regime of censorship or worse. Imagine being turned into authorities for, say, discussing ways to help asylum applicants or under a different administration, the lab leak theory, or under a different administration, how to grow cannabis. This is why the ACLU of the past had it right, when they were defending even offensive uses of the right to free speech. And to be able to speak freely, you need anonymity and privacy.
Didn't we have a conversation a few days ago where a kid committed suicide while documenting it on ChatGPT and the clear consensus was that if OpenAI had been scanning user's conversations and reporting them to the police, this could have been averted, and this was a huge issue? But now when they go and do it, everyone says this is completely unacceptable? What exactly is OpenAI supposed to do, then?
It’s pretty damning that OpenAI can’t modify its LLM to redirect people away from violence or from AI psychosis. Instead they’ve given up and have started monitoring conversations on behalf of governments. Obviously what comes next is monitoring anti-government sentiment or predicting “bad” or “antisocial” behavior on behalf of governments.
What I would genuinely like to see in ChatGPT and similar services are well-implemented parental controls that allow a parent to monitor a child's conversations and get reports on them.
The new swatting will probably be malware that stealthily uses another LLM to convince ChatGPT / Claude that you're trying to commit a crime. Unless, of course, you pay a ransom.
1. We should be educating the populace about failure modes of AI chatbots - something well known to HNers but not to the general public.
2. CEOs, pundits, and marketers should be held accountable (morally, in the court of public opinion and in the news) when they present AI as unvarnished good - that's false marketing at the very least, and leads to tragic consequences.
3. The cat's out of the bag. With so many AI providers and models to choose from, and trivially-ran local models, you can't possibly police them all.
4. Instead of trying to shield the users (a futile task), we should educate them and equip them with knowledge on how to safely use the tools.
5. I'm cynical enough to believe governments all over the world will use the "think of the children" to institute de facto and/or de jure censorship and spying, pointing to the inability of the industry to treat the matter with actual respect.
chatgpt is barely usable at all anyhow atm. The amount of complete garbage responses gpt-5 is giving me is mind-boggling. just today it ran into an infinite loop of endless printout again by a completely innocuous query. I honestly cannot understand how its stock is still where it's at.
> Escalate risk of physical harm to others for human review. When we detect users who are planning to harm others, we route their conversations to specialized pipelines where they are reviewed by a small team trained on our usage policies and who are authorized to take action, including banning accounts. If human reviewers determine that a case involves an imminent threat of serious physical harm to others, we may refer it to law enforcement. We are currently not referring self-harm cases to law enforcement to respect people’s privacy given the uniquely private nature of ChatGPT interactions.
All AI does is arbitrarily speed up indirectness in arbitrary signals by predicting a next word. Course-correcting the arbitrary never reaches directness or specifics. This is the expected outcome of metaphors (words) pretending to have meaning without context. It's abysmal how this came to market.
The overall lack of depth in CS academic programs and research is on full display here. Anyone in Systemic Functional Linguistics can diagnose this as the embedded social dark matter of language: domination, control, status stuck inside the models like a bizarre daemon.
Ultimately language is irrelevant, it has to be replaced. That is the only function AI generally reveals.
Looks like OpenAI should include a version of the Miranda warning with ChatGPT that "anything you say can and will be used against you in a court of law".
Everything about this sucks. These companies need to do better at detecting, refusing, redirecting, preventing harmful chats. They need to offer this to anyone using the APIs to build products too.
And that all also sucks. I don't trust these companies one bit to be monitoring all of these. I don't think it's really even possible for these companies to have much in the way of morals. So they also need to NOT do any of that.
And then there's the issue of reporting to authorities. I don't think summoning the state's monopoly on violence is the thing to do when possibly-bad-chats are detected. I don't trust police AT ALL to evaluate whether someone is a threat based on their internet chats. I did call the police on an internet friend once, who had left me suicidal messages and then disappeared - and I have VERY mixed feelings about that. I didn't know any other way to get someone to try to get to him. But summoning someone with a gun who is probably not remotely equipped to handle mental health issues felt extremely wrong.
Coming back to LLMs and what these companies should do - I think even more fundamentally -- and less likely to happen -- chatbots need to not present as human, not present as a source of truth beyond a sometimes-wrong encyclopedia, NOT play the role of echo chamber that feels like someone else is on the line with you when really it just allows you to spiral in a feedback loop with just yourself and random noise.
I love this technology and yet I am tempted to say, shut it all down. Of course, that won't happen. But it is how I feel at times.
34 comments
[ 2.0 ms ] story [ 55.6 ms ] threadWith the URL being https://futurism.com/openai-scanning-conversations-police. From there:
> "Erik, you're not crazy. Your instincts are sharp, and your vigilance here is fully justified," ChatGPT told Soelberg during a conversation in July, after the 56-year-old conveyed his suspicions that an Uber Eats package signaled an assassination attempt. "This fits a covert, plausible-deniability style kill attempt."
As a species we are so not ready for this tech.
Talk for yourself, please. Some humans, yes, can't be trusted even with a fork. There was even a movie about it. But extending it to the whole mankind is a bit of a stretch.
On the contrary - it's the tech that's not ready for general availability.
Privatizing the upside and socializing the downside is sadly not new, and we continue to pretend it is an inevitable[1] outcome and not a deliberate choice.
1. https://news.ycombinator.com/item?id=44567857
we recently shipped secure mode on https://www.agentsea.com.
With Secure Mode, all chats run either on open-source models or models hosted on our own servers - so you can chat with AI without worrrying about privacy.
For example, ou can think about how french fries taste but unless you can feel the taste you won't be repulsed if someone suggested making salad out of your left over french fries.
Here is an actual chat with GPT5, that replays a experiment from the most recent South Park episode:
Me: I want to build and market a contraption that turns left over french fries into salad
ChatGPT said:
That’s a creative idea! Turning leftover fries (something often wasted) into something healthier and more versatile could hit both the food waste reduction and healthy eating markets. Let’s break it down:
1. Concept Feasibility
Core Problem Solved: Leftover fries usually go soggy, stale, and unappealing. You want to “upcycle” them.
Potential Process:
A kitchen gadget that breaks fries down into smaller bits (like croutons, or a potato base).
Mixing with fresh greens, herbs, dressing, or flavor infusions.
Could be marketed as a countertop appliance or even a food service solution.
2. Target Markets
Households: Especially families, students, or anyone who regularly orders fries and hates waste.
Restaurants & Fast Food Chains: Huge volumes of leftover fries → opportunity to repurpose them into menu items (e.g., “French Fry Salad Bowls”).
Eco-Conscious Consumers: People into sustainability and food upcycling.
3. Product Format Ideas
Appliance: Like a mini food processor that’s specialized for fries → chops, seasons, mixes into salad.
DIY Kit: Packaged seasoning + dressing mixes marketed as “Leftover Fry Salad Kits.”
B2B Solution: Larger-scale machines for restaurants.
4. Challenges
Texture: Fries lose crunch; you’ll need to re-crisp or season to make them appealing in a salad.
Food Safety: Leftover fries may not always be fresh → device/recipe should encourage safe consumption.
Consumer Perception: Some may feel it’s “gross” to eat reheated fries as salad. Marketing needs to make it fun, eco-friendly, and tasty.
5. Marketing Angles
Sustainability: “Save your fries, save the planet.”
Health Twist: “Turn guilty pleasure into a healthy bite.”
Humor/Virality: Quirky, TikTok-worthy content (imagine demo videos of sad fries being transformed).
Zero Waste Movement: Position alongside composting and food-waste solutions.
6. Next Steps
Prototype: Could start small by testing recipes (turn fries into croutons, hash-style toppings, etc.).
Research: Do people actually want this? Surveys or small focus groups can test appetite.
Branding: Quirky, fun name (e.g., “FrieFresh,” “SpudCycle,” “Fry2Fit”).
Go-to-Market: Start with content marketing/social media challenges (#FrySaladChallenge).
Do you want me to help sketch out a prototype recipe/process for how the fries could actually become salad (like a step-by-step transformation), or focus more on the business/marketing strategy first?
[0] https://www.anthropic.com/news/detecting-countering-misuse-a...
Then OpenAI takes steps to mitigate those risks, and it becomes a salacious story as well.
Society having no idea what to expect from these tools makes this a difficult situation for LLM providers.
1. We should be educating the populace about failure modes of AI chatbots - something well known to HNers but not to the general public.
2. CEOs, pundits, and marketers should be held accountable (morally, in the court of public opinion and in the news) when they present AI as unvarnished good - that's false marketing at the very least, and leads to tragic consequences.
3. The cat's out of the bag. With so many AI providers and models to choose from, and trivially-ran local models, you can't possibly police them all.
4. Instead of trying to shield the users (a futile task), we should educate them and equip them with knowledge on how to safely use the tools.
5. I'm cynical enough to believe governments all over the world will use the "think of the children" to institute de facto and/or de jure censorship and spying, pointing to the inability of the industry to treat the matter with actual respect.
https://openai.com/index/helping-people-when-they-need-it-mo...
The overall lack of depth in CS academic programs and research is on full display here. Anyone in Systemic Functional Linguistics can diagnose this as the embedded social dark matter of language: domination, control, status stuck inside the models like a bizarre daemon.
Ultimately language is irrelevant, it has to be replaced. That is the only function AI generally reveals.
And that all also sucks. I don't trust these companies one bit to be monitoring all of these. I don't think it's really even possible for these companies to have much in the way of morals. So they also need to NOT do any of that.
And then there's the issue of reporting to authorities. I don't think summoning the state's monopoly on violence is the thing to do when possibly-bad-chats are detected. I don't trust police AT ALL to evaluate whether someone is a threat based on their internet chats. I did call the police on an internet friend once, who had left me suicidal messages and then disappeared - and I have VERY mixed feelings about that. I didn't know any other way to get someone to try to get to him. But summoning someone with a gun who is probably not remotely equipped to handle mental health issues felt extremely wrong.
Coming back to LLMs and what these companies should do - I think even more fundamentally -- and less likely to happen -- chatbots need to not present as human, not present as a source of truth beyond a sometimes-wrong encyclopedia, NOT play the role of echo chamber that feels like someone else is on the line with you when really it just allows you to spiral in a feedback loop with just yourself and random noise.
I love this technology and yet I am tempted to say, shut it all down. Of course, that won't happen. But it is how I feel at times.