Just connect everything folks, we'll proactively read everything, all the time, and you'll be a 10x human, trust us friends, just connect everything...
I'm immediately thinking of all the ways this could potentially affect people in negative ways.
- People who treat ChatGPT as a romantic interest will be far more hooked as it "initiates" conversations instead of just responding. It's not healthy to relate personally to a thing that has no real feelings or thoughts of its own. Mental health directly correlates to living in truth - that's the base axiom behind cognitive behavioral therapy.
- ChatGPT in general is addicting enough when it does nothing until you prompt it. But adding "ChatGPT found something interesting!" to phone notifications will make it unnecessarily consume far more attention.
- When it initiates conversations or brings things up without being prompted, people will all the more be tempted to falsely infer a person-like entity on the other end. Plausible-sounding conversations are already deceptive enough and prompt people to trust what it says far too much.
For most people, it's hard to remember that LLMs carry no personal responsibility or accountability for what they say, not even an emotional desire to appear a certain way to anyone. It's far too easy to infer all these traits to something that says stuff and grant it at least some trust accordingly. Humans are wired to relate through words, so LLMs are a significant vector to cause humans to respond relationally to a machine.
The more I use these tools, the more I think we should consciously value the output on its own merits (context-free), and no further. Data returned may be useful at times, but it carries zero authority (not even "a person said this", which normally is at least non-zero), until a person has personally verified it, including verifying sources, if needed (machine-driven validation also can count -- running a test suite, etc., depending on how good it is). That can be hard when our brains naturally value stuff more or less based on context (what or who created it, etc.), and when it's presented to us by what sounds like a person, and with their comments. "Build an HTML invoice for this list of services provided" is peak usefulness. But while queries like "I need some advice for this relationship" might surface some helpful starting points for further research, trusting what it says enough to do what it suggests can be incredibly harmful. Other people can understand your problems, and challenge you helpfully, in ways LLMs never will be able to.
Maybe we should lobby legislators to require AI vendors to say something like "Output carries zero authority and should not be trusted at all or acted upon without verification by qualified professionals or automated tests. You assume the full risk for any actions you take based on the output. [LLM name] is not a person and has no thoughts or feelings. Do not relate to it." The little "may make mistakes" disclaimer doesn't communicate the full gravity of the issue.
Google's edge obvious here is the deep integration it already has with calendar, apps, and chats and what not that lets them surface context-rich updates naturally. OpenAI doesn't have that same ecosystem lock-in yet, so to really compete they'll need to get more into those integrations. I think what it comes down to ultimately is that being "just a model" company isn't going to work. Intelligence itself will go to zero and it's a race to the bottom. OpenAI seemingly has no choice but to try to create higher-level experiences on top of their platform. TBD whether they'll succeed.
The very models they pioneered are far better at writing code for web than they are at any other domain, leveling the very playing field they're now finding they must compete on. Ironic.
I was wondering how they'd casually veer into social media and leverage their intelligence in a way that connects with the user. Like everyone else ITT, it seems like an incredibly sticky idea that leaves me feeling highly unsettled about individuals building any sense of deep emotions around ChatGPT.
At what point do you give up thinking and just let LLMs make all your decisions of where to eat, what gifts to buy and where to go on holiday? all of which are going to be biased.
My pulse today is just a mediocre rehash of prior conversations I’ve had on the platform.
I tried to ask GPT-5 pro the other day to just pick an ambitious project it wanted to work on, and I’d carry out whatever physical world tasks it needed me to, and all it did was just come up with project plans which were rehashes of my prior projects framed as its own.
I’m rapidly losing interest in all of these tools. It feels like blockchain again in a lot of weird ways. Both will stick around, but fall well short of the tulip mania VCs and tech leaders have pushed.
I’ve long contended that tech has lost any soulful vision of the future, it’s just tactical money making all the way down.
They’re more like synthesizers or sequencers: if you have ideas, they are amazing force multipliers, but if you don’t have ideas they certainly won’t create them for you.
Try turning off memory. I've done a lot of experiments and find ChatGPT is objectively better and more useful in most ways with no memory at all. While that may seem counter-intuitive, it makes sense the more you think about it:
(1) Memory is primarily designed to be addictive. It feels "magical" when it references things it knows about you. But that doesn't make it useful.
(2) Memory massively clogs the context window. Quality, accuracy, and independent thought all degrade rapidly with too much context -- especially low-quality context that you can't precisely control or even see.
(3) Memory makes ChatGPT more sychophantic than it already is. Before long, it's just an echo chamber that can border on insanity.
(4) Memory doesn't work the way you think it does. ChatGPT doesn't reference everything from all your chats. Rather, your chat history gets compressed into a few information-dense paragraphs. In other words, ChatGPT's memory is a low-resolution, often inaccurate distortion of all your prior chats. That distortion then becomes the basis of every single subsequent interaction you have.
Another tip is to avoid long conversations, as very long chats end up reproducing within themselves the same problems as above. Disable memory, get what you need out of a chat, move on. I find that this "brings back" a lot of the impressiveness of the early version of ChatGPT.
Oh, and always enable as much thinking as you can tolerate to wait on for each question. In my experience, less thinking = more sychophantic responses.
Replies you get tell me it is more akin to Agile movement at this point. People are mixing “lines of code per minute” with tangible results.
Are you more personally ‘productive’ if your agent can crunch out PoCs of your hobby projects at night when you sleep? Is doing more development iterations per month making your business more ‘productive’?
It is like expecting that achieving x10 more sunny side ups cooked per minute will make your restaurant more profitable. In reality amount of code delivered is rarely a bottleneck for value delivered, but for ‘productivity’ everyone has their own subjective definition.
Comparing this to blockchain and Web3 is fucking hilarious. Web3 literally has like no real use case, LLM tech has a billion. It literally passes the Turing test.
I think you're expecting too much from it? It's great at "interpolating" across its training data, but it can't really "extrapolate" (think of brand new, ambitious ideas). If you gave it some more constraints then it could definitely come up with an ambitious project for you, but it wouldn't be something entirely new to the world.
I see some pessimism in the comments here but honestly, this kind of product is something that would make me pay for ChatGPT again (I already pay for Claude, Gemini, Cursor, Perplexity, etc.).
At the risk of lock-in, a truly useful assistant is something I welcome, and I even find it strange that it didn't appear sooner.
158 comments
[ 4.0 ms ] story [ 146 ms ] threadBy their own definition, its a feature nobody asked for.
Also, this needs a cute/mocking name. How about "vibe living"?
- People who treat ChatGPT as a romantic interest will be far more hooked as it "initiates" conversations instead of just responding. It's not healthy to relate personally to a thing that has no real feelings or thoughts of its own. Mental health directly correlates to living in truth - that's the base axiom behind cognitive behavioral therapy.
- ChatGPT in general is addicting enough when it does nothing until you prompt it. But adding "ChatGPT found something interesting!" to phone notifications will make it unnecessarily consume far more attention.
- When it initiates conversations or brings things up without being prompted, people will all the more be tempted to falsely infer a person-like entity on the other end. Plausible-sounding conversations are already deceptive enough and prompt people to trust what it says far too much.
For most people, it's hard to remember that LLMs carry no personal responsibility or accountability for what they say, not even an emotional desire to appear a certain way to anyone. It's far too easy to infer all these traits to something that says stuff and grant it at least some trust accordingly. Humans are wired to relate through words, so LLMs are a significant vector to cause humans to respond relationally to a machine.
The more I use these tools, the more I think we should consciously value the output on its own merits (context-free), and no further. Data returned may be useful at times, but it carries zero authority (not even "a person said this", which normally is at least non-zero), until a person has personally verified it, including verifying sources, if needed (machine-driven validation also can count -- running a test suite, etc., depending on how good it is). That can be hard when our brains naturally value stuff more or less based on context (what or who created it, etc.), and when it's presented to us by what sounds like a person, and with their comments. "Build an HTML invoice for this list of services provided" is peak usefulness. But while queries like "I need some advice for this relationship" might surface some helpful starting points for further research, trusting what it says enough to do what it suggests can be incredibly harmful. Other people can understand your problems, and challenge you helpfully, in ways LLMs never will be able to.
Maybe we should lobby legislators to require AI vendors to say something like "Output carries zero authority and should not be trusted at all or acted upon without verification by qualified professionals or automated tests. You assume the full risk for any actions you take based on the output. [LLM name] is not a person and has no thoughts or feelings. Do not relate to it." The little "may make mistakes" disclaimer doesn't communicate the full gravity of the issue.
Edit: Downvote all you want, as usual. Then wait 6 months to be proven wrong. Every. Single. Time.
Or more likely: `[object, object]`
But a device that reaches out to you reminds you to hook back in.
This reads like the first step to "infinite scroll" AI echo chambers and next level surveillance capitalism.
On one hand this can be exciting. Following up with information from my recent deep dive would be cool.
On the other hand, I don't want to it to keep engaging with my most recent conspiracy theory/fringe deep dives.
I tried to ask GPT-5 pro the other day to just pick an ambitious project it wanted to work on, and I’d carry out whatever physical world tasks it needed me to, and all it did was just come up with project plans which were rehashes of my prior projects framed as its own.
I’m rapidly losing interest in all of these tools. It feels like blockchain again in a lot of weird ways. Both will stick around, but fall well short of the tulip mania VCs and tech leaders have pushed.
I’ve long contended that tech has lost any soulful vision of the future, it’s just tactical money making all the way down.
(1) Memory is primarily designed to be addictive. It feels "magical" when it references things it knows about you. But that doesn't make it useful.
(2) Memory massively clogs the context window. Quality, accuracy, and independent thought all degrade rapidly with too much context -- especially low-quality context that you can't precisely control or even see.
(3) Memory makes ChatGPT more sychophantic than it already is. Before long, it's just an echo chamber that can border on insanity.
(4) Memory doesn't work the way you think it does. ChatGPT doesn't reference everything from all your chats. Rather, your chat history gets compressed into a few information-dense paragraphs. In other words, ChatGPT's memory is a low-resolution, often inaccurate distortion of all your prior chats. That distortion then becomes the basis of every single subsequent interaction you have.
Another tip is to avoid long conversations, as very long chats end up reproducing within themselves the same problems as above. Disable memory, get what you need out of a chat, move on. I find that this "brings back" a lot of the impressiveness of the early version of ChatGPT.
Oh, and always enable as much thinking as you can tolerate to wait on for each question. In my experience, less thinking = more sychophantic responses.
Are you more personally ‘productive’ if your agent can crunch out PoCs of your hobby projects at night when you sleep? Is doing more development iterations per month making your business more ‘productive’?
It is like expecting that achieving x10 more sunny side ups cooked per minute will make your restaurant more profitable. In reality amount of code delivered is rarely a bottleneck for value delivered, but for ‘productivity’ everyone has their own subjective definition.
But well I guess they have committed 100s of billions of future usage so they better come up with more stuff to keep the wheels spinning.