Ask HN: Are people in tech inside an AI echo chamber?

398 points by freelanddev ↗ HN
I recently spoke with a friend who is not in the tech space and he hadn’t even heard of ChatGPT. He’s a millennial & a white collar worker and smart. I have had conversations with non-tech people about ChatGPT/AI, but not very frequently, which led me to think, are we just in an echo chamber? Not that this would be a bad thing, as we’re all quite aware that AI will play an increasing role in our lives (in & out of the office), but maybe AI mainstream adoption will take longer than we anticipate. What do you think?

754 comments

[ 3.3 ms ] story [ 335 ms ] thread
I have a friend who is a senior electronics engineer at a fairly large company. A few months ago, I mentioned ChatGPT in conversation and he had never heard of it. I was quite surprised since I've seen ChatGPT become a major topic of concern even outside tech among those in the humanities. I think the explanation is that since the concerns about LLM AI revolve around language use, the communities paying attention to it are not the same as what we normally categorize as tech.
Very much so. But also a hype wave. It's the successor the the big data wave, the ML wave, and countless others
Seems like the successor the crypto craze after that crashed. I'm expecting a wave of grifters fairly soon.
You haven't seen them already? The "AI Lawyer", all of the people trying to sell LLMs as search engines, and just generally hundreds of projects that are outright dangerous uses of LLMs but seem like they might be feasible.
(comment deleted)
What I love is the same guys that were pushing crypto are now big into AI. The true entrepreneurs.
I see that comparison pop up quite often, and I really don't see the connection.

Cryptocurrencies have tried since 2009 to present some kind of problem to which they are supposedly the solution.

The problems that generative ML models can solve are pretty clear.

> maybe AI mainstream adoption will take longer than we anticipate

I mean, I'm in "tech", and I don't anticipate it happening soon. People want something they can trust, and current solutions are nowhere near that.

Students want someone to do writing for them, and they only have to do a trivial amount of vetting.

Now you can hand Lorem Ipsum in for homework!

That probably speaks more to the quality of our homework assignments than the quality of the student.

In this day and age, why is our education still based on rote regurgitation?

I agree. They should be chatting to ChatGPT instead.
They would probably learn more, honestly.
There was a fun example from a history lecturer a while back. Did you know that the Romans developed special anti-elephant crossbows to fight Hannibal? Because ChatGPT knows that. I think people who rely on this sort of thing for essay writing may be in for a rude awakening, especially when they have to write a thesis.
I saw other waves and this one is different. This is the first time we can unlock increasing cognitive capabilities by just basically letting capital grow its own ai. It's at the level of like invention of internet or currency or printing press or writing or maybe even human language.
I don’t see any evidence that it’s there yet. I do see that potential if AI were to improve dramatically, but I think we’re tricking ourselves into thinking it’s already here because what we’re seeing is so compelling.

The reality as far as I can tell is that we still have an incredible distance to go to have anything comparable to unlocking spoken or written language.

I guess it's your right to believe that as long as you want, the evidence is that it's beating people at every exam like AP exams and GRE exams and bar exams and medical certification exams and not only like regurgitating facts but it's also superhuman level in theory of mind. And all that comes from just an architecture that reads every text and uses a lot of GPU power.
Yeah right, it can answer written form exams that are similar to previous ones, and are expressed in formulaic ways. That's impressive, don't get me wrong, but it's also already clear that ChatGPT makes for a terrible lawyer (inventing cases looks bad) and literally can't act as a doctor cause it doesn't have, like, a body, or a sense of empathy. It's like saying that it got the theory part of a driving test correct so now we have self driving figured out.
I think these things are amazing, but I don’t think they’re comprehensively intelligent nor do I think we can easily reproduce this spark in other domains of intelligence.

When we can, we will be creating something superhuman in the truest sense. What we have now isn’t that at all. I think it appears magical because we’re so enthralled by language. It’s a major interface into our world and we’re extremely stimulated by it. It conveys so much meaning to us with so little. In fact, when it conveys too little meaning we begin to search for it and fill in the gaps! Our brains are extremely eager to engage with language.

I’m excited about this stuff, but what the models are doing isn’t as incredible as all of that seems.

I've had plenty of non tech people talk about chat gpt.

Especially college students

This isn't just a tech echo chamber, it's the YC/SV echo chamber. Nobody is seriously looking at AI for normal products, because it's not trustworthy. Never really will be. Why do you think subways still have human operators? The combined investment & risk of full automation aren't worth it. It'll be useful for customer support bots, and backend things like spam filtering and other statistical analysis. But otherwise it's a flash in the pan.
That's been said about countless tasks that are now done by machines. Computer was a job title. Printer was a job title. Very important jobs that didn't have room to risk errors.
Not sure why this is downvoted but I feel the SV bubble is its own bubble within the tech bubble without a doubt. And the biggest issue is that most of my friends in it seem to think everyone lives like them.
It's an incredible tool for augmenting what humans already do. It shifts your role from grunt to output supervisor, imperative to declarative.
> Why do you think subways still have human operators?

They don’t always. This sucked the wind out of your whole argument. https://en.wikipedia.org/wiki/List_of_driver-less_train_syst...

Driverless trains doesn't mean no drivers. It means remote drivers.

The DLR for example still has drivers, its just they are located somewhere other than the train.

No, it doesn't mean "remote drivers". Yes, an automated network is going to have people monitoring it and the ability of remote control for emergencies or failures, but that's not the same as "remote drivers" for normal operations.

(in the case of DLR, the staff on the train can take control if needed, but other networks don't have staff on every train)

I know this is getting into "technically correct" territory, but I think its reasonably important to qualify. As we are getting into AI automates <Hardthing> when it turns out that actually <Hardthing> was largely automated already.

For the tube, places like the victoria, jubilee, the driver doesn't actually "drive" if that makes any sense. They open the doors and hold down a button that indicates the line is clear. The driver has no real control over speed under normal circumstances.

for the tubes, the real blocker to "Driverless" trains is re-boring the tunnel to allow a walkway for evacuation.

The DLR is centrally controlled for most of the time. In the sense that there is a central operator that opens and closes the doors and tells the train to move to the next station.

the only difference is that there is no requirement for someone to hold down a button for the train to continue, the deadmans switch as it were.

> In the sense that there is a central operator that opens and closes the doors and tells the train to move to the next station.

Is it that remote controlled? Other systems I'm familiar with don't need a human for that.

> As we are getting into AI automates <Hardthing> when it turns out that actually <Hardthing> was largely automated already.

Oh, I fully agree with that point. You don't need AI-hype to automate a subway system, it's in many ways in the real of "classic" industrial automation.

It really didn't affect their argument.
It's not just about it being trustworthy but about quality.

AI looks to be great for ideation and development. But it's not there if you value a high quality output.

> AI looks to be great for ideation and development.

Agreed. I don't see this current version of AI as a calculator, but as an adaptable companion to work ideas with. The calculator side will continue to get better, but there's already so much value in the ideation side. Obvious is augmenting writing. The other I have found is in planning. If we expand to images, there's things like Firefly.

Getting an email just right is so much easier/faster now. It's increased my productivity from that one thing alone. Coding is also very helpful, and while not always 100%, it presents ideas which then I can use to get to the solution.

This is the first time I've seen a fairly straight line to a Star Trek like computer assistant at some point in the future.

If you see specific aspects of a chat reply where quality is bad, tell it.

It will improve the answer.

Iterating this you can get way beyond the quality of the first answer. You have full control over the quality but you have to bring something to the collaboration.

That's part of the problem. People think that using their own human bias to alter the machine is a good thing, when it's clear that human bias leads to negative outcomes. You need a separate thing to remove the human input bias.

Quality is a complex thing that requires design, safeguards, process, as well as iteration. You can't design test suites or checklists with just AI, because you don't know if the AI will decide to override them because of some other instruction or signal. And how do you determine the quality of the signal anyway? The quality of input of each human varies.

It's "too adaptive" for QA. You would need a QA for the QA.

Human bias is a whole huge area apart from just quality, not sure we want to get into that. I mean is it human bias, or is it reality bias. e.g. Poland hasn't had any terrorist attacks (or very few?), is that human bias, or is it reality?

I definitely don't treat its output with full trust, but I've been pleasantly surprised that even when I give it bad or incorrect guidance (unintentionally) it has caught my mistake, corrected me, and I've learned things.

For the QA case, I suppose what you're getting at is that if it can't be fully trusted, you might get incorrect QA results -- false negatives, false positives -- I'd agree but I think you just have to find an effective way to use the tool. Perhaps the obvious way most people would want to use the tool, is not in fact the best way to use the tool.

But just because the tool isn't delivering the perfection people hope for, doesn't mean there isn't some other way to use it that still catches (some) mistakes and adds value.

Subways have human operators for difference reasons not because trains cannot be fully automated. Some are partially automated because they run on old infrastructure which is difficult and expensive to revamp. Others are fully automated but regulations prohibit full autonomy, or labor is unionized and operators are guaranteed work. Automation is not AI anyway, automation today is done through control systems, that is sensors, some type of control like a PID loop (or more advanced Level 2 models) and devices that allow for controlled movement/positioning. There's no artificial intelligence there.
I recently went to Cleveland to visit a friend who has nothing to do with technology. Him and his girlfriend had heard of ChatGPT and she uses it at work. They work in chemical engineering and health. I took that as a sign that it has gone mainstream.
Yes. There's still a lot of people who aren't that aware of this stuff, and frankly, its because it still doesn't have much practical use.

Its making headlines left and right, and businesses are all trying to figure out what this stuff does, but if you're not watching much news and not in the tech side of business you probably don't know or care?

It has unbelievable practical use. ChatGPT is a personal tutor on any subject you can think of. It's more that 90% of people can't even be bothered to google the answer to a question they have, despite having a personal computer and oracle in their front pocket.

I'm using an LLM to teach me interactively how LLMs work and how to integrate LLMs into our products. It's replaced 90% of my googling/stack overflow. Every engineer in our company is using Copilot and ChatGPT to write software.

How can it replace 90% of your Googling if you have to go check everything it tells you afterward?
Our blind spot in vetting information is kind of like our blind spot when looking for something we have lost. We don't double check what we are sure of. To me, that is the most serious challenge when trying to learn using LLMs.
It's not like Googling typically gives you truthful answers either.
The difference to me is that with search you can explicitly search only in trusted sources (with e.g. site:...) if you choose to and you get transparency about where the data is coming from.

New Bing is a slight improvement in that you get some transparency about where it's getting information from and can tell it to prioritise accuracy, but can't actually explicitly tell it to use a subset of the web that you personally trust.

I have yet to see someone apply ChatGPT or other LLM in this way and not regret it or ditch it later.

Honestly i'm more worried about the ones who will use it this way and assume it's always correct. It is too often incorrect for anything I've ever done with it.

> i'm more worried about the ones who will use it this way and assume it's always correct

There are plenty of folks who don't fall into this trap.

Example: Me. I am fully aware that LLM generated code can be wrong or worse, contain disastrous errors. But being aware of that, and looking for the things I know from experience it commonly gets wrong, allows me to use it as an incredibly powerful tool in my workflow.

I guess it’s a matter of taste, I’d find that fucking horrible.
Why would that be any more horrible than handing routine tasks and boilerplate to a junior and having to review his PR?

With the junior, I need a meeting.

With the LLM, I just let it redo.

I know a lot of people who are using ChatGPT who aren't in tech.

My spouse works in early childhood education and they use ChatpGPT routinely for low-value boilerplate stuff (social media posts that no one reads, etc).

A relative is in commercial real-estate management, and they also use ChatGPT routinely (in fact they started using it before I did).

So I don't think it's an echo chamber.

My partner runs an ecommerce business and her team uses ChatGPT dozens of time a day.

Everything from writing emails to suppliers, correcting grammar, responding to customers, brainstorming new product ideas, explaining contracts etc.

Where as for me I can't find a use for it. Even as a coding assistant I find that I spent more time trying to understand/correct what it did than if I just wrote it myself.

The majority of my usage of ChatGPT as a dev has been to synthesize examples of APIs/tools/etc in usage when those aren't easy to find and/or when documentation is sparse/scarce.
Same to me so far, synthesise or summarise examples of tools I'm not familiar with to then explore the documentation.

Lately I've been using it quite a bit for Arduino on a ESP32 board, I had toyed around with Arduino previously but since I got this board for a small hobby project it's been great to ask ChatGPT to generate some examples of the kind of data I want to read from a few sensors. Even when it hallucinates something wrong it's been helpful for my learning.

Another way I've been using ChatGPT is to be a personal tutor to correct me when learning foreign languages, it's pretty easy to create a prompt asking ChatGPT to have a conversation with me in a given language while correcting any mistakes it believes I made, I've been getting feedback from these corrections from some native speakers and so far haven't got any case of "this is absurd and wrong", unsure why it works so well to correct my broken grammar but it does and without a fault.

I have noticed that there is a decent amount of regional (esp Salvadoran) slang that native Spanish speakers I know use that it doesn't recognize. This isn't a huge problem and it is still incredibly useful though, as learning a few random slang words isn't exactly a challenge.
As a supplier, I would be furious if one of my partners was sending AI emails to me.
I’m a software engineer and I use it everyday. I haven’t gone to StackOverflow in months. I use it like interactive documentation. I don’t even have to leave the terminal to use it as I have a cli tool called shell_gpt.
Those are people talking to echo chambers.
I've found that any non-techie I know who uses ChatGPT uses it in a similar way, and for quite a bit of stuff. The ones that don't usually haven't really used it much, if at all.

But I still feel most of them are in the 'google but nicer' stage.

My kids and all their friends are crazy about AI. They’re bent on making it do as much of their work as possible.

It’s fascinating. Though it probably isn’t intentional, AI service providers are already hooking kids early to have customers later.

Or they're hooking kids because they're least likely to have the domain experience required to recognize how confidently wrong AI can be.
Well, I wonder what’s wrong more often: GPT or a 13 year old :)
Bet a 13 year old with total confidence in GPT beats them both individually on any subject of reasonable complexity.
That’s the thing. If a kid approaches learning with a language model responsibly, they stand to learn a lot very quickly and solve difficult problems that would otherwise be next to impossible for them.

The thing is, we need to teach them that today rather than tell them it’s cheating and try to catch them using it on essays and deal some kind of consequence.

I now use it professionally fairly regularly and it’s an easily justified expense. I’ve already delivered things to clients faster because of it. Most recently I reasoned through prototyping a sort of minimal CMS experience using a self hosted CMS API connected to Next.JS, and had a viable plan and prototype at the proposal stage in as much time as I’d normally just do the research on something like this.

If it’s feasible to accelerate learning and research for real world work, I think we should seriously consider how it integrates with education rather than encourage kids to avoid it entirely. Of course, we don’t have that awareness in our education workforce in Canada, but I wonder if it’s harmful to discourage the use entirely rather than accepting it and ensuring kids are still producing the work that’s expected. If it’s clearly GPT regurgitation with hallucinations and no bibliography, the kid has still failed to deliver. If they manage to do their work faster with technology (the main difference here is that they haven’t googled bunch of stuff, frankly) then great, they’re still learning something.

And of course, the more you tell kids not to use it, the more they’ll want to (which I’ve come to love, honestly).

> What do you think?

What do you think?

(meaning "yes")

Young people all know about AI. There are 24/7 live AI generated SpongeBob shows on TikTok, character.ai is immensely popular, ChatGPT is used by students for homework and stuff, Snapchat AI was all posted all over (mostly memes about how bad and out of touch it is), I get ads for AI anime girl generators, etc.

I think there is certainly more talk about it inside tech circles, but I feel like it's more of a generational divide than anything else.

All my students us it a lot. We use it a lot in the office - we have a lot of digital paperwork to do, ChatGPT is fine at doing it all.

It can write lectures, generate worksheets, come up with interesting quizzes, D&D stuff.

It's just a tool, some people will find it useful, some won't. A bit like a chainsaw.

AI discussion has definitely made it mainstream and you can corroborate that just by looking at things like the ongoing Writer's Guild strike where AI generated content is a huge sticking point. That goes doubly for the Acting Guild which looks increasingly likely to strike as well.
This American Life and Planet Money have in recent months covered chat-gpt, though that audience may be skewed.
I think the AI space is getting broad attention from the public, largely due to generative AI. However, I think this is a bit of a double-edged sword. AI has a trust problem - people know it lies to you. It lies to me all the time. Convincingly! This is hurting people's initial impressions and I suspect that we're going to lose traction for a while before we once again re-gain it as the trust problem is solved. Time will tell.
There's definitely a few echo chambers around AI, but it's definitely not something that "just techies" are onto.

ChatGPT made some waves at the end of last year. My in-laws were wanting to talk to (at) me about it at Christmas. There's plenty of awareness outside of the tech circles, but most of the discussion (both out and in of the tech world) seems to miss what LLMs actually _are_.

The reason why ChatGPT was impressive to me wasn't the "realism" of the responses... It was how quickly it could classify and chain inputs/outputs. It's super impressive tech, but like... It's not AI. As accurate as it may ever seem, it's simply not actually aware of what it's saying. "Hallucinations" is a fun term, but it's not hallucinating information, it's just guessing at the next token to write because that's all it ever does.

If it was "intelligent" it would be able to recognise a limitation in its knowledge and _not_ hallucinate information. But it can't. Because it doesn't know anything. Correct answers are just as hallucinatory as incorrect answers because it's the exact same mechanism that produces them - there's just better probabilities.

AlphaGo is not aware that it’s playing a game either, but it’s better than humans at it. Awareness is not necessary to make people lose their jobs.
> Awareness is not necessary

Wasn't it the plot of a sci-fi novel by Vernor Vinge or someone at least as popular?

You might be thinking of Blindsight by Peter Watts. Great book.
I don't really know anything about AlphaGo. There's more types of "AI" than LLMs, but that's not really the point. You don't need AI for people to lose their jobs... but nobody is losing their jobs to AlphaGo, and in the grand scheme of things it's unlikely that people are going to lose their jobs to GPT, too.
If you make people who produce text 25% more productive you can fire one in four and increase your profits.
In your opinion, how does the "hallucination" issue differ from the same behaviour we see in humans?

I don't claim or believe that any LLM is actually intelligent. It just seems that we (at least on an individual basis) can also meet the criteria outlined above. I know plenty of people who are confidently incorrect and appear unwilling to learn or accept their own limitations, myself included.

In my opinion, even if we did have AGI it would still exhibit a lot of our foibles given that we'd be the only ones teaching it.

Yeah I've never gotten this argument at all. "Humans aren't actually intelligent they're just machines designed to optimize their probability of reproducing "
Yes, you may be. But you still have an internal world model - through conditioning or otherwise that you're playing off against.

An LLM doesn't have that. It's very impressive parlour trick (and of course a lot more), but it's use is hence limited (albeit massive) to that.

Chaining and context assists resolving that to some extent, but it's a limited extent.

That's the argument anyway, that doesn't mean it's not incredibly impressive, but comparing it to human self-awareness, however small, isn't a fair comparison.

It's next token prediction, which is why it does classification so well.

So humans have a level of knowledge, understanding, and reasoning ability that LLMs simply don't have. I'm writing a response to you right now, and I "know" a certain amount of information about the world. That knowledge has limits, and I can expand it, I can forget it, all sorts of things...

"Hallucination" is a term that works well for actual intelligence - when you "know" something that isn't true, and has no path of reasoning, you might have hallucinated the base "knowledge".

But that doesn't really work for LLMs, because there's no knowledge at all. All they're doing is picking the next most likely token based on the probabilities. If you interrogate something that the training data covers thoroughly, you'll get something that is "correct", and that's to be expected because there's a lot of probabilities pointing to the "next token" being the right one... but as you get to the edge of the training data, the "next token" is less likely to be correct.

As a thought experiment, imagine that you're given a book with every possible or likely sequence of coloured circles, triangles, and squares. None of them have meaning to you, they're just colours and shapes that are in random seeming sequences, but there's a frequency to them. "Red circle, blue square, gren triangle" is a much more common sequence than "red circle, blue square, black triangle", so if someone hands you a piece of paper with "red circle, blue square", you can reasonably guess that what they want back is a green triangle.

Expand the model a bit more, and you notice that "rc bs gt" is pretty common, but if there's a yellow square a few symbols before with anything in between, then the triangle is usually black. Thus the response to the sequence "red circle, blue square" is usually "green triangle", but "black circle, yellow square, grey circle, red circle, blue square" is modified by the yellow square, and the response is "black triangle"... but you still don't know what any of these things _mean_.

When you get to a sequence that isn't covered directly by the training data, you just follow the process with the information that you _do_ have. You get "red triangle, blue square" and while you've not encountered that sequence before, "green" _usually_ comes after "red, blue", and "circle" is _usually_ grouped with "triangle, square", so a reasonable response is "green circle"... but we don't know, we're just guessing based on what we've seen.

That's the thing... the process is exactly the same whether the sequence has been seen before or not. You're not _hallucinating_ the green circle, you're just picking based on probabilities. LLMs are doing effectively this, but at massive scale with an unthinkably large dataset as training data. Because there's so much data of _humans talking to other humans_, ChatGPT has a lot of probabilities that make human-sounding responses...

It's not an easy concept to get across, but there's a fundamental difference between "knowing a thing and being able to discuss it" and "picking the next token based on the probabilities gleaned from inspecting terabytes of text, without understanding what any single token means"

(comment deleted)
What you're describing is very close to what the thought experiment of Chinese Room (https://en.wikipedia.org/wiki/Chinese_room).

But yes, it's unfortunate that when the next tokens are joined token and laid out in the form of a sentence it appears "intelligent" to people. However if you instead lay out the individual probabilities of each token instead then it'll be more obvious what ChatGPT/LLMs actually do.

Yep, the "chinese room" is the classic thought experiment, but I feel like it fails to get the point across because the characters still represent language, so you could conceivably "learn" the language. I prefer the idea of symbols that aren't inherently language, as it really nails in the idea that it doesn't matter how long you spend, there's not something that you can ever learn to "speak" fluently.
What do you think your brain does when deciding the next word to speak? It is scoring words based on the appropriateness considering context and all the relevant known facts, as well as your communicative intent. But it is not obvious that there is nothing like communicative intent in LLMs. When you prompt it, you are engaging some subset of the network relevant to the prompt that induces a generative state disposed to produce a contextually appropriate response. But the properties of this "disposition to contextually appropriate responses" is sensitive to the context. In a Q&A context, the disposition is to produce an acceptable answer, in a therapeutic context, the disposition is to produce a helpful or sensitive response. The point is that communicative intent is within the solution space of text prediction when the training data was produced with communicative intent. We should expect communicative intent to improve the quality of text prediction, and so we cannot rule out that LLMs have recovered something in the ballpark of communicative intent.
> What do you think your brain does when deciding the next word to speak? It is scoring words based on the appropriateness considering context and all the relevant known facts

I mean, it's not. It's visualizing concepts internally and then using a grammar model to turn those into speech.

>It's visualizing concepts internally and then using a grammar model to turn those into speech.

First off, not everyone "visualizes" thought. Second, what do you think "using a grammar model to turn those into speech" actually consists of? Grammar is the set of rules by which sequences of words are mapped to meaning and vice-versa. But this is implemented mechanistically in terms of higher activation for some words and lower activation for other words. One such mechanism is scoring each word explicitly. Brains may avoid explicitly scoring irrelevant words, but that's just an implementation detail. All such mechanisms are computationally equivalent.

> I'm writing a response to you right now, and I "know" a certain amount of information about the world.

How do you know? And more importantly, how do you prove it to others? The only way to prove it is to say: "OK, you are human, I am human, each of us know this is true for ourselves, let's be nice and assume it's true for each other as well".

> But that doesn't really work for LLMs, because there's no knowledge at all.

How do you know? I know your argument saying that the LLM "is just" guessing probabilities, but surely, if the LLM can complete the sentence "The Harry Potter book series was written by ", the knowledge is encoded in its sea of parameters and probabilities, right?

Asserting that it does not know things is pretty absurd. You're conflating "knowledge" with the "feeling" of knowing things, or the ability to introspect one's knowledge and thoughts.

> As a thought experiment, imagine that you're given a book with every possible or likely sequence of coloured circles, triangles, and squares.

I'd argue thought experiments are pretty useless here. The smaller models are quantitatively different from the larger models, at least from a functional perspective. GPT with hundreds of parameters may be very similar to the one you're describing in your thought experiment, but it's well known that GPT models with billions of parameters have emergent properties that make them exhibit much more human-like behavior.

Does your thought experiment scale to hundreds of thousands of tokens, and billions of parameters?

Also, as with the Chinese Room argument, the problem is that you're asserting the computer, the GPU, the bare metal does not understand anything. Just like how our brain cells don't understand anything either. It's _humans_ that are intelligent, it's _humans_ that feel and know things. Your thought experiment would have the human _emulate_ the bare metal layer, but nobody said that layer was intelligent in the first place. Intelligence is a property of the _whole system_ (whether humans or GPT), and apparently once you get enough "neurons" the behavior is somewhat emergent. The fact that you can reductively break down GPT and show that each individual component is not intelligent does not imply the whole system is not intelligent -- you can similarly reductively break down the brain into neurons, cells, even atoms, and they aren't intelligent at all. We don't even know where our intelligence resides, and it's one of the greatest mysteries.

Imagine trying to convince an alien species that humans are actually intelligent and sentient. Aliens opens a human brain and looks inside: "Yeah I know these. Cells. They're just little biological machines optimized for reproduction. You say humans are intelligent? But your brains are just cleverly organized cells that handles electric signals. I don't see anything intelligent about that. Unlike us, we have silicon-based biology, which is _obviously_ intelligent."

You sound like that alien.

You can figure out if someone knows what they’re talking about or not by asking them questions about a subject. A bullshitter will come up with plausible answers; an honest person will say they don’t know.

ChatGPT isn’t even a bullshitter when it hallucinates – it simply does not know when to stop. It has no conceptual model that guides its output. It parrots words but does not know things.

(Unless you're intentionally going on a tangent --)

The discussion is whether LLMs have "knowledge, understanding, and reasoning ability" like humans do.

Your reply suggests that a bullshitter has the same cognitive abilities as an LLM, which seems to validate that LLMs are on-par with some humans. The claim that "it simply does not know when to stop" is wrong (it does stop, of course, it has a token limit -- human bullshitters don't). The claim that "It has no conceptual model that guides its output." is just an assertion. "It parrots words but does not know things." is just begging the question.

Lots of assertions without back up. Thanks for your opinion, I guess?

"Picking the most likely token based on probabilities" doesn't accurately describe their architecture. They are not intrinsically statistical, they are fully deterministic. The next word is scored (and then normalized to give something interpretable as a probability), But the calculation performed to determine the score for the next token considers the full context window and features therein, while leveraging the meaning of the terms by way of semantic embeddings and its trained knowledge base. It is not obvious that the network does not engage with the meaning of the terms in the context window when scoring the next word, and it certainly can't be dismissed by characterizing it as just engaging with probabilities. There is reason to believe that the network does understand to some degree in some cases. I go into some detail here: https://www.reddit.com/r/naturalism/comments/1236vzf/on_larg...
> how does the "hallucination" issue differ from the same behaviour we see in humans?

In humans “hallucination” means observing false inputs. In GTP it means creating false outputs.

Completely different with massively different connotations.

That's kind of the point, but also kind of not.

GPT isn't making true or false outputs. It's just making outputs. The truthiness or falseness of any output is irrelevant because it has no concept of true or false. We're assigning those values to the outputs ourselves, but like... it doesn't know the difference.

It's like blaming a die for a high or a low roll - it's just doing rolls. It has no knowledge of a good or a bad roll. GPT is like a Rube Goldberg machine for rolling dice that's _more likely_ to roll the number that you want, but really it's just rolling dice.

You could apply the same logic to humans.

Whenever a human speaks, it's just vibrations of wave molecules, triggered by the mouth and throat, which in turn are controlled by electric signals in the human's neural network. Those neurons, they just make muscles move. They don't have any concept of true of false. At least nobody has found a "true of false" neuron in the brain.

all of it coheres to consciousness, we know what it's like to be a human, but I think it'd be hubris to think we've cracked the code and made a blueprint of anything other than a word calculator
Hubris goes both ways. It is also hubris to assume our intelligence is special, instead of a boring neural network with sufficient number of neurons that exhibit emergent properties.
There's probably more dimensions to hubris but typically I understand it as flying too close to the sun, the other way for me is humility.
It’s more than next-word prediction though. The supervised fine tuning and RLHF steps are ways to possibly train it to favor truthful answers. Not sure whether this is currently the emphasis of ChatGPT though…
> It's just making outputs.

Yeah, one way to conceive of the issue is that GPT doesn't know when to shut up. Intuitively, you can kind of understand how this might be the case: the training data reflects when someone did produce output, not when they didn't, which is going to bias strongly toward producing confident output.

A lot of the conversation about GPT hallucinations has felt like an extended rehash of the conversations we've been having out the difference between plausible and accurate machine translations since like, 2016ish.

Great point, perhaps “confabulation” is a better way of describing it, which means “the replacement of a gap in a person's memory by a falsification that they believe to be true”. For example, the term is sometime used to describe dementia patients, who might wander somewhere and forget how they got there. The patient then might confabulate a story about why they are there, e.g. they were getting their keys so they could drive to the store to run an errand, despite the fact they no longer have a car.
> In humans “hallucination” means observing false inputs.

How do you know that? You can only observe the output of the humans (other than yourself).

A person can hallucinate under the effects of drugs or mental disorder and then tell you about it after they've recovered from it.

This experience is available to you and is well documented.

How do you know they are observing false inputs, as opposed to creating false outputs? (acting as if they have seen halucinations)

How do you know that the LLM is not observing false inputs but creating false outputs? Would an LLM which tells you very convincingly about how it obtained a false information make you change your mind?

> This experience is available to you and is well documented.

You are misunderstanding what I'm asking. Sure, drug induced hallucinations in humans is very well documented. What I'm asking if this purported difference between "hallucinating on the inputs" vs "creating false outputs" is meaningful distinction.

> In your opinion, how does the "hallucination" issue differ from the same behaviour we see in humans?

I feel like if you have any belief in philosophy then LLMs can only be interpreted as a parlour trick (on steroids). Perhaps we are fanciful in believing we are something greater than LLMs but there is the idea that we respond using rhetoric based on trying to find reason within in what we have learned and observed. From my primitive understanding, LLMs rhetoric and reasoning is entirely implied based on an effectively (compared to the limitations of human capacity to store information) infinite amount of knowledge they've consumed.

I think if LLMs were equivalent to human thinking then we'd all be a hell of a lot stupider, given our lack of "infinite" knowledge compared to LLMs.

> if you have any belief in philosophy [...]

You're going to have to explain which part of philosophy you mean, because what came after this doesn't follow from that premise at all. It's like saying a Chinese Room is fundamentally different from a "real" solution even though nobody can tell the difference. That's not a "belief in philosophy", that's human exceptionalism and perhaps a belief in the soul.

The belief that your thoughts are constructed based on an understanding of principles such as logic, rationality, ethics. That your interactions are built from a solid understanding of these ideas. As opposed to every train of thought just being glued together from pertinent fragments you can recall from your knowledge in response to a prompt provided by the circumstances of reality.

> that's human exceptionalism and perhaps a belief in the soul.

I would also argue that LLMs are not proven to be equivalent to what's going on in our minds. Is it really "human exceptionalism" to state that LLMs are not yet and perhaps never will be what we are? I feel like from their construction it is somewhat evident that there are differences, since we don't raise humans the same way we raise LLMs. In terms of CPU years babies require significantly less time to train.

Not to be difficult but wouldn't "confabulating" be a preferable description for this behaviour? Hallucinating doesn't quite feel right but I can't exactly articulate why confabulate is superior in this context
Nah, my issue with both terms is that they imply that when the answer is "correct" that's because the LLM "knows" the correct answer, and when it's wrong it's just a brain fart.

It doesn't matter if the output is correct or not, the process for producing it is identical, and the model has the exact same amount of knowledge about what it's saying... which is to say "none".

This isn't a case of "it's intelligent, but it gets muddled up sometimes". It's more of the case that it's _always_ muddled up, but it's accidentally correct a lot of the time.

>It doesn't matter if the output is correct or not, the process for producing it is identical

I don't see how this differs from a human earnestly holding a mistaken belief.

"Hallucinating" (normally) means having a subjective experience of the same type as a sensory perception, without the presence of a stimulus that would normally cause such a perception. I agree it's weird to apply this term to an LLM because it doesn't really have sensory perception at all.

Of course it has text input, but if you consider that to be equivalent to sensory perception (which I'd be open to) then a hallucination would mean to act as if something is in the text input when it really isn't, which is not how people use the term.

You could also consider all the input it got during training as its sensory perception (also arguable IMHO), but then a proper hallucination would entail some mistaken classification of the input resulting in incorrect training, which is also not really what's going on I think.

Confabulation is a much more accurate term indeed, going by the first paragraph of wikipedia.

It's not AI. As accurate as it may ever seem, it's simply not actually aware of what it's saying.

Conflating intelligence and awareness seems to me the biggest confusion around this topic.

When non-technical people ask me about it, I ask them to consider three questions:

- is alive?

- thinks?

- can speak (and understand)?

A plant, microbe, primitive animals... are alive, don't think, can't speak.

A dog, a monkey... are alive, think, can't speak.

A human is alive, thinks, can speak.

These things aren't alive, think, can speak.

I know some of the above will be controversial, but clicks for most people, that agree: if you have a dog, you know what I mean whith "a dog thinks". Not with words, but they're capable intricate reasoning and strategies.

Intelligence can be mechanical, the same as force. For a man from the ancient times, the concept of an engine would have been weird. Only live beings were thought to move on their own. When a physical process manifested complex behaviour, they said that a spirit was behind it.

Intelligence doesn't need awareness. You can have disembodied pieces of intelligence. That's what Google, Facebook, etc. have been doing for a long time. They're AI companies.

It doesn't help with the confusion that speaking is a harder condition than thinking and thinking seems to be harder than being alive: "these things aren't alive so they can't think" but they speak, so...

Ehh... my dog is alive, thinks, and "speaks" in a manner - not a cute term for barking, but he communicates (with relatively high effectiveness) his wants and desires. Maybe not using human words, but he certainly has his own sort of crude language, as does my cat.

The problem is that LLMs aren't alive, and they _don't think_. The speaking is arguable.

You might be onto something (or not, I'm not sure), but its extremely well-documented that both dogs and monkeys can speak.

They can't speak English like a human, but they both can understand a good deal of English, and they both can speak in their own ways (and understand the speaking of others).

I think the key thing about these LLMs is that they upend the notion that speaking requires thinking/understanding/intelligence.

They can "speak", if you mean emit coherent sentences and paragraphs, really well. But there is no understanding of anything, nor thinking, nor what most people would understand as intelligence behind that speaking.

I think that is probably new. I can't think of anything that could speak on this level, and yet be completely and obviously (if you give it like, an hour of back and forth conversation) devoid of intelligence or thinking.

I think that's what makes people have fantastical notions about how intelligent or useful LLMs are. We're conditioned by the entirety of human history to equate such high-quality "speech" with intelligence.

Now we've developed a slime mold that can write novels. But I think human society will adapt quickly, and recalibrate that association.

I can't think of anything that could speak on this level, and yet be completely and obviously (if you give it like, an hour of back and forth conversation) devoid of intelligence or thinking.

It's not devoid of intelligence or thinking. You're just using "what I'm doing right now" as the definition of intelligence and thinking. It isn't alive so it can't be the same. You are noticing that its intelligence is not centralized in the same way as your own mind.

But that's not the same as saying it's dumb. Try an operational definition that involves language and avoid vague criteria that try to judge internal states. Your dog might understand some words, associate them to the current situation and react, but can't understand a phrase.

These things can analyze the syntax of a phrase, can follow complex instructions, can do what you tell them to do. How is that not "understanding"?

If that isn't intelligence for you, I don't know what else to say.

There definitely is an echo chamber, but not related to the existence of ChatGPT and other AI technologies (even the "menial" laborers I know being relatively aware of it, and my parents and family members frequently talking about it in the context of deepfakes/cheating (at school)/automation/healthcare.

Common reactions to ChatGPT and a lot of the fear are definitely overemphasized in the tech field, but that makes sense.

No. Quite the opposite. Everyone else is in an echo chamber telling them that AI is a cool fad, a toy that will never fully replace people, etc., when in reality AI is the most disruptive technology since the Internet, and the most dangerous technology since the atomic bomb.

I can't stop shaking my head whenever I read any article on AI written by non-tech journalists (and even many by tech journalists). AI is vastly more dangerous and more urgent than climate change, than Russia and China, than literally any other hot topic today, and it's being treated like a combination of a science fiction story and an entertainment tool.

Not really.

AI -might- kill us if there is some quantum leap that makes it sentient.

Climate change -will- kill us if we do nothing.

For now one is simply not like the other, it may change but there is no guarantee we every breach that barrier.

> Climate change -will- kill us if we do nothing.

Horseshit. There is not a single scientific model that predicts human extinction from climate change. Parts of humanity, yes. All of humanity, no chance.

There a plenty of models that predict human extinction from AGI.

Climate change will kill millions in the next ~40-60 years and if course isn't corrected 100s of millions will follow. Seems like it will do a pretty damn good job of killing us even if it fails to kill all of us.

I agree AGI would be horrendously dangerous and if achieved has a higher chance of complete extinction. However, we don't have AGI and it's still not clear we ever will.

I don't think it's likely, but climate change could easily wipe out humanity by exacerbating pre-existing geopolitical tensions. For instance, Pakistan and India famously share a river, and the treaty that governs its water rights was written with the assumption that the river wouldn't reduce in output. A sufficiently severe drought could cause a war between India and Pakistan, and both countries have nukes.

You could dispute that WW3 would cause human extinction, and while I'm not remotely certain of it I think that WW3 could cause extinction, if there's a sufficient combination of 1) climate collapse and 2) worldwide economic collapse that makes high-tech systems impossible to sustain.

These idiots will believe anything if you put “climate change” in front of it…

“Let’s block out the sun, it will be good for global warming” says the billionaire as plants and animals freeze and die. WE NEED THE SUN THEY ARE TRYING TO KILL US OFF. Duh if this is not self evident then you are already dead.

> AI is vastly more dangerous and more urgent than climate change, than Russia and China, than literally any other hot topic today

No, it's not. And most (not quite all, there are some genuine nutballs) of the people selling that idea are selling it to push a political agenda attached to their financial interest (mostly, in AI: either by pushing AI danger to advance competition-limiting regulatiom or by pushing the kind of AI danger that is not actually imminent with hyperbolic language to distract from the real and present issues with AI, and sometimes both.)

> AI is vastly more dangerous and more urgent than climate change, than Russia

No it isn't, and I say that as a user and Integrator of ML tech.

Climate Change: Literally our planetary habitat becoming uninhabitable for our species.

Russia: A country with one of the largest nuclear arsenals literally threatening a large portion of the world with nuclear annihilation.

Excuse me when the prospect of changes in the economic landscape for white collar jobs doesn't look particularly frightening compared to such problems. Especially since we live in a day and age where 2 entire generations have lived most of their lives in almost constant economic upheavals anyway.

As for all the AI-doomerism that's flying around the net: As long as no one can even give me a precise, quantifieable definition of "general intelligence", aka. one that doesn't include pointing at ourselves, and a method to measure how far AI is from that, I will work under the assumptions confirmed by what is measureable and observable: that what we have are still stochastic inference engines.

Russia isn't "literally threatening a large portion of the world with nuclear annihilation". Any 'threat' they have issued has been in response to 'threats' issued by 'our' side... including at the time UK PM Liz Truss being willing to push the nuclear button against Russia. If that's not a threat then neither is Russia saying that they'll be willing to use nukes too.

Our politicians together with our sycophantic media and their weapons salesmen talking heads really do spread the most egregious disinformation throughout every wartime situation we are involved in, by proxy or otherwise.

Glass houses... stones...

> Any 'threat' they have issued has been in response to 'threats' issued by 'our' side

Then please, link me the relevant statements. Liz Truss's (who btw. isn't Britains PM any more) remarks were made in late August 2022 [1]. The russian nuclear sabre-rattling started in February 2022 [2].

So who exactly has threatened russia with nuclear weapons to elicit these responses? Helping a souvereign nation defend itself against an invasion and protecting their land and people, is not a threat. Offering a souvereign nation to join a military pact is also not a threat.

[1]: https://www.wsws.org/en/articles/2022/08/26/jfvn-a26.html

[2]: https://en.wikipedia.org/wiki/Nuclear_threats_during_the_Rus...

AGI isn't going to kill us.

But enough unemployment caused by the next wave of automation is sure going to cause civil unrest.

Look at the chartists, luddite and the saboteurs. Some of them were weavers who up until that point had been in high society, running parts of countries through the guilds system. Then over a couple of years, the bottom fell out and they were cast into the mills like the unlanded labours.

That, was not a smooth transition.

The people that claim "oh there will be new jobs" I mean sure, there probably will be, but they forget to mention the important qualifier: "eventually"

In my circle of friends, I'm the only one working in technology. But ChatGPT is something that everyone has tried and has been amazed by. For my non-tech friends, it's been mostly used for playing around, generating poems, etc - not really for productivity purposes.
Given ChatGPT reached 100M users shortly after launch, it must have a broader reach than just techies.

There are regular articles on it in mainstream news. Most of the people I know have tried it. Some of them can't see it will be useful, and some of them already find it useful. Some of them are techies and some aren't. Some techies are in the set that don't think it is useful.

> are we just in an echo chamber?

Yes.

> but maybe AI mainstream adoption will take longer than we anticipate

It will take longer than some "evangelists" anticipate, but other than a lot of tech hypecycles (like web3), this one is different for 3 reasons:

    a) It's firmly in the public mindspace, including mass media
    b) It has already proven it's usefulness and applicability to real world problems
    c) Politics are taking it very seriously
There are people who don't use it, for various reasons, but it is increasingly hard to never have heard of it.

A large part of that drive is due to the simple accessibility: Making ChatGPT as a convenient and simple webapp that appeals to non-techies was a brilliant move, and Microsoft driving integration into it's product suite will further adoption as well.

I don't think it's people in tech specifically - I think that "fire your workers because they're going to be replaced by AI" is a great excuse for firing workers without looking bad. Thus, anyone who might use that excuse has a strong incentive to play up the potential of AI (and thus the plausibility of their excuse), regardless of the actual value of AI.

Also, we're all calling it "AI" for some dumbass reason. That infects our thoughts with unwarranted credit toward the technology.

We (some of us at least) have been calling simple logic in computer games AI. And an artificial plant means some plastic shaped like a plant, it can't do photosynthesis, it can't grow, it can't reproduce. Meanwhile current AIs can code, explain a joke, etc.

After all an artifice: (the use of) a clever trick or something intended to deceive

they can't "code", they can't "explain a joke", they can output cached human data in such a way that it seems like they're generating these things themselves.
what do you mean by this exactly? like whats an experiment which establishes the difference? a lot of human behavior is also only liminally generative cached retrieval
On October 17th, 1979, VisiCalc was released.

I'm sure pretty much every accountant in the world got up and went to work that day exactly like they'd been doing all of their career. It probably didn't feel different for that many of them. Most of them had probably never used a computer then, and a lot of them probably didn't feel any particular need to, at least until they tried it. There were probably more than a few who were near the end of the career managed to keep doing their job the old way for another half or whole decade or so, because even when the future moves fast it's never evenly distributed.

There were probably also more than a few who saw VisiCalc and bought an Apple II to start doing their own books and ended up regretting it. I don't know where we are and how things will pan out, but I think there's a parallel.

You have to go back a whole lot further if you're making an analogy between chatGPT and spreadsheets. Back to Ada Lovelace, probably. VisiCalc did already useful and already transformative, the only things missing was better user experience and adoption. ChatGPT is a nice thought provoking toy, but not really useful other than as a novel search engine interface.
Agreed. On my Twitter timeline lots of people are saying "the world will be completely different in 6-12 months' time". On October 17th, 1980, most accountants' work day still looked the same as a year before. It took decades for spreadsheets to get widespread adoption. I think LLMs are revolutionary, but the (r)evolution will take decades to materialize. Our work days will look completely different... in 2043.
> I think LLMs are revolutionary, but the (r)evolution will take decades to materialize.

1980s was a different world. Today, the distribution that BigTech has, enabled by this thing called the Internet, changes the dynamic completely.

Given the investment, I believe LLMs will change some of the industries within 5 years, simply because it is a new but natural way to interact with Computers; and Google/Apple already put an Internet-connected Computer in everyone's hands.

This LLM hype is very real and is nothing like web3 (which lacked 10x utility over web2, imo).

When low/no-code started its hype? 5-10 years ago? Or something like that. First release dates:

Mendix 2005 Outsystems 2001 Appian 2004 (no info before) Quickbase 2000 Zoho Creator 2006

~20 years ago. I agree with the idea that it’s a new cool way of interaction with systems, but I also think you trust too much in our development cycles. Five years is nothing. Companies will try and fail, try and pivot, try and barely swim for years. Few of them will arrive alive at stage 2 where the real deal starts.

If it were easy, it would hit frontpages last week, because small companies and single developers can prototype things 10-100x faster than any bigcorp.

The primary issue that I see here is that as a natural language interface to a computer, I don’t know that there’s enough profit to support the hardware and training costs. Also, there are copyright questions involved. There also issues of hallucination, and limiting these neuters the creative aspect of the LLMs.

Personally, I truly hope that this technology gets cheaper and cheaper, and that it can be run on a device the size of a pager in pocket with a pair of AirPods on. If that were my computing environment 90% of the time, I’d be thrilled.

I hate and am terrified of what LLMs will do to society, and I do think they're overhyped, but the revolution won't take years.

They can already fully replace human proofreaders and content marketers. They're very close to being able to replace graphic designers, voice actors, and commercial photographers. I expect we'll see many more unsexy jobs like those being replaced within 5 years.

Throwing out the hot take (for HN) that computer intuition is about to be a much stronger destabilizing force than spreadsheets were. And that’s really where change comes from, IMO - the status quo showing cracks
A young man in his 20's starting his career will never know understand how things worked without the tech. A man in his 30's see's an opportunity to boost his career, and an older man in his 50's see's a long fought career's worth of expertise being thrown away.
You don't need to put an apostrophe before every s that ends a word. Only ones that are possessive, like "career's"

It's 20s, not 20's. It's sees, not see's.

This might be an autocorrect issue. My iphone 7 would do this rampantly with the swipe keyboard to the point where i concluded that no one at apple actually used it.