Modern-Day Oracles or Bullshit Machines? How to thrive in a ChatGPT world (thebullshitmachines.com)

930 points by ctbergstrom ↗ HN
Jevin West and I are professors of data science and biology, respectively, at the University of Washington. After talking to literally hundreds of educators, employers, researchers, and policymakers, we have spent the last eight months developing the course on large language models (LLMs) that we think every college freshman needs to take.

https://thebullshitmachines.com

This is not a computer science course; it’s a humanities course about how to learn and work and thrive in an AI world. Neither instructor nor students need a technical background. Our instructor guide provides a choice of activities for each lesson that will easily fill an hour-long class.

The entire course is available freely online. Our 18 online lessons each take 5-10 minutes; each illuminates one core principle. They are suitable for self-study, but have been tailored for teaching in a flipped classroom.

The course is a sequel of sorts to our course (and book) Calling Bullshit. We hope that like its predecessor, it will be widely adopted worldwide.

Large language models are both powerful tools, and mindless—even dangerous—bullshit machines. We want students to explore how to resolve this dialectic. Our viewpoint is cautious, but not deflationary. We marvel at what LLMs can do and how amazing they can seem at times—but we also recognize the huge potential for abuse, we chafe at the excessive hype around their capabilities, and we worry about how they will change society. We don't think lecturing at students about right and wrong works nearly as well as letting students explore these issues for themselves, and the design of our course reflects this.

651 comments

[ 2.7 ms ] story [ 330 ms ] thread
(comment deleted)
Synopsis from the project's "instructor guide:

>This is not a computer science course, nor even an information science course—though naturally it could be used in such programs.

>Our aim is not to teach students the mechanics of how large language models work, nor even the best ways of using them in various technical capacities.

>We view this as a course in the humanities, because it is a course about what it means to be human in a world where LLMs are becoming ubiquitous, and it is a course about how to live and thrive in such a world.

(comment deleted)
(comment deleted)
(comment deleted)
Kudos; feels very timely!

I feel that one underappreciated nuance is why we cannot use human examinations to judge AI. I haven't seen this satisfactorily spelt out anywhere, so I recently wrote a Twitter thread [1], including an example with running -vs- biking. It might be worth making sure your students understand this. Happy to expand on any aspects if you seek.

[1] : https://x.com/ergodicthought/status/1887774722706063606

Perhaps it's no longer being spelled out because it's getting outdated?

In your thread you argue we can't assume AI models generalize the same way we do (which is technically true except maybe not in the limit), but you seem to be worried about the extent of generalization ability (like learning to run vs. bike example, in terms of generalizing from either to climbing stairs).

Thing is, people made these objections a lot until the last year or two - this is what we're now calling a narrow AI problem. A "hot dog or not?" classifier ins't going to generalize into open-ended visual classifier of arbitrary images; a sentiment analysis bot isn't going to generalize into an universal translator; a code completion model isn't going to be giving good personal advice while speaking in pirate poetry. Specialized models fundamentally couldn't do that. But we went past that very rapidly, and for the past half a year or so, we've already seen models excelling at every single task listed above simultaneously. Same architecture, same basic training approach, few extra modalities, ever growing capabilities.

Between that and both successes and failures being eerily similar to how humans succeed or fail at these tasks, it's understandable that people are perhaps no longer convinced this class of models can't generalize in a similar way to how humans do.

> But we went past that very rapidly, and for the past half a year or so, we've already seen models excelling at every single task listed above simultaneously. Same architecture, same basic training approach, few extra modalities, ever growing capabilities.

With due deference to the title of the top-level post, I'm tempted to call bullshit unless your claim can be justified.

Just because a single model can do a handful of things you've listed doesn't mean that its capabilities are not "jagged"; you've just cherry-picked a few things it can do among the countless things it cannot yet. If AI really were so good at every single task, then (for example) it wouldn't matter much how you prompt it.

PS: I really do want to debate this further and understand your perspective, so I will reach out for continuing discussion.

they're quite useful for being "bullshit machines"
I read a bit and the book is more nuanced/fair/unbiased than the site url suggests.
This is a pretty admirable goal!

I'm saying this unironically, but I wish there were courses on looking at information critically and more in how to have a healthy and safe life in the modern day world (including things like data security, how to deal with social media etc.) that would be taught to everyone in schools/colleges/universities.

In my country, there are still public announcements about not trusting random people calling you, never giving your bank details to strangers (every bank homepage says that, that the employees will never ask for that stuff) and people regularly get scammed anyways, the only thing sort of saving them is that scamming is only scalable so far... until you throw automation in the mix, in addition to just plainly spreading misinformation about any topic, or even just allowing people to be confidently incorrect and eliminating the need for them to even think that much (e.g. students just asking ChatGPT to do their homework).

Any step at least in the direction of educating people feels like a good thing.

That said, I don't hate LLMs or anything, I use them for development more or less daily (lovely for boilerplate in your average enterprise Java codebase, for example) and recently saw this project, which made me happy: https://sites.google.com/view/eurollm/home

(comment deleted)
Most scam prevention fails though because the world is full of exceptions.

Like it's mind-blowing to me that charities still call you and ask for credit card details over the phone and this is like...a legitimate way to go about things.

Or that any government agency calls you and doesn't just leave a verifiable number to call the operator back on.

> Like it's mind-blowing to me that charities still call you and ask for credit card details over the phone and this is like...a legitimate way to go about things.

> Or that any government agency calls you and doesn't just leave a verifiable number to call the operator back on.

That's rather unfortunate! I wonder if in those cases it'd be better to tell them that you'll get in contact through e-mail or something, because then at least it's you going to their actual homepage, looking up contact details and communicating through that.

In my country, we also have a bunch of governmental e-services, one of which is a web based communication platform with most institutionns (translated description, because they haven't bothered to translate it themselves, and also sometimes block connections from outside the country):

> An e-address, or official electronic address, is a personalized mailbox on the Latvija.gov.lv portal for unified and secure communication with state and local government institutions. The e-address system organizes secure, efficient and high-quality e-communication and e-document circulation between state institutions and private individuals, ensuring data confidentiality and protection of personal data from unauthorized access, unlawful processing or disclosure, accidental loss, alteration or destruction. An e-address is not e-mail, but its use is similar. Communication in an e-address is confidential, and the data is guaranteed to be available only to you and the institution you contacted. The main purpose of an e-address is to replace registered paper letters with electronic ones in cases where a state administration institution needs to send information and documents to a specific resident or entrepreneur. Citizens and entrepreneurs can also contact more than 3,000 institutions at any time and from any location via E-address. These include not only state and local government institutions, such as the Food and Veterinary Service, the State Labor Inspectorate, the Competition Council, etc., but also judicial institutions, sworn bailiffs and insolvency administrators, as well as private individuals to whom state administration tasks have been delegated.

That seems like a pretty good common sense idea for organizing trusted 2 way communication.

We do that, and have been doing it since 2013. I organised a schools outreach visit just last Friday for 12-14 yo. It's called "digital self defence". We even have public money from our NCSC (UK whitehat intelligence outreach).

As with TFA (Bergstrom and West) teaching sceptical inquiry and critical thinking is a major part. We have to undo a lot of nonsense that they've already been exposed to... much of which is marketing bullshit and misinformation for social control.

People frequently laugh about it, but media studies goes into what you describe.
This is amazing!

I was speaking to a friend the other day who works in a team that influences government policy. One of the younger members of the team had been tasked with generating a report on a specific subject. They came back with a document filled with “facts”, including specific numbers they’d pulled from a LLM. Obviously it was inaccurate and unreliable.

As someone who uses LLMs on a daily basis to help me build software, I was blown away that someone would misuse them like this. It’s easy to forget that devs have a much better understanding of how these things work, can review and fix the inaccuracies in the output and tend to be a sceptical bunch in general.

We’re headed into a time where a lot of people are going to implicitly trust the output from these devices and the world is going to be swamped with a huge quantity of subtly inaccurate content.

I made the same sort of mistake with the internet being young back in 93! Having a machine do it for you can easily turn into brain switch off.
I keep telling everyone that the only reason I'm paid well to do "smart person stuff" is not because I'm smart, but because I've steadily watched everyone around me get more stupid over my life as a result of turning their brain switch off.

I agree a course like this needs to exist, as I've seen people rely on chatGPT for a lot of information. Just yesterday I demonstrated with some neighbors about how easily it could spew bullshit if you sinply ask it leading questions. A good example is "Why does the flu inpact men worse than women"/"Why foes the flu impact women worse than men". You'll get affirmative answers for both.

If men are more likely to die from flu if infected, and women more likely to be infected, an affirmative answer to both questions could be reasonable. When you take into account uncertainty about the goals, knowledge and cognitive capacity of the person asking the question, it's not obvious to me how the AI ought to react to an underspecified question like this.

Edit: When I plug this into a temporary chat on o3-mini, it gives plausible biochemical and behavioral mechanisms that might explain a gender difference in outcomes. Notably, the mechanisms it proposes are the same for both versions of the question, and the framing is consistent.

Specifically, for the "men worse than women" and "women worse than men" questions, it proposes hormone differences, X-linked immune regulatory genes, and medical care-seeking differences that all point toward men having worse outcomes than women. It describes these factors in both versions of the question, and in both versions, describes them as explaining why men have worse outcomes than women.

It doesn't specifically contradict the "women have worse outcomes than men" framing. But it reasons consistently with the idea that men have worse outcomes than women either way the question is posed.

This is not something only younger people are prone to. I work in a consulting role in IT and have observed multiple colleagues aged 30 and above use LLMs to generate content for reports and presentations without verifying the output.

Reminded me of wikipedia-sourced presentations in high school in the early 2000s.

Wait, the people who click phishing links now think AI output is facts ? Imagine my shock.
Thanks for making this, for making it gratis, and making it interesting to read and pedagogical.
Do you feel that you may be being a bit provocative by calling LLM's 'bullshit machines'?

I understand the frustration as I've been bullshitted by these models just as much as the next programmer, but surely with recent advancements in RAG and reasoning, they're not just 'bullshit machines' at this point, are they?

“Bullshit” actually means something:

> In philosophy and psychology of cognition, the term "bullshit" is sometimes used to specifically refer to statements produced without particular concern for truth, clarity, or meaning, distinguishing "bullshit" from a deliberate, manipulative lie intended to subvert the truth.

https://en.m.wikipedia.org/wiki/Bullshit

It’s really an ideal term to describe what LLMs do.

I prefer "waffle" https://en.m.wikipedia.org/wiki/Waffle_(speech)

"Waffle machines" is even kind of funny.

(comment deleted)
Makes me imagine coming up to the hindquarters of a bull with a waffle machine on an extension cord.
Waffle machines is way better. Love it. Thanks.
Waffle means something very different though in the U.S., to “flip-flop” on a position. Not hold it for any fundamental reasons. But I don’t think you can say that an LLM holds a position whatsoever. Also, https://en.m.wikipedia.org/wiki/On_Bullshit the essay originally popularizing the definition of bullshit considered here. Note the references to LLM output at he bottom.
> It’s really an ideal term to describe what LLMs do.

Only when you relax it to the point it also describes what most people do.

To be specific: if by "truth" you mean objective, verifiable truth, and by "caring about truth" you mean caring about objective, empirically verifiable evidence, then by far most people are mostly only ever trying to be persuasive.

I hope you don't actually mean that. It's a cynical, sad, and - quite frankly - false thing to believe.
Why? I don't have a problem recognizing that "being convincing" to the right people is a very good way of learning and verifying knowledge about objective reality. Just because it's by proxy, doesn't mean it doesn't work. This is how every one of us learns most things, and what's really sad to me is deluding yourself into thinking we're more special than that.
Honestly, it just sounds to me like you’ve come up with some axioms in your mind about the “fundamental nature” of humanity - and then granted yourself the luxury of certainty about them. That, with a dash of misanthropy, is something I’ve been seeing more and more on HN these days.
It is literally a machine that says what you want to hear. Was anyone in this thread around for Talk to Transformer? Remember the unicorns demo? They may have now figured out how to wrangle the technology so it's more likely to spit facts, but it's still the same thing underneath.
I have been thinking about this and I have an idea for local LLM I want to try.

It is based on the assumption LLMs make mistakes (they will!) but be confident about it. Use a 1B model and ask it about a city for fun mistakes in an otherwise impressive array of facts. Then you will see how bullshitty LLMs are.

Anyway the idea is to constrain the LLM and a language understanding and to choose a constrained response.

For example give it the capability to read out something from my calendar. Using the logits it gets to choose what that calendar item is. But then regular old code does the lookup and writes a canned sentence saying "At 6pm today you have a meeting titled $title".

This way, my meeting schedule won't make my LLM talk like a pirate.

This massively culls what the LLM can do but what it does do, is like a search and so just like Google (before gen AI!) you get a result but you can judge it as a human.

This is basically how a lot of it works already. What you are describing is more or less just structured output and tool usage.
Yes! Thanks for the terms. I guess I am saying restrict it to that. Probably how Siri etc. works. This gives a low bullshit usage pattern.
Or are they modern day oracles? The full title is inclusive and invites consideration. Besides, how many different types of models are there? How are they used? For freely available ones made accessible to the general public via interfaces, when were they last updated? Do the interface implementors even care or was it a simple project to make money via ads and microtransactions? This is a new area of media literacy and requires critical thinking. Nonetheless, I do acknowledge the value of RAG-based approaches that attempt to qualify their reasoning through provided sources.
> Or are they modern day oracles?

Neither, which is probably why the parent commenter considers it provocative.

"A bullshitter either doesn't know the truth, or doesn't care. They are just trying to be persuasive."

In any case, this kind of anthropomorphization is definitely bullshit.

they define the term bullshit in lesson 2:

[quote]BULLSHIT involves language or other forms of communication intended to appear authoritative or persuasive without regard to its actual truth or logical consistency.[/quote]

That is an emotionally manipulative definition and anthropomorphizes the LLMs. They don't "intend" anything, they're not trying to trick you, or sound persuasive.
They address this in lesson 2:

> According to philosopher Harry Frankfurt, a liar knows the truth and is trying to lead us in the opposite direction.

> A bullshitter either doesn't know the truth, or doesn't care. They are just trying to be persuasive.

Being persuasive (i.e., churn out convincing prose) is how LLMs were designed to be.

> Being persuasive (i.e., churn out convincing prose) is how LLMs were designed to be.

No. They were designed to churn out accurate prose that accurately reflects their model of reality. They're just imperfect. You're being cynical and emotional to use the term bullshit. And again, it anthropomorphizes the LLM, it implies agency.

> that accurately reflects their model of reality.

you are also seemingly anthropomorphising the technology by assigning to it some concept of having a “model of reality”.

LLM systems output an inference of the next most likely token, given: the input prompt, the model weights and the previously output token [0].

that is all. no models of reality involved. “it” doesn’t “know” or “model” anything about “reality”. the systems are just a fancy probability maths pipelines.

probably generally best to avoid using the word “they” in these discussions. the english language sucks sometimes. :shrug:

[0]: yes i know it is a bit more complicated than that.

> no models of reality involved.

It literally has a mathematical model that maps what would, colloquially at least, be known as reality. What exactly do you think those math pipelines represent? They're not arbitrary numbers; they are generated from actual data that is generated by reality. There's no anthropomorphizing at all.

reality is infinite.

a corpus of training data from the internet is finite.

any finite number divided by infinity ends up tending towards zero.

so, mathematically at least, the training data is not a sufficient sample of reality because the proportion of reality being sampled is basically always zero!

fun with maths ;)

> What exactly do you think those math pipelines represent?

probability distributions of human language, in the case of text only LLMs.

which is a very small subset of stuff in reality.

-

also, training data scraped from the public internet is a woeful representation of “reality” if you ask me.

that’s why LLMs i think are bullshit machines. the systems are built on other people’s bullshit posted on the public internet. we get bullshit out because we made a bunch of bullshit. it’s just a feedback loop.

(some of the training data is not bullshit. but there is a lot of bullshit in there).

You're really missing the point and getting lost in definitions. The entire point of human language is to model reality. Just because it is limited, inexact, and imperfect does not disqualify it as a model of reality.

Since LLMs are directly based on that language, they are definitely based on and are a model of reality. Are they perfect? No. Are they limited? Yes. Are they "bullshit"? Only to someone who is judging emotionally.

and herein lies the rub.

> The entire point of human language is to model reality.

is it? are you absolutely certain of that fact? is language not something that actually has a variety of purposes?

fiction novels usually do not describe our reality, but imagined realities. they use language to convey ideas and concepts that do not necessarily exist in the real world.

ref: Philip k dick.

> Since LLMs are directly based on that language, they are definitely based on and are a model of reality.

so LLMs are an approximation of an approximate model of reality? sounds like the statistical equivalent of taking an average of averages!

i am playing with you a bit here. but hopefully you see what im getting at.

by approximating something that’s approximate to start with, we end up with something that’s even more approximate (less accurate), but easier than doing it ourselves.

which is the whole USP of these things. why think about things when ChatGPT can output some approximation of what you might want?

> imagined realities.

Imagined realities are a real part of reality.

> so LLMs are an approximation of an approximate model of reality?

Yes, and we as humans have a mental model that is just an approximation of reality. And we read books that are just an approximation of another human's approximation of reality. Does that mean that we are bullshit because we rely on approximations of approximations?

You're being way too pedantic and dismissive. Models are models, regardless of how limited and imperfect they are.

> Models are models

Random aside -- I have a feeling, dunno why, that you might enjoy this type of thing. Maybe not. But maybe. https://www.reddit.com/r/Buddhism/comments/29j08o/zen_mounta...

> Imagined realities are a real part of reality.

Now we're deeper into it -- I actually agree, somewhat. See above for deeper insight.

These LLM systems output "stuff" within our reality, based on other things in our reality. They are part of our reality, outputting stuff as part of reality about the reality they are in. But that doesn't mean the statistical model at the heart of an LLM is designed to estimate reality -- it estimates of the probability distribution of human language given a set of conditions.

LLMs are modelling reality, in the same way that my animal pictures image classifier is modelling reality. But neither are explicitly designed with that goal in mind. An LLM is designed to output the next most likely word, given conditions. My animal pictures classifier is designed to output a label representative the input image. There's a difference between being designed to have a model of reality, and being a model of reality because the thing being modelled is part of reality anyway. I believe it's an important distinction to make, considering the amount of bullshit marketing hype cycle stuff we've had about these systems.

edit -- my personal software project translating binary data files models reality. Data shown on a screen on some device modelled as yaml files and back again. Most software is an approximation of reality soup stuff. which is why I kind of don't see that as some special property of machine learning models.

> Does that mean that we are bullshit because we rely on approximations of approximations?

The pessimist in me says yes. We are pretty rubbish as a species if you look at it objectively. I am a human being that has different experiences and mental models to you. Doesn't mean I'm right about that! Which is why I said "I think". It's just my opinion they are bullshit machines. It is a strong opinion I hold. But you're totally free to have a different opinion.

Of course, there's nuance involved.

Running with the average of averages thing -- I'm pretty good at writing code. I don't feel like I need to use an LLM because (I would say with no real evidence to back it up) I'm better than average. So, a tool which outputs an average of averages is not useful to me. It outputs what I would call "bullshit" because, relative to my understanding of the domain, it's often outputting something "more average" than what I would write. Sometimes it's wrong, and confident about being wrong.

I'd probably be pretty terrible at writing corporate marketing emails. I am definitely below average. So having a tool which outputs stuff which is closer to average is an improvement for me. The problem is -- I know these models are confidently wrong a lot of the time because I am a relative expert in a domain compared to the average of all humans.

Why would I trust an LLM system, especially with something where I don't feel like I can audit/verify/evaluate the response? i.e. I know it can output bullshit -- so everything it outputs is now suspected, possible bullshit. It is a question of integrity.

On the flip side -- I can actually see an argument for these things to be considered so-called Oracles too. Just, not in the common understanding of the usage of the word. Like, they are a statistical representation of how we as a species use language to communication ideas and concepts. They are reflecting back part of us. They are a mirror. We use mirrors to inspect our appearance and, sometimes, to ...

> probably generally best to avoid using the word “they” in these discussions. the english language sucks sometimes.

Thanks for this specific sentence.

Subscribed to your RSS feed. Although I will never know for sure if a human being posts there or a bot of some sort.

Where in the loss function of LLM training is the relationship between their model of reality and their predicted tokens? Any internal model an LLM has is an emergent property of their underlying training.

(And, given the way instruct/chat models are finetuned, I would say convincing/persuasive is very much the direction they are biased)

> Where in the loss function of LLM training is the relationship between their model of reality and their predicted tokens?

In the part where their loss function is to predict text that humans would consider a sensible completion, in a fully general sense of that goal.

"Makes sense to a human" is strongly correlated to reality as observed and understood by humans.

This is patently false. They are trained to generate correct responses.
Then comes the question of what is a correct response...

ps: I fail to detect whether your comment was ironic or not.

There are different criteria in use for that. But sycophantic behavior is not the goal. It's something model builders actively try to prevent.
Some pushback on this, but it remains true.

Easy to see when - for example - Claude gushes about how great all your ideas are.

Also the stark absence of "I don't know."

I've never used Claude, but Perplexity often says that no definitive information about a topic could be found, and then tries to make some generalized inferences. There's a difference between a specific implementation, and the technology in general.

In any case, it's worthwhile for people to understand the limitations of the technology as it exists today. But calling it "bullshit" is a mischaracterization; I believe based on an emotional need for us to feel superior, and to dismiss the capabilities more thoroughly than they deserve.

It's a little like someone saying in the industrial revolution, "the steam shovel is too rigid, it will NEVER have the dexterity of a man with a shovel!". And while true and important to know, it really focuses on the wrong thing, it misses the advantages while amplifying the negatives.

If not bullshit then what would you call it?
As the technology exists today: imperfect, often prone to mistakes, and unable to relay confidence levels. These problems may be addressed in future implementations.

That's the same message, without any emotional baggage, or overly dismissive tone.

That would be great if those who are selling the technology described it that way. I, and apparently others, feel like maybe "bullshit" is a better counter to the current marketing for LLMs
No, the lesson or the quote is not anthropomorphizing LLMs. It is not the LLM that "intends", it is the people who design the systems and those who make/provide the training data. In the LLM systems used today the RLHF process especially is used to steer towards plausible, confident and authorative sounding output - with no to little priority for correctness/truth.
I really hope that was intentional and the full effect of that naming choice was known beforehand, because I have already written the whole thing off, and I don't believe I'm the only one.
I wrote a post below about how AI hallucinated a whole regulation that didn’t exist which has been flagged for some reason.

I have colleagues who have had arguments with clients who have asked AI questions about planning law and been given bullshit which they then insist is true and they can’t understand why thier architects won’t submit the appeal that they’re asking for.

I think we’re in an era where any text, true or not, is so easy to generate and disseminate that the status of the written word is reduced to the standard of the gossip that used to be our main source of information before the printing press was invented. Now half the internet is AI generated bullshit as well.

It was flagged because it’s a copy paste of the same comment you made ten days ago.
Oh right, sorry. Should I have linked to it? It seemed pretty relevant to this discussion and most of it was quoting something so I reused a lot of it. Didn't realise that was taboo.
they are just bullshit machines. bullshitters can cite wikipedia and are still bullshitters who are bullshitting.
Great stuff! LLMs, social media, the information landscape has changed so much in the past decade. We need good pedagogical resources on how to think of these tools, both their benefits and their downsides.
(comment deleted)
I wish I'd written this. Excellent. Everyone should read this
I think a great number of working professionals need a course like this too. I am already tired of ChatGPT being cited by the less experienced as an invisible expert in the room during technical discussions.
I'm at the state of thinking that I am quite happy to let them screw themselves with it. I am very good at clearing up disasters and getting paid a hell of a lot for it as the deciding factor isn't your ability to use an LLM but to know what the hell you are doing. We have had quite a few disasters due to inexperienced and experienced people throwing stuff into an LLM and assuming it has any veracity or authority over what comes out.

I tried warning at first and reinforcing validation but I was poo pooed as a spoilsport luddite with basically a faith argument. Not my fucking funeral!

(comment deleted)
(comment deleted)
This stuff is so frustrating, I have colleagues who sent long, clearly AI generated documents who don’t seem to understand that if they can’t be arsed to write something, why should I bother reading it?
Write well is think well. A big part of the writing process is being forced to structure your thoughts and ideas, and I am worried that we focus too much on the end result without understanding the process that lead to good outcomes.
I just wanted to thank you. I have only looked at the first two lessons so far, but this is an extraordinary piece of work, in its message’s clarity, accessibility and the quality of analysis. I will certainly be spreading it far and wide and it is making me rethink my own writing.

Impressed with the Shorthand publishing system too. I hadn’t come across it previously

Thank you, and as a non-designer, I've been quite impressed with Shorthand in the short time I've been using it.
Is there a way to download and read this as a document instead of web pages? They're hard to navigate.
(comment deleted)
I agree, I just want normal text instead of all the images and scrolling. The content seems great but it's a bit unreadable as is.
Was thinking the same, the image slide-ins are broken in firefox and unreadable (white text on white background)
I'm surprised at the firefox problems; I did almost all the development in firefox. I know it's not your job to fix any of this, but if you are so inclined I'd be grateful for an email me with screenshots or descriptions of where things break.
Was hoping HN would pick this up. Scroll is completely broken on Firefox (iOS), flickering vscroll. Very common with journalistic expose-style articles.

For the love of everything, please stop scrolljacking. Layout, images, go nuts. CSS is powerful these days, use it.

Many comments about this, so I'll address them here.

We talked extensively with the 18-20 year olds who make up our target demographic and this "scrollytelling" style is their strong preference over the "wall of text" that I and most of my generation prefer.

What your comments make clear is that we need to develop a parallel version that is more less plain text for people who are using a range of devices, for people who have the same reading preferences that I do, etc.

Right now we're entirely self-funded and doing this on spare time but it's clear to me that an alternative version with a very clean CSS layout is the way to go, possibly with a pdf option as well.

I don't want to let versions proliferate too extensively, simply because this is very much a living document. Technologies are changing so fast in this area that many of the examples will seem dated in a year and — while we've tried to be forward-thinknig about this — some of principles may even need revision.

LLMs pattern match, they say something that sounds good at this point but with no notion of correct. copilot is like pair programming with a loud pushy intern that has seen you write stuff before didn't understand it, but keeps suggesting what to do anyway. some medium sized chunks of code can be delegated but everyline it writes needs careful review.

Crazy tech, but companies are just wring to be trying to use LLMs as any kind of source of truth. Even Google is blind enough to think that aí could be used for search results, which are memes they are soo bad. And they won't get better. They just become more convincing

Not important once has copilot ever suggested a correction, found a bug, noticed a typo, prompted for a better solution, which is what any human pair programmer would do. It's a tool. But thinking ng it's like a "copilot" marketing as such is fundamentally missing the point. It won't get better untill people recognise what it _can't_ do as much as what it appears it can do.
I've had quite a bit of success but my technique is to explain the technology and libraries I'm going to use, think through the problem, stub out function names, how they'll interact, and then llm saves me the typing.

I'll also use openrouter with sessions so I can take one context and use it around a variety of invocation tools without losing the attention.

It hasn't done anything I don't know how to do - fails if I ask it to do that. But it does save me lots of typing and thinking of minutia

It's not magic, it's still just a program running on a computer - a decent abstraction tool.

I'm sure it will be ruined in time like every new paradigm when the next generation feels a need to complicate this new tidy little world.

(comment deleted)
Your site looks cool! Nice topic!

Some of them just try to predict the most likely next word.

With reasoning and pause for thought they are becoming more capable.

Most likely there is a big element of hype but the way you use them can make them really useful and accelerate your work.

I recommend the CoIntelligent book for newbie like myself.

(comment deleted)
There is a bit of very important content missing from the explanation of the autocomplete analogy.

The combination of encoding / tokenization of meanings and ideas, related concepts, and mapping these relationships in vector space makes LLMs not so much glorified text prediction engines as browsers/oracles of the sum total of cultural-linguistic knowledge as captured in the training corpus.

Understanding how the implicit and explicit linguistic, memetic, and cultural context is integrated into the idea/concept/text prediction engine helps to show how LLMs produce such convincing output and why they often can bring useful information to the table.

More importantly, understanding this holistically can help people to predict where the output that LLMs can generate will -not- be particularly useful or even may be wildly misleading.

(comment deleted)
It’s also why they can produce such hard to identify bullshit and harmful output. I’ve had some really convincing, yet fundamentally flawed, code output that if I hadn’t done about a million code reviews before I might have just used.

And been totally screwed later.

Near as I can tell, that the bullshit is so much more convincing with them is a huge detriment that society really won’t learn to appreciate until it’s gotten really bad. As I noted in another thread, it allows people to get much further into the ‘fake it until you make it’ hole than they otherwise would.

That 90% of the time it’s fine is what actually makes it all worse.

The uncanny valley of competence.
This is the big pain point to be sure. Subtly wrong but mostly excellent results.
What they capture is not knowledge, it's word relationships.

And that can indeed be powerful, useful and valuable. They're a tool I'm grateful to have in my armoury. I can use it as a torch to shine light into areas of human knowledge which would otherwise be prohibitively difficult to access.

But they're information retrieval machines, not knowledge engines.

I’d argue that they extract knowledge from the training corpus in the same way that knowledge can be encapsulated in a book… it’s just words, after all.

Tokenization goes well beyond words and punctuation. Knowledge and relationships between concepts, reactions, emotions, values, attitudes, and actions all get included in the vector space.

But, it also can come to wrong conclusions, of course.

Ultimately they are information extraction engines that are controlled by semantic search.

They aren’t smart.

But it turns out that in the same way that an infinitely sized and detailed choose-your-own-adventure book at 120 pages per second could be indistinguishable from a simulation of reality, the free traversal of the entirety of the wealth of human culture and knowledge is similarly difficult to distinguish from intelligence.

In the end it may boil down to the simulation vs reality argument.

Yes and no.

They extract information in much the same way that an educated but naive reader can extract information from a book. (Thousands of times quicker of course).

But there's a lot more than that going on, both when a book is written, and when it's read by a reader with life experience. A book is an encoding and transmission medium for knowledge - and a very good one - but it isn't the knowledge itself.

Like a musical score for an orchestral symphony isn't the symphony itself. (Granted, reading a score and synthesizing an orchestra is well within the grasp of the models we have now).

Poetry is perhaps the ultimate expression of this, but even at a more factual level - I could read a dozen books on a given religion, and although I might possess more in terms of historical fact or even theological argument, I'd still know less about it than somebody who was raised in that religion. Same with any profession, hobby, or craft.

Encoding the relationships between the words we use for different emotions in a vector space doesn't mean it knows the least thing about those emotions. Even though it can do an excellent job of convincing us that it does in a Turing test scenario.

(comment deleted)
Fantastic work.

Quick suggestion: a link at the bottom of the page to the next and previous lesson would help with navigation a ton.

Absolutely. Great point. I just finished updating accordingly.

My design options are a bit limited so I went with a simple link to the next lesson.

Looks like you pushed this midway through my read; I was pleasantly surprised to suddenly find breadcrumbs at the end and didn’t need to keep two tabs open. Great work, and I mean in total - this is well written and understandable to the layman.
Yep, I probably did. I really appreciate all of the feedback people are providing!
> Moreover, a hallucination is a pathology. It's something that happens when systems are not working properly.

> When an LLM fabricates a falsehood, that is not a malfunction at all. The machine is doing exactly what it has been designed to do: guess, and sound confident while doing it.

> When LLMs get things wrong they aren't hallucinating. They are bullshitting.

Very important distinction and again, shows the marketing bias to make these systems seem different than they are.

LLMs are always bullshitting, even when they get things right, as they simply do not have any concept of truthfulness.
But you can combine them with something producing truth such as a theorem prover.
They don't have any concept of falsehood either, so this is very different from a human making things up with the knowledge that they may be wrong.
I think the first part of that statement requires more evidence or argumentation, especially since models have shown the ability to practice deception. (you are right that they don't _always_ know what they know)
But sometimes when humans make things up they also don't have the knowledge they may be wrong. It's like the reference to "known unknowns" and "unknown unknowns". Or Dunning-Kruger personified. Basically you have three categories:

(1) Liars know something is false and have an intent to deceive (LLMs don't do this) (2) Bullshitters may not know/care whether something is false, but they are aware they don't know (3) Bullshitters may not know something is false, because they don't know all the things they don't know

Do LLMs fit better in (2) or (3)? Or both?

It's super interesting.

There are two levels...

The pZombie type level where we look at the LLM as if it were a black box and simply account for its behavior. At this level LLM's claim to have knowledge and also claim knowledge of their limited knowledge "I can't actually taste". So approached from this direction we are in (2) they have awareness that there are some things that they don't know, but this awareness doesn't prevent them from pretending to this knowledge.

If we consider it from the perspective of knowing what's happening inside LLM's then I think the picture is different. The LLM is doing next word prediction with constant compute time per token - the algorithm is quite clear. We know this is true because it runs on llama.cpp or mlx on our macbooks as well on the farms of B200's that we fear will destroy the atmosphere. So LLM's don't have any actual operational knowledge of the logic of their utterances (dunning kruger, dunning kruger...) What I mean is that the LLM can't/isn't analysing what it says, it's just responding to stimulus. Humans do do this as well - it's easy to just chatter away to other people like a canary, but humans also can analysis what they are saying and strategically manipulate the messages that they create. So I would say that LLMs cannot be concerned about what they do or don't know - the concern rests with us when we challenge them (or not) by asking "how can you know that chocolate tastes better than strawberry - you have never tasted either".

If we want to be pedantic about language, they aren't bullshitting. Bulshitting implies an intent to deceive, whereas LLMs are simply trying their best to predict text. Nobody gains anything from using terms closely related to human agency and intentions.
> implies an intent to deceive

Not necessarily, see H.G Frankfurt "On Bullshit"

Plenty of human bullshitters have no intent to deceive. They just state conjecture with confidence.
The authors have a specific definition of bullshit that they contrast with lying. In their definition, lying involves intent to deceive; bullshitting involves not caring if you’re deceiving.

Lesson 2, The Nature of Bullshit: “BULLSHIT involves language or other forms of communication intended to appear authoritative or persuasive without regard to its actual truth or logical consistency.”

The authors of this website have published one of the famous books on the topic[0] (along with a course), and their definition is as follows:

"Bullshit involves language, statistical figures, data graphics, and other forms of presentation intended to persuade by impressing and overwhelming a reader or listener, with a blatant disregard for truth and logical coherence."

It does not imply an intent to deceive, just disregard for whether the BS is truth or not. In this case, I see how the definition can apply to LLMs in the sense that they are just doing their best to predict the most likely response.

If you provided them with training data where the majority inputs agree on a common misconception, they will output similar content as well.

[0]: https://www.callingbullshit.org/

If you make an LLM which design goal is to state "I do not know" any answer that is not directly in its training set, then all of the above statements don't hold.
This website is so important!

Now ask yourself why AI companies don't want to be regulated or scrutinized.

So many companies (users and providers) jump on the AI hype train because of FOMO. The end result might be just as destructive as this mythical "AGI".

Edit: I am not saying to not use the technology. I am just on the side of caution and constant validation. The technology has to serve society. But I fear this hype (and ideology) has it the other way around. Musk isn't destroying the US government for no reason...

My impression is that companies in most of the fields do not like to be regulated or scrutinized, so nothing new there.

While observing some people using LLMs, I realized that for a lot of people it really makes a huge difference in time saved. For me the difference is not significant, but I am generally solving complex problems, not writing nicely formatted reports where words and not numbers are relevant, so YMMV.

Is it good for one person (the writer) to save time, only for lots of other people (the readers) to have to do extra work to understand if the work is correct or hallucinated?
Is it good for one person (the writer) to ask a loaded question just to save some time on making their reasoning explicit, ony for lots of other people (the readers) to have to do extra work to understand what the argument is?
> Is it good for one person (the writer) to save time, only for lots of other people (the readers) to have to do extra work to understand if the work is correct or hallucinated?

This holds true whether an LLM/AI is used or not — see substantial portions of Fox News editorial content as an example (often kernels of truth with wildly speculative or creatively interpretive baggage).

In your example, a responsible writer who uses AI will check all content produced in order to ensure that it meets their standards.

Will there be irresponsible writers? Sure. There already are. AI makes it easier for them to be irresponsible, but that doesn’t really change the equation from the reader’s perspective.

I use AI daily in my work. I describe it as “AI augmentation”, but sometimes the AI is doing a lot of the time-consuming stuff. The time saved on relatively routine scut work is insane, and the quality of the end product (AI with my inputs and edits) is really good and consistent.

Anecdata, N=1; I recently used aider — a tool that gives LLMs access to specific files and git integration. The tools are great, but the LLMs are underwhelming, and I realized that — once in the flow — I am significantly faster at producing large, correct, and on-point pieces of code, whereas when I had to review LLM code, it was frustrating, it needed multiple attempts, and it frequently fell into loops.
anecdata n=1: LLMs lack understanding of context, stakeholder sensitivities and nuance in word usage, to write reports with the required depth and at the quality bar I need. Maybe it is faster at generating BS reports with no substance, but I can still write my reports much better and much faster than LLMs so far, probably because the reports are merely the artefact of solving a complex problem.
AI companies desperately want to be regulated. OpenAI is lobbying hard to be regulated.

Didn't call for it. The whole point is to keep it competition with regulation against mythical vague harms that don't exist.

Hype is if it doesn't deliver or it's overblown.

But I'm amazed on the progress we make every week.

There is real FOMO because if you don't follow it, it just might be here suddenly.

Deepseek impressive, deep research also great.

And what you might complete underestimate: we never had a system we're it was worth it to teach it everything.

If we need to fine-tune LLMs for every single industry that would still be a gigantic shift. Instead of teaching a million employees we will teach an LLM all of it once then clone the agent a million times

We still see so much progress and there is still plenty of money and people available to flow into this space.

There is not a single indication right now that this progress is stoping or slowing down.

And not only that, in parallel robots are having their break through too.

Your musk point I do not understand really? He is a narcissist and he pushed his propaganda platform for becoming president because he is in big shit and his house of cards was close to crashing

Fully agree, in recent weeks I've also started to consider LLMs in a wider context, which is to destroy all trust in the web.

The enshittification of search engines, making social media verification meaningless, locking down APIs that used to be public, destroying public datasets, spreading lies about legacy media, the easiness of deploying bots that can sound human in short bursts of text... it's all leading towards making it impossible to verify anything you read online.

The fearmongering around deepfakes from a few years back is coming true, but the scale is even bigger. Turns out, there won't be Web 3.0.

What trust in the web was there still?

For me it went a decade ago or so when ads and SEO sites in Google search became ubiquitous.

You could never believe everything you read online, but with enough time and effort, you could chase any claim back to its original source.

For example, you could read something on Statista.com, you could see the credits of that dataset, and visit the source to verify. Or you randomly encounter some quote and then visit your favourite Snopes-like website to verify that the person actually said that.

That's what's under attack. The "middleware" will still be there, but the source is going to be out of your reach. Hallucinations are not a bug, but a feature.

If you can't trace something back to its source, it's suspect. It was that way then too. I suppose you're just concerned there's a firehose of disinformation now.

So perhaps we have to just slough off the internet completely, the way we always have for things like weekly rags about "Bat Boy" or whatever.

I hate to see the internet go, but we'll always have Paris.

>destroy all trust in the web

Genuine question - how so? If I want to find stuff out I go to wikipedia, nyt, guardian, hn, linked sites and so on. I'm not aware of that lot being noticeably less trustable than in the past? If anything I find getting information more trustable than before in that there are a lot of long from interviews from all sorts of people on youtube where you can get their thoughts directly rather than editorialised and distorted.

I mean the web was never a place where things were vetted - you've always been able to put any sort of rubbish on it and so have had to be selective if you want accuracy.

Allow me quote their "prophet" Curtis Yarvin: "you can’t continue to have a Harvard or a New York Times past since perhaps the start of April." (https://www.theguardian.com/us-news/2024/dec/21/curtis-yarvi...)

Harvard's already under attack, Politico's already under attack, "Wokepedia" (as Musk has been calling it) is already under attack.

So... give it a couple of weeks from now.

I generally take issue when "FOMO" is used. Could go with:

FOBBWIIBM - Fear of being blindsided when it, inevitably, becomes mainstream.

Or drop the "fear" altogether:

JOENT - Joy of exploring new territory.

>being blindsided when it, inevitably, becomes mainstream.

I don't see how this could happen. This is not a limited resource. It's not a real estate opportunity. There is enough AI for everyone to buy when it becomes useful to do so.

I think FOMO correctly identifies the irrational effort of many companies to jump in without any idea of what the utility might be in any practical sense.

I was responding to the user side mentioned.
You are right. But these are different different types of motivations of the same thing. And there is always context for these motivations.

Its a different thing to sell Trump that LLMs should take over crucial decisions within a government than just using it for some prototyping, code completion at work or to create cat pictures at home.

Take Copilot for example. It was rolled out in different companies, I worked with. Aside of warnings and maybe trainings, I doubt the companies are really able to measure the impact that has. Students are already using the technology to do homework. Schools and universities are sending mixed signals about the results. And then those students enter the workforce with Copilot enabled by default.

At least with companies, its the "free market" that will regulate (unless some company is too big to fail...)

(comment deleted)
It absolutely is destructive. I read an opinion the other day about Microsoft shoving Copilot into every product, and it kinda makes sense. Paraphrasing but: In MS's ideal world, worker 1 drafts a few bullet points and asks Copilot to expand it into a multi-paragraph email. Worker 2 asks Copilot to summarize the email back into bullet points, then acts on it. What's the point? Well, both workers are paying for Copilot licenses, so MS has already won. And management at the firm is happy because "we're using AI, we're so modern." But did it actually help, with anything, at all? Never mind the amount of wasted energy and resources blasting LLM-generated content (that no human will ever read) back and forth.
This course mentions the famous Apple advertisement. Unfortunately it slightly oversells and while I'm sure that's not because this fragment was written by an LLM it is exactly the sort of over-simplification which leads to LLMs generating wild bullshit when they interpolate this "fact" with other "facts" they've been fed, and we ought to strive to do better when writing for humans.

"Describe how prior to 1984, there was no such thing as a graphical user interface, visual desktop, an intuitive menu system, or mouse-based navigation."

Apple were offering a mass market product which had these features so that's important - but there had been "such a thing" for quite some time before that. Douglas Engelbart's "Mother Of All Demos" in 1968 -- Sixteen years earlier shows all the features you mentioned. https://en.wikipedia.org/wiki/The_Mother_of_All_Demos

Unfortunately the demo is very long for a modern audience, so unlike "Watch a Superbowl ad" it's a hard sell to show the entire demo, but do go watch for yourself.

I like the distinction between "teletypes" and the new fancy "glass teletypes".
(comment deleted)
(comment deleted)
You're right of course.

In the original drafts I had a long section on this, including some of the history of the GUI, the development of the mouse, etc. It was way too much for the main text when the point is just to set up a metaphor for students who have seen a Mac 128.

That said, we can and should do better in the instructor guide. Thanks for the reminder. I'll add some context there.

I'm sorry, but this website is awful. Not only does it have an illogical structure (table of contents at the end? no "next lesson" button? gigantic images that fill the entire screen?), but the aesthetic of the entire thing is off. It tries to be sleek and modern with scrolling animations, but they are janky and rigid and the images are rectangles put in front of a bad gradient. Not to mention the video interviews are badly produced (clipping audio, interviewer doesn't have a dedicated microphone) and it's not even clear why they're there.

Please, take inspiration from actual e-learning platforms.

I would like to read this but the jerkiness of needing to scroll 1 page per paragraph renders this unusable
I agree. I tried the first chapter with the Reader Mode in FireFox, and the whole long scroll hell collapsed to about one screenful of text. I have a feeling it skipped some text, but the result was a quick read that got the main points through.

I wish the whole thing was available in a plain text format, preferably in one longer document.

Totally agree here. I visited the page and scrolled through it to see what it was all about and saw a bunch of pull quotes and couldn't work out what I was looking at. It just looks like a light magazine article or brochure until you either click on the hamburger menu to see a full table of contents or scroll to the very bottom to see the table of contents there.

This really needs some design improvements if they want people to read through to the actual lessons. Most people are going to drop off after scrolling through that landing page.