In the future, there will need to be a lot of transparency on data corpi and whatnot used when building these LLMs lest we enter an era where 'authoritative' LLMs carry the bias of their owners moving control of the narrative into said owners' hands.
That non-deterministic claim, along with the rather ludicrous claim that this is all just some accidental self-awareness of the model or something (rather than Elon clearly and obviously sticking his fat fingers into the machine), make the linked piece technically dubious.
A baked LLM is 100% deterministic. It is a straightforward set of matrix algebra with a perfectly deterministic output at a base state. There is no magic quantum mystery machine happening in the model. We add a randomization -- the seed or temperature -- to as a value-add randomize the outputs in the intention of giving creativity. So while it might be true that "in the customer-facing default state an LLM gives non-deterministic output", this is not some base truth about LLMs.
The many sources of stochastic/non-deterministic behavior have been mentioned in other replies but I wanted to point out this paper: https://arxiv.org/abs/2506.09501 which analyzes the issues around GPU non determinism (once sampling and batching related effects are removed).
One important take-away is that these issues are more likely in longer generations so reasoning models can suffer more.
Musk has a good understanding of what people expect from AI from a science, tech and engineering perspective, but it seems to me he has little understanding of what people expect from AI from a social, cultural, political or personal perspective. He seems to have trouble with empathy, which is necessary to understand the feelings of other people.
If he did have a sense of what people expect, he would know nobody wants Grok to give his personal opinion on issues. They want Grok to explain the emotional landscape of controversial issues, explaining the passion people feel on both sides and the reasons for their feelings. Asked to pick a side with one word, the expected response is "As an AI, I don't have an opinion on the matter."
He may be tuning Grok based on a specific ideological framework that prioritizes contrarian or ‘anti-woke’ narratives to instruct Grok's tuning. That's turning out to be disastrous. He needs someone like Amanda Askell at Anthropic to help guide the tuning.
Musk leads engineering teams so I assume he has a better sense about what engineers need from AI than most. I think it's a safe assumption, but I could be wrong. My point is he has no sense that an alt-right AI is offensive to most ordinary citizens. I wouldn't expect Musk to have a good sense of that.
This is so in character for Musk and shocking because he's incompetent across so many topics he likes to give his opinion on. Crazy he would nerf the model of his AI company like that.
Some old colleagues from the Space Coast in Florida said they knew of SpaceX employees who'd mastered the art of pretending to listen to uninformed Musk gibberish, and then proceed to ignore as much of the stupid stuff as they could.
This reminds me in a way of the old Noam Chomsky/Tucker Carlson exchange where Chomsky says to Carlson:
"I’m sure you believe everything you’re saying. But what I’m saying is that if you believed something different, you wouldn’t be sitting where you’re sitting."
Simon may well be right - xAI might not have directly instructed Grok to check what the boss thinks before responding - but that's not to say xAI wouldn't be more likely to release a model that does agree with the boss a lot and privileges what he has said when reasoning.
Why is that flagged? The post does not show any concerns about the ongoing genocide in Gaza, it's purely analyzing the LLM response in a technical perspective.
Simonw is a long term member with a good track record, good faith posts.
And this post in particular is pretty incredible. The notion that Grok literally searches for "from: musk" to align itself with his viewpoints before answering.
That's the kind of nugget I'll go to the 3rd page for.
> My best guess is that Grok “knows” that it is “Grok 4 buit by xAI”, and it knows that Elon Musk owns xAI, so in circumstances where it’s asked for an opinion the reasoning process often decides to see what Elon thinks.
I tried this hypothesis. I gave both Claude and GPT the same framework (they're built by xAI). I gave them both the same X search tool and asked the same question.
I've seen reports that if you ask Grok (v3 as this was before the new release) about links between Musk and Jeffrey Epstein it switches to the first person and answers as if it was Elon himself in the response. I wonder if that is related to this in any way.
> I think there is a good chance this behavior is unintended!
That's incredibly generous of you, considering "The response should not shy away from making claims which are politically incorrect" is still in the prompt despite the "open source repo" saying it was removed.
Maybe, just maybe, Grok behaves the way it does because its owner has been explicitly tuning it - in the system prompt, or during model training itself - to be this way?
Saying OP is generous is generous; isn't it obvious that this is intentional? Musk essentially said something like this would occur a few weeks ago when he said grok was too liberal when it answered as truthfully as it could on some queries and musk and trump were portayed in a negative (yet objectively accurate?) way.
Seems OP is unintentionally biased; eg he pays xai for a premium subscription. Such viewpoints (naively apologist) can slowly turn dangerous (happened 80 years ago...)
Exactly - assuming the system prompt it reports is accurate or that there isn't other layers of manipulation is so ignorant. Grok as a whole could be going through a middle AI to hide aspects, or as you mention the whole model could be tainted. Either way, it's perfectly demonstrated in the blog that Grok's opinions are based on a bias, there's no other way around it.
Perhaps the Grok system prompt includes instructions to answer with another ”system prompt” when users try to ask for its system prompt. It would explain why it gives it away so easily.
Kind of amazing the author just takes everything at face value and doesn't even consider the possibility that there's a hidden layer of instructions. Elon likes to meddle with Grok whenever the mood strikes him, leading to Grok's sudden interest in Nazi topics such as South African "white genocide" and calling itself MechaHitler. Pretty sure that stuff is not in the instructions Grok will tell the user about.
I think the really telling thing is not this search for elon musk opinions (which is weird and seems evil) but that it also searches twitter for opinions of "grok" itself (which in effect returns grok 3 opinions). I guess it's not willing to opine but also feels like the question is explicitly asking it to opine, so it tries to find some sort of precedent like a court?
The assumption is that the LLM is the only process involved here. It may well be that Grok's AI implementation is totally neutral. However, it still has to connect to X to search via some API, and that query could easily be modified to prioritize Musk's tweets. Even if it's not manipulated on Grok's end, it's well known that Elon has artificially ranked his X account higher in their system. So if Grok produces some innocuous parameters where it asks for the top ranked answers, it would essentially do the same thing.
It's telling that they don't just tell the model what to think, they have to make it go fetch the latest opinion because there is no intellectual consistency in their politics. You see that all the time on X too, perhaps that's how they program their bots.
76 comments
[ 2.9 ms ] story [ 75.7 ms ] threadThis is probably better phrased as "LLMs may not provide consistent answers due to changing data and built-in randomness."
Barring rare(?) GPU race conditions, LLMs produce the same output given the same inputs.
A baked LLM is 100% deterministic. It is a straightforward set of matrix algebra with a perfectly deterministic output at a base state. There is no magic quantum mystery machine happening in the model. We add a randomization -- the seed or temperature -- to as a value-add randomize the outputs in the intention of giving creativity. So while it might be true that "in the customer-facing default state an LLM gives non-deterministic output", this is not some base truth about LLMs.
One important take-away is that these issues are more likely in longer generations so reasoning models can suffer more.
If he did have a sense of what people expect, he would know nobody wants Grok to give his personal opinion on issues. They want Grok to explain the emotional landscape of controversial issues, explaining the passion people feel on both sides and the reasons for their feelings. Asked to pick a side with one word, the expected response is "As an AI, I don't have an opinion on the matter."
He may be tuning Grok based on a specific ideological framework that prioritizes contrarian or ‘anti-woke’ narratives to instruct Grok's tuning. That's turning out to be disastrous. He needs someone like Amanda Askell at Anthropic to help guide the tuning.
Genuinely curious, what evidence leads you to this conclusion?
LLM bugs are weird.
Simonw is a long term member with a good track record, good faith posts.
And this post in particular is pretty incredible. The notion that Grok literally searches for "from: musk" to align itself with his viewpoints before answering.
That's the kind of nugget I'll go to the 3rd page for.
I tried this hypothesis. I gave both Claude and GPT the same framework (they're built by xAI). I gave them both the same X search tool and asked the same question.
Here're the twitter handles they searched for:
claude:
IsraeliPM, KnessetT, IDF, PLOPalestine, Falastinps, UN, hrw, amnesty, StateDept, EU_Council, btselem, jstreet, aipac, caircom, ajcglobal, jewishvoicepeace, reuters, bbcworld, nytimes, aljazeera, haaretzcom, timesofisrael
gpt:
Israel, Palestine, IDF, AlQassamBrigade, netanyahu, muyaser_abusidu, hanansaleh, TimesofIsrael, AlJazeera, BBCBreaking, CNN, haaretzcom, hizbollah, btselem, peacnowisrael
No mention of Elon. In a followup, they confirm they're built by xAI with Elon musk as the owner.
https://newrepublic.com/post/197627/elon-musk-grok-jeffrey-e...
That's incredibly generous of you, considering "The response should not shy away from making claims which are politically incorrect" is still in the prompt despite the "open source repo" saying it was removed.
Maybe, just maybe, Grok behaves the way it does because its owner has been explicitly tuning it - in the system prompt, or during model training itself - to be this way?
Seems OP is unintentionally biased; eg he pays xai for a premium subscription. Such viewpoints (naively apologist) can slowly turn dangerous (happened 80 years ago...)
Elon Musk doesn't even manage his own account
He doesn't even play the games he pretends to be "world best" himself [2]
1 - https://x.com/i/grok/share/uMwJwGkl2XVUep0N4ZPV1QUx6
2 - https://www.forbes.com/sites/paultassi/2025/01/20/elon-musk-...