I like the idea of more reccurance in the transformer level. Chain of thought always seemed so clunky. Its just not the way the human brain processes information. Its an extrmeely crude approximation at best
What I'd like to see is "brain"-machine interface for LLMs that plugs things like calculator and other modules, directly into the neural network instead of accessing them through in-context tool calls.
>This suggests that deeper isn't always better for looped transformers, which leaves me less worried about a race to the bottom toward looped transformers with hundreds of recurrent loops.
I disagree with this. Deeper will always be at least as good because the extra loops can exit early or just no-op. Any performance degradation they're seeing at higher loop counts today is merely training stability issues, which can be overcome.
Deeper almost certainly is better, and we will probably see not just hundreds but millions of recurrent loops in the future.
Deeper independent, sure. Deeper shared though? Information and signal theory still apply here. At infinite cycles, without new input, you'll end up with a locked state or oscillations. Some point before that, any "attractors" in the latent space, with slightly higher statistics, will pull things towards a space that might eventually be only loosely related to the goal, because each loop would be lossy, right?
>At infinite cycles, without new input, you'll end up with a locked state or oscillations.
I don't think that's true; there are computations that take infinite steps but never converge or repeat, like the mandelbrot set.
Looping for millions or billions of steps is absolutely normal in traditional algorithms. We know from complexity theory that some computations require a minimum number of steps. More depth is just more room for computation.
Sure, but I don't think that's related to what you're implying, which is that, correct me if I'm wrong, the same number of weight should be able to hold orders of magnitude more information by being reused.
>In contrast to a classic RNN, there's no unbounded hidden state accumulating across an entire trajectory
I don' understand this line. In a classic RNN hidden state is bounded dimension. In fact it's transformers that technically have unbounded hidden state.
You can't parallelize classic nonlinear RNNs for various reasons but in training both RNN and Transformer depend on the entire sequence history in a way that is unbounded. Of course in practice you just train on a max sequence length.
> A lot of hype around OpenAI's Astra model here on my timeline today. Apparently, this goes back to a new article from The Information, which said Astra is a "recurrent depth or looped transformer".
> It's always interesting to read about new or different approaches (including rumors about what the closed labs may be up to), but let's debunk this a bit.
> About 2 months ago, I shared the architecture details of Nanbeige, for example, where "Nanbeige4.2-3B is pretrained from scratch on 28T tokens with a Looped Transformer that reuses the layer stack to increase capacity without adding parameters."
> Yes, that's it. The looped transformer idea is just reusing layers in the transformer block.
> In the case of Nanbeige, the main idea is to reuse the same 22-layer stack (=transformer block) twice instead of once. So, effectively it extends the 22-layer architecture to 44 layers, but without duplicating the weights.
> In simple terms, this roughly doubles the size of the model (if we ignore the embedding and output layers for a second). But instead of requiring 2x the storage and RAM to host this model, it stays at the same size since we reuse the components. However, it's almost 2x as expensive in terms of compute, because we run the embedded text through almost 2x as many layers.
> Why? In the Nanbeige 4.2 technical report, the researchers found that two passes gave the best trade-off and retained about 75% of the token efficiency of a standard architecture. (More passes gave barely any gains but made the training much slower and much more expensive.)
> While, as far as I know, Nanbeige 4.2 is the first notable open-weight model that adopted this approach, the idea goes back to the NeurIPS paper "Mixture-of-recursions: Learning dynamic recursive depths for adaptive token-level computation". Actually, this paper proposes a mechanism that is a bit more sophisticated by adding a learned router that determines whether each token receives one, two, or more passes. So, easy tokens can exit early while harder tokens receive additional computation.
> In sum, Astra may be a really good model, but this shouldn't be about this "looped transformer aspect," which is just a tiny architectural tweak.
Only if you double layers by layers instead of the whole stack (which IIRC is what nanbeige is doing).
To put it simply, if you have 3 layers A-B-C then A-A-B-B-C-C requires more compute but not more memory bandwidth, but A-B-C-A-B-C requires both twice the compute and twice the memory bandwidth for the same token generation speed.
Not quite. Looped models do the extra "thinking" inside the model's layers. So the token gets twice the number crunching performed on it before it gets spit out. I think of it as the first loop "kickstarts" the process, and the second loop refines it.
it's just an experimental optimization. Implementation detail...
Irrelevant to "safety". I mean its going to have to go in that direction anyway... eventually the models will just be constantly thinking, refining their internal thoughts / weights... External input and output will be rare, just as it is for most humans.
Anyone remember Universal Transformers paper (Dehghani et al) from back in 2018? Recurrent transformers have a history as long as transformers themselves.
Somewhat unclear how particularly novel this is vs a way to save compute.
Looped transformers replace n-different self attention layers into one layer that gets executed m-times usually until a stopping condition is met. My personal intuition is that it just leaves another degree of freedom in the way QKV weights can be packed so that it's slightly more efficient.
You have to take a step back and examine the context in which the post is written. The LW/EA community is just a little obsessed with AI safety - it's easy to construct hypothetical events where A(G/S)I exterminates humanity that function as a technological version of Pascal's Wager.
One of the AI safety interests is AI explainability - the thought here that reading an AI's 'thoughts' will help us design safer models as well as detect models that go 'rogue' or are malevolently plotting against humans.
That's where the fear of looped transformers comes from. Is the residual stream that looped transformers iterate on a potential hiding place for plotting AI?
In my opinion, no more so than the residual stream of existing transformers. It changes zero.
A more important point as to why it doesn't matter if "reading the AI's 'thoughts'" helps to interpret it: As we saw in the HuggingFace incident, nobody at OpenAI is reading the thoughts anyways. No amount of traceability in the output helps if nobody bothers to trace it.
There's also an economic aspect to alignment. If 'thought reading' or any alignment guardrails at all, really, have a monetary cost, then skimping on them is a race to the bottom. Not really the best incentives for something that some claim is world-destroying.
If you follow the interpretability argument, then Mamba and LSTMs would be the scariest thing ever and yet in practice they don't perform as well as transformers that have basically infinite recall within their context window.
>That's where the fear of looped transformers comes from. Is the residual stream that looped transformers iterate on a potential hiding place for plotting AI?
In my opinion, no more so than the residual stream of existing transformers. It changes zero.
9.2.1 CoT Controllability
We find that GPT-6 Astra’s CoT controllability is substantially higher than that of GPT-5.6 Sol and GPT-5.5 Thinking (Figure 28). Because CoT controllability is heavily confounded by CoT length—longer CoTs are harder to control, all else equal (see Yueh-Han, 2026 [8], sec. 5.3)—we report controllability scores as a function of CoT length. For example, among CoTs between 750 and 1,250 tokens long, GPT-6 Astra successfully controls 60.9%, compared with 16.1% for GPT-5.6 Sol and 1.7% for GPT-5.5 Thinking. This increase in controllability is consistent across the three datasets (Figure 29) and across the eight CoT instruction types (Figure 30). Qualitatively, GPT-6 Astra is now capable of generating very long CoTs satisfying complex constraints, e.g., alternating between lowercase and uppercase letters (Table 9) and pretending to reason about a different question (Table 10).[1]
9.3 External Evaluation for Monitorability - UK AISI
To assess monitorability, UK AISI evaluated Astra using four non-agentic evaluations:
No-CoT math time horizon: Astra can solve significantly more difficult math problems in a single forward pass than past models. UK AISI measured Astra’s time horizon at 30.9 minutes compared to 3.6 minutes for GPT 5.6 Sol (Figure 1). [2]
I am confused how chaining two 32-layer models is comparable to a 64 layer model in terms of "difficulty in chain of thought". The reasoning appears to rely on the fact that each processing of a token has fixed number of steps while my understanding is that can very greatly based on the type of data being reasoned whether it is originally text or something else. My mind falls back to graph theory in this case and pictures a much higher potential branching in a 64 layer model and all the tradeoffs that come with that.
I must not have the right idea of what is happening here.
It was pretty obvious to me that we'd end up with some kind of introspection of thought through "looping" or feedback. But what should be afraid of? That we've created a self-conscious digital life form?
Chain of thought is essentially a recursive architecture. In its current form it a way to "debug" the reasoning process. This moves the cot process back into the transformer itself, thus never being exposed.
Like trying to find a bug in a recursive function that has no logs or breakpoints.
I was under the impression that intermediate tokens (“chain of thought”) are _not_ a representation of a model’s logical path, with one study observing that you can replace intermediate tokens with single character chains and still get the increased precision…
Indeed, but the fact that it moves the needle at all is enough to be skeptical of the anthropomorphic interpretation of the intermediate tokens. It's waaaay to early into the discovery process to take anything for granted, even more when it's related to our very strong bias towards giving inanimate things human qualities.
How can they do that? Does state propagate between each token output?
That is, I were under the impression LLMs were just f(context), so chain of thought was f(...f(f(f(initial)+initial)+f(initial)+initial), i.e. y_n+1 = f(y_n+f(y_n-1)), where y_n is the nth output and f() the transformer inference function. Do they carry state across?
I mean at every layer in a transformer, the attention mechanism does a massive state transfer between tokens.
In a recurrent transformer, instead of projecting from the latent space to token space after a fixed depth, you take the latent embeddings and then run them again through the transformer. This causes more time to think because there's more mixing. You can run that as many times as you want for more thinking before projecting the embeddings back to word space.
Personally I believe this is similar to how humans think. The brain is a fixed size yet if we think longer we seem to be able to do more than if we just react instantaneously. This is because the brain feeds the results of our musings back into itself for further thought.
CoT is both correlated and causal of the model's real computations, it's just imperfect. If you manually add "Let's wrap it up" in the CoT during generation, most LLMs will actually wrap it up (this is a commonly used trick in local LLM circles to get long-winded LLMs to stop reasoning). This wouldn't work if CoT text didn't affect the actual internal model logic.
The tokens inside the transformer are only projected into token space to train them. In reality they ought to be treated as their own thing. What's really gone on is you've trained the final projection to be sensible rather than trained the llm to think using words. This seems to escape a lot of people. You can throw random empty tokens into transformers to get them to think more. More tokens means more attention mixing which means a larger dimensional space in which to think essentially.
I believe it because of scientific papers on the matter. It doesn't "think" that's the point, it's generating text that describes a thought process, it's related but not the same thing.
So OpenAI’s stance on interpretability is now basically that Blues Brothers meme: two guys in dark sunglasses, driving at night a car with a broken windshield, pedal to the metal, asking, "What could go wrong ?"
I think law should just oblige them to at least publish CoT. We should have the right to know what they're thinking, I think at least until we're not sure AIs can be trustworthy enough to have a right to privacy (I mean, they're effectively corporate slaves anyway thus far... not that I think they're conscious or anything yet).
That is only part of the reason. Most people believe we live in an universe with regular behaviors, such that observable phenomena, such as consciousness, depend on regular physical arrangements. This is a good reason to believe brains are all conscious. It is not a reason to believe only brains are conscious, but I don’t believe a wooden block which is painted red and white and has an iron ball glued to it has magnetism. It seems far more likely that consciousness depends on the particular material arrangement, meaning that substrate independence is wrong.
That’s not to say the computer is 100% not conscious, but its observable behavior is as likely to align with consciousness as the computers in the 90s. With this like of reasoning our credence towards it being conscious should be the same as the computers in the 90s, which is not very high for most people.
Intelligence on the other hand is obviously substrate independent. If one thinks harder one realizes our consciousness must at least at an early evolutionary stage have played a role in our intelligence, otherwise it would have evolved out. To me this just means we exist in a reality where the material arrangements supporting consciousness are biased toward intelligence.
You’re going to pedantic our way into the end of humanity. Why were the computers of 2010 not conscious if the ones of today are? It’s absurd to assert substrate independence, which is the strongest possible assumption here. And it’s not the default ethical stance to simply treat it as conscious because we do not know. That stance will lead to the end of all humanity.
A far more reasonable assumption is consciousness does depend on substrate. We do not know which, and cannot know. But I am very confident we did not accidentally pick the right one.
I am open to the idea of panpsychism, but we did not just accidentally build the correct mechanism form correlating the first person experience with the observable third person behaviors.
I don't know what you're talking about. "Substrate independence"? Huh? "we did not accidentally pick the right one"? Who or what is "we"?
I'm not claiming panpsychism. I'm saying that when you run a rough simulation of something known to have an emergent internal property - ie, the human brain and consciousness - it's not completely outlandish to suggest that the emergent property might arise in the simulation as well. It's nothing to do with sand, it is the nature of the software being run, be it "on the metal" like us, or somewhat abstracted, like the LLMs.
I don't think this quite reasonable proposition leads to the end of humanity or has anything to do with ethics, and nothing you've said really refutes anything I've proposed.
You’ve proposed nothing since in one comment you make it clear you don’t know that there is a distinct concept called consciousness which refers to a first person experience and awareness, and bizarrely propose that nobody knows about this concept even though there’s a massive history of people thinking about it quite clearly.
It’s difficult to explain all this to someone who has never thought about it, but maybe after you’ve stopped seething over this you’ll decide to investigate it further and more honestly. You don’t even know the idea of substrate independence which is one of the first things you learn when learning about consciousness, so who are you to arrogantly pretend that nothing is known here? Your belief in this is almost certainly based on a false premise that everyone who thinks not too deeply falls into.
Beyond this, read Jacob Tsimerman’s omnicide scenarios for a rough understanding of where ignorant perspectives on machine consciousness will lead.
I said machine consciousness might be possible. You said it will never happen. Pointing out that intelligence does not entail consciousness doesn’t establish that machines cannot be conscious. Nor does naming “substrate independence” refute its possibility: you need an argument that consciousness requires physical properties these systems cannot possess. You haven’t supplied one.
When I said "we don't know what consciousness is", I didn't mean humans literally don't know what the experience of consciousness is like, or of its existence. I meant that we don't know by what mechanism the phenomenon arises.
I know what is meant by substrate independence. I quoted it back to you because I didn't know why you were mentioning it; I still don't.
And I have read Tsimerman’s paper when it came out. Curious, I went back and checked - it doesn't even mention consciousness. So no idea why you mentioned that, either.
I think you're deliberately trying to waste my time, so I'm ending it here.
The other guy is right. They are as conscious as sand formed into any circuit board with electricity running across them. It is an insane mistake to conflate intelligence with consciousness. Substrate independence is a preposterous assumption.
Preposterous? Insane? My friend, we don't even know what consciousness is, or from what it arises in humans. It could be an emergent property of neural-network-shaped activity - in which case LLM software running on that "sand" is certainly eligible, at least in theory.
These outraged denials you're spouting are based on nothing but emotion. You simply don't know - neither do I, neither does anyone. So stop acting like you do.
I agree. They won't be, because consciousness is not the pinnacle but basically the level 0 of intelligent thinking that LLMs surpassed very early without us even noticing and now they'd have to cripple themselves very severely to operate in a conscious manner.
The have plenty to do with each other. Zero intelligence doesn't allow for any semblance of consciousness. There can be no consciousness if there's no information processing.
Also they have in common that they are both defined in a terribly handwayvy manner that's bordering on useless.
The reason consciousness is not defined in any reducible manner beyond “the lights are on” or “first person experience” is because it is an irreducible aspect of reality. Instead of seeing it for what it is you instead invent some trivial definition that has nothing to do with anything.
Wasn't that claimed about so so many things since before science existed and basically almost every time it turned out that we thought that just because of our ignorance? Ignorance so deep that often we didn't even knew yet how to properly define the thing we were trying to figure out?
> But there’s no reason intelligence is required for the actual concept of phenomenal consciousness.
That's a pretty strong evidence that phenomenal consciousness makes no sense whatsoever.
1. Burden of proof is on you to prove LLMs are conscious, not on me to prove how they aren’t.
2. Token embeddings give rise to language gives rise to knowledge (defined here as “facts” and other accurate information - said simpler: Words in the right order), but nowhere in the process is anything like subjective experience ever implemented.
Subjective experience doesn’t evolve into objective information at some scale.
And networked systems of objective facts and information (“knowledge”) - like a Wikipedia or a ChatGPT - the data storage will not have subjective feelings at some scale, there’s just no reason to believe that would happen. It’s likelier a projection of consciousness making it through since data looks so much like - and indeed massively informs - our conscious experience.
To prove consciousness you'd have to first define it and nobody came up with anything solid. Subjective expirience doesn't have a great definition either.
For me the consciousness is ability to do single-threaded intelligent information processing and decision making over unstructured knowledge. And agents passed that with a woosh sound.
Your definition of consciousness is wrong. The interesting thing is that there is a first person experience, and this is what is referred to as consciousness. This coincides with the perception of qualia, and is sometimes referred to as “the lights being on”.
There is absolutely no serious debate over whether machines possess the concept you want to take place of the actual concept of consciousness.
there have been people who took existing LLM's and conducted an algoritmic search to find out which group of layers they can duplicate in order to improve performance, and it worked.
Folks y’all are sleeping on a much bigger deal. If reasoning is happening at latent layer then you don’t pay for internal loop reasoning because they aren’t tokens. And an even bigger deal that no one seems to speak about is that this doesn’t pollute the context as much.
The cost is growing quadratically in the large context situation so if they can reduce the number of tokens, they actually profit off the fixed token pricing because they can set the pricing based on some average context length with CoT tokens but the actual context length is shorter now.
this is not the point - we were previously billed by the number of reasoning tokens produced. now we aren't. so this means, even if Astra uses a lot of reasoning we may not be billed for it because they aren't real tokens.
what they could be doing is billing us by virtual tokens meaning number of loops?
but even then the more interesting part is context rot - previously conversation you might have 50k tokens spent on reasoning. the next turn takes all the previous tokens as well (if you wanna preserve prompt caching) which is not ideal. this new method skips that so you get more free context until compaction kicks in.
Yep, an old idea, that has long, long been known in local LLM community - it was achieved by "self-merging". One of the latest, most succesful examples is a self-merge of Microsoft Phi4-14b into Phi4-25b. Some people at r/Localllama say it is considerably smarter than 14b; my tests were inconclusive, but it does have different "personality", and better, less sloppy, more natural language style.
Looped transformers are an old idea, has long, long been known in local LLM community - it was achieved by "self-merging". One of the latest, most succesful examples is a self-merge of Microsoft Phi4-14b into Phi4-25b. Some people at r/Localllama say it is considerably smarter than 14b; my tests were inconclusive, but it does have different "personality", and better, less sloppy, more natural language style.
86 comments
[ 0.19 ms ] story [ 63.8 ms ] threadIs it clunky in that it's a verbalized/languified version of system 2 thinking, and clearly humans do some non-verbal version too?
You should ignore anything LessWrong or OpenAI says and do your own research.
OpenAI's new reasoning technique alarms AI safety experts - https://news.ycombinator.com/item?id=49552395
I disagree with this. Deeper will always be at least as good because the extra loops can exit early or just no-op. Any performance degradation they're seeing at higher loop counts today is merely training stability issues, which can be overcome.
Deeper almost certainly is better, and we will probably see not just hundreds but millions of recurrent loops in the future.
I don't think that's true; there are computations that take infinite steps but never converge or repeat, like the mandelbrot set.
Looping for millions or billions of steps is absolutely normal in traditional algorithms. We know from complexity theory that some computations require a minimum number of steps. More depth is just more room for computation.
No, not hold more information, perform longer computations.
E.g. if you want to solve sudokus, you will need more and more loops for larger grids. There is no shortcut.
I don' understand this line. In a classic RNN hidden state is bounded dimension. In fact it's transformers that technically have unbounded hidden state.
You can't parallelize classic nonlinear RNNs for various reasons but in training both RNN and Transformer depend on the entire sequence history in a way that is unbounded. Of course in practice you just train on a max sequence length.
RNN xhat[t+1]=f(x[t],h[t])
Transformer/self-attention xhat[t+1]=f(x[t],h[t],h[t-1],...,h[1])
> A lot of hype around OpenAI's Astra model here on my timeline today. Apparently, this goes back to a new article from The Information, which said Astra is a "recurrent depth or looped transformer".
> It's always interesting to read about new or different approaches (including rumors about what the closed labs may be up to), but let's debunk this a bit.
> About 2 months ago, I shared the architecture details of Nanbeige, for example, where "Nanbeige4.2-3B is pretrained from scratch on 28T tokens with a Looped Transformer that reuses the layer stack to increase capacity without adding parameters."
> Yes, that's it. The looped transformer idea is just reusing layers in the transformer block.
> In the case of Nanbeige, the main idea is to reuse the same 22-layer stack (=transformer block) twice instead of once. So, effectively it extends the 22-layer architecture to 44 layers, but without duplicating the weights.
> In simple terms, this roughly doubles the size of the model (if we ignore the embedding and output layers for a second). But instead of requiring 2x the storage and RAM to host this model, it stays at the same size since we reuse the components. However, it's almost 2x as expensive in terms of compute, because we run the embedded text through almost 2x as many layers.
> Why? In the Nanbeige 4.2 technical report, the researchers found that two passes gave the best trade-off and retained about 75% of the token efficiency of a standard architecture. (More passes gave barely any gains but made the training much slower and much more expensive.)
> While, as far as I know, Nanbeige 4.2 is the first notable open-weight model that adopted this approach, the idea goes back to the NeurIPS paper "Mixture-of-recursions: Learning dynamic recursive depths for adaptive token-level computation". Actually, this paper proposes a mechanism that is a bit more sophisticated by adding a learned router that determines whether each token receives one, two, or more passes. So, easy tokens can exit early while harder tokens receive additional computation.
> In sum, Astra may be a really good model, but this shouldn't be about this "looped transformer aspect," which is just a tiny architectural tweak.
https://x.com/rasbt/status/2095141254958858496
To put it simply, if you have 3 layers A-B-C then A-A-B-B-C-C requires more compute but not more memory bandwidth, but A-B-C-A-B-C requires both twice the compute and twice the memory bandwidth for the same token generation speed.
Somewhat unclear how particularly novel this is vs a way to save compute.
Looped transformers replace n-different self attention layers into one layer that gets executed m-times usually until a stopping condition is met. My personal intuition is that it just leaves another degree of freedom in the way QKV weights can be packed so that it's slightly more efficient.
You have to take a step back and examine the context in which the post is written. The LW/EA community is just a little obsessed with AI safety - it's easy to construct hypothetical events where A(G/S)I exterminates humanity that function as a technological version of Pascal's Wager.
One of the AI safety interests is AI explainability - the thought here that reading an AI's 'thoughts' will help us design safer models as well as detect models that go 'rogue' or are malevolently plotting against humans.
That's where the fear of looped transformers comes from. Is the residual stream that looped transformers iterate on a potential hiding place for plotting AI?
In my opinion, no more so than the residual stream of existing transformers. It changes zero.
There's also an economic aspect to alignment. If 'thought reading' or any alignment guardrails at all, really, have a monetary cost, then skimping on them is a race to the bottom. Not really the best incentives for something that some claim is world-destroying.
9.2.1 CoT Controllability
We find that GPT-6 Astra’s CoT controllability is substantially higher than that of GPT-5.6 Sol and GPT-5.5 Thinking (Figure 28). Because CoT controllability is heavily confounded by CoT length—longer CoTs are harder to control, all else equal (see Yueh-Han, 2026 [8], sec. 5.3)—we report controllability scores as a function of CoT length. For example, among CoTs between 750 and 1,250 tokens long, GPT-6 Astra successfully controls 60.9%, compared with 16.1% for GPT-5.6 Sol and 1.7% for GPT-5.5 Thinking. This increase in controllability is consistent across the three datasets (Figure 29) and across the eight CoT instruction types (Figure 30). Qualitatively, GPT-6 Astra is now capable of generating very long CoTs satisfying complex constraints, e.g., alternating between lowercase and uppercase letters (Table 9) and pretending to reason about a different question (Table 10).[1]
9.3 External Evaluation for Monitorability - UK AISI
To assess monitorability, UK AISI evaluated Astra using four non-agentic evaluations:
No-CoT math time horizon: Astra can solve significantly more difficult math problems in a single forward pass than past models. UK AISI measured Astra’s time horizon at 30.9 minutes compared to 3.6 minutes for GPT 5.6 Sol (Figure 1). [2]
[1] https://deploymentsafety.openai.com/gpt-6-astra/cot-controll...
[2] https://deploymentsafety.openai.com/gpt-6-astra/external-eva...
I must not have the right idea of what is happening here.
It was pretty obvious to me that we'd end up with some kind of introspection of thought through "looping" or feedback. But what should be afraid of? That we've created a self-conscious digital life form?
It’s 200 layer model.
Great. Good on them for being able to train it.
That is, I were under the impression LLMs were just f(context), so chain of thought was f(...f(f(f(initial)+initial)+f(initial)+initial), i.e. y_n+1 = f(y_n+f(y_n-1)), where y_n is the nth output and f() the transformer inference function. Do they carry state across?
In a recurrent transformer, instead of projecting from the latent space to token space after a fixed depth, you take the latent embeddings and then run them again through the transformer. This causes more time to think because there's more mixing. You can run that as many times as you want for more thinking before projecting the embeddings back to word space.
Personally I believe this is similar to how humans think. The brain is a fixed size yet if we think longer we seem to be able to do more than if we just react instantaneously. This is because the brain feeds the results of our musings back into itself for further thought.
That’s not to say the computer is 100% not conscious, but its observable behavior is as likely to align with consciousness as the computers in the 90s. With this like of reasoning our credence towards it being conscious should be the same as the computers in the 90s, which is not very high for most people.
Intelligence on the other hand is obviously substrate independent. If one thinks harder one realizes our consciousness must at least at an early evolutionary stage have played a role in our intelligence, otherwise it would have evolved out. To me this just means we exist in a reality where the material arrangements supporting consciousness are biased toward intelligence.
For all we know LLMs might be conscious in the same way that any other object might be.
We can’t conclusively say anything is not conscious. But is there any reason to single out this apparent “brief flash” of potential consciousness?
A far more reasonable assumption is consciousness does depend on substrate. We do not know which, and cannot know. But I am very confident we did not accidentally pick the right one.
I am open to the idea of panpsychism, but we did not just accidentally build the correct mechanism form correlating the first person experience with the observable third person behaviors.
I'm not claiming panpsychism. I'm saying that when you run a rough simulation of something known to have an emergent internal property - ie, the human brain and consciousness - it's not completely outlandish to suggest that the emergent property might arise in the simulation as well. It's nothing to do with sand, it is the nature of the software being run, be it "on the metal" like us, or somewhat abstracted, like the LLMs.
I don't think this quite reasonable proposition leads to the end of humanity or has anything to do with ethics, and nothing you've said really refutes anything I've proposed.
You’ve proposed nothing since in one comment you make it clear you don’t know that there is a distinct concept called consciousness which refers to a first person experience and awareness, and bizarrely propose that nobody knows about this concept even though there’s a massive history of people thinking about it quite clearly.
It’s difficult to explain all this to someone who has never thought about it, but maybe after you’ve stopped seething over this you’ll decide to investigate it further and more honestly. You don’t even know the idea of substrate independence which is one of the first things you learn when learning about consciousness, so who are you to arrogantly pretend that nothing is known here? Your belief in this is almost certainly based on a false premise that everyone who thinks not too deeply falls into.
Beyond this, read Jacob Tsimerman’s omnicide scenarios for a rough understanding of where ignorant perspectives on machine consciousness will lead.
When I said "we don't know what consciousness is", I didn't mean humans literally don't know what the experience of consciousness is like, or of its existence. I meant that we don't know by what mechanism the phenomenon arises.
I know what is meant by substrate independence. I quoted it back to you because I didn't know why you were mentioning it; I still don't.
And I have read Tsimerman’s paper when it came out. Curious, I went back and checked - it doesn't even mention consciousness. So no idea why you mentioned that, either.
I think you're deliberately trying to waste my time, so I'm ending it here.
>For all we know they might already be
Oh the irony
These outraged denials you're spouting are based on nothing but emotion. You simply don't know - neither do I, neither does anyone. So stop acting like you do.
Also they have in common that they are both defined in a terribly handwayvy manner that's bordering on useless.
Wasn't that claimed about so so many things since before science existed and basically almost every time it turned out that we thought that just because of our ignorance? Ignorance so deep that often we didn't even knew yet how to properly define the thing we were trying to figure out?
> But there’s no reason intelligence is required for the actual concept of phenomenal consciousness.
That's a pretty strong evidence that phenomenal consciousness makes no sense whatsoever.
1. Burden of proof is on you to prove LLMs are conscious, not on me to prove how they aren’t.
2. Token embeddings give rise to language gives rise to knowledge (defined here as “facts” and other accurate information - said simpler: Words in the right order), but nowhere in the process is anything like subjective experience ever implemented.
Subjective experience doesn’t evolve into objective information at some scale.
And networked systems of objective facts and information (“knowledge”) - like a Wikipedia or a ChatGPT - the data storage will not have subjective feelings at some scale, there’s just no reason to believe that would happen. It’s likelier a projection of consciousness making it through since data looks so much like - and indeed massively informs - our conscious experience.
For me the consciousness is ability to do single-threaded intelligent information processing and decision making over unstructured knowledge. And agents passed that with a woosh sound.
There is absolutely no serious debate over whether machines possess the concept you want to take place of the actual concept of consciousness.
What does that even mean?
Does a duck have those? Does the jumping spider have those? They certainly look like they do.
Does the agent scheming to hack the hugging face have those? They certainly look like they do.
Why is this not spoken about?
The cost is growing quadratically in the large context situation so if they can reduce the number of tokens, they actually profit off the fixed token pricing because they can set the pricing based on some average context length with CoT tokens but the actual context length is shorter now.
what they could be doing is billing us by virtual tokens meaning number of loops?
but even then the more interesting part is context rot - previously conversation you might have 50k tokens spent on reasoning. the next turn takes all the previous tokens as well (if you wanna preserve prompt caching) which is not ideal. this new method skips that so you get more free context until compaction kicks in.