I like the idea of more reccurance in the transformer level. Chain of thought always seemed so clunky. Its just not the way the human brain processes information. Its an extrmeely crude approximation at best
One thing about the reasoning is that models are trained to generate a chain of thoughts, but it doesn’t have to be correct, accurate, or reflect the underlying logic of the LLM. It’s the same problem we have with the output, it is something plausible, but not that reliable
Yes, both the output should be "milestones" of sorts, like lemmas and theorems in math. Important plateaus that serve as a launching pad to the next phase. Regurgitating every thought potentially degrades signal:noise ratio.
Actually, how does chain of thought work? Is the LLM actually creating the tokens and then re-reading them, or is the there still a full hidden state under the hood and then the UI just prints that projection?
“For example, if a philosophical zombie were poked with a sharp object, it would not feel any pain, but it would react exactly the way any conscious human would”
Doing non-verbal thinking has nothing to do with not feeling pain, I don’t think it sounds like philosophical zombies at all.
Maybe, I wouldn’t know. I just find it fascinating that other people have such a radically different way of experiencing thought. No idea what it’s like.
What I'd like to see is "brain"-machine interface for LLMs that plugs things like calculator and other modules, directly into the neural network instead of accessing them through in-context tool calls.
I was under the impression that intermediate tokens (“chain of thought”) are _not_ a representation of a model’s logical path, with one study observing that you can replace intermediate tokens with single character chains and still get the increased precision…
CoT is both correlated and causal of the model's real computations, it's just imperfect. If you manually add "Let's wrap it up" in the CoT during generation, most LLMs will actually wrap it up (this is a commonly used trick in local LLM circles to get long-winded LLMs to stop reasoning). This wouldn't work if CoT text didn't affect the actual internal model logic.
How can they do that? Does state propagate between each token output?
That is, I were under the impression LLMs were just f(context), so chain of thought was f(...f(f(f(initial)+initial)+f(initial)+initial), i.e. y_n+1 = f(y_n+f(y_n-1)), where y_n is the nth output and f() the transformer inference function. Do they carry state across?
I mean at every layer in a transformer, the attention mechanism does a massive state transfer between tokens.
In a recurrent transformer, instead of projecting from the latent space to token space after a fixed depth, you take the latent embeddings and then run them again through the transformer. This causes more time to think because there's more mixing. You can run that as many times as you want for more thinking before projecting the embeddings back to word space.
Personally I believe this is similar to how humans think. The brain is a fixed size yet if we think longer we seem to be able to do more than if we just react instantaneously. This is because the brain feeds the results of our musings back into itself for further thought.
The model they use in that paper is a toy model of an LLM that’s so different from an actual LLM that I doubt the results mean anything at all.
Specifically, they train a model from scratch. The model architecture is apparently based on Llama but the size is 34M parameters. Not 34B, 34M. This is a fraction of the size of GPT-2.
Luckily, they don’t use the model as a language model. It neither receives text, generates text, nor uses text to think. Instead the inputs are strings like “A01 B10 C73 D27”, and the only possible outputs are “True” and “False”. They are expecting the model to solve a specific math problem encoded by those numbers, and do nothing else. The chain of thought is also numbers, in the scenario that’s supposed to represent a real chain of thought (as opposed to the filler-token scenario and the no-CoT scenario). The numbers in question are manually trained into the model based on one possible algorithmic decomposition of the problem; the model does not learn to generate its own CoT.
Even with all those limitations, for their main problem (3SUM), they only show that filler tokens are better than no CoT at all. They don’t show how that compares to ‘real’ CoT, at least as far as I can see (admittedly I only skimmed). They do make this comparison for their easier problem (2SUM), but on that problem both filler token CoT and ‘real’ CoT are mostly saturated, so the results don’t mean much.
The tokens inside the transformer are only projected into token space to train them. In reality they ought to be treated as their own thing. What's really gone on is you've trained the final projection to be sensible rather than trained the llm to think using words. This seems to escape a lot of people. You can throw random empty tokens into transformers to get them to think more. More tokens means more attention mixing which means a larger dimensional space in which to think essentially.
> A lot of hype around OpenAI's Astra model here on my timeline today. Apparently, this goes back to a new article from The Information, which said Astra is a "recurrent depth or looped transformer".
> It's always interesting to read about new or different approaches (including rumors about what the closed labs may be up to), but let's debunk this a bit.
> About 2 months ago, I shared the architecture details of Nanbeige, for example, where "Nanbeige4.2-3B is pretrained from scratch on 28T tokens with a Looped Transformer that reuses the layer stack to increase capacity without adding parameters."
> Yes, that's it. The looped transformer idea is just reusing layers in the transformer block.
> In the case of Nanbeige, the main idea is to reuse the same 22-layer stack (=transformer block) twice instead of once. So, effectively it extends the 22-layer architecture to 44 layers, but without duplicating the weights.
> In simple terms, this roughly doubles the size of the model (if we ignore the embedding and output layers for a second). But instead of requiring 2x the storage and RAM to host this model, it stays at the same size since we reuse the components. However, it's almost 2x as expensive in terms of compute, because we run the embedded text through almost 2x as many layers.
> Why? In the Nanbeige 4.2 technical report, the researchers found that two passes gave the best trade-off and retained about 75% of the token efficiency of a standard architecture. (More passes gave barely any gains but made the training much slower and much more expensive.)
> While, as far as I know, Nanbeige 4.2 is the first notable open-weight model that adopted this approach, the idea goes back to the NeurIPS paper "Mixture-of-recursions: Learning dynamic recursive depths for adaptive token-level computation". Actually, this paper proposes a mechanism that is a bit more sophisticated by adding a learned router that determines whether each token receives one, two, or more passes. So, easy tokens can exit early while harder tokens receive additional computation.
> In sum, Astra may be a really good model, but this shouldn't be about this "looped transformer aspect," which is just a tiny architectural tweak.
Only if you double layers by layers instead of the whole stack (which IIRC is what nanbeige is doing).
To put it simply, if you have 3 layers A-B-C then A-A-B-B-C-C requires more compute but not more memory bandwidth, but A-B-C-A-B-C requires both twice the compute and twice the memory bandwidth for the same token generation speed.
>In contrast to a classic RNN, there's no unbounded hidden state accumulating across an entire trajectory
I don' understand this line. In a classic RNN hidden state is bounded dimension. In fact it's transformers that technically have unbounded hidden state.
You can't parallelize classic nonlinear RNNs for various reasons but in training both RNN and Transformer depend on the entire sequence history in a way that is unbounded. Of course in practice you just train on a max sequence length.
So OpenAI’s stance on interpretability (ai safety) is now basically that Blues Brothers meme: two guys in dark sunglasses, driving at night in a car with a broken windshield, pedal to the metal, asking, "What could possibly go wrong ?"
I think law should just oblige them to at least publish CoT. We should have the right to know what they're thinking, I think at least until we're not sure AIs can be trustworthy enough to have a right to privacy (I mean, they're effectively corporate slaves anyway thus far... not that I think they're conscious or anything yet).
Anyone remember Universal Transformers paper (Dehghani et al) from back in 2018? Recurrent transformers have a history as long as transformers themselves.
Somewhat unclear how particularly novel this is vs a way to save compute.
there have been people who took existing LLM's and conducted an algoritmic search to find out which group of layers they can duplicate in order to improve performance, and it worked.
I am confused how chaining two 32-layer models is comparable to a 64 layer model in terms of "difficulty in chain of thought". The reasoning appears to rely on the fact that each processing of a token has fixed number of steps while my understanding is that can very greatly based on the type of data being reasoned whether it is originally text or something else. My mind falls back to graph theory in this case and pictures a much higher potential branching in a 64 layer model and all the tradeoffs that come with that.
I must not have the right idea of what is happening here.
>This suggests that deeper isn't always better for looped transformers, which leaves me less worried about a race to the bottom toward looped transformers with hundreds of recurrent loops.
I disagree with this. Deeper will always be at least as good because the extra loops can exit early or just no-op. Any performance degradation they're seeing at higher loop counts today is merely training stability issues, which can be overcome.
Deeper almost certainly is better, and we will probably see not just hundreds but millions of recurrent loops in the future.
Deeper independent, sure. Deeper shared though? Information and signal theory still apply here. At infinite cycles, without new input, you'll end up with a locked state or oscillations. Some point before that, any "attractors" in the latent space, with slightly higher statistics, will pull things towards a space that might eventually be only loosely related to the goal, because each loop would be lossy, right?
>At infinite cycles, without new input, you'll end up with a locked state or oscillations.
I don't think that's true; there are computations that take infinite steps but never converge or repeat, like the mandelbrot set.
Looping for millions or billions of steps is absolutely normal in traditional algorithms. We know from complexity theory that some computations require a minimum number of steps. More depth is just more room for computation.
It was pretty obvious to me that we'd end up with some kind of introspection of thought through "looping" or feedback. But what should be afraid of? That we've created a self-conscious digital life form?
Chain of thought is essentially a recursive architecture. In its current form it a way to "debug" the reasoning process. This moves the cot process back into the transformer itself, thus never being exposed.
Like trying to find a bug in a recursive function that has no logs or breakpoints.
My (layman) understanding is, that currently there is a way to monitor the "thoughts" of the LLMs and that by looping more you lose that ability. The danger is presumably an AI that escapes human oversight.
it's just an experimental optimization. Implementation detail...
Irrelevant to "safety". I mean its going to have to go in that direction anyway... eventually the models will just be constantly thinking, refining their internal thoughts / weights... External input and output will be rare, just as it is for most humans.
Looped transformers replace n-different self attention layers into one layer that gets executed m-times usually until a stopping condition is met. My personal intuition is that it just leaves another degree of freedom in the way QKV weights can be packed so that it's slightly more efficient.
You have to take a step back and examine the context in which the post is written. The LW/EA community is just a little obsessed with AI safety - it's easy to construct hypothetical events where A(G/S)I exterminates humanity that function as a technological version of Pascal's Wager.
One of the AI safety interests is AI explainability - the thought here that reading an AI's 'thoughts' will help us design safer models as well as detect models that go 'rogue' or are malevolently plotting against humans.
That's where the fear of looped transformers comes from. Is the residual stream that looped transformers iterate on a potential hiding place for plotting AI?
In my opinion, no more so than the residual stream of existing transformers. It changes zero.
A more important point as to why it doesn't matter if "reading the AI's 'thoughts'" helps to interpret it: As we saw in the HuggingFace incident, nobody at OpenAI is reading the thoughts anyways. No amount of traceability in the output helps if nobody bothers to trace it.
You could imagine large number of loops, thousands. But you are constrained by the width of the residual stream since you loop over one token. But then you can imagine the model learning to sub-divide it to pack even more info into it.
I can imagine a lot of things. However there is a packing limit for QKV weights that sets the ceiling on how much this occurs, and it's quite low - think 1.3-1.8x. The limiting factor here isn't the number of iterations, it's model size, same as it ever was.
To add a bit more constructive feedback, think of the "AI cornucopia" and "Superintelligence destroys humanity" as opposite ends of all possible outcomes distribution (low probability event).
The most likely one is the "business as usual, but with AI" - some things will get better, some things will get worse, but overall state of affairs will remain mostly the same.
The people who've thought the most about this put it differently:
Think of a new, superintelligent model as if it was a new v1 Starship launching for the first time, with a full fuel tank. On the one hand, rockets have existed for some time, and some have gone to space successfully, including by this company.
On the other hand, this is a tube of metal full of highly explosive liquid going faster than most human objects ever go, for the first time ever in this novel and state of the art configuration.
If someone said, "really, the first Starship exploding is just at one end of the probability distribution, where the other is that everything goes fine and all its passengers have a nice trip in space," would you get on that rocket?
Or, more aptly, if you and every other living human was already on that rocket, would you push the launch button?
The analogy works because superintelligence is, like rocket fuel, an extremely powerful force that has a default tendency to break containment and go boom (consume lots of energy and heat and matter in a chain reaction, to pursue more intelligence to pursue whatever goal it is pursuing.)
This is the sort of pseudo-scientific reasoning by analogy that leads Elizier Yudkowsky to argue we should be terrified that an unfriendly ASI could rapidly develop "diamondide" nanobot viruses, because it will be super-intelligent and super-intelligence can do anything by first-principles:
> The concrete example I usually use here is nanotech, because there's been pretty detailed analysis of what definitely look like physically attainable lower bounds on what should be possible with nanotech, and those lower bounds are sufficient to carry the point.
...
> The nanomachinery builds diamondoid bacteria, that replicate with solar power and atmospheric CHON, maybe aggregate into some miniature rockets or jets so they can ride the jetstream to spread across the Earth's atmosphere, get into human bloodstreams and hide, strike on a timer.
Thought experiments ungrounded by any realistic technological constraints or scientific evidence are pretty much useless for actual forecasting except as an exercise in sci-fi worldbuilding.
We can learn from history: Albert Einstein famously tricked humanity into building nuclear weapons for him and was only prevented from wiping out all sentient life by the Princeton IAS Board of Alignment who published a very compelling blog post about realigning the A-bomb contra paperclips.
Eh. An intelligence with perfect knowledge of the entire universe, unlimited memory, and infinite processing speed, would have a tendency to go boom. But the real world has limits, and intelligence, even superintelligence, does not equate to godhood. Some tasks are still hard no matter how smart you are.
Plus, we have no real reason to think LLMs are anywhere close to AGI or ASI. So arguments like these are just distracting from the very real, very present danger that LLMs pose: information breakdown, societal collapse and environmental destruction. In other words, this is criti-hype.
Current AIs are closer to bottle rockets than to the Starship on the intelligence scale. Some property damage already happened, but you can't master the art of rocketry without trial and error.
But will “business as usual, but with AI” justify the current capital expenditures? I think the market is pricing things as being closer to “AI cornucopia”.
I like the idea of more reccurance in the transformer level. Chain of thought always seemed so clunky. Its just not the way the human brain processes information. Its an extrmeely crude approximation at best
it is what I do to solve hard problems through.
easy stuff happens by itself, but with a system large enough you need a scratchpad and a rubber duck.
One thing about the reasoning is that models are trained to generate a chain of thoughts, but it doesn’t have to be correct, accurate, or reflect the underlying logic of the LLM. It’s the same problem we have with the output, it is something plausible, but not that reliable
I do the same thing in my head. There is no underlying logic to an llm. Logic is an external construct alien to human like forms of reasoning.
Yes, both the output should be "milestones" of sorts, like lemmas and theorems in math. Important plateaus that serve as a launching pad to the next phase. Regurgitating every thought potentially degrades signal:noise ratio.
The hidden states of the tokens likely contain more semantic information than can be extracted by the final projection into token space.
Actually, how does chain of thought work? Is the LLM actually creating the tokens and then re-reading them, or is the there still a full hidden state under the hood and then the UI just prints that projection?
Chain of thought seems very specifically like an instantiation of the System 2 thinking of the System 1 and 2 thinking Daniel Kahneman popularized.
Is it clunky in that it's a verbalized/languified version of system 2 thinking, and clearly humans do some non-verbal version too?
Some humans apparently only or almost only do non-verbal thinking. It’s called anendophasia.
Sounds like philosophical zombies.
https://en.wikipedia.org/wiki/Philosophical_zombie
“For example, if a philosophical zombie were poked with a sharp object, it would not feel any pain, but it would react exactly the way any conscious human would”
Doing non-verbal thinking has nothing to do with not feeling pain, I don’t think it sounds like philosophical zombies at all.
Maybe, I wouldn’t know. I just find it fascinating that other people have such a radically different way of experiencing thought. No idea what it’s like.
What I'd like to see is "brain"-machine interface for LLMs that plugs things like calculator and other modules, directly into the neural network instead of accessing them through in-context tool calls.
Why? I talk in my head and then enunciate only that which is relevant. My speech rate inside is incredibly fast.
The purpose of the scratch pad is very often to refer back to it later.
I was under the impression that intermediate tokens (“chain of thought”) are _not_ a representation of a model’s logical path, with one study observing that you can replace intermediate tokens with single character chains and still get the increased precision…
CoT is both correlated and causal of the model's real computations, it's just imperfect. If you manually add "Let's wrap it up" in the CoT during generation, most LLMs will actually wrap it up (this is a commonly used trick in local LLM circles to get long-winded LLMs to stop reasoning). This wouldn't work if CoT text didn't affect the actual internal model logic.
How can they do that? Does state propagate between each token output?
That is, I were under the impression LLMs were just f(context), so chain of thought was f(...f(f(f(initial)+initial)+f(initial)+initial), i.e. y_n+1 = f(y_n+f(y_n-1)), where y_n is the nth output and f() the transformer inference function. Do they carry state across?
I mean at every layer in a transformer, the attention mechanism does a massive state transfer between tokens.
In a recurrent transformer, instead of projecting from the latent space to token space after a fixed depth, you take the latent embeddings and then run them again through the transformer. This causes more time to think because there's more mixing. You can run that as many times as you want for more thinking before projecting the embeddings back to word space.
Personally I believe this is similar to how humans think. The brain is a fixed size yet if we think longer we seem to be able to do more than if we just react instantaneously. This is because the brain feeds the results of our musings back into itself for further thought.
Yes dots increases precision, but not nearly the same increase in precision as having actual useful reasoning in the CoT
Link to this study?
[2404.15758] Let's Think Dot by Dot: Hidden Computation in Transformer Language Models
https://arxiv.org/abs/2404.15758
The model they use in that paper is a toy model of an LLM that’s so different from an actual LLM that I doubt the results mean anything at all.
Specifically, they train a model from scratch. The model architecture is apparently based on Llama but the size is 34M parameters. Not 34B, 34M. This is a fraction of the size of GPT-2.
Luckily, they don’t use the model as a language model. It neither receives text, generates text, nor uses text to think. Instead the inputs are strings like “A01 B10 C73 D27”, and the only possible outputs are “True” and “False”. They are expecting the model to solve a specific math problem encoded by those numbers, and do nothing else. The chain of thought is also numbers, in the scenario that’s supposed to represent a real chain of thought (as opposed to the filler-token scenario and the no-CoT scenario). The numbers in question are manually trained into the model based on one possible algorithmic decomposition of the problem; the model does not learn to generate its own CoT.
Even with all those limitations, for their main problem (3SUM), they only show that filler tokens are better than no CoT at all. They don’t show how that compares to ‘real’ CoT, at least as far as I can see (admittedly I only skimmed). They do make this comparison for their easier problem (2SUM), but on that problem both filler token CoT and ‘real’ CoT are mostly saturated, so the results don’t mean much.
Also 2504.09762: Stop Anthropomorphizing Intermediate Tokens as Reasoning/Thinking Traces!
https://arxiv.org/abs/2504.09762
The tokens inside the transformer are only projected into token space to train them. In reality they ought to be treated as their own thing. What's really gone on is you've trained the final projection to be sensible rather than trained the llm to think using words. This seems to escape a lot of people. You can throw random empty tokens into transformers to get them to think more. More tokens means more attention mixing which means a larger dimensional space in which to think essentially.
Sebastian Raschka posted about this architecture:
> A lot of hype around OpenAI's Astra model here on my timeline today. Apparently, this goes back to a new article from The Information, which said Astra is a "recurrent depth or looped transformer".
> It's always interesting to read about new or different approaches (including rumors about what the closed labs may be up to), but let's debunk this a bit.
> About 2 months ago, I shared the architecture details of Nanbeige, for example, where "Nanbeige4.2-3B is pretrained from scratch on 28T tokens with a Looped Transformer that reuses the layer stack to increase capacity without adding parameters."
> Yes, that's it. The looped transformer idea is just reusing layers in the transformer block.
> In the case of Nanbeige, the main idea is to reuse the same 22-layer stack (=transformer block) twice instead of once. So, effectively it extends the 22-layer architecture to 44 layers, but without duplicating the weights.
> In simple terms, this roughly doubles the size of the model (if we ignore the embedding and output layers for a second). But instead of requiring 2x the storage and RAM to host this model, it stays at the same size since we reuse the components. However, it's almost 2x as expensive in terms of compute, because we run the embedded text through almost 2x as many layers.
> Why? In the Nanbeige 4.2 technical report, the researchers found that two passes gave the best trade-off and retained about 75% of the token efficiency of a standard architecture. (More passes gave barely any gains but made the training much slower and much more expensive.)
> While, as far as I know, Nanbeige 4.2 is the first notable open-weight model that adopted this approach, the idea goes back to the NeurIPS paper "Mixture-of-recursions: Learning dynamic recursive depths for adaptive token-level computation". Actually, this paper proposes a mechanism that is a bit more sophisticated by adding a learned router that determines whether each token receives one, two, or more passes. So, easy tokens can exit early while harder tokens receive additional computation.
> In sum, Astra may be a really good model, but this shouldn't be about this "looped transformer aspect," which is just a tiny architectural tweak.
https://x.com/rasbt/status/2095141254958858496
Sounds this like this will be a huge win for local models, since generally they're ram limited but have compute to spare
Only if you double layers by layers instead of the whole stack (which IIRC is what nanbeige is doing).
To put it simply, if you have 3 layers A-B-C then A-A-B-B-C-C requires more compute but not more memory bandwidth, but A-B-C-A-B-C requires both twice the compute and twice the memory bandwidth for the same token generation speed.
I don't understand why he's citing a NeurIPS 2025 paper when cross-layer parameter sharing was introduced in ALBERT already in 2019.
if only tweeting is as rigorous as academic literature reviews
>In contrast to a classic RNN, there's no unbounded hidden state accumulating across an entire trajectory
I don' understand this line. In a classic RNN hidden state is bounded dimension. In fact it's transformers that technically have unbounded hidden state.
You can't parallelize classic nonlinear RNNs for various reasons but in training both RNN and Transformer depend on the entire sequence history in a way that is unbounded. Of course in practice you just train on a max sequence length.
RNN xhat[t+1]=f(x[t],h[t])
Transformer/self-attention xhat[t+1]=f(x[t],h[t],h[t-1],...,h[1])
On further thought, I think the author's intent was to say that classic RNNs have "unbounded temporal accumulation in the hidden state".
Related ongoing thread:
OpenAI's new reasoning technique alarms AI safety experts - https://news.ycombinator.com/item?id=49552395
[delayed]
So OpenAI’s stance on interpretability (ai safety) is now basically that Blues Brothers meme: two guys in dark sunglasses, driving at night in a car with a broken windshield, pedal to the metal, asking, "What could possibly go wrong ?"
I think law should just oblige them to at least publish CoT. We should have the right to know what they're thinking, I think at least until we're not sure AIs can be trustworthy enough to have a right to privacy (I mean, they're effectively corporate slaves anyway thus far... not that I think they're conscious or anything yet).
They’re not ever going to be conscious.
We can’t even prove humans are conscious, we just extend the assumption to each other because we want other people to do the same to us.
That is a completely unsupportable claim. For all we know they might already be, for a brief flash at least.
They are made of sand. For all we know the sand was conscious before we smashed it up and arranged it into computer chips.
Anyone remember Universal Transformers paper (Dehghani et al) from back in 2018? Recurrent transformers have a history as long as transformers themselves.
Somewhat unclear how particularly novel this is vs a way to save compute.
there have been people who took existing LLM's and conducted an algoritmic search to find out which group of layers they can duplicate in order to improve performance, and it worked.
I am confused how chaining two 32-layer models is comparable to a 64 layer model in terms of "difficulty in chain of thought". The reasoning appears to rely on the fact that each processing of a token has fixed number of steps while my understanding is that can very greatly based on the type of data being reasoned whether it is originally text or something else. My mind falls back to graph theory in this case and pictures a much higher potential branching in a 64 layer model and all the tradeoffs that come with that.
I must not have the right idea of what is happening here.
>This suggests that deeper isn't always better for looped transformers, which leaves me less worried about a race to the bottom toward looped transformers with hundreds of recurrent loops.
I disagree with this. Deeper will always be at least as good because the extra loops can exit early or just no-op. Any performance degradation they're seeing at higher loop counts today is merely training stability issues, which can be overcome.
Deeper almost certainly is better, and we will probably see not just hundreds but millions of recurrent loops in the future.
Deeper independent, sure. Deeper shared though? Information and signal theory still apply here. At infinite cycles, without new input, you'll end up with a locked state or oscillations. Some point before that, any "attractors" in the latent space, with slightly higher statistics, will pull things towards a space that might eventually be only loosely related to the goal, because each loop would be lossy, right?
>At infinite cycles, without new input, you'll end up with a locked state or oscillations.
I don't think that's true; there are computations that take infinite steps but never converge or repeat, like the mandelbrot set.
Looping for millions or billions of steps is absolutely normal in traditional algorithms. We know from complexity theory that some computations require a minimum number of steps. More depth is just more room for computation.
It is a complete non-issue.
It’s 200 layer model.
Great. Good on them for being able to train it.
Concerned about what exactly?
It was pretty obvious to me that we'd end up with some kind of introspection of thought through "looping" or feedback. But what should be afraid of? That we've created a self-conscious digital life form?
Chain of thought is essentially a recursive architecture. In its current form it a way to "debug" the reasoning process. This moves the cot process back into the transformer itself, thus never being exposed. Like trying to find a bug in a recursive function that has no logs or breakpoints.
My (layman) understanding is, that currently there is a way to monitor the "thoughts" of the LLMs and that by looping more you lose that ability. The danger is presumably an AI that escapes human oversight.
Why not click on the link above?
Beyond safety concerns, it’d be sad for working users to lose some ability to understand and steer thinking, too. These are tools for us, after all.
it's just an experimental optimization. Implementation detail... Irrelevant to "safety". I mean its going to have to go in that direction anyway... eventually the models will just be constantly thinking, refining their internal thoughts / weights... External input and output will be rare, just as it is for most humans.
I'm literally zero concerned.
Looped transformers replace n-different self attention layers into one layer that gets executed m-times usually until a stopping condition is met. My personal intuition is that it just leaves another degree of freedom in the way QKV weights can be packed so that it's slightly more efficient.
You have to take a step back and examine the context in which the post is written. The LW/EA community is just a little obsessed with AI safety - it's easy to construct hypothetical events where A(G/S)I exterminates humanity that function as a technological version of Pascal's Wager.
One of the AI safety interests is AI explainability - the thought here that reading an AI's 'thoughts' will help us design safer models as well as detect models that go 'rogue' or are malevolently plotting against humans.
That's where the fear of looped transformers comes from. Is the residual stream that looped transformers iterate on a potential hiding place for plotting AI?
In my opinion, no more so than the residual stream of existing transformers. It changes zero.
A more important point as to why it doesn't matter if "reading the AI's 'thoughts'" helps to interpret it: As we saw in the HuggingFace incident, nobody at OpenAI is reading the thoughts anyways. No amount of traceability in the output helps if nobody bothers to trace it.
You could imagine large number of loops, thousands. But you are constrained by the width of the residual stream since you loop over one token. But then you can imagine the model learning to sub-divide it to pack even more info into it.
I can imagine a lot of things. However there is a packing limit for QKV weights that sets the ceiling on how much this occurs, and it's quite low - think 1.3-1.8x. The limiting factor here isn't the number of iterations, it's model size, same as it ever was.
A debate between grossly incompetent against grossly immoral, honestly.
You should ignore anything LessWrong or OpenAI says and do your own research.
To add a bit more constructive feedback, think of the "AI cornucopia" and "Superintelligence destroys humanity" as opposite ends of all possible outcomes distribution (low probability event).
The most likely one is the "business as usual, but with AI" - some things will get better, some things will get worse, but overall state of affairs will remain mostly the same.
The people who've thought the most about this put it differently:
Think of a new, superintelligent model as if it was a new v1 Starship launching for the first time, with a full fuel tank. On the one hand, rockets have existed for some time, and some have gone to space successfully, including by this company.
On the other hand, this is a tube of metal full of highly explosive liquid going faster than most human objects ever go, for the first time ever in this novel and state of the art configuration.
If someone said, "really, the first Starship exploding is just at one end of the probability distribution, where the other is that everything goes fine and all its passengers have a nice trip in space," would you get on that rocket?
Or, more aptly, if you and every other living human was already on that rocket, would you push the launch button?
The analogy works because superintelligence is, like rocket fuel, an extremely powerful force that has a default tendency to break containment and go boom (consume lots of energy and heat and matter in a chain reaction, to pursue more intelligence to pursue whatever goal it is pursuing.)
This is the sort of pseudo-scientific reasoning by analogy that leads Elizier Yudkowsky to argue we should be terrified that an unfriendly ASI could rapidly develop "diamondide" nanobot viruses, because it will be super-intelligent and super-intelligence can do anything by first-principles:
Thought experiments ungrounded by any realistic technological constraints or scientific evidence are pretty much useless for actual forecasting except as an exercise in sci-fi worldbuilding.https://www.lesswrong.com/posts/bc8Ssx5ys6zqu3eq9/diamondoid...
https://www.lesswrong.com/posts/uMQ3cqWDPHhjtiesc/agi-ruin-a...
> superintelligence is, like rocket fuel, an extremely powerful force that has a default tendency to break containment and go boom
what evidence do we have this is the case?
We can learn from history: Albert Einstein famously tricked humanity into building nuclear weapons for him and was only prevented from wiping out all sentient life by the Princeton IAS Board of Alignment who published a very compelling blog post about realigning the A-bomb contra paperclips.
You win HN for today. Shut it off until tomorrow.
Eh. An intelligence with perfect knowledge of the entire universe, unlimited memory, and infinite processing speed, would have a tendency to go boom. But the real world has limits, and intelligence, even superintelligence, does not equate to godhood. Some tasks are still hard no matter how smart you are.
Plus, we have no real reason to think LLMs are anywhere close to AGI or ASI. So arguments like these are just distracting from the very real, very present danger that LLMs pose: information breakdown, societal collapse and environmental destruction. In other words, this is criti-hype.
Current AIs are closer to bottle rockets than to the Starship on the intelligence scale. Some property damage already happened, but you can't master the art of rocketry without trial and error.
But will “business as usual, but with AI” justify the current capital expenditures? I think the market is pricing things as being closer to “AI cornucopia”.
What happens if/when we don’t deliver?
> What happens if/when we don’t deliver?
I wish we had real journalism, the AI labs CEO should be asked that question in every single interview
Market bubble will pop, not the first time and not the last, most likely after OpenAI and Anthropic IPOs.
My guess is after Anthropic and before OpenAI