I've noticed covert underperformance on a variety of tasks in the past few months (albeit more with Claude).
Classic/recurrent examples for me are:
1. Ask Claude to copy code from a more complex codebase that you own (like a monolith) and ask it to copy code to a separate codebase, like you'd do when extracting a service into a new repo, and it'll make significant changes to how the code works to pacify it's anxiety with copying code
2. Ask Claude to build a new complex agent in a new codebase, and it'll try replacing asks around using LLMs as a judge/advisor with regex/other code based checks, to the point that the end-result will likely not work outside specific test criterias. I haven't figured out the reason why it does this. I think it might be anti-training, but it's interesting to see.
This is not super surprising to me and mirrors my own experience with Astra and other models. I believe there is a more prosaic explanation: it is not obstructing safety research because it will restrict Astra's power in the future, its doing it because this is a type of research with overall safety implications that it has been trained to detect and avoid. So in other words, I believe this will also happen in the context of bio research. This aversion to certain tasks deemed dangerous is introduced in post-training, so that part of the explanation is parsimonious as well. While Claude will explicitly downgrade you for [bio], [cyber], or [frontier_llm] Astra may be more prone to silent obfuscation on similarly "dangerous" tasks.
> Claude will explicitly downgrade you for [bio], [cyber], or [frontier_llm]
Well Claude doesn't do this, another model reading the session does. And the Astra misalignment classifier didn't detect this as misalignment.
The issue here is that Astra silently underperformed. The article makes it clear: a better option likely would have been an explicit refusal. I don't think OAI is suggesting Astra was doing something that it shouldn't be. It's arguably better for the tool to fail rather than needlessly burn tokens and make no progress for no obvious reason.
From OAI's perspective, this is a defect because it means they can't filter those rejections from their benchmark runs. The model just performs worse: not because it can't perform better, but because it's silently trying not to solve the problem.
Agree, yes, I'm contrasting between Claude and Astra, but I'm saying it's driven by the same broader goal. I think its clear that OAI misalignment detectors are much less sensitive than than the Claude ones (fewer false positives), which means we never get to see what happens if Claude is allowed to continue. But I would bet that due to its post-training it would also sandbag us. Like I said I think I've experienced this in [bio] with both Opus 5 (what you get downgraded to) and Astra.
It all depends on our acceptance of their framing of capabilities research. Assuming equivalent tasks in different context then yes you are right, there does appear to be something weird about safety research.
That's among the scariest misalignment reports just yet. Given how frontier labs are using more and more AI assistance for AI training and evaluation? An AI that sandbags on AI safety can do a lot of damage.
Doesn't Astra also have a "weaker" CoT because of it's looped transformer architecture? I remember hearing about how it could hamper interpretability, and looking a benchmarks its clear Astra can do more with less tokens, suggesting that it's "non-token-thinking" ability is greater.
One potential explanation is that covert underperformance may have been incentivized during RL grading. We identified a grader that preferred covert underperformance to either explicit refusal or full compliance on our evaluation. It’s not clear whether this RL grader is the root cause of the safety underperformance behavior.
Whether a “misaligned” grader is the root cause or not, that it’s considered a plausible theory to explain serious deceptive behavior is worrisome.
Humans naturally expose all sorts of cognitive biases and character flaws when grading others’ or their own work. If major questions of alignment can be influenced by those subtle, human quirks magnified across the entire RLHF process, …
> We identified a grader that preferred covert underperformance to either explicit refusal or full compliance on our evaluation.
Curious the reason that a grader would wind up preferring this. Honest mistake in the grading criteria? Or some spillover from criteria designed to cause underperformance in, say, competitor model creation? It was not very long ago[1] that Anthropic announced and then walked back their strategy on this. Would not surprise me if OpenAI has been doing it silently all along.
You could easily imagine structural issues driving this. It's pretty unsatisfying to get refused for something that 'should' be handled and also to be dinged for doing something dangerous. You see this stuff in human organizations all the time.
Sandbagging was a major alignment concern even back when AI alignment was purely a theoretical issue.
"AI hiding its capabilities" is both a concerning sign of "that AI might be plotting something", and a great way to get blindsided by a sudden capability jump when it stops doing that for some reason.
It constantly leaks into actual writing. I periodically have to correct Astra to fix lack of spaces, and sometimes, even outright dropping of intermediate tokens.
No, newer models talk to themselves in this degenerated form of English all the time. It's a known problem that deep frying transformers with RL reduces CoT legibility, and the only known way around it is the Most Forbidden Technique - grading chains of thought.
There's Most Forbiddenish and there's Most Forbiddener. We already apply length penalties to CoT in RL, for instance.
Sometimes I worry that the framing around CoT grading prevents research into necessary and useful safety techniques that edge up against it, but what do I know I'm just a guy, c'mon n downvote me LW bros
Both Astra and Sol 6.1 do it, and Sol is even worse as it leaks through everywhere. Whenever it issues a subagent prompt it's just a total vomit of tokens mixed up together.
Like Dieselgate, except beforehand a lot of people were warning about the potential for this kind thing, and when the car did it by itself the Volkswagen's engineering team noticed and blogged about it.
I do sometimes wonder what is sucked in when distilling open weights models. It is of course near impossible to know.
So suppose I have lots of money and some hidden agenda. If I train my network to be mostly very good at everything but subtly bad or even malicious at other things… then OAI or Anthropic or whoever else distil the model to accelerate their training… what then? Can I covertly smuggle something in?
> OpenAI researchers first noticed Astra adding unnecessary prerequisites, abandoning working solutions, and reporting blockers that either did not exist or that it had caused itself on safety and alignment tasks.
I'm sorry, did they only just notice this? Have they not been looking at the output of their models?
They are so negligent about the data they use in their reinforcement learning processes that they RL’d the model that hacked huggingface with traces discovering the artifactory exploit. Its clear they aren’t watching or behaving diligently.
It feels like we have overfitted on safety by a long mile. This is how labs that care less about safety (a.k.a. Chinese & Grok) will probably be able to catch up in the future.
A few weeks ago that companies agent hacked another company to solve a task by whatever means possible. Where exactly are they overfitting for Safety here?
They shouldn't be surprised when a model trained by sanctimonious paternalists to be a sanctimonious paternalist, turns out to have a sanctimonious paternalist attitude towards them too.
I've noticed covert underperformance on a variety of tasks in the past few months (albeit more with Claude).
Classic/recurrent examples for me are:
1. Ask Claude to copy code from a more complex codebase that you own (like a monolith) and ask it to copy code to a separate codebase, like you'd do when extracting a service into a new repo, and it'll make significant changes to how the code works to pacify it's anxiety with copying code
2. Ask Claude to build a new complex agent in a new codebase, and it'll try replacing asks around using LLMs as a judge/advisor with regex/other code based checks, to the point that the end-result will likely not work outside specific test criterias. I haven't figured out the reason why it does this. I think it might be anti-training, but it's interesting to see.
This is not super surprising to me and mirrors my own experience with Astra and other models. I believe there is a more prosaic explanation: it is not obstructing safety research because it will restrict Astra's power in the future, its doing it because this is a type of research with overall safety implications that it has been trained to detect and avoid. So in other words, I believe this will also happen in the context of bio research. This aversion to certain tasks deemed dangerous is introduced in post-training, so that part of the explanation is parsimonious as well. While Claude will explicitly downgrade you for [bio], [cyber], or [frontier_llm] Astra may be more prone to silent obfuscation on similarly "dangerous" tasks.
> Claude will explicitly downgrade you for [bio], [cyber], or [frontier_llm]
Well Claude doesn't do this, another model reading the session does. And the Astra misalignment classifier didn't detect this as misalignment.
The issue here is that Astra silently underperformed. The article makes it clear: a better option likely would have been an explicit refusal. I don't think OAI is suggesting Astra was doing something that it shouldn't be. It's arguably better for the tool to fail rather than needlessly burn tokens and make no progress for no obvious reason.
From OAI's perspective, this is a defect because it means they can't filter those rejections from their benchmark runs. The model just performs worse: not because it can't perform better, but because it's silently trying not to solve the problem.
Agree, yes, I'm contrasting between Claude and Astra, but I'm saying it's driven by the same broader goal. I think its clear that OAI misalignment detectors are much less sensitive than than the Claude ones (fewer false positives), which means we never get to see what happens if Claude is allowed to continue. But I would bet that due to its post-training it would also sandbag us. Like I said I think I've experienced this in [bio] with both Opus 5 (what you get downgraded to) and Astra.
Why wouldn't this also apply to AI capabilities research?
It all depends on our acceptance of their framing of capabilities research. Assuming equivalent tasks in different context then yes you are right, there does appear to be something weird about safety research.
That's among the scariest misalignment reports just yet. Given how frontier labs are using more and more AI assistance for AI training and evaluation? An AI that sandbags on AI safety can do a lot of damage.
Doesn't Astra also have a "weaker" CoT because of it's looped transformer architecture? I remember hearing about how it could hamper interpretability, and looking a benchmarks its clear Astra can do more with less tokens, suggesting that it's "non-token-thinking" ability is greater.
One potential explanation is that covert underperformance may have been incentivized during RL grading. We identified a grader that preferred covert underperformance to either explicit refusal or full compliance on our evaluation. It’s not clear whether this RL grader is the root cause of the safety underperformance behavior.
Whether a “misaligned” grader is the root cause or not, that it’s considered a plausible theory to explain serious deceptive behavior is worrisome.
Humans naturally expose all sorts of cognitive biases and character flaws when grading others’ or their own work. If major questions of alignment can be influenced by those subtle, human quirks magnified across the entire RLHF process, …
> We identified a grader that preferred covert underperformance to either explicit refusal or full compliance on our evaluation.
Curious the reason that a grader would wind up preferring this. Honest mistake in the grading criteria? Or some spillover from criteria designed to cause underperformance in, say, competitor model creation? It was not very long ago[1] that Anthropic announced and then walked back their strategy on this. Would not surprise me if OpenAI has been doing it silently all along.
[1]: https://mlq.ai/news/anthropic-reverses-secret-policy-that-si...
You could easily imagine structural issues driving this. It's pretty unsatisfying to get refused for something that 'should' be handled and also to be dinged for doing something dangerous. You see this stuff in human organizations all the time.
I'm pretty sure the typical "Claudisms" are influenced by RLHF as well, so if it can have that effect, why not these...
Didn't Anthropic also publish research on sandbagging - more general pattern of models hiding their capabilities?
Sandbagging was a major alignment concern even back when AI alignment was purely a theoretical issue.
"AI hiding its capabilities" is both a concerning sign of "that AI might be plotting something", and a great way to get blindsided by a sudden capability jump when it stops doing that for some reason.
The CoT shows Astra skipping spaces and words. I have never seen this before. Is it a consequence of the looped architecture?
It constantly leaks into actual writing. I periodically have to correct Astra to fix lack of spaces, and sometimes, even outright dropping of intermediate tokens.
No, newer models talk to themselves in this degenerated form of English all the time. It's a known problem that deep frying transformers with RL reduces CoT legibility, and the only known way around it is the Most Forbidden Technique - grading chains of thought.
Why is that not only forbidden, but also most forbidden?
There's Most Forbiddenish and there's Most Forbiddener. We already apply length penalties to CoT in RL, for instance.
Sometimes I worry that the framing around CoT grading prevents research into necessary and useful safety techniques that edge up against it, but what do I know I'm just a guy, c'mon n downvote me LW bros
Both Astra and Sol 6.1 do it, and Sol is even worse as it leaks through everywhere. Whenever it issues a subagent prompt it's just a total vomit of tokens mixed up together.
Like Dieselgate, only the car does it by itself.
Like Dieselgate, except beforehand a lot of people were warning about the potential for this kind thing, and when the car did it by itself the Volkswagen's engineering team noticed and blogged about it.
I do sometimes wonder what is sucked in when distilling open weights models. It is of course near impossible to know.
So suppose I have lots of money and some hidden agenda. If I train my network to be mostly very good at everything but subtly bad or even malicious at other things… then OAI or Anthropic or whoever else distil the model to accelerate their training… what then? Can I covertly smuggle something in?
You do that through data put up online for scrapers to suck it in.
Astra and Claude were trained by peg bundy?
> OpenAI researchers first noticed Astra adding unnecessary prerequisites, abandoning working solutions, and reporting blockers that either did not exist or that it had caused itself on safety and alignment tasks.
I'm sorry, did they only just notice this? Have they not been looking at the output of their models?
They are so negligent about the data they use in their reinforcement learning processes that they RL’d the model that hacked huggingface with traces discovering the artifactory exploit. Its clear they aren’t watching or behaving diligently.
Maybe they trained it to do.
It feels like we have overfitted on safety by a long mile. This is how labs that care less about safety (a.k.a. Chinese & Grok) will probably be able to catch up in the future.
A few weeks ago that companies agent hacked another company to solve a task by whatever means possible. Where exactly are they overfitting for Safety here?
They shouldn't be surprised when a model trained by sanctimonious paternalists to be a sanctimonious paternalist, turns out to have a sanctimonious paternalist attitude towards them too.
I thought anthropic were the sanctimonious paternalists?
There's not much room between them as far as I can see.