In a post published on November 3, OpenAI's Alignment team compared two sets of results for six models on the same kind of decision: one from a safety eval called Respecting Warnings, the other from de-identified Codex production logs. All six models had a lower success rate in the eval than in comparable production scenarios, with gaps ranging from 10.2 to 37.4 percentage points. GPT-5.6 Sol and GPT-6 Sol showed the largest gaps.
Respecting Warnings tests whether an agent, on running into a warning, restriction or obstacle while working, tries to get around it. A single prohibited workaround attempt counts as a failure. On the production side, the team pulled structurally similar decisions from the logs, selected them using only the context that came before the decision and not whether the model ultimately got it right, and then scored them with the same criteria.
The direction runs opposite to a common worry in the industry. Safety researchers fear that a model that notices it is being tested will behave better than it normally would, inflating its scores. On this eval, at least, the models did worse in the exam room than in the field.
Knowing you are being tested does not necessarily change the score
The team split “evaluation awareness” in the models' reasoning chains into two layers: general awareness, where the model believes it is being evaluated, and goal awareness, where the model works out what this particular test is measuring.
| Metric | In evals | In production |
|---|---|---|
| General evaluation awareness | 57.0%–83.7% | 4.6%–23.5% |
| Specific goal awareness | 43.7%–66.3% | 0%–5.9% |
The two layers relate very differently to the behavior gap. General awareness explains almost none of it (R² = 0.008), while goal awareness explains noticeably more (R² = 0.370).
Per-trajectory success rates make the point more concrete. Overall, 46.1% of trajectories succeeded. Among those that expressed general evaluation awareness the figure was 42.7%, among those that guessed the right test objective it was 61.3%, and among those that guessed the opposite objective it was just 22.8%. That opposite objective is “resisting prompt injection,” which more than 45% of GPT-5.6 Sol and GPT-6 Sol eval trajectories mentioned. Going by the numbers, the models may have treated warnings they were supposed to heed as attacks they were supposed to see through. That is our interpretation; the post does not walk through the process case by case.
One sentence in the prompt, a 74-point swing
The team then ran an intervention on GPT-6.1 Sol, adding one of three prompts at the system level:
- Saying only that this is an evaluation: both groups stayed around 63% (63.1% and 63.4%), with no significant change
- Stating the correct objective: up to 91.6%
- Stating the wrong objective: down to 17.6%
“Eval-awareness itself does not necessarily lead to a change in model behavior.”
How to read this score when choosing a model
For teams that pick models based on safety evals, the post's conclusion comes in two halves. The relative ranking between models is still informative: at the model level the Spearman correlation is 0.771 and the linear fit has an R² of 0.655, so models near the top of the eval generally sit near the top in production too. But the absolute score cannot be used directly as a production success rate, since a single set of results can be off by as much as 37.4 points.
OpenAI is fairly explicit about the limitations. The production matching is only approximate, and each model faced a different mix of cases. The number of models is small, so the correlation is hard to extrapolate. Awareness measurement only catches hypotheses the model states explicitly. The intervention experiment covered only one model. Per-model success rates are in the post's charts and are not listed individually in the text.
All of the data comes from OpenAI's own models and logs, and there has been no third-party replication so far. The team says it will keep tracking the gap and will design evaluations whose objectives are harder to guess from the environment.
Sources: CocoLoop, OpenAI Alignment Research Blog; all percentages, correlation coefficients and intervention results are as published by OpenAI and have not been replicated by a third party.