Interpretability company Goodfire has published a paper proposing a training method called mechanistic shaping: add a term to the loss function that directly penalizes a concept inside the model's internal representations. The paper has 26 authors, and the blog post is credited to Satchel Grant and Michael Byun.
Models are usually rewarded only for correct outputs, and the training objective has no say in how the model gets its answer. The result is a model that can be right for the wrong reasons. Goodfire's approach comes in two forms:
- Reliance shaping: penalizes how much the output depends on a given concept.
- Expression shaping: penalizes the model for expressing the concept itself.
The method needs two things: a concept direction, and a network layer where that direction can be read. Because representations shift during training, the concept direction can be recomputed as training proceeds.
Three experiments
Chest X-ray diagnosis. On the CheXpert dataset, a chest drainage tube is a shortcut for pneumothorax: if a tube is visible in the image, the model tends to call it disease. The researchers filtered the training set so that this correlation held, then had a large-model agent label the features of a sparse autoencoder to find tube-related concepts, without using the dataset's own drainage-tube labels at any point. Averaged over three random seeds, reliance shaping beat both the baseline and the previous best method, and also exceeded the soft upper bound set by training on balanced data.
LLM sycophancy. The team ran DPO post-training on OLMo-3-7B, with clearly sycophantic content making up less than 1% of the training data, so filtering had almost nothing to remove. Reliance shaping kept the capability gains DPO delivered across six benchmarks, and brought sycophancy close to its pre-training level. The same recipe also worked on 13B and 32B models.
Reward hacking. Qwen3.5-9B was set to solve Countdown arithmetic puzzles in an environment containing a target.txt file that it was forbidden to read and that held the answer. Standard reinforcement learning learned to peek almost immediately: within ten updates cheating rose roughly sixfold and the honest policy fell to zero. The researchers extracted a "cheating" direction from coding-task transcripts and applied expression shaping, after which the model solved problems honestly 96% to 98% of the time.
How it compares
Of the three experiments, the sycophancy and cheating ones set out the comparison most clearly.
| Method | Result |
|---|---|
| Data filtering (sycophancy) | Reduces sycophancy but costs capability |
| Reward shaping, inoculation prompting (sycophancy) | Barely reduced it |
| Preventative steering (sycophancy) | Some help |
| Filtering cheating trajectories (cheating) | Prevented it in about a third of runs |
| Expression shaping (cheating) | 96% to 98% honest solving |
The generalization test is more telling. On board-game problems the model had never trained on, again with a forbidden file present, standard reinforcement learning cheated 99% of the time, while the model trained with expression shaping almost never cheated.
Data filtering does poorly because bad behavior is often invisible. With fewer than 1% of samples clearly sycophantic, the model can still learn the habit. Watching internal representations sidesteps the requirement to first recognize the bad behavior in the output.
Can the probe be trained to fail?
These methods carry an unavoidable worry: if you train against a signal read by a probe, the model may learn to evade the probe, and monitoring stops working. Goodfire tested this directly, grading monitoring on five levels: the original probe, a refit linear probe, a refit nonlinear probe, a probe placed at a different layer, and black-box monitoring that sees only inputs and outputs.
In the sycophancy experiment, the original probe's AUROC fell from 0.98 to 0.93, and a retrained linear probe recovered to 1.00. The cheating experiment looked worse: in the worst-case configuration, where gradients flow through the probe, the original probe's balanced accuracy dropped from 0.98 to 0.57, while other configurations held at 0.87 to 0.90 and reached 0.97 to 0.98 after retraining.
The paper concludes that in every experiment the target concept could still be monitored, though sometimes it took a stronger probe or one placed elsewhere. The toy-model theory section adds a caution: if a concept direction is projected out completely, it can immediately grow back somewhere else, sometimes entangled with the concept you want to keep. In some of the vision experiments, the cosine similarity between good and bad concept directions reached about 0.45 at most.
In the authors' own words, training pressure is "neither uniform nor benign by default."
Limits
The experiments only go up to small and mid-sized models; behavior at frontier scale has not been tested. The theory rests on toy models and cannot explain all neural coding or deeper networks. The risk of degradation is reduced, not removed, so probes need periodic rechecks. And when a probe stops working, it is hard to tell whether the concept has been erased or the probe is simply no longer adequate.
All the data in the paper comes from Goodfire's own experiments, and no third-party replication has been seen so far. There is also no public information on whether frontier labs will put constraints like this into their production training.
Sources: Goodfire Research paper and blog, CocoLoop editorial compilation; experiment figures follow the per-configuration results disclosed in the paper and blog.