Answer-Conditioned Chains of Thought Degrade Verifiable-Reasoning Distillation in Large Language Models

TL;DR: A hugely common trick in reasoning distillation — when a model can't solve a problem, show it the gold answer and ask it to write a chain of thought that reaches it — quietly poisons the training data in a way no correctness filter can detect. In a tightly controlled experiment that varies literally one bit (was the generator shown the answer?), fine-tuning Qwen3-8B on its own answer-conditioned chains costs 16.2 points on MATH-500 and 27.2 points on AIME versus answer-blind chains from the same model on the same problems with the same filter. The mechanism is legible: answer-conditioned chains rationalize backward, stating the answer early and narrating toward it, and this "answer-first" habit transfers to the student. Best of all, the damage is predictable before any training from a cheap statistic on unlabeled generations, and largely fixable with a one-line prompt change.

The recipe everyone uses, and its hidden assumption

The standard loop for building open reasoning models — the one behind STaR, rejection sampling, ReST-style pipelines, MetaMath-style augmentation, and much of the DeepSeek-R1 distillation lineage — is: sample chains of thought from a model, keep the ones whose final answer checks out, fine-tune on the survivors.

There's an annoying failure mode: for hard problems, sampling never produces a correct chain, so those problems contribute nothing. The popular fix, going back to the original STaR paper (Zelikman et al. 2022), is rationalization: show the model the gold answer and ask it to write a chain that arrives at it. The resulting chain reaches the correct answer by construction, passes the correctness filter, and gets mixed into the training set. Problem solved — you've rescued the hard problems.

The implicit assumption is that a correct chain is a good training target regardless of how it was produced. This paper's claim is that the assumption is false, badly, and — crucially — that the failure is invisible to outcome verification. Every poisoned chain is correct. Every poisoned code trace passes its unit tests. The filter you're relying on cannot see the problem, because the problem isn't in the destination; it's in how the chain got there.

The one-bit experiment

The reason this is convincing rather than anecdotal is the experimental design. Prior work (e.g., TRICE) had noted theoretically that answer-conditioning biases the learning signal, and concurrent work has started bolting entropy filters onto hint-based pipelines. But nobody had cleanly isolated the causal cost. Here, everything is held fixed — the generator (Qwen3-8B), the 935 math problems, the correctness filter, chain lengths (via a matched control), the SFT recipe — and only one thing varies: whether the generation prompt includes the gold answer a(q) with a request to reach it.

The leakage penalty is defined simply as \Delta = \operatorname{acc}(S_{\text{blind}}) - \operatorname{acc}(S_{\text{leaked}}), where the two students differ only in which corpus they were fine-tuned on. On MATH-500 (three-seed mean), the blind student sits at 95.1 — indistinguishable from the untouched base model's 95.2, so answer-blind self-distillation is essentially neutral. The leaked student drops to 78.9. And the penalty grows with difficulty:

Within MATH-500 the same pattern holds by difficulty level: zero penalty on Level 1, ~21 points on Level 4. A length-matched control (blind chains truncated to their leaked twin's length) scores 95.0, so it's the content, not the length. Paired-bootstrap 95% CI on the headline penalty: [12.2, 19.2]; McNemar discordant items split 86 to 8, p<10^{-17}. The statistics are done carefully.

The mechanism: rationalization has a fingerprint

Why would a correct chain be toxic? Think of it distributionally. An answer-blind generator samples from P(c \mid q) — the search distribution, where c is the chain and q the question. That's the distribution a deployed solver must occupy, because at test time the answer is unknown. An answer-conditioned generator samples from the much lower-entropy P(c \mid q, a(q)): with the destination known, search collapses into post-hoc justification. SFT on those chains is off-policy in the sense that matters — it teaches the student to imitate a narrator who already knows the ending, a role the student can never play at test time.

The fingerprint of this collapse is delightfully simple: the chain states the final answer near the beginning. Define the answer-first rate

where \mathrm{pos}(c) \in [0,1] is the relative position of the first gold-answer mention inside the think block. On matched pairs, answer-conditioning shifts the whole position distribution toward the start: median first-answer position drops from 0.587 to 0.297, and the fraction stating the answer within the first 5% of the chain jumps from 13.3% to 41.9% (KS test D = 0.291, p < 10^{-34}).

Two things make this more than a correlate. First, the habit transfers: the leaked-trained student, evaluated with no answer in sight, states its own boxed answer early 48.7% of the time versus 36.5% for the blind-trained student (a +9.4 gap even restricted to problems all students solve). Second, the failure decomposes into a derivation deficit (9.5 points worse even when both students finish) and a termination deficit: the leaked student fails to produce any extractable answer within the 32K-token budget on 7.4% of problems, versus 0.4% for blind. This comes with a striking train/test length inversion — leaked training chains are shorter (seeing the answer compresses the reasoning), but the leaked student generates longer chains at test time, rambling without converging. It imitated the confident narration but never learned the search that's supposed to precede it.

Nailing the cause: three progressively sharper ablations

The paper doesn't stop at "leaked corpus bad." It runs three dissections, each isolating a smaller carrier of the harm.

1. Instruction vs. visibility. Is the problem that the model sees the answer, or that it's told to rationalize toward it? Holding the answer visible in all arms and varying only the instruction:

Mere visibility costs about a point. The rationalize-toward instruction carries essentially everything else. A "derive from scratch, only then state the answer" instruction recovers roughly two-thirds of the penalty while keeping every example — the cheap mitigation if your pipeline genuinely must show the answer.

2. Which chains carry it. Split the 935 leaked chains at the AFR threshold: 438 answer-first (Leaked-Early), 497 derivation-first (Leaked-Late). Since the split might just track problem difficulty, they add blind-twin control arms on the same problem subsets and compute a difference-in-differences. Result: Leaked-Early trains a student at 64.2 while Leaked-Late reaches 86.2; the DiD is +16.9 points (three-seed mean, bootstrap p<10^{-4}). The rationalized chains carry most — not all — of the harm, so filtering them is a legitimate partial repair. Note the length account predicts the opposite ordering here (the rationalized chains are the longer ones), which kills length as an explanation from a second direction.

3. Down to a single line. The sharpest cut: from each Leaked-Early chain, excise the one line where the gold answer is first stated — a median of 302 characters, ~1.4% of the chain — and retrain on the otherwise-identical 438 chains. The score recovers from 64.2 to 88.8, i.e., 83% of the gap to the blind twin. A sentence-scale surgical edit recovers most of a 30-point collapse. Whatever else is subtly different about rationalized chains, teaching the student "state the conclusion, then justify" is the dominant poison.

Predicting the damage before you train anything

Here's the part with the most practical leverage. Different models are hurt very differently by this recipe — Qwen3-4B loses 29.2 points, DeepSeek-R1-Distill-Llama-8B loses 1.2 — and it turns out you can rank the vulnerability before any fine-tuning. Compute \Delta\mathrm{AFR} = \mathrm{AFR}_{\text{leaked}} - \mathrm{AFR}_{\text{blind}} on a small sample of the candidate teacher's unlabeled generations: how much more often does this model blurt the answer early when it can see it?

Across eight thinking models from four families, \Delta\mathrm{AFR} correlates with the realized penalty at r = 0.96 (permutation p = 0.0003; partial correlation 0.91 after controlling for base accuracy). More convincingly, the fit made genuine out-of-sample predictions, recorded before the held-out models were trained:

The Nemotron prediction is almost eerie — 17.4 predicted, 17.6 observed, on a from-scratch Llama lineage unrelated to either family in the fit. The caveats: eight models is a small sample (Fisher-z interval on r is [0.79, 0.99]), all points share this paper's generation and evaluation harness, and one model (Phi-4-reasoning, with an extreme \Delta\mathrm{AFR} of +45.5) had to be discarded because both trained arms degenerated. Read it as a strong small-sample trend, which is how the authors themselves frame it.

Boundaries: where the effect lives and dies

The generalization section maps the effect's edges, and the edges match the mechanism:

• Code: with "answer" = reference solution and "correct" = passing tests, the penalty is 9.5 points on MBPP+ and 9.2 on HumanEval+ — even though every kept chain passes its tests. The signature transfers too: leaked code chains echo a verbatim fragment of the reference solution early 13.0% of the time vs. 1.3% for blind.
• Cross-student and cross-family: the harm travels with the data. Qwen3-8B's leaked corpus hurts a Qwen3-1.7B student (~7 points, though noisily); Nemotron and GLM teachers' leaked corpora hurt a Qwen3-8B student by 14.8 and 10.2 points.
• STaR-style rescue — leaking only on the failed problems, the exact deployed use case — still costs 4.0 points versus training on the blind successes alone. The rescued chains are negative-value data despite being verified correct and covering problems the blind arm couldn't solve.
• Where it vanishes: MMLU and GPQA-Diamond show no penalty; both arms sit at the base model. Multiple-choice selection requires no derivation, so there's no search behavior to corrupt. The effect is confined to tasks where the answer must be derived, exactly as the off-policy story predicts.

What to make of it, and what to be skeptical about

If this holds, the actionable conclusions are unusually crisp. Generate answer-blind — it's free, avoids the pathology entirely, and even beats gold-data SFT (95.1 vs. 90.8) at a matched budget. If you must show the answer, use a derive-first instruction (recovers ~2/3). Screen candidate teachers with \Delta\mathrm{AFR} before spending fine-tuning compute. Filter or excise answer-first chains from existing hint-conditioned corpora. And more broadly: audit how your chains were generated, not just whether they end correctly — this is a data-provenance failure, complementary to the process-vs-outcome supervision literature, which worried about verifying chains but not about the generation-time conditioning of chains that verify perfectly.

Reasons for caution. The core experiment is 935 problems, one SFT recipe, mostly one student family; several generalization rows are single-seed ("consistent sign, not exact magnitude," as the limitations section honestly puts it). The headline setting is self-distillation on a model already at 95.2 on MATH-500, where blind SFT has nothing to gain — though the headroom-regime result (DS-R1-Distill-Qwen-7B: blind gains +10 over base while leaked forfeits 7.3 of it, positive in every seed) and the AIME numbers address the ceiling worry. The cross-student transfer is the shakiest cell (seed penalties of 16.8, −0.2, 4.0). The triage question of whether the effect survives modern filtering beyond correctness checks is only partially answered: the paper shows the AFR filter itself is a partial repair, but doesn't test, say, PRM-based reranking against leaked corpora — plausibly a process reward model would catch some rationalized chains, and that's a real open question. And AFR requires an answer that appears as a matchable string mid-chain, which is easy for boxed math and echoed code, less obvious for open-ended reasoning.

None of that undercuts the central causal claim, which is about as well-controlled as SFT-data experiments get: same generator, same problems, same filter, one bit flipped, three seeds, length-matched, with the carrier identified down to a single excisable sentence.

Where to spend your reading time: Section 4 (mechanism) plus Table 1 — the instruction ablation, the within-corpus carrier design, and the train/test length inversion are where the paper earns its causal language. Then the single-line excision result in Section 6, which is the most surprising sentence in the paper. Skim Section 5 for the predictor, and glance at Appendix I if you want to check that the small-n statistics were handled as carefully as claimed (they were).