Mind the Cap: Output-Budget Regimes Change the Measured Multilingual Reasoning Gap

TL;DR: When you measure whether an LLM reasons better in English translation than in the native language, you always pick a maximum output length — and this paper shows that this seemingly innocent hyperparameter can be the result. On MGSM (multilingual grade-school math), the measured native-vs-translate gap swings by up to 57 points depending on the token cap, because Thai needs ~2.5× the tokens of English to say the same thing, so a "fair" cap silently truncates native reasoning mid-thought. The authors reduce the whole artifact to a one-line identity, confirm it with 540,000 independently hard-capped decodes under preregistered-style frozen tests, and — as a bonus — show that merely telling the model its budget changes accuracy even when the enforced cap never moves.

The hidden experimental variable

The standard finding in multilingual evaluation goes like this: take a math word problem in Thai, ask the model to solve it in Thai (call this NATIVE), or first translate it to English and solve it there (TRANSLATE-ACT). Translation usually wins, and the difference gets reported as a "multilingual reasoning gap" — evidence that the model reasons worse in low-resource languages.

But every such evaluation runs with a max_tokens cap, and languages are not tokenized equally. Using Qwen3-8B's own tokenizer on FLORES-200 parallel text, Thai costs 2.55× as many tokens as the English translation of the same content; Swahili 1.94×; German 1.56×. A chain-of-thought that fits comfortably in English can hit the cap in Thai before the model ever writes its answer line. Under the standard strict scoring (exact match on #### <integer>, truncated traces score 0), that trace counts as wrong — indistinguishable from a reasoning failure.

So the cap is not background furniture. It is an uncontrolled independent variable that interacts with language in a systematic, directional way. The question the paper asks is narrow and falsifiable: how much of the MGSM native-vs-translate gap is this truncation artifact, and where does it live?

The aha: the artifact collapses to one curve

Here is the move that makes the whole paper tractable. Define the gap at budget B as \operatorname{gap}(B) = \operatorname{acc}_T(B) - \operatorname{acc}_N(B) (translate minus native accuracy). The "budget artifact" is how much the gap shrinks when you give the native arm its length-normalized budget \lfloor r B\rfloor, where r is the language's token premium. Write it out and the translate term cancels:

The two-arm comparison disappears. The entire normalization effect is a finite increment of the native accuracy curve — how many native answers become correct inside the interval (B, \lfloor rB\rfloor]. Three consequences fall out for free: the artifact peaks wherever native answers are being emitted; it is exactly zero once the native curve saturates; and it can be predicted from a single long-cap run, because \Delta_L(B) = G(\lfloor rB\rfloor) - G(B), where G(t) = P(\text{correct} \wedge \text{answer emitted by token } t) is the correct-emission sub-CDF. You never need to run the budget sweep at all — one 4096-token run tells you where the cap will bite.

The evidence: sweep, freeze, replicate

The methodology is unusually careful for this literature. The authors stored 48,000 full 4096-token generations (2 models × 4 prompting strategies × 3 languages × 250 items × 8 samples) and scored every prefix — a "ledger" that lets them evaluate any budget without re-sampling noise. That sweep locates where the artifact lives. Then, because prefix scoring shares trajectories across budgets, they froze the sweep's three peak predictions in git tags and ran 540,000 fresh decodes, each independently hard-capped at its own budget, to test whether the effect survives genuinely independent generation.

It does, almost exactly:

chart error: Expecting ',' delimiter: line 1 column 430 (char 429)
{"type": "bar", "title": "Length-normalization effect Δ(B) for Qwen3-8B, points of accuracy", "ylabel": "Δ (accuracy points)", "categories": ["German (peak B=192)", "Thai (peak B=256)", "Swahili (peak B=128)"], "series": [{"name": "Discovery sweep peak", "values": [34.2, 38.9, 15.0]}, {"name": "Independent 540k decodes", "values": [34.65, 38.60, 13.70]}, {"name": "At B=1024 (saturated)", "values": [0.00, 0.15, 0.05]}, "note"][0] ? [] : [], "note": "Table 1 and Section 4.2. All six frozen Holm-corrected tests on the independent sample reject (p=0.0001 for peaks); the argmax lands on the predicted budget in all three languages."}

At B=256, Qwen Thai NATIVE scores 6.2% while TRANSLATE-ACT scores 47.8% — a 41.6-point "reasoning gap." Give the native arm its premium-scaled 652 tokens and 38.9 of those points vanish. That is not a reasoning deficit; it is a trace that wasn't allowed to finish.

Two more regime effects sharpen the picture. First, at very tight caps the ranking can reverse: Swahili NATIVE beats TRANSLATE-ACT at budgets 64–192 (bootstrap probability 1.00), because the translate arm burns its budget on the translation preamble before it can reason. The answer-emission timing explains it — Swahili NATIVE's 10th-percentile emission is 75 tokens earlier than translate's. Second, once native accuracy saturates (Qwen: 79.0/47.1/33.7% for de/th/sw at B=1024), \Delta_L goes to essentially zero, exactly as the identity requires: between budget 1024 and Thai's scaled budget of 2611, only 3 of 2000 traces become newly correct. The prospectively frozen confirmatory test at B^*=1024 therefore fails to reject — a negative result that the authors correctly interpret as "the artifact has moved out of range," not "there is no artifact."

The stranger result: announcing the budget changes accuracy

Everything above assumes accuracy is a function of the enforced cap. It isn't. In a separate frozen experiment (192,000 decodes), the enforced cap was held at a non-binding 2048 tokens while only the announced budget in the prompt varied. Telling Qwen "you have 128 tokens" — while silently letting it use 2048 — raised Thai NATIVE accuracy from 58.1% to 63.2% (+5.1 points, p=0.0029, the one cell that survives Holm correction), with a monotone dose response across the announced grid:

Announcing a tight budget compressed the traces (median Thai output drops from 350 to 198 tokens) and improved accuracy — the model apparently reasons more efficiently when told to be brief, at least in this cell. The authors are admirably restrained: one cell, one model, one benchmark, reported as such. But the methodological implication is general and nasty: \operatorname{acc}(B) is not a function of the enforced cap alone, so any evaluation that puts the budget in the prompt (as budget-forcing setups like s1 do) is measuring a different quantity than one that enforces silently. A machine-readable TOKEN_BUDGET: tag, by contrast, was inert in all twelve cells — the effect requires natural-language disclosure.

Predicting the artifact from one run

The sub-CDF identity gets its own validation, and it's the part most likely to be adopted as a tool. Estimate G(t) from one long-cap run, and you predict the entire budget-sweep artifact analytically:

The naive factorization p_{\text{correct}} \times P(\text{emit in window}) — assuming correctness and emission time are independent — misses by 3.1 points; they are correlated (non-emitters are 0% correct by construction, emitters 60.3%). The joint sub-CDF is the right object.

Pricing interventions with the same lens

Section 6 uses the identity to price the standard fix for tokenizer inequity: extend the vocabulary with target-language merges so the same text costs fewer tokens. The authors actually built cross-fitted extended Qwen tokenizers (~6.7k new Thai tokens, cutting the Thai premium from 2.55 to 2.21) and retokenized the stored traces. The punchline: vocabulary extension acts only through the truncation channel, so it buys nothing once traces fit. Its maximum gap closure anywhere in the data is 4.9 points (Thai at B=512, where 19% of traces still truncate), and exactly 0.0 at B=1024. Meanwhile simply raising the cap recovers 40+ points of native accuracy in the binding regime. The triage heuristic is clean: before paying for tokenizer work, extend the cap on a sample and measure what longer prefixes actually recover; if that's small, no token-count intervention can help.

What survives after the artifact is removed

Importantly, the paper does not claim multilingual gaps are fake. At saturation, Qwen Thai still trails TRANSLATE-ACT by ~41 points and Swahili by ~23. But the design deliberately reframes what that residual is: prompt language, reformulation, answer-format compliance, translation quality, and reasoning-trace language are all confounded in the "strategy" bundle, so the residual is a strategy-performance gap, not an identified reasoning deficit. Translation quality, for one, doesn't explain it — COMET scores correlate with correctness gains at |\rho| \le 0.28 across cells. That reframing is the paper's conceptual contribution: most published "reasoning gap" numbers conflate at least two things (a budget artifact and a strategy bundle), and neither is the causal quantity people think they're citing.

What to be skeptical about

The scope is genuinely narrow: three languages, two 8B models, one benchmark family, numeric exact-match scoring. Llama-3.1-8B is nearly useless here — native Thai accuracy sits at 3.9% with 93% of traces never emitting a parseable answer, so its cells mostly measure format compliance at floor. Qwen3's thinking mode was disabled, forcing all reasoning into the visible budgeted channel; reasoning models with separate thinking budgets may behave differently, and frontier-scale models with far better low-resource fluency may have much smaller emission-timing asymmetries to begin with. The "preregistration" is an internal git-tag freeze, not a public registry filing (the authors say so). Human validation of the language-ID audit is still outstanding, and the FLORES premium as a normalizer is itself contestable — the behavioral trace-length ratio is smaller in all six cells, and under it the Swahili effect drops below the 5-point threshold. And the announcement result, the most tantalizing finding, rests on a single significant cell.

None of that undermines the central methodological claim, which is well-supported and immediately actionable: the output cap is an independent variable. Any multilingual (or more broadly, any cross-condition) evaluation under a token budget should report a sweep, mark where answer emission binds, and never let a single cap carry the claim. The G(t) trick makes this cheap — one long run per condition predicts the whole regime.

If you read one part of the paper, read Section 3 (the estimand and its three consequences — the whole paper is downstream of Equation 1) and then Appendix J, where the sub-CDF prediction is validated. Section 4.4 on announcement effects is short and worth the detour for anyone building budget-aware inference systems.