Judging Is Not Enumerating: Silent Omissions in LLM-Authored Acceptable Sets

TL;DR: The same LLM that can judge "is this answer correct?" at F1 0.74–0.90, and can state the rule defining correctness at F1 ≈ 0.99, cannot write out the set of things that rule accepts — it silently omits huge chunks of it. Since test suites, answer keys, rubrics, and RLVR reward functions are exactly "written-out acceptable sets," LLM-authored verifiers are systematically too strict: they reject 58–81% of oracle-correct code, and their errors are omissions, which no review process can see. The paper measures this cleanly against constructed ground truth (no LLM judge grades anything), shows the gap doesn't close over a 24× scale range, quantifies what it costs inside an RL loop, falsifies the obvious self-repair, and then supplies two fixes that actually work.

The role reversal nobody stress-tested

The field has quietly promoted LLMs from examinees to examiners. A model writes the unit tests from a docstring, the answer key for a generated quiz, the rubric that becomes a reward signal. Once authored, that artifact is the definition of correct for everything downstream — benchmark labels, agent feedback, policy gradients.

There's a large literature on whether verifiers can be gamed, and how verifier noise propagates into RL. Almost all of it treats the verifier's origin as exogenous. This paper asks the prior question: can a model produce an artifact whose acceptance region actually matches the specification it was given?

The formal core is simple. A specification over candidate space V has an acceptable set S^* \subseteq V. There are three ways to expose a model's knowledge of it:

1. Judge (execution): given one candidate x, answer "is x \in S^*?" — one forward pass, one bit.
2. Author (materialize): produce an artifact inducing a region \hat S \subseteq V. Either list the members directly (extensional — what an answer key is), or write a test suite whose acceptance region is \hat S (intensional — the only option when V is unbounded, as for programs).
3. Restate: emit the defining rule as an executable predicate, letting machinery enumerate.

Same weights, same knowledge, three interfaces. The paper's finding is that they are wildly unequal — and the deployed role of "verifier author" uses the weak one.

Keys, rubrics, and test suites are produced through the weak middle interface.

The scissors, measured without a judge

The obvious methodological trap: you can't use an LLM judge to grade LLM verifier-authorship. The paper builds four constructions with mechanical ground truth. Two have complete finite truth: a presented list of 15 words plus a decidable predicate ("contains a double letter"), and the same design over integers. Nothing can be missing; every authoring error is genuine. One has executable truth: HumanEval+/MBPP+, with a pool of 2,464 solutions independently certified by the hardened EvalPlus oracle. One is WordNet — intuitive but incomplete, kept only for triangulation, with the confounds explicitly flagged.

On the complete-truth string construction (Qwen2.5, 3B→72B): judging rises 0.60→0.77 F1, enumerating rises 0.26→0.48. Authoring does scale — but the gap settles at ~+0.29 and never closes; every CI pair is disjoint. On the numeric variant, judging is 0.94–1.00 (trivial pointwise) yet authoring plateaus at 0.66–0.79 — a clean ~0.25 residual that's almost pure set-emission cost, since the candidate list is shown to both arms.

Code is the widest scissors. Models judging correctness at F1 0.74–0.90 author suites admitting only 19–42% of oracle-correct solutions, while still catching 94–99.8% of wrong ones. The failure mode is diagnostic: 70% of suites execute cleanly and reject the canonical reference solution on an assertion — syntactically healthy verifiers confidently encoding wrong expected outputs, often inventing requirements (like a ValueError on malformed input) the spec never states.

The control that locates the deficit is the paper's sharpest move (§3.11). Ask the same models, on the same items, to write def accepts(w): instead of listing the words: F1 = 0.999 / 0.978 / 0.999 at 3B/7B/14B, every predicate executable — above the model's own judging. So the failure isn't missing knowledge or an inability to specify. The 3B model that emits 26% of the set can state the rule generating it essentially perfectly. What fails is materializing the extension: emitting an unordered set token-by-token under a causal mask, deciding element-by-element what to include and when to stop, with no lookahead over unsampled candidates.

Test suites look intensional, so why do they fail too? The paper's decomposition: a suite requires choosing inputs (extensional selection) and computing each expected output (a derivation from prose, not a restatement of a given rule). The trace-repair experiment later shows exactly which half breaks.

Why omissions are silent

An authored set errs in two directions: over-inclusions (\hat S \setminus S^*) and omissions (S^* \setminus \hat S). These are not symmetric under review, and the asymmetry is structural, not empirical accident. A reviewer interrogates tokens that exist: an over-inclusion is a written entry, exposed by one membership query. An omission names nothing. Observation 2 makes it information-theoretic: even a reviewer with a perfect membership oracle, queryable only on candidates it can name, cannot distinguish two ground-truth sets that agree on \hat S but differ outside it. Producing the missing witness is the authoring problem — the capability under audit.

oracle S* authored Ŝ

omission no token exists to challenge

TP

over-inclusion a written entry: one query exposes it

detected 10–37% → RLVR starvation detected 71–86% → RLVR pollution

Empirically the asymmetry is 6–7× at 7–14B: planted over-inclusions are caught 71–86% of the time, planted omissions recovered only 10–15%. At 72B the ratio "improves" to 2.3× — but by flooding: false-alarm additions on clean keys quintuple to 5.46 per item. A frontier reviewer (GPT-5.1) looks symmetric in raw rates, but netting out its clean-key false-alarm floor, over-inclusion detection is +0.30 above chance and omission recovery +0.09 (CI covers zero) — the same sevenfold ordering. Capability increases willingness to add, not ability to find what's missing.

The corollary is nasty: any subtractive review loop (self-critique, adversarial revision) preferentially sheds the visible error and keeps the invisible one, walking the artifact monotonically toward under-acceptance. Formally, review maps \hat S_{t+1} = \hat S_t \setminus D_t where D_t contains only over-inclusions, so precision rises, recall is a fixed point. The paper's production corpus — 43,227 K-12 assessment items — shows this signature verbatim: key errors omission-dominated 316:33 (~10:1), over-inclusions falling round-over-round through the pipeline's adversarial review stage while omissions persist. (Caveat the paper itself makes: that taxonomy is diagnosed by a commercial grader that reproduces only 25% of its own fail verdicts on open-format items, so it's directional, not a rate.)

One more escape hatch is closed and it matters: sample-then-verify self-repair was registered as a prediction and falsified. Sample the set 10× at T=0.9, union, filter with the model's own (strong!) membership judgment — repaired F1 (0.13–0.21) never beats one-shot. Two binds: the union covers at most 55% of S^* (the missing members aren't in the sampling distribution), and the membership judge, strong on curated contrasts, collapses on the model's own near-boundary proposals. The boundary isn't sharply represented under any interface tested. External ground truth is a requirement, not an optimization.

What it costs inside RLVR

Wire an authored key in as R(x) = \mathbb{1}[x \in \hat S] and the disagreement with the oracle reward is exactly the two crescents: correct outputs in S^* \setminus \hat S get zero reward (starvation), wrong outputs in \hat S \setminus S^* get paid (pollution). Crucially, nothing in the loop can distinguish "the policy is wrong" from "the key is missing the answer" — the key is the loop's only definition of correct.

The paired experiment: identical GRPO runs on a Qwen2.5-3B policy, differing only in the reward key. On the lexical task (authored key recall 0.51), the authored arm ends 18.5 points below the oracle arm in synset-membership accuracy, in all six paired seeds (p=0.031), and the trained policy stays ~5× more polluted (0.179 vs 0.041 of rewarded rollouts oracle-invalid). On a complete-truth numeric task, where the authored key is nearly perfect (recall 0.926), the tax shrinks to 1.9 points — still positive in all six seeds. The honest reading: the tax scales with how badly the key is authored, and authoring is worst exactly on fuzzy semantic sets and code.

Note the caveat the authors state themselves: the lexical numbers are reference-relative and the oracle arm is criterion-aligned (trained on the metric that scores it), so 18.5 bounds rather than measures semantic loss. The clean causal estimate is the 1.9-point complete-truth run.

The two fixes that work

Gate. The authored-suite population is bimodal: a small sane minority and a broken majority that rejects even the reference solution. So gate on any known-correct probe: one execution. False rejection collapses from 58–92% to ≤5% — but only 5–39% of suites survive, and for weak authors (Mistral-7B) a quarter of survivors are vacuous.

Repair instead of discard. Since 87.5% of broken-suite rejections are assertion failures — the model picked an input, then miscomputed the expected output — rewrite each wrong expected value to what a reference execution returns, and re-gate. Yield rises 3.3× (Qwen2.5-14B: 0.207→0.690), 4.7× (Llama-3.2-3B), 6.9× (Llama-3.1-8B), 10.6× (Mistral-7B) — largest exactly where the gate keeps least. All measured on held-out solutions the repair never saw; catch-wrong stays ≥0.94, so repair isn't making suites vacuous (Mistral's survivors go from 43% vacuous at catch-wrong 0.47 to none at 1.00). This also completes the diagnosis beautifully: with expectations fixed by execution, the model's chosen inputs still reject ~95% of wrong solutions. Input selection is fine; expected-output computation is what fails. The repairing agent is the interpreter, not the model — consistent with the falsified self-repair.

Two honest scopes. Both fixes need a reference implementation, which on a real task may be the thing being built. And part of the low gated FRR was stylistic proximity: against a solution pool from non-Qwen families, gated FRR rises 0.010→0.064 and repaired FRR to 0.075–0.161 — still far below ungated 0.58–0.92, but authored suites do encode incidental properties of one implementation style.

Boundaries, and what to be skeptical of

The paper is unusually candid about its scope condition: everything above is one-shot greedy authoring with no test-time reasoning — which is, to be fair, how keys and suites are produced at scale. With reasoning enabled, GPT-5.1 closes the algorithmic gap outright (+0.259→+0.008, CI covers zero): a scratchpad lets the model apply the rule candidate-by-candidate, reconstructing the restate-then-execute route internally. But on test suites — where no compact rule exists to restate — reasoning shrinks Opus's gap (+0.274→+0.096, still excludes zero) and doesn't help GPT-5.1 at all (recall falls; the scratchpad over-specifies more). So the deficit is escapable precisely where a mechanical rule exists, and bites where it doesn't — which is where verification is hardest.

Other things to weigh. The RLVR runs are one LoRA epoch on single-word rollouts: great for attribution, narrow in realism; the scale-sweep arms are n=3 seeds with no attainable significance. The complete-truth predicates are deliberately trivial, so magnitudes shouldn't be extrapolated to compositional specs. The predicate-emission escape was measured with the rule given in the prompt; inferring a predicate from examples alone — the realistic condition — is the one experiment conspicuously not run, and the authors say so. The field evidence is judge-relative. Credit where due: a falsified registered prediction, a retracted scaling trend, and two self-reported instrumentation bugs with before/after numbers (Appendix G) are reported in full — rare and confidence-inspiring.

If the result holds, the practical rules are crisp: elicit the rule, not the roster (predicate emission is the difference between F1 0.26 and 0.99); never treat a one-shot authored key or suite as ground truth — it's directionally wrong in a way its own review provably cannot certify; gate every authored verifier on known-correct behavior, then repair rather than discard what fails, using execution to supply the expected values. And for RLVR designers: verifier noise from machine authoring has a direction (false-negative-heavy, fail-closed) and now a magnitude to budget for.

Where to spend your reading time: §3.11 (the intensional control — the paper's "aha" in three tables), §4 with Observation 2 (why omission structurally resists audit), and §3.6 (trace repair, which doubles as the cleanest mechanistic decomposition). Table 1's claim–evidence map, with its flagged falsification and retractions, is worth a look purely as a model of how to report empirical ML.