19PINE-AI/programmable-kv

TL;DR: A single-author preprint plus full code arguing that the KV cache should be treated as editable, composable program state. The interpretability finding is the interesting part: surgically refreshing one field's key/value vectors in a cached prefill fails not because of some numerical fragility, but because the transformer has already computed field-conditioned conclusions at prefill and written them onto downstream delimiter/aggregator tokens — at decode, the model reads those "notes," not the field itself. That mechanism motivates two serving primitives: append-only errata that correct stale facts without recomputation, and RoPE-repositioned "skill" caches you can transplant into any context.

The core finding

The setup is the standard agent-serving pain point: you cache a long prefill (user profile, tool docs, order state), then one token inside it changes. Naively you'd recompute just that field's KV and keep the rest. The paper shows causally that this recovers almost nothing — the field's own KV drives under 1% of the downstream decision. The reason: prefill attention has already aggregated the field into conclusions stored on later tokens ("distributed write, concentrated read"), resolved here to a component-level circuit and replicated across four model families.

This flips the usual mental model of the KV cache from "compressed replay of the prompt" to "notebook of memoized conclusions." Once you accept that, the fix is obvious in hindsight: don't rewrite the notes, amend them.

Decode: concentrated read field notes decision field KV: <1% of decision → surgical edit fails

Fix: append an erratum erratum (append) cache-aligned; matches hoist-to-end oracle

The two primitives

Editable. Instead of invalidating the cache on a field change, append a one-line salient erratum. Because it's append-only, everything upstream stays prefix-cache-aligned. Claimed effect in online serving: 98.5% vs 1% prefix-cache hit rate, up to 14.5× throughput, 53–398× lower p90 TTFT, matching a "hoist the update to the end" oracle without prompt surgery. Note this is closer to smart context engineering with a mechanistic justification than to literal cache surgery — the discovery is precisely that literal surgery can't work.

Composable. Precompile a "skill" (a reusable context block) once, cache its KV, and RoPE-reposition it into arbitrary positions in new contexts. Claimed to be behaviorally indistinguishable from full recompute (logit cosine 0.90–0.999) with O(L) instead of O(L²) TTFT — 13.9× at 32k. A keystone experiment edits a field inside a transplanted skill, arguing both operations act on the same notes; a unified edit+compose agent stays decision-identical to recompute across thirteen models.

What's actually in the repo

Unusually complete for a preprint: LaTeX source and PDF, all experiment harnesses (e1/, e2/, esys/, mem/), a standalone editkv module exposing an EditableContext (in-place edit + erratum with per-edit diagnostics), raw result JSONs that drive every paper number, figure-generation scripts, and an interactive companion site whose build asserts 22 numbers against the paper. Claims span scale, quantization, MoE, and multimodal image caches, with adapters for MLA, interleaved M-RoPE, and sliding-window attention. Everything ran on one RTX PRO 6000 (96 GB), so reproduction is plausible on a single big GPU. Apache-2.0.

Caveats

Days old, single author, 13 stars, zero external validation. The behavioral-equivalence evidence is logit cosine and decision-identity, not exact-match generation — coherence over long generations after edits, and interference between many stacked errata, are the obvious things to stress-test. "Composable skills" overlaps conceptually with prior KV-transplant/prompt-cache work (e.g., PromptCache-style block reuse); the differentiator here is the causal mechanism story and the edit-inside-transplant unification, which is the part worth scrutinizing. The claimed 14.5×/398× serving wins are against full-recompute baselines in the paper's own harness.

Try it

The interactive companion at https://01.me/research/programmable-kv/ is the fastest way to evaluate the mechanism claims — every figure is backed by released result records. To run the probes yourself:

git clone https://github.com/19PINE-AI/programmable-kv && cd programmable-kv
pip install -r requirements.txt
pip install -e editkv

# mechanism probes on Qwen3-8B; records land in results/
python esys/mech_suite.py --model Qwen/Qwen3-8B
# editing-recovery experiment
python e2/run_recovery.py --model Llama-3.1-8B

Worth an afternoon: if the mechanism replicates on a model you care about, the erratum trick is immediately usable in any agent-serving stack, and the "notes, not replay" framing is a genuinely useful lens on what prefill computes.