TacPAC: Tactile Prediction and Real-Time Action Correction in World-Action Models for Contact-Rich Manipulation

TL;DR: World-action models plan robot actions by first imagining the future — but the imagined future is frozen the moment the plan is committed, while the tactile feedback that decides contact-rich tasks arrives during execution. TacPAC's move: after planning an action chunk, cache the predicted contact the plan was conditioned on (as attention key/value states), then let a small "tactile expert" read each incoming tactile frame against that cached expectation and emit a residual correction to the unexecuted part of the chunk. One correction is a 30.4 ms forward pass — 20.7× cheaper than replanning — and on five real-robot tasks it lifts the vision-only base from 22% to 64% average success. The most interesting finding is negative: naively predicting tactile frames as extra video views recovers only a third of that gain, and reactive tactile correction without an expectation actually hurts on fragile-object tasks.

The timing mismatch nobody was pricing in

A quick tour of the setup. Modern robot policies (VLAs like \pi_0, ACT-style imitation) map observations directly to a chunk of future actions — here H = 48 steps issued at 30 Hz, so each chunk is 1.6 seconds of committed motion. World-action models (WAMs) go one step further: alongside the action chunk, they generate predicted future observations, and the action tokens attend to those predictions at every layer. The plan is thus conditioned on the consequences the model expects its own actions to have. Think of it as planning with a mental simulation attached.

The predictions in these models are almost entirely visual. That's fine for pick-and-place, but contact-rich manipulation — plugging in a charger, seating a PCIe card, lifting a potato chip without cracking it — is decided by signals a camera barely sees: incipient slip, a half-millimeter insertion misalignment, grip force on a compliant object. Tactile sensors (vision-based fingertip sensors like GelSight; here, InTac S1) see these directly.

So the obvious fix is to add tactile images as extra views in the WAM's predicted video stream. The model then predicts future contact alongside future pixels, and the action chunk is conditioned on both. The authors do exactly this — and it helps, raising their base model from 22% to 37% average success. But the full system reaches 64%. Prediction alone captures roughly a third of the available gain.

Why? Two reasons, and the paper quantifies the first with a nice diagnostic (Appendix E). Tactile information is temporally sparse: measuring frame-to-frame tactile change over 296 training episodes, contact transitions occupy 4.9% of frames but carry 18.6% of the total tactile change, with activity 4.1× the episode mean inside those windows versus 0.58× during free-space approach. A generative objective over the whole tactile stream is dominated by frames where nothing happens, so it needn't make action generation sensitive to the transitions that matter.

The second reason is structural and is the paper's core observation: the prediction is made before execution, but the feedback arrives during it. Once the chunk is committed, the WAM's anticipated contact is an assumption fixed in the past. When reality diverges — the plug catches on the socket rim, the chip is more brittle than expected — nothing in the architecture can act on the discrepancy until the next planning cycle, 1.6 seconds later. And replanning the whole chunk (628.6 ms of diffusion sampling here) is far too slow to track a tactile stream.

The key idea: feedback needs an expectation to be read against

Here's the framing that makes the mechanism click. A tactile image, by itself, is ambiguous. "The fingers are loaded" can mean everything is going to plan (you expected to be pressing the plug in right now) or something is wrong (you expected free space). The same reading calls for opposite corrections depending on what the plan anticipated at this point in execution. A corrector that sees only the current reading is doing blind reactive control; a corrector that sees the reading and the expectation is doing error-driven control — the tactile analog of comparing an efference copy against reafference.

The WAM already contains the expectation. The chunk was denoised jointly with a predicted tactile stream and attended to it at every layer. TacPAC's insight is to keep that prediction around and hand it to a correction module.

The mechanism, concretely

The architecture is a three-expert Mixture-of-Transformers (all sharing 30 layers and identical attention geometry, so they can consume one another's per-layer states). A video expert generates future visual and tactile latents via flow matching; an action expert generates the chunk, with an asymmetric attention mask so action queries attend to everything while video queries never see noisy actions. This base model is trained in Stage 1, and is essentially LingBot-VA with tactile views tiled into the predicted video canvas.

The new parts:

The cache. After the ten denoising steps finish, the model runs one more forward pass over the clean video latent and planned chunk \hat{\mathbf{a}} at flow time \tau=0, storing per-layer attention keys and values. (They pay for this extra pass deliberately — the KVs from the final denoising step still encode a slightly noisy prediction.) The cache \mathcal{C} keeps two token groups: the predicted tactile views (what contact the plan expects across the horizon) and the action tokens (the plan as the model internally represented it while attending to that prediction). Visual keys are dropped — the scene changes slowly over a 1.6 s chunk and its influence is already baked into the retained tokens. This cache is built once (62.6 ms) and read unchanged by every correction.

The tactile expert. When a tactile frame arrives after m actions have executed, the expert's query sequence contains the encoded frame plus an updated copy of the chunk: the first m entries overwritten with the commands actually sent, the rest keeping their planned values, each tagged with a learned executed/unexecuted status embedding. Every layer attends jointly to \mathcal{C} and its own tokens, and the output is a per-step delta \Delta\mathbf{a} added to the original plan. Two design choices matter here: corrections are conditioned on the execution offset m (not a flow time — this is a deterministic regressor, not a diffusion model), and each new delta replaces the previous one on the remaining steps rather than compounding, so a spurious correction doesn't propagate.

Training the corrector on the model's own mistakes. Stage 2 freezes everything and trains only the tactile expert (initialized from the action expert, output head zero-initialized so the first correction is the identity). Crucially, the frozen base model plans a chunk under the test-time denoising schedule, the cache is prefetched from that plan, and the supervision target is the residual between the plan and the ground-truth demonstration on the unexecuted suffix:

with K=4 offsets m sampled per episode, all sharing one plan and one prefill. Training on the model's own plans rather than ground-truth chunks is what keeps the training and deployment distributions matched: the quantity being learned is precisely "how the base model's plans go wrong, given tactile evidence at this stage." This is a residual-policy / DAgger-flavored idea executed cleanly inside the WAM's own representation space.

Asynchrony. The robot starts executing as soon as the chunk exists; the prefill computes while the first actions fly. A correction conditioned at offset m is written back from m+d onward, where d is the recent worst-case correction latency in action steps. In practice d \approx 0: only 1.6% of corrections see the robot advance a control step while computing, and the authors report no correction-induced jerks.

The evidence

The five tasks — charger-plug insertion, three-fruit transfer, fragile potato-chip transfer, empty-bottle uprighting, expansion-card insertion — span tight tolerance, force regulation, and damage sensitivity. Twenty trials per method per task, binary success, damage counts as failure.

The comparison against the two closest tactile WAMs is where the argument gets sharp. Dream-Tac predicts contact but commits its chunk in a single generative pass — it leads baselines on the damage-scored tasks (chip 85%, fruit 40%), where the grip must be right as contact is established, but loses on the insertions (plug 55%, card 20%), where the misalignment only becomes observable mid-chunk. T-Rex corrects a chunk from the live tactile stream but predicts nothing — the opposite pattern: strongest baseline on both insertions, but on damage tasks it reacts only after the error is physically sensed, and a crushed chip can't be un-crushed. TacPAC leads both groups (chip 90%, fruit 65%, plug 80%, card 45%), exactly the pattern you'd expect if the corrector responds to how contact departs from expectation rather than to the raw reading.

The ablation (Table 1) is the paper's key table, and it's unusually well-designed — each row severs one pathway while holding the backbone, corrector, tactile input, and parameter count fixed:

Read this chart from left to right and the argument assembles itself. Prediction alone: +15 points. Reactive correction alone: +11 — and note that the reactive variant is worse than vision-only on the damage-sensitive tasks (chip 45% vs. 60%, fruit 25% vs. 30%), direct evidence that tactile feedback without an expectation can drag actions away from an already-correct plan. Both together but with the corrector blinded to the cached prediction: 47%. Grant it that single attention pathway and success jumps 17 points to 64%. That last comparison is as clean as ablations get in real-robot work — same model, same parameters, same inputs, one attention mask changed.

On efficiency: chunk generation is 628.6 ms (1.59 Hz), one correction is 30.4 ms (32.9 Hz), the claimed 20.7× ratio. The point isn't the specific number — it's that corrections keep pace with the tactile stream while replanning fundamentally can't.

What to take away, and what to squint at

The transferable idea is bigger than tactile: in any model that plans by predicting consequences, the prediction shouldn't be discarded once the plan is committed — it's a free, already-computed expectation that turns ambiguous execution-time feedback into an interpretable error signal, and caching it as KV states lets a lightweight module consume it at sensor rate. This is closed-loop control with the "reference signal" supplied by the world model's own imagination, and the cache trick makes it nearly free. It would apply just as naturally to audio feedback, force-torque, or proprioceptive surprise. There's also a pleasing predictive-coding resonance — corrections driven by prediction error rather than raw observation — which the paper doesn't belabor but which readers with a neuroscience bent will notice.

The caveats. The vision-only base at 22% is weak — \pi_{0.5} scores 36% without tactile — so the 42-point headline gain is measured against a low anchor; the honest comparison is +16 over T-Rex, still substantial but less dramatic. Twenty trials per cell means ±10-point noise on individual task numbers; the average over five tasks and the consistent ablation ordering carry most of the statistical weight. Everything is single-arm, single-gripper, single tactile sensor type, no pretraining, and the corrector is trained per-setup — generalization is untested. Dream-Tac's gating hyperparameters were adapted to the authors' sensors, which is fair but means baseline tuning effort wasn't fully symmetric. And the "no cache" ablation still plans under tactile prediction, so the 47→64 gap is specifically about the corrector's read access, not about prediction per se — the paper is careful about this, but it's easy to misquote.

Where to spend your time: Section 3.2 (the cache and correction mechanism, ~2 pages, very readable) and the Table 1 ablation in Section 4.3. Appendix E's tactile-sparsity diagnostic is a nice methodological bonus — including the honest accounting of HEVC codec artifacts in their tactile-change measurements, a detail most papers would quietly omit.