showlab/Show-Harness

TL;DR: Show-Harness (Show Lab @ NUS) is a bet against end-to-end VLA policies: instead of training a model to emit continuous actions, put a small, discrete vocabulary of incremental "action units" between a vision-language model and the robot, and let deterministic per-embodiment interpreters turn each unit into motion. Through that one interface a frontier VLM controls a real Franka zero-shot, and a 2B open model becomes a working policy with under a few H200 GPU-hours of LoRA. The release is unusually complete — code, six adapters, data, training pipeline, and a browser-based demonstration collector that needs no teleop hardware.

The idea

The core object is the "harness": a compact semantic action vocabulary (discrete, incremental units — think small moves, grasp/release, view/step modifiers) that the VLM reasons over at every step. Embodiment-specific interpreters ground each unit deterministically — Franka via impedance control (Polymetis), AgileX Piper via joint streaming, plus ManiSkill and Isaac Lab in sim. Switching robots swaps only the interpreter and a primitives_*.yaml; the model-facing vocabulary and prompts are identical. The VLM stays "directly responsible for every physical decision" — there is no learned low-level policy hiding the hard parts.

Two modes share this interface:

• Zero-shot: a frontier VLM (Gemini by default, any OpenAI-compatible endpoint) operates the full plugin harness with no robot-specific training.
• Fine-tuned: a small open VLM (Qwen3.5 0.8B–9B, Gemma 4) LoRA-tuned on demonstrations emits one action token per step, planner-free. That's a striking data point: the same interface degrades gracefully from "big model + scaffolding" to "tiny model + cheap fine-tune."

GUMI is the quietly clever part

Because actions are discrete units, demonstration collection collapses to a browser UI: every action unit maps to a key or button, and a human (or, notably, a GUI-driving agent) "plays" the robot with WASD/arrows. Every step is recorded as an (observation, action) pair in the exact format the fine-tune consumes — no teleop rig, no retargeting, no post-processing. The same bindings power live human takeover during autonomous rollouts. A synthetic --sim tabletop lets you try it with no hardware at all.

What's actually released

• Code: the full loop, GUMI, interpreters for four embodiments, and a plugin suite (subgoal planning, action chunking, proprioception, failure recovery, visual affordance prompts, etc.), each toggled by one boolean and byte-identical to no-plugin when off — deliberately ablation-friendly.
• Weights: six LoRA adapters on HF (Show-Harness-VLMs): Qwen3.5 at 0.8B/2B/4B/9B, Gemma 4 E4B, plus one sim policy covering both simulators.
• Data: the demonstration corpus (Show-Harness-Data) — real Franka/Piper rollouts plus RoboLab and ManiSkill.
• Training: LLaMA-Factory-based pipeline, self-contained, with "prompt contracts that must not drift" versioned per checkpoint.

Caveats

The README contains no quantitative results — no success rates, no task list, no baselines. Evidence is a demo video and the arXiv paper (2609.10522), which you'd need to read for numbers. The obvious question is how far a discrete incremental vocabulary stretches: this looks well-suited to quasi-static tabletop manipulation (pick, place, open, press) and poorly suited to anything dynamic, contact-rich, or dexterous — the same ceiling that killed earlier "LLM emits primitives" systems like Code-as-Policies, though the incremental per-step grounding here is finer-grained than one-shot code generation. Zero-shot mode also means a frontier-VLM API call per step, so expect slow, deliberate motion. Setup is real: three separate venvs (pins conflict), ROS/RealSense layer for hardware, and chat-template mismatches between training and vLLM serving fail silently (they warn you). Apache-2.0 throughout.

Try it

No hardware needed to feel the interface — drive the synthetic tabletop from a browser:

git clone https://github.com/showlab/Show-Harness && cd Show-Harness
bash scripts/setup.sh base
.venv/bin/python gumi/collect_rollouts_web.py data/rollouts_demo --sim
# open http://localhost:8600, drive the gripper with WASD / arrow keys

To run the fine-tuned 2B policy on a real Franka:

ADAPTER=qwen3_5_2b WITH_BASE=1 bash scripts/model/download_vlm_model.sh
MODEL=Qwen/Qwen3.5-2B \
  LORA=qwen3_5_2b_showharness_ft=models/Show-Harness-VLMs/qwen3_5_2b \
  FAMILY=qwen3_5 bash scripts/serve_vlm.sh
python scripts/run_real_mvtoken.py --robot-config configs/robot_franka_ft.yaml

Worth a look if you care about the "harness vs. end-to-end VLA" debate — this is the most complete open artifact on the harness side so far, even if the README makes you go to the paper for the evidence.