H-EmbodVis/TurboVLA

TL;DR: TurboVLA is a 0.2B-parameter vision-language-action policy that throws out the "project images into an LLM, decode actions from it" pipeline and instead fuses a DINOv3 vision encoder with a BERT text encoder via GroundingDINO-style bidirectional cross-attention, feeding a compact action-chunk decoder. Result: 97.7% average on LIBERO at 31.2 ms/inference and 0.9 GB VRAM on an RTX 4090 — i.e., genuine 32 Hz closed-loop control on a consumer GPU, matching much larger VLA policies. Code, training recipes, and checkpoints are all released under Apache-2.0.

The idea: V + L → A instead of V → L → A

The prevailing VLA recipe (OpenVLA, π0, RT-2 lineage) routes visual tokens through a multi-billion-parameter LLM before an action head, so every control tick pays the full LLM forward-pass cost. TurboVLA's claim is that for manipulation policies, the LLM as a perception–action interface is mostly overhead: what matters is task-conditioned visual features, and you can get those by directly exchanging information between independently-encoded vision and language streams.

Concretely, from the README's assets and acknowledgments: DINOv3 (ViT-B for LIBERO, ViT-L for RoboTwin) encodes camera views, BERT-base encodes the instruction online (no cached text features), and the two are fused with bidirectional vision-language interaction layers taken from — and initialized by — GroundingDINO (Swin-T OGC checkpoint). A small decoder then predicts continuous action chunks: 12-step chunks of 7-D actions for LIBERO, a 50-step ACT-style head with 14-D absolute joint positions for RoboTwin. This is essentially "open-vocabulary detection architecture repurposed as a policy," which is a neat transfer: GroundingDINO's fusion layers were already trained to ground language in pixels, which is most of what a manipulation policy needs from language.

TurboVLA (V + L → A) DINOv3 vision encoder BERT-base text encoder Bidirectional V–L fusion (GroundingDINO init) Chunk decoder 0.2B params · 31.2 ms · 0.9 GB VRAM

What's released and what the evidence looks like

Released: full training and evaluation code for LIBERO and RoboTwin 2.0, checkpoints plus normalization stats on Hugging Face (H-EmbodVis/TurboVLA), and reproducible recipes (LIBERO: 4 GPUs, 80k steps, batch 256, two camera views; RoboTwin: 4 GPUs, 55k steps, batch 192, three camera views, 50 clean tasks). Ascend NPU support is a TODO.

Evidence: 97.7% average success across the four LIBERO suites with 0.2B parameters — competitive with 7B-class policies at ~35× fewer parameters, per the abstract. The headline latency figures (31.2 ms, 0.9 GB) are measured on an RTX 4090. RoboTwin 2.0 numbers are claimed via released checkpoints but not quoted in the README (performance figures are in images that didn't survive the text export). Real-robot rollouts appear only as demo video placeholders; there are no quantitative real-world results here.

Caveats

• Simulation-heavy. LIBERO is close to saturated across recent methods, and there are no hardware numbers. The interesting open question — does dropping the LLM hurt instruction generalization? — is untestable from these benchmarks, whose language is templated. BERT + GroundingDINO fusion should handle "pick up the red mug" grounding well, but compositional or novel instructions are unproven.
• Per-suite checkpoints. The LIBERO eval uses suite-specific checkpoints (e.g., libero_object.pth), though training mixes all four suites' data; worth checking the paper for whether the 97.7% comes from a single policy.
• Dependency stack. You need DINOv3, BERT, and the GroundingDINO checkpoint separately; DINOv3 weights are gated.
• Note this isn't the first small-policy result — VLA-Adapter and ACT-lineage work also questioned LLM necessity — but the GroundingDINO-as-policy-backbone framing and the latency/VRAM package are a clean, usable contribution.

Try it

git clone https://github.com/H-EmbodVis/TurboVLA.git && cd TurboVLA
conda create -n turbovla-libero python=3.10 -y && conda activate turbovla-libero
pip install torch==2.3.1 torchvision==0.18.1 --index-url https://download.pytorch.org/whl/cu121
pip install -e ".[libero]"   # plus LIBERO itself
hf download H-EmbodVis/TurboVLA --local-dir pretrained/TurboVLA

python experiments/libero/evaluate.py \
  --ckpt_path pretrained/TurboVLA/checkpoints/libero/libero_object.pth \
  --dinov3_path facebook/dinov3-vitb16-pretrain-lvd1689m \
  --bert_path google-bert/bert-base-uncased \
  --stats_path experiments/libero/configs/libero_all4_stats.json \
  --stats_key libero_all4_no_noops \
  --task_suite_name libero_object --num_trials_per_task 50 \
  --chunk_size 12 --num_open_loop_steps 12 --precision bf16

Worth a clone if you do manipulation research: at <1 GB VRAM you can run the policy alongside a simulator on one consumer GPU, which alone changes the iteration loop.