Multimedia-Semantic-Analytics-Lab/PerceptionDLM

TL;DR: PerceptionDLM is an 8B multimodal diffusion language model (built on LLaDA) that exploits the one property diffusion LMs genuinely have over autoregressive models — parallel decoding — for a task where it actually matters: given an image and N region masks, it captions all N regions in a single denoising pass instead of N sequential generations. The result is ~2.9× tokens-per-forward and 276s vs 479s against an AR region captioner on their new benchmark, at a real but bounded quality cost (62.4 vs 69.5 avg). Everything is released: code, weights, training data, and the benchmark, under Apache-2.0.

Why this is interesting

Diffusion LLMs have mostly been a solution in search of a problem: in single-response chat, parallel decoding buys little because you still pay many denoising steps and quality drops when you decode too many tokens per step. This project picks a task with natural structure for parallelism — dense region captioning, where an image with 20 masks means 20 short, largely independent captions. AR VLMs pay latency linear in region count; a diffusion LM can denoise all captions as one batch of masked spans inside one forward pass. That's a genuinely different decoding regime for perception, not another point on a benchmark, and it's the clearest "diffusion LM earns its keep" demonstration I've seen.

PerceptionDLM: all regions in one denoising process image + N masks shared denoising steps caption span 1 (masked → text) caption span 2 (masked → text) caption span N (masked → text) N captions, one pass latency ≈ constant in N

How it works

Two models. PerceptionDLM-Base is a diffusion VLM: LLaDA-8B-Instruct as the language backbone, trained through a 4-stage LLaVA-style pipeline on open corpora (Bee stages 1–2, LLaVA-OneVision-1.5, Honey-Data-15M). It's positioned as a new SOTA open discrete-diffusion VLM, beating LLaDA-V on 15/16 multimodal benchmarks — the README shows this as an image so exact numbers aren't in the text.

PerceptionDLM proper is fine-tuned from Base on region mask/caption data (DAM, COCONut, SAM-derived annotations, released as PerceptionDLM-Data, ~2 days on 32×H100). At inference it takes an image plus multiple binary masks and lays out one masked caption span per region, then runs a single shared denoising schedule that fills all spans simultaneously. TPF (tokens per forward) of 2.9 vs 1.0 for both AR and naive diffusion baselines quantifies the parallelism.

They also release ParaDLC-Bench, a multi-region localized captioning benchmark that scores quality and efficiency jointly — a sensible artifact, since existing dense-captioning evals ignore latency.

Evidence and honest caveats

The headline table:

Read this carefully: PerceptionDLM does not match the AR specialist on quality — 62.4 vs GAR-8B's 69.5, a ~7-point gap bought with a 1.7× wall-clock speedup (the "3.4×" highlight presumably refers to the densest multi-region regime; the table shows 2.9× TPF). The fairer framing is a new point on the accuracy–latency Pareto frontier, not a free win. The more striking number is LLaDA-V collapsing to 35.2 at 3241s — a generic diffusion VLM handles neither the region grounding nor the parallel layout, so the fine-tuning and span-layout design are doing real work. Also note the caveat you'd expect: parallel spans can't condition on each other, so cross-region consistency (e.g., distinguishing similar objects) is exactly where I'd probe for failures.

The release is unusually complete: 8B weights for both models on HF, the converted LLaDA backbone, training data, training configs and scripts, the benchmark, VLMEvalKit integration, and a Gradio demo. Reproducing Base from scratch is heavy (~3 weeks on 32×H100), but inference of an 8B model is single-GPU territory. Repo is days old (77 stars), so expect rough edges; the paper is arXiv-only.

Try it

curl -LsSf https://astral.sh/uv/install.sh | sh
git clone https://github.com/MSALab-PKU/PerceptionDLM.git
cd PerceptionDLM
uv sync --extra=pdmllm && source .venv/bin/activate

python demo/infer_pdmllm.py \
  --model-path MSALab/PerceptionDLM \
  --image assets/demo.jpg \
  --masks assets/demo_mask_0.jpg assets/demo_mask_1.jpg assets/demo_mask_2.jpg \
  --gen-length 32 --steps 32 --temperature 0.0 --top-p 1.0

Models and data: HF collection · paper · project page.