z-lab/Qwen3.8-27B-DFlash2

TL;DR: DFlash 2 is a block-diffusion draft model for lossless speculative decoding of Qwen3.8-27B — it drafts a whole block of tokens in one parallel pass instead of autoregressively, and a lightweight selector picks one coherent candidate path for the target to verify. On an H200 it beats both Qwen's built-in MTP head and a community EAGLE-style drafter, hitting 3.43× over plain autoregressive at concurrency 1 and, notably, still 1.16–1.45× at concurrency 32 where the other drafters go below 1×. Already merged into SGLang and landing in vLLM, with 141K downloads in days.

The idea

Standard speculative drafters (EAGLE-style, MTP heads) are themselves autoregressive: seven draft tokens cost seven sequential draft passes. DFlash 2 replaces the drafter with a small block-diffusion model that predicts all positions of an 8-token block in a single forward pass, keeping the top candidates at each position. A lightweight selector then traces one coherent path through that candidate lattice, and the target model verifies it in one pass as usual. Decoding stays lossless — greedy output matches the target exactly, sampling preserves its distribution — so this is purely an inference-speed play, not an approximation.

The README mentions one architectural fix worth noting: "two-tap dynamic convolutions in the backbone keep the draft from decaying toward the end of the block." That addresses the known failure mode of parallel/diffusion drafting, where positions far from the committed context become low-quality and tank acceptance length. The acceptance numbers suggest it works: DFlash 2 accepts more tokens per verification step than autoregressive MTP (5.46 vs 5.02 on GSM8K, 4.39 vs 3.91 on HumanEval), despite drafting in parallel. That's the surprising bit — parallel drafting usually trades acceptance for draft speed; here it wins on both.

One drafter pass proposes the whole 8-token block with alternatives at each position; the selector's path is what the target verifies, so draft latency stops scaling with block length.

The evidence

Benchmarks are on one H200, SGLang, FlashAttention 3, temperature-1.0 sampling with Qwen's recommended parameters, 7 draft tokens per step, across GSM8K, MATH-500, HumanEval, MBPP, MT-Bench. The high-concurrency behavior is the headline for anyone running a serving fleet:

Speculative decoding classically dies under batching because GPU compute is saturated and drafting is pure overhead. DFlash 2's single-pass drafting cuts that overhead enough to stay at or above break-even at concurrency 32, where MTP and DSpark are actively slower than plain decoding (0.74–0.94× on most tasks). MT-Bench at 32 is a wash for DFlash 2 too (1.01×), so this isn't magic — just a much better overhead curve.

What's actually here, and caveats

The repo contains only the drafter weights (safetensors, Apache-2.0) — it's useless standalone and is target-specific to Qwen3.8-27B. The method details live in a blog post and an ICML 2026 DFlash paper; the model card itself is thin on drafter size, training data, and selector mechanics, so I can't tell you the parameter count or training cost. Engine support is bleeding-edge: SGLang from git main, and vLLM from an unmerged PR branch (#52816), so expect churn. Numbers are from the vendor (Inco AI) on one H200 with FA3; the selection note flags that you should reproduce on your own hardware and batch sizes before believing the curves. The same technique reportedly ships for other targets (a Muse-Glimmer drafter claims up to 4.6×), which suggests the training recipe generalizes — worth watching for a drafter-training release.

Try it

pip install "sglang[all] @ git+https://github.com/sgl-project/sglang.git#subdirectory=python"

python -m sglang.launch_server \
  --model-path Qwen/Qwen3.8-27B \
  --speculative-algorithm DFLASH \
  --speculative-draft-model-path incoai/Qwen3.8-27B-DFlash2 \
  --speculative-num-draft-tokens 8

Model card: https://huggingface.co/z-lab/Qwen3.8-27B-DFlash2