IFM/K2-Horizon-MoVA-36B-A4B

TL;DR: K2-Horizon-MoVA-36B-A4B is an Apache-2.0 MoE language model with 36B total / ~4B active parameters, native 512K context, and a new attention-side sparsity mechanism the IFM team calls Mixture-of-Values attention (MoVA). The headline claim — matching or beating open models up to ~15× its active size on agentic benchmarks — would be a real efficiency jump, but the model card explains almost nothing about how MoVA works, and the "fully open" parts (data, training code, intermediate checkpoints) are promised, not shipped.

What it is and why you should care

This is the sparse member of IFM's K2-Horizon family. Two things make it worth your time. First, MoVA suggests routing sparsity applied inside attention (presumably over value projections) rather than only in FFN experts — a second sparsity axis that, if it works, other labs will copy. Second, the release philosophy: intermediate checkpoints plus training data and code would make this one of the few frontier-adjacent models you can actually study across training, à la Pythia/OLMo but at a much higher capability level.

The catch: the model card is a benchmark table plus serving recipes. There is no architecture section, no paper link, no explanation of what MoVA does, how many experts there are, or how the router works. Everything about the mechanism is currently inferable only from the name and the custom_code in the repo.

How it (apparently) works

What the card actually states: MoE architecture, 36B stored / 4B activated per token, Mixture-of-Values attention, 524,288-token context native from midtraining onward. Reasoning depth is controlled per-request (reasoning_effort in the chat template), thinking is emitted separately in reasoning_content, and tool calls come in three switchable formats (json, xml, xml_typed) with a dedicated k2_horizon parser. The SGLang recipe includes a router-numerics override (xllm_source_router_gemm_partitions), which hints the routing math is numerically fragile enough to need serving-side care — mildly unusual and worth watching if you deploy it.

If you want the mechanism details, read the modeling code in the repo; the README won't tell you.

The evidence

The benchmark table pits it against open models with 3B–55B active parameters, scored via Artificial Analysis, all in reasoning mode. Where it genuinely stands out is agentic work:

Beating a 55B-active MoE on Terminal-Bench 2.1 (58.6 vs 53.9) and nearly doubling everyone on tau3-Banking at 4B active parameters is the striking result. Elsewhere the picture is honest but mixed, and credit to them for including it: HLE 25.2 vs Nemotron Ultra's 28.4, GPQA Diamond 80.8 is actually the lowest in the row, SciCode trails the dense 30B models, and long-context reasoning (AA-LCR 66.3) loses clearly to Muse Glimmer-30B's 80.0 — notable given the 512K-context headline. Non-hallucination rate (69.2) is decent but well behind G9v3's 87.0. So: read this as "exceptional agentic tool-use per active parameter," not "frontier across the board."

What's actually there, and maturity

Shipped now: final checkpoint (safetensors, bf16), custom modeling code (trust_remote_code required), chat template with reasoning-effort control, validated vLLM and SGLang recipes with dedicated reasoning/tool-call parsers. Apache-2.0, English-only tag. 262 likes and #12 trending in week one.

Promised, not shipped: training data, training recipe, training code, intermediate checkpoints. Until those land, the "radically open" framing is aspirational — worth checking back in a month.

Hardware: recipes assume 2× H200 with TP=2 + expert parallel; bf16 weights alone are ~72GB, so realistically two 80GB-class GPUs minimum. The vLLM recipe caps at 128K context, so the 512K claim likely needs more memory or a different config than what's documented. llama.cpp-family support is immature. Recommended sampling is temperature=1.0, top_p=0.95, always reasoning_effort="high" — and note all reported numbers use high effort, so budget for long generations (the examples set max_tokens=32768).

Shortest path to trying it

python3 -m sglang.launch_server \
  --model-path IFM/K2-Horizon-MoVA-36B-A4B \
  --tp 2 --ep 2 --dtype bfloat16 \
  --attention-backend fa3 \
  --json-model-override-args '{"xllm_source_router_gemm_partitions":2}' \
  --reasoning-parser k2_horizon --tool-call-parser k2_horizon \
  --host 0.0.0.0 --port 30000

Then hit it with any OpenAI client, passing extra_body={"chat_template_kwargs": {"reasoning_effort": "high"}}. Model card: https://huggingface.co/IFM/K2-Horizon-MoVA-36B-A4B

Bottom line: treat the agentic numbers as a claim worth independently verifying, poke at the custom code to figure out what MoVA actually is, and watch whether the training artifacts ship. If both hold up, this is the most interesting sparsity result of the season.