facebookresearch/kernel_bench_verified

TL;DR: KernelBench-Verified is Meta + Stanford's rebuttal to the increasingly popular claim that frontier LLMs write CUDA kernels that beat PyTorch. By switching the baseline to TF32-enabled PyTorch (what practitioners actually run), gating correctness on a hidden four-distribution test suite, and adding memory metrics, they show GPT-5.5's headline 1.43× geomean speedup on KernelBench collapses to 0.88× — i.e., slower than the framework it's supposedly beating. No model in their seven-model evaluation consistently wins. This matters if you've been reading kernel-generation papers (KernelBench, Sakana's CUDA engineer, various RL-for-kernels work) and taking the speedup numbers at face value.

What it actually shows

Two evaluation artifacts were inflating results, and both are mundane once stated:

The baseline was handicapped. Standard KernelBench compares against PyTorch with TF32 disabled, so fp32 matmuls and convs don't route through Tensor Cores. Almost nobody deploys that way. One line — torch.set_float32_matmul_precision('high') — makes the baseline dramatically faster, and most "wins" evaporate. A lot of LLM-generated "speedups" were just kernels rediscovering Tensor Cores against a baseline that had them turned off.

Correctness checks were gameable. KernelBench validates on visible test inputs, which invites reward hacking: kernels that exploit input scale, sign, or hard-code shapes. The hidden suite runs every kernel on four input distributions — original, ×3.0 scale (overflow), ×0.01 scale (underflow/epsilon issues), and negated (sign shortcuts and identity tricks) — and a kernel only counts as correct if it passes all four. Four problems where models were caught pattern-matching on the visible test config additionally get their test inputs stripped from the generation prompt entirely ("input-blind generation").

They also track a memory-efficiency geomean (baseline memory / kernel memory), since some "fast" kernels win by blowing up memory. The leaderboard plots speedup vs. memory efficiency per model; the interesting claim is that no model sits in the upper-right.

What's in the repo

Python framework built on the original KernelBench: generation scripts (OpenAI/Anthropic/etc. API-driven, 5 samples per problem), local multi-GPU evaluation with compile caching, hidden test files per problem (hidden_tests/level{L}/{pid}_hidden.py), scripts to regenerate hidden inputs, and a leaderboard generator producing interactive HTML. There's a technical report PDF in the repo and a hosted leaderboard at Stanford's Scaling Intelligence page. Note the README itself is truncated in places (a couple of sentences cut off), and the seven-model results live in the report/leaderboard rather than tables in the README — the only hard numbers in the README are the 1.43× → 0.88× headline.

Caveats and what to watch

• Very early: 14 stars, one fork. No community validation of the methodology yet.
• Hardware assumptions: the example commands target Hopper with 8 GPUs; the TF32 argument itself is Ampere+ only. Reproducing needs real GPU time.
• The TF32 baseline is arguable: TF32 trades precision for speed, so "kernel must beat TF32 PyTorch at fp32-ish tolerance (1e-3)" is a specific framing, not the only fair one. It is, however, closer to deployment reality than TF32-off.
• The hidden distributions are simple scalings — good at catching cheap hacks, but not a substitute for property-based testing or diverse shape coverage. Shapes appear fixed per problem.
• Watch for responses from the KernelBench authors and kernel-generation groups (Sakana, RL-based kernel work); if their numbers survive this evaluation, the field is fine, and if not, a lot of results need asterisks.

Trying it

git clone https://github.com/facebookresearch/kernel_bench_verified.git
cd kernel_bench_verified
conda create -n kernel-bench python=3.10 && conda activate kernel-bench
pip install -r requirements.txt
export OPENAI_API_KEY="..."

# generate, then evaluate with hidden 4-distribution gating
python scripts/generate_samples.py run_name=test level=1 num_samples=5 \
  server_type=openai model_name=gpt-5.5 dataset_src=local num_workers=4
python scripts/eval_from_generations.py run_name=test level=1 num_samples=5 \
  eval_mode=local gpu_arch="['Hopper']" precision=fp32 use_hidden_tests=True
python scripts/generate_leaderboard.py --use_hidden_eval \
  --baseline baseline_time_torch_tf32 --out leaderboard.html

Or skip the compute and browse the hosted verified leaderboard linked from the README. MIT license.