0xShug0/audio.cpp

TL;DR: audio.cpp is llama.cpp for the entire audio stack — a single ggml-based C++ runtime covering TTS, voice cloning, voice conversion, ASR, diarization, VAD, source separation, forced alignment, codecs, and music generation, with no Python at inference time. It already spans 62 model families / 85+ variants (Qwen3-TTS/ASR, VibeVoice, IndexTTS2, Higgs Audio, Voxtral, Parakeet, ACE-Step, MiniMax Music 3…), ships GGUF quantized packages, and reports 1.8–8x speedups over Python reference paths on CUDA. Two months old, moving extremely fast, and likely to become default local-audio infrastructure.

What it is and why it matters

The audio model ecosystem has the dependency problem LLMs had in 2022: every model ships its own Conda environment, its own vocoder fork, its own incompatible torch pin. audio.cpp replaces that with one native runtime: shared GGUF loading, reusable sessions, common DSP utilities (STFT/ISTFT, resampling, denoise), and unified CLI/server/WebUI entry points across all tasks. Backends are ggml's usual suspects — CUDA, HIP/ROCm, Vulkan, Metal, CPU — so the same binary story works on an RTX 5090, an AMD iGPU, or a MacBook.

The interesting bet is architectural: rather than one-off ports (whisper.cpp, bark.cpp, etc.), it builds shared components — a Conv1DTranspose CUDA kernel, streaming session machinery, LoRA loading, batch inference — that every new model family reuses. That's why coverage went from first release to 62 families in roughly two months.

Evidence and numbers

All figures are self-reported, but there's real parity tooling (tests/warmbench.py runs long-lived sessions against Python references) and new-model PRs require reproducible validation. Headline claims:

• Multiple CUDA TTS paths run 1.8–8x faster than Python references, with 45–85% lower end-to-end latency.
• VibeVoice 1.5B: a 93.9-minute podcast in 18.2 minutes (5.15x realtime) at 10 diffusion steps, no quantization.
• Supertonic 3: ~10 hours of audio in 3 minutes on an RTX 5090 (200x+ realtime), 47 ms TTFT streaming.
• Nemotron 3.5 ASR matched other implementations' WER on messy French meeting audio (TranscrIA benchmark) at ~1/4 the wall time — the one third-party-ish datapoint.
• Q8 GGUF packages run up to 1.53x faster with ~37% less peak VRAM vs 16-bit.

What's actually there

Code (C++), prebuilt binaries for Windows (CPU/Vulkan/CUDA), Ubuntu (CPU/Vulkan), and macOS (Metal), plus Docker images and a Homebrew tap. GGUF packages are published under audio-cpp/audio.cpp-gguf, installable via a native model manager (audiocpp_model_manager install ...). The server embeds a compiled SvelteKit WebUI — including an "Arena" tab for side-by-side model comparison with shared inputs and metrics — and exposes OpenAI-style endpoints (/v1/audio/speech, /v1/audio/transcriptions). Composite builds let you compile only the families you need. Experimental JSON pipelines chain steps (the included redub pipeline: chunked Qwen3-ASR → merge → Qwen3-TTS in a target voice over a 7-minute input).

Caveats

• No license visible in the README — check the repo before shipping anything; note also that individual model weights carry their own licenses (many of these families are non-commercial).
• Almost all performance numbers are the author's own; "parity" coverage varies by family, and the community-model tier explicitly has lighter review. Verify parity status for any model you care about in docs/.
• Some model preparation jobs (HF download/conversion) still shell out to Python; only inference and the UI are Python-free.
• The project is ~2 months old with a very high commit velocity — expect API churn. Streaming is per-model, not universal.

Quickest path

macOS:

brew tap 0xShug0/audio-cpp && brew install audio-cpp
audiocpp_server --ui --ui-management --backend metal
# open http://127.0.0.1:8080 — browse the catalog, download a model, run it

Or from source with just the models you want:

git clone https://github.com/0xShug0/audio.cpp && cd audio.cpp
scripts/build_linux.sh --backend cuda --model-set custom \
  --models qwen3_tts,qwen3_asr --target audiocpp_cli
build/linux-cuda-release/bin/audiocpp_cli --task tts --family qwen3_tts \
  --model models/qwen3-tts --backend cuda --text "Hello from audio.cpp" --out out.wav