Organization of the Hearsay II speech understanding system

TL;DR — In 1975, the Hearsay-II team proposed that speech understanding should emerge from many independent "knowledge sources" — acoustics, phonetics, lexicon, syntax, semantics — reading and writing hypotheses on a shared blackboard, with a scheduler that opportunistically runs whichever module can most reduce global uncertainty. It failed against brute-force search because every module was weak and hand-built. Fifty years later we have absurdly strong modules (foundation models) and we are rebuilding the blackboard badly, as ad hoc multi-agent frameworks with no scheduling theory. The part of Hearsay-II worth resurrecting is precisely the part everyone forgot: the control layer.

The idea as they had it

The ARPA Speech Understanding Research program (1971–1976) set a target that sounds quaint now: understand connected speech from a 1000-word vocabulary, in a constrained task domain, with roughly 90% semantic accuracy, in something within shouting distance of real time. Nobody knew how to do it. The signal was ambiguous at every level — segments blurred by coarticulation, words with no acoustic boundaries, syntax that constrained but didn't determine — and no single algorithm could resolve it.

Lesser, Fennell, Erman, and Reddy's answer was architectural rather than algorithmic. Picture a blackboard divided into horizontal levels of abstraction: parametric signal, segments, syllables, words, word sequences, phrases. Hypotheses live on the blackboard as data structures with time extents and credibility ratings. Around the blackboard sit knowledge sources (KSs): independent specialist programs, each a condition–action pair. A KS watches for a pattern of hypotheses it knows something about (its condition), then fires, creating or re-rating hypotheses at its own or adjacent levels (its action). A word hypothesizer sees a plausible syllable string and posts word candidates; a syntax KS sees two words that could be adjacent in the grammar and predicts a third; a rating policy propagates the new evidence up and down.

Crucially, nothing is pipelined. Processing is bidirectional and asynchronous — bottom-up from acoustics, top-down from semantic predictions — and the KSs never call each other. They communicate only through the blackboard. This gives you what we'd now call loose coupling: you can swap a KS, add one, or run several in parallel (Hearsay-II was explicitly designed for multiprocessing, which in 1975 was radical) without touching the rest of the system.

The second half of the idea, the half that got lost, is control. At any moment dozens of KS instantiations are triggered. Which do you run? Hearsay-II kept an agenda of pending KS activations, each scored by an estimate of how much it would improve the best global interpretation — roughly, the credibility of the hypotheses it would touch, the reliability of the KS, and the "stimulus" region's importance. A focus-of-attention mechanism steered work toward islands of reliability: find a high-confidence word anchor mid-utterance, then grow the interpretation outward from the island in both directions, rather than marching left-to-right through the noisiest part of the signal. Search was opportunistic: driven by where the uncertainty was, not by a fixed order of operations.

Syntax KS Lexicon KS Phonetics KS

Scheduler agenda of triggered KS activations, ranked by expected uncertainty reduction state run KS Island of reliability: grow the interpretation outward from a confident anchor

Why it could not work then

Put numbers on the wall it hit. Hearsay-II ran on a DEC PDP-10 class machine: roughly 0.5 MIPS and on the order of a megabyte of memory. A modern GPU node delivers on the order of 10^{14}–10^{15} operations per second — roughly nine to ten orders of magnitude more compute — plus terabytes of fast storage. Under the 1975 budget, every knowledge source had to be a small hand-crafted program: phonetic rules written by phoneticians, a hand-built grammar with a deliberately low branching factor (roughly 30–40 words possible at each point), a vocabulary of about 1,000 words. Processing an utterance took many times real time even so.

The deeper problem was that weak modules poison the architecture's own logic. The blackboard's scheduling depends on hypothesis ratings meaning something: "run the KS that most reduces uncertainty" is only coherent if credibility scores are roughly calibrated. But the acoustic-phonetic KSs of 1975 were unreliable enough that ratings were noisy, islands of reliability were sometimes islands of confident error, and the opportunistic search wandered. The team spent enormous effort tuning rating and focus policies by hand — control knowledge that was as brittle as the domain knowledge.

And there is an honest, uncomfortable footnote: within the same ARPA program, CMU's Harpy — which compiled all knowledge into one enormous finite-state network and ran a beam search through it — was the only system to clearly meet the ARPA goals, with sentence error around 5% on its constrained task, versus noticeably worse for Hearsay-II. Integrated brute-force search beat modular cooperation. If that sounds familiar, it should: it is the same lesson end-to-end deep learning taught us against modular pipelines in the 2010s, delivered 40 years early. The blackboard lost not because the idea was wrong but because, when your components are weak, fusing everything into a single searchable structure and spending compute on search wins.

What changed

Two things, one obvious and one underappreciated.

The obvious one: the knowledge sources are now strong. A streaming ASR model, a speech-emotion or speaker model, a vision-language model, an LLM with world knowledge and task context — each of these is individually superhuman relative to any 1975 KS, and each naturally emits scored hypotheses. The failure mode "every module is too weak to contribute reliably" is gone.

The underappreciated one: the field has spontaneously reinvented the blackboard, minus its theory. Look at any 2024–2025 multi-agent LLM framework — AutoGen, LangGraph, CrewAI, open "society of agents" setups. The shared conversation history or scratchpad is a blackboard: heterogeneous agents post partial results into a common workspace and trigger on each other's outputs. But scheduling is almost always trivial — round-robin, a fixed graph, or "ask another LLM who should speak next." There is no explicit uncertainty representation, no expected-value-of-computation reasoning, no focus of attention, no principled account of when to stop. Hearsay-II and its descendants (especially Barbara Hayes-Roth's BB1, which put control knowledge itself on a blackboard) spent a decade on exactly these questions. That literature is the missing piece, and almost nobody in the agents community has read it.

There is also a neuroscience echo the 1975 authors couldn't have anticipated: Bernard Baars's Global Workspace Theory of consciousness was explicitly modeled on blackboard architectures — specialist processes competing for access to a shared broadcast workspace. The modern "global workspace for AI" proposals (Goyal, Bengio, VanRullen, Dehaene-adjacent work) are Hearsay-II's architecture wearing a cortical costume.

A serious 2026 revival

Take streaming multimodal understanding as the testbed — a live meeting, a robot's sensorium, a long video — where no single end-to-end model comfortably covers all levels and latency budgets differ per module.

Keep from the paper:

• The layered blackboard as the only communication medium: typed, time-extended hypotheses (audio events, transcript spans, entities, intents, scene states), each carrying a confidence and provenance. This is a structured world-model, not a chat log.
• Condition–action KSs: each foundation model wrapped with a trigger pattern ("a low-confidence transcript span overlapping a detected face turn") and an action ("re-decode with speaker-conditioned ASR; post revised span").
• Opportunistic, uncertainty-driven scheduling with islands of reliability: spend your expensive LLM calls where confident anchors border confused regions, exactly as Hearsay-II grew interpretations outward from reliable words.

Replace:

• Hand-tuned rating functions with calibrated probabilities. Modern models give you logprobs, verbalized confidences, ensemble disagreement; calibration is imperfect but tractable, and it's the load-bearing assumption of the whole architecture.
• Hand-crafted focus-of-attention heuristics with a learned scheduler: framed properly, "which KS activation to run next" is a budgeted RL / value-of-computation problem — state is the blackboard, actions are KS invocations with known dollar-and-latency costs, reward is final task accuracy minus cost. This is where the interesting research lives, and it connects directly to current work on adaptive test-time compute, routing, and speculative execution. Hearsay-II's scheduler was a hand-coded policy for this exact MDP.
• Serial firing with genuine asynchrony: cheap fast KSs (VAD, diarization, keyword spotters) run continuously and post anchors; expensive KSs (long-context LLM reasoning) fire sparsely where the scheduler sends them. The 1975 paper's multiprocessing design maps cleanly onto heterogeneous fleets of small and large models.

The experiment that would settle whether the control theory matters: fix the KSs, compare (a) a fixed pipeline, (b) an LLM-orchestrator choosing what to call, and (c) a learned uncertainty-driven scheduler, on accuracy-per-dollar under a compute budget. Hearsay-II's implicit claim is that (c) dominates, and increasingly so as the number and cost-spread of KSs grows.

Has it been vindicated? What's open?

Partially. Blackboards had a real career: HASP/SIAP for sonar interpretation, BB1, crisis-management and sensor-fusion systems through the 1990s. Global Workspace Theory carried the architecture into neuroscience. And today's agent frameworks are convergent evolution toward the same shape. In that sense the architecture is vindicated as an attractor.

What is not vindicated is the control theory, because nobody has seriously ported it. Open problems: getting confidences from heterogeneous foundation models onto a common calibrated scale (the hardest part, and the one that sank Hearsay-II); credit assignment through long chains of blackboard writes; stopping criteria; and the standing Harpy question — whether, for any fixed task, a single end-to-end omni-modal model will simply eat the modular system again. The honest bet is that end-to-end wins wherever you can afford to train it, and the blackboard wins where components must be heterogeneous, updated independently, priced differently, or auditable — which describes most deployed agent systems.

Where to read it

The paper is at doi.org/10.1109/tassp.1975.1162648 (IEEE TASSP, 1975; bibliographic details verified). Read it alongside Erman, Hayes-Roth, Lesser & Reddy's 1980 ACM Computing Surveys retrospective "The Hearsay-II Speech-Understanding System," which is the fuller and more readable account; Klatt's 1977 review of the ARPA SUR program for the brutal Harpy comparison; Barbara Hayes-Roth's 1985 "A Blackboard Architecture for Control" for where the scheduling theory went; and Baars's A Cognitive Theory of Consciousness (1988) to see the same diagram drawn on a brain. Then open the source of whichever multi-agent framework you used last week and notice what's missing.