Revisited · 1953 Ripe now Speech / Video ✓ read
Some Experiments on the Recognition of Speech, with One and with Two Ears
original ↗· The Journal of the Acoustical Society of America, 1953 ·doi 10.1121/1.1907229·3,267 citations ·verified in OpenAlex/Crossref ·7 min read
TL;DR — In 1953, Colin Cherry asked how a listener plucks one voice out of a babble of others, named it the "cocktail party problem," and — unusually for a psychoacoustics paper — framed it as a statistical filtering problem a machine might someday solve. Seventy years on, deep networks separate clean two-speaker mixtures almost perfectly, yet no device gives you Cherry's ability in a real room: low-latency, steerable, ego-centric selective hearing. The paper is ripe because the missing pieces — on-device separation under 10 ms and a control signal telling the device whom you want — both became plausible in the last few years.
The idea as Cherry had it
Cherry, an information theorist by temperament, took Shannon's framing and pointed it at human hearing. His question: what statistics of a speech signal let a listener "recognize what one person is saying when others are speaking at the same time"?
He ran two families of experiments. In the first, two messages from the same speaker are mixed and played to both ears — no spatial cue at all. Subjects, asked to transcribe one message, could do it, slowly and phrase by phrase, exploiting only continuity of meaning, syntax, and voice dynamics. When Cherry stitched the messages from clichés so that transitional probabilities gave no help, separation broke down. That's a beautifully direct demonstration that the "filter" is partly a language model.
In the second family — the famous dichotic shadowing experiments — a different message goes to each ear, and the subject repeats ("shadows") one of them aloud. The rejected ear turns out to be a nearly closed channel: listeners noticed when the unattended voice changed from male to female, or became a pure tone, but did not notice when it switched to German or was played backwards, and retained essentially nothing of its content. Attention, Cherry showed, is a filter with a crude front-end pass (coarse acoustic features get through) and a locked gate on everything semantic.
He also played the same message to both ears with a time offset and found listeners fuse or notice the repetition depending on the delay — an early probe of the temporal window of binaural integration.
The computational reading, which Cherry states explicitly, is the important part: the brain solves a source-separation-plus-recognition problem using (1) binaural/spatial cues, (2) speaker voice characteristics, (3) continuity and predictability of language, and (4) top-down selection of a target. That is, almost line for line, the spec sheet of a modern target-speaker extraction system.
Why it could not work then
Cherry could pose the machine version of the problem but not attempt it, and the gap was total rather than incremental.
No digital signal processing. The FFT algorithm was published in 1965; digital filtering as a discipline dates from the late 1960s. In 1953 "processing audio" meant analog filter banks, tape splicing (which Cherry used ingeniously), and the sound spectrograph. There was no way to represent a waveform as data and operate on it adaptively.
No compute. Machines of the era executed roughly 10^3–10^4 operations per second with a few kilobytes of memory. Real-time separation of even telephone-band audio (8 kHz sampling) with a modern small model needs on the order of 10^8–10^9 multiply-accumulates per second — roughly ten orders of magnitude beyond what existed. Bell Labs' "Audrey" (1952), the state of the art in recognition, was an analog circuit that recognized ten digits from one cooperative speaker.
No estimation framework. Wiener filtering existed for stationary signals in noise, but separating two nonstationary, spectrally overlapping signals of the same class is a different beast. Blind source separation (ICA) arrived in the 1990s and, notably, still fails Cherry's hardest case — one microphone, or fewer microphones than sources — because it needs spatial diversity. Cherry's Experiment A, separation with zero spatial cues, requires strong learned priors over speech and language. Nothing before deep learning had them.
What changed
Three things, in sequence.
Separation got solved on clean benchmarks. Deep clustering (Hershey et al., 2016) and permutation-invariant training (Yu et al., 2017) cracked the label-permutation problem — which output slot is which speaker. TasNet-style time-domain models (Luo & Mesgarani, 2018–19) then showed you don't need long STFT windows: learned filterbanks with ~2 ms frames enable causal, low-latency separation. On WSJ0-2mix, SI-SDR improvements went from near-zero to roughly 20 dB — for anechoic, full-overlap, offline mixtures. Real rooms, moving talkers, and streaming constraints remain much harder; on realistic reverberant data, gains drop sharply.
Selection became conditionable. Instead of separating everyone and picking afterward, target-speaker extraction (SpeakerBeam, VoiceFilter, ~2018–19) conditions the network on an enrollment embedding of the desired voice — Cherry's "voice characteristics" cue made differentiable. Direction-of-arrival conditioning does the same for his spatial cue.
The brain's attention signal became readable. O'Sullivan, Mesgarani and colleagues (2014–2015) showed that scalp EEG reveals which of two talkers a listener attends: the cortical response tracks the attended speech envelope far better than the ignored one. Decoding accuracy is roughly 80–90% with multi-second windows, degrading as windows shrink. Ear-EEG (electrodes in or around the ear canal) makes this wearable, at some cost in signal quality. This closes Cherry's loop: the top-down selection he could only infer behaviorally is now a measurable control signal.
Meanwhile compute moved onto the ear. Modern hearing aids and earbuds carry NPUs running tens to hundreds of millions of MACs per second within a few milliwatts, and the hearing-aid literature gives a hard latency budget: much beyond ~10 ms of processing delay, the mix of processed sound and bone-conducted own-voice becomes disturbing.
A serious 2026 revival
The experiment Cherry could not run is now a buildable system: an attention-steerable neural hearable. Concretely:
- Front end: 2–4 microphones per ear; a causal time-domain separation/extraction network (TasNet-lineage or a streaming state-space model) with total algorithmic-plus-hardware latency under 10 ms, quantized to run in a few milliwatts.
- Selection channel — the interesting part. Fuse Cherry's cue list as conditioning inputs: (a) enrollment embeddings for known voices, (b) head pose / eye gaze from IMU and earbud-adjacent sensors as a proxy for spatial attention, (c) ear-EEG attention decoding as the slow arbiter. Gaze gives fast but ambiguous steering; EEG gives unambiguous but slow (seconds) confirmation; a small policy network reconciles them.
- What to reuse from the 1953 paper: the evaluation philosophy. Cherry measured behavior in continuous, meaningful speech, not tones and clicks. The revival should be judged by speech intelligibility and effort of a human wearing the device in a real multi-talker room with movement and attention switches — not by SI-SDR on synthetic mixtures. His attention-switching findings also set the target: the device must re-lock onto a new talker in well under a second, because humans do.
- What to replace: everything mechanical. Cherry's tape recorders and clichés become on-device models; his "transitional probabilities" prior is now literally a language model, which suggests the boldest variant — coupling the separator to a streaming speech LM so that linguistic continuity (his Experiment A cue) helps disambiguate acoustically confusable moments.
The open scientific question the system would answer: which combination of cues, at which timescales, suffices to match human selective listening — and does the human strategy (coarse acoustic gating early, semantic gating late) fall out of the learned policy?
Already tried, and what's still open
Much of this is in motion. Denmark's hearing-aid industry and academic groups (Tanveer, Zaar, and others; the Clarity Challenges in the UK) have pushed low-latency deep noise reduction into shipping products — but mostly noise suppression, not multi-talker selection. "Neuro-steered" hearing prototypes combining EEG attention decoding with separation have been demonstrated in labs (Columbia/Mesgarani, KU Leuven, DTU) since roughly 2017, but with offline or high-latency processing and seconds-long decoding windows. Meta and academic AR groups have shown gaze- and "look-once" steered target-speech hearing on glasses-style hardware; the University of Washington's "Target Speech Hearing" earbud work (2024) let a user enroll a speaker by looking at them for a few seconds. Cherry's psychology, meanwhile, was vindicated and refined for decades — Broadbent's 1958 filter model, Treisman's attenuation theory, Bregman's Auditory Scene Analysis (1990) — and modern auditory neuroscience (Mesgarani & Chang's 2012 finding that attended speech dominates cortical representations) confirmed his gating story at the neural level.
Still open: sub-second EEG attention decoding; robustness to reverberation, overlap, and talker motion at hearing-aid latency; graceful handling of the switch (Cherry's subjects could reorient instantly; today's systems cannot); and an honest human-in-the-loop benchmark. Nobody has yet demonstrated a wearable that beats an unaided normal-hearing human at a real cocktail party. That is the exact bar Cherry set, and it is finally the right decade to clear it.
Where to read it
The paper is at doi.org/10.1121/1.1907229 (JASA, 1953; bibliographic details verified). It's short and a pleasure — the cliché-splicing experiment alone is worth it. Read alongside: O'Sullivan et al., Cerebral Cortex 2015 (EEG attention decoding); Luo & Mesgarani's Conv-TasNet (2019) for the low-latency separation lineage; Bregman's Auditory Scene Analysis for the bridge between Cherry's psychology and computational modeling; and the UW Target Speech Hearing paper (CHI 2024) for the closest thing yet to Cherry's problem solved in hardware.