Learning by watching: extracting reusable task knowledge from visual observation of human performance

TL;DR: In 1994, Kuniyoshi, Inaba, and Inoue built the complete "learning from human video" pipeline that every VLA lab is now betting on: a robot watches a human do an assembly task once, compiles the video into a symbolic plan of grasps, moves, and placements with dependency structure, and re-executes it in a workspace where nothing is in the same place. It worked — for blocks — and then stalled for 25 years on perception alone. Perception is now solved enough that the paper's other idea, the structured action-graph intermediate, is the part worth stealing, and it's conspicuously absent from today's end-to-end approaches.

The idea as they had it

The framing in the paper is already the modern one: robot programming is the bottleneck, humans demonstrate tasks effortlessly, so let the robot watch. But Kuniyoshi and colleagues made a specific, sharp claim about what should be extracted from the demonstration — and it is not the trajectory.

Their system, built at the University of Tokyo on Inoue's real-time vision hardware, watched a human hand assemble block structures on a table via stereo cameras. It ran three stages:

1. Seeing. Track the hand and objects continuously, using an active attention mechanism that switches between watching the hand's motion and watching the objects it approaches. Segment the continuous stream into discrete events at moments of qualitative change — hand contacts object, object detaches from support, object comes to rest on another.

2. Understanding. Map segments to a vocabulary of symbolic operations (roughly: reach, grasp, transfer, place, release) organized hierarchically, and — crucially — extract the dependency structure among them: this placement requires that block to already be there; this grasp is only valid after that clearance.

3. Doing. At execution time, re-instantiate the plan against the actual workspace: objects in new positions, different initial layout. Because the plan is stated in terms of object relations rather than coordinates, the robot re-derives motions from its own perception and executes the task.

The elegance is in what the representation throws away. If the human fumbles, repositions a block twice, or takes a detour, the dependency analysis prunes it: an action that produces no state change the final structure depends on simply doesn't appear in the plan. The demonstration is treated as evidence about task structure, not as a motion to be replayed. My recollection is that the paper explicitly demonstrates this redundancy elimination, though I'd want the PDF in hand to quote the exact examples.

To feel how far ahead of its time this was: 1994's state of the art in imitation was teach pendants and kinesthetic playback, and the learning-from-demonstration literature that followed (behavioral cloning of trajectories, DMPs in the early 2000s) mostly went the other way — copying motion, exactly what this paper argued against.

Why it could not work then

Every stage of the pipeline was throttled by perception and compute, and the authors knew it.

Vision. There were no learned object detectors, no hand-pose estimators, no action recognition. Tracking meant edge-based and correlation-based methods on engineered scenes: matte polyhedral blocks, controlled lighting, known geometry. The "recognize a grasp" step worked because a grasp in blocks-world is a hand blob converging on a block blob followed by joint motion. Recognizing a grasp of a spatula in a cluttered kitchen was science fiction. ImageNet-scale recognition arrives in 2012; usable in-the-wild 3D hand-mesh recovery only really lands around 2019–2023 (FrankMocap, HaMeR era).

Compute. The workstations of the day delivered roughly tens to a couple hundred MIPS; the group's real-time vision relied on specialized multi-window tracking hardware to hit frame rate at all. A modern GPU delivers on the order of 10^{14}–10^{15} FLOP/s — call it roughly 7–9 orders of magnitude more arithmetic than a 1994 workstation — and that's before clusters.

Data. The deeper issue: the perception their pipeline needed is exactly the kind of thing that can't be hand-engineered and must be learned, and there was nothing to learn from. No labeled video corpora, no pretraining, no notion that 3,600+ hours of egocentric everyday video (Ego4D, 2021) would one day exist as a public dataset, let alone the effectively unbounded supply of humans doing manipulation on the internet.

So the pipeline was demonstrated end-to-end on toy blocks and could not leave that sandbox. It's a clean case of an architecture stalling on one component: the symbolic machinery in stages 2–3 was fine; stage 1 was 25 years early.

What changed

Three things, and they're exactly the three the paper needed.

General perception exists. VLMs can describe what a human is doing in arbitrary video; open-vocabulary detectors and segmenters (the SAM/DINO lineage) localize any object; hand–object reconstruction recovers 3D hand pose, contact, and grasped-object motion from ordinary RGB. Stage 1 of the pipeline — the part that confined the system to blocks — is now roughly a solved commodity for everyday scenes.

The data thesis inverted. In 1994, human demonstration video was something you had to stage. Today, internet-scale human video is the largest untapped corpus for robot learning, and "learning from human video" is arguably the central bet of the field — precisely because robot teleoperation data is scarce and expensive while YouTube is not.

The consumers exist. In 1994 there was nothing downstream to feed. Now there are VLA policies (RT-2, OpenVLA, π0 and successors) hungry for exactly the kind of supervision human video could provide — if we knew what representation to extract from it. Which is where the paper's second idea comes back.

What a 2026 revival looks like

The live question the paper speaks to: when transferring from human video to robot execution, what should the intermediate be? Current answers are mostly unstructured — latent video features (R3M-style), predicted future frames or subgoal images (Gen2Act-style), point tracks and hand trajectories (Track2Act, VideoDex, WHIRL), or nothing at all (end-to-end conditioning). Kuniyoshi's answer was: a symbolic action graph with dependencies.

A concrete revival experiment:

1. Extraction at scale. Run a VLM + hand–object reconstruction pipeline over a large human-video corpus (Ego4D, Epic-Kitchens, HowTo100M-style instructional video) to auto-label each clip with a Kuniyoshi graph: nodes are (grasp, move, place, release, plus contact-rich verbs the 1994 vocabulary lacked — pour, wipe, insert, twist), edges are precondition dependencies inferred from object-state changes. This annotation was impossible in 1994 and is nearly free now.

2. Pretraining with structure. Train a manipulation policy that conditions on the graph — either hierarchically (graph planner on top, learned skill policies below) or by injecting the graph as tokens into a VLA.

3. The test. One-shot generalization, exactly the paper's demo scaled up: show one human video of a multi-step task, then execute in a rearranged scene with distractors and swapped object instances. Compare against raw behavior cloning / video-conditioned policies on (a) success, (b) robustness to rearrangement, (c) whether demonstrator mistakes get copied. The paper's redundancy-pruning result predicts the graph-conditioned policy should be markedly better on (b) and (c) — a falsifiable, publishable claim.

Reuse from the paper: the event-segmentation-by-qualitative-change principle (segment where object relations change, not where motion changes — this is a genuinely good inductive bias that most learned segmenters ignore); the dependency analysis; the plan-then-reground execution scheme. Replace: all of the hand-engineered vision; the closed symbolic vocabulary (make node types open-vocabulary, VLM-grounded); the purely symbolic bottom layer (contact-rich skills like insertion need learned visuomotor policies, not geometric replanning — this is where a pure 1994-style system would still fail today).

Has it been tried? What's still open?

Partially. The paper is a founding citation of the whole learning-from-demonstration lineage (~487 citations per OpenAlex, which if anything understates its influence). Direct descendants of the structured idea exist: Neural Task Programming and Neural Task Graphs (2018–2019) learned graph-structured task representations from demonstration video; the TAMP community lives on symbolic operators over learned perception; recent VLM-planner-over-skills systems (SayCan and its many successors) are Kuniyoshi's stage 2–3 with a language model in place of the parser.

But the dominant bets — end-to-end VLAs, video-prediction subgoals, trajectory transfer — mostly skip explicit dependency structure, and the head-to-head question has not been cleanly answered: does the graph intermediate beat unstructured conditioning on one-shot, rearranged-scene generalization at scale? Individual small-scale results suggest structure helps sample efficiency and mistake-pruning, but no one has run the internet-scale version. The counterargument is real too: hard symbolic bottlenecks discard information (forces, timing, style) that end-to-end methods preserve, so the right answer may be soft — graphs as an auxiliary prediction target or attention scaffold rather than a hard bottleneck. That design space is open.

There's also a sociological point worth naming: the field abandoned this architecture when perception failed, then rebuilt learning-from-human-video from scratch in a perception-first, structure-last order. Whether the structure gets re-added is one of the more interesting open questions in robot learning right now.

Where to read it

The paper: Kuniyoshi, Inaba, Inoue, "Learning by watching: extracting reusable task knowledge from visual observation of human performance," IEEE Trans. Robotics and Automation, 1994 — doi.org/10.1109/70.338535 (bibliographic details verified; my account of the paper's internals is from knowledge of the literature, so verify specifics like the exact action vocabulary against the PDF).

Read alongside: Ikeuchi & Suehiro's "Assembly Plan from Observation" (early 1990s, the closest sibling, more geometric); Bahl et al., WHIRL (2022) and Bharadhwaj et al., Gen2Act (2024) for the modern unstructured bets; Huang et al., Neural Task Graphs (2019) for the structured lineage; and the Ego4D paper (2021) for the corpus that makes the revival possible. The contrast between the 1994 paper and Gen2Act, read back-to-back, is the open research question.