The Ecological Approach to Visual Perception

TL;DR — In 1979 James J. Gibson argued that vision does not exist to reconstruct a 3D model of the world; it exists to serve action, and its natural output is affordances: graspable, traversable, sit-on-able, openable — properties of the environment relative to an acting body, picked up from the structured optic flow generated by the observer's own movement. The field went with Marr instead, partly because reconstruction decomposed into pipelines that 1970s compute could at least attempt, while "pick up affordances from flow" was uncomputable. Robot learning has now quietly re-run the experiment: end-to-end action-conditioned perception routinely beats reconstruct-then-plan, affordance prediction is a standard target, and world models learn flow-like latents. Nobody has yet built the robot foundation model that takes Gibson's thesis fully seriously — affordances as the content of the representation, evaluated only by action generalization. That's the ripe part.

The idea as Gibson had it

Gibson's book (a book, not a paper; the standard citation is Houghton Mifflin, 1979, though I haven't verified edition details) is the capstone of thirty years of "ecological optics." Its claims, stripped to essentials:

1. The stimulus for vision is the ambient optic array, the structured pattern of light converging on a point of observation — not a retinal image. When the observer moves, the array transforms lawfully, and those transformations (optic flow) carry information that a static image does not.

2. Information is in invariants. Under all the transformations produced by movement, some structure stays constant: texture gradients specifying slant, horizon ratios specifying object height relative to eye height, the rate-of-expansion variable later formalized as \tau — roughly, the optical size of an approaching surface divided by its rate of dilation, which specifies time-to-contact without ever estimating distance or speed separately. (The \tau formalization is usually credited to David Lee in the mid-1970s, building on Gibson.)

3. What is perceived is affordances. Not "a horizontal plane at height 0.45 m" but sit-on-able-for-me. Affordances are relational — they depend jointly on environment and body — and Gibson insisted they are directly perceived, not inferred from a prior geometric description.

4. Perception is active. The observer moves in order to generate the flow that carries the information. Perceiving and acting form one loop, not a pipeline.

The radioactive claim was "direct perception": no inference, no internal representation, the nervous system "resonating" to invariants. Cognitive scientists reasonably asked what "resonating" was computationally, got no answer, and mostly moved on.

The world it was written against

The state of the art in 1979 machine vision was the blocks world and its descendants: edge detection, line labeling (Waltz filtering), and the emerging Marr program — image → primal sketch → 2.5-D sketch → object-centered 3D model, each stage a well-posed(ish) inverse problem. Marr explicitly engaged Gibson in Vision (published 1982) and dismissed direct perception as begging the question: detecting invariants is information processing, and hard information processing at that.

Marr was right about that, and it mattered that his framing came with a research program you could actually execute on a VAX. Gibson's framing came with none.

Why it could not work then

Put numbers on the wall Gibson's program hit:

• Compute. A VAX-11/780 (1978) delivered roughly 1 MIPS and well under a MFLOP. Dense optic flow — the raw material of ecological optics — was not even formulated as an algorithm until Horn & Schunck in 1981, and running it on a single 256×256 image pair took minutes to hours depending on hardware. Gibson's theory needed dense flow at video rate on a moving platform. That is roughly 9–10 orders of magnitude away from what a modern GPU does (a current accelerator delivers on the order of 10^{14}–10^{15} FLOP/s; RAFT-class flow runs at real time comfortably).

• Sensors and memory. Framegrabbers of the era captured maybe 512×512 at 8 bits; a single digitized frame (~0.25 MB) was a meaningful fraction of a machine's RAM. Storing seconds of egocentric video to learn invariants from was fantasy.

• No learning machinery. Gibson's "resonance" is, with hindsight, a placeholder for learned feature extraction. But there was no backprop at scale, no self-supervised learning, no way to let statistics of flow shape a representation. Marr's program could be hand-engineered stage by stage; Gibson's could only be learned, and learning didn't exist yet in usable form.

• No embodiment to close the loop. Affordances are defined relative to a body that acts. Robots of 1979 (Shakey's generation) took minutes per perception-action cycle. You cannot validate "vision for action" without action.

So the field's choice was overdetermined: Marr's theory was the one you could make progress on. It's a clean case of compute selecting the paradigm.

What changed

Three converging lines, each partially vindicating Gibson:

End-to-end robot learning. From deep RL on pixels (~2015) through today's vision-language-action models (RT-2, OpenVLA, π0 and kin), the dominant recipe maps observations to actions with no explicit reconstruction stage. When explicit geometry is added, it's as an auxiliary signal or feature, not as the mandatory intermediate Marr envisioned. The empirical pattern — task-conditioned representations beat task-agnostic reconstructions for control — is Gibson's central bet.

Affordances became a standard prediction target. Grasp-point and contact-affordance networks, traversability estimation for legged robots, "where/how would a human interact" models trained on egocentric video (e.g., Bahl et al.'s work on affordances from human videos, 2023). The word "affordance" appears in robotics abstracts constantly, usually with the relational meaning intact.

World models learn flow-like structure. Dreamer-style latent dynamics models and video-prediction world models learn representations organized around how the array transforms under action — which is a fair paraphrase of ecological optics. Self-supervised video models discover motion and depth cues without depth labels, i.e., invariants emerge from the statistics of a moving observer, exactly the learning story Gibson lacked.

Neuroscience has been kinder to Gibson than cognitive science was: looming detectors (locust LGMD, and looming-sensitive circuits in mouse), MST neurons tuned to flow fields, and \tau-like time-to-contact control in diving birds and braking humans are all reasonably well established, though the strong "\tau alone controls action" claim has been disputed.

What a serious 2026 revival looks like

The honest observation is that current robot foundation models are implicitly Gibsonian — no reconstruction bottleneck — but not explicitly so. Their latents are whatever gradient descent found. The untried experiment: make affordances the structured content of the representation and see if that beats both unstructured end-to-end latents and 3D-reconstruction features.

Concretely:

• Backbone: video model pretrained on egocentric human video plus robot interaction data, with self-supervised objectives that exploit ego-motion (flow prediction, forward dynamics) — reusing Gibson's insight that the observer's own movement is the teacher.
• Affordance-structured latent: explicit heads producing spatial fields — graspability (with hand/gripper pose distributions), traversability, articulation (openable/pushable, with axis), support (place-on-able) — conditioned on an embodiment descriptor. This is the part almost nobody does: Gibson's affordances are body-relative, so the same scene should yield different affordance fields for a parallel-jaw gripper, a five-finger hand, and a quadruped. Body-conditioning is the crisp, testable, novel claim.
• Policy: acts from the affordance fields plus a thin residual stream, so you can ablate whether the affordance structure is doing the work.
• Evaluation: only manipulation and navigation generalization — new objects, new embodiments, new scenes. No depth error, no reconstruction PSNR, no segmentation mIoU. Gibson's thesis is precisely that those metrics measure the wrong thing.

What to reuse from the book: affordances as content; invariants over transformations as the learning target; self-motion as supervision; the relational, body-indexed definition. What to discard: the anti-representationalism. "Direct pickup" was Gibson refusing to specify a mechanism; a learned network that maps flow to affordance fields without an intermediate metric reconstruction honors the spirit while abandoning the untenable letter.

Tried, vindicated, still open

Tried in pieces: active/animate vision (Bajcsy, Aloimonos, Ballard, late 1980s–90s) and Brooks' behavior-based robotics were explicit Gibsonian revolts, ahead of their compute. Affordance prediction, VLAs, and world models are the modern descendants, mostly without citing him.

Partially vindicated: reconstruction-free control works, often better. But there is genuine counter-evidence to over-strong readings: 3D-aware features and depth pretraining measurably help many manipulation policies, and mapping-based navigation still beats pure reactive policies on long-horizon tasks. The live question is not "Gibson or Marr" but where the boundary sits: which tasks need persistent geometry (a Marrian residue) and which are best served by affordance fields over flow. Body-conditioned affordance representations, and any demonstration that affordance structure improves cross-embodiment transfer, remain essentially open.

Where to read it and what to read alongside

The Ecological Approach to Visual Perception is a book, widely available in reprint (a Psychology Press "Classic Editions" reissue exists); no link was provided and I haven't verified edition specifics. The affordances chapter and the ecological-optics chapters are the essential core; the polemics against "snapshot vision" are skimmable.

Read alongside: Marr's Vision (1982) for the opposing program and his direct critique of Gibson; Lee's 1976 time-to-contact paper for the one Gibsonian invariant that got fully formalized; Ballard's "Animate Vision" (1991) and Brooks' "Intelligence Without Representation" (1991) as the first computational revival; and any current VLA paper (π0, OpenVLA) plus Bahl et al.'s human-video affordance work, reading them as ecological psychology that forgot its own name.