Active perception

TL;DR. In 1988 Ruzena Bajcsy argued that perception is not a signal-processing problem but a control problem: an agent should choose its sensing actions—where to look, how to focus, what to touch—so that interpretation becomes tractable, making the sensing strategy part of the percept itself. The field ignored this for thirty years because it had neither the actuators nor the learning algorithms to close the loop, and standardized on fixed image datasets instead. Now that passive internet pretraining has visibly hit a ceiling for manipulation, and RL can actually train sensing policies, the paper reads like a spec sheet for what embodied VLMs are missing.

The idea as Bajcsy had it

The mainstream 1980s view, inherited from David Marr, treated vision as an inverse problem: given an image, recover the scene. This is famously ill-posed—infinitely many 3D worlds project to the same 2D image—so the field's energy went into regularization: adding smoothness priors and constraints until a unique solution popped out.

Bajcsy's move was to reject the problem statement. The inverse problem is only ill-posed if you insist on solving it from a fixed image. A perceiver that can act gets to choose its data. Depth from a single image requires priors; depth from a deliberate lateral head motion is nearly trivial geometry. Ambiguity is not something to regularize away in software—it is something to resolve by moving.

Her definition is careful and worth restoring, because "active" was already an overloaded word. She did not primarily mean active sensors (sonar, structured light), and she explicitly distinguished her proposal from "active vision" as mere camera motion. Active perception, in her framing, is the intelligent control of the data-acquisition process: a hierarchy of feedback loops, from local ones (control focus, aperture, gain—what she called parameter adaptation within a sensor) up to global ones (select the next viewpoint, choose which surface to press a tactile sensor against, decide whether vision or touch answers the current question). Crucially, the paper treats haptics as a first-class sense: her lab's work on tactile exploration—pressing, tracing, gripping to estimate compliance and shape—was part of the same program. The percept is the outcome of a strategy, not a function of an image.

Active (Bajcsy, 1988) Sensing action: look, zoom, touch Well-posed measurement Update belief; enough for the task? no: act Left: recover everything from fixed data. Right: choose data so recovery is easy.

A modern way to write her thesis: given belief history h, choose the sensing action a^\* = \arg\max_a \, I(S; O_a \mid h), where S is the task-relevant state and O_a the observation that action produces—maximize expected information gain about what the task needs, not about the scene in general. Bajcsy didn't have this notation, but the paper's framing of perception as feedback control with a task-dependent stopping criterion is exactly this.

Why it could not work in 1988

Every piece of the loop was missing.

Compute. A high-end research workstation delivered roughly 1–10 MIPS with a few megabytes of RAM. A single convolution pass over a 512×512 frame took on the order of seconds; anything resembling a modern perception stack was minutes per frame. A control loop that must look, decide, move, look again dies when each "look" costs minutes.

Actuation and sensing. Cameras were monochrome, 512×512-ish, digitized through frame grabbers. Steerable camera heads were one-off lab hardware; the famous binocular heads (Rochester, KTH) that made "active vision" a subfield arrived just after this paper and cost tens of thousands of dollars each. Robust tactile arrays essentially did not exist outside a handful of labs.

Learning—the fatal gap. Active perception is a sequential decision problem under partial observability. In 1988 there was no machinery for this: Q-learning was published in 1989, policy gradients for neural policies came a decade later, and deep RL two decades after that. Every sensing strategy had to be hand-engineered as an explicit control law, per task, per sensor. That does not scale past demos, and it didn't.

So the field took the path that its tools allowed: fixed datasets of passive images, first small (COIL, MNIST-era), then huge (ImageNet). This was the right engineering call at the time—it gave us supervised deep learning—but it quietly redefined "vision" as "processing images someone else took," and the sensing-strategy half of Bajcsy's percept was amputated.

What changed

Three things, and they arrived in the right order.

The learning machinery exists. RL with recurrent policies and world models handles exactly the class of problem Bajcsy posed: partially observed, sequential, with actions valued for information rather than reward. Imitation learning gives a cheaper on-ramp: humans demonstrate where they look and what they touch, not just what they do, and wrist-camera teleoperation data captures that for free.

The hardware is disposable-cheap. A pan-tilt camera is under $100; wrist cameras are standard on every serious manipulation platform; vision-based tactile sensors (GelSight-style, DIGIT) turn touch into images that the same networks can consume. Bajcsy's "haptics as co-equal sense" is finally cheap enough to instrument.

Passive pretraining hit its ceiling in exactly the place she predicted. Internet video teaches a lot about what scenes contain and almost nothing about how to interrogate one. The symptom is everywhere in manipulation: VLA policies that collapse under occlusion, grasping systems that fail on the object behind the object, inspection tasks where the answer is only visible from a viewpoint no photographer would choose. The field's quiet fixes—adding wrist cameras, multi-view setups, "zoom in on the crop" tool calls in reasoning models—are all unacknowledged Bajcsy.

That last point deserves emphasis. When a modern reasoning model crops and re-examines part of an image before answering, or a guided-visual-search system like V* decides where to look next in a high-resolution scene, that is active perception—executed in image space rather than physical space, because image space is where we could give the model actions. The concept was vindicated the moment we let perception systems have actions at all.

What a 2026 revival looks like

The serious version is not "add RL to a camera." It is a study with a control condition, which the embodied-AI literature mostly lacks.

Architecture. Take an embodied VLM/VLA and extend its action space with sensing actions: pan/tilt/dolly a head camera, reposition the wrist camera, zoom, and trigger a tactile probe (press, slide, grip) with a GelSight-style fingertip. The policy is trained in two stages: imitation from teleoperation that records gaze/viewpoint choices, then RL fine-tuning where the reward is task success minus a per-glance cost—the cost term forces the policy to learn when sensing is worth it, which is Bajcsy's stopping criterion. An explicit information-gain bonus, I(S; O_a \mid h) estimated by an ensemble or a learned world model's predictive variance, is the principled version; task-success RL alone may discover it implicitly, and comparing the two is itself a good experiment.

Benchmark. Occlusion-heavy manipulation and inspection, scored against a passive baseline given the same total pixel budget but no control over acquisition: find-and-retrieve in cluttered bins, cable/connector inspection, distinguishing objects that are visually identical but haptically different (ripe vs. unripe, full vs. empty container). The last category is the clean kill shot: no passive visual policy can solve it in principle, so it measures the value of the sensing action directly.

What to reuse from the paper. The hierarchy of loops (local sensor-parameter control nested inside global strategy), the task-dependent stopping rule, and the insistence that touch and vision share one framework. What to replace. All the hand-designed control laws and the 1980s modeling formalism—learned policies and learned belief states do that job now.

Already tried, partially vindicated, still open

Much of this has been tried in fragments. "Animate vision" (Ballard, 1991) and the active-vision heads of the early 1990s proved the geometry arguments, then stalled on learning. Next-best-view planning is standard in 3D reconstruction and active SLAM. Interactive perception—pushing objects to segment them—was surveyed by Bohg et al. (2017). Hard visual attention (Mnih et al., 2014) was an early learned-sensing-policy result. Embodied-AI platforms (Habitat, AI2-THOR) made "where to move to see" a trainable problem. And Bajcsy herself, with Aloimonos and Tsotsos, published "Revisiting Active Perception" (Autonomous Robots, 2018), arguing the agenda was unfinished—correctly, in my view.

What remains open is the integration: no widely-adopted foundation model today treats sensing actions as first-class citizens of its action space, trained end-to-end with manipulation. Viewpoint selection, zooming, and touching live in separate literatures with separate baselines. The open scientific question is whether information-seeking behavior emerges from task-reward RL at scale in embodied models, or must be built in via explicit information-gain objectives—a live debate with real architectural consequences, and one Bajcsy's framing lets you state crisply.

Where to read it

The paper is Bajcsy, "Active Perception," Proceedings of the IEEE, 1988 (doi.org/10.1109/5.5968); bibliographic details verified. Read it alongside Bajcsy, Aloimonos & Tsotsos, "Revisiting Active Perception" (2018) for the authors' own thirty-year retrospective; Bohg et al., "Interactive Perception" (2017) for the manipulation-side descendants; and Mnih et al., "Recurrent Models of Visual Attention" (2014) as the first modern learned sensing policy. If you want the psychology that inspired the haptics half, Lederman & Klatzky's work on exploratory procedures (late 1980s) shows humans already implement Bajcsy's control laws with their hands.