Visual routines

TL;DR. In the mid-1980s Shimon Ullman argued that seeing spatial relations—inside/outside, connectedness, counting—cannot be done by parallel feedforward vision alone: it requires serially executed "visual routines" built from elemental operations like shifting attention, indexing, marking, and curve tracing. Forty years later, frontier VLMs fail on precisely the tasks Ullman used as his examples, and we finally have the machinery he lacked: learned controllers, callable vision tools, and program induction. This is one of the cleanest cases of an old framework diagnosing a modern failure mode in advance.

The idea as Ullman had it

Ullman's picture of vision has two stages. First, early vision: a battery of parallel, bottom-up processes that run everywhere in the image at once and produce "base representations"—edge maps, local orientation, depth, motion. This was the Marr program, and by 1984 it was the respectable part of computational vision.

The second stage was his contribution. Some visual properties are not local features you can read off any map. Is point x inside contour C? Are these two blobs part of one connected squiggle? How many objects are there? Ullman argued these are computed by visual routines: short sequential programs, assembled per task, executed over the base representations. The elemental operations he proposed:

• Shift of the processing focus (covert attention movement),
• Indexing: jumping the focus to an odd-man-out location that pops out,
• Marking: tagging a location so you can return to it or avoid recounting it,
• Bounded activation ("coloring"): spreading activation over a region until it hits a boundary—region filling,
• Boundary tracing: following a curve from a point.

Inside/outside becomes: color outward from x; if activation reaches the frame, x is outside. Counting becomes: index, mark, index the next unmarked item, repeat. Connectivity becomes tracing or coloring. The output of a routine is written into an "incremental representation" that later routines can reuse.

The theoretical backbone was already twenty years old when Ullman wrote: Minsky and Papert's Perceptrons (1969) proved that connectedness cannot be computed by any parallel predicate of bounded order—no fixed-depth, local-receptive-field parallel machine can do it, regardless of size. Ullman took the theorem seriously as a statement about biology and engineering: relations of unbounded complexity demand serial computation whose runtime grows with the problem. Psychophysics agreed—Jolicoeur, Ullman and Mackay showed that judging whether two dots lie on the same curve takes time increasing with distance along the curve, not straight-line distance. Something inside your head is literally tracing.

Why it could not work then

Every box in that diagram was out of reach.

Compute. The reference machine was a VAX-11/780 at roughly 1 MIPS with a few megabytes of RAM; a Lisp machine was not dramatically better. Running a Canny-class edge detector on a 512×512 image took on the order of minutes. Executing a routine means running early vision plus dozens or hundreds of serial focus operations, each touching the image. A single inside/outside query was a coffee break; a learning loop over millions of queries was fantasy. Today one GPU delivers roughly 10^{14}–10^{15} FLOP/s—about nine to ten orders of magnitude more arithmetic than the machines Ullman's contemporaries used.

No controller, and no way to learn one. The routine processor is a sequential decision-maker: observe partial state, pick the next operation, repeat. In 1984 there was no practical reinforcement learning (TD-learning was just emerging, Q-learning was 1989, and none of it scaled to pixels until 2013), no backprop-through-attention, no differentiable relaxations of discrete ops. Routines would have had to be hand-coded per task—which is exactly what happened in the few implementations attempted (Chapman's instruction-following agent Sonja in the late 80s hand-built routines for a video game). Ullman himself flagged the "assembly problem"—how routines are composed and selected—as the open question, and it stayed open.

Weak primitives. Even the elemental operations were unsolved. Robust region filling requires closed boundaries, which 1984 edge detectors did not reliably produce. Curve tracing through gaps, junctions and clutter is still nontrivial. "Indexing" presupposes a saliency mechanism; Koch and Ullman's saliency-map proposal is from 1985, and the first working implementation (Itti–Koch) is 1998.

So the paper functioned as a framework and a psychophysics research program, not an engineering blueprint. Its Annual Meeting of the OSA appearance (the DOI above, 1986) is a talk abstract; the full treatment is the 1984 Cognition article and the earlier MIT AI Memo.

What changed

Three things, and they interlock.

First, the failure Ullman predicted showed up, at scale. Modern VLMs—models that ace chart QA and describe complex scenes—fail embarrassingly on toy tasks: counting nested squares, deciding whether two lines intersect, following a path in a subway map, judging whether circles overlap. The "Vision Language Models Are Blind" line of work (2024) catalogued exactly this, and the tasks read like Ullman's figure captions. The diagnosis fits: a transformer over a fixed grid of patch tokens is a bounded-depth, mostly-parallel predicate. Minsky–Papert applies in spirit: no fixed-depth parallel machine computes connectedness for arbitrary inputs. VLMs interpolate relational judgments from texture-level statistics and break when the relation requires unbounded serial work.

Second, the primitives now exist as commodity tools. Segment Anything is Ullman's "coloring" operation, industrialized. Crop-and-zoom is "shift." Saliency and open-vocabulary detection give "indexing." Drawing onto a scratchpad image is "marking"—literally. Curve tracing is the weakest link but tractable.

Third, the assembly problem has plausible solutions. LLM-driven visual programming (VisProg, ViperGPT, 2023) already does the crude version: a language model writes a short program that calls vision modules, and the program is a visual routine, assembled on the fly from a task description. Chain-of-thought over images—models that crop, zoom, annotate, and re-inspect during reasoning, as in "thinking with images" systems—is routine execution inside the forward loop. And RL on verifiable outcomes gives a way to learn routines rather than prompt for them.

Neuroscience, meanwhile, quietly vindicated the psychology. Roelfsema and colleagues showed that when a monkey traces a curve, attentional enhancement spreads along the curve's V1 representation over tens of milliseconds—bounded activation, implemented in cortex. Pylyshyn's FINST visual indexes and Kahneman–Treisman object files are the marking/indexing operations under other names. The claim that relational vision is serial, attention-mediated, and runtime-scaling is now mainstream in visual neuroscience even though the engineering side forgot it.

What a serious 2026 revival looks like

Keep from the paper: the decomposition into base representations, a small closed set of elemental operations, incremental representations as working memory, and runtime that scales with relational complexity. Replace: hand-coded assembly with a learned controller.

Concretely: a multimodal model with a visual-routine interpreter as its tool environment. The action space is Ullman's ops, made executable—shift(x, y, scale) returns a re-encoded crop; index(property) returns pop-out candidates; mark(x, y, tag) and color(seed) and trace(seed, direction) write onto a persistent raster canvas that is re-encoded into the context each step. The canvas is the incremental representation: it lets the model externalize state instead of holding it in attention, which is where monolithic VLMs die. Marking as literal drawing matters—counting fails in VLMs largely because nothing prevents double-counting; a mark does.

Training: RL with verifiable rewards on procedurally generated relational tasks (connectivity, insideness, counting, maze-solving, same-curve judgments) where difficulty is a dial—curve length, nesting depth, object count. The generalization test is the crucial one: train on complexity \le k, test at 2k and 4k. A genuine routine executor should degrade gracefully with more steps (inference-time compute scaling with problem size); a pattern-matcher falls off a cliff. Then add routine caching: distill successful op-sequences into named subroutines and grow a library, DreamCoder-style—this is the learned answer to the assembly problem, and it directly addresses Ullman's distinction between universal routines (always available) and acquired ones.

There is also a stronger, more interesting hypothesis to test: whether routines can be pushed inside the network—recurrent latent computation with an attention pointer and a differentiable canvas—rather than staged as tool calls. If serial relational competence can emerge in-weights given the right training pressure, that says something about what pretraining alone cannot provide.

Already tried? Vindicated? Open?

Partially tried, largely vindicated, not yet unified. The recurrent attention models of 2014 (Mnih et al.'s glimpse networks) were routines without the vocabulary. ViperGPT/VisProg are routines with an LLM assembler but frozen, non-learning execution. "Visual sketchpad"-style systems (2024) rediscovered marking. Ullman's own group has published on atomic visual routines and program-like scene understanding. The neuroscience side is solid: serial, attention-mediated computation of relations in cortex is well-evidenced.

What is open: (1) a learned, general controller that beats monolithic VLMs on relational benchmarks and transfers to natural images, not just synthetic diagrams; (2) whether routine competence internalizes with scale plus the right RL, or genuinely requires an external interpreter; (3) the complete op vocabulary—Ullman guessed five operations from armchair analysis, and nobody has established the real basis set. A side benefit worth stating: routines are legible. A model that answers "connected?" by exhibiting the trace it performed is interpretable in a way a softmax over patch tokens never will be.

Where to read it

The linked DOI (10.1364/OAM.1986.ThC4) is the 1986 OSA annual-meeting talk abstract—bibliographically verified, but it is a one-page summary. Read the full paper: S. Ullman, "Visual routines," Cognition 18 (1984) 97–159 (also MIT AI Memo 723, freely available from MIT). Read alongside it: Minsky & Papert's Perceptrons (the connectedness theorem that motivates everything); Roelfsema's work on curve tracing and incremental grouping in visual cortex (e.g., his 2005–2006 reviews); Pylyshyn on visual indexes; and on the modern side, "Vision Language Models Are Blind" (Rahmanzadehgervi et al., 2024) next to ViperGPT (Surís et al., 2023)—the failure and the fix, each half of which Ullman wrote down in 1984.