Über die Entropieverminderung in einem thermodynamischen System bei Eingriffen intelligenter Wesen

TL;DR. In 1929 Leo Szilard reduced Maxwell's demon to its minimal form — a one-molecule gas engine — and argued that an intelligent being can only extract work from thermal fluctuations by paying for the information it acquires, at roughly kT\ln 2 per binary decision. This was the first time anyone priced a bit in joules, nearly two decades before Shannon defined the bit. It matters to you now because that price, about 3\times10^{-21} J at room temperature, sits eight-ish orders of magnitude below what a GPU pays per operation, and because the workloads dominating AI — sampling, stochastic optimization, diffusion — are exactly the workloads a Szilard-style machine that harnesses thermal noise is naturally suited to.

The demon, shrunk to one molecule

Maxwell's 1867 demon watches molecules and sorts fast from slow, building a temperature gradient for free and breaking the second law. For sixty years the standard responses were hand-wavy ("no such being exists"). Szilard's move was to strip the demon of everything except the one thing it irreducibly does: acquire one bit of information and act on it.

His engine: a single molecule in a box at temperature T, in contact with a heat bath. Insert a frictionless partition in the middle. The molecule is now on the left or the right — you don't know which. If you measure which side it's on, you can attach a piston to the empty side, let the one-molecule "gas" expand isothermally back to the full volume, and extract work

where k is Boltzmann's constant. Repeat forever: you're converting heat from a single bath into work, a perpetual motion machine of the second kind — unless something in the cycle pays. Szilard argued the measurement itself must generate at least kT\ln 2 of entropy (k\ln 2 in entropy units), exactly cancelling the gain. Remarkably, he also gestured at what Bennett would make precise fifty years later: the cost is tied to the demon's memory, the physical record of the measurement, which must be reset each cycle.

The conceptual leap, easy to miss from 2026: information here is not a metaphor. It is a physical quantity with a conversion rate to energy, appearing in a physics paper as a term in an entropy budget. Shannon's 1948 theory would give information its mathematics; Szilard had already given it a thermodynamic price and, in his memory register, effectively invented the bit as a physical object.

Why it could only be a thought experiment in 1929

Three walls, each absolute at the time:

No formal information theory. Szilard had no notion of a channel, a code, or entropy as expected log-probability. He computed the entropy of one binary alternative from statistical mechanics and stopped there. Without Shannon there was no way to generalize from "one molecule, one partition" to arbitrary computations, so the result stayed an isolated curiosity in the Maxwell's-demon literature for decades.

No devices anywhere near the scale. kT\ln2 at room temperature is \sim3\times10^{-21} J. The best laboratory instruments of 1929 could resolve energies many orders of magnitude above this; single-molecule manipulation, optical tweezers, and single-electron electrometry were 60–80 years away. You could not build the engine, and you could not even measure whether an existing process was close to or far from the bound.

No computers to make the question urgent. The first electronic computers were still ~15 years off, and when they arrived they were grotesquely far from thermodynamic limits — ENIAC burned roughly 150 kW to do a few thousand operations per second, on the order of 10^{-2}–10^{1} J per operation, twenty-plus orders of magnitude above kT\ln2. Asking "what is the minimum energy of a logical operation" was like asking about the rocket equation before gunpowder.

What changed: the theory got fixed, then the experiments arrived

Landauer (1961) made the principle precise for computing: it is logically irreversible operations — erasing a bit, merging two computational paths into one — that must dissipate at least kT\ln2. Bennett (1973, 1982) then resolved the demon properly: measurement can in principle be done reversibly at zero cost; the unavoidable payment is resetting the demon's memory, i.e., Landauer erasure. Bennett also showed any computation can be embedded in a logically reversible one, so there is no fundamental lower bound on the energy of computation itself — only on information destruction. Szilard's accounting was slightly misplaced (he charged the measurement) but his total was exactly right, and his instinct about the memory reset was prescient.

Then, strikingly, the thought experiment became an experiment:

• Toyabe et al. (2010, Nature Physics): feedback control on a Brownian particle demonstrated information-to-energy conversion — a working demon.
• Bérut et al. (2012, Nature): direct measurement of the Landauer bound, erasing a bit stored in a colloidal particle in a double-well potential and measuring dissipated heat approaching kT\ln2.
• Koski et al. (2014): a Szilard engine implemented with a single electron, extracting kT\ln2 per cycle to good precision.

So the paper is not merely vindicated in theory; the exact apparatus Szilard imagined has been built and behaves as he calculated. Alongside this, stochastic thermodynamics (Jarzynski, Crooks, Sagawa–Ueda's generalized second law \langle W \rangle \ge \Delta F - kT\,I, where I is the mutual information gained by measurement) turned his one-off argument into a full theory of information engines.

Meanwhile the economics inverted. Energy is now the binding constraint on AI — training runs measured in tens of megawatts, inference fleets in gigawatts — and the gap between practice and the Landauer floor is the largest known untapped efficiency margin in computing:

Even at the device level, a modern transistor switching event dissipates roughly 10^4 kT; at the system level (memory movement, clocks, interconnect), a useful FLOP costs closer to 10^8 kT. Almost all of that is spent fighting thermal noise — charging capacitances far above kT so bits stay deterministic. Szilard's engine points at the alternative: let the noise do work for you.

What a serious 2026 revival looks like

The key observation: the dominant AI workloads are not deterministic arithmetic. Sampling from a diffusion model, Gibbs sampling in an energy-based model, MCMC, stochastic optimization, Bayesian posterior inference — all of these want a physical source of properly-shaped randomness. A GPU synthesizes randomness expensively (deterministic PRNG + arithmetic at 10^8 kT per op) and then simulates a stochastic process. A thermodynamic computer instead builds a physical system whose natural fluctuations are the target distribution.

The revival architecture, concretely:

• Substrate: networks of probabilistic bits (p-bits) — stochastic magnetic tunnel junctions, or noisy analog CMOS cells — each fluctuating between 0 and 1 at rates set by a local bias. Camsari, Datta and colleagues showed p-bit networks natively implement Boltzmann-machine dynamics; a 2019 Nature paper demonstrated integer factorization by sampling. Startups (Extropic, Normal Computing) are building thermodynamic sampling ASICs on this premise.
• What to reuse from Szilard: the framing itself — treat the machine as an information engine and do explicit entropy accounting per cycle. Charge every bit erased at kT\ln2; everything else is in principle free. The Sagawa–Ueda inequality gives the design rule: work extracted per feedback cycle is bounded by the mutual information the measurement actually acquires, so sensor precision should be budgeted like energy.
• What to replace: the single molecule with millions of coupled p-bits; the demon with a learned energy function programmed into coupling weights; the piston with readout of equilibrium (or driven nonequilibrium) samples.
• The experiment: benchmark in joules per effective sample — energy divided by effective sample size after autocorrelation — on tasks GPUs actually run: sampling a trained EBM, the discrete inner loop of a diffusion sampler, combinatorial optimization. The honest comparison includes I/O and programming energy, which is where analog accelerators historically die. The claim to beat: 3–5 orders of magnitude better joules/sample than an H100-class GPU on distributions that map naturally to the hardware.

Status: vindicated in physics, open in engineering

Already done: the physics is fully vindicated — Landauer's bound measured, Szilard engines built at the single-electron and single-particle scale, the Maxwell's demon debate essentially closed by Bennett. Reversible/adiabatic computing (Bennett's Brownian machines, Michael Frank's adiabatic CMOS) is the other direct descendant, attacking the same gap from the deterministic side, though it has never escaped niche status because reversibility costs speed and area.

Still open, and genuinely so:

• Does the joules-per-effective-sample advantage survive at system level, with digital I/O, weight programming, and the mismatch between the distributions hardware natively samples and the ones models actually need?
• Can training (not just inference/sampling) be moved onto thermodynamic hardware — e.g., contrastive learning where both phases are physical relaxations?
• How close to kT\ln2 per irreversible bit can a practical machine operate before thermal errors overwhelm error correction, whose own cost is erasure? There is a real, largely uncharted trade-off surface here connecting Szilard directly to modern nonequilibrium thermodynamics.

My uncertainty: the physics claims above are solid; the startup landscape and specific p-bit benchmark numbers are moving targets and should be checked against current literature. No published system yet demonstrates an end-to-end, wall-plug-measured win on an ML workload — that demonstration is the revival.

Where to read it

The original: Szilard, "Über die Entropieverminderung in einem thermodynamischen System bei Eingriffen intelligenter Wesen," Zeitschrift für Physik (1929), doi:10.1007/bf01341281 (bibliography verified). An English translation ("On the decrease of entropy in a thermodynamic system by the intervention of intelligent beings") appears in Leff & Rex's anthology Maxwell's Demon: Entropy, Information, Computing — the single best companion volume, since it also collects Landauer (1961) and Bennett (1982). Read alongside: Bennett's "The Thermodynamics of Computation — a Review" (1982) for the resolution; Parrondo, Horowitz & Sagawa, "Thermodynamics of information" (Nature Physics, 2015) for the modern theory; Bérut et al. (2012) and Koski et al. (2014) for the engine made real; and the recent p-bit and thermodynamic-computing literature (Camsari et al.; Coles et al.'s thermodynamic AI perspective) for the hardware program that Szilard, without knowing it, chartered.