Agora: Collective and Permissionless Internet-Scale Pretraining of Large Language Models

TL;DR: Pluralis trained an 8.6B-parameter LLM on 500B tokens of FineWeb-Edu over the public internet, using 330 churning contributor nodes — mostly consumer GPUs like RTX 4090s on residential links — with no participant ever holding the full model weights. The run sustained ~170k tokens/s at 63% of the efficiency of a centralized H100 cluster, and the loss curve tracked a centralized reference. The two enabling ideas are worth understanding on their own: a reparameterization that confines a transformer's residual stream to a low-rank subspace (cutting pipeline-parallel traffic ~100×), and the observation that weights — unlike gradients — tolerate stale, sparse, asynchronous averaging, which removes the per-step synchronization barrier entirely. If this holds up, it changes who can pretrain large models.

Why the internet is the wrong place to train a model

Start with the numbers, because they make the problem visceral. Datacenter training assumes NVLink inside a node (hundreds of GB/s) and InfiniBand between nodes (tens of GB/s). A residential connection gives you maybe 200 Mbit/s — four to five orders of magnitude less — with 20–80 ms latencies and no reliability guarantees.

Now count what standard parallelism wants to move. Pipeline parallelism (splitting the model's layers into sequential stages on different machines) must ship activations across every stage boundary: for this 8.6B model with hidden dimension d = 5120 and 4096-token sequences, that's a 4096 \times 5120 FP32 tensor, about 671 Mbit per microbatch. Several seconds of transfer on a home link, per microbatch, per boundary. Data parallelism is worse: keeping replicas synchronized means exchanging gradients the size of the weights every step. One Llama-style transformer block at d = 5120 is ~310M parameters, roughly 10 Gbit in FP32 — around a minute of transfer for a synchronization that a datacenter completes in milliseconds, and that must finish before the optimizer step can proceed.

And that's the no-failure case. Real contributor nodes join and leave constantly (this run saw 669 joins and 607 departures). Standard frameworks fix their communicator ranks at initialization; one departing rank stalls the collective, times out, and kills the job. Restart-from-checkpoint recovery is useless when nodes leave every few minutes.

So there are three independent walls: pipeline traffic is too big, data-parallel synchronization is too frequent and blocking, and the participant set is not stable enough for any synchronous program to run at all. Agora's contribution is a solution to each, composed into a working system.

Insight one: make the residual stream live in a small subspace

The pipeline-parallel fix is the most interesting piece of ML in the paper. A transformer's hidden state is a residual stream: each layer adds a contribution,

where {\bm{W}}_{p_1}^i, {\bm{W}}_{p_2}^i are the attention and MLP output projections of layer i. Here's the observation (from the group's earlier "subspace networks" work): if the rows of every output projection lie in a common k-dimensional subspace with orthonormal basis {\bm{U}}_k \in \mathbb{R}^{d \times k}, then every layer's contribution to the residual stream also lies in that subspace. The parts of the hidden state that live outside the subspace — the positional embeddings and a fixed token-embedding component — are known to both sender and receiver. So a stage boundary only needs to transmit the k coordinates per token:

and the receiver reconstructs losslessly with {\bm{U}}_k^\top. With k \approx 51 against d = 5120, that 671 Mbit activation becomes ~7 Mbit — a 100× compression that isn't compression at all, but a constraint on the model's function class that makes the inter-stage signal intrinsically low-dimensional.

The paper's new twist over the original subspace-network formulation is reparameterization: instead of keeping full-width matrices and projecting them back into the subspace after each optimizer step, they factor each projection as {\bm{W}} = {\bm{Z}}{\bm{U}}_k^\top and train {\bm{Z}} directly (the embedding table is factored the same way). Same function class, but the constraint is now structural: no projection step, ~35% lighter body stages, ~50% lighter head, and — per their 1B ablation (Figure 40) — much better-behaved gradient norms, since the compression lives inside the computational graph. Convergence at 1B is identical to the projection-based variant.

Insight two: gradients demand synchrony; weights don't

The data-parallel wall is not really volume — it's frequency. A gradient is only meaningful for the step that produced it; it must be exchanged synchronously, now, before anyone steps. But weights move slowly: each AdamW step nudges them by a small, momentum-smoothed increment. Slow-moving state tolerates exactly what gradients can't — stale, sparse, infrequent, non-blocking exchange.

So Agora abandons gradient synchronization altogether. Each replica of a stage accumulates gradients locally and steps its own optimizer once the stage collectively hits the global batch of 2,048 sequences. Consensus among replicas is maintained by AsyncSPARTA: every 20 steps, replicas average a rotating 5% slice of their parameters in a background all-reduce that overlaps with training (every parameter gets averaged once per 400 steps). Per round that's ~0.5 Gbit per block instead of 10 Gbit per step — and crucially, it never blocks the optimizer.

Two subtleties make this work, and both show up sharply in ablations:

Backward pinning. Workers are stateless — they recompute the forward when a backward request arrives (stage-level activation checkpointing). Under asynchronous averaging, replicas drift apart between rounds, so the recomputation only reproduces the original activations if the backward runs on the same worker that ran the forward. Routing backwards stochastically instead costs ~0.4 nats of loss by step 8k in the 1B ablation (Figure 41) — a compounding gradient bias, not a subtle effect.

Delta-rule application. An averaging round can span up to five optimizer steps. Writing the returned average back in place would erase the local progress made in the meantime, so each replica applies only the consensus shift: {\bm{\theta}}[\mathcal{I}] \mathrel{+}= \bar{{\bm{\theta}}}[\mathcal{I}] - {\bm{\theta}}_\star[\mathcal{I}], where {\bm{\theta}}_\star is the snapshot it contributed when the round began.

The system: no one holds the model

The roles are cleanly split. Workers (GPU nodes) each hold exactly one pipeline stage — Pluralis-8B has seven: head (embeddings + 6 blocks), five body stages (4 blocks each, open to contributors), tail (6 blocks + LM head, the compute-heaviest). Trainers (no GPU, ~745 of them) hold the dataset shards and the full pipeline topology but zero parameters; they route each microbatch through one worker per stage, hold the boundary activations, and unwind the backward along the same path. Workers advertise themselves on a DHT with short-expiry entries; trainers load-balance via a min-heap of speed-weighted virtual runtimes, so a 4090 and an L40S can coexist in one stage, each fed at its own pace. A slow node just gets fewer microbatches; a dead node simply stops being routed to. Fault tolerance is a property of the step, not a recovery layer around it.

The permissionless machinery is the least glamorous but arguably the most operationally novel part: an Authorizer gates admission (24 GB VRAM, 200 Mbit/s, ≤80 ms latency floors, signed-credential peer auth so the central service stays off the training path), joining nodes pass through a two-phase warm-up — 400 steps absorbing averaged weights with zero contribution weight, then 100 steps warming Adam moments on live gradients before counting toward anything — and workers self-terminate after failing 2 of 3 averaging rounds, since a replica that keeps missing consensus drags everyone down.

Did it work?

The run held ~170k tokens/s across 40 days and 60k optimizer steps, through continuous churn (the 60-contributor cap was saturated within two days; demand was 3.3M join requests, 99.98% rejected). Final training loss: 2.41. The loss curve tracks a centralized reference throughout — jaggier early when partial averaging hurts most, with one visible perturbation at step 40k when many contributors exited simultaneously, recovering within a few thousand steps with no intervention.

Where does the missing 37% go? Mostly into the contributor body stages, which were deliberately over-provisioned as churn insurance and ran latency-bound:

An interesting empirical finding: contributor throughput was governed by network latency and GPU class, not bandwidth — throughput was flat above the 200 Mbit/s floor but fell ~40% from the 0–20 ms latency band to the 60–80 ms band (Figure 36). And the 1B ablations give the system a quantified operating envelope: batch-size heterogeneity of 10:1 across workers is tolerated; convergence measurably degrades only when all-reduce participation drops below ~85%, and fully recovers once failing nodes are pruned (Figure 45) — the empirical basis for the self-termination rule.

The scaling analysis (Section 3.4.5) is the strategic claim: per-stage compute grows as \Theta(nL_s d^2) (with L_s blocks per stage) while boundary traffic grows as \Theta(nd/c), so the compute-to-communication ratio improves as \Theta(L_s d c) — doubling the width doubles the traffic but quadruples the compute. The sparse-averaging duty cycle is scale-invariant. If correct, this regime gets easier at 70B than at 8B, which is the opposite of most distributed-training pain.

What to be skeptical about

The convergence comparison is internal, not absolute. The "centralized reference" is a TorchTitan run modified to use the same subspace compression and SPARTA averaging. So the near-matching curves demonstrate that decentralized execution (churn, partial averaging, heterogeneity) costs little given these ML changes — they do not tell you what the rank-51 subspace constraint costs against standard dense pretraining. The model has 8.6B trainable parameters against a "12B dense-equivalent," and no downstream benchmark comparison against a conventionally trained 8B appears in the text. This is the biggest open question: is the subspace constraint nearly free, or is it a real capability tax? The reference was also only run for 27k of the 60k steps, and sits "slightly lower."

The 63% figure has soft edges. The denominator is spec-sheet dense-BF16 peak across all pooled GPUs, compared against a Llama 3.1 8B / H100 / FSDP2 baseline that trains a different (dense, uncompressed) model. Reasonable as a headline, but not a controlled apples-to-apples measurement.

"Permissionless" has a substantial operator backbone. The head and tail stages ran exclusively on Pluralis cloud nodes (8 and 16 respectively), each body stage kept 4 Pluralis anchors, only Pluralis nodes served as all-reduce reducers, the Authorizer is a central singleton, and contributors were capped at 60 concurrent. The 330 contributor nodes are real and the churn tolerance is genuinely demonstrated, but this is operator-anchored decentralization, ~39.7 pooled PFLOP/s total — around three orders of magnitude below a frontier cluster. Whether the DHT coordination (whose latency already rose ~10× when contributors joined, Section 5.9) survives thousands of nodes is untested.

Incentives and adversaries are deferred. The economic story ("collectively owned models") rests on Unextractable Protocol Models cited from separate work; gradient poisoning by admitted-but-malicious contributors is handled only by build verification and clipping, not by any Byzantine-robust aggregation.

Where to spend your time

Read Section 4 first — the reparameterized SSN and AsyncSPARTA (with backward pinning and the delta rule) are the transferable ideas, compactly presented. Then Section 6 for the ablations that actually justify the design (Figures 41 and 45 are the load-bearing plots), and Section 3.4.5 for the scaling argument that determines whether this direction has legs. Section 5 is a rich systems postmortem — the DHT degradation and all-reduce timeout forensics (5.9) are the most honest parts of the paper — but skimmable if you trust the headline numbers.

The fair summary: this is the most serious end-to-end demonstration yet that internet-grade, churning, heterogeneous consumer hardware can sustain a real multi-billion-parameter pretraining run, built on two clean ideas rather than brute-force engineering. What it hasn't yet shown is that the resulting model is as good as one trained the ordinary way — and that's the comparison that will decide whether Protocol Learning is a new paradigm or an elegant niche.