RinDig/cost-of-remembering

TL;DR: A controlled experiment answering a question a lot of people have been hand-waving about: does the "just give the agent a folder of Markdown files" memory pattern (CLAUDE.md, memory banks, Obsidian-style vaults) actually work, and what does it cost relative to stuffing the whole history into a long context? On LongMemEval, accuracy is statistically indistinguishable from full long-context, while the filesystem approach reads 97% fewer tokens and costs 95% less per question — with no embeddings and no vector DB. The whole thing is released: harness, every run config, per-question traces, and 129 memories the agents actually built.

What it is

The setup is deliberately minimal. Give an agent an empty directory plus a set of filing conventions (the authors' "ICM" spec, arXiv:2603.16021): a CLAUDE.md at the root that says where things live, one folder per kind of thing, each with its own CONTEXT.md describing itself. The agent ingests conversations one at a time and writes notes into this structure. At question time, a fresh agent walks the folders — plain file reads, no retrieval model — and answers. The baseline is the standard approach: put the entire conversation history in the context window every time.

The pitch is that long-context memory has a pathological cost curve: the longer an assistant has known you, the more every single question costs, because you re-read the whole relationship each time. A filesystem amortizes that into a one-time build cost plus cheap targeted reads.

Filesystem memory Session 1 Session 2…n file CLAUDE.md folders + CONTEXT.md Answer Build once; ~3% of tokens per question Same accuracy on LongMemEval (statistically indistinguishable)

What's actually released

Unusually complete: the harness (run_cost.py drives every arm in the paper), raw per-question traces and per-session token records, the memories themselves under results/*/brains/, and LaTeX source whose tables and figures are regenerated from results/ on every build — nothing hand-typed. There's also a RUNBOOK.md mapping each paper number to the exact command that produced it, and offline/mocked smoke tests that exercise the real code paths for free.

One nice artifact worth two minutes of browsing before running anything: the brains/ directories. Compare cost_guided_sonnet5_n40/brains/ (conventions) against cost_autonomous_sonnet5_n40/brains/ (same model, same tools, no conventions) — the paper's claimed effect of the conventions is, per the README, visible by eye.

How much to trust it

The README is refreshingly explicit about its own weaknesses, which raises confidence in the parts it does claim:

• The cost numbers are solid; they moved by under 4% as the sample grew from 5 to 39 questions.
• The accuracy numbers are fragile. The accuracy ordering between arms reversed twice across sample sizes, and the authors report all three sizes for that reason. "Statistically indistinguishable" here means the study lacks power to separate them, not that equivalence is established.
• Scope is narrow: one benchmark (LongMemEval), one user, one writer, one reader, synthetic English chat, a single ingestion pass. Multi-user sharing, human auditability, and months-long drift — arguably the properties this pattern exists for — are explicitly untested.

Practical notes: Python ≥3.10, MIT license, needs API keys (Anthropic/OpenAI-style; most single-arm runs need one). LongMemEval is not vendored — you clone it and fetch the Sept-2025 cleaned dataset from HF separately. Full reproduction of all four arms at n=40 is ~$330, build-dominated; a 4-question single-arm run is a few cents.

Why it matters

This is the first serious number attached to a pattern the community has been using on vibes. It's also indirect evidence in the "do agents need memory infrastructure?" debate: the answer here is that a directory tree plus conventions beats both raw long context (on cost) and unstructured agent note-taking (visibly, in the artifacts), without any retrieval stack. Whether it survives denser cross-session dependencies is the open question — but the harness to test that is now public.

Try it

git clone https://github.com/RinDig/cost-of-remembering.git
cd cost-of-remembering
git clone https://github.com/xiaowu0162/LongMemEval.git   # + fetch cleaned data → LongMemEval/data/
python -m venv .venv && . .venv/bin/activate
pip install -r lme-icm/requirements.txt
cp lme-icm/.env.example lme-icm/.env                       # add keys

cd lme-icm
python smoke_offline.py && python smoke_mock.py            # free
python run_cost.py --arm guided --model sonnet5 --n 4 --keep-brains  # a few cents

Then open the brains/ directory it leaves behind and see what the agent filed.