The Cost of Remembering

Everything behind the paper The Cost of Remembering: Filesystem Memory Against Long Context on LongMemEval: the harness, every run configuration, the raw token records, the memories the agents actually built, and the LaTeX source that turns them into tables and figures.

The problem it starts from is that the longer an assistant has known you, the more every question costs. Putting the whole history in the context window means re-reading the entire relationship to answer one thing, at full price, every time. This measures what a folder costs instead.

The short version. Give an agent an empty folder and a set of conventions for filing things in it. Have it read one conversation at a time and write notes. Later, ask it a question and let it walk the folders to answer. Compare that against putting the whole conversation history in the model's context window. On LongMemEval, the two are statistically indistinguishable on accuracy, and the folder reads 97% fewer tokens and costs 95% less per question.

No embedding model. No vector database. Markdown files in directories.

---

Why this repo exists

People have been giving language models a folder of markdown files for a while now and reporting that it works. CLAUDE.md, AGENTS.md, memory banks, Obsidian vaults, Karpathy's LLM wiki. What was missing was a number. This is the number, plus the harness that produced it, so you can disagree with it using data rather than intuition.

The conventions under test are ICM, which is one attempt to write down the layer these approaches have in common.

What is in here

lme-icm/          the harness
  lme/            library: ingestion, tools, agent loops, judging, cost
  run_cost.py     the driver for every run in the paper
  run_eval.py     the earlier read-only evaluation (phase 1)
  smoke_*.py      offline and mocked checks that cost nothing to run
  results/        every run: config, per-question traces, per-session tokens,
                  and the memories themselves under results/*/brains/
paper/            LaTeX source
  scripts/        regenerates every table and figure from results/
  main.pdf        the built paper
docs/RUNBOOK.md   the exact commands that produced each number

Nothing in the paper was typed in by hand. paper/scripts/make_tables.py and make_figures.py read lme-icm/results/ and write paper/tables/ and paper/figures/ on every build. If you re-run an arm, the paper changes.

Look at a memory before you run anything

The most direct way to understand the claim costs nothing:

lme-icm/results/probe_guided/brains/          a memory after 8 conversations
lme-icm/results/cost_guided_sonnet5_n40/brains/   after ~48, forty of them

Open one. There is a CLAUDE.md at the root that says where things live, a folder per kind of thing, each describing itself in its own CONTEXT.md. That is the whole storage layer. Compare it with results/cost_autonomous_sonnet5_n40/brains/, which is the same model with the same tools and no conventions, and the difference the paper measures is visible by eye.

Setup

Python 3.10 or newer.

git clone https://github.com/RinDig/cost-of-remembering.git
cd cost-of-remembering

# 1. the benchmark. Not vendored here; it is a separate MIT-licensed project.
git clone https://github.com/xiaowu0162/LongMemEval.git
# then fetch the data it points at. We used the September 2025 cleaned release:
#   https://huggingface.co/datasets/xiaowu0162/longmemeval-cleaned
# and put longmemeval_s_cleaned.json under LongMemEval/data/

# 2. dependencies
python -m venv .venv && . .venv/Scripts/activate   # or bin/activate on POSIX
pip install -r lme-icm/requirements.txt

# 3. keys
cp lme-icm/.env.example lme-icm/.env    # then fill it in

.env is gitignored. The harness fails with a message naming the missing variable rather than a stack trace, and most single-arm runs need only one of the two keys.

Run it

Start with the things that cost nothing. Both exercise the real code paths against fixtures and a scripted fake model:

cd lme-icm
python smoke_offline.py      # ingestion, sandboxing, metrics, cost arithmetic
python smoke_mock.py         # the full agent loop under a fake model

Then a small live run. Four questions, one arm, a few cents:

python run_cost.py --arm guided --model sonnet5 --n 4 --keep-brains

--keep-brains is what leaves the memory on disk afterwards. Without it the workspace is deleted and you only get the numbers.

To reproduce a specific figure from the paper, see docs/RUNBOOK.md. Budget first: the four arms at n=40 cost roughly $330 in total, and build dominates. Start small.

Rebuild the paper

cd paper
make            # regenerates tables from results/, then runs pdflatex

Needs a TeX distribution. make tables alone regenerates the LaTeX tables without building the PDF, which is enough to check that a run you did changed what you expected it to change.

What this does not show

Stated plainly here because it is stated plainly in the paper. The benchmark is one user, one writer, one reader, in synthetic English chat history. It cannot show whether several people can share a structure without it degrading, whether a person can audit one under time pressure, or whether any of this holds over months rather than one ingestion pass. The properties the method exists for are mostly the ones LongMemEval has no way to express. Section 5 of the paper is about that, and section 6 describes the studies that would settle it.

The accuracy ordering in this study reversed twice as the sample grew from 5 questions to 39. We report all three sample sizes for that reason. Cost figures moved by under 4% across the same range. The two halves of this paper do not deserve equal confidence.

Citing

@misc{vanclief2026costofremembering,
  title  = {The Cost of Remembering: Filesystem Memory Against Long Context
            on {LongMemEval}},
  author = {Van Clief, Jake and McDermott, David and Kumar, Kay},
  year   = {2026}
}

If you use the benchmark, cite Wu et al., ICLR 2025 as well. If you use the ICM conventions, arXiv:2603.16021.

Provenance and redaction

The memories under results/ are derived from LongMemEval, which is assembled partly from real scraped conversations. Contact details that the build agent extracted from them have been redacted before publication, and docs/PROVENANCE.md says exactly what was replaced and why. No number in the paper depends on any of it.

Licence

MIT, for the harness and the paper source. Run outputs under lme-icm/results/ are derived from LongMemEval, also MIT, Copyright (c) 2024 Di Wu. See LICENSE.