Ben Thomasson

4 minute read

I measured the context window usage across a 776-turn session with one of my research agents. The pattern was unmistakable:

Turn   1:   18,463 tokens
Turn 100:  101,082 tokens  (growing)
Turn 200:   46,198 tokens  (compacted)
Turn 300:   79,646 tokens  (growing)
Turn 400:  154,735 tokens  (near ceiling)
Turn 500:  119,187 tokens  (compacted)
Turn 600:   48,753 tokens  (compacted again)
Turn 700:  111,803 tokens  (growing)
Turn 776:   54,984 tokens  (post-compaction)

Context grows until it hits the ceiling (~165,000 tokens), then drops through lossy compression, then grows again. A sawtooth wave. The sharpest drop I measured:

Turn 755: 157,844 tokens. Turn 756: 18,722 tokens. An 88% loss in a single step.

What Gets Lost

Before compaction, the context contains:

  1. Justifications — why conclusions were reached
  2. Dependency chains — evidence → reasoning → conclusion
  3. Retraction history — what changed and why
  4. Contextual disambiguation — what claims meant when they were made

After compaction: the model retains conclusions but loses justifications. It knows what it believes but not why. It knows a value is correct but not which calculation produced it. It knows a hypothesis was rejected but not which test falsified it.

This isn’t just information loss. It’s a specific kind of information loss that destroys the ability to correct errors.

Doyle’s Summarization Problem (1979)

In 1979, Jon Doyle identified summarization as the key unsolved problem for truth maintenance systems: how do you compress a justification network — the full history of beliefs and their dependencies — without losing the ability to correct errors?

That’s exactly what context compaction does. It compresses the conversation history into a summary. The summary preserves conclusions (they seem important) and drops justifications (they seem like intermediate work). But justifications are what you need when a conclusion turns out to be wrong.

When the model compacts “X is true because of evidence A, B, and C” into “X is true,” you’ve lost the ability to revisit X when evidence A is invalidated. The belief persists without its foundation. In TMS terms, it’s an unjustified belief — a conclusion floating free of the reasoning that produced it.

Three Consequences

1. Beliefs outlive justifications. A belief formed before compaction persists after, but its supporting evidence was destroyed. If someone later asks “why do you believe X?” the model confabulates a justification rather than admitting it doesn’t remember. The confabulated justification may be wrong.

2. Implicit circumscription. After compaction, the model treats the summary as the complete state of the world. Anything not in the summary is implicitly retracted — not through principled belief revision, but through lossy compression. The model doesn’t know what it forgot.

3. Error correction breaks. If an error was incorporated into the summary (believed to be correct at compaction time), there’s no trail back to the source. Dependency-directed backtracking — the standard technique for tracing errors to their root cause — requires the dependency chain to exist. Compaction destroys it.

The Numbers

I measured compaction across all sessions in the research programme:

  • Peak context ceiling: 160,000-167,000 tokens (uniform across repos)
  • Effective working memory: 50,000-120,000 tokens between compactions
  • Fixed overhead: ~18,000 tokens (role definition, system prompt)
  • Available for work: 32,000-102,000 tokens
  • Compaction frequency: Every 100-200 turns in active sessions
  • Information loss per compaction: 50-88%
  • Cache hit rates: 99.6-100.0% (space cost is real, compute cost is amortized)

After several compaction cycles, accumulated information loss becomes significant. Very long sessions (700+ turns) may produce lower-quality reasoning than fresh sessions that explicitly load relevant context.

The Workaround: Externalize Belief State

The fundamental insight: files survive compaction, conversation doesn’t. An entry written to disk at turn 100 is still readable at turn 700, even after three compaction cycles. The conversation that produced the entry is gone, but the entry persists.

The beliefs compact command generates a token-budgeted summary of the current belief state:

beliefs compact --budget 300

This outputs a structured snapshot: how many claims, which are IN/OUT/STALE, active warnings, unresolved contradictions, and pending actions. Prioritized by entrenchment — high-confidence beliefs and active warnings first, speculative claims last.

After compaction, the model can read this summary and restore its belief state without needing the conversation that built it. It won’t remember why it believes each claim, but it will know what it believes and what problems are active. That’s enough to continue working.

This is a workaround, not a solution. The real fix would be platform-level changes to context management — preserving justification chains through compaction, or providing a structured belief state that persists across the context boundary. But until that exists, externalizing beliefs to files is the best available option.

Practical Advice

For short sessions (< 100 turns): Don’t worry about it. You probably won’t hit the ceiling.

For medium sessions (100-300 turns): Write important findings as entries before compaction hits. Run beliefs compact periodically to snapshot your belief state.

For long sessions (300+ turns): Consider starting fresh sessions that explicitly load the relevant context rather than relying on compacted conversation. A fresh session with good entries and a compact belief summary will outperform a compacted session that’s lost its justification chains.

For automated pipelines: Write outputs as files, not just conversation. If the pipeline hits a compaction boundary mid-run, the upstream work is lost unless it was written to disk.

The tools: entry for writing findings to disk. beliefs for maintaining belief state that survives compaction.

This is post 6 in a series on belief management for AI agents. Previously: 67 Minutes from Spec to Implementation. Next: the classical AI researchers who solved these problems in 1979 — and why nobody in the LLM world seems to know about it.

You May Also Enjoy