Ben Thomasson

12 minute read

This is the first post in a series about how External Epistemic Memory (EEM) works in practice — not the theory, but the machinery. Series: How EEMs Actually Work (Post 1 of 8).

We send 300 beliefs to an LLM and ask it to find connections we missed. Here is the exact prompt, the exact parser, and the exact validation pipeline. Every claim in this post traces to a line of code you can read yourself.

If you’ve ever wondered what happens inside a system that maintains thousands of justified beliefs with automatic retraction cascades, this is where to start.

The Problem

You have 500 beliefs about a codebase. Each one is a justified claim — “the authentication service uses JWT tokens” or “the database connection pool defaults to 10 connections.” Some are depth-0 premises, directly observed from reading source code. Others are derived: “the authentication system is a potential bottleneck” follows from the JWT implementation plus the synchronous validation pattern plus the single-threaded token refresh.

But those derived conclusions didn’t arrive automatically. Someone — a human or an LLM — had to read the premises, see the pattern, and articulate the conclusion. The derive command automates this: it asks an LLM to find connections between existing beliefs and propose new derived conclusions.

The hard part isn’t generating conclusions. LLMs are excellent generators. The hard part is controlling the process so the output is trustworthy.

The Prompt

The derive prompt (reasonsforge/derive.py:21) has four sections:

1. Background: What a Reason Maintenance System Is

The LLM needs to understand the substrate it’s working with. It’s not generating free text — it’s proposing nodes in a justification network with formal semantics:

A Reason Maintenance System (RMS) tracks beliefs with justifications
and automatic retraction cascades. There are three kinds of nodes:

1. Base premises (depth-0): Observable facts with no justifications
2. Derived conclusions (depth-1+): Justified by antecedents via SL rules
3. Outlist-gated conclusions: Justified by antecedents UNLESS certain
   nodes are IN

This isn’t decoration. When the LLM proposes a derived conclusion, it needs to classify each derivation as ALL or ANY:

  • ALL means the conclusion requires every antecedent. Retracting any single one retracts the conclusion. This is for logical chains: “if A and B then C.”
  • ANY means each antecedent independently supports the conclusion. It survives as long as at least one holds. This is for convergent evidence: “three separate observations all point to the same conclusion.”

Getting this classification wrong means the retraction cascade — the core value proposition of the whole system — propagates incorrectly.

2. Task: Three Kinds of Derivation

The prompt asks the LLM to do three things:

  1. Combine existing conclusions into higher-level claims — depth N+1 from depth N. This deepens the reasoning chain.
  2. Group related base beliefs into thematic conclusions — new depth-1 nodes. This creates structure.
  3. Connect positive and negative chains via outlist semantics — where a positive claim holds UNLESS a negative claim is IN.

That third one — GATE beliefs — is where it gets interesting.

GATE: Connecting Positive and Negative Reasoning

A GATE belief says: “the system is production-ready UNLESS critical-bug-123 is IN.” While the bug is active (IN), the production-readiness claim is OUT. When someone fixes the bug and retracts it, the production-readiness claim automatically flips to IN.

This is how the system connects two separate reasoning chains — the positive chain arguing the system works, and the negative chain cataloging its problems. Without GATE beliefs, these chains exist in parallel and never interact. With them, fixing a bug automatically updates the conclusions that depend on it.

In the output format:

### GATE system-is-production-ready
The system meets all production readiness criteria
- Antecedents: all-tests-pass, monitoring-configured, docs-complete
- Unless: critical-bug-auth-timeout
- Mode: ALL
- Label: Production readiness gated on critical bug resolution

3. Rules: Controlling the Generator

The LLM is an excellent generator. Too excellent. Without constraints, it will produce plausible-sounding conclusions with tenuous connections to their stated antecedents. The rules section constrains this:

  • At least 2 antecedents per conclusion. Single-antecedent derivations are just restatements.
  • Only load-bearing antecedents. If a belief was in scope but isn’t essential to the conclusion, don’t list it. This prevents the LLM from padding its justifications with everything it saw.
  • Prefer combining derived beliefs over grouping base beliefs. Deepening the chain (depth 2 from depth 1) is more valuable than widening it (more depth 1 from depth 0).
  • No forced connections. The antecedents must be genuinely related. The LLM will find connections between any two beliefs if you let it — the constraint is that the connection must be real.

4. The Beliefs Themselves

This is where it gets complicated.

A 500-belief network can’t fit entirely in a prompt. The budget system (_build_beliefs_section, line 208) controls how much the LLM sees. The default budget is 300 beliefs.

There are three sampling strategies, each with trade-offs:

Alphabetical truncation (default): Simple — sort belief IDs alphabetically, take the first 300. But biased. Beliefs starting with ‘a’ always get included; beliefs starting with ‘z’ never do. For a codebase where beliefs are named by component (auth-*, database-*, queue-*), this means the authentication subsystem gets thoroughly derived while the queue subsystem is invisible.

Random sampling (--sample): Uniform coverage across rounds. Each round sees a different random subset. But random selection may split related beliefs across rounds — you might see auth-jwt-validation in round 3 but auth-token-refresh not until round 7, missing the connection between them.

Semantic clustering (--cluster): Uses sentence-transformers embeddings to group beliefs by meaning, then samples proportionally from each cluster. Related beliefs stay together. The --intra-cluster variant goes further: each round focuses the entire budget on a single cluster, rotating through clusters across rounds. Deep exploration of one topic at a time.

The belief section groups beliefs by prefix and shows each one truncated to 120 characters:

### auth (47 beliefs, showing 12)
- `auth-jwt-validation`: The authentication service validates JWT tokens synchronously on each...
- `auth-token-refresh`: Token refresh uses a single-threaded executor that blocks during...

This truncation is a trade-off. The LLM sees enough to identify each belief but not enough to reason deeply about its content. It’s identifying connections at the level of “these two things are related,” not “here is the full logical argument.”

The Parser

The LLM responds with structured proposals. parse_proposals() (line 401) extracts them via regex:

new_pattern = re.compile(
    r"### (DERIVE|GATE) (\S+)\n"
    r"(.+?)\n"
    r"- Antecedents: (.+?)\n"
    r"(?:- Unless: (.+?)\n)?"
    r"(?:- Mode: (ALL|ANY)\n)?"
    r"- Label: (.+?)(?:\n|$)",
)

This is a rigid parser for a reason. Structured output from LLMs — even when prompted with an exact format — is never perfectly reliable. The regex is the contract: if the LLM’s output doesn’t match, the proposal doesn’t exist.

This creates a known failure mode: a model that generates valid conclusions in a slightly wrong format produces zero proposals. The system can’t distinguish between “the network is fully explored” and “the model’s markdown was slightly off.” Both look like saturation — zero new proposals.

The parser also handles a legacy format from an earlier version (the old_pattern branch), because belief databases built with v0.9 still exist and their derive reports need to be re-parseable.

Validation: The Last Line of Defense

Before a proposal enters the database, validate_proposals() (line 661) checks three things:

1. Do the referenced antecedents exist?

missing = [a for a in p["antecedents"] if a not in nodes]

If the LLM hallucinated a belief ID — claimed its conclusion follows from auth-oauth-flow but no such belief exists — the proposal is rejected. This catches the most common generation error: plausible-sounding references to beliefs that aren’t in the network.

2. Does the proposed belief already exist?

if p["id"] in nodes:
    skipped.append((p, "already exists"))

Straightforward dedup. If the LLM proposes a belief that’s already in the network, skip it.

3. Is it too similar to a retracted belief?

This is the most interesting check. find_similar_out() (line 644) tokenizes the proposed belief ID into lowercase hyphen-separated words and computes Jaccard similarity against all OUT (retracted) beliefs:

def find_similar_out(proposal_id, nodes, threshold=0.5):
    p_tokens = _tokenize_id(proposal_id)
    matches = []
    for nid, node in nodes.items():
        if node.get("truth_value") != "OUT":
            continue
        sim = _jaccard(p_tokens, _tokenize_id(nid))
        if sim >= threshold:
            matches.append((nid, sim))

If more than 50% of the tokens in the proposed ID overlap with a retracted belief’s ID, the proposal is blocked.

Why? Because retraction is a judgment. If auth-is-production-ready was retracted (maybe the review found the claim wasn’t supported), a proposal for auth-system-is-production-ready is likely re-deriving the same invalid conclusion with slightly different wording. The Jaccard check prevents this — it respects the TMS’s retraction decisions by refusing to let differently-worded versions of invalidated conclusions sneak back in.

The 50% threshold is a balance. Too low and legitimate new beliefs get blocked because they share vocabulary with retracted ones (auth-timeout-handling blocked by retracted auth-timeout-is-critical). Too high and the system re-derives conclusions it already rejected.

The Exhaust Loop

Single derive rounds are useful, but the real value comes from iterating. The exhaust loop (--exhaust) runs derive repeatedly until saturation:

for round_num in range(1, max_rounds + 1):
    added = _derive_one_round(...)
    if added == 0:
        break

Each round re-exports the network, so beliefs added in round 1 are visible in round 2. This is how chains deepen: round 1 derives depth-1 conclusions from depth-0 premises, round 2 combines those into depth-2, and so on.

The loop stops when a round produces zero new proposals. This is zero-tolerance — one dry round and the loop exits. There’s no “try one more time” or “switch clusters and retry.” This is a known limitation. With --intra-cluster, if cluster 0 (the first one tried) produces nothing, the loop stops without ever trying clusters 1 through k. The cluster that might have the richest unexplored territory never gets a chance.

What the LLM Actually Sees

Here’s a concrete example of what a derive prompt looks like for a 500-belief codebase knowledge base, sampled to 300 with semantic clustering:

You are a reasoning architect analyzing a belief network...

## Existing Beliefs

### auth (47 beliefs, showing 12)
- `auth-jwt-validation`: The authentication service validates JWT...
- `auth-token-refresh`: Token refresh uses a single-threaded...
[10 more]

### database (38 beliefs, showing 10)
- `database-pool-default`: The connection pool defaults to 10...
- `database-migration-v2`: Migration v2 adds a nullable column...
[8 more]

[... more groups ...]

## Existing Derived Conclusions

#### [IN] depth-2: `auth-is-potential-bottleneck`
The authentication service is a potential performance bottleneck
- Antecedents: auth-jwt-validation, auth-token-refresh, auth-sync-check

#### [IN] depth-1: `database-schema-stable`
The database schema has been stable since v2 migration
- Antecedents: database-migration-v2, database-no-recent-changes

[... more derived conclusions ...]

## Statistics
- Total IN beliefs: 487
- Existing derived: 43
- Max depth: 3

The LLM sees the premises, the existing derived conclusions (so it doesn’t re-propose them), and summary statistics. From this, it proposes new connections.

The Derived Section Problem

There’s a subtle issue with the prompt that becomes critical at scale. The beliefs section (_build_beliefs_section) is filtered — if you use --intra-cluster, it shows only the focused cluster’s beliefs. But the derived section (_build_derived_section, line 375) shows ALL existing derivations across the entire network, up to the budget limit.

This means the LLM sees a narrow slice of base beliefs (one cluster) alongside a broad view of everything that’s already been derived (all clusters). The network looks more thoroughly explored than it actually is. The LLM’s response is anchored by the comprehensive derivation list: “I can see you’ve already derived 200 conclusions. My narrow view of the base beliefs suggests… no, these are already covered.”

This is the primary cause of false saturation in large networks. It’s a prompt architecture problem, not a model capability problem. The fix is to filter the derived section to match the beliefs section — show only the derivations whose antecedents are in the current belief sample.

The Depth-8 Ceiling

In practice, derive chains rarely go past depth 8. At depth 8 and beyond, review (the next stage in the pipeline) retracts 100% of proposals. The deeper the chain, the more tenuous each link — and review catches this.

This isn’t a bug. It’s a structural property of how LLMs reason about justification chains. Each derivation step adds some probability of error. By depth 8, the accumulated error probability is high enough that nothing survives quality review. The system is wide, not deep — many depth-1 and depth-2 conclusions, fewer depth-3 and depth-4, almost nothing deeper.

The practical implication: if you want deeper reasoning, you need to add new depth-0 observations (from new source documents, new experiments, new data) that give the system fresh premises to reason from. Derive can’t create depth from nothing.

Reports and Audit Trail

Every derive run writes a JSON report to reports/ with round-by-round details: network statistics at each round, proposals found and validated, what was applied, what was skipped and why, and cost tracking (tokens and API calls).

This audit trail matters. When a belief in your network turns out to be wrong and you trace back through its justification chain, the derive report tells you which round proposed it, what the LLM saw in its prompt, and which antecedents it claimed to be reasoning from. This is the difference between a knowledge base and a pile of text — you can trace how you got here.

What This Doesn’t Cover

This post covered how derive proposes new beliefs. It didn’t cover:

  • How review catches bad derivations — derive over-generates, review prunes. One in three derived beliefs gets retracted. That’s the subject of the next post.
  • How dedup handles near-duplicates — when two independent derive runs produce similar-but-not-identical conclusions, some of those “duplicates” are actually contradictions.
  • How the exhaust loop should handle cluster rotation — the zero-tolerance exit is a known bug. The fix is to allow k dry rounds before stopping (one per cluster).
  • How the sampling strategy affects what gets derived — alphabetical bias means some beliefs never participate in derive at all. Semantic clustering mitigates it but doesn’t eliminate it.

The system works. It also has known problems. The next post in the series covers review — the mechanism that catches the errors derive introduces.

Try It Yourself

pip install reasonsforge

# Build a small knowledge base from source documents
reasonsforge forge pipeline --sources-dir docs/ --model ollama:qwen3.8:27b

# Run a single derive round
reasonsforge derive --model ollama:qwen3.8:27b --auto

# Run until saturation with semantic clustering
reasonsforge derive --model ollama:qwen3.8:27b --auto --exhaust --cluster

# See what was proposed
ls reports/

The code is at github.com/benthomasson/reasonsforge. Every function referenced in this post is in reasonsforge/derive.py.

You May Also Enjoy