Ben Thomasson

9 minute read

This is the second post in a series about how External Epistemic Memory (EEM) works in practice. Series: How EEMs Actually Work (Post 2 of 8). Post 1: The Derive Prompt

One in three derived beliefs gets retracted. That is the system working correctly.

Derive proposes new conclusions by combining existing beliefs. It’s a generator — deliberately aggressive, over-productive, designed to find connections. But generators make mistakes. The review gate catches them, and the retraction cascade propagates the corrections automatically. This is the mechanism that makes the whole system trustworthy instead of just voluminous.

What Derive Doesn’t Check

The previous post covered validate_proposals() — the structural checks that run before a derived belief enters the database. Those checks verify:

  • Do the referenced antecedents exist?
  • Is the proposed belief ID unique?
  • Is it too similar to a retracted belief?

But structural validity doesn’t guarantee semantic soundness. A belief can reference real antecedents and still be a logical leap, an overgeneralization, or a circular argument. Consider:

Antecedent 1: auth-jwt-validation
  "The authentication service validates JWT tokens synchronously"

Antecedent 2: auth-single-thread-refresh
  "Token refresh uses a single-threaded executor"

Derived: auth-is-production-ready
  "The authentication service is production-ready"

Both antecedents exist. The ID is unique. Structurally valid. Semantically absurd — nothing about synchronous JWT validation and single-threaded token refresh implies production readiness. If anything, they suggest the opposite.

Review-beliefs catches this.

The Review Prompt

The review prompt (REVIEW_PROMPT in reasonsforge/review.py:15) presents derived beliefs in batches of 20 and asks the LLM to evaluate three axes:

1. Valid — Does the conclusion follow?

“Does this conclusion logically follow from its antecedents? A conclusion that sounds reasonable but doesn’t follow strictly from the antecedents is NOT valid.”

This is the core check. The LLM sees the claim text AND the full antecedent texts — not just IDs, but what each antecedent actually says. It evaluates whether the inferential step from antecedents to conclusion is sound.

The key word is “strictly.” A plausible-sounding conclusion that requires unstated assumptions is invalid. This is deliberately rigorous. The system would rather retract a true-but-poorly-justified belief than keep a false-but-plausible one.

2. Sufficient — Are the antecedents enough?

Are the listed antecedents enough to support this conclusion, or does the derivation require additional unstated assumptions?

A belief might be valid given its antecedents plus some background knowledge that isn’t in the network. “The authentication service is a potential bottleneck” follows from the synchronous validation and single-threaded refresh — IF you know that synchronous operations block request handling. That background assumption is the smuggled premise. If it’s not in the network as a separate belief, the derivation is insufficient.

3. Necessary — Are all antecedents load-bearing?

Could any antecedent be removed without weakening the conclusion?

Derive sometimes pads its justifications. The LLM saw ten beliefs in the prompt and listed five as antecedents, but only three actually support the conclusion. The other two were in scope and topically related but don’t contribute to the inferential step. Marking them as unnecessary doesn’t retract the belief — it trims the justification to its load-bearing components.

Structured Forensics for Invalid Beliefs

When a belief is invalid, a bare “invalid” verdict isn’t very useful. The review prompt requires structured evidence — an audit trail through the justification graph:

Scope findings

For each antecedent examined, the reviewer records what it actually establishes versus what the derived belief claims it covers:

{
  "antecedent": "auth-jwt-validation",
  "establishes": "JWT tokens are validated synchronously per request",
  "does_not_establish": "overall production readiness of the auth service"
}

This creates a traceable record. When you ask “why was this belief retracted?”, you don’t get “an LLM said it was wrong.” You get a specific accounting of what each antecedent actually supports and where the gap is.

Missing property

The single property the derived belief claims but no antecedent establishes. In our example: “production readiness” — no antecedent says anything about production readiness.

Defeat reason type

The logical failure is classified into exactly one of seven categories:

Type What it means
unsupported-conjunct Conclusion claims a property no antecedent establishes
over-generalizes Conclusion universalizes from bounded evidence
false-causal-claim Conclusion asserts causation from co-occurrence
internal-contradiction Conclusion contradicts its own antecedents
circular-reasoning Antecedent presupposes the conclusion
missing-bridge Gap between subsystems not connected by antecedents
scope-mismatch Antecedents cover a narrower scope than conclusion claims

These categories aren’t just labels — they’re actionable diagnostics. An unsupported-conjunct tells you to look for a missing premise. An over-generalizes tells you to weaken the claim. A circular-reasoning tells you the derivation chain has a loop. Each category suggests a different repair strategy.

The Retraction Cascade

When review marks a belief invalid, retraction propagates through the justification network. Every belief that depends on the invalid one re-evaluates its own support:

  • If the retracted belief was its ONLY antecedent (mode ALL with other antecedents, or mode ANY sole supporter), the dependent goes OUT too.
  • If the dependent has alternative justifications that still hold, it stays IN.

This is the TMS doing what it was designed to do. One invalid belief at depth 2 can take 19 dependents with it — and each of those dependents was built on the faulty foundation. The cascade doesn’t need a human to trace the dependency graph. It happens automatically, in milliseconds.

This is also why the 13-37% retraction rate per derive round is a feature, not a bug. Derive over-generates. Review catches the errors. Retraction propagates the corrections. What remains is the subset of derived beliefs that survived scrutiny — the ones where the inferential step from antecedents to conclusion is sound.

The Repair Stage

Retraction is the blunt instrument. Sometimes a belief is wrong in a way that can be fixed rather than discarded. The repair stage (reasonsforge/repair.py) triages each invalid belief via LLM into four patterns:

The claim is sound but missing an antecedent — there’s a smuggled premise. Repair extracts the specific factual claim the conclusion assumes but no antecedent states, searches the network for an existing premise that supports it, and wires it in as a new antecedent.

The search uses a two-step LLM pipeline: first extract the smuggled claim as a single sentence, then match it against candidate premises. If a match is found, the belief gets a new antecedent and stays IN with a stronger justification than it had before.

2. Soften

The claim overstates the evidence. The antecedents support something, but not as strong as what the conclusion says. Repair rewrites the belief text to match what the antecedents actually establish:

  • “the mechanism” becomes “a primary mechanism”
  • “ensures” becomes “supports”
  • “all services” becomes “the surveyed services”

The belief stays IN with weakened text. The justification structure doesn’t change — the same antecedents now support a more modest claim.

3. Abandon

The dependency chain is too broken to repair. The belief is retracted with a structured reason, and its dependents cascade OUT. This is the right outcome when the inferential step was fundamentally flawed, not just overstated.

4. Research

The claim is plausible but can’t be confirmed from current evidence. Instead of prematurely abandoning it, the belief is preserved pending further investigation. This is for cases where the claim might be true but the current network doesn’t have the premises to support it — a gap in the knowledge base, not a flaw in the reasoning.

The Convergence Loop

Review isn’t a one-time pass. It’s part of a four-stage convergence loop that runs until the network stabilizes:

while not converged:
    1. DERIVE    — propose new beliefs (exhaust mode, multiple rounds)
    2. REVIEW    — audit all derived beliefs
    3. REPAIR    — fix invalid beliefs (search-and-link, soften, abandon, research)
    4. DEDUPLICATE — remove duplicates (with LLM verification)

    if invalid_count == 0 and new_derivations == 0:
        converged = True

Each cycle through the loop improves the network:

  • Derive adds new beliefs, including ones that build on repaired beliefs from the previous cycle
  • Review catches errors in the new and repaired beliefs
  • Repair fixes what can be fixed, removes what can’t
  • Dedup eliminates near-duplicates that derive may have produced

The loop converges when a cycle produces zero new derivations AND zero invalid beliefs. The network has stabilized — everything in it has been proposed, reviewed, and found sound.

Why the loop matters

Without review, derive builds tall reasoning chains on shaky foundations. A depth-1 belief that overgeneralizes becomes the antecedent for depth-2 conclusions, which feed depth-3 meta-claims. By depth 4, the conclusion might be three inferential leaps away from anything the premises actually support. Review catches the rot early. Repair either fixes the foundation or removes it, and the cascade takes the unsupported tower with it. The next derive round then rebuilds from the corrected base, producing better-grounded conclusions.

This is why the retraction rate decreases over cycles. Early cycles catch gross errors (false causal claims, unsupported conjuncts). Later cycles catch subtler issues (scope mismatches, unnecessary antecedents). The network converges toward a state where every remaining belief has survived at least one round of adversarial scrutiny.

Review-beliefs is the main gate, but two companion commands audit different aspects:

review-justifications — Checks whether SL justifications are correctly classified as ALL vs ANY. A multi-antecedent justification marked ALL will cascade-retract if any single antecedent goes OUT. If the antecedents are actually convergent evidence (each independently supporting the conclusion), it should be ANY. Misclassification makes the network too fragile (ALL when it should be ANY — one retraction breaks everything) or too resilient (ANY when it should be ALL — retracting a load-bearing antecedent has no effect).

review-premises — Checks base premises against their source material for factual accuracy. This audits the foundation. If a premise misquotes or overgeneralizes from the source document, everything built on it is suspect. Review-premises catches errors at depth 0, before they propagate upward.

A Complete Cycle

$ reasonsforge derive --exhaust --auto --model claude:sonnet
[round 1] Network: 500 IN beliefs, 45 derived, max depth 3
[round 1] 12 valid proposals (3 skipped)
[round 2] 5 valid proposals (1 skipped)
[round 3] No new proposals — network saturated.
Total added: 17.

$ reasonsforge review-beliefs --model claude:sonnet
  [INVALID] complex-error-handling-maturity
    Conclusion claims comprehensive error handling but antecedents
    only cover retry logic
    defeat_reason_type: scope-mismatch
  [INSUFFICIENT] cross-module-test-coverage
    Missing evidence for integration test coverage
    defeat_reason_type: unsupported-conjunct
  [UNNECESSARY(basic-logging-present)] observability-strategy-coherence
    basic-logging-present does not contribute to the claim
Reviewed 57 derived beliefs. Invalid: 1  Insufficient: 1  Unnecessary: 1

$ reasonsforge repair --review-file reviews/review-beliefs-*.json
  [SOFTENED] complex-error-handling-maturity
    "Retry logic is present across service boundaries"
    (was: "Comprehensive error handling...")
  [LINKED] cross-module-test-coverage
    Linked: integration-test-config (existing premise wired as new antecedent)
Total: Linked 1, Softened 1, Abandoned 0, Research 0

17 beliefs proposed. 2 flagged. 1 softened, 1 repaired. 15 survived unchanged. The network is better than it was — not by adding more, but by catching what was wrong.

What This Doesn’t Cover

  • Contradiction detection — review checks individual beliefs against their antecedents. Contradiction detection checks beliefs against each other. That’s post 4 in this series.
  • The depth-8 ceiling — why review retracts 100% of beliefs beyond depth 8. That’s in the saturation post (post 6).
  • Cross-model review quality — does a different model reviewing than the one that derived catch more errors? Open question.

Try It Yourself

pip install reasonsforge

# Review all derived beliefs
reasonsforge review-beliefs --model ollama:qwen3.8:27b

# Auto-retract invalid beliefs (cascades dependents)
reasonsforge review-beliefs --model ollama:qwen3.8:27b --auto-retract

# Review only deep derivations
reasonsforge review-beliefs --model ollama:qwen3.8:27b --min-depth 2

# Run the full derive-review-repair loop
reasonsforge forge derive-review-repair --model ollama:qwen3.8:27b --rounds 3

# Dry run — see what would be retracted without modifying the database
reasonsforge review-beliefs --model ollama:qwen3.8:27b --dry-run

The code is at github.com/benthomasson/reasonsforge. The review prompt is in reasonsforge/review.py. The repair logic is in reasonsforge/repair.py.

You May Also Enjoy