Dedup: When Similar Beliefs Are Actually Contradictions

This is the third post in a series about how External Epistemic Memory (EEM) works in practice. Series: How EEMs Actually Work (Post 3 of 8). Post 1: The Derive Prompt / Post 2: The Review Gate

Auto-merging duplicate beliefs sounds safe until you realize some duplicates are actually contradictions.

As beliefs accumulate — from document ingestion, multiple derive rounds, or multi-agent imports — the same claim often appears in different words. “The API uses JWT tokens for auth” and “Authentication is handled via JSON Web Tokens in the API” are the same claim. Without dedup, they waste the derive budget (the LLM sees both), inflate belief counts, and create fragile justification chains where two antecedents are really one.

But not every pair of similar-looking beliefs is a duplicate. Some are orthogonal properties that share vocabulary. Some are scope differences. And some are genuine contradictions — conflicting claims that word similarity alone cannot distinguish from duplicates. The dedup pipeline exists to tell the difference.

The Three-Stage Pipeline

Stage 1: Candidate Clustering

deduplicate() in reasonsforge/api.py:4329 finds clusters of similar IN beliefs using union-find. Two modes:

Jaccard mode (default): Tokenizes each belief ID by splitting on hyphens, then computes Jaccard similarity between token sets. Pairs above the threshold (default 0.5) are unioned into clusters.

_tokenize_id("api-jwt-auth-mechanism") → {"api", "jwt", "auth", "mechanism"}
_tokenize_id("api-auth-jwt-tokens")    → {"api", "auth", "jwt", "tokens"}
Jaccard = |{api, jwt, auth}| / |{api, jwt, auth, mechanism, tokens}| = 3/5 = 0.6 ✓

Semantic mode (--semantic): Uses sentence-transformers embeddings (default all-MiniLM-L6-v2) to compute cosine similarity on belief text rather than IDs. Higher quality but more expensive.

Both modes use union-find to build transitive clusters: if A~B and B~C, all three form one cluster even if A and C aren’t directly similar. The belief with the most dependents is chosen as the survivor — it’s the one the rest of the graph relies on.

Stage 2: LLM Verification

This is where dedup stops being a string-matching exercise. verify_dedup_clusters() in reasonsforge/api.py:4432 sends each candidate cluster to an LLM for classification into exactly one of three categories:

SAME_CLAIM — genuine duplicates, safe to merge. “The API uses JWT tokens for auth” and “Authentication is handled via JSON Web Tokens in the API.”

DIFFERENT_CLAIMS — false duplicates, leave alone. “api-error-handling-retry” and “api-error-handling-logging” share tokens but describe different error-handling aspects.

CONTRADICTION — conflicting claims that word similarity surfaced. “cache-ttl-300-seconds” and “cache-ttl-3600-seconds” share structure but state conflicting values.

The verify prompt is deliberately simple:

You are classifying a group of beliefs from a knowledge base that
were flagged as potential duplicates based on word similarity.

Beliefs in this cluster:
  - ID: api-jwt-auth-mechanism
    Text: The API uses JWT tokens for authentication
  - ID: api-auth-jwt-tokens
    Text: Authentication is handled via JSON Web Tokens in the API

Classify this cluster as ONE of:
- SAME_CLAIM: These beliefs make the same factual claim in
  different words. They are genuine duplicates.
- DIFFERENT_CLAIMS: These beliefs share vocabulary but make
  distinct, non-redundant claims. They should NOT be deduplicated.
- CONTRADICTION: These beliefs contradict each other. They
  should be recorded as a conflict, not deduplicated.

Response parsing is loose — any line containing “SAME” maps to SAME_CLAIM, “CONTRA” to CONTRADICTION, anything else to DIFFERENT_CLAIMS. LLM failures default to rejected (safe — no data lost).

Stage 3: Application

For verified duplicates: rewrite dependents to point at the survivor, then retract the duplicate. For contradictions: record as nogoods via add_nogood(), which triggers TMS backtracking.

The critical safety mechanism is _rewrite_dependents(). Before retracting a duplicate, it rewires every belief that depends on it:

def _rewrite_dependents(net, old_id, new_id):
    for dep_id in list(old_node.dependents):
        dep = net.nodes[dep_id]
        for j in dep.justifications:
            if old_id in j.antecedents:
                j.antecedents = [new_id if a == old_id else a
                                 for a in j.antecedents]
                new_node.dependents.add(dep_id)

Without this, retracting a duplicate would cascade-retract every derived conclusion that cited it, even though the same claim still exists under the kept ID. Rewriting prevents false cascades.

The False Duplicate Problem

Word similarity alone has a ~10% false-positive rate. The failures cluster into three patterns:

Orthogonal properties: auth-jwt-api-gateway and auth-jwt-internal-services share tokens but apply to different system boundaries. Merging them would lose the distinction between gateway-level and service-level authentication.

Scope differences: api-error-handling-retry and api-error-handling-logging share vocabulary but describe different error-handling aspects. Merging them would collapse two independent observations into one.

Contradictions: cache-ttl-300-seconds and cache-ttl-3600-seconds share structure but state conflicting values. Merging them would silently discard the conflict — the worst possible outcome. Contradictions are discoveries, not noise.

LLM verification catches all three: the first two as DIFFERENT_CLAIMS, the third as CONTRADICTION. Without verification, 10% of merges would be wrong, and some of those wrong merges would hide genuine contradictions.

The Human Review Path

Without --auto, dedup writes a plan file for human review:

# Deduplication Plan

Review each cluster below. Delete any cluster you want to skip,
or change which belief is KEEP vs RETRACT. Then run:
  reasonsforge deduplicate --accept proposed-dedup.md

---

## Cluster 1 (2 beliefs)

- [KEEP] `api-jwt-auth-mechanism`  (3 dependents)
  The API uses JWT tokens for authentication
- [RETRACT] `api-auth-jwt-tokens`
  Authentication is handled via JSON Web Tokens in the API

The plan file is both human-readable and machine-parseable. Users can delete entire clusters, swap KEEP/RETRACT labels, or remove individual RETRACT lines. reasonsforge deduplicate --accept proposed-dedup.md applies the edited plan.

This matters because the LLM isn’t always right either. An LLM that has never seen your codebase might classify two beliefs as SAME_CLAIM when they actually describe different services with similar names. The plan file gives the domain expert final say.

Dedup in the Convergence Loop

Since v0.64.0, LLM-verified dedup is the default in all forge pipeline stages. The convergence loop runs derive → review → repair → dedup, and verified dedup runs after each repair cycle:

while not converged:
    1. DERIVE    — propose new beliefs
    2. REVIEW    — audit all derived beliefs
    3. REPAIR    — fix invalid beliefs
    4. DEDUPLICATE — remove duplicates (with LLM verification)

    if invalid_count == 0 and new_derivations == 0:
        converged = True

Dedup runs last because derive can produce near-duplicates of existing beliefs (the LLM rephrased a conclusion slightly), and repair can produce near-duplicates of each other (two softened beliefs landing on similar wording). Running dedup after repair catches both.

Contradictions as Discovery

The most valuable output of the dedup pipeline isn’t the duplicates it removes — it’s the contradictions it surfaces. When the LLM classifies a cluster as CONTRADICTION, the belief IDs are passed to add_nogood(). This records the conflict in the TMS and triggers backtracking — the system retracts the least-supported belief.

Contradictions masquerading as duplicates are a real discovery. “The service is stateless” and “The service maintains session state” look like duplicates to Jaccard similarity. To the LLM, they’re a conflict. To you, they reveal an inconsistency in the source material or in the derive chain that produced them.

This is why dedup isn’t just cleanup — it’s an error-detection channel. The next post in this series covers what happens when contradictions are found in the wild, including a cascade that retracted 19 beliefs from a single discovery.

Try It Yourself

pip install reasonsforge

# Find duplicate clusters (no action, writes plan file)
reasonsforge deduplicate

# LLM-verified dedup (recommended)
reasonsforge deduplicate --verify --auto --model ollama:qwen3.8:27b

# Semantic similarity instead of ID tokens
reasonsforge deduplicate --semantic --threshold 0.85

# Accept a reviewed plan
reasonsforge deduplicate --accept proposed-dedup.md

# Custom Jaccard threshold (default 0.5)
reasonsforge deduplicate --threshold 0.6

The code is at github.com/benthomasson/reasonsforge. The dedup logic is in reasonsforge/api.py. The verify prompt is inline in verify_dedup_clusters().

What This Doesn’t Cover

• Contradiction cascades — what happens when a discovered contradiction triggers backtracking that retracts 19 beliefs. That’s post 4.
• Parallel construction — two independent runs on the same codebase produce <1% exact overlap but ~4.5% near-duplicates. The dedup pipeline is what makes parallel construction practical.
• Semantic dedup quality — embedding-based dedup catches paraphrases that Jaccard misses, but adds latency and cost. When to use which mode is an open question.