DeltaLogic: Minimal Premise Edits Reveal Belief-Revision Failures in Logical Reasoning Models

Abstract: Reasoning benchmarks typically evaluate whether a model derives the correct answer from a fixed premise set, but they under-measure a closely related capability that matters in dynamic environments: belief revision under minimal evidence change. We introduce DeltaLogic, a benchmark transformation protocol that converts natural-language reasoning examples into short revision episodes. Each episode first asks for an initial conclusion under premises P, then applies a minimal edit δ(P), and finally asks whether the previous conclusion should remain stable or be revised. We instantiate DeltaLogic from FOLIO and ProofWriter and evaluate small causal LLMs with constrained label scoring. On a completed 30-episode Qwen evaluation subset, stronger initial reasoning still does not imply stronger revision behavior: Qwen3-1.7B reaches 0.667 initial accuracy but only 0.467 revision accuracy, with inertia rising to 0.600 on episodes where the gold label should change, while Qwen3-0.6B collapses into near universal abstention. There, Qwen3-4B preserves the same inertial failure pattern (0.650 initial, 0.450 revised, 0.600 inertia), whereas Phi-4-mini-instruct is substantially stronger (0.950 initial, 0.850 revised) but still exhibits non-trivial abstention and control instability. These results suggest that logical competence under fixed premises does not imply disciplined belief revision after local evidence edits. DeltaLogic therefore targets a distinct and practically important reasoning capability that complements existing logical inference and belief-updating benchmarks.

• The paper introduces DeltaLogic, a benchmark protocol that quantifies language models' capacity for disciplined belief revision using minimal premise edits.
• It shows that high initial inference accuracy does not ensure reliable revision, with models exhibiting failure modes like inertia, over-flip, and abstention.
• The study demonstrates that increasing model scale alone is insufficient for robust revision, highlighting the need for targeted strategies in dynamic evidence updates.

DeltaLogic: Evaluating Minimal Belief Revision in Logical Reasoning Models

Overview and Motivation

The paper "DeltaLogic: Minimal Premise Edits Reveal Belief-Revision Failures in Logical Reasoning Models" (2604.02733) introduces DeltaLogic, a benchmark transformation protocol aimed at quantifying the ability of LLMs to perform disciplined local belief revision in response to minimal, controlled edits to premise sets. Unlike static reasoning benchmarks which evaluate a model's inference ability under fixed premises, DeltaLogic systematically tests models' capacity for precise belief revision—a critical requirement for systems that operate in dynamic environments where evidence evolves incrementally.

DeltaLogic converts standard natural language reasoning examples (from FOLIO and ProofWriter) into concise revision episodes. Each episode consists of two stages: inferring an initial answer from a premise set $P$, then inferring a revised answer after a minimal edit $\delta(P)$ is applied to the premises, with a deterministic target change in the gold label. This protocol makes errors in belief revision measurable, rather than anecdotal, and supports fine-grained failure mode analysis: inertia, over-flip, and abstention.

DeltaLogic Benchmark Protocol

The DeltaLogic protocol articulates a formal approach to measuring local belief revision. Each episode tuple $(P, q, y_0, P', y_1, t)$ is constructed as follows:

• $P$ and $q$ denote the original premises and query.
• $y_0$ is the initial gold label.
• $\delta(P)$ yields a minimally edited premise set $P'$ and revised gold label $y_1$.
• $t$ indexes the edit type.

DeltaLogic employs four edit types: support insertion, defeating-fact insertion, support removal, and irrelevant-fact addition. These edits respectively probe positive update, belief retraction, and stability under irrelevant evidence. By leveraging shallow examples with semantically deterministic effects, the protocol isolates the causal impact of minimal premise changes and supports rigorous evaluation of revision discipline.

Episodes are instantiated by systematically transforming public logical reasoning datasets into controlled test cases that require explicit revision or stability. The process ensures that revision errors (stale commitment, unnecessary revision, degenerate abstention) are quantifiable, enabling precise empirically grounded diagnosis.

Evaluation Metrics and Experimental Setup

DeltaLogic's evaluation metrics differentiate between initial reasoning performance and revision reliability:

• Initial Accuracy: $\delta(P)$0, quantifying correctness under the original premise set.
• Revision Accuracy: $\delta(P)$1, quantifying correctness post-edit.
• Inertia: Frequency of maintaining a prior answer when the gold label changes.
• Over-flip: Rate of unnecessary revision when the gold label is stable.
• Abstention: Frequency of retreating into uncertainty.

The revision gap, $\delta(P)$2, summarizes degradation in performance due to the evidence update.

The experimental protocol evaluates Qwen3-0.6B, Qwen3-1.7B, Qwen3-4B, and Phi-4-mini-instruct models using closed-label scoring. For each episode, the model scores the token log-likelihoods for labels True, False, and Uncertain, without chain-of-thought or task-specific fine-tuning. Episodes are divided into two subsets due to computational constraints: a 30-episode main Qwen evaluation and a 20-episode near-4B extension.

Empirical Results and Failure Mode Analysis

DeltaLogic exposes three strong empirical claims:

• Initial reasoning competence does not imply disciplined revision: Qwen3-1.7B achieves $\delta(P)$3 initial accuracy but only $\delta(P)$4 revision accuracy—the revision gap is pronounced. Qwen3-4B, despite being larger, yields similar revision failures ($\delta(P)$5 initial, $\delta(P)$6 revised). Phi-4-mini-instruct is markedly stronger ($\delta(P)$7 initial, $\delta(P)$8 revised) but still exhibits notable abstention and control instability.
• Failure is multi-modal: Qwen3-0.6B collapses into universal abstention ($\delta(P)$9 revised abstention rate), avoiding stale commitments by defaulting to uncertainty. Qwen3-1.7B and Qwen3-4B, in contrast, demonstrate inertia, failing to revise their beliefs when warranted ($(P, q, y_0, P', y_1, t)$0 inertia). Phi-4-mini-instruct trades lower inertia for higher, though non-trivial, over-flip and abstention.
• Edit-type breakdown reveals structural asymmetry: The models are revision-competent on support-insertion episodes (reacting to explicit positive evidence), but fail on support-removal and defeating-fact edits requiring retraction or reversal of prior beliefs. Phi-4-mini-instruct attains $(P, q, y_0, P', y_1, t)$1 revision accuracy on ProofWriter support-removal episodes, outstripping Qwen3-4B ($(P, q, y_0, P', y_1, t)$2 revision accuracy).

These results collectively refute the hypothesis that scale alone is a sufficient mechanism for developing robust belief revision—Qwen3-4B does not outperform Qwen3-1.7B in revision discipline, despite its larger parameter count.

Practical and Theoretical Implications

DeltaLogic delineates belief revision as a distinct and measurable reasoning capability. Practically, models that perform well on inference from static premises may still fail unpredictably when evidence changes—a critical vulnerability for systems deployed in dynamic environments (e.g., real-time document processing, dynamic rule management, or incremental evidence aggregation). The benchmark provides granular failure analysis, supporting the development of models with targeted revision discipline.

Theoretically, the findings underscore that logical competence under fixed premises is insufficient for disciplined belief revision after localized evidence edits. Model development must explicitly target revision regimes rather than rely on brute-force scale or static inference performance. The taxonomy of failure modes—stale commitments, degenerate abstention, unnecessary revision—presents concrete targets for architectural and training interventions.

Limitations and Future Directions

DeltaLogic achieves precise edit semantics by sacrificing ecological breadth; the current episodes derive from FOLIO and ProofWriter rather than open-world interactive contexts. Experiments employ CPU-only hardware, restricting evaluation to small subsets. The benchmark addresses single-step revisions and label correctness, not multi-step revision chains or qualitative justification. Future work may extend DeltaLogic to broader environments, larger models, interactive multi-step revision, and justification generation.

There is scope for integrating DeltaLogic's revision metrics into model training (e.g., curriculum learning with revision-specific reward structures) and for developing architectures tailored to local evidence tracking and disciplined belief update.

Conclusion

DeltaLogic establishes minimal belief revision as a distinct reasoning challenge inadequately captured by standard benchmarks. Across reported runs, small causal LMs exhibit diverse failure modes: inertia, universal abstention, and sporadic over-flip. Strong static reasoning does not guarantee reliable revision discipline, and scale alone is not sufficient for desirable update behavior. The practical imperative is clear: reasoning models must reliably update their commitments in response to evidence edits, and DeltaLogic provides an analytic protocol for direct measurement and development of this capability.