Quantifying Cross-Query Contradictions in Multi-Query LLM Reasoning

Abstract: LLMs frequently produce mutually inconsistent answers when reasoning over multiple related queries. We study case-file logical consistency: maintaining a globally satisfiable belief state across interdependent queries. We introduce a benchmark of 390 multi-query reasoning instances with entailment/contradiction/unknown labels and propose set-level metrics including Case Satisfiability Rate, Contradiction Density and Revision Cost. Our solver-augmented approach extracts commitments, verifies global satisfiability and performs counterexample-guided repair. Across four reasoning domains, our method substantially reduces cross-query contradictions (SetCons: 0.56 to 0.94) while preserving per-query accuracy, demonstrating that global coherence is critical for robust multi-query reasoning.

• The paper introduces a solver-augmented framework that improves global consistency by detecting and minimally repairing cross-query contradictions.
• It presents a novel multi-query benchmark for evaluating logical consistency across interdependent queries in domains such as temporal, legal, and clinical reasoning.
• Results show enhanced SetCons rates and reduced revision costs, proving that symbolic verification significantly mitigates non-monotonic reasoning issues in LLMs.

Quantifying Cross-Query Contradictions in Multi-Query LLM Reasoning

Motivation and Problem Formulation

This work rigorously analyzes the phenomenon of cross-query contradictions in LLM-based reasoning scenarios, where models must answer bundles of interdependent queries posed over a shared natural language case file. While per-query logical ability is increasingly stress-tested in contemporary benchmarks, most evaluations overlook the requirement for case-file logical consistency: ensuring that the induced global belief state remains satisfiable as multiple related queries are answered. The authors identify that strong single-query accuracy does not guarantee global coherence; LLMs systematically produce sets of answers that, when jointly considered, can be mutually inconsistent even though each appears plausible in isolation.

The paper formally defines case-file logical consistency as a set-level property: given shared premises $P$ and a bundle of queries $Q$, the LLM must output labels $$(a_1, ..., a_n) \in \{\mathrm{ENTAILED}, \mathrm{CONTRADICTED}, \mathrm{UNKNOWN}\}^n$$ such that the union of these commitments is satisfiable under a symbolic logic encoding. This failure mode is particularly pertinent in high-stakes, multi-step reasoning settings such as clinical or legal analysis, where users require not only accurate but also non-contradictory sets of model outputs.

Benchmark Construction and Metrics

To enable systematic study, the authors introduce a multi-query benchmark composed of 390 case files spanning four reasoning domains (Relational/SAT, Temporal/SMT, Policy/Rules, and Abductive/Underspecified) and 2,450 bundled query instances. Each bundle consists of 5–8 queries sharing entities and inducing cross-constraints, assigned entailment labels via three-way expert annotation and solver-verified for logical soundness. Inter-annotator agreement is substantial (Fleiss' $\kappa = 0.82$) and further corroborated by solver-based gold validation.

Crucially, the benchmark supports both unordered/offline and sequential/online reasoning protocols, stressing long-range dependency management. The authors introduce several set-level evaluation metrics:

• Set Satisfiability Rate (SetCons): the fraction of bundles with a globally satisfiable belief state,
• Contradiction Density,
• Revision Cost (RevCost): the minimal number of label changes required to restore consistency,
• and AUC Prefix Consistency (AUC) in sequential settings.

Per-query metrics such as Macro-F1 and Unknown-F1 are also reported, but these do not reflect cross-query coherence.

Solver-Augmented Commitment Tracking and Repair

The core methodology is a model-agnostic, lightweight belief-state tracking and repair pipeline. Model outputs are systematically mapped to symbolic commitments using LLM-driven extraction prompts that yield canonical CNF or SMT-LIB encodings. The current belief state is incrementally updated with each query response; a solver (CaDiCal for SAT, Z3 for SMT) checks satisfiability after each step.

When an inconsistency is detected, a minimal-change repair routine is invoked. This process uses solver-generated unsatisfiable cores to localize the incompatible commitments, then prompts the LLM to suggest minimally disruptive repairs (flipping the current label, weakening the commitment, or retracting prior commitments). The objective function prioritizes minimal revision of prior commitments, minimal change to the current label, and constraint simplicity—operationalizing classical principles from belief revision theory.

An optional enhancement combines self-consistency sampling with logic-based filtering, only retaining candidate traces that are globally satisfiable.

Results: Baselines vs. Solver-Augmented Pipelines

The evaluation comprises both strong prompting baselines (Direct Decoding, Chain-of-Thought, Self-Consistency, Sequential Conditioning) and the proposed solver-augmented methods (Check-only, Check+Repair).

Key quantitative results:

• SetCons increases substantially, from baseline levels (e.g., 0.56) to 0.94 using Check+Repair, without materially increasing UNKNOWN prediction rates (stable at 16–19%). This demonstrates that the gains result from contradiction detection and repair, not over-abstention.
• Revision Cost is reduced by 2–3x vs. No-CoT, with the minimal belief revision requirement driven primarily by core-guided repair.
• Overhead is limited: belief tracking and repair incur only 1.3–1.6x wall-clock time of No-CoT; this is significantly lower than self-consistency, which requires 2.5–2.7x compute due to repeated sampling.
• CoT and self-consistency improve per-query metrics but do not ensure set-level satisfaction; contradiction rates remain high without explicit global verification.

The method's efficacy is most pronounced in structure-heavy domains (e.g., temporal/capacity reasoning), where single errors can globally break feasibility. For underspecified (abductive) cases, repair converts overconfident entailments to UNKNOWN, further stabilizing global consistency.

Analysis, Ablations, and Limitations

Ablations on the deep model subset show that core-guided repair is the largest individual contributor to lowering RevCost and improving SetCons. Allowing limited retraction of past commitments (vs. only revising the most recent) yields marginal further gains, aligning with the belief revision literature. Extraction failures and solver timeouts are mapped to conservative UNKNOWN predictions, ensuring robustness against parsing or tool limitations.

The authors recognize some limitations: extraction accuracy (91–95%) can still result in spurious SAT/UNSAT determinations, and solver/vendor/tool sensitivity remains a practical concern. The current framework is restricted to relatively lightweight logics and bundle sizes; scaling to richer fragments or larger context windows (e.g., probabilistic logic, nested quantification, or bundles of length tens–hundreds) is a stated objective for future work.

Theoretical and Practical Implications

The findings provide clear evidence that logical reliability in LLMs is not solely a per-query phenomenon but has an essential cross-query, set-level dimension. Robustness in real-world deployment scenarios requires explicit verification and repair of the global belief state—a property not guaranteed by state-of-the-art prompting or sampling strategies. The techniques and benchmark introduced constitute a reproducible foundation for further algorithmic and architectural improvements.

From a practical perspective, these results suggest that LLM-based reasoning systems interfacing with high-stakes, multi-turn workflows (e.g., legal, clinical, or scientific) must be equipped with mechanisms for cross-query contradiction detection and repair to ensure usable global consistency. Integration of symbolic reasoning solvers is feasible with modest compute costs and yields substantial dividends in reliability.

Theoretically, this work connects LLM reasoning research with classical belief revision theory and establishes that even gold-standard LLMs operate in a fundamentally non-monotonic regime at the set level unless supported by exogenous verification/repair. Future extensions to richer logical languages and training-time integration of case-file consistency objectives are promising research directions.

Conclusion

This paper systematically demonstrates that enforcing case-file logical consistency using solver-augmented belief state tracking and counterexample-guided repair yields major improvements in global coherence for multi-query LLM reasoning. Cross-query contradiction detection and minimal revision are necessary primitives for robust deployment. The benchmark and metrics provided will likely become standard tools for the evaluation of next-generation reasoning models, with implications for both research and high-reliability applications (2604.14525).