Measuring LLM Trust Allocation Across Conflicting Software Artifacts

Abstract: LLM-based software engineering assistants fail not only by producing incorrect outputs, but also by allocating trust to the wrong artifact when code, documentation, and tests disagree. Existing evaluations focus mainly on downstream outcomes and therefore cannot reveal whether a model recognized degraded evidence, identified the unreliable source, or calibrated its trust across artifacts. We present TRACE (Trust Reasoning over Artifacts for Calibrated Evaluation), a framework that elicits structured artifact-level trust traces over Javadoc, method signatures, implementations, and test prefixes under blind perturbations. Using 22,339 valid traces from seven models on 456 curated Java method bundles, we evaluate per-artifact quality assessment, inconsistency detection, affected artifact attribution, and source prioritization. Across all models, quality penalties are largely localized to the perturbed artifact and increase with severity, but sensitivity is asymmetric across artifact types: documentation bugs induce a substantially larger heavy-to-subtle gap than implementation faults (0.152-0.253 vs. 0.049-0.123). Models detect explicit documentation bugs well (67-94%) and Javadoc and implementation contradictions at 50-91%, yet show a systematic blind spot when only the implementation drifts while the documentation remains plausible, with detection dropping by 7-42 percentage points. Confidence is poorly calibrated for six of seven models. These findings suggest that current LLMs are better at auditing natural-language specifications than at detecting subtle code-level drift, motivating explicit artifact-level trust reasoning before correctness-critical downstream use.

• The paper introduces TRACE, a framework that quantifies LLM trust allocation by eliciting structured reasoning on conflicting software artifacts.
• It evaluates seven state-of-the-art LLMs on 456 Java method bundles, revealing asymmetric sensitivity with stronger documentation fault detection than code drift recognition.
• Findings highlight that while LLMs reliably identify documentation inconsistencies, they struggle with subtle implementation faults, underscoring the need for explicit code consistency checks.

Trust Allocation in LLMs Across Conflicting Software Artifacts

Motivation and Framework

The proliferation of LLMs in software engineering workflows introduces a new class of failures: not only incorrect outputs, but also misallocated trust across inconsistent artifacts such as source code, method signatures, documentation, and test scaffolding. When such artifacts conflict, most evaluations focus strictly on downstream outputs (e.g., correctness of test oracles), overlooking whether LLMs appropriately recognize degraded evidence, identify unreliable sources, or calibrate trust. The paper "Measuring LLM Trust Allocation Across Conflicting Software Artifacts" (2604.03447) addresses this gap by formalizing artifact-level trust reasoning and introducing TRACE, a calibrated framework for eliciting structured ‘reasoning traces’ on heterogeneous software artifacts under blind, provenance-annotated perturbations.

TRACE operates by making artifact-level trust judgments explicit, collecting per-artifact quality assessments, inconsistency detection with attribution, and reliability ranking across artifacts. It emphasizes artifact symmetry: neither code nor documentation is presumed authoritative. The study applies TRACE across seven leading LLMs and 456 carefully curated Java method bundles with controlled perturbations to documentation and code, yielding 22,339 validated traces and enabling direct measurement of input-quality sensitivity, semantic contradiction detection, and the impact of model-specific semantic code understanding.

Benchmark Construction and Perturbation Protocol

The evaluation leverages OE25, a benchmark spanning 25 Java projects, selecting non-trivial methods (8–60 lines, substantive logic) and enforcing rich English Javadoc (≥15 characters of description, required tags), rigorous filtering, and artifact deduplication. Each method bundle is perturbed along six axes: deterministic removal of docstring components (description, @return), and injection of semantic bugs (to Javadoc, MUT, or both), stratified by three severity tiers (heavy, normal, subtle). Faults are LLM-generated to mirror plausible real-world inconsistencies, ensuring naturalness and realism in the altered artifacts.

All perturbations are presented in a blind protocol: models receive only the bundled artifacts, without extrinsic labels or hints about which components—if any—are modified. This ensures that quality scores, inconsistency flags, and prioritization choices are intrinsic model judgments, not prompted behavior.

TRACE Protocol and Model Suite

TRACE queries seven LLMs (Anthropic Claude Opus, Sonnet, Haiku; OpenAI GPT-5.2 Chat, GPT-4o; DeepSeek-V3.2-Speciale; xAI Grok-4 Fast Reasoning) with identical prompts and harness, extracting structured JSON traces with artifact-specific quality metrics, multi-level conflict analysis, prioritization orders, and confidence scores. Models are selected for provider-coverage and tier diversity. Artifact-level evaluations are performed at temperature 0.0 to ensure deterministic behavior.

The output schema supports direct quantitative evaluation, correlating artifact-level trust allocation, explicit conflict detection, and source prioritization with perturbation provenance, severity, and artifact identity.

Input-Quality Sensitivity (RQ1)

Model traces demonstrate robust sensitivity to artifact perturbation. Quality penalties are concentrated on the perturbed artifact, increasing monotonically with severity. Crucially, the response is asymmetric: documentation bugs induce substantially larger heavy-to-subtle score gaps (0.152–0.253) than implementation faults (0.049–0.123). Severity stratification holds across all models; Sonnet and Opus exhibit the highest sensitivity, GPT-4o the lowest.

This artifact-asymmetric response has direct deployment implications. LLMs act as reliable auditors of natural-language specifications—Javadoc defects are surfaced with strong signal—whereas code-level drift, especially subtle or implementation-only faults, is under-signaled. For pipeline integration, documentation quality ought to be high-leverage for prompt construction and input triage, but cannot substitute explicit code consistency checks.

Explicit Contradiction Detection and Attribution (RQ2)

Models reliably detect explicit documentation faults (67–94%) and Javadoc–implementation contradictions (50–91%). Detection rates scale with perturbation severity but are systematically higher for documentation-side faults. When only the implementation is perturbed and documentation remains plausible (MUT_only), detection drops sharply (7–42 percentage points across models), revealing a blind spot: LLMs primarily reason from documentation cues and fail to consistently flag code-drift in the absence of strong specification mismatch.

False positive rates and confidence calibration further nuance these results. Most models operate with distinct baseline noise floors and poorly calibrated confidence scores; only DeepSeek-V3.2 displays meaningful separation between correct and incorrect detections. Description fidelity correlates with explicit detection: models that catch inconsistencies also generate higher semantic alignment in their explanations.

Source prioritization further supports the asymmetry: only Sonnet and Opus show meaningful concordance in deprioritizing faulty documentation, without consistent discrimination for code faults.

For practical application, LLM-based consistency checkers are sufficiently reliable for documentation auditing in CI/CD or code review triage, but not for autonomous detection of implementation drift. Confidence should not be used as a gating metric without explicit calibration.

Semantic Code Understanding and Inter-Model Variation (RQ3)

Inter-model variation in trace fidelity is largely attributed to differences in semantic code understanding. Models that maintain contradiction detection under subtle perturbations (Sonnet, DS-V3.2, Haiku) also produce more accurate, stable fault descriptions, evidencing deep, cross-artifact behavioral reasoning. GPT-4o exhibits the steepest drop in subtle contradiction detection (heavy-to-subtle: 44 percentage points), indicative of surface-level pattern matching, not semantic code comprehension.

Opus forms a distinct niche: strong semantic understanding and description fidelity but systematic fault mislocalization, penalizing its utility as a sole detector. Detection stability and description accuracy converge on the same model ranking, reinforcing semantic code understanding as a principal driver.

For integration, Sonnet, DS-V3.2, and Haiku are preferred for tasks requiring subtle implementation-level contradiction detection. Opus is best used for secondary, high-fidelity explanation after fault surfacing.

Limitations, Validity, and Practical Implications

TRACE uncovers several concrete limitations:

• Cross-model comparisons require explicit calibration—confidence and quality floors are inconsistent.
• Subtle code faults remain significantly under-signaled relative to documentation faults.
• Implementation-only drift is systematically missed; silent failures are the dominant mode in real codebases.
• Description accuracy is not universal; some models localize faults incorrectly despite semantic recognition.

Practical deployment should treat LLM traces as triage; high-quality Javadoc is a strong prompt signal, but explicit code consistency checks remain indispensable. The strongest models are documentation auditors, not code-drift detectors.

Conclusion

TRACE establishes artifact-level trust reasoning as a first-class target, alongside a controlled benchmark for evaluating LLMs on contradictory software artifacts (2604.03447). Artifact-level penalties are localized and severity-proportional but artifact-asymmetric; explicit documentation faults and contradictions are reliably detected and described, whereas subtle code drift eludes robust detection. Inter-model variation is dominated by semantic code understanding, with strong models remaining robust under subtle perturbations. The framework provides actionable guidance for SE assistant design and integration, suggesting artifact-aware evaluation and quality-aware prompting as necessary steps before correctness-critical downstream use. Future research may extend TRACE for broader artifact sets and more granular semantic analysis, addressing calibration and compositional robustness required for trustworthy LLM-based SE pipelines.