- The paper introduces a DQA framework that uses a persistent diagnostic state to aggregate evidence and systematically track diagnostic hypotheses.
- It employs retrieval aggregation and state-conditioned action selection, achieving a 78.7% trajectory success rate and reducing interactions from 8.4 to 3.9 turns.
- Experimental results highlight improved resolution completeness and efficiency over traditional RAG methods, emphasizing aggregated hypothesis-level evidence.
Diagnostic Question Answering for IT Support: DQA Framework
Motivation and Problem Setting
Enterprise IT support scenarios present an inherently diagnostic, multi-turn interaction challenge, where resolving user-reported issues requires systematic evidence gathering and disambiguation of competing hypotheses. Traditional retrieval-augmented generation (RAG) techniques, particularly in multi-turn dialogue contexts, fail to maintain explicit diagnostic state—a deficiency that impedes evidence accumulation, hypothesis tracking, and principled action selection. The core limitations are context window constraints, redundant retrievals due to large, noisy historical case repositories, and a lack of mechanisms for systematic uncertainty reduction.
DQA Framework and Methodology
DQA (Diagnostic Question Answering) is introduced to explicitly address these deficiencies via persistent diagnostic state modeling and evidence aggregation at the root-cause level, rather than at the level of individual documents. The framework is characterized by three core innovations:
- Retrieval Aggregation via RAggG: Instead of retrieving and conditioning on potentially redundant individual cases, DQA retrieves a candidate neighborhood of tickets and clusters them based on root-cause (resolution) embeddings. Aggregation yields compact, hypothesis-level evidence representations, dramatically improving signal-to-noise under context and latency constraints.
- Persistent Structured Diagnostic State: DQA encodes the evolving diagnostic process in a turn-persistent, structured state object. This state contains a dynamically updated hypothesis-weight vector over candidate root-cause clusters, representative evidence sets, accumulated user-reported and system-elicited symptoms, and ranked supporting knowledge base artifacts.
- State-Conditioned Action Selection: Leveraging the diagnostic state, DQA guides LLM-driven response generation via a policy abstraction comprising clarifying questions (to reduce entropy), investigative steps (to validate dominant hypotheses), and resolution proposals (when uncertainty is minimized).
The dialogue engine includes a lightweight conversational query rewriting module that normalizes terminology and maintains semantic coherence with the current diagnostic state, efficiently mitigating retrieval drift prevalent in multi-turn interactions.
Experimental Results
DQA is evaluated on 150 anonymized, real-world enterprise IT support scenarios using a controlled replay protocol where simulated users respond to system prompts in situ. Systems compared include standard RAG without query rewriting, RAG with standalone conversational query rewriting, and RAG with clustering-based aggregation but no persistent state.
Key numerical findings:
- DQA yields a 78.7% trajectory-level success rate, versus 41.3% for the best non-aggregating RAG baseline—a relative improvement of 90.6 percentage points.
- Diagnosis and resolution completeness (fraction of required facts/steps provided) reach 0.99 and 0.94, respectively.
- DQA reduces interaction length, averaging 3.9 turns to resolution versus 8.4 for the baseline.
- Ablation indicates the largest performance gain is driven by the introduction of persistent diagnostic state (+24.9pp), over and above query rewriting (+0.9pp) or semantic clustering (+12.5pp).
These results highlight the value of explicit state tracking and aggregated evidence modeling. Maintaining an evolving, structured diagnostic context enables both superior uncertainty reduction and computational efficiency, as DQA focuses system effort on high-value discriminatory actions.
Theoretical and Practical Implications
DQA operationalizes a key principle for retrieval-augmented agents operating in diagnostic domains: compressing retrieved information into dynamically maintained, task-relevant hypothesis-level aggregates, and tightly coupling evidence aggregation with decision policy. By eschewing complex symbolic or Bayesian belief propagation in favor of retrieval-induced empirical priors, DQA achieves scalability and tractable context maintenance while retaining flexibility for real-world, unscripted troubleshooting.
From a practical viewpoint, this architecture generalizes beyond IT support to any domain that exhibits a large solution space, ambiguous inputs, and the need for evidence-driven disambiguation—e.g., fault triage in complex technical systems or even clinical decision support. However, the explicit reliance on high-quality, adequately structured historical case data, and the use of simulated user interactions for benchmarking, delineate current generalizability and real-world deployment boundaries.
Limitations and Future Directions
DQA’s evaluation, while systematic, is confined to simulated user settings. Uncertainty persists regarding live user satisfaction, trust, and adaptability to domains where root-cause annotation is sparse or noisy. Its performance on non-IT diagnostic domains, e.g. medical or hardware fault triage, remains to be empirically established. Future research should focus on live deployment studies, more robust unsupervised root-cause induction methods under weak supervision, and hierarchical or compositional diagnostic state representations suitable for more complex environments.
Conclusion
DQA represents a formal step forward in retrieval-augmented diagnostic reasoning by integrating persistent, structured state tracking with aggregated evidence modeling and policy-driven multi-turn interaction. The framework’s results suggest significant potential for enterprise troubleshooting agents and future retrieval-augmented LLM architectures deployed in diagnostic and other evidence-accumulative artificial intelligence contexts (2604.05350).
