Disambiguating the causes of unsuccessful outcomes after intervention

Ascertain whether unsuccessful trial outcomes after applying DoVer’s orchestrator-level interventions in LLM-based multi-agent systems arise from incorrect failure attribution hypotheses or from system limitations that prevent faithful execution of the intervention, and develop diagnostics that reliably differentiate between these two causes to resolve the ambiguity in intervention evaluation.

Background

Within the DoVer pipeline, trial outcomes after targeted interventions are classified as Validated, Partially Validated, Refuted, or Inconclusive. The authors note that an Inconclusive outcome frequently occurs because agents fail to follow the intervened instruction, which makes it ambiguous whether the intervention was mislocalized (i.e., based on an incorrect failure hypothesis) or whether the system lacked the capability to carry out the intervention. This ambiguity complicates evaluation and motivates the need for more robust diagnostics to distinguish between these causes.

To mitigate the ambiguity, the paper uses an LLM-as-a-judge to check whether the intervention was faithfully executed (is_intervention_fulfilled) and combines this with success/progress measures. However, the underlying cause of failure in the Inconclusive category remains a core uncertainty, and the paper highlights it explicitly as unclear, suggesting a need for methods that can definitively attribute unsuccessful outcomes to either mislocalization or capability limitations.