Interactive Evaluation Requires a Design Science

Abstract: AI evaluation is undergoing a structural change. LLMs are increasingly deployed as systems that act over time through tools, environments, users, and other agents, while many evaluation practices still inherit assumptions from response-centered benchmarks (e.g., fixed inputs, isolated outputs, and outcome judgments that can be made from a single response). The field has begun to build interactive benchmarks, but the resulting landscape is fragmented: benchmarks differ in what interaction artifacts they admit, how trajectories are scored, and what claims their results support. This position paper argues that interactive evaluation should be treated as a principled evaluation paradigm, not merely a new family of agent benchmarks. Simply adopting previous evaluation paradigms does not suffice. We define evaluation as an autonomous mapping from evidence to judgments, and show that interactive evaluation changes both sides of this mapping: the evidence becomes interaction-generated trajectories, while the evaluation procedure must assess process, recoverability, coordination, robustness, and system-level performance. Building on this definition, we propose a two-axis taxonomy, derive design principles and reporting standards, examine representative scenarios, and analyze how longstanding evaluation challenges reappear at the trajectory level.

• The paper's main contribution is establishing a design science framework that redefines evaluation from static, response-centered benchmarks to dynamic, trajectory-based analysis.
• It introduces a comprehensive taxonomy distinguishing interaction artifacts from evaluation programs to assess process quality, recoverability, and system-level performance.
• The study proposes practical principles for benchmark design, advocating explicit protocol specification, perturbation testing, and multi-metric reporting to address deployment risks.

Interactive Evaluation as a Design Science: Position and Framework

Motivation and Problem Statement

Current evaluation protocols in AI and NLP have been historically dominated by response-centered paradigms, which predominantly measure model performance on fixed input-output mappings. As LLMs and agentic systems have rapidly evolved toward acting over time in tool-rich, user-interfaced, environment-based, or multi-agent settings, there is now a pronounced mismatch between what traditional benchmarks measure and what deployed systems require. "Interactive Evaluation Requires a Design Science" (2605.17829) critiques inherited assumptions from response-centered benchmarks and posits that interactive evaluation must be formalized as a scientific discipline, not merely a family of incremental benchmarks.

Figure 1: Benchmarks have evolved from fixed-instance response evaluation to executable, grounded, and interactive settings, revealing gaps and motivating a design-science approach in evaluation protocol.

The authors' analysis identifies an increasing divergence between academic and industry evaluation, with frontier-lab reports adopting more interactive protocols while academic works lag in adopting genuine trajectory-centric evaluations. This motivates the central claim that interactive evaluation requires explicit design principles, explicit evidence curation, and principled mapping from interaction-derived artifacts to evaluative judgments.

Formal Definition and Distinction

The conceptual core is the formalization of evaluation as an autonomous mapping: $E: \mathcal{X} \rightarrow \mathcal{Y}$, where $\mathcal{X}$ is the admissible evidence (previously final answers; now potentially full trajectories) and $\mathcal{Y}$ is the set of evaluative judgments (scores, diagnostics, pass/fail, etc.). Interactive evaluation is defined as the paradigm where $\mathcal{X}$ consists of interaction-generated trajectories, and $E$ must implement trajectory-to-judgment programs that can assess process, recoverability, coordination, safety, robustness, and system-level properties, not merely outcome or end-state correctness.

Figure 2: Response-centered evaluation focuses on judging final responses; interactive evaluation judges the entire trajectory for system-level performance and process characteristics.

The paper precisely demarcates boundary cases: not all multi-turn or tool-augmented benchmarks qualify as interactive. The key distinction is whether the interaction is consequential for the evidence and whether evaluation claims can only be supported via trajectory-level scrutiny.

Taxonomy: Evaluation Input vs. Evaluation Program

A central contribution is a two-axis taxonomy structuring interactive evaluation:

• Axis 1 (Interaction Artifact/Input): Categorizes the type of trajectory—tool/environment interaction, user interaction, multi-agent/social context, or hybrid/dynamic systems with persistent state.
• Axis 2 (Evaluation Program): Specifies the measurement logic—task success, process quality and efficiency, recoverability and robustness, safety/alignment/social competence.

  Figure 3: A two-dimensional design space mapping the landscape of interactive evaluation benchmarks by type of interaction artifact (input) and evaluative program (judgment protocol).

This framework exposes critical shortcomings in the current interactive benchmark landscape:

• Concentration on Task/Outcome: Despite collecting rich interaction traces, most benchmarks collapse these into scalar end-task labels without exploiting process-level evidence.
• Substrate-Bound Evaluation: Evaluation programs remain tightly coupled to the environment type; there is weak cross-pollination of process, robustness, or safety criteria across substrates.
• Sparse Robust Hybrid Evaluation: Long-horizon, persistent, and hybrid/dynamic system evaluations—where accumulated errors and process qualities are deployment-critical—are underexplored.

Principles and Roadmap for Benchmark Design

To move the field toward interpretable, robust, and comparable interactive evaluation science, the paper provides implementable principles:

• System and Protocol Specification: Explicitly define system scaffolds (tools, memory, orchestration) and the precise protocol by which interaction is generated.
• Perturbation/Repair Testing: Evaluate recoverability, robustness, and adaptation by actively perturbing interaction (e.g., ambiguous inputs, state drift, strategic counterparts).
• Multi-Metric Reporting: Separate final outcome, process quality, trajectory cost, risk, and failure recovery, rather than reporting aggregated metrics only.
• Infrastructure with Flexibility: Advocate for shared replay, logging, and scoring infrastructure but caution against protocol ossification that would limit domain diversity or mask deployment-relevant risks.

Risks and Open Issues in Interactive Benchmarking

The trajectory-level paradigm introduces new forms of risk:

• Gaming and Leakage at Trajectory Level: Policy overfitting and gaming can now target patterns in simulator regularities, state transitions, or judge heuristics, not just answer memorization.
• Protocol Sensitivity: Small changes in tool API, user modeling, or environment versioning can invalidate evaluation comparability unless fully documented and controlled.
• Construct Validity and Standardization Risks: Over-standardization or evaluator dependence can bias the field toward performative compliance rather than genuine capability and adaptation.
• Process/Outcome Entanglement: Reliance on human or LLM-as-a-judge trajectories may incentivize surface-level "good" behavior that is not robust or transferable, requiring fine-grained evidence-based metrics.

Theoretical and Practical Implications

By treating interactive evaluation as a scientific design problem, the field can systematically classify what can and cannot be inferred from trajectory-level evidence. The taxonomy and roadmap inform not only benchmark construction but meta-evaluation of agentic system performance, generalization, and safety. For instance, without trajectory-level diagnostics, brittle agents that overfit test harnesses cannot be distinguished from genuinely robust and human-aligned systems.

Future directions include: (1) protocol-agnostic infrastructure to enable cross-benchmark comparison, (2) systematic perturb-and-repair evaluations to expose non-i.i.d. environment risks, and (3) rigorous integration of evaluator and counterpart diversity to establish the generality and construct validity of interactive scores. Furthermore, this design science program can guide regulatory standards by enabling fine control and auditability of evidence, addressing societal and deployment risks in closed-loop AI systems.

Conclusion

This position paper provides a formal, taxonomy-driven, and risk-conscious framework for interactive evaluation, arguing that action-dependent trajectory evidence mandates explicit, principled mapping to system-level judgments. The authors demonstrate that mere adoption of multi-step or tool-based benchmarks is insufficient, and that a design science approach is necessary to ensure evaluation is interpretable, robust, and scientifically informative. The outlined taxonomy, roadmap, and risk analysis chart a clear path for advancing the methodological foundations of AI evaluation.