Interactive Benchmarks Overview

• Interactive Benchmarks are evaluation frameworks that assess systems by tracking entire interaction trajectories instead of isolated outputs.
• They integrate diverse evidence artifacts, such as runtime logs and coder feedback, to measure efficiency, robustness, and recoverability.
• These benchmarks propel advancements in interactive proofs, tool utilization, and user-centered evaluations for comprehensive system assessment.

Searching arXiv for papers on interactive benchmarks and related benchmark design. {"query":"interactive benchmarks evaluation design science interactive evaluation arXiv 2026", "max_results": 10} {"query":"\"Interactive Benchmarks\" arXiv", "max_results": 10} {"query":"interactive benchmarks trajectory evaluation arXiv 2026 design science", "max_results": 10} Interactive benchmarks are evaluation frameworks in which admissible evidence includes trajectories generated by consequential interaction, and the evaluation program maps those trajectories to judgments about system-level performance rather than only a single response to a fixed prompt (Xuan et al., 18 May 2026). They treat model behavior as a budgeted sequential process, often formalized by histories $h_t=(o_1,a_1,\dots,o_t)$, policies $a_t\sim\pi(\cdot\mid h_t)$, and explicit interaction budgets $\sum_t c(a_t)\le B$, so that active information acquisition, adaptation, and long-horizon strategy become first-class evaluation targets (Yue et al., 5 Mar 2026).

1. From response-centered testing to trajectory-centered evaluation

The defining shift is from evaluating isolated outputs to evaluating trajectories. In response-centered benchmarks, the evaluator typically sees a fixed input–output pair and produces a score. In interactive benchmarks, the admissible evidence is a trajectory such as

$$\tau = (s_0,a_0,o_1,s_1,a_1,o_2,\dots,s_T,a_T,o_{T+1}),$$

where later observations depend on earlier actions in ways that matter for evaluation. This makes process properties visible: recoverability after an error, robustness under perturbation, coordination with other agents, and the ability to decide what information to acquire next (Xuan et al., 18 May 2026).

This reconceptualization is partly a response to the limits of standard benchmarks. Static leaderboards are vulnerable to saturation, contamination, and preference noise, and they do not test whether a model can operate under under-specification or limited feedback. The interactive alternative asks whether a system can improve its belief state through action, not merely whether it can map a complete prompt to a plausible answer. In the formulation used by "Interactive Benchmarks" (Yue et al., 5 Mar 2026), the distinction is explicit: interactive proofs evaluate correctness under a turn budget, whereas interactive games evaluate long-horizon utility under repeated interaction.

2. Design dimensions and interaction substrates

A general taxonomy follows from the mapping $E:\mathcal{X}\to\mathcal{Y}$: one axis specifies what interaction-generated artifacts enter as evidence, and the other specifies how those artifacts are judged (Xuan et al., 18 May 2026). On the evidence side, interactive benchmarks span tools and environments, users, other agents, and hybrid dynamic systems. On the evaluation side, they span task success, process quality and efficiency, recoverability and robustness, and safety, alignment, or social competence.

This taxonomy is instantiated across a wide range of substrates. "InterCode" models coding as a POMDP with code as actions and execution feedback as observations in Bash, SQL, and Python environments, using Docker-backed execution to preserve state and expose runtime consequences (Yang et al., 2023). "NARRA-Gym" places a LLM inside an executable narrative environment in which every turn updates memory, pacing, planning, and optional artifact synthesis, thereby turning story generation into a logged, long-horizon agent benchmark (Huang et al., 8 May 2026). "AsgardBench" isolates visually grounded interactive planning in AI2-THOR by removing navigation and low-level control, so that agents must revise high-level plans from images, action history, and lightweight success or failure signals (Tupini et al., 16 Mar 2026). "LiViBench" extends the substrate to omnimodal livestreams with video, audio, ASR, and real-time comments, making audience–streamer interaction part of the benchmarked evidence (Wang et al., 21 Jan 2026).

Other domains reveal the same pattern. "TriMediQ" evaluates interactive medical QA as iterative consultation between an expert system and a patient system, with clinical facts revealed over multiple turns (Meng et al., 3 Oct 2025). "BAIT" unifies multiple interactive theorem-proving environments and supervised proof tasks under shared Data, Model, Environment, and Experiment modules, so that intermediate representations and end-to-end proving can be compared across systems (Lamont et al., 2024). The common feature is not domain but consequence: each action changes what can be known, done, or evaluated next.

3. Evaluation programs, metrics, and evidence quality

Interactive benchmarks differ sharply in how they convert trajectories into judgments. Some emphasize end-state success, while others explicitly separate outcome, process, and risk. The design-science view argues that these dimensions should not be collapsed into a single opaque score, because two trajectories with the same final outcome can differ in efficiency, safety, and recoverability (Xuan et al., 18 May 2026).

Several benchmark families make this separation concrete. "NARRA-Gym" distinguishes textual narrative quality from user experience through two composite metrics,

$$\text{StoryQ}=\frac{1}{7}\sum_{d\in\{Rel,Coh,Emp,Sur,Eng,Cpx,Char\}}\text{score}_d,\qquad \text{UX}=\frac{1}{4}\sum_{d\in\{Sat,PQual,Help,Reuse\}}\text{score}_d,$$

with scores assigned by a three-judge LLM ensemble and complemented by human evaluation (Huang et al., 8 May 2026). "Triangulating LLM Progress through Benchmarks, Games, and Cognitive Tests" defines an interactive game score as

$\text{GameScore}=\frac{q}{100}\times p,$

where $q$ is average quality across played games and $p$ is the fraction of games completed, thereby combining outcome with interaction reliability (Momentè et al., 20 Feb 2025). "TETRIS" replaces single-trajectory reporting for interactive segmentation with robustness-aware quantities such as IoU-Min, IoU-Max, and IoU-D, where IoU-D measures the gap between best-case and worst-case valid click trajectories (Moskalenko et al., 2024).

A second issue is whether the benchmark can actually support its own claims. "Evidence-Supported Bounds for Interactive-Agent Evaluation" shows that native success labels may be based on weak proxies or incomplete post-state evidence, and introduces an outcome-evidence layer with labels Evidence Pass, Evidence Fail, and Unknown, together with partial-identification bounds

$$\mathrm{Perf}(a,d)\in\left[\frac{P}{N},\frac{P+U}{N}\right],\qquad \mathrm{Width}(a,d)=\frac{U}{N},$$

where $a_t\sim\pi(\cdot\mid h_t)$0, $a_t\sim\pi(\cdot\mid h_t)$1, and $a_t\sim\pi(\cdot\mid h_t)$2 are the counts of evidence-supported passes, fails, and undecidable cases (Gao et al., 11 May 2026). This reframes score reporting itself as an interactive-benchmark design problem: the environment may log trajectories, but unless the retained artifacts decide the claimed outcome, the headline number is not fully warranted.

4. Representative benchmark families

A major family evaluates reasoning by constraining how information can be acquired. "Interactive Benchmarks" instantiates this with interactive proofs and interactive games. In interactive proofs, models query a judge to solve situation puzzles or challenging mathematics under a turn budget of 20, receiving restricted answers such as yes, no, both, or irrelevant before submitting a final answer (Yue et al., 5 Mar 2026). In interactive games, they play Texas Hold’em or an iterated Trust Game, where the evaluation target is long-horizon utility rather than objective truth. The point is not merely multi-turn dialogue but strategic query selection and action under budget.

A second family centers tool use and executable feedback. "InterCode" standardizes interactive coding as an RL-style environment in which the agent writes code, executes it, receives runtime observations, and refines its behavior across turns (Yang et al., 2023). "When Benchmarks Talk" shows how static code benchmarks can be converted into interactive ones by obfuscating the problem statement and requiring the model to recover critical information through simulated user feedback, with up to 5 refinement steps and feedback types including sentence feedback, paragraph feedback, query rephrasing, and code feedback (Pan et al., 25 Feb 2025). "EnIGMA" further expands this paradigm for cybersecurity by adding Interactive Agent Tools that let an LM agent run interactive utilities such as a debugger and a server connection tool inside CTF environments, making benchmark difficulty depend on whether the interface itself supports the task’s interaction structure (Abramovich et al., 2024).

A third family is explicitly user-centered. "NARRA-Gym" treats narrative assistance as a long-horizon, user-adaptive benchmark over eight predefined personas, with resistance-sensitive personalization, pacing control, and optional story-grounded artifact synthesis all logged in trajectory form (Huang et al., 8 May 2026). "Is This It?" argues that ecologically valid situated-collaboration benchmarks should be derived from real interactions with end-to-end systems such as Sigma on HoloLens 2, because questions that arise in context differ sharply from post hoc QA over static images or videos (Bohus et al., 2024). "TriMediQ" occupies a related position in clinical reasoning: the expert system must gather patient facts across turns rather than answer from a complete record in one shot (Meng et al., 3 Oct 2025).

A fourth family isolates interaction in embodied, visual, or multimodal settings. "AsgardBench" targets visually grounded interactive planning under minimal feedback, with 108 task instances spanning 12 task types and requiring plan repair from visual observations rather than navigation skill (Tupini et al., 16 Mar 2026). "LiViBench" defines interactive livestream understanding over 3,168 videos, 24 tasks, and 3,175 multiple-choice questions, with audio, speech, and real-time comments treated as first-class modalities (Wang et al., 21 Jan 2026). Interactive segmentation benchmarks reveal another aspect of the field: "Localized Interactive Instance Segmentation" argues that the click protocol itself is part of the benchmark and proposes localized clicking near the object rather than arbitrary global negatives (Majumder et al., 2020), while "TETRIS" shows through a real user study that common click-generation heuristics do not match real user behavior and that robustness should be measured over multiple valid click trajectories rather than a single deterministic one (Moskalenko et al., 2024).

5. Instrumentation, reproducibility, and benchmark infrastructure

Interactive benchmarks are unusually sensitive to implementation choices because the evaluation target is a process, not a file of predictions. As a result, infrastructure is part of the scientific object. "NARRA-Gym" is exemplary in this respect: it logs construction stages, per-turn memory state, pacing signals, planning output, artifacts, prompts, responses, and latencies, so that failures can be localized to premise construction, memory drift, stagnation, or personalization errors rather than treated as undifferentiated losses (Huang et al., 8 May 2026). "BAIT" makes the same point for theorem proving through its explicit modularization into Data, Model, Environment, and Experiment components (Lamont et al., 2024).

Executable environments and containerization are another recurring pattern. "InterCode" uses self-contained Docker environments and a Gym-like interface with reset, step, and close, making code execution, filesystem changes, and interpreter state part of the benchmark itself (Yang et al., 2023). "Codabench" generalizes this platform logic: tasks are defined by ingestion and scoring programs bundled with data and a Docker image, allowing arbitrary evaluation loops, including reinforcement-learning style agent–environment interaction, while keeping software and hardware conditions fixed across submissions (Xu et al., 2021). In security benchmarks, "EnIGMA" shows that not only the environment but also the agent–computer interface matters: non-blocking interactive sessions and narrowly scoped commands for debuggers or remote server tools make the difference between nominal access to a tool and effective interactive use (Abramovich et al., 2024).

These implementation choices are not incidental. In interactive evaluation, logging schemas determine what evidence exists, interface design shapes what behaviors are possible, and reproducibility depends on being able to replay not just a prompt but a full trajectory. The benchmark is therefore partly an environment, partly an evaluator, and partly an instrumentation layer.

6. Limitations, controversies, and future directions

A persistent controversy concerns validity. Interactive benchmarks promise greater realism, but protocol choices can silently determine what the scores mean. The design-science position argues that every benchmark must make explicit what evidence is logged, what claims its scores support, how counterpart behavior is generated, and how stopping rules, perturbations, and persistence affect evaluation (Xuan et al., 18 May 2026). Without that documentation, an interactive score is no more self-interpreting than a static one.

Several recurring failure modes follow from this. One is evaluator dependence. "NARRA-Gym" reports different strictness profiles across its three LLM judges and notes that human and judge rankings diverge most on user-experience dimensions, indicating that trajectory-level qualities are difficult to approximate from transcript text alone (Huang et al., 8 May 2026). Another is evidence mismatch: the evidence-supported-bounds framework documents Unknown cases, target-set construction errors, omitted task requirements, and proxy-based false successes across AndroidWorld, AgentDojo, AppWorld, $a_t\sim\pi(\cdot\mid h_t)$3-bench retail, and MiniWoB (Gao et al., 11 May 2026). A third is shortcutting through feedback. In "AsgardBench", adding detailed failure explanations can make text-only agents competitive with, or stronger than, image-based agents on some settings, showing that rich feedback can substitute for the very perception the benchmark is meant to test (Tupini et al., 16 Mar 2026).

A separate line of criticism concerns ecological validity. "Is This It?" argues that post hoc question generation from static data misses response obligation, self-talk, deictic reference, and proactive assistance, all of which emerge in real situated collaboration (Bohus et al., 2024). "TETRIS" similarly shows that user-click simulation based on the center of the largest error region is not empirically grounded and can overstate real-world performance (Moskalenko et al., 2024). These findings suggest that interactive benchmarks must often pair simulation with human validation, not because simulation is unnecessary, but because interaction protocols themselves are hypotheses about how systems will be used.

The field’s near-term trajectory is therefore methodological as much as empirical. The literature points toward broader persona and genre coverage in narrative systems, more live-user studies, stronger logging and evidence reporting, perturbation-and-repair protocols, richer multimodal interaction, and shared infrastructure that standardizes reporting without freezing the design space (Huang et al., 8 May 2026). Taken together, these developments define interactive benchmarks not as a narrow family of agent tasks but as a general evaluation paradigm for systems whose competence is only visible when acting over time.