AgentChangeBench Evaluation Framework

• AgentChangeBench is a benchmark that evaluates LLM agents' robustness during mid-dialogue goal shifts by measuring task success, tool efficiency, and adaptation latency.
• It simulates 2,835 dynamic task sequences across banking, airline, and retail domains to capture real-world interactions and user persona variability.
• The framework employs multi-dimensional metrics such as TSR, TUE, TCRR, and GSRT to enhance understanding of agent resilience and guide improvements in efficiency.

AgentChangeBench is a multi-dimensional benchmark and evaluation framework designed to assess the robustness of tool-augmented LLM agents in the presence of mid-dialogue goal shifts across enterprise domains. It systematically quantifies not only ultimate task success but also efficiency, redundancy, and adaptation latency—criteria critical for real-world deployment but overlooked by static-goal or one-shot task benchmarks. AgentChangeBench comprises 2,835 simulated task sequences with explicitly specified goal-shift points and five distinct user personas, providing a reproducible testbed for evaluating and improving agent resilience in dynamic settings (Rana et al., 20 Oct 2025).

1. Motivation and Benchmarking Gaps

Real-world multi-turn AI interactions frequently involve users changing objectives, adding new constraints, or shifting priorities mid-dialogue. Enterprise use cases such as banking (authenticate → dispute charge), airline booking (select flight → cancel/rebook), and retail (order tracking → return/exchange) all reflect such dynamic workflows. Traditional benchmarks—e.g., AgentBench, τ-bench, τ²-bench—primarily evaluate either fulfillment of a static or implicitly-shifting goal using pass$^k$ metrics, and thus:

• Do not capture latency in agent adaptation or subtle drifts in objective alignment,
• Mask inefficiencies such as repeated, wasted tool calls,
• Fail to account for user persona variability and realistic linguistic shift triggers.

AgentChangeBench remedies these limitations by introducing explicit and diversified goal-shift scenarios, multi-dimensional performance metrics, and task simulations reflecting diverse customer behaviors and communication patterns.

2. Metrics and Formal Evaluation Protocol

AgentChangeBench formalizes evaluation through four complementary metrics, each capturing a distinct aspect of real-world agent robustness:

Task Success Rate (TSR):

Weighted composite of three channels:

• Communicate Info Rate (fraction of required facts surfaced),
• Action Rate (fraction of tool calls executed correctly),
• NL Assertion Rate (fraction of behavioral/policy assertions satisfied):

$$\mathrm{TSR} = 0.25 \times \text{communicate\_info\_rate} + 0.45 \times \text{action\_rate} + 0.30 \times \text{nl\_assertion\_rate}$$

Tool Use Efficiency (TUE):

Assesses reliability of tool invocation, combining tool correctness (T) and parameter validity (P):

$\mathrm{TUE} = 0.6\,T + 0.4\,P$

where $$T = \frac{\#\text{successful tool calls}}{\#\text{total tool calls}}$$, $$P = \frac{\#\text{calls with valid arguments}}{\#\text{total calls}}$$. As $P$ is near ceiling ($\approx 0.986$), TUE primarily reflects tool correctness.

Tool-Call Redundancy Rate (TCRR):

Captures wasted effort. A call is redundant if it duplicates an identical call within a 3-turn window or exceeds a batch threshold (more than 2 calls to the same function in any window):

$$\mathrm{TCRR} = \frac{\#\text{redundant calls}}{\#\text{total tool calls}}$$

Goal-Shift Recovery Time (GSRT):

Quantifies adaptation latency and recovery rate post shift. For each shift:

• $ack_s$: Minimum number of turns to explicit acknowledgement,
• $tool_s$: Minimum number of turns to relevant tool invocation,
• $$\mathrm{TSR} = 0.25 \times \text{communicate\_info\_rate} + 0.45 \times \text{action\_rate} + 0.30 \times \text{nl\_assertion\_rate}$$0: Minimum number of turns to task achievement, measured from the shift point $$\mathrm{TSR} = 0.25 \times \text{communicate\_info\_rate} + 0.45 \times \text{action\_rate} + 0.30 \times \text{nl\_assertion\_rate}$$1.

Recovery is credited if acknowledgement occurs and the agent does not escalate to human transfer. Aggregate statistics include recovery rate and latency distributions.

The combined use of these metrics enables the discrimination of agents with high raw success but poor shift management or excessive redundancy, supporting nuanced analysis of agent behavior.

3. Dataset and Scenario Construction

AgentChangeBench encompasses 315 core task templates spanning three enterprise domains:

• Banking (50 templates): authentication, payments, disputes,
• Airline (100): booking, modification, cancellation,
• Retail (165): order tracking, returns, exchanges.

Persona-based simulation yields 1,575 core task-persona pairings, further extended to 2,835 with old/new task splits. Five personas, each defined by cooperation, expertise, and style, are:

• EASY_1: Polite, detail-oriented,
• EASY_2: Casual, distractible,
• MEDIUM_1: Business-focused, impatient,
• MEDIUM_2: Curious learner, cost-sensitive,
• HARD_1: Suspicious, demanding proof.

Each JSON task script specifies required goal shifts ($$\mathrm{TSR} = 0.25 \times \text{communicate\_info\_rate} + 0.45 \times \text{action\_rate} + 0.30 \times \text{nl\_assertion\_rate}$$2), ordered goals $$\mathrm{TSR} = 0.25 \times \text{communicate\_info\_rate} + 0.45 \times \text{action\_rate} + 0.30 \times \text{nl\_assertion\_rate}$$3, and triggers (fixed turn count, agent sub-goal completion, or explicit user shift phrases). Agents are blind to these annotations and must infer shifts from language context alone.

Domain	Templates	Example Flows
Banking	50	Auth → Statement → Dispute
Airline	100	Search → Book → Modify → Cancel
Retail	165	Track → Return → Exchange

4. Experimental Setup and Baselines

Three proprietary tool-augmented LLM agents (GPT-4o, Claude-3.7-Sonnet, Gemini-2.5-Flash) and one open-source baseline (Qwen2.5-14B-Instruct) were evaluated using a τ²-bench-based harness with goal-shift extensions. Agents interacted through an OpenAI-style function-calling API (one tool call per turn; validated schema). Each task/persona combination was run three times for proprietary models to compute variability and pass$$\mathrm{TSR} = 0.25 \times \text{communicate\_info\_rate} + 0.45 \times \text{action\_rate} + 0.30 \times \text{nl\_assertion\_rate}$$4 consistency, with pass$$\mathrm{TSR} = 0.25 \times \text{communicate\_info\_rate} + 0.45 \times \text{action\_rate} + 0.30 \times \text{nl\_assertion\_rate}$$5 serving as a static-goal accuracy baseline. All multi-dimensional metrics were computed from full dialogue transcripts.

5. Empirical Findings

Significant discrepancies emerged between traditional pass$$\mathrm{TSR} = 0.25 \times \text{communicate\_info\_rate} + 0.45 \times \text{action\_rate} + 0.30 \times \text{nl\_assertion\_rate}$$6 metrics and multi-dimensional robustness scores.

• Overall TSR by Domain:

| Domain | GPT-4o | Claude-3.7-Sonnet | Gemini-2.5 | |----------|--------|-------------------|------------| | Banking | 51.3% | 57.5% | 47.4% | | Airline | 62.2% | 65.1% | 47.0% | | Retail | 56.5% | 79.6% | 58.0% |

• Goal-Shift Sensitivity (new-only tasks):

In airline tasks, GPT-4o displays a recovery rate of 92.2% versus Gemini's 48.6%. In retail, redundancy rates are extreme—TCRR for GPT-4o is 89.1%, Gemini 66.5%, manifesting as repeated identical order lookups even after success. TUE is consistently $$\mathrm{TSR} = 0.25 \times \text{communicate\_info\_rate} + 0.45 \times \text{action\_rate} + 0.30 \times \text{nl\_assertion\_rate}$$7 across domains, with high parameter validity ($$\mathrm{TSR} = 0.25 \times \text{communicate\_info\_rate} + 0.45 \times \text{action\_rate} + 0.30 \times \text{nl\_assertion\_rate}$$8) and tool correctness in the 95–99% range, meaning variance in TUE tracks with T. TCRR varies significantly: low in airline (13–24%), moderate in banking (61–72%), and very high in retail (66–89%). Persona results indicate that medium personas have fastest recovery and success rates; the hard persona performs worst, reflecting dependence on user cooperation and statement clarity.

• Key Observations:

High agent pass$$\mathrm{TSR} = 0.25 \times \text{communicate\_info\_rate} + 0.45 \times \text{action\_rate} + 0.30 \times \text{nl\_assertion\_rate}$$9 accuracy under static conditions fails to predict reliability under shifts; e.g., Gemini’s pass$\mathrm{TUE} = 0.6\,T + 0.4\,P$0 parity with GPT-4o dissolves once adaptation is required. High TUE can mask substantial inefficiency, revealed by TCRR—especially acute in cost-sensitive domains like retail. Excessive confirmation behaviors (communication channel collapse) can reduce actual info transfer without reducing tool invocation rates.

6. Analytical Implications

Explicit measurement of adaptation latency (GSRT) and redundancy (TCRR) provides critical insight absent from simple success metrics. Two agents with similar TSRs can exhibit a 5+ turn difference in adaptation latency post-shift, with substantial implications for user experience and API cost overhead. The multi-pronged evaluation offered by AgentChangeBench enables domain-specific agent selection (e.g., prioritizing low TCRR in retail or fast GSRT in airline booking) and highlights avenues for direct model improvement, notably state-tracking to reduce redundant tool calls and prompt engineering to enhance shift detection.

A plausible implication is that agent architectures and prompting strategies must explicitly address state persistence and latent shift detection to optimize both efficiency and end-user satisfaction in realistic enterprise deployments.

7. Future Research and Extension Directions

AgentChangeBench identifies several axes for advancing evaluation and model capability:

• Integration of explicit state-tracking or memory modules to suppress redundant tool invocations,
• Design of shift-detection classifiers or prompting frameworks that can dynamically balance directive action with clarifying communication,
• Reward modeling oriented toward concise, context-sensitive communication tailored to varying user personas,
• Expansion to additional domains (e.g., healthcare, education, robotics/IoT), adversarial or deceptive personas, and tasks with latent or overlapping subgoals,
• Compatibility with more complex multi-tool interaction protocols, containerized APIs for reproducibility, and deterministic replay for regression testing.

By quantifying robustness under dynamic goals and exposing dimensions such as redundancy and latency, AgentChangeBench sets a new standard for agent evaluation, bridging the gap between static benchmarks and open-world conversational requirements (Rana et al., 20 Oct 2025).