SAGE: A Service Agent Graph-guided Evaluation Benchmark

Abstract: The development of LLMs has catalyzed automation in customer service, yet benchmarking their performance remains challenging. Existing benchmarks predominantly rely on static paradigms and single-dimensional metrics, failing to account for diverse user behaviors or the strict adherence to structured Standard Operating Procedures (SOPs) required in real-world deployments. To bridge this gap, we propose SAGE (Service Agent Graph-guided Evaluation), a universal multi-agent benchmark for automated, dual-axis assessment. SAGE formalizes unstructured SOPs into Dynamic Dialogue Graphs, enabling precise verification of logical compliance and comprehensive path coverage. We introduce an Adversarial Intent Taxonomy and a modular Extension Mechanism, enabling low-cost deployment across domains and facilitating automated dialogue data synthesis. Evaluation is conducted via a framework where Judge Agents and a Rule Engine analyze interactions between User and Service Agents to generate deterministic ground truth. Extensive experiments on 27 LLMs across 6 industrial scenarios reveal a significant Execution Gap'' where models accurately classify intents but fail to derive correct subsequent actions. We also observeEmpathy Resilience'', a phenomenon where models maintain polite conversational facades despite underlying logical failures under high adversarial intensity. Code and resources are available at https://anonymous.4open.science/r/SAGE-Bench-4CD3/.

• The paper introduces a novel evaluation framework that formalizes SOPs as directed dialogue graphs for dynamic, multi-turn service agent assessment.
• It employs dual-axis metrics combining logical compliance and chat quality to pinpoint execution gaps in LLMs under adversarial conditions.
• Extensive experiments across diverse domains reveal performance trade-offs between linguistic fluency and accurate procedural adherence.

SAGE: A Service Agent Graph-guided Evaluation Benchmark — Technical Essay

Motivation and Problem Statement

The automated customer service domain increasingly relies on LLMs for multi-turn, high-fidelity dialog management. However, existing evaluation paradigms are fundamentally limited: they are typically static, single-dimensional, and poorly aligned with the multifaceted requirements of real-world deployments, especially around strict adherence to domain-specific Standard Operating Procedures (SOPs). These deficiencies impede fine-grained diagnosis of procedural versus linguistic failures and cloud agentic performance under adversarial user behaviors. SAGE (Service Agent Graph-guided Evaluation) directly addresses these gaps by formalizing SOP compliance as a graph traversal task, introducing adversarially-tuned user simulation, and delivering a dual-axis assessment for both logic and conversational dimensions.

Benchmark Framework Overview

SAGE introduces a rigorous three-stage pipeline for service agent evaluation: (1) Dynamic Multi-Turn Dialogue Graph Modeling, (2) Graph-Guided Multi-Agent Evaluation, and (3) a Scenario Extension Mechanism.

The benchmark formalizes unstructured SOPs into annotated, directed dialogue graphs. Nodes denote dialogue states such as decision or action points; edges represent logical transitions, explicitly parameterized by classification fields and environmental variables. This structure disambiguates procedural requirements and serves as the canonical environment for both evaluated agents and deterministic ground-truth rollouts.

Figure 1: Overview of the SAGE evaluation framework detailing graph formalization, agent interaction, and dual-axis assessment modules.

Multi-turn dialogues are synthesized via the real-time interaction between the User Agent (simulating diverse, adversarially-controllable personas) and the Service Agent (LLM under test). The Service Agent must output structured JSON comprising field classifications, explicit path traversal, action execution, and a contextually-appropriate chat utterance. This complexity ensures the model’s policy is not only linguistically plausible but also procedurally sound per business logic.

Evaluation leverages an ensemble of LLM Judge Agents for semantic labeling and conversation quality scoring — mitigating single-model bias as shown empirically (cf. main paper heatmaps) — followed by deterministic logic checking against the SOP graph via a Rule Engine.

Methodology and Technical Design

SOP Formalization and Path Coverage

Each industrial SOP is parsed and transformed into a computable directed graph $G = (V, E)$, where $V$ includes all decision and action points, and $E$ encodes transitions contingent on classification fields and explicit system variables. This abstraction eliminates ambiguity in compliance assessment.

Path coverage is enforced by enumerating all valid graph paths via a hybrid of intent-balanced user simulations and targeted edge-case construction, ensuring that agents are stress-tested across the entire SOP space and not just nominal flows.

Figure 2: Service Agent SOP Example (Telecom Scenario), illustrating conversion of textual SOP to an explicit decision-action graph.

Multi-Agent Dual-Axis Evaluation

SAGE’s evaluation is dual-axis:

• Logical Compliance (80% weight in final score): Decomposed into field classification accuracy, path correctness, and action correctness. This separation exposes the frequent failure mode (“Execution Gap”) where models correctly classify intents yet fail in resultant action selection due to non-trivial conditional dependencies.
• Chat Quality (20% weight): Five sub-dimensions (Linguistic Quality, Anthropomorphism, Content Utility, User Satisfaction, Instruction Compliance) measured by Judge Agent ensemble, ensuring robustness against egocentric or systematic model bias.

This procedure allows SAGE to localize defects to specific reasoning or dialogic weaknesses. The final evaluation is performed at multiple conversation depths (turns 1, 5, 10, 15, final), surfacing both cold-start and long-context limitations.

Figure 3: Logic performance gap analysis demonstrating classification vs action correctness discrepancies across six scenarios (“Execution Gap”).

Adversarial User Simulation

SAGE operationalizes user adversariality via a taxonomy spanning zero (fully cooperative), weak, and strong (directly confrontational or cognitively demanding) adversarial intents and personas. The User Agent is prompt-driven, modulating both goal alignment and communication traits to create high-fidelity stress tests for the Service Agent.

This compositional adversariality is critical for exposing models’ logical fragility and error escalation under real-world hostile conditions. Notably, SAGE analyzes how models maintain chat quality (so-called “Empathy Resilience”) despite pronounced logical failures as adversarial load increases.

Figure 4: Correlation analysis of Chat Quality composite score versus its sub-dimensions, showing both overall linearity and outlier model-specific capability tradeoffs.

Experimental Results and Analytical Insights

Cross-Model and Cross-Scenario Evaluation

SAGE benchmarks 27 LLMs (open and closed-source; 3B–1T parameters) across six complex customer service domains (e.g., e-commerce refund, airline, logistics, telecom, property service, online education).

Closed-source models (Claude, Gemini series, GPT-4) top the charts in logical compliance and overall assessment, though DeepSeek-V3.2 (open-source, 671B) nearly matches leaderboard performance — evidencing rapid open model catchup. MiniMax-M2.1 shows maximal Chat Quality, highlighting agent-specific optimization trade-offs between empathic dialog and SOP rigor.

The “Execution Gap” (classification accuracy vs action correctness) is strongly scenario-dependent and most pronounced in high-branching scenarios (e.g., e-commerce refund: 88%→29.5%). In more linear settings, path accuracy is inflated as models partially fulfill early branch conditions yet miss critical latter steps.

Figure 5: Overall Average Score (OA) heatmap, visualizing differential scoring behaviors across model-judge pairs and demonstrating variance due to single-judge bias.

Multi-Turn Robustness and Contextual Degradation

A distinctive “Inverted-U” performance trajectory emerges across dialogue turns: logic and OA scores improve up to Turn 5 (information gain phase), then deteriorate sharply by Turn 15 (context fatigue and “lost-in-the-middle” errors). Notably, chat quality degradation lags logic breakdown, underlining the necessity of dual-axis assessment.

Adversarial Intensity Analysis

Increasing adversarial scenario intensity leads to steeper logic drops (e.g., LogisticsDelivery: 84.0→54.5), higher JSON formatting failure, and wider score variance between models (discriminative filter effect). However, conversational scores remain stable — agents maintain polite, de-escalating language (“Empathy Resilience”) despite breakdowns in actual SOP policy reasoning.

Subdimension Correlation and Model Profiling

Regression analysis across chat quality subdimensions evidences strong linearity ($R^2 > 0.9$) — validating SAGE’s composite scoring. Residual analysis pinpoints model idiosyncrasies: some (e.g., Doubao-Seed-1.8, DeepSeek-R1) display anomalously high linguistic scores but underperform on instruction adherence, indicative of alignment deficiencies not apparent via aggregate scores.

Practical Impact, Limitations, and Future Directions

SAGE’s design has direct industrial applicability: the modular scenario extension mechanism supports rapid deployment to new domains, and automated data synthesis enables dynamic, high-fidelity dialogue corpora generation for fine-tuning or safety validation. The multi-agent, ensemble-judge protocol provides a more objective, bias-resilient assessment than single-model LLM-judge frameworks.

Practically, SAGE enables:

• Reliable agentic capability comparison across diverse, high-complexity real-world scenarios.
• Fine-grained diagnostics of procedural, instruction-following, and rhetorical (chat) strengths/weaknesses.
• Stress-testing of context retention and adversarial resilience, informing both deployment and research prioritization.

Theoretically, the “Execution Gap” phenomenon formalized by SAGE will likely inform future work on hybrid policy architectures able to bridge intent classification and conditional action planning. As LLM architectures evolve toward longer context and explicit reasoning modules, SAGE’s dual-axis, multi-turn, adversarially-tunable setup will serve as a critical empirical reference.

Future extensions encompass: supporting nested graph structures (for recursive workflows), multimodal inputs/outputs, and fine-grained real-time error annotation for RL-finetuning and alignment.

Conclusion

SAGE advances the methodological rigor of LLM service agent evaluation by unifying dynamic procedural logic modeling, robust dual-axis (logic/chat) metrication, and scalable scenario extension in a single framework. Extensive results elucidate current model strengths, critical pathologies (particularly the logic-action execution gap), and the intricate dissociation between conversational fluency and true SOP adherence under complex, adversarially-tuned customer service conditions. The SAGE framework establishes both a practical industry benchmark and a fertile platform for future agentic reasoning research.