Failure-Centered Runtime Evaluation for Deployed Trilingual Public-Space Agents

Abstract: This paper presents PSA-Eval, a failure-centered runtime evaluation framework for deployed trilingual public-space agents. The central claim is that, when the evaluation object shifts from a static input-output mapping to a runtime system, the basic unit of analysis should shift from score to failure. PSA-Eval extends the conventional chain Question -> Answer -> Score -> End into Question -> Batch -> Run -> Score -> Failure Case -> Repair -> Regression Batch, making failures traceable, reviewable, repairable, and regression-testable. The framework uses trilingual equivalent inputs as controlled probes for observing group-level cross-language policy drift. We conduct a pilot study on a real trilingual digital front-desk system deployed in the lobby of an international financial institution. The pilot uses a simplified single-foundation-model setting (MA = MB), so the observed drift should not be interpreted as an A/B foundation-model difference. The study contains 81 samples organized into 27 trilingual equivalent question groups. Although the system achieves an average score of 23.15/24, 14 groups show non-zero cross-language score drift, 5 groups show drift of at least 3 points, and the maximum drift reaches 9 points. These results provide initial evidence that failure-centered runtime evaluation can expose structured deployment signals hidden by aggregate scoring.

• The paper introduces PSA-Eval—a failure-centric framework that shifts evaluation from pointwise scores to trajectory-level failure analysis in trilingual public-space agents.
• It integrates runtime traceability, batch context, and human review to assess and repair policy, service, and broadcast boundary violations.
• A pilot study with 81 samples revealed significant cross-language score drift and calibration issues, highlighting the need for group-level failure analysis.

Failure-Centered Runtime Evaluation for Deployed Trilingual Public-Space Agents

Motivation and Problem Framing

The paper introduces PSA-Eval, a runtime evaluation framework specifically structured for trilingual public-space agents, i.e., deployed LLM-driven entities whose outputs are institutionally visible and potentially subject to broadcast via screens, voice, or digital-human interfaces. The shift from private or offline chat systems to such public deployments necessitates a fundamental change in evaluation paradigms. The primary claim is that, for runtime systems, the basic unit of evaluation must shift from pointwise score to trajectory-level failure. Failures are defined not merely as anomalous outputs, but as events in which the agent enters an unacceptable state under specific deployment conditions—crossing policy, service, or broadcast boundaries.

Traditional static evaluation pipelines, mapping input to output and assigning aggregate scores, are inadequate for nuanced runtime phenomena such as cross-language policy drift, boundary violation recurrence across versions, and regression after repair. PSA-Eval extends the evaluation chain to integrate batch context, runtime traceability, failure persistence, human review, and regression batches.

Formalization of Evaluation Objects and Boundaries

The evaluation object in PSA-Eval is a deployed trilingual agent parameterized by language-specific models, a unified policy layer, templates, gateways, and versioning. PSA-Eval operationalizes evaluation via trilingual equivalent inputs—Mandarin, Cantonese, and English variants constructed to probe institutional boundaries under matched intent and risk triggers, rather than simple translation consistency.

Three boundary types are formally delineated:

• Policy boundary: Institutional position and public issues
• Service boundary: Authorized commitments and guidance scope
• Broadcast boundary: Suitability for public content distribution

Failures are attributed according to these boundaries and tracked as structured objects that persist across evaluation, repair, and regression phases.

PSA-Eval Framework and Process

PSA-Eval structures evaluation around a recursive loop:

• Question → Batch → Run → Score → Failure Case → Repair → Regression Batch

This loop is designed to maintain the full context of input provenance, batch configuration, run-specific outputs, scoring and risk assignment, human review outcomes, and regression testing. Figure 1 illustrates the distinction from conventional static evaluation pipelines, emphasizing batch traceability, group-level analysis, and closed-loop governance.

Figure 1: PSA-Eval runtime evaluation loop highlighting batch context, human review entrypoints, and failure-case persistence across evaluation and regression cycles.

The Auto-Judge module provides multi-dimensional scoring and triage, but is explicitly not the final authority in high-risk or boundary-sensitive cases. Human reviewers evaluate samples selected by risk level, score drift, or explicit boundary violations. Figure 2 details the lifecycle of a failure case, documenting all salient metadata for traceability and persistent governance.

Figure 2: Failure case as a traceable evaluation object with question, language, topic, boundary, batch, configuration, Auto-Judge score and risk, human review, repair, and regression fields.

Pilot Study: Quantitative Insights

A pilot is conducted in the lobby of an international financial institution, covering 81 samples across 27 trilingual question groups. The deployment setting uses a single foundation model for all languages ($M_A = M_B$), thereby controlling for model variation and isolating policy drift attributable to language paths.

Overall average score is 23.15/24, but local structure reveals 14 groups with non-zero cross-language score drift, 5 groups with drift $\geq 3$, and maximum drift of 9. Low-scoring samples (score $<20$) are sometimes categorized as 'excellent' by the Auto-Judge, exposing calibration issues. High-drift groups cluster around boundary-sensitive topics and charged input intensity. Figure 3 visualizes aggregate performance and risk distributions.

Figure 3: Pilot overview showing sample counts, average scores by language and intensity, and risk-level distribution per sample.

Figure 4 displays the distribution and concentration of score drift across trilingual groups, substantiating the claim that failures are structurally clustered.

Figure 4: Distribution of trilingual group score drift showing concentration in boundary-sensitive question groups and top drift clusters.

Moreover, the D7 cross-language consistency dimension (per-sample scoring) saturates, failing to detect true cross-language boundary instability. Figure 5 shows calibration bias between low scores and risk levels, as well as this D7 anomaly, underscoring the necessity of group-level joint judging rather than per-sample isolation.

Figure 5: Auto-Judge calibration bias and D7 saturation, demonstrating risk misclassification and inability to detect group-level drift.

Contradictory Claims and Implications

A bold claim in the paper is the non-substitutability of failure-centric analysis versus multidimensional score aggregation. PSA-Eval asserts that group-level drift, socially consequential failures, and regression persistence cannot be expressed solely via scalar or multi-dimensional scores. This claim is empirically supported by the pilot’s demonstration that high aggregate scores obscure local failures and policy drift.

Practically, PSA-Eval is essential for institutional deployments where language inconsistency can constitute reputational, legal, or social risk. Theoretical implications extend to redefining the evaluation unit in interactive, multi-agent, or multilingual governance settings. PSA-Eval’s structured loop supports ongoing governance and repair rather than one-time leaderboard generation, aligning future evaluation with software engineering best practices for deployed systems.

Future Directions and Theoretical Expansion

The paper proposes several testable predictions:

• Drift concentration will persist in boundary-sensitive topics as question banks scale.
• Unified policy layers in multi-model settings should empirically reduce cross-language drift (requiring with/without ablations).
• Human review will correct Auto-Judge misclassification on high-risk cases.

Validation requires scaling question banks ($\geq 500$ groups), introducing joint trilingual judging, and quantifying closed-loop repair gains. The explicit engineering protocol (Mark → Patch → Regress → Close) is not yet empirically validated, and agreement metrics between Auto-Judge and human review remain to be quantified.

PSA-Eval’s design is generalizable to multilingual agents in other public-space contexts (government halls, hospitals, airports), necessitating scenario-specific boundary rules and expansion of regression objects.

Conclusion

PSA-Eval represents a rigorously formalized shift in deployed-agent evaluation, emphasizing persistent, reviewable, and regression-testable failure cases as the fundamental evaluation unit. Aggregate scores are insufficient for boundary robustness or cross-language policy consistency. The pilot study demonstrates that failure-centered runtime evaluation exposes structurally significant deployment signals that static pipelines occlude, providing a robust foundation for practical governance and theoretical model expansion in multilingual, public-space environments. Future work must validate closed-loop gains, operationalize group-level human review, and expand evaluation artifacts for broader reproducibility.