SEATauBench: Adapting Tool-Agent-User Evaluation Into Low-Resource Southeast Asian Languages

Abstract: While AI development and evaluation for Southeast Asia (SEA) has grown rapidly, agent capabilities in regional languages are still poorly understood despite its importance to sovereign AI. To fill this gap, we introduce SEATauBench, the first agent-focused evaluation framework for SEA sovereign AI. SeaTau adapts TauBench to five languages -- Mandarin, Vietnamese, Thai, Indonesian, and Filipino -- and evaluates agents across progressively localized settings that vary the language of user-agent interaction, tool specifications, and task domains. Across three recent models, we find that English agent capabilities transfer reasonably well when only the conversation language changes, but quality and robustness degrade sharply as more task contexts are localized, with the largest losses in full domain adaptation. We also the limits of English-only agent assessment for measuring agent capabilities in SEA languages. More broadly, SeaTau provides a diagnostic benchmark and reusable adaptation pipeline for building reliable multilingual agents for linguistically diverse regions. Data and code can be accessed at github.com/SEACrowd/SEATauBench.

• The paper introduces SEATauBench, a novel agent-centric benchmark designed to assess multilingual AI performance in underrepresented Southeast Asian languages.
• It presents a two-phase adaptation pipeline that combines LLM-driven offline translation with runtime localization to ensure evaluation robustness across tool-agent-user interactions.
• Experimental results indicate notable performance drops during full localization and highlight divergent trends between accuracy and robustness metrics, emphasizing challenges for sovereign AI deployment.

SEATauBench: Agent-Oriented Multilingual Evaluation for Southeast Asian Sovereign AI

Motivation and Problem Statement

Contemporary AI development in Southeast Asia (SEA) is confronted by profound linguistic diversity, with over 700 million individuals speaking languages that are often underrepresented in mainstream AI benchmarks. The advent of Sovereign AI—systems that embody both technological independence and cultural/linguistic alignment—necessitates robust assessment and adaptation pipelines capable of supporting these languages. Existing benchmarks in SEA primarily emphasize static, English-centric evaluation or focus exclusively on language understanding, reasoning, safety, and multimodal context. However, there remains a substantial gap in evaluating agentic systems using localized tool-driven, multi-turn tasks that more faithfully reflect deployment environments in service, commerce, and travel. SEATauBench directly addresses this deficit by offering the first agent-focused, multilingual benchmark for SEA languages, facilitating rigorous, scenario-driven evaluation of interactive agents across Mandarin, Vietnamese, Thai, Indonesian, and Filipino.

Framework and Methodology

SEATauBench builds upon the established T2-Bench/72-Bench agent evaluation paradigm and introduces a two-phase adaptation pipeline: offline translation of evaluation artifacts and runtime localization for simulation.

• Offline Translation: Leveraging LLM-driven translation with targeted human validation, the pipeline systematically extracts and translates natural-language spans from task definitions, domain policies, tool docstrings, schemas, and database entries. Canonical runtime tokens (e.g., IDs, enum literals, tool names) are masked to ensure both executional integrity and terminological consistency across languages. The translation pipeline establishes a glossary guaranteeing consistent rendering of interface literals, with artifacts versioned and checksummed for reproducibility.
• Runtime Localization: At inference, tool schemas and user/agent dialogues are localized per scenario, and all agent-facing payloads are appropriately mapped between canonical English and L2 forms. This ensures metrics are directly comparable between languages while minimizing spurious localization-induced failures (e.g., untranslated literals triggering lookup exceptions).

Four evaluation scenarios are instantiated to gradate localization difficulty: (1) English-only baseline, (2) localized agent-user conversation (L2. Interaction), (3) localized tool schemas (L2. Tool), and (4) fully localized domain (L2. Domain), progressively challenging the model's robustness and competence.

Experimental Evaluation and Key Findings

Experiments span the retail, airline, and telecom domains using three prominent agentic LLMs—GPT-5-Mini4, Qwen3-235B-A22B-Instruct-2507, and Kimi-K2.5. Metrics comprise both one-shot success rate (pass@1) and a newly proposed robustness score ($p_3$), reflecting the consistency of successful completions across repeated trials.

Major empirical observations:

• Agentic Degradation with Localized Contexts: English-to-L2 transfer is reasonable when only response language changes (~0.65 pass@1), but severe degradation manifests as tool interfaces and domain context localize ($\Delta$pass@1 often $>0.2$). Full domain adaptation (L2. Domain) precipitates the largest drops, especially in low-resource and non-Latin script languages such as Thai (pass@1 ≈ 0.4).
• Divergence of Quality and Robustness: Robustness and accuracy are not consistently correlated. Certain language/model pairs show that while average success declines, successful completions exhibit increased internal consistency, i.e., $p_3$ stays stable even as pass@1 drops. This highlights error modes distinct from pure language understanding (e.g., policy following, tool argument normalization).
• Heterogeneous Model Sensitivity: Kimi-K2.5 generally achieves the highest pass@1 in high-resource languages but does not uniformly dominate across SEA languages or under robustness metric. Qwen3-235B provides superior robustness in some L2 contexts even when its accuracy is lower, and performance disparities widen in more challenging domains (e.g., retail).
• Simulated User and Code-Switching Effects: Agent and user simulator performance both degrade in SEA languages, with critical error rates exceeding 40% of trials in L2. Domain for the hardest settings. Language correctness (i.e., remaining in the target L2 throughout a turn) has negligible explanatory power for task success ($R^2=0.014$ for pass3), indicating that syntactic code-switching is not the primary bottleneck—rather, semantic misalignment at the tool/policy level and error propagation dominate.
• Multilingual Tool Specifications: Providing tool documentation/interfaces in mixed languages yields only a modest decrement in task performance, with pass@1 dropping by ≈0.13 when moving from English-only to five-language tool mixes. This suggests relative agentic tool-use stability in the face of interface diversity.

Theoretical and Practical Implications

SEATauBench exposes a critical deficiency in English-centric agentic evaluation as a proxy for true multilingual agent reliability. English-only agent scores overestimate Southeast Asian generalization, particularly under full localization, thereby risking premature deployment and misplaced user trust in sovereign application domains. The framework sets a new bar for evaluating task-oriented, tool-using conversational agents in linguistically diverse, real-world environments. It is especially significant for sovereign AI initiatives prioritizing autonomy and cultural alignment.

On the methodological front, SEATauBench’s pipeline offers a reproducible, non-trivial adaptation process suited for any benchmark combining user-agent-tool evaluation. The findings motivate the need for multilingual pretraining/finetuning strategies, robust cross-lingual tool-use policies, and error diagnosis focusing on executional and policy-following behaviors rather than surface fluency alone.

Future Research Opportunities

Future work should extend SEATauBench coverage to additional low-resource SEA languages and other modalities (e.g., voice), integrate fully human-evaluated user-agent interactions, and support hard policy enforcement and error type annotation at scale. Rapid advances in multilingual LLM pretraining, cross-lingual retrieval-augmented generation, and tool-centric adaptation methods could leverage SEATauBench as a critical testbed for measuring practical sovereignty and reliability. Optimizing for robustness under real-world domain shifts, especially for service deployments in global South and non-Latin scripts, is a priority.

Conclusion

SEATauBench constitutes a foundational diagnostic resource for sustainable sovereign AI in Southeast Asia, rigorously quantifying the agentic capability gap between English-centric evaluation and actual multilingual deployment. Its scenario-based evaluation, structured adaptation pipeline, and extensive empirical study provide an essential framework for the rapid advancement and reliable assessment of sovereign, multilingual agentic AI.

---

Reference:

"SEATauBench: Adapting Tool-Agent-User Evaluation Into Low-Resource Southeast Asian Languages" (2606.28715)