SEATauBench: Multilingual AI Evaluation

• The paper introduces SEATauBench, a multilingual, agent-focused benchmark that adapts TauBench for low-resource Southeast Asian languages.
• It implements a two-phase localization pipeline that translates both offline artifacts and runtime tool interfaces while preserving execution semantics.
• Empirical findings show that while conversation-only localization maintains much of the English capability, full domain localization produces significant quality and robustness losses.

Searching arXiv for SEATauBench and closely related TauBench work to ground the article in papers. SEATauBench is a multilingual, agent-focused evaluation framework for Southeast Asian sovereign AI that adapts TauBench into low-resource Southeast Asian languages and localized deployment settings. Its defining objective is to test whether tool-using agents remain effective when the full agent-facing interface becomes multilingual, including user dialogue, tool specifications, policies, domain data, and tool-return text, rather than only the conversational surface. The benchmark covers Mandarin Chinese, Vietnamese, Thai, Indonesian, and Filipino, and evaluates agents under progressively localized settings ranging from an English baseline to full domain adaptation. Across these settings, the central empirical pattern is that English capabilities transfer only partially: conversation-only localization is often manageable, whereas localization of tools, policies, and domain context produces materially larger losses in both quality and robustness (Nguyen et al., 27 Jun 2026).

1. Motivation and problem setting

SEATauBench was introduced to address a gap in the evaluation of Southeast Asian sovereign AI. The paper argues that English-centric benchmarks are insufficient for regional deployment because Southeast Asian languages remain underrepresented in AI development, while useful sovereign AI requires linguistic and cultural relevance for more than 700 million people. Existing Southeast Asian benchmarks such as SEA-Exam, SEA-Bench, SEA-HELM, SEA-VL, NusaCrowd, NusaX, and NusaDialogue largely target static language understanding, reasoning, translation, safety, or multimodal tasks. They do not evaluate whether agents can perform multi-turn, tool-mediated tasks in regional languages, which is the capability required in service, commerce, telecom, airline, and travel systems (Nguyen et al., 27 Jun 2026).

The benchmark is therefore organized around a stronger notion of deployment realism. It evaluates whether an agent can converse with a user over multiple turns, follow domain policies, interpret tool schemas, call tools correctly, inspect tool outputs, and maintain consistent state under multilingual conditions. The paper’s broader claim is not merely that localization affects model quality, but that localization affects different layers of the agent stack differently. In particular, English-only assessment can overestimate readiness, because conversation translation alone does not capture failures induced by localized schemas, localized literals, localized policy text, and localized domain constraints (Nguyen et al., 27 Jun 2026).

A further conceptual point is that SEATauBench is not limited to translation of benchmark text. Its scope is the localization of what may be called the entire interaction stack: visible task descriptions, structured databases, executable interfaces, and tool-return messages. This makes it a benchmark of multilingual agent execution rather than only multilingual dialogue behavior.

2. Benchmark scope, languages, and localization architecture

The benchmark extends the underlying tool-agent-user framework into five target languages: Mandarin Chinese (zh), Vietnamese (vi), Thai (th), Indonesian (id), and Filipino (tl). The paper characterizes these languages as spanning distinct resource and script profiles: Mandarin Chinese is relatively high-resource and logographic; Vietnamese uses Latin script but is lower-resource than English; Thai is lower-resource and non-Latin; Indonesian is a regional language in a medium/low-resource setting; and Filipino is low-resource in this evaluation setup. The results repeatedly show that all languages degrade under localization, but not uniformly: Thai is often among the most degraded, Filipino also degrades strongly despite using the Latin alphabet, Chinese and Vietnamese often retain somewhat better performance, and Indonesian sometimes exhibits unusual robustness behavior (Nguyen et al., 27 Jun 2026).

The localization pipeline has two phases. In the offline translation phase, SEATauBench constructs language-specific artifacts by translating task setup text, policy and workflow documents, tool docstrings, schema literals, database leaves, and tool-return message templates. Before translation, the system masks canonical executable tokens so that IDs, status values, tool names, structural markers, and related tokens remain consistent and executable. The paper specifies four translation principles for this process: Runtime invariance, Terminological consistency, Format fidelity, and Bidirectional transparency (Nguyen et al., 27 Jun 2026).

The runtime localization phase modifies the interface exposed to the agent during execution. Tool descriptions, enum choices, examples, and tool outputs are localized. Before tool execution, localized arguments are normalized back to canonical English values; after execution, outputs are localized again; and before scoring, the final result is canonicalized back to English. This architecture is designed to preserve execution semantics while varying the language visible to the model, thereby separating semantic capability from surface-language presentation (Nguyen et al., 27 Jun 2026).

The paper presents this design as a reusable adaptation pipeline in addition to a benchmark. A plausible implication is that the framework is intended not only for measurement but also for systematic multilingual stress-testing of agent stacks that were originally designed in English.

3. Evaluation scenarios and task domains

SEATauBench evaluates agents under four progressively localized scenarios. The progression is intended to isolate different sources of multilingual difficulty rather than treating “multilinguality” as a single variable.

Setting	Localized components	Diagnostic role
S1. English Only	Nothing; everything remains English	Reference baseline
S2. L2 Interaction	User-agent conversation only	Isolates dialogue localization
S3. L2 Tool	Tool schemas in L2; conversation and domain context remain English	Isolates tool-interface localization
S4. L2 Domain	Dialogue, tool schemas, policies, task descriptions, and visible database	Full domain adaptation

In S2, the user-agent conversation is in the target language while tools, policies, databases, and task context remain English. In S3, tool schemas are translated into the target language while conversation and domain context remain English. The paper also examines mixed-language tools, including single-L2, bilingual, trilingual, quadlingual, and multilingual tool specifications. In S4, dialogue, tool schemas, policies, task descriptions, and visible database content are all localized; this is presented as the most realistic and hardest setting (Nguyen et al., 27 Jun 2026).

The task domains are Retail, Airline, and Telecom. The paper notes that SEATauBench uses updated versions of Retail and Airline, while Telecom follows the original benchmark because it was not updated in the newest release. Domain difficulty is not uniform. Retail tends to be the hardest because of policy constraints, item-level state updates, and more complex decision rules, while Airline and Telecom are often easier than Retail for some models and settings (Nguyen et al., 27 Jun 2026).

This benchmark structure distinguishes SEATauBench from multilingual benchmarks that evaluate static competencies only. It also distinguishes it from later dual-control work: $\tau^2$-Bench frames itself as a generalization of $\tau$-style single-control evaluation into settings where both agent and user can act on a shared environment, whereas SEATauBench localizes the tool-agent-user evaluation stack across languages while retaining the original interaction paradigm (Barres et al., 9 Jun 2025).

4. Metrics, protocol, and scoring methodology

SEATauBench uses the same task-success framework as the underlying benchmark: pass@1 as mean success rate over independent trials, and p3 as a robustness measure over three independent trials. The robustness metric is defined as

$p_3 = \frac{pass^3}{pass@1},$

and the paper also states that

$pass@1 = \text{pass}^1 = \mathbb{E}[r],$

where $r$ is success in a trial. In this formulation, pass@1 measures whether the model can solve tasks at all, whereas p3 measures repeatability across runs (Nguyen et al., 27 Jun 2026).

The benchmark supplements these outcome metrics with language correctness, defined as the fraction of eligible turns whose detected language matches the expected target language, and with qualitative error analysis that distinguishes critical from benign errors. Critical errors are irrecoverable or intent-breaking; benign errors are not fatal to task completion. The paper also includes detailed error tagging (Nguyen et al., 27 Jun 2026).

The experimental setup evaluates three recent agent models: GPT-5-Mini, Qwen3-235B-A22B-Instruct-2507, and Kimi-K2.5. Qwen3-235B-A22B-Inst serves as the user simulator, GPT-4.1 serves as the natural-language assertion judge, and robustness is computed with q = 3, hence p3. The experiments use default provider hyperparameters with no tuning. The absence of model-specific tuning is deliberate and is meant to reflect out-of-the-box capability in multilingual deployment (Nguyen et al., 27 Jun 2026).

The scoring design is important because the paper’s conclusions rely not only on average success but also on consistency. A model that succeeds intermittently may appear strong under pass@1 while remaining unsuitable for deployment under repeated trials.

5. Main empirical findings

The principal quantitative result is monotonic degradation as more of the environment is localized. English is consistently best; performance drops when only the conversation becomes L2; drops further when tools are localized; and is lowest under full L2-domain adaptation. The paper presents this as the core benchmark finding: the more of the benchmark environment is localized, the more agent performance degrades (Nguyen et al., 27 Jun 2026).

The English baseline already shows substantial domain and model variation. In Airline, GPT-5-mini reaches pass@1 $= 0.693$ and p3 $= 0.540$, Qwen3-235B reaches pass@1 $= 0.500$ and p3 $= 0.340$, and Kimi-K2.5 reaches pass@1 $= 0.707$ and p3 $\tau$0. In Retail, GPT-5-mini records pass@1 $\tau$1 and p3 $\tau$2, Qwen3-235B records pass@1 $\tau$3 and p3 $\tau$4, and Kimi-K2.5 records pass@1 $\tau$5 and p3 $\tau$6. In Telecom, GPT-5-mini records pass@1 $\tau$7 and p3 $\tau$8, Qwen3-235B records pass@1 $\tau$9 and p3 $p_3 = \frac{pass^3}{pass@1},$0, while Kimi-K2.5 is unusually strong with pass@1 $p_3 = \frac{pass^3}{pass@1},$1 and p3 $p_3 = \frac{pass^3}{pass@1},$2 (Nguyen et al., 27 Jun 2026).

Under L2 Interaction (S2), conversation-only localization is often survivable but not free. In Airline, GPT-5-mini ranges from pass@1 $p_3 = \frac{pass^3}{pass@1},$3 in Vietnamese to $p_3 = \frac{pass^3}{pass@1},$4 in Filipino, Qwen3-235B ranges from $p_3 = \frac{pass^3}{pass@1},$5 in Vietnamese to $p_3 = \frac{pass^3}{pass@1},$6 in Filipino, and Kimi-K2.5 remains strongest with examples including $p_3 = \frac{pass^3}{pass@1},$7 in Vietnamese and $p_3 = \frac{pass^3}{pass@1},$8 in Filipino. In Retail, GPT-5-mini falls roughly in the $p_3 = \frac{pass^3}{pass@1},$9 to $pass@1 = \text{pass}^1 = \mathbb{E}[r],$0 range, Qwen3-235B in the $pass@1 = \text{pass}^1 = \mathbb{E}[r],$1 to $pass@1 = \text{pass}^1 = \mathbb{E}[r],$2 range, and Kimi-K2.5 in the $pass@1 = \text{pass}^1 = \mathbb{E}[r],$3 to $pass@1 = \text{pass}^1 = \mathbb{E}[r],$4 range. In Telecom, GPT-5-mini ranges from $pass@1 = \text{pass}^1 = \mathbb{E}[r],$5 to $pass@1 = \text{pass}^1 = \mathbb{E}[r],$6, Qwen3-235B from $pass@1 = \text{pass}^1 = \mathbb{E}[r],$7 to $pass@1 = \text{pass}^1 = \mathbb{E}[r],$8, and Kimi-K2.5 from $pass@1 = \text{pass}^1 = \mathbb{E}[r],$9 to $r$0, with p3 around $r$1 to $r$2 (Nguyen et al., 27 Jun 2026).

Under L2 Tool (S3), localizing tool schemas introduces additional degradation. For monolingual tool specifications, GPT-5-mini reaches approximately $r$3 to $r$4 in Airline, $r$5 to $r$6 in Retail, and $r$7 to $r$8 in Telecom; Qwen3-235B reaches approximately $r$9 to $= 0.693$0 in Airline, $= 0.693$1 to $= 0.693$2 in Retail, and $= 0.693$3 to $= 0.693$4 in Telecom. In the mixed-language tool ablation, moving from English-only tools to mixed-language tools causes a modest drop: pass@1 decreases by about $= 0.693$5 on average from $= 0.693$6 to $= 0.693$7, and p3 decreases by about $= 0.693$8. The effect plateaus after a second language is added, suggesting that tool multilingualism itself is not the dominant source of failure once translation has entered the interface (Nguyen et al., 27 Jun 2026).

Under L2 Domain (S4), full localization is hardest. In Airline, GPT-5-mini records pass@1 roughly $= 0.693$9 to $= 0.540$0, Qwen3-235B roughly $= 0.540$1 to $= 0.540$2, and Kimi-K2.5 roughly $= 0.540$3 to $= 0.540$4. In Retail, GPT-5-mini can fall to $= 0.540$5 in Thai, Qwen3-235B can fall to $= 0.540$6 in Thai, and Kimi-K2.5 also degrades though it remains higher. In Telecom, GPT-5-mini ranges from $= 0.540$7 to $= 0.540$8, Qwen3-235B from $= 0.540$9 to $= 0.500$0, and Kimi-K2.5 from $= 0.500$1 to $= 0.500$2 (Nguyen et al., 27 Jun 2026).

The paper’s interpretation is that surface multilingual ability transfers more readily than deeper multilingual execution. Conversation-only localization often preserves much of the English competence, but localized tools, localized policies, localized schemas, and localized domain state expose brittle dependencies on English-formatted execution environments.

6. Diagnostics, interpretation, and broader significance

SEATauBench includes several analyses intended to explain why localization harms performance. One line of evidence comes from correlation analysis. The paper reports that pass@1 correlations between English and Southeast Asian languages are often high, above $= 0.500$3 in many cases, but p3 correlations vary much more. The highest p3 correlation is in Chinese at $= 0.500$4, and the lowest is in Thai at $= 0.500$5. Filipino shows the strongest cross-language correlation profile, with the lowest pass@1 correlation still at $= 0.500$6 and the lowest p3 correlation at $= 0.500$7. The authors therefore conclude that English performance is a decent proxy for one-shot success but not for robustness in Southeast Asian settings (Nguyen et al., 27 Jun 2026).

A second line of evidence concerns error analysis. Using an LLM-as-a-judge framework, the paper finds that critical errors rise in L2 settings, Thai often exhibits the most severe degradation, user simulation becomes less stable in L2 settings especially for Filipino, and full domain localization increases critical errors further. This shifts the interpretation away from simple translation mistakes and toward broader multilingual instability in agent execution (Nguyen et al., 27 Jun 2026).

A third analysis studies language drift. In L2 Tool, language correctness is nearly perfect at around $= 0.500$8; in L2 Domain, it remains high at around $= 0.500$9 to $= 0.340$0; and in L2 Interaction, it drops to around $= 0.340$1 to $= 0.340$2. Off-target text accumulates over turns, most drift is toward English, Filipino has the highest English code-switching, and drift is strongest in L2 Interaction. However, language correctness explains very little of task performance, with only $= 0.340$3 for pass3 versus language correctness. The paper’s conclusion is that language drift is real but is not the main driver of performance loss (Nguyen et al., 27 Jun 2026).

These findings have direct implications for multilingual evaluation and sovereign AI. For multilingual agents, the benchmark shows that high English scores do not guarantee reliable multilingual deployment. For sovereign AI, it shows that sovereignty requires not just model ownership but linguistic inclusion, region-specific evaluation, and tool-mediated systems that function in local languages. For evaluation methodology, it shows that robust multilingual benchmarks must preserve execution semantics, localize visible interface content, canonicalize scoring, and test multiple degrees of localization rather than a single translated setting (Nguyen et al., 27 Jun 2026).

The paper also notes several limitations. Only five languages are covered; the benchmark remains tied to the TauBench or “72-Bench” paradigm described in the paper; Malay, Lao, and Cambodian were not included because of annotator limitations; and generalization beyond Southeast Asia or beyond this class of task structures is uncertain. Within those limits, SEATauBench establishes a diagnostic benchmark and adaptation pipeline for measuring how multilingual agent competence changes as localization moves from the conversational surface to the full tool-using domain environment (Nguyen et al., 27 Jun 2026).