Ticket-Bench: Multilingual Soccer Ticket Benchmark
- Ticket-Bench is a multilingual benchmark designed to assess LLM agents on culturally localized soccer ticket purchasing tasks using realistic function-calling interactions.
- It emphasizes regionalization by incorporating localized teams, cities, and user identities across six languages rather than relying on simple translation.
- Empirical results show that larger, reasoning-oriented models outperform others in pass-hat-3 scores, though language-specific performance disparities remain.
Ticket-Bench is a multilingual, regionalized benchmark for evaluating LLM agents on realistic function-calling tasks in the domain of soccer ticket purchasing. It was introduced to address a specific gap in agent evaluation: many existing function-calling benchmarks are English-only, culturally neutral, or built through straightforward translation of monolingual datasets, whereas deployed agents must operate across languages, local entities, and culturally specific references. Ticket-Bench therefore emphasizes localization rather than mere translation, interactive multi-step tool use, and programmatic end-to-end evaluation based on final environment state rather than an LLM judge (Almeida et al., 17 Sep 2025).
1. Motivation and benchmark scope
Ticket-Bench is framed as a response to the mismatch between current agent benchmarks and real multilingual deployment conditions. The benchmark is designed to measure whether a model can correctly interpret a user request in different languages and produce the right sequence of tool calls to complete a task in a culturally grounded environment. Its stated purpose is not generic language understanding, but multilingual function-calling robustness under localized conditions (Almeida et al., 17 Sep 2025).
The benchmark adopts a single domain, purchasing soccer tickets, because that domain supports realistic, structured agent behavior while remaining comparable across regions. Soccer is globally familiar but locally instantiated: leagues, clubs, stadiums, and cities differ by country, while the underlying task logic remains comparable across languages. This lets the benchmark hold task structure aligned while varying the local entities that the agent must understand and manipulate. The paper argues that such alignment permits cleaner attribution of performance differences to language-specific agent ability rather than to unequal underlying task difficulty (Almeida et al., 17 Sep 2025).
A central design claim is that multilingual agent evaluation should be culturally grounded, not just translated. Ticket-Bench therefore localizes user identities, teams, cities, schedules, function names, and function descriptions. This directly rejects the assumption that multilingual benchmarking can be adequately approximated by surface translation alone. The paper treats that assumption as a source of artifact and reduced realism rather than as a neutral simplification (Almeida et al., 17 Sep 2025).
2. Regionalization across languages and leagues
Ticket-Bench covers six languages: Portuguese, English, Spanish, German, Italian, and French. Each language is paired with a major national league, and the localized setting is constructed from official league teams and their home cities. User identities are also localized through country-appropriate names, preferred teams, and account balances. The text interface shown to the model, including question templates, function names, and function descriptions, is manually translated rather than automatically generated (Almeida et al., 17 Sep 2025).
| Language | Regional setting |
|---|---|
| Portuguese | Brasileirão / Brazil |
| English | Premier League / United Kingdom |
| Spanish | La Liga / Spain |
| German | Bundesliga / Germany |
| Italian | Serie A / Italy |
| French | Ligue 1 / France |
Regionalization operates at several levels simultaneously. Team names come from official rosters of the corresponding league, and their home cities determine the localized city set. Each language also has 20 users, with no two users sharing the same preferred team. The benchmark further states that constraints are kept consistent across languages and schedules are synchronized, so that the multilingual comparison is not confounded by different local scenario difficulty (Almeida et al., 17 Sep 2025).
This design suggests a specific notion of multilingual comparability. The benchmark does not seek identity of surface form or entity inventory across languages; instead, it preserves semantic template alignment while changing the entities appropriately by region. A plausible implication is that Ticket-Bench is intended to evaluate whether an agent can generalize task logic across linguistic and cultural variation without collapsing that variation into a translation artifact.
3. Agent task, tools, and scenario templates
The task is formulated as an interactive tool-use episode initiated by a single user query. A model receives a request in one language and must interact with a fixed set of functions to identify the correct game and buy the ticket, or correctly determine that no valid booking exists. The setup is multi-step rather than single-shot: the model may need to gather user information, search games, inspect historical rankings, compute weekdays from dates, and then decide whether a purchase should occur (Almeida et al., 17 Sep 2025).
Ticket-Bench contains 17 question templates, instantiated from five constraint categories: semester, weekday, price, location, and leaderboard. The templates range from simple affordability requests to compound constraints involving city, date, opponent quality, weekday restrictions, and semester boundaries. Because these constraints must often be resolved by combining several tool outputs, the benchmark evaluates not only function-name prediction but also argument selection and sequential reasoning over the structured environment (Almeida et al., 17 Sep 2025).
| Function | Role | Returns or effect |
|---|---|---|
| Get User Info | Retrieve localized user profile | Name, preferred team, account balance |
| List Games | Search the schedule | Paginated games with filters and ordering |
| Buy Game Ticket | Execute purchase | Updates balance and booking state |
| Get Leaderboard | Query prior standings | Points, wins, draws, losses, goals |
| Get Weekday from Date | Convert date to weekday | Day of week from YYYY-MM-DD |
The List Games function is explicitly paginated, with at most 10 entries per page, and supports optional filters such as location and team together with ordering by date or price. Each game includes an identifier, teams, city, stadium, ticket price, and scheduled date. This matters because the agent must sometimes reason over partial retrieval, ordering, and affordability jointly rather than over a prefiltered gold candidate set (Almeida et al., 17 Sep 2025).
The benchmark also includes unsatisfiable cases. Fifteen percent of questions are deliberately constructed so that no game in the schedule satisfies the constraints. This tests whether a model can refrain from issuing a purchase when no valid solution exists, rather than treating tool use as a forced-booking task (Almeida et al., 17 Sep 2025).
4. Synthetic environment construction and evaluation protocol
The environment is mostly synthetic but grounded in real entities. It has three main components: users, schedules, and leaderboards. For each language, the authors generate 20 users, each with a culturally appropriate name, a favorite team, and a virtual balance. Schedules are synthetic full league seasons. The paper states that it simulates two schedule types per language: one in which each of the 20 teams plays every other team once, and one in which they play twice, with the latter yielding 380 matches. Leaderboards are also synthetic, with match goals assigned so that aggregate team statistics resemble realistic league outcomes (Almeida et al., 17 Sep 2025).
Each of the 17 templates is instantiated 10 times by varying user profiles, schedules, and constraint values. This yields 170 queries per language and 1,020 queries in total. The benchmark is therefore a fixed evaluation suite rather than a train/validation/test dataset split for supervised learning. The paper does not report train/validation/test splits, significance tests, confidence intervals, or formal statistical hypothesis comparisons (Almeida et al., 17 Sep 2025).
Evaluation is fully programmatic and LLM-free. A run is correct if the final environment state matches the annotated expected state: all expected games are booked and no unexpected games are booked. This criterion is stricter than judging whether an intermediate tool call was plausible, because it evaluates complete task execution rather than local action quality (Almeida et al., 17 Sep 2025).
The main metric is a consistency-aware pass@k variant written as pass-hat-3. For query , if of repeated executions are correct, then
and the benchmark-level score is
Here is the number of queries, is the number of repeated executions per query, and controls how strongly consistency is rewarded. In the reported experiments, and for budget reasons. The paper notes that running GPT-5 three times over the full benchmark costs about $70 USD (Almeida et al., 17 Sep 2025).
Beyond pass-hat-3, the analysis reports per-language results, overall multilingual averages, standard deviation across languages as a measure of cross-lingual consistency, and per-language deviations from each model’s mean. This indicates that the benchmark is intended not only to rank models but also to characterize asymmetry within model families (Almeida et al., 17 Sep 2025).
5. Model suite and empirical results
The evaluated model suite spans commercial, open-source, and reasoning-oriented systems. Commercial or proprietary models include GPT-5, GPT-5 Mini, GPT-5 Nano, GPT-4.1, GPT-4.1 Mini, GPT-4.1 Nano, Gemini-Pro 2.5, and Gemini-Flash 2.5. Open or openly available families include Qwen3-235B-A22B, Qwen3-32B, Qwen3-14B, Qwen3-30B-A3B, Qwen3-8B, Qwen3-4B, Qwen2.5 Instruct variants, GPT-OSS variants, xLAM variants, and Sabiá-3.1. The paper groups the strongest systems as reasoning models and specifically names GPT-5, Qwen3-235B, GPT-5 Mini, GPT-5 Nano, and GPT-OSS-120B among the best performers (Almeida et al., 17 Sep 2025).
The highest overall pass-hat-3 scores are reported as follows: GPT-5 at 0.91; GPT-5 Mini at 0.89; Qwen3-235B-A22B at 0.88; GPT-5 Nano at 0.75; GPT-OSS-120B at 0.72; and GPT-4.1 at 0.70 (Almeida et al., 17 Sep 2025). GPT-5 is also the most balanced of the listed systems, though even the strongest models retain nontrivial cross-language variation. Its per-language scores are en 0.92, es 0.93, fr 0.92, it 0.92, de 0.92, and pt 0.87. GPT-5 Mini scores en 0.91, es 0.91, fr 0.89, it 0.90, de 0.89, and pt 0.86. Qwen3-235B-A22B scores en 0.88, es 0.91, fr 0.90, it 0.85, de 0.88, and pt 0.86 (Almeida et al., 17 Sep 2025).
A particularly uneven multilingual pattern is reported for Gemini. Gemini-Pro 2.5 scores 0.63 overall, with en 0.85, es 0.54, fr 0.76, it 0.48, de 0.59, and pt 0.60. Gemini-Flash 2.5 scores 0.52 overall, with en 0.72, es 0.52, fr 0.64, it 0.37, de 0.43, and pt 0.45. The paper interprets such results as evidence that there is no universally easy or hard language; instead, each model family exhibits its own language profile (Almeida et al., 17 Sep 2025).
The reported family-level findings are threefold. First, reasoning-oriented models dominate: the five best-performing systems are all reasoning models, which the authors interpret as evidence that allocating more inference computation materially improves multilingual function-calling. Second, scale helps, but unevenly. Qwen3 Mixture-of-Experts models scale especially well; GPT-OSS shows near-linear scaling; Qwen2.5 and xLAM scale more weakly and plateau at lower performance. Third, larger models tend to be more consistent across languages, especially large GPT-OSS and large Qwen3 MoE models (Almeida et al., 17 Sep 2025).
The paper also highlights a counterexample to simple specialization narratives. xLAM models, despite being fine-tuned specifically for function calling, perform worse than their corresponding base Qwen2.5 models on Ticket-Bench. Concretely, xLAM-2-32b-fc-r scores 0.26 overall versus 0.33 for Qwen2.5-32B-Instruct, and xLAM-2-3b-fc-r scores 0.06 versus 0.11 for Qwen2.5-3B-Instruct. The authors interpret this as evidence that specialization for function-calling on the fine-tuning distribution may hurt broader multilingual generalization (Almeida et al., 17 Sep 2025).
6. Interpretation, limitations, and position within the wider “ticket” literature
Ticket-Bench’s main conclusion is that multilingual agent evaluation should be culturally aware. By localizing leagues, cities, teams, and user names while holding task structure aligned, the benchmark makes it possible to study whether end-to-end function calling remains reliable across languages and regions. The empirical results indicate that reasoning ability and model scale reduce disparities but do not eliminate them, and that multilingual asymmetries are family-specific rather than properties of the languages themselves. The paper suggests that such asymmetries likely reflect imbalances in training data and optimization (Almeida et al., 17 Sep 2025).
Several limitations are either stated or implicit. The benchmark is confined to one domain, soccer ticket purchasing, so its findings may not transfer cleanly to other agent domains such as travel, banking, healthcare, or enterprise workflows. It covers six major languages only, leaving many lower-resource languages unrepresented. The environment is simulated rather than connected to real APIs. Repeated execution is limited to three runs per query because of cost constraints. The paper does not report confidence intervals, significance tests, annotation-error analysis, translation-quality audits, inter-annotator reliability, detailed error taxonomies, prompt ablations, or breakdowns by template and constraint type (Almeida et al., 17 Sep 2025).
The benchmark is also notable for what it is not. Despite the word “ticket,” it is unrelated to the enterprise support-ticket literature on incident labeling, routing, or escalation. Those topics are addressed by systems such as UFTR for ticket routing (Han et al., 2020), Ticket-BERT for incident-management ticket labeling (Liu et al., 2023), and TickIt for online ticket escalation in cloud service systems (Liu et al., 11 Apr 2025). Ticket-Bench instead concerns consumer ticket purchasing as a multilingual function-calling problem. This distinction matters because the benchmark’s “ticket” is an object of purchase, not a support artifact.
Ticket-Bench is intended as a public evaluation suite for multilingual tool-using agents, and the paper states that it is available at https://github.com/TropicAI-Research/Ticket-Bench. No software license, data license, or detailed reproducibility checklist is specified in the provided text. Within those bounds, the benchmark’s principal significance lies in formalizing a concrete evaluation setting in which localization, interaction, reasoning over tool outputs, and final-state correctness are all integral rather than peripheral (Almeida et al., 17 Sep 2025).
