AI Search Arena

• AI Search Arena is a framework of evaluation infrastructures where models or agents engage in pairwise battles using Bradley–Terry and Elo-based approaches.
• It integrates offline, live, and automated systems to rank performance, employing controlled comparisons and uncertainty estimation for robust results.
• The approach serves multiple purposes, from generating training data through post-training data flywheels to supporting real-world application leaderboards in AI evaluations.

Searching arXiv for papers on AI search arenas, search-augmented model evaluation, arena-based ranking, and related evaluation infrastructures. Tool call: arxiv_search(query="AI Search Arena search-augmented LLM arena evaluation Bradley-Terry Elo", max_results=10) Across recent work on LLMs, search-augmented systems, and agentic evaluation, “AI Search Arena” refers to a family of arena-style infrastructures in which models or agents are compared through pairwise battles or task-level competitions, and the resulting judgments are converted into rankings, selection signals, or training data. The pattern appears in offline simulated arenas calibrated to LMSYS-style Elo rankings, live application-integrated leaderboards, crowd-sourced evaluation of search-augmented LLMs, and automated evaluation of deep research agents; in each case, pairwise preference data, Bradley–Terry or Elo-like estimation, and repeated comparison against a shared population are the core organizing ideas (Luo et al., 2024, Wang et al., 15 Aug 2025, Miroyan et al., 5 Jun 2025, Gao et al., 15 Jan 2026).

1. Statistical and procedural foundations

The canonical arena workflow is pairwise comparison under anonymity. In chatbot and search settings, a prompt is shown to two systems, users or judges select the preferred answer, and the platform aggregates many such outcomes into a global ranking. The dominant statistical model is Bradley–Terry, often presented in Elo-compatible form. If model $i$ has latent strength $\theta_i$, then the probability that $i$ beats $j$ is

$$P(i \succ j)=\frac{\exp(\theta_i)}{\exp(\theta_i)+\exp(\theta_j)}.$$

In Inclusion Arena, the same idea is written as an Elo-compatible logistic model over latent skills $u_i$, with $\alpha=\ln(10)/400$, so that a 400-point difference corresponds to approximately $10\times$ odds (Wang et al., 15 Aug 2025).

Arena systems differ mainly in where the comparisons come from and how they are sampled. Arena Learning and WizardArena use offline pairwise battles judged by a strong LLM; Inclusion Arena injects pairwise comparisons into real application traffic; Search Arena collects human preferences over search-augmented LLMs in a dedicated side-by-side interface; DR-Arena replaces human adjudication with an automated Examiner that generates tasks and judges answers against explicit rubrics (Luo et al., 2024, Wang et al., 15 Aug 2025, Miroyan et al., 5 Jun 2025, Gao et al., 15 Jan 2026).

System	Primary focus	Distinctive mechanism
WizardArena / Arena Learning	Offline arena and post-training	LLM judge, calibrated Elo prediction, data flywheel
Inclusion Arena	Live real-world app ranking	Placement Matches, Proximity Sampling
Search Arena	Search-augmented LLM evaluation	Multi-turn side-by-side battles with full search traces
DR-Arena	Deep research agent evaluation	Information Trees and Adaptive Evolvement Loop
SAGEO Arena	Generative search visibility analysis	Full retrieval–reranking–generation pipeline

Beyond ranking, arena methodology often includes uncertainty estimation and consistency diagnostics. WizardArena reports Spearman rank correlation, Human Agreement with $95\%$ confidence intervals, and Differentiation with $95\%$ confidence intervals; Inclusion Arena analyzes Fisher information through a weighted graph Laplacian; DR-Arena evaluates agreement against LMSYS Search Arena via Spearman and Pearson correlations (Luo et al., 2024, Wang et al., 15 Aug 2025, Gao et al., 15 Jan 2026).

2. Offline, simulated, and automated arenas

A major branch of AI Search Arena research replaces human adjudication with offline or automated judges. Arena Learning formalizes this as an offline pairwise battle arena over a large instruction set $\theta_i$0, a set of battle models $\theta_i$1, and a judge model, Llama-3-70B-Instruct, which scores both answers from 1 to 10 and explains the decision. To reduce position bias, each pair is judged twice with swapped positions. The resulting WizardArena benchmark is built from a large cleaned instruction pool, then split into WizardArena-Diverse (1k), WizardArena-Hard (1k), and WizardArena-Mix (2k). WizardArena-Mix achieves Spearman $\theta_i$2, Human Agreement $\theta_i$3, and Differentiation $\theta_i$4 against LMSYS Chatbot Arena EN Elo, substantially above MT-Bench and Arena-Hard v1.0. For 1M battles, the reported runtime is approximately 9 days on $\theta_i$5G GPUs, versus roughly 12 months of human annotation at LMSYS scale, a reported $\theta_i$6 speedup (Luo et al., 2024).

DR-Arena extends the same logic to deep research agents rather than general chatbots. Its evaluation objects are not fixed prompts but dynamically generated tasks derived from live web “Information Trees.” The automated Examiner samples fresh topics, builds a tree of linked web pages, generates “Deep reasoning” and “Wide coverage” questions, and judges competing agents with a rubric consisting of checklist_depth and checklist_width. An Adaptive Evolvement Loop then changes task difficulty depending on the loser’s failure type: DEEP, WIDE, BOTH, or NONE. In experiments with six advanced DR agents, DR-Arena reports a Spearman correlation of $\theta_i$7 with the LMSYS Search Arena leaderboard; using only round 1 yields $\theta_i$8, while round 2 already reaches $\theta_i$9. Human audits further report Cohen’s $i$0 for verdict alignment with human judges (Gao et al., 15 Jan 2026).

These systems treat the arena not merely as a leaderboard but as a calibrated surrogate for human preference. This suggests that, when calibration is strong enough, offline or automated arenas can support large-scale internal model comparison that would be infeasible with direct human adjudication alone.

3. Live product arenas and search-augmented comparison

A second line of work embeds the arena directly into real usage. Inclusion Arena is designed explicitly for internet applications rather than crowdsourced demo sites. Its platform comprises an App Pool, Model Center, Management Layer, and Execution Layer. Applications integrate through a Conversation API and a User Preference Feedback API; for each live query, the system selects $i$1 models, usually 2–3, runs them concurrently, presents anonymized outputs, and converts user selections into pairwise comparisons. The reported deployment spans two applications with 46,611 users, 49 models, and approximately 0.5M valid pairwise comparisons. For cold start, Inclusion Arena introduces Placement Matches, a binary-search-style procedure that runs $i$2 battles against ranked reference models to estimate an initial rating. For ongoing comparison, it introduces Proximity Sampling: models are compared primarily within a trust region defined by $i$3, motivated by the fact that information gain is highest when outcomes are near $i$4-$i$5. Simulations report that, at $i$6, total variance $i$7 is reduced by approximately $i$8 compared to random sampling (Wang et al., 15 Aug 2025).

Search Arena applies the arena paradigm specifically to search-augmented LLMs. Implemented as a dedicated “Search” tab inside Chatbot Arena, it records 24,069 conversations over seven weeks, including approximately 5,000 multi-turn interactions and 12,652 human preference votes, across 71 languages and 9 intent categories. Each record includes full conversation history, web search queries, retrieved URLs, inline citations, and, for some models, reasoning traces. The dataset therefore exposes not only which model won, but also how each system searched, cited, and synthesized. Cross-arena experiments show that a search-augmented model and its non-search counterpart are at overall parity in a general-purpose chat environment, while in Search Arena the non-search model underperforms significantly; the quality in search settings is “significantly affected if solely relying on the model’s parametric knowledge” (Miroyan et al., 5 Jun 2025).

The live-arena model changes the meaning of evaluation. Instead of testing systems against a fixed prompt set, it treats ranking as an online statistical summary of performance under organic use. In production-oriented settings, this shifts the arena from benchmark artifact to infrastructural component.

4. Arena learning, data flywheels, and post-training

Arena Learning turns the arena into a post-training mechanism. Its central idea is a data flywheel: run battles between a target model and stronger opponents, identify where the target loses, convert those failures into supervised or preference data, retrain, and repeat. The framework uses three stages—SFT, DPO, and PPO. The SFT objective is standard token-level cross-entropy,

$i$9

where $j$0 is the winning response selected by the judge. DPO uses chosen–rejected pairs mined from battles, and PPO uses a reward model trained on the same preference data (Luo et al., 2024).

The empirical point is not merely that more data helps, but that battle-selected data behaves as targeted hard-example mining. For WizardLM-$j$1-7B, the selected SFT subset shrinks across iterations while its average difficulty increases: SFT-$j$2 uses 14.6k selected samples with average difficulty 5.8, SFT-$j$3 uses 11.3k with difficulty 6.5, and SFT-$j$4 uses 7.8k with difficulty 7.4. Thresholding by score gap improves efficiency: in SFT-$j$5, $j$6 uses all 30k samples and yields WizardArena Elo 1047, whereas $j$7 uses about 14.6k samples and yields Elo 1117; in DPO-$j$8, $j$9 uses about 18.1k samples and yields Elo 1187, above the $$P(i \succ j)=\frac{\exp(\theta_i)}{\exp(\theta_i)+\exp(\theta_j)}.$$0 baseline of 1165. Pair-judge selection outperforms the original 30k unfiltered set by +29 Elo and +0.35 MT-Bench (Luo et al., 2024).

End-to-end gains are large. WizardLM-$$P(i \succ j)=\frac{\exp(\theta_i)}{\exp(\theta_i)+\exp(\theta_j)}.$$1-7B improves from WizardArena-Mix Elo 871 and MT-Bench 6.41 at $$P(i \succ j)=\frac{\exp(\theta_i)}{\exp(\theta_i)+\exp(\theta_j)}.$$2 to Elo 1274 and MT-Bench 8.16 at $$P(i \succ j)=\frac{\exp(\theta_i)}{\exp(\theta_i)+\exp(\theta_j)}.$$3; WizardLM-$$P(i \succ j)=\frac{\exp(\theta_i)}{\exp(\theta_i)+\exp(\theta_j)}.$$4-8x22B improves from Elo 889 and MT-Bench 6.78 to Elo 1349 and MT-Bench 8.85. The paper states that Arena Learning “plays a pivotal role in the success of WizardLM-2” (Luo et al., 2024).

In this formulation, an AI Search Arena is not only a ranking device. It is also a continual alignment and selection engine in which evaluation data is recycled into new supervision, and model search becomes inseparable from post-training.

5. Bias, robustness, and strategic behavior

Arena outputs are shaped by judges, users, and strategic participants. Search Arena shows this directly in citation behavior. User preference is strongly influenced by the number of citations, with a positive Bradley–Terry coefficient $$P(i \succ j)=\frac{\exp(\theta_i)}{\exp(\theta_i)+\exp(\theta_j)}.$$5. More strikingly, when claim–citation pairs are labeled as Support, Irrelevant, or Contradict, both supporting and irrelevant citation counts are positively associated with user preference, with $$P(i \succ j)=\frac{\exp(\theta_i)}{\exp(\theta_i)+\exp(\theta_j)}.$$6 and $$P(i \succ j)=\frac{\exp(\theta_i)}{\exp(\theta_i)+\exp(\theta_j)}.$$7, while contradicting citations are statistically insignificant. Users also prefer citations from tech/coding platforms ($$P(i \succ j)=\frac{\exp(\theta_i)}{\exp(\theta_i)+\exp(\theta_j)}.$$8), community platforms ($$P(i \succ j)=\frac{\exp(\theta_i)}{\exp(\theta_i)+\exp(\theta_j)}.$$9), and social media ($u_i$0), while “wiki” sources are negatively associated ($u_i$1) (Miroyan et al., 5 Jun 2025).

The citation-level audit of the AI Search Arena dataset sharpens the same point. Across 24,069 conversations and 65,768 model responses, the dataset contains 366,087 citations from 83,533 unique domains; 32,865 citations, or $u_i$2, go to 1,795 news domains. Citation concentration is high, with Gini coefficients $u_i$3 for OpenAI, $u_i$4 for Perplexity, and $u_i$5 for Google. The study reports a “pronounced liberal bias,” with right-leaning outlets comprising $u_i$6, $u_i$7, and $u_i$8 of news citations for Perplexity, Google, and OpenAI respectively, while high-quality sources dominate. Yet user preference models find that neither the political leaning nor the quality of cited news sources significantly influences user satisfaction once basic response features are controlled (Yang, 7 Jul 2025).

Robustness problems also arise from the mechanism itself. “Strategic Candidacy in Generative AI Arenas” proves that, under the status-quo Bradley–Terry maximum-likelihood ranking rule, producers can benefit by submitting clones of competitive models. The proposed mechanism, You-Rank-We-Rank (YRWR), requires producers to submit rankings over their own models and then applies a monotonicity correction to fitted scores. The paper proves approximate clone-robustness and shows in simulation that ranking accuracy can improve when producers rank their own models correctly or approximately correctly (Hays et al., 27 Mar 2026).

A separate sociotechnical analysis describes “arena-ization” as a coupling of evaluation, spectacle, and commercialization. In that account, leaderboards serve as attention infrastructures, while bespoke comparisons, preferential access to private testing, and marketing-oriented optimization become forms of “arena gaming.” This does not negate the technical utility of arenas, but it establishes that their ranking rules and access policies are part of the object being evaluated (Hind, 17 Dec 2025).

6. Extensions, neighboring infrastructures, and scope

AI Search Arena has expanded beyond model-vs-model preference ranking into adjacent evaluation problems. SAGEO Arena studies Search-Augmented Generative Engine Optimization under a full retrieval–reranking–generation pipeline. It builds a corpus of 171,003 web documents from 2,700 queries, preserves structural fields such as title, meta description, headings, and schema/JSON-LD, and evaluates visibility with document-level rank, Hit Rate, and Citation Rate. For a target document $u_i$9, document rank is defined as the best passage rank,

$\alpha=\ln(10)/400$0

The main finding is stage-specific: body-only GEO-style rewriting is “largely impractical under realistic conditions and often degrade[s] performance in retrieval and reranking,” while structural information helps mitigate these limitations, and “effective SAGEO requires tailoring optimization to each pipeline stage” (Kim et al., 12 Feb 2026).

Mobile and systems-level arenas show how far the pattern can generalize. Android Agent Arena (A3) evaluates mobile GUI agents over 21 widely used third-party apps and 201 tasks, including operation tasks, single-frame query tasks, and multi-frame query tasks. It provides a unified action space spanning AitW, AitZ, AMEX, and AndroidControl, and complements element- and action-matching evaluators with LLM-based judging that reaches $\alpha=\ln(10)/400$1 correctness for GPT-4o and $\alpha=\ln(10)/400$2 for Gemini 1.5 Pro on a 50-task sample (Chai et al., 2 Jan 2025). At internet-service scale, AIBench approaches the problem from a different angle: it identifies learning to rank as one of sixteen AI component benchmarks and implements an end-to-end e-commerce search-and-recommendation service with Search Planer, Recommender, Searcher, and Ranker microservices, providing a systems complement to preference-based arenas (Gao et al., 2019).

Taken together, these extensions indicate that AI Search Arena is not a single benchmark but a design pattern. It can serve as a calibrated offline evaluator, a live production leaderboard, a post-training data flywheel, a document-visibility testbed, a mobile task arena, or a full internet-service benchmark. The shared structure is persistent: heterogeneous agents compete on realistic tasks, comparisons are normalized into statistically interpretable scores, and the resulting signals are used either to select, to optimize, or to govern AI systems.