WildChat-AQA Benchmark

• WildChat-AQA is a benchmark for aggregative question answering that infers collective insights from over 182K real-world user-chatbot conversations.
• It employs a multi-stage pipeline combining metadata filtering, semantic inference, and the PROBE retrieval method to rank aggregate responses.
• Empirical results show that enhanced retrieval techniques and summary-based inputs significantly outperform traditional single-dialogue QA approaches.

WildChat-AQA is a benchmark for Aggregative Question Answering (AQA) over large-scale user–chatbot interaction logs. Rather than asking about a single dialogue, it asks models to infer collective insights—for example, topic distributions, temporal trends, or demographic correlations—by reasoning across many conversations. The benchmark contains 6,027 aggregative questions derived from 182,330 real-world user–chatbot conversations from WildChat, and it evaluates systems as a ranking problem over candidate answers rather than as span extraction or single-dialogue retrieval (Zhang et al., 29 May 2025). Its source corpus inherits the central properties of WildChat itself: real opt-in chat logs, timestamped transcripts, metadata such as location and language, and broad multilingual, topical, and behavioral diversity (Zhao et al., 2024).

1. Conceptual scope and task definition

WildChat-AQA was introduced to address a limitation of conventional QA over conversational data: most existing methods treat interactions as independent examples, whereas large chat corpora also encode population-level structure. In this setting, the target is not a local fact such as what a specific user asked, but an aggregate pattern such as what topics are associated with a region, time window, or user cohort (Zhang et al., 29 May 2025).

The task therefore differs from standard QA, summarization, and structured database aggregation. Standard QA usually assumes that a question can be answered by retrieving a small amount of local evidence. WildChat-AQA instead requires combining broad evidence distributed across many dialogues, often conditioned on metadata and inferred semantic attributes. The paper explicitly frames the task as answering query-conditioned aggregate questions, and it evaluates systems by asking them to rank 10 candidate answers with NDCG@1, NDCG@3, NDCG@5, and NDCG@10 (Zhang et al., 29 May 2025).

This formulation makes retrieval breadth central. A system must identify not merely the most relevant single conversation, but a sufficiently representative subset of conversations from which aggregate tendencies can be inferred. This suggests that WildChat-AQA is structurally closer to conversational analytics than to ordinary document QA.

2. Source corpus and attribute schema

The benchmark is built from WildChat-Full, which the benchmark construction pipeline describes as containing about 990K conversations before filtering (Zhang et al., 29 May 2025). WildChat itself was collected through public chatbot services on Hugging Face Spaces backed by GPT-3.5-Turbo and GPT-4 APIs, under a two-step affirmative consent flow. The public release contains 1,039,785 conversations and 2,639,415 turns, spanning April 9, 2023, 12 AM to May 1, 2024, 12 AM, with metadata including hashed IP-derived geographic information and request headers (Zhao et al., 2024).

WildChat-AQA uses both direct metadata and inferred semantic attributes. The benchmark’s attribute schema is as follows (Zhang et al., 29 May 2025):

Attribute	Source type	Value type
Location	direct metadata	single-valued
User Name	direct metadata	single-valued
Time	direct metadata	single-valued
Language	direct metadata	single-valued
Topic	inferred	multi-valued
Subtopic	inferred	multi-valued
Keywords	inferred	multi-valued

This mixed schema is one of the benchmark’s defining properties. Time, location, language, and user identity signals come from WildChat metadata, while topic, subtopic, and keywords are inferred from conversation content. The benchmark therefore tests systems on queries that combine explicit fields with latent semantic structure.

The underlying WildChat corpus is itself unusually broad. It contains 68 languages in total; turn-level language shares include 52.94% English, 13.38% Chinese, and 11.61% Russian; about 41% of conversations are multi-turn; and the dataset has substantial topical and behavioral diversity, including ambiguous requests, code-switching, and topic shifts within a conversation (Zhao et al., 2024). Those properties matter because aggregative reasoning becomes more difficult when the underlying corpus is multilingual, noisy, and behaviorally heterogeneous.

3. Benchmark construction pipeline

WildChat-AQA is created through a multi-stage pipeline that progressively transforms raw chat logs into aggregative QA instances (Zhang et al., 29 May 2025).

The starting point is WildChat-Full with about 990K conversations. The first stage applies MinHash + LSH deduplication, a 4,096-token length cap, and a filter retaining only active users with at least 10 interactions. User IDs are derived from hashed IP addresses and HTTP headers. This reduces the corpus to about 520K conversations after deduplication and then about 220K after filtering by length and user activity. After later annotation and quality filtering, the final benchmark uses 182,330 conversations (Zhang et al., 29 May 2025).

For each conversation, GPT-4o is prompted to produce a summary and keywords. The extracted keywords span semantic types such as persons, technologies, scientific terms, foods, organizations, locations, events, artworks, programming languages, brands, and financial terms. Topic induction then proceeds using a TnT-LLM-style taxonomy induction pipeline: conversation summaries are embedded, clustered, and iteratively refined into a hierarchy of 28 high-level topics and 455 fine-grained subtopics (Zhang et al., 29 May 2025).

Some subtopics are highly frequent, especially in areas such as Programming, Fanfiction and crossover, and Creative writing-related content. To make question generation more discriminative, keywords are further grouped into coherent categories such as Programming Language, Video Games, Tabletop Games, Manga/Anime, Film, TV Show, Western Cartoon/Comic, Book, Musical, and Public Figure (Zhang et al., 29 May 2025).

Question generation begins by enumerating structured queries with 0 to 3 conditions. The authors manually select 73 meaningful combinations that can be phrased naturally. Candidate instances are retained only when the condition matches at least 50 documents, or at least 10 when username is involved, and when the top 3 target values collectively account for at least 15% of occurrences. Natural-language realization is then performed by GPT-4.1, initially yielding 6,177 questions (Zhang et al., 29 May 2025).

A final quality-control stage uses o4-mini to test whether conversational context actually helps. The filtering compares no-context ranking performance, performance with raw conversation context, and performance with summarized context. Instances are removed if context does not improve over no-context and contextual performance is not sufficiently above random. After this stage, 6,027 valid questions remain (Zhang et al., 29 May 2025).

This pipeline implies that WildChat-AQA is not merely generated from metadata templates. It is also difficulty-filtered to retain questions for which conversation-derived evidence materially affects answerability.

4. Retrieval and reasoning formulation

The benchmark’s central methodological claim is that ordinary top-$k$ retrieval is poorly matched to aggregative questions. Conventional RAG seeks a small number of highly relevant documents, but aggregative QA often depends on multiple weak signals spread over a large corpus. To address this, the paper proposes PROBE, short for Probing Retrieval Of Broad Evidence (Zhang et al., 29 May 2025).

PROBE operates in two stages. Given a question $\mathbf{Q}$, an LLM generates both strict filters $\mathbf{F} = \{f_1, f_2, \dots, f_m\}$ and a set of diverse short queries $\{q_1, q_2, \cdots, q_n\}$: $$\mathbf{F}, \{q_1, q_2, \cdots, q_n \} = \text{LLM} (\mathbf{p}, \mathbf{Q})$$ Each query retrieves documents independently, the retrieval lists are merged, and duplicate documents are pooled by max pooling over relevance scores before top-$k$ evidence selection (Zhang et al., 29 May 2025).

In implementation, PROBE uses GPT-4.1-mini for query generation and OpenAI text-embedding-3-large for retrieval embeddings. The benchmark infrastructure also uses MongoDB v8.0.4 for question generation and ground-truth retrieval, Elasticsearch v8.18 for BM25 and dense kNN retrieval, 3072-dimensional OpenAI embeddings, vLLM for open-source inference, and HuggingFace Transformers for finetuning Qwen3-8B (Zhang et al., 29 May 2025).

The benchmark compares raw conversations with summarized conversations as retrievable units. The average raw conversation length is 1,143.4 tokens, whereas summaries average 21.5 tokens (Zhang et al., 29 May 2025). This compression ratio is important because aggregative reasoning is simultaneously retrieval-intensive and context-window-intensive; summarization can therefore change not only efficiency but also answer quality.

5. Empirical findings

WildChat-AQA reports several consistent empirical patterns (Zhang et al., 29 May 2025).

First, stronger models perform better. Among the evaluated systems, o4-mini is best overall, GPT-4.1-mini is close behind, and among open-source models Qwen3-32B-think performs best. Second, PROBE beats standard RAG by a large margin. On raw data, the reported NDCG@1 improvements include +14.8 points for Gemma 3-4B, +23.7 for Qwen3-8B-think, +24.7 for Qwen3-32B-think, +23.1 for GPT-4.1-mini, and +23.8 for o4-mini (Zhang et al., 29 May 2025).

Third, summaries beat raw conversations across models and retrieval methods. The paper attributes this to the fact that raw chat logs are noisy and verbose, whereas summaries reduce token burden and make signal aggregation easier. Fourth, finetuning does not solve the problem: finetuning Qwen3-8B does not substantially beat random performance, indicating that the benchmark is not easily reduced to parametric memorization (Zhang et al., 29 May 2025).

Ablation results reinforce the retrieval diagnosis. Better retrieval recall leads to better ranking performance, and removing either the generated queries or the filtering conditions from PROBE reduces retrieval recall. Oracle-context experiments show that even with oracle ground-truth documents, performance remains far from perfect, especially with raw context. The paper also reports that providing more retrieved documents helps and that explicit reasoning mode (“think”) improves results across oracle, RAG, and PROBE settings (Zhang et al., 29 May 2025).

These findings suggest that WildChat-AQA is simultaneously a retrieval problem, a long-context reasoning problem, and a collective inference problem. Improvements in any one component are insufficient if the others remain weak.

6. Annotation quality, representativeness, and benchmark caveats

Because WildChat-AQA relies on inferred semantic labels, annotation reliability is an important concern. For topic labeling, the reported agreement is human-human $\kappa = 0.581$ and human-model $\kappa = 0.617$; for subtopic labeling, the values are human-human $\kappa = 0.576$ and human-model $\kappa = 0.609$ (Zhang et al., 29 May 2025). These values support the use of automatic annotation, but they also indicate that the semantic layer is not noise-free.

The benchmark paper is explicit about several limitations. It notes possible model-derived annotation errors in keywords, topics, subtopics, and keyword categories; possible artificiality in GPT-4.1-generated natural-language questions; and the underlying noise and ambiguity of real chat logs (Zhang et al., 29 May 2025). These are substantive caveats because the benchmark depends on both metadata accuracy and semantic inference quality.

A second layer of caveats comes from the WildChat data source itself. WildChat was collected through frictionless public access on Hugging Face Spaces, with no account requirement but affirmative opt-in consent, and the original dataset paper explicitly notes that the user population may skew toward the IT community and may overrepresent coding-oriented or technically savvy users (Zhao et al., 2024). A later longitudinal analysis of WildChat-4.8M found additional distortions: it reported that the dataset is “extremely unusual”, over-represents power users, contains a large fraction of API-like templated usage, and “does not represent typical user-AI interactions” (Hicke et al., 27 May 2026). That analysis retained 2,522,330 conversations from 1,830,631 hashed IPs after filtering, identified 774 templates comprising 39% of the dataset, and noted that the top 10 templates account for over 15% of WildChat-4.8M (Hicke et al., 27 May 2026).

This suggests a careful interpretive boundary for WildChat-AQA. It is a benchmark over real chatbot logs, but not necessarily over a sample that is demographically or behaviorally representative of average consumer chatbot usage. Its aggregate patterns are therefore best read as patterns in WildChat-derived usage, not automatically as population-level facts about all user–LLM interaction.

7. Position in the research landscape

WildChat-AQA occupies a distinct position within research on question answering over conversational data. It is not a benchmark for single-dialogue conversational QA of the type emphasized by systems such as ChatQA, which focuses on dialogue-following, retrieval grounding, tabular reasoning, and refusal behavior in multi-turn question answering (Liu et al., 2024). Nor is it a metadata-filtering benchmark like AMAQA, where questions are answered from a document collection enriched with structured metadata such as timestamps, topics, emotional tone, and toxicity indicators (Bruni et al., 19 May 2025). Its defining property is instead aggregation across many conversations.

This distinction is important because WildChat-AQA operationalizes a different research objective: extracting collective insights from a conversational corpus. The benchmark therefore emphasizes ranking over candidate aggregate answers, broad-evidence retrieval, and conditioning on metadata-plus-semantics. In that sense, it extends the WildChat dataset from a corpus for behavior analysis, instruction tuning, multilingual modeling, and toxicity study into a benchmark for population-level reasoning over chat logs (Zhao et al., 2024).

The broader implication is that WildChat-AQA formalizes a new use of LLM interaction data. Rather than treating chat logs only as training examples or local retrieval targets, it treats them as a substrate for queryable collective knowledge. The benchmark’s results indicate that this objective remains unsolved: standard RAG is too narrow, finetuning is ineffective, oracle evidence still leaves a large reasoning gap, and high performance remains computationally expensive (Zhang et al., 29 May 2025). That combination of scale, retrieval breadth, and reasoning difficulty is the central technical significance of WildChat-AQA.