---
title: WildChat-AQA Benchmark
url: https://www.emergentmind.com/topics/wildchat-aqa
type: topic
---

# WildChat-AQA Benchmark

WildChat-AQA is a benchmark for **Aggregative Question Answering (AQA)** over large-scale user–chatbot interaction logs. Rather than asking about a single dialogue, it asks models to infer **collective insights**—for example, topic distributions, temporal trends, or demographic correlations—by reasoning across many conversations. The benchmark contains **6,027 aggregative questions** derived from **182,330 real-world user–chatbot conversations** from WildChat, and it evaluates systems as a **ranking problem** over candidate answers rather than as span extraction or single-dialogue retrieval [2505.23765]. Its source corpus inherits the central properties of WildChat itself: real opt-in chat logs, timestamped transcripts, metadata such as location and language, and broad multilingual, topical, and behavioral diversity [2405.01470].

## 1. Conceptual scope and task definition

WildChat-AQA was introduced to address a limitation of conventional QA over conversational data: most existing methods treat interactions as **independent** examples, whereas large chat corpora also encode **population-level structure**. In this setting, the target is not a local fact such as what a specific user asked, but an aggregate pattern such as what topics are associated with a region, time window, or user cohort [2505.23765].

The task therefore differs from standard QA, summarization, and structured database aggregation. Standard QA usually assumes that a question can be answered by retrieving a small amount of local evidence. WildChat-AQA instead requires combining **broad evidence** distributed across many dialogues, often conditioned on metadata and inferred semantic attributes. The paper explicitly frames the task as answering **query-conditioned aggregate questions**, and it evaluates systems by asking them to **rank 10 candidate answers** with **NDCG@1**, **NDCG@3**, **NDCG@5**, and **NDCG@10** [2505.23765].

This formulation makes retrieval breadth central. A system must identify not merely the most relevant single conversation, but a sufficiently representative subset of conversations from which aggregate tendencies can be inferred. This suggests that WildChat-AQA is structurally closer to conversational analytics than to ordinary document QA.

## 2. Source corpus and attribute schema

The benchmark is built from **WildChat-Full**, which the benchmark construction pipeline describes as containing about **990K conversations** before filtering [2505.23765]. WildChat itself was collected through public chatbot services on Hugging Face Spaces backed by **GPT-3.5-Turbo** and **GPT-4** APIs, under a two-step affirmative consent flow. The public release contains **1,039,785 conversations** and **2,639,415 turns**, spanning **April 9, 2023, 12 AM to May 1, 2024, 12 AM**, with metadata including hashed IP-derived geographic information and request headers [2405.01470].

WildChat-AQA uses both **direct metadata** and **inferred semantic attributes**. The benchmark’s attribute schema is as follows [2505.23765]:

| Attribute | Source type | Value type |
|---|---|---|
| Location | direct metadata | single-valued |
| User Name | direct metadata | single-valued |
| Time | direct metadata | single-valued |
| Language | direct metadata | single-valued |
| Topic | inferred | multi-valued |
| Subtopic | inferred | multi-valued |
| Keywords | inferred | multi-valued |

This mixed schema is one of the benchmark’s defining properties. Time, location, language, and user identity signals come from WildChat metadata, while topic, subtopic, and keywords are inferred from conversation content. The benchmark therefore tests systems on queries that combine **explicit fields** with **latent semantic structure**.

The underlying WildChat corpus is itself unusually broad. It contains **68 languages** in total; turn-level language shares include **52.94% English**, **13.38% Chinese**, and **11.61% Russian**; about **41%** of conversations are multi-turn; and the dataset has substantial topical and behavioral diversity, including ambiguous requests, code-switching, and topic shifts within a conversation [2405.01470]. Those properties matter because aggregative reasoning becomes more difficult when the underlying corpus is multilingual, noisy, and behaviorally heterogeneous.

## 3. Benchmark construction pipeline

WildChat-AQA is created through a multi-stage pipeline that progressively transforms raw chat logs into aggregative QA instances [2505.23765].

The starting point is **WildChat-Full** with about **990K conversations**. The first stage applies **MinHash + LSH deduplication**, a **4,096-token length cap**, and a filter retaining only **active users** with at least **10 interactions**. User IDs are derived from **hashed IP addresses and HTTP headers**. This reduces the corpus to about **520K** conversations after deduplication and then about **220K** after filtering by length and user activity. After later annotation and quality filtering, the final benchmark uses **182,330 conversations** [2505.23765].

For each conversation, **GPT-4o** is prompted to produce a **summary** and **keywords**. The extracted keywords span semantic types such as persons, technologies, scientific terms, foods, organizations, locations, events, artworks, programming languages, brands, and financial terms. Topic induction then proceeds using a **TnT-LLM-style taxonomy induction** pipeline: conversation summaries are embedded, clustered, and iteratively refined into a hierarchy of **28 high-level topics** and **455 fine-grained subtopics** [2505.23765].

Some subtopics are highly frequent, especially in areas such as **Programming**, **Fanfiction and crossover**, and **Creative writing-related content**. To make question generation more discriminative, keywords are further grouped into coherent categories such as **Programming Language**, **Video Games**, **Tabletop Games**, **Manga/Anime**, **Film**, **TV Show**, **Western Cartoon/Comic**, **Book**, **Musical**, and **Public Figure** [2505.23765].

Question generation begins by enumerating structured queries with **0 to 3 conditions**. The authors manually select **73 meaningful combinations** that can be phrased naturally. Candidate instances are retained only when the condition matches at least **50** documents, or at least **10** when username is involved, and when the **top 3 target values** collectively account for at least **15%** of occurrences. Natural-language realization is then performed by **GPT-4.1**, initially yielding **6,177 questions** [2505.23765].

A final quality-control stage uses **o4-mini** to test whether conversational context actually helps. The filtering compares **no-context ranking performance**, performance with **raw conversation context**, and performance with **summarized context**. Instances are removed if context does not improve over no-context and contextual performance is not sufficiently above random. After this stage, **6,027 valid questions** remain [2505.23765].

This pipeline implies that WildChat-AQA is not merely generated from metadata templates. It is also **difficulty-filtered** to retain questions for which conversation-derived evidence materially affects answerability.

## 4. Retrieval and reasoning formulation

The benchmark’s central methodological claim is that ordinary top-\(k\) retrieval is poorly matched to aggregative questions. Conventional RAG seeks a small number of highly relevant documents, but aggregative QA often depends on **multiple weak signals** spread over a large corpus. To address this, the paper proposes **PROBE**, short for **Probing Retrieval Of Broad Evidence** [2505.23765].

PROBE operates in two stages. Given a question \( \mathbf{Q} \), an LLM generates both strict filters \( \mathbf{F} = \{f_1, f_2, \dots, f_m\} \) and a set of diverse short queries \( \{q_1, q_2, \cdots, q_n\} \):
\[
\mathbf{F}, \{q_1, q_2, \cdots, q_n \} = \text{LLM} (\mathbf{p}, \mathbf{Q})
\]
Each query retrieves documents independently, the retrieval lists are merged, and duplicate documents are pooled by **max pooling** over relevance scores before top-\(k\) evidence selection [2505.23765].

In implementation, PROBE uses **GPT-4.1-mini** for query generation and **OpenAI text-embedding-3-large** for retrieval embeddings. The benchmark infrastructure also uses **MongoDB v8.0.4** for question generation and ground-truth retrieval, **Elasticsearch v8.18** for BM25 and dense kNN retrieval, **3072-dimensional** OpenAI embeddings, **vLLM** for open-source inference, and **HuggingFace Transformers** for finetuning **Qwen3-8B** [2505.23765].

The benchmark compares **raw conversations** with **summarized conversations** as retrievable units. The average raw conversation length is **1,143.4 tokens**, whereas summaries average **21.5 tokens** [2505.23765]. This compression ratio is important because aggregative reasoning is simultaneously retrieval-intensive and context-window-intensive; summarization can therefore change not only efficiency but also answer quality.

## 5. Empirical findings

WildChat-AQA reports several consistent empirical patterns [2505.23765].

First, **stronger models perform better**. Among the evaluated systems, **o4-mini** is best overall, **GPT-4.1-mini** is close behind, and among open-source models **Qwen3-32B-think** performs best. Second, **PROBE beats standard RAG by a large margin**. On raw data, the reported **NDCG@1** improvements include **+14.8 points** for **Gemma 3-4B**, **+23.7** for **Qwen3-8B-think**, **+24.7** for **Qwen3-32B-think**, **+23.1** for **GPT-4.1-mini**, and **+23.8** for **o4-mini** [2505.23765].

Third, **summaries beat raw conversations** across models and retrieval methods. The paper attributes this to the fact that raw chat logs are noisy and verbose, whereas summaries reduce token burden and make signal aggregation easier. Fourth, **finetuning does not solve the problem**: finetuning **Qwen3-8B** does **not** substantially beat random performance, indicating that the benchmark is not easily reduced to parametric memorization [2505.23765].

Ablation results reinforce the retrieval diagnosis. Better retrieval recall leads to better ranking performance, and removing either the generated queries or the filtering conditions from PROBE reduces retrieval recall. Oracle-context experiments show that even with **oracle ground-truth documents**, performance remains far from perfect, especially with raw context. The paper also reports that providing **more retrieved documents** helps and that explicit reasoning mode (“**think**”) improves results across oracle, RAG, and PROBE settings [2505.23765].

These findings suggest that WildChat-AQA is simultaneously a **retrieval problem**, a **long-context reasoning problem**, and a **collective inference problem**. Improvements in any one component are insufficient if the others remain weak.

## 6. Annotation quality, representativeness, and benchmark caveats

Because WildChat-AQA relies on inferred semantic labels, annotation reliability is an important concern. For **topic** labeling, the reported agreement is **human-human \( \kappa = 0.581 \)** and **human-model \( \kappa = 0.617 \)**; for **subtopic** labeling, the values are **human-human \( \kappa = 0.576 \)** and **human-model \( \kappa = 0.609 \)** [2505.23765]. These values support the use of automatic annotation, but they also indicate that the semantic layer is not noise-free.

The benchmark paper is explicit about several limitations. It notes possible **model-derived annotation errors** in keywords, topics, subtopics, and keyword categories; possible **artificiality** in GPT-4.1-generated natural-language questions; and the underlying **noise and ambiguity** of real chat logs [2505.23765]. These are substantive caveats because the benchmark depends on both metadata accuracy and semantic inference quality.

A second layer of caveats comes from the WildChat data source itself. WildChat was collected through frictionless public access on Hugging Face Spaces, with no account requirement but affirmative opt-in consent, and the original dataset paper explicitly notes that the user population may skew toward the **IT community** and may overrepresent **coding-oriented or technically savvy users** [2405.01470]. A later longitudinal analysis of **WildChat-4.8M** found additional distortions: it reported that the dataset is **“extremely unusual”**, over-represents **power users**, contains a large fraction of **API-like** templated usage, and **“does not represent typical user-AI interactions”** [2605.29018]. That analysis retained **2,522,330 conversations from 1,830,631 hashed IPs** after filtering, identified **774** templates comprising **39%** of the dataset, and noted that the top 10 templates account for **over 15%** of WildChat-4.8M [2605.29018].

This suggests a careful interpretive boundary for WildChat-AQA. It is a benchmark over **real chatbot logs**, but not necessarily over a sample that is demographically or behaviorally representative of average consumer chatbot usage. Its aggregate patterns are therefore best read as patterns in **WildChat-derived usage**, not automatically as population-level facts about all user–LLM interaction.

## 7. Position in the research landscape

WildChat-AQA occupies a distinct position within research on question answering over conversational data. It is not a benchmark for **single-dialogue conversational QA** of the type emphasized by systems such as ChatQA, which focuses on dialogue-following, retrieval grounding, tabular reasoning, and refusal behavior in multi-turn question answering [2401.10225]. Nor is it a metadata-filtering benchmark like **AMAQA**, where questions are answered from a document collection enriched with structured metadata such as timestamps, topics, emotional tone, and toxicity indicators [2505.13557]. Its defining property is instead **aggregation across many conversations**.

This distinction is important because WildChat-AQA operationalizes a different research objective: extracting **collective insights** from a conversational corpus. The benchmark therefore emphasizes ranking over candidate aggregate answers, broad-evidence retrieval, and conditioning on metadata-plus-semantics. In that sense, it extends the WildChat dataset from a corpus for behavior analysis, instruction tuning, multilingual modeling, and toxicity study into a benchmark for **population-level reasoning over chat logs** [2405.01470].

The broader implication is that WildChat-AQA formalizes a new use of LLM interaction data. Rather than treating chat logs only as training examples or local retrieval targets, it treats them as a substrate for **queryable collective knowledge**. The benchmark’s results indicate that this objective remains unsolved: standard RAG is too narrow, finetuning is ineffective, oracle evidence still leaves a large reasoning gap, and high performance remains computationally expensive [2505.23765]. That combination of scale, retrieval breadth, and reasoning difficulty is the central technical significance of WildChat-AQA.

Source: https://www.emergentmind.com/topics/wildchat-aqa