WildChat-50M: A Comparative Chat Dataset

• WildChat-50M is a large public chat dataset that extends the original corpus by incorporating responses from over 50 open-weight models to benchmark synthetic data quality.
• It enables controlled comparisons of synthetic responses across diverse post-training models with detailed metrics on scale, speed, and efficiency.
• Its design transforms observational logs into experimental infrastructure, providing actionable insights for supervised fine-tuning and downstream LLM evaluation.

Searching arXiv for WildChat-50M and closely related WildChat papers. WildChat-50M is a large public chat dataset for studying synthetic data in LLM post-training. It extends the original WildChat corpus of real user–ChatGPT conversations by preserving the original WildChat prompts and augmenting them with responses from over 50 open-weight models, thereby enabling controlled comparisons of synthetic data generating models on a shared prompt distribution (Feuer et al., 30 Jan 2025). The dataset is presented as the largest public chat dataset to date, and its central methodological premise is that open post-training research requires a common substrate in which the same real user prompts are answered by many different models rather than by a single proprietary system (Feuer et al., 30 Jan 2025).

1. Lineage within the WildChat family

WildChat-50M is an extension of the original WildChat release rather than a new prompt corpus. The base WildChat paper introduced a public dataset of real user–ChatGPT interaction logs collected through free public access to GPT-3.5-Turbo- and GPT-4-based services hosted on Hugging Face Spaces, with explicit two-step opt-in consent, timestamped transcripts, hashed IP addresses, request headers, and coarse geographic metadata (Zhao et al., 2024). In its public release, WildChat reported 1,039,785 conversations and 2,639,415 turns, with 68 languages appearing in more than 100 user prompts and an average conversation length of approximately 2.54 turns (Zhao et al., 2024).

WildChat-50M inherits this “real prompts, real usage” substrate but shifts the research target from descriptive analysis of deployed chatbot use to comparative analysis of synthetic responses for post-training (Feuer et al., 30 Jan 2025). The paper explicitly states that it considered both WildChat-1M and LMSYS-Chat-1M as prompt sources and chose WildChat because it offered a richer variety of use cases, potentially toxic or difficult content, regional and temporal diversity, and relatively low contamination on common test sets (Feuer et al., 30 Jan 2025). In that sense, WildChat-50M is neither a replacement for the original WildChat release nor merely a larger snapshot of the same logs; it is a multi-model augmentation layer built over WildChat prompts.

This design matters because it converts WildChat from an observational corpus into experimental infrastructure. Instead of asking how users interacted with one deployed assistant, WildChat-50M asks how different open models behave when confronted with the same in-the-wild conversational inputs. That distinction is central to its relevance for supervised fine-tuning, distillation, and synthetic preference data research.

2. Dataset construction, scale, and composition

The dataset extends WildChat by collecting synthetic responses from 54 checkpoints, comprising 19 unique pre-trained models and 35 post-trained variants, with parameter scales ranging from 0.5B to 104B and release dates spanning July 2023 to November 2024 (Feuer et al., 30 Jan 2025). The paper notes a slight accounting nuance: its high-level summary refers to “50 different open-weight models,” while the technical description reports 54 checkpoints represented (Feuer et al., 30 Jan 2025).

Each model participates in over 1M multi-turn conversations, with 2 to 3 turns per conversation on average, yielding over 125 million chat transcripts in aggregate (Feuer et al., 30 Jan 2025). The corpus includes original WildChat user prompts and conversation context, synthetic responses from open-weight models, and the original WildChat GPT responses as a reference source; the paper also mentions judgments generated using models, although it does not provide a full released schema for judgment artifacts in the main text (Feuer et al., 30 Jan 2025).

Quantity	Reported value
Checkpoints represented	54
Unique pre-trained models	19
Post-trained variants	35
Parameter range	0.5B to 104B
Conversations per model	over 1M multi-turn conversations
Average turns per conversation	2 to 3
Aggregate scale	over 125 million chat transcripts

The model roster spans multiple families and alignment recipes. Named examples in the paper include Qwen2.5-72B-Instruct, Llama-3.1-8B-Instruct, Llama-3.3-70B, Qwen2-7B-Instruct, Command-R-Plus-104B, AI21-Jamba-1.5-Mini, Gemma, Mixtral, Mistral derivatives, InternLM, GLM, NVLM-D-72B, Athene-70B, and post-training variants associated with DPO, IPO, KTO, ORPO, RDPO, RRHF, SLiC-HF, SimPO, Magpie, OpenHermes, Tulu, Ultrachat, WizardLM, and ShareGPT (Feuer et al., 30 Jan 2025).

The paper is notably less explicit about low-level preprocessing than about scale. It does not provide a detailed step-by-step cleaning pipeline for the extended dataset, does not specify prompt filtering or deduplication beyond what was inherited from WildChat, and does not print exact decoding hyperparameters such as temperature, top-p, max new tokens, or stop sequences in the main text (Feuer et al., 30 Jan 2025). It also does not provide a field-by-field per-record schema. This makes WildChat-50M unusually strong as a comparative response-generation resource while leaving some release-level implementation details external to the manuscript.

3. Inference infrastructure and analytic affordances

WildChat-50M was collected over approximately two months on a 12×8 H100 shared research cluster, at an estimated cost of 10,000 H100-hours (Feuer et al., 30 Jan 2025). All responses and judgments were generated using vLLM; models were distributed across up to 8 GPUs, and no model was run across more than one node (Feuer et al., 30 Jan 2025). The authors report that they first minimized the number of GPUs required per model and then heuristically maximized usable context length, resulting in context windows ranging from 2,048 to 20,000 tokens (Feuer et al., 30 Jan 2025).

Precision settings were heterogeneous but explicitly documented. The largest models were queried with FP8 quantization using Neural Magic checkpoints, while all other models ran in bfloat16; the paper also states that it does not ablate the effect of quantization on output quality (Feuer et al., 30 Jan 2025). This is important because the dataset is not only a textual artifact but also a benchmarking substrate for throughput, latency, and memory-efficiency comparisons under a relatively homogeneous serving stack.

On a random subset of 5,000 conversations, the paper reports two efficiency metrics: average combined input and output tokens per second and average elapsed time per 1,000 conversations (Feuer et al., 30 Jan 2025). The slowest model in that setup was Qwen2.5-72B-Instruct, with a 20,000-token context window and 3,163 Tok/s, while the fastest was Llama-2-7B-Chat, with a 2,048-token context window and 37,357 Tok/s (Feuer et al., 30 Jan 2025). The mean input/output speed ratio across unique pre-trained models was 4.68:1 with standard deviation 3.3, and both throughput metrics were reported as strongly correlated with a proxy based on context window length multiplied by parameter count, with Time $\sigma = 0.90, \rho = 0.73$ and Tok/s $\sigma = -0.41, \rho = -0.80$ (Feuer et al., 30 Jan 2025).

These engineering details are not incidental. They enable WildChat-50M to function as a comparative systems benchmark for synthetic data generation, not only as a static corpus. The same infrastructure also underlies the paper’s downstream evaluation suite, which includes MTBench, AlpacaEval2 length-controlled win rate, BBH, GPQA, MATH, MUSR, IFEval, MMLU Pro, and MixEval (Feuer et al., 30 Jan 2025).

4. Comparative analysis of synthetic data generating models

A central contribution of the WildChat-50M paper is the controlled comparison of synthetic data generating models by holding prompts fixed and varying only the response source (Feuer et al., 30 Jan 2025). The main downstream experiment fine-tunes Llama-3.1-8B-Base on 250k samples generated by different DGMs and then evaluates the resulting student models on a nine-benchmark suite (Feuer et al., 30 Jan 2025).

The paper uses shorthand names for several especially discussed generators: Q72 for Qwen2.5-72B-Instruct, L8I for Llama-3.1-8B-Instruct, L70 for Llama-3.3-70B, Q7 for Qwen2-7B-Instruct, CRP for Command-R-Plus, and JMB for Jamba-Mini (Feuer et al., 30 Jan 2025). Selected results illustrate the heterogeneity of DGM quality. Q72-generated data produced MTBench 6.86, AlpacaEval 41.00, and MixEval 64.50; L8I produced MTBench 6.26 and AlpacaEval 21.12; L70 produced MTBench 6.23 and AlpacaEval 24.91; Q7 produced MTBench 6.03, AlpacaEval 17.26, and BBH 48.72; CRP produced BBH 49.27 and GPQA 30.82; JMB produced AlpacaEval 25.14 (Feuer et al., 30 Jan 2025).

The paper’s interpretation is unusually specific. It concludes that DGM choice strongly affects downstream performance, that the effect is large and unpredictable, that parameter count is not a reliable proxy for synthetic data quality, and that no single model dominates all benchmarks (Feuer et al., 30 Jan 2025). On 3 of 9 benchmarks, the best DGM had fewer than 10B parameters (Feuer et al., 30 Jan 2025). This directly opposes a simple “larger teacher implies better student” narrative.

The paper also reports several ablations. Scaling studies at 100k, 250k, and 500k samples indicate that performance generally improves with more data, but the rate of improvement depends on synthetic data quality; the apparent asymptote, if one exists, lies beyond 500k samples in the reported experiments (Feuer et al., 30 Jan 2025). A context-length ablation on Qwen2.5-72B truncates its responses to match Llama-like length and yields a slightly positive effect on downstream average score, reported as 0.404 versus 0.400 (Feuer et al., 30 Jan 2025).

A separate blending analysis tests whether mixing outputs from multiple DGMs helps. The reported 500k-sample results show no gain from such mixtures: for example, L8B:Q72 averages 0.37, L8B:L70 averages 0.36, and L8B:Q72+L70 remains 0.36 (Feuer et al., 30 Jan 2025). The paper concludes that blending DGMs offers no benefit and that mixtures tend to perform between their components rather than above them (Feuer et al., 30 Jan 2025). A plausible implication is that prompt diversity, already supplied by WildChat, contributes more than generator diversity once the prompt distribution is sufficiently broad.

5. Style inheritance, response similarity, and RE-WILD

WildChat-50M is used not only to compare DGMs but also to study what students inherit from them. The paper reports that benchmark-specific strengths and weaknesses are not reliably inherited: across six benchmarks, the agreement rate between a teacher’s benchmark strength and the student’s corresponding strength is 0.5, i.e. chance level (Feuer et al., 30 Jan 2025). By contrast, stylistic inheritance is strong.

For style analysis, the paper examines 80 MTBench turns, converts Markdown-like outputs to HTML-like styling tags, and counts features such as emphasis, ordered and unordered lists, headers, paragraphs, and response length (Feuer et al., 30 Jan 2025). It defines proportional frequency for models $A$ and $B$ over feature $F$ as

$\mathrm{PF}(A,B;F) = \frac{A[F]}{B[F]}.$

The reported mean proportional frequency between an SFT model and its own DGM is close to 1: Qwen SFT versus Qwen DGM yields MPF 0.91, and Llama SFT versus Llama DGM yields MPF 1.05 (Feuer et al., 30 Jan 2025). Cross-SFT comparison is much more divergent, with MPF 2.61 (Feuer et al., 30 Jan 2025). The paper therefore argues that style, formatting, and presentation are strongly heritable under SFT even when benchmark-specific capability patterns are not.

The appendix also reports inter-model response similarity. Over 500 responses from 25 randomly selected models compared against reference responses from 4 other randomly sampled models, similarity scores computed with ROUGE-1, ROUGE-L, and METEOR are interpreted as showing that LLM outputs are more similar to one another than one might expect (Feuer et al., 30 Jan 2025). Named examples include ROUGE-1 values of approximately 0.37 for Mixtral-8x7B-Instruct and Llama-3.1-Nemotron-70B-Instruct, approximately 0.34 for Qwen2.5-72B-Instruct, approximately 0.24 for Ministral-8B-Instruct-2410, and approximately 0.19 for Mistral-7B-Base-SFT-RRHF (Feuer et al., 30 Jan 2025). The paper further notes that larger models tend to generate more similar responses and may be closer to a “consensus response” (Feuer et al., 30 Jan 2025).

The practical downstream demonstration is RE-WILD. This SFT mixture is composed of 246,750 WildChat-Q72 conversations, 99,800 MMLU Auxiliary Train examples, and 20,000 Tulu 3 Persona Hub Algebra examples, for a total of 366,550 conversations/examples (Feuer et al., 30 Jan 2025). The mixture is explicitly described as about 40% the size of the Tulu-3 SFT blend (Feuer et al., 30 Jan 2025). Training uses a modified Axolotl framework with AdamW, learning rate $2 \times 10^{-5}$, 1 epoch, a cosine scheduler, 8 steps of gradient accumulation, bf16 precision, and speed optimizations including gradient checkpointing, flash attention, and sometimes FSDP, on 1 node with 4×H100; average time to fine-tune on 250,000 conversations is reported as approximately 5.5 hours (Feuer et al., 30 Jan 2025). The paper claims that RE-WILD outperforms the recent Tulu-3 SFT mixture from Allen AI while using only 40% as many samples (Feuer et al., 30 Jan 2025).

6. Governance, caveats, and interpretation

WildChat-50M is accompanied by explicit governance constraints. The paper states that dataset access will require approval, that release will occur under the AI2 ImpACT License, and that upstream model-license restrictions also apply (Feuer et al., 30 Jan 2025). It also warns that the dataset may contain toxic, offensive, sexually explicit, violent, bigoted, or otherwise disturbing content, and that original user inputs are preserved without modification, so some conversations may inadvertently contain personal information despite prior safeguards (Feuer et al., 30 Jan 2025).

These concerns connect directly to earlier and later analyses of the broader WildChat family. The original WildChat release anonymized conversation text with Microsoft Presidio, spaCy, and custom rules and released hashed IPs, country, state, and request-header information under AI2 licensing (Zhao et al., 2024). However, later privacy analysis on English WildChat samples argued that PII detection alone is insufficient to capture the sensitive topics common in human–LLM interaction and reported that detected PII remained highly prevalent even after prior redaction (Mireshghallah et al., 2024). A separate longitudinal comparison using WildChat-4.8M argued that WildChat is significantly skewed toward highly proficient “power” users and contains a substantial amount of API-like or templated activity, rather than only natural conversational use (Hicke et al., 27 May 2026). This suggests that WildChat-50M should not automatically be treated as a faithful proxy for average mainstream chatbot behavior.

A second interpretive caveat concerns scope. Although the WildChat-50M abstract refers to “large-scale comparative analyses of synthetic data generating models and LLM judges,” the main body is much richer on DGM comparisons than on systematic judge benchmarking (Feuer et al., 30 Jan 2025). LLM-judged benchmarks such as MTBench and AlpacaEval2 are used, and GPT-4o-mini is explicitly used for MTBench judging, but the paper does not provide a standalone judge-analysis section with agreement studies or a general judge-aggregation formalism (Feuer et al., 30 Jan 2025).

A third caveat concerns the broader WildChat ecosystem. Many later works build filtered or transformed derivatives from the original WildChat family rather than directly from WildChat-50M. WildClaims, for example, begins from English, non-math, non-coding WildChat conversations and constructs a 3,000-conversation derivative resource with 121,905 extracted factual claims (Joko et al., 22 Sep 2025). A separate routing benchmark constructs 3,000 prompts over a fixed 12-agent catalog from public WildChat prompts and explicitly warns that its rebalanced evaluation set is not distribution-faithful to natural WildChat usage (Bala et al., 27 Jun 2026). RLHI-based work similarly builds derived artifacts such as WildLlamaChat and WildChat UserEval from WildChat-1M rather than using a 50M-scale release directly (Jin et al., 29 Sep 2025). These derivative studies underscore a recurring pattern: WildChat-derived resources often prioritize controlled task construction over raw distributional fidelity.

In aggregate, WildChat-50M is best understood as infrastructure for open synthetic-data research in post-training. Its distinctive contribution is not merely that it is large, but that it aligns a real-world prompt distribution with many open-weight generators, enabling controlled study of synthetic data quality, throughput, stylistic inheritance, and downstream supervised fine-tuning performance (Feuer et al., 30 Jan 2025). Its most durable claim is that post-training quality depends strongly on which model generated the synthetic responses, and that this dependence is not well captured by model scale alone.