NB-WildChat: LLM Benchmarking Suite

Updated 2 July 2026

NB-WildChat is a benchmark suite for open-ended and multi-agent language evaluation that uses both real-world and synthetic conversational data.
It employs stratified sampling and multi-turn conversation datasets to assess response diversity, diagnostic metrics like DivCov, and cost-aware model routing.
The suite integrates various routing strategies, including embedding-based methods and fine-tuned classifiers, to optimize LLM selection and tool-use prediction.

NB-WildChat is a suite of benchmarks and evaluation strategies centered on open-ended and multi-agent language understanding, response diversity, and cost-aware routing, addressing critical post-training challenges for LLMs using both real-world and synthetic conversational data. NB-WildChat datasets and methodologies are closely associated with the WildChat(Zhao et al., 2024) and WildChat-50M(Feuer et al., 30 Jan 2025) corpora, providing reproducible, diagnostic testbeds for LLM selection, tool-use prediction, and synthetic data quality evaluation.

1. Dataset Foundations and Construction

NB-WildChat inherits its core data from WildChat(Zhao et al., 2024) and WildChat-50M(Feuer et al., 30 Jan 2025). WildChat contains 1 million multi-turn ChatGPT conversations, with richly diverse, opt-in user queries spanning open-ended requests, coding problems, factual information, tips, jokes, and language learning. WildChat-50M further extends this resource to 50 million prompt-response pairs using outputs from over 50 open-weight LLMs with model sizes ranging from 0.5B to 104B parameters.

The NB-WildChat benchmarks utilize stratified sampling techniques to ensure balanced coverage across prompt difficulty (quantified via LLM perplexity) and diverse topics: world knowledge, coding, mathematics, creative writing, opinion, and miscellaneous. For diagnostic benchmarking, NB-WildChat includes:

Open-ended prompts supporting infinite valid answers (e.g., mnemonic generation)
Multi-agent routing scenarios (e.g., tool selection for complex tasks)
Behavioral filtering (removal of policy-violating prompts and separation of edge cases)

A typical NB-WildChat dataset instance consists of either:

1,000 open-ended prompts (for diversity coverage tasks)(Liu et al., 2 Apr 2026)
3,000 multi-agent prompts with fixed agent catalogs for set-valued routing evaluations(Bala et al., 27 Jun 2026)
Stratified 5 million prompt-response pairs with per-prompt responses from up to 10 LLMs, for synthetic data benchmarking and per-prompt teacher variance analysis(Feuer et al., 30 Jan 2025)

2. Response Diversity and Coverage Metrics

A central feature of NB-WildChat is its evaluation of model output diversity via the diversity coverage (DivCov) metric(Liu et al., 2 Apr 2026). For a query $q$ and a model’s predicted answer set $A_{\mathrm{pred}}$ (with sample budget $B$ ), DivCov formalizes joint assessment of distinctiveness and quality of responses:

$\mathrm{DivCov}(A_{\mathrm{pred}}) = \frac{1}{\max_{|A|=B} \sum_{a\in A} Q(a)} \sum_{a\in\mathrm{uniq}(q, A_{\mathrm{pred}})} Q(a)$

$Q(a)$ is the reward-model-derived quality score for answer $a$ , linearly mapped to $[1,10]$ using Skywork‐Reward‐Gemma‐2-27B.
$\mathrm{uniq}(q, A)$ deduplicates responses using an equivalence classifier trained from 1,100 labeled answer pairs.
The denominator represents the “oracle” sum for $B$ best distinct answers.

Empirical studies on NB-WildChat show that no single LLM dominates in DivCov across prompts, but per-prompt optimal LLMs exist, motivating route-by-query model selection. The highest single-model DivCov is 23.8%; the per-query oracle achieves 33.0%(Liu et al., 2 Apr 2026).

3. Multi-Agent Routing and Cost-Aware Evaluation

NB-WildChat supports structured multi-agent routing experiments(Bala et al., 27 Jun 2026). In these, each prompt $x$ requires a subset $A_{\mathrm{pred}}$ 0 from a catalog of 12 agents. The benchmark dataset is balanced to ensure stable multi-label evaluation (50% require one agent, 33% two, 10% three) with rebalancing across both label count and agent frequency.

Set-valued prediction metrics include:

Precision: $A_{\mathrm{pred}}$ 1
Recall: $A_{\mathrm{pred}}$ 2
F1-score: $A_{\mathrm{pred}}$ 3
Jaccard index: $A_{\mathrm{pred}}$ 4
Exact Match (EM): binary indicator for $A_{\mathrm{pred}}$ 5

Cost-sensitive performance is captured by a utility function:

$A_{\mathrm{pred}}$ 6

Here, $A_{\mathrm{pred}}$ 7 is the ordinal cost tier of agent $A_{\mathrm{pred}}$ 8.

Weighted Agent Routing (WAR) enables decision-time cost–coverage trade-offs without retraining: for per-agent scores $A_{\mathrm{pred}}$ 9, scores are adjusted as $B$ 0, with $B$ 1 determined by dev-set sweeps.

Supervised routers—especially a fine-tuned MPNet encoder—achieve the highest set-valued accuracy (F1 = 89.6%, utility = 0.654) and, when augmented with WAR, further improve utility (to 0.670) and success rate (92.1%) at low latency (49 ms/query)(Bala et al., 27 Jun 2026).

4. Routing Architectures and Algorithmic Approaches

NB-WildChat benchmarks a variety of routing strategies for both model and agent selection(Liu et al., 2 Apr 2026, Bala et al., 27 Jun 2026):

Embedding-based K-nearest-neighbors (KNN): Using model-agnostic (inf-retriever-v1) or model-specific LLM embeddings.
Fine-tuned classifiers:
- Multi-way MLP for softmax over candidate models
- 18-way binary MLP for per-model best-prediction
- BERT fine-tuned for model selection
- One-vs-rest linear SVM for agent selection
- Classifier Chains and ML-kNN for label dependency
- Fine-tuned MPNet encoder for multi-agent multilabel prediction
Prompt-based routing: Zero-shot LLM approaches, where GPT-4o is prompted to output minimal JSON agent lists (substantially underperforming supervised methods).
Set construction: All methods, except majority/zero-shot, use threshold-based set selection ( $B$ 2), with “top-1 fallback” to guarantee non-empty predictions.

The best model selection router (Binary MLP with model-specific encodings) achieves DivCov = 26.3% on the 1K prompt NB-WildChat (vs. 23.8% top model baseline), closing ≈20% of the oracle–baseline gap at only ~2–3x top-model inference cost(Liu et al., 2 Apr 2026).

5. Synthetic Data, Evaluation Protocols, and Recommendations

NB-WildChat also serves as a synthetic data benchmarking suite, leveraging the scale and variance of the WildChat-50M resource(Feuer et al., 30 Jan 2025). The recommended protocol samples prompts stratified by difficulty and topic, filters policy-violating content, and ensures response diversity by sampling from 10 representative data-generating models (DGMs).

Key evaluation practices include:

Reporting both “ground-truth” and LLM-judge (GPT-4o-mini, Claude-3.5-Sonnet) scores with bootstrap confidence intervals
Using per-prompt teacher variance analysis to study synthetic data impacts
Mixing small sets of auxiliary ground-truth data (e.g., MMLU) to anchor factuality
Ship datasets with reproducible code for data loading, SFT-fine-tuning, embedding computation, and evaluation (e.g., via Evalchemy)

Empirical ablations reveal that DGM heterogeneity provides limited marginal benefit beyond prompt diversity; parameter count is a poor proxy for synthetic data quality. The NB-WildChat design thus prioritizes prompt stratification and multi-judge evaluation.

6. Applications, Limitations, and Future Directions

NB-WildChat is applicable to benchmarking LLM sample diversity, optimizing model selection/routing strategies, evaluating synthetic data post-training regimes, and studying the accuracy–cost trade-offs in set-valued decision making. Its structure supports systematic advances in:

Diversity-oriented LLM benchmarking without finite “gold lists”
Multi-agent routing and capability-coverage simulations under constrained resource scenarios
Diagnostics for instruction tuning, safety, and policy adherence with real and synthetic data

Future directions highlighted include scaling agent catalogs ( $B$ 3), leveraging real invocation costs, moving from simulation to true user studies, and developing adaptive cost/coverage trade-offs using bandit or RL techniques.

NB-WildChat thus provides a reproducible, extensible foundation for the study of compositionality, diversity, routing accuracy, and efficiency in real-world LLM systems and post-training evaluation(Liu et al., 2 Apr 2026, Feuer et al., 30 Jan 2025, Bala et al., 27 Jun 2026).