---
title: 'NB-WildChat: LLM Benchmarking Suite'
url: https://www.emergentmind.com/topics/nb-wildchat
type: topic
---

# NB-WildChat: LLM Benchmarking Suite

NB-WildChat is a suite of benchmarks and evaluation strategies centered on open-ended and multi-agent language understanding, response diversity, and cost-aware routing, addressing critical post-training challenges for large language models (LLMs) using both real-world and synthetic conversational data. NB-WildChat datasets and methodologies are closely associated with the WildChat[2405.01470] and WildChat-50M[2501.18511] corpora, providing reproducible, diagnostic testbeds for LLM selection, tool-use prediction, and synthetic data quality evaluation.

## 1. Dataset Foundations and Construction

NB-WildChat inherits its core data from WildChat[2405.01470] and WildChat-50M[2501.18511]. WildChat contains 1 million multi-turn ChatGPT conversations, with richly diverse, opt-in user queries spanning open-ended requests, coding problems, factual information, tips, jokes, and language learning. WildChat-50M further extends this resource to 50 million prompt-response pairs using outputs from over 50 open-weight LLMs with model sizes ranging from 0.5B to 104B parameters.

The NB-WildChat benchmarks utilize stratified sampling techniques to ensure balanced coverage across prompt difficulty (quantified via LLM perplexity) and diverse topics: world knowledge, coding, mathematics, creative writing, opinion, and miscellaneous. For diagnostic benchmarking, NB-WildChat includes:

- Open-ended prompts supporting infinite valid answers (e.g., mnemonic generation)
- Multi-agent routing scenarios (e.g., tool selection for complex tasks)
- Behavioral filtering (removal of policy-violating prompts and separation of edge cases)

A typical NB-WildChat dataset instance consists of either:
- 1,000 open-ended prompts (for diversity coverage tasks)[2604.02319]
- 3,000 multi-agent prompts with fixed agent catalogs for set-valued routing evaluations[2606.28925]
- Stratified 5 million prompt-response pairs with per-prompt responses from up to 10 LLMs, for synthetic data benchmarking and per-prompt teacher variance analysis[2501.18511]

## 2. Response Diversity and Coverage Metrics

A central feature of NB-WildChat is its evaluation of model output diversity via the **diversity coverage** (DivCov) metric[2604.02319]. For a query $q$ and a model’s predicted answer set $A_{\mathrm{pred}}$ (with sample budget $B$), DivCov formalizes joint assessment of *distinctiveness* and *quality* of responses:

\[
\mathrm{DivCov}(A_{\mathrm{pred}}) = 
\frac{1}{\max_{|A|=B} \sum_{a\in A} Q(a)}
\sum_{a\in\mathrm{uniq}(q, A_{\mathrm{pred}})} Q(a)
\]

- $Q(a)$ is the reward-model-derived quality score for answer $a$, linearly mapped to $[1,10]$ using Skywork‐Reward‐Gemma‐2-27B.
- $\mathrm{uniq}(q, A)$ deduplicates responses using an equivalence classifier trained from 1,100 labeled answer pairs.
- The denominator represents the “oracle” sum for $B$ best distinct answers.

Empirical studies on NB-WildChat show that no single LLM dominates in DivCov across prompts, but per-prompt optimal LLMs exist, motivating route-by-query model selection. The highest single-model DivCov is 23.8%; the per-query oracle achieves 33.0%[2604.02319].

## 3. Multi-Agent Routing and Cost-Aware Evaluation

NB-WildChat supports structured multi-agent routing experiments[2606.28925]. In these, each prompt $x$ requires a subset $G(x)$ from a catalog of 12 agents. The benchmark dataset is balanced to ensure stable multi-label evaluation (50% require one agent, 33% two, 10% three) with rebalancing across both label count and agent frequency. 

Set-valued prediction metrics include:

- Precision: $P = \frac{|\hat{Y}\cap Y|}{|\hat{Y}|}$
- Recall: $R = \frac{|\hat{Y}\cap Y|}{|Y|}$
- F1-score: $F1 = 2\frac{P\,R}{P+R}$
- Jaccard index: $J = \frac{|\hat{Y}\cap Y|}{|\hat{Y}\cup Y|}$
- Exact Match (EM): binary indicator for $\hat{Y}=Y$

Cost-sensitive performance is captured by a utility function:

\[
U(\hat{S}) = \frac{|G\cap \hat{S}|}{|G|} - 0.10\sum_{a\in\hat{S}} c(a) - 0.05|\hat{S}\setminus G|
\]

Here, $c(a)$ is the ordinal cost tier of agent $a$.

Weighted Agent Routing (WAR) enables decision-time cost–coverage trade-offs without retraining: for per-agent scores $s_i(x)$, scores are adjusted as $\tilde{s}_i(x) = s_i(x) - \lambda c(a_i)$, with $\lambda$ determined by dev-set sweeps.

Supervised routers—especially a fine-tuned MPNet encoder—achieve the highest set-valued accuracy (F1 = 89.6%, utility = 0.654) and, when augmented with WAR, further improve utility (to 0.670) and success rate (92.1%) at low latency (49 ms/query)[2606.28925].

## 4. Routing Architectures and Algorithmic Approaches

NB-WildChat benchmarks a variety of routing strategies for both model and agent selection[2604.02319][2606.28925]:

- **Embedding-based K-nearest-neighbors** (KNN): Using model-agnostic (inf-retriever-v1) or model-specific LLM embeddings.
- **Fine-tuned classifiers**:
  - Multi-way MLP for softmax over candidate models
  - 18-way binary MLP for per-model best-prediction
  - BERT fine-tuned for model selection
  - One-vs-rest linear SVM for agent selection
  - Classifier Chains and ML-kNN for label dependency
  - Fine-tuned MPNet encoder for multi-agent multilabel prediction
- **Prompt-based routing**: Zero-shot LLM approaches, where GPT-4o is prompted to output minimal JSON agent lists (substantially underperforming supervised methods).
- **Set construction**: All methods, except majority/zero-shot, use threshold-based set selection ($\{a_i: s_i \ge t\}$), with “top-1 fallback” to guarantee non-empty predictions.

The best model selection router (Binary MLP with model-specific encodings) achieves DivCov = 26.3% on the 1K prompt NB-WildChat (vs. 23.8% top model baseline), closing ≈20% of the oracle–baseline gap at only ~2–3x top-model inference cost[2604.02319].

## 5. Synthetic Data, Evaluation Protocols, and Recommendations

NB-WildChat also serves as a synthetic data benchmarking suite, leveraging the scale and variance of the WildChat-50M resource[2501.18511]. The recommended protocol samples prompts stratified by difficulty and topic, filters policy-violating content, and ensures response diversity by sampling from 10 representative data-generating models (DGMs).

Key evaluation practices include:

- Reporting both “ground-truth” and LLM-judge (GPT-4o-mini, Claude-3.5-Sonnet) scores with bootstrap confidence intervals
- Using per-prompt teacher variance analysis to study synthetic data impacts
- Mixing small sets of auxiliary ground-truth data (e.g., MMLU) to anchor factuality
- Ship datasets with reproducible code for data loading, SFT-fine-tuning, embedding computation, and evaluation (e.g., via Evalchemy)

Empirical ablations reveal that DGM heterogeneity provides limited marginal benefit beyond prompt diversity; parameter count is a poor proxy for synthetic data quality. The NB-WildChat design thus prioritizes prompt stratification and multi-judge evaluation.

## 6. Applications, Limitations, and Future Directions

NB-WildChat is applicable to benchmarking LLM sample diversity, optimizing model selection/routing strategies, evaluating synthetic data post-training regimes, and studying the accuracy–cost trade-offs in set-valued decision making. Its structure supports systematic advances in:

- Diversity-oriented LLM benchmarking without finite “gold lists”
- Multi-agent routing and capability-coverage simulations under constrained resource scenarios
- Diagnostics for instruction tuning, safety, and policy adherence with real and synthetic data

Future directions highlighted include scaling agent catalogs ($M\gg 12$), leveraging real invocation costs, moving from simulation to true user studies, and developing adaptive cost/coverage trade-offs using bandit or RL techniques.

NB-WildChat thus provides a reproducible, extensible foundation for the study of compositionality, diversity, routing accuracy, and efficiency in real-world LLM systems and post-training evaluation[2604.02319][2501.18511][2606.28925].

Source: https://www.emergentmind.com/topics/nb-wildchat