UnpredictaBench: A Benchmark for Evaluating Distributional Randomness in LLMs

Abstract: We introduce UnpredictaBench, an evaluation that tests the ability of LLMs to capture true underlying distributions. As LLMs are increasingly used as substitutes for other entities (e.g., for humans in economic simulations), the tendency of many models to collapse towards a single plausible answer means a failure to capture the unpredictability of real systems. Recent work on improving output diversity is insufficient for this setting: simulation requires samples that are calibrated to a target distribution, not merely varied outputs. UnpredictaBench isolates a simplified but fundamental version of this problem: sampling outcomes from individual target distributions, including canonical statistical distributions, distributions induced by stochastic programs, and natural-language scenarios that describe random processes. We introduce 448 such problems together with KS@N, a general-purpose evaluation metric that quantifies how well a model outputs approximate black-box target distributions via the Kolmogorov-Smirnov statistical test. This is the rate at which we fail to reject model samples of size N against ground-truth samples, with larger N indicating greater difficulty. Tested across open and proprietary models, we find a large spread in distributional capabilities. For instance, when models generate samples of size 100 (KS@100, our standard metric), scores range from near 0 to over 20%. No model is able to achieve over 40% at KS@100, showing significant headroom in distributional sampling as a capability. Although adding reasoning can somewhat increase scores, we find no immediate solution for this issue. UnpredictaBench shows that even simple distributional simulation remains challenging, making it a necessary first step toward using LLMs as stand-ins for complex systems.

• The paper introduces UnpredictaBench, a benchmark that uses 448 tasks spanning 40 distributions to assess the calibrated randomness of LLM outputs.
• The methodology employs robust metrics like KS@N, WDZ, and JSD to quantify support collapse, mode-seeking, and tail deviation in generated samples.
• Empirical findings reveal that even state-of-the-art models fall short of pseudo-random baselines, highlighting the urgent need for improved stochastic calibration.

UnpredictaBench: Evaluating Distributional Randomness in LLMs

Motivation and Problem Formulation

The generation of calibrated outputs reflecting authentic probability distributions remains a requirement for the increasing use of LLMs as simulators and proxies in stochastic environments (economic simulations, agent-based systems, scientific modeling). While recent work demonstrates LLMs’ ability to reason about probabilities, rigorous empirical evaluation shows systematic failures in distributional generation—such as output collapse, mode-seeking, and poor randomness—even when models can formally articulate statistical properties (Zhao et al., 8 Jan 2026, Koevering et al., 2024, Karanjai et al., 14 Oct 2025). UnpredictaBench provides an exhaustive, statistically grounded framework for benchmarking LMM distributional fidelity, isolating whether a model can sample outcomes according to the prescribed distribution rather than merely exhibit output diversity or plausible text.

UnpredictaBench introduces a suite of 448 problems spanning 40 statistical and program-induced distributions, with tasks scaffolded in code, natural language, and real-world stochastic scenario formats. This construction addresses diverse stochastic processes—unimodal, multimodal, discrete, continuous, real, and sequence/permutation-valued—probing beyond numeric prompt-response setups. Figure 1 graphically demonstrates typical model failure modes, with many models clamping output support or misrepresenting distribution shape, and only Nemotron-3-Super-120B manifesting partial adherence to multimodal requirements.

Figure 1: Most evaluated LLMs fail to reproduce target distributions, due to either insufficient distributional understanding or degeneracy into a constrained output range; Nemotron-3-Super-120B, notably, recovers multimodal structure, whereas OLMo-3-7B exhibits severe support collapse.

This benchmark addresses the deficits of prior approaches that conflate output variety with calibrated randomness or focus only on distributional reasoning rather than stochastic realization [hopkins2023can, gu-etal-2025-LLMs, gu2026illusion].

Benchmark Construction and Task Categories

UnpredictaBench’s construction pipeline (Figure 2) draws 40 canonical distributions covering the major classes from Wikipedia, with prompt generation diversified across explicit/implicit and code/text dimensions. Additionally, 50 human-curated real-world and shuffling tasks augment coverage, particularly for permutation-valued and complex stochastic system behaviors.

Figure 2: Pipeline for UnpredictaBench dataset generation: tasks are instantiated from Wikipedia-sourced distributions and hand-crafted scenarios, and evaluated by querying models 100 times and comparing to large ground-truth samples.

Bespoke categories probe distinct facets of LLMs’ stochastic expressivity:

• Text Explicit/Implicit: Sampling from named or scenario-induced distributions, requiring explicit mapping or reverse engineering of the generative process.
• Code Explicit/Implicit: Emulation of outputs for direct or indirect stochastic Python programs, introducing necessary reasoning over code semantics.
• Multimodal: Mixtures/combinations of distributions, evaluating if models can manifest complex multimodal supports without mode collapse.
• Shuffling: Permutation-level randomness; measured via Lehmer-encoded permutation marginals as a scalar proxy.
• Real-World: Simulation of nondeterminism in OS, networks, MCMC, etc., reflecting practical stochasticity demands.

The dataset comprises 398 GPT-5.4-authored and 50 human-authored tasks, balancing parameter range (concentrated/spread) and distribution family, resulting in strong coverage across statistical modalities.

Evaluation Methodology

UnpredictaBench introduces KS@N, a robust metric quantifying the match between generated and ground-truth distributions using the Kolmogorov-Smirnov hypothesis test. For each task, $N=100$ i.i.d. model samples are compared with 10,000 ground-truth draws per task; the KS test determines whether the hypothesis that the samples originate from the same distribution can be rejected at $p < 10^{-4}$. KS@N summarizes the fraction of tasks where the model ‘passes’.

Complementary metrics include debiased Wasserstein-1 Z-score (WDZ), capturing global and tail mismatches, and Jensen-Shannon Divergence (JSD), offering a density-level characterization. These metrics provide orthogonal diagnostic perspectives, reinforcing the statistical sufficiency of KS@N for calibration tasks.

Empirical Findings

Model Comparison and Distributional Failures

Cross-model evaluation on UnpredictaBench (Figure 3) reveals substantial variance: Nemotron-3 Super 120B leads with 32.64% at KS@100, yet no model exceeds 40%—well below the optimal pseudo-random baseline (100%). There is a major drop-off even among highly-tuned “frontier” models (e.g., GPT-5.4 and Claude Sonnet 4.6, 15.18% and 4.7% respectively), and some small open models (Qwen-3.5-2B: 17.67%) outperform much larger proprietary alternatives.

Figure 3: KS@100 for all models grouped by family; Nemotron-3 Super 120B demonstrates clear separation from the rest, with significant drop-off for most state-of-the-art LLMs.

Ablation by task category, distribution class, and prompt format shows that:

• Shuffling and code tasks stress LLM randomness most, with high collapse rates.
• Real-world tasks expose surficial diversity; some small models achieve high scores via narrow but uniform support, not true calibration.
• Multimodal and spread-out distributions drive the largest performance decrements, revealing poor mode maintenance and tail sampling.
• Explicit prompting improves most models’ KS@N; however, notable exceptions suggest tendency to anchor on memorized prototypes under explicit instructions.

Qualitative and Distribution-Specific Analysis

Figure 4 illustrates how model decodings (samples and logits) on Beta and Poisson-Binomial tasks fail to capture global distributional shape, with instruct-tuned variants exhibiting further suppression of diversity.

Figure 4: Llama-3.2-1B-base and -instruct: even when prompted, sample and logit mass fail to recapitulate U-shaped or complex discrete ground truth, with instruct-tuned models showing exacerbated degeneration.

Tabled evaluations across many target distributions confirm that small-support discrete distributions (Bernoulli, Categorical) are tractable, but heavy-tailed and joint-structure classes (Fréchet, Dirichlet, Negative Binomial) consistently expose calibration errors, especially at increased sample size. Cross-metric correlation analysis (Figure 5) establishes alignment between high KS@100 and utility on creativity/novelty benchmarks (CREATE, NoveltyBench), supporting the claim that UnpredictaBench’s protocol measures useful model characteristics distinct from “raw” lexical diversity.

Figure 5: KS@100 aligns tightly with external utility/creativity metrics, reinforcing the link between statistically measurable distributional fidelity and broader LLM generative quality.

Interventions: Tuning, Decoding, and Prompting

Instruction tuning shows only minor gains (Table: base vs instruct), and can degrade diversity by penalizing responses outside dense training distribution support. List output prompting enhances KS@100 by encouraging mutual exclusivity across outputs in a forward pass, but is not a panacea (see Table: list size ablation).

Increasing temperature ($T>1.0$) boosts diversity and thus KS@N for strong models, but for weak models it tends to amplify out-of-support outputs and tail deviation (JSD, WDZ). Expanding budget (more samples per prompt) improves short-horizon KS@100 but reveals deeper collapse as N increases due to sampling impoverishment.

Prompt format matters: requesting explicit distributions helps most model families, but certain models perform paradoxically better on implicit prompts due to distribution-agnostic reasoning, rather than overfit strategy under explicit instructions.

Implications for LLM Simulation and Modeling

UnpredictaBench rigorously demonstrates that current LLMs lack robust stochastic sampling capability, even for canonical and structurally simple distributions. This limitation persists across diverse architectures, scales, and training paradigms, and is not substantially remediated by common decoding strategies, alignment protocols, nor scaling laws.

Practical implications are pronounced: downstream simulation, scientific modeling, and agentic reasoning tasks relying on calibrated randomness will suffer from biased estimates, overconfident predictions, and unreliable uncertainty quantification if using off-the-shelf LLMs. The results invalidate the simplistic assumption that output diversity equates to distributional fidelity and challenge the assumption that instruction-following or high model scale alone resolves the calibration gap.

Theoretically, these findings indicate a disconnect between the discrete, next-token training objective and the realization of distributional sampling under compositional or stochastic prompting. Mechanistically, output mode collapse, support underreporting, and degeneration result more from internal representation and semantic biases than merely from suboptimal decoding noise.

Future Directions

UnpredictaBench delineates a concrete target for development: genuine stochastic modeling and sample-level calibration. Directions for progress include:

• Distribution-aware objective functions and post-training approaches that enforce output support and statistical fidelity beyond output variety.
• Dedicated calibration layers or hierarchical sampling atop standard LLM architectures.
• Benchmarks with progressively complex compositional, multidimensional, or real-world simulation tasks, scaling from this foundation.
• Exploration of in-context “random seed” conditioning for improved empirical randomness [gu2026illusion].

Given the large headroom evidenced, UnpredictaBench is anticipated to be adopted as a primary testbed for model-based simulators in science, economics, and agent modeling, as well as for diagnostic refinements in future LLM generation methods.

Conclusion

UnpredictaBench provides a statistically principled, comprehensive benchmark for LLMs' distributional generation ability. Strong empirical evidence indicates that current models are severely limited: the frontier is far from the random machine ceiling, and even best-in-class LLMs fail to support true stochastic calibration across a wide array of tasks. Success on UnpredictaBench provides a necessary condition for deployment of LLMs as stand-ins for entities and systems governed by uncertainty, and the KS@N protocol establishes a reusable standard for evaluation of distributional fidelity in generative models.