LLMs as Statistical Models

• LLMs as statistical models are deep neural architectures that assign probability distributions over token sequences through likelihood maximization.
• They employ autoregressive sampling and uncertainty quantification methods to elicit full numerical predictive distributions for varied inferential tasks.
• LLMs are integrated into broader statistical pipelines, necessitating careful calibration, alignment, and robust evaluation for reliable and interpretable outputs.

LLMs are fundamentally statistical systems that assign probability distributions over discrete token sequences and are trained using likelihood-based objectives. Their generative architecture, immense data dependency, and intrinsic stochasticity position them as modern incarnations of high-dimensional statistical models, distinguished from classical counterparts by both scale and operational modality. The statistical treatment of LLMs—encompassing model characterization, uncertainty quantification, downstream inferential use, and control—has progressed rapidly, parallel to their adoption in domains historically dominated by explicit probabilistic models.

1. The Statistical Foundation of LLMs

LLMs parameterize a probability distribution $P_\theta$ over token sequences $(w_1, \ldots, w_T)$ such that

$$P_\theta(w_1,\ldots,w_T) = \prod_{t=1}^T P_\theta(w_t|w_{<t})$$

where each conditional is a multinomial over the vocabulary $V$, computed by propagating the prefix $w_{<t}$ through a deep neural architecture (typically a Transformer). The training objective is to maximize data likelihood or, equivalently, minimize cross-entropy:

$$L(\theta) = -\sum_{i=1}^N \sum_{t=1}^{T_i} \log P_\theta(w_{i,t}|w_{i,<t})$$

At generation, sampling is stochastic: $w_t \sim P_\theta(\cdot|w_{<t})$, potentially with temperature scaling or nucleus/top-k filtering to modulate diversity. This inherent randomness is exploited for creative text production and is a defining feature differentiating LLMs from deterministic programmatic systems (Su, 25 May 2025).

Classical statistical paradigms such as Bayesian updating or likelihood-based inference are challenging to apply directly due to LLMs’ black-box character and high complexity (billions of parameters, intricate pretraining distributions, and nonlinear interactions inaccessible to full mechanistic analysis). Therefore, statistical methodologies—uncertainty quantification, estimation theory, latent variable modeling—are essential for both understanding and operationalizing LLMs (Su, 25 May 2025).

2. Eliciting and Utilizing Numerical Predictive Distributions

Unlike traditional regression or generative models, LLMs can be elicited to produce full numerical predictive distributions at arbitrary query points, conditional on both data and free-form text encoding prior knowledge. The 'LLM Process' ("LLMP") formalism defines a stochastic process via joint densities

$P(y^*_{1:N} | x^*_{1:N}, D, T)$

with $D = \{(x_i, y_i)\}_{i=1}^M$ and $T$ encoding natural language priors. Two constructions are central (Requeima et al., 2024):

• Independent-Marginal LLMP (I-LLMP):

$$P_I(y^*_{1:N}|x^*_{1:N},D,T) = \prod_{n=1}^N P(y^*_n|x^*_n, D, T)$$

Yielding exchangeable, Kolmogorov consistent inference.

• Autoregressive LLMP (A-LLMP):

$$P_A(y^*_{1:N}|x^*_{1:N},D,T) = \prod_{n=1}^N P(y^*_n | y^*_{1:n-1}, x^*_{1:n}, D, T)$$

Capturing joint dependencies but not exchangeability.

Numbers are elicited either via repeated sampling—producing an empirical measure—or via logit-based continuous likelihoods: fixed-precision numeric strings are parsed to retrieve the model’s token log-probabilities, then aggregated to yield a pseudo-PDF. Best-practices include explicit end-of-number delimiters and calibration of prompt separators and scaling to ensure coherence. Empirically, such recipes enable LLMs to exceed classical Gaussian process baselines in regression, forecasting, black-box optimization, and image completion (Requeima et al., 2024).

Natural language context $T$ modulates priors over functions, with prompt engineering demonstrably shaping extrapolations (e.g., enforcing plausible physical bounds, seasonality, or economic constraints) without explicit parameterization (Requeima et al., 2024).

3. LLMs in Statistical Estimation, Measurement, and Causal Inference

LLMs’ output stochasticity exposes systematic measurement error when viewed as automated decision instruments. For example, when classifying text, repeated LLM queries yield a distribution of responses, with non-negligible false positive and false negative rates. A Bayesian latent state model reframes each LLM output as a noisy observation of an unobserved ground-truth label $Y_i$, with inferable error rates ($\varepsilon_0$, $\varepsilon_1$) and population parameters ($\theta$) (Zhang et al., 27 Oct 2025):

• Each data point $i$ is queried $K$ times, with binary outcomes $X_{ik}$.
• Model likelihood for true state $Y_i$ incorporates error parameters; posterior inference (via MCMC) recovers calibrated estimates of $\theta$, $\varepsilon_0$, and $\varepsilon_1$ with full uncertainty quantification.
• In simulation, such latent models outperform common heuristics (e.g., majority voting) in parameter recovery, producing posterior credible intervals that propagate LLM-induced measurement uncertainty into downstream models.

For quantitative data analysis, benchmarks such as QRDATA show that LLMs internalize formulas for means, correlations, regression coefficients, and hypothesis tests, but lack robustness in application—systematic error occurs in equation selection, arithmetic, and table interpretation. Reliable statistical estimation (with valid confidence intervals or error bars) is not achieved: output variance across prompts and seeds is high, and confidence intervals cannot be guaranteed through LLM-intrinsic mechanisms (Liu et al., 2024).

Causal reasoning remains a challenge: even advanced models (e.g., GPT-4 with code execution) achieve only ≈51% accuracy on causal inference tasks, and performance degrades when required to synthesize causal reasoning with tabular data. LLMs display better recall of causal “commonsense” in text-only scenarios than in data-linked settings, underscoring limitations as unconditional statistical estimators (Liu et al., 2024).

4. Time Series and Portfolio Modeling: Sequence, Signal, and Limitation

LLMs have been adapted to time series forecasting by framing numerical prediction as a sequence modeling task—encoding observations as quantized tokens and eliciting future values from the model distribution. The LLMTIME approach treats next-point prediction as conditional next-token generation, using scale/offset normalization and digit-level tokenization. While this enables coherent generation for strictly periodic signals, performance declines markedly with trend components or multifrequency structures. Classical statistical models such as ARIMA and ETS provide lower mean squared error (MSE) and mean absolute error (MAE) across a broad spectrum of real and synthetic benchmarks, reflecting superior inductive biases for trend and seasonality (Cao et al., 2024).

Advanced models such as Chronos (a patched-decoder T5 variant) can be fine-tuned online in financial forecasting contexts. Chronos achieves gross Sharpe ratios surpassing naively applied ARIMA, and is able to transfer pretraining-acquired pattern recognition to near-random data (e.g., equity residual returns). It does not, however, surpass models explicitly tailored to mean-reversion or low-parameter convolutional architectures, indicating that full LLM capacity is not yet systematically exploitable without domain-specific curriculum or further architectural refinement (Valeyre et al., 2024).

Model/Task	Setting/Dataset	Metric (Best)	Key Limitation
LLMP (A-LLMP)	1D Regression, Forecasting	NLL, MAE: Beats GP 10/12 sets	Context length, computation, black-box priors
LLMTIME	Univariate time series	MSE/MAE ≫ ARIMA (trends)	Poor for trend/multifrequency signals
Chronos (LLM)	US stocks returns, PCA factors	SR ≈ 3.97 (volatility-resized)	Underperforms mean-reversion CNN-Transformer

LLMs’ utility in time series and portfolio construction arises from autoregressive structure, pretraining transfer, and in-context meta-learning, but is circumscribed by unaddressed inductive bias and architectural idiosyncrasy (Requeima et al., 2024, Valeyre et al., 2024, Cao et al., 2024).

5. Statistical Alignment, Security, and Evaluation

LLMs’ outputs are stochastic measures subject to statistical alignment interventions. Preference alignment (e.g., RLHF) is conceptualized through statistical modeling, e.g., fitting a Bradley–Terry model on pairwise human preference data and regularizing generation policies by KL-divergence to ensure the output distribution aligns with reference-safe behaviors. Lower bounds on 'jailbreaking' success—the probability an adversary can induce a harmful response despite alignment—are derivable as unavoidable, even as improved alignment objectives (e.g., E-RLHF, substituting safety-modified prefixes for KL anchors) statistically reduce attack rates (Su et al., 2024).

Statistical tools underpin watermarking (as formal hypothesis testing on next-token distributions), uncertainty quantification (conformal prediction, entropy-based risk measures), calibration, and robust evaluation (bootstrap CI, item response theory). As LLMs remain opaque at the mechanistic layer, these statistical controls furnish the only tractable framework for safety, fairness, and interpretability (Su, 25 May 2025).

6. LLMs as Components of Broader Statistical Models

LLM outputs are increasingly harnessed as features or predictors within formal statistical pipelines. In Bayesian frameworks, LLM-generated scores (e.g., log-probabilities of candidate responses) serve as pseudo-likelihoods or predictors for human response distributions. Aggregation strategies (e.g., average-scores, average-probabilities, or winner-take-all across items) critically influence fit to empirical data, with aggregate-level modeling providing better alignment to human choice frequencies than item-level modeling, which overstates variance and idiosyncrasy. Nevertheless, LLMs’ fit depends on protocol, aggregation, and calibration; they do not constitute generative models of human behavior de novo but can serve as proxy summary statistics under careful regime selection (Franke et al., 2024).

7. Outlook and Specialized Statistical Research Directions

The diversity and scale of LLM architectures preclude a unifying statistical theory. Instead, the field is developing as a mosaic of specialized statistical subdomains: alignment, provable watermarking, uncertainty quantification, principled evaluation, data mixture optimization, and causal inference. Each area requires methodological innovation, capturing the idiosyncrasies of black-box, high-variance stochastic generative systems. Statistical engagement is necessary for guiding LLM applications toward fairness, trustworthiness, and actionable uncertainty. As LLMs permeate new application domains, early adoption of statistical frameworks will underwrite reliable, interpretable, and robust deployed models (Su, 25 May 2025).