- The paper introduces a novel sample-aware framework that decomposes non-determinism using Factor Variance Attribution (FVA).
- It demonstrates that aggregate metrics mask critical sample-level variability and error diversity induced by both model and system factors.
- The evaluation reveals task-dependent non-determinism with code generation showing notably higher variability than question answering.
Fine-Grained Non-Determinism Evaluation in Diffusion LLMs
Introduction
This paper addresses a critical and under-explored aspect of diffusion LLMs (DLMs): the prevalence and structure of non-determinism during inference. While the stochasticity inherent to DLMs is recognized, existing evaluations predominantly employ dataset-level metrics (e.g., accuracy, pass@k), which aggregate prediction quality across inputs and runs. These aggregate metrics systematically mask input-conditional variability and error patterns, potentially misrepresenting model reliability, especially in applications demanding reproducibility and robust decision-making.
Limitations of Dataset-Level Metrics
The authors demonstrate that dataset-level metrics attenuate the apparent non-determinism by averaging out correctness flips and prediction inconsistencies at the sample level. Configurations with nearly identical aggregate scores can yield distinct predictions for individual inputs, exhibiting divergent error modes and failure paths. The empirical evidence shows that sample-level flip rates and error diversity persist across inference configurations, even when aggregate metrics convey stability. As such, dataset-level evaluation protocols are inadequate for characterizing the true behavior and reliability of DLMs.
Fine-Grained Evaluation Paradigm
To address these shortcomings, the authors introduce a sample-aware and factor-aware evaluation framework:
- Sample-level analysis: Each input is evaluated across multiple inference-time configurations, explicitly tracking prediction correctness and error diversity.
- Factor and setting decomposition: Evaluation variability is attributed to (i) factor-level effects (dimensions of the evaluation protocol, such as guidance scale or diffusion steps), and (ii) setting-level sensitivity (specific parameter choices within each factor).
The core metric proposed is Factor Variance Attribution (FVA), which decomposes the observed variability into variance attributable to factor identity versus variance within factor settings. This variance-based analytic approach enables rigorous quantification of non-determinism sources and their task-dependent impact.
Empirical Analysis and Key Findings
The study systematically varies both model-related (e.g., classifier-free guidance scale, diffusion steps, MC sampling) and system-related (e.g., batch size, GPU type, numerical precision) factors. Strong empirical findings include:
- Sample-level variability: Pronounced sample-level prediction variability is induced by both model and system factors, with standard deviations typically in the range 0.30–0.50, even as dataset-level accuracy standard deviations are nearly zero.
- MC sampling: Increasing the number of MC samples stabilizes mean estimates but does not monotonically reduce sample-level variability, indicating the presence of structured, input-dependent stochastic effects beyond mere noise averaging.
- System effects: Changes in precision, GPU architecture, or batch size can introduce non-determinism comparable in magnitude to stochastic model factors, underscoring the role of hardware and numerical execution in evaluation reliability.
Task Dependency
A marked task dependency is observed: code generation tasks (HumanEval, MBPP) exhibit substantially higher sensitivity to both factor selection and setting, as quantified by FVA, compared to question answering tasks (PIQA, WinoGrande, ARC-Challenge). Specifically, FVA for code generation exceeds 0.79, indicating that between-factor effects dominate evaluation variability, whereas FVA for QA tasks is lower (around 0.5), reflecting more balanced contributions from within-factor variability.
Backbone Comparison
Comparing LLaDA and LLaDA-1.5 backbones, backbone improvements (e.g., variance reduction, preference optimization) can reduce overall sample-level variability but do not eliminate substantive non-determinism, particularly for highly sensitive factors such as the number of diffusion steps. The qualitative structure of variability remains consistent across backbones, with magnitudes affected by architecture and implementation choices.
Robustness of FVA
Sensitivity analyses confirm that FVA is robust to the expansion of factor ranges and alternative value choices, with minor variations (<0.03 absolute change) and consistent relative patterns across datasets and tasks. This supports the validity of FVA as an intrinsic measure of configuration-induced evaluation variability.
Comparison with Autoregressive Models
The authors extend analysis to autoregressive backbones (e.g., LLaMA-2-7B, Qwen2.5-7B). While dataset-level variability is generally smaller for autoregressive models under deterministic decoding, sample-level variability and correctness flips remain present. The manifestation of non-determinism in autoregressive models is largely hidden at the aggregate level, while visible at the sample level, thus corroborating the central thesis.
Practical and Theoretical Implications
The results decisively establish that dataset-level metrics are insufficient for reliable non-determinism assessment in DLMs and, by extension, LLMs. Fine-grained, factor-aware evaluation protocols are necessary for trustworthy benchmarking, reproducible assessment, and deployment in applications with stringent stability requirements. The findings highlight specific areas—such as code generation and hardware-induced stochasticity—where rigorous configuration control and reporting are essential.
Theoretically, the work motivates the study of structure in stochastic inference beyond noise averaging, including cross-factor interaction effects and hierarchical error decomposition. The variance-based methodologies, such as FVA, provide a principled foundation for such analyses.
Future Directions
The authors suggest extending the framework to more diverse tasks (e.g., long-horizon reasoning, structured prediction), additional model architectures, and integrating training-time sources of variability. Interaction effects between factors (synergistic or antagonistic) warrant detailed study. Joint modeling of training and inference variability could further enhance interpretability and robustness.
Conclusion
This paper provides a rigorous demonstration that dataset-level metrics systematically attenuate non-determinism in diffusion LLMs, masking substantial input-conditional variability and error diversity. Through sample-level analysis and variance decomposition—specifically Factor Variance Attribution—the authors reveal pervasive, structured non-determinism, especially in code generation, across both model-related and system-related configurations. The implications are clear: reliable, reproducible assessment of DLMs demands fine-grained, factor-aware evaluation protocols that transcend conventional aggregate metrics (2604.13413).