- The paper introduces GROVE, a graph-based visualization that represents LM outputs as merged token paths to analyze distributional diversity and failure modes.
- It utilizes interactive D3 force simulation and token clustering to compactly summarize outputs and support dynamic comparative analysis.
- Empirical studies show that while graph views improve high-level diversity assessments, list views yield better accuracy for detail-oriented comparisons.
Distributional Visualization of LLM Outputs: Summary and Analysis of GROVE
The paper addresses a fundamental challenge in LM evaluation: the inadequacy of single-sample inspection for capturing the true stochastic nature of generative models. It highlights how practitioners are forced to reason about distributions—central tendency, spread, modes, and outliers—yet have minimal tooling for visualizing and analyzing such distributions effectively. Through formative interviews with 13 researchers, the work characterizes the ambiguity of 'distribution' in natural language and the limitations of existing metrics, revealing demands for tools supporting both qualitative sensemaking and iterative hypothesis testing.
Key findings indicate that practitioners treat LMs as infrastructural components, often out of necessity rather than preference. Distributional failures—mode collapse, semantic redundancy, and culturally homogenized outputs—were noted across domains (synthetic data, creative writing, reasoning traces). Practitioners require within-input diversity for creative tasks, controlled consistency for reliability, and ability to identify failure modes and modes of convergence/divergence. Manual inspection scales poorly, hampered by cognitive overload and unstructured lists, motivating a more compact and structured interface.
GROVE: System Architecture and Design Principles
Building on these requirements, the authors introduce GROVE, an interactive visualization system representing multiple generations as overlapping token paths in a merged graph. The token graph is constructed by tokenizing outputs (space, sentence, phrase), merging semantically similar tokens using local embedding similarity, and collapsing linear chains to reduce visual clutter. Each output corresponds to a unique path, preserving sequence information and avoiding spurious recombinations.
The layout employs a D3 force simulation for spatial ordering; horizontal positions are based on parent nodes, vertically centered by frequency, and collided to avoid overlap. Node color encodes prompt provenance, and node size reflects generation frequency. Users interactively filter outputs by token, adjust simplification parameters (hiding long-tail, tuning merge thresholds), and toggle raw-text views.
Comparison mode enables merged or side-by-side visualization of outputs from different prompts, models, or decoding settings. The key design goals are: compact summarization (DG1), surfacing shared/diverging structure (DG2–DG3), supporting comparative analysis (DG4), retaining raw access (DG5), and facilitating rapid iterations (DG6).
Empirical Evaluation: User Study Outcomes
Three controlled experiments (N=47, 44, 40) focus on complementary distributional tasks: diversity comparison, single-distribution comprehension, and two-distribution comparison. The primary manipulation is presentation mode—graph view versus scrollable list view. Objective measures include accuracy and completion time; subjective measures probe workload, usability, and preference.
Strong Numerical Results:
- Diversity comparison: Graph view yields significantly higher accuracy (mean Δ = 0.12, p = 0.012) and faster completion, with strong preference for graph.
- Single-distribution comprehension: List view outperforms graph in accuracy (mean Δ = -0.057, p = 0.009); preferences are polarized.
- Two-distribution comparison: List view again outperforms graph (mean Δ = -0.10, p = 0.002); preferences are distributed.
Participants consistently favored the graph for high-level diversity assessments; list view was optimal for detail-oriented, granular comparison tasks. The combined interface received positive qualitative feedback (>80% favorable), indicating a hybrid need for summary and exhaustive inspection.
Practical and Theoretical Implications
The study formalizes the significance of distributional reasoning in LM workflows, exposing gaps in standard evaluation paradigms. Visualization of output clusters, modes, and divergences enables rapid identification of mode collapse, template repetition, and distributional artifacts—phenomena linked to cultural homogenization [10.1145/3706598.3713564], open-ended homogeneity (Jiang et al., 27 Oct 2025), and non-humanlike output [reinhart2025LMs]. GROVE's graph encoding is particularly effective for temporally/syntactically aligned outputs and domains with strong modal structure; effectiveness diminishes for long, highly divergent outputs ('hairball' phenomenon).
This research demonstrates that visual encodings outperform baseline list inspection for distributional sensemaking, but only for certain tasks and output regimes. It reinforces prior findings on the need for 'mesoscale' inspection [10.1145/3613904.3642139] and expands on clustering approaches in text visualization [Sevastjanova2023-lq]. The approach offers an analytic lens for LM researchers to investigate homogeneity, uncover rare modes, and audit reliability in open-ended generative settings.
Limitations and Directions for Future Research
Several limitations are noted: lab-only evaluation, hairball complexity in large/divergent samples, reliance on finite output sampling, and lack of integration with quantitative distributional metrics. The graph invariant (one path per output) constrains aggregation; relaxing this or integrating token-level probabilities may enhance scalability and interpretability.
Further work could explore longitudinal deployment, notebook integration, domain-specific pipelines, and hybrid metrics bridging qualitative visualizations with quantitative measures (semantic uncertainty [kuhn2023semantic], novelty (Zhang et al., 7 Apr 2025)). Investigating which properties of text distributions best inform visualization encodings remains essential—particularly for operationalizing diversity, consistency, and reliability in LM-driven workflows.
Conclusion
This paper provides a rigorous characterization of distributional reasoning needs in LM practice and introduces GROVE, a graph-based visualization harnessing structural overlap to summarize, compare, and analyze output distributions. Empirical results validate the tool's advantages for rapid diversity assessment, with detail-focused tasks favoring list inspection. The research advances the interface design of generative AI systems, advocating for hybrid visualization paradigms that expose the underlying stochastic structures crucial for robust LM evaluation and prompt iteration (2604.18724).
