Semantic Interaction for Narrative Map Sensemaking: An Insight-based Evaluation

Abstract: Semantic interaction (SI) enables analysts to incorporate their cognitive processes into AI models through direct manipulation of visualizations. While SI frameworks for narrative extraction have been proposed, empirical evaluations of their effectiveness remain limited. This paper presents a user study that evaluates SI for narrative map sensemaking, involving 33 participants under three conditions: a timeline baseline, a basic narrative map, and an interactive narrative map with SI capabilities. The results show that the map-based prototypes yielded more insights than the timeline baseline, with the SI-enabled condition reaching statistical significance and the basic map condition trending in the same direction. The SI-enabled condition showed the highest mean performance; differences between the map conditions were not statistically significant but showed large effect sizes (d > 0.8), suggesting that the study was underpowered to detect them. Qualitative analysis identified two distinct SI approaches-corrective and additive-that enable analysts to impose quality judgments and organizational structure on extracted narratives. We also find that SI users achieved comparable exploration breadth with less parameter manipulation, suggesting that SI serves as an alternative pathway for model refinement. This work provides empirical evidence that map-based representations outperform timelines for narrative sensemaking, along with qualitative insights into how analysts use SI for narrative refinement.

• The paper demonstrates that SI-enabled narrative maps significantly increase insight yield, with high-level insights rising from 13.5 to 19.7 per user.
• The paper employs a multi-model SI pipeline, integrating Transformer encodings, UMAP, and HDBSCAN to extract semantically coherent narratives.
• The paper reveals that while SI enhances coverage and reduces parameter search, it introduces a moderate usability penalty that must be addressed.

Semantic Interaction for Narrative Map Sensemaking: An Insight-Based Evaluation

Context and Motivation

Sensemaking over large document collections presents ongoing challenges within visual analytics (VA), particularly for narrative extraction. Traditional systems generally tether users to indirect parameter tuning, inhibiting the integration of nuanced cognitive strategies. Semantic interaction (SI) reverses this paradigm, enabling users to directly manipulate visualized representations, and thus incrementally encode their reasoning into AI models. Despite theoretical advances and SI frameworks tailored for narrative extraction, quantitative evaluation of their practical utility in real analyst workflows remains sparse.

System Architecture

The evaluated system operationalizes a multi-model SI pipeline for narrative extraction. Documents are treated as discrete events, which are embedded at the sentence level via Transformer encodings (all-MiniLM-L6-v2), aggregated, reduced to two dimensions through UMAP, and clustered with HDBSCAN. The narrative map forms a directed acyclic graph (DAG), achieved through a linear programming scheme optimizing for semantic coherence and topical coverage.

Analysts can manipulate three core parameters: map size, topic coverage, and temporal sensitivity. The SI implementation introduces direct manipulation: analysts add or remove event connections, pin or discard nodes, and restructure topical clusters. These actions backpropagate to model parameters and structure via inverse transformations, providing a tight human-in-the-loop feedback cycle.

Experimental Design

A study with 33 participants compared three conditions: timeline baseline (TL), a non-interactive map baseline (BM), and a fully interactive narrative map with SI (IM). Each participant analyzed a corpus of 160 news articles covering the 2021 Cuban protests, aiming to extract insights about causes and effects through free exploration. The evaluation protocol prioritized an insight-based methodology, meticulously coding verbal, written, and interaction-inferred insights.

Figure 1: Hierarchical coding scheme for insights and frequency distributions across conditions.

Insight Generation and Comparative Efficacy

Figure 1 details the analytical ontology used to classify insights, mapping participant performance granularly across experimental groups. Quantitative analysis evidences that narrative maps (BM and IM) facilitate higher levels of insight generation relative to the timeline baseline.

Figure 2: (a) High-level insights results; (b) Detailed insights results, including significance indications for post-hoc pairwise comparisons.

For high-level insights, mean counts were TL = 13.5, BM = 16.9, IM = 19.7. For detailed insights, TL = 40.5, BM = 51.6, IM = 63.6. ANOVA revealed strong group effects ($p < 0.01$ in both cases). Post-hoc comparisons indicate that IM outperforms TL with statistical significance ($p < 0.001$ for high-level; $p < 0.01$ for detailed), and BM trends similarly but does not reach significance after correction. Effect sizes are robust (IM–TL $d = 1.75$, IM–BM $d = 0.88$ for high-level), highlighting a likely under-powered study—the IM–BM difference is large but did not achieve statistical significance due to small group sizes.

The SI condition facilitates more insights per unit effort, and when controlling for implicit (interaction-inferred) insights, the IM–TL advantage persists ($p < 0.05$ for both hierarchical levels). This evidence supports the claim that SI-enabled map systems yield net improvements in analytic throughput over parameter-only and timeline interfaces.

Reading Behavior and Coverage

Empirical evidence indicates that map-based systems guide analysts toward more uniform data coverage. As Figure 3 demonstrates, timeline users exhibit systematic bias toward recent or early events, a consequence of interface ordering and search heuristics, whereas both map-based systems approximate an ideal even coverage.

Figure 3: Reading distributions by condition showing that map-based systems yield greater dataset coverage uniformity than timeline baselines.

This effect is consequential: sufficient coverage is a prerequisite to robust narrative synthesis, especially in intelligence and investigative analytics contexts.

Interaction Strategies and SI

Qualitative analysis reveals two primary SI-driven strategies: corrective and additive. Corrective SI involves aggressive removal or transformation of nodes/edges identified as biased, irrelevant, or spurious (e.g., culling unreliable news sources). Additive SI comprises integrating new organizational structure—such as custom clusters—imposing a user-generated taxonomy atop the model output.

Notably, analysts using SI (IM condition) exhibited reduced parameter search yet produced equivalent numbers of maps and more insights, indicating that SI supports alternative—but equally effective—modes of model refinement. Thus, SI appears to reduce superfluous parameter exploration, allowing focus on direct structural manipulation.

Usability and Trust Dynamics

While SI capabilities raised the perceived utility and reusability of the narrative map tools, they introduced a moderate usability penalty. IM scored highest on usefulness (4.80/5) and reusability (4.90/5), but lowest on ease of use (3.60/5), reflecting the increased complexity and learning requirements of semantic interaction. Importantly, trust ratings for the SI model trailed usefulness (by ∼0.4 Likert points), mirroring patterns well-documented in the human-AI trust literature (cf. [ZERILLI2022100455], [Dzindolet2003]). This outcome suggests a need for interpretable feedback and transparency mechanisms to mitigate ambiguity in SI intent inference—a well-known challenge in interactive model steering.

Limitations and Scalability

The sample is drawn primarily from student populations, not professional analysts, and dataset size (160 documents) is moderate by intelligence standards. The SI pipeline is contingent on the tractability of its linear programming and projection components; scale-up to corpora several orders of magnitude larger would require further architectural refinement (e.g., hierarchical or sampling-based extraction [keith2026interactive]).

Theoretical and Practical Implications

The results substantiate several theoretical positions: (1) direct manipulation through SI systematically increases insight yield and analytic coverage, (2) map-based representations outperform timeline approaches for exploratory sensemaking in high-complexity domains, and (3) SI not only augments model refinement but enables diverse reasoning strategies, in line with incremental formalism hypotheses ([shipman1999formalism]).

Practically, the findings endorse investments in SI-enabled visual analytics, especially for high-stakes, nonlinear investigative analysis. However, adoption may be constrained by increased system complexity and learning curve; explainable interaction feedback is a clear direction for future system design ([LIME], [shap2017]).

Conclusion

"Semantic Interaction for Narrative Map Sensemaking: An Insight-Based Evaluation" provides the first controlled, empirical evidence that SI-integrated narrative maps enhance both the thoroughness and efficacy of analytical sensemaking tasks relative to timeline and parameter-based baselines (2603.29651). The SI advantage is pronounced in effect size, and qualitative findings show SI supports granular, user-driven refinement and interpretability. These results have direct implications for the design of future analyst-in-the-loop systems—highlighting both the promise and essential complexity of SI. Extensions should address trust calibration, SI transparency, and scalability, advancing toward robust, explainable, and user-steerable narrative analytics for real-world deployment.