HCI Visual Storytelling Education

• HCI visual storytelling education is an interdisciplinary field combining visual and narrative modalities to optimize engagement and comprehension.
• It employs cognitive theories, dual-coding, and design thinking through frameworks like QUAVER and interactive digital storytelling to structure learning.
• Collaborative and creative workflows, including human–AI partnerships and agile pedagogical methods, personalize instruction and enhance visualization literacy.

Human–Computer Interaction (HCI) visual storytelling education encompasses the design, implementation, and assessment of systems, curricula, and methodologies that enable learners to comprehend complex phenomena through integrated visual and narrative modalities. This field synthesizes algorithmic frameworks, cognitive theory, multimodal HCI, and collaborative workflows, with critical emphasis on personalization, creative agency, and systematic evaluation. Research spans STEM instruction, digital heritage, exploratory visualization, and human–AI creative partnerships, grounded in methods such as heuristic modeling, dual-coding theory, interactive digital storytelling, and design thinking.

1. Algorithmic Frameworks for Visual–Narrative Integration

The QUAVER framework (Bangroo et al., 2023) formalizes instructional design as an optimization problem, seeking a configuration of visual elements $V$ and narrative segments $N$ that maximizes engagement $E$ and comprehension $C$. The system employs an Engagement Analysis Model (EAM)—a neural architecture combining transformer-based text encoding, CNN-driven image semantic analysis, and cross-modal attention—to predict user outcomes based on candidate scripts. The utility function is expressed:

$$U(x) = \alpha \cdot \mathrm{Engagement}(x) + \beta \cdot \mathrm{Coherence}(x) - \gamma \cdot \mathrm{Load}(x)$$

Parameters $\alpha, \beta, \gamma$ are determined via cross-validation, balancing predicted click-through rates, narrative continuity, and cognitive load constraints. Content selection is iteratively explored using gradient-free heuristics (e.g., covariance matrix adaptation), with explicit constraints on the maximal number of visuals ($V_{\max}$) and narratives ($N_{\max}$):

$$\max_{x} U(x), \;\; \text{subject to} \;\; \sum_{i=1}^k \mathbf{1}[v_i\in x] \le V_{\max},\; \sum_{j=1}^\ell \mathbf{1}[n_j\in x] \le N_{\max}$$

EAM training involves minimization of squared prediction error across engagement and comprehension labels, with batchwise backpropagation and validation-controlled checkpointing.

2. HCI Principles and Design Guidelines

HCI visual storytelling systems operationalize foundational cognitive and narrative theories:

• Dual-Coding Theory: Mandates pairing textual information exclusively with meaningful, explanatory visuals.
• DeFT Model: Visual components are classified as providing complementary views, juxtaposed clues, or cognitive scaffolding.
• Cognitive Load (Mayer): Instructional design reduces extraneous media while promoting germane processing.
• Narrative Arc: Lesson units follow structured arcs—setting, conflict, resolution—to instill conceptual progression and emotional resonance.

Practical recommendations include limiting per-page visual density ($V_{\max}=2$), ensuring spatial and temporal contiguity between captions and figures, signaling references, segmenting explanations into discrete interactive scenes, and employing emotionally resonant hooks. Concrete archetypes such as comic strips and interactive animations reinforce quantum concepts (e.g., superposition, entanglement) (Bangroo et al., 2023).

In interactive digital storytelling (IDS) for heritage education (Rizvic et al., 2020), narrative construction draws from classical rule sets (Aristotle’s “seven golden rules,” Campbell’s Hero’s Journey) and modular “hyper-storytelling” graphs. Multimedia presentation incorporates visual, auditory, and kinesthetic modalities, with attention to segmentation and system usability heuristics, expanding Nielsen’s canon for narrative-enriched virtual environments.

3. Collaborative and Creative Workflows

Human–AI collaboration in visual storytelling education is exemplified by the gap-and-fill methodology (Nie et al., 23 Dec 2025). The workflow segments creation into three phases: human-led narrative foundation ($H$), identification of strategic gaps ($G$), and AI-based enhancement ($E$), so the final sequence is $S = H \cup E$. Gaps are formalized as feature-space differences $\Phi(T_i) - \Phi(h_i)$, where $T_i$ is the ideal content for narrative node $h_i$. AI-generated fills $e_i$ are prompted with stylistic continuity constraints.

Eastern aesthetic principles (Chinese leave-blank, Japanese ma, minimalism) deliberately embed structural voids for AI or human intervention, indexed quantitatively via a creative-agency ratio (ACR):

$$\mathrm{ACR}_k = \frac{\|h^{(k)}\|}{\|h^{(k)}\| + \|e^{(k)}\|}$$

Best practices discourage excessive AI dependence (target ACR $\geq$ 0.6), integrate reflection checkpoints, and archive portfolios for longitudinal assessment.

4. Pedagogical Approaches and Visualization Literacy

Visualization literacy curricula, such as that outlined in “Dear Data” (Krekhov et al., 2019), adapt design-thinking models—empathize, define, ideate, prototype, test—to visual story creation. Assignments revolve around personal data collection, rapid divergent sketching (3–6 concepts per theme), iterative visual refinement, and peer critique. The pipeline is formalized:

$$\begin{aligned} &\text{Acquire: } D \leftarrow \mathrm{Track}(\text{self},\ \text{theme})\ &\text{Explore: } E \leftarrow \mathrm{Analyze}(D)\ &\text{Define: } M \leftarrow \mathrm{Message}(E)\ &\text{Ideate: } \{I_i\} \leftarrow \mathrm{GeneratePrototypes}(D, M)\ &\text{Select: } R^* \leftarrow \arg\max_{I_i} \mathrm{Clarity}(I_i) + \mathrm{Appeal}(I_i)\ &\text{Deliver: } R^* \end{aligned}$$

Pain points are reported in novice practice: omission of systematic data analysis, premature selection of narrative messages, technical limits in craftsmanship, and reduced creativity in large-group settings. Instruction balances technical mastery of chart grammars with extensive creative lab sessions. Programs are advised to reserve open-ended project time, seed inspirational exemplars, and introduce data exploration workshops early.

5. Evaluation Methodologies and Empirical Findings

Visual storytelling education is frequently subject to mixed-methods evaluation:

• QUAVER (Bangroo et al., 2023):
  • Two-sample t-test contrasts engagement and comprehension across control (text-only, $N_1=45$) and QUAVER (visual + narrative, $N_2=48$); statistical significance achieved ($p<0.05$, t-statistic/degrees of freedom not specified).
  • Quantitative improvements: mean engagement score 3.1 $\to$ 4.2, comprehension 68% $\to$ 82%, Cohen’s $d \approx 0.75$, EAM–actual engagement correlation $r=0.68$.
  • Qualitative themes stress narrative memorability, visualization realism, curiosity induction, and potential for confusion in overcomplicated pacing.
• IDS (Rizvic et al., 2020):
  • Heuristic usability audits, cognitive load analyses, and Likert-scale user surveys.
  • Visualization of results through diverging bar charts, group comparison by discipline.
• Gap-and-fill (Nie et al., 23 Dec 2025):
  • Proposed metrics: narrative coherence score, creative agency index, AI-dependence ratio; peer-rubric inter-rater reliability via Cronbach’s $\alpha$.
  • Self-report and peer evaluation on collaboration and artifact creativity.
• Dear Data (Krekhov et al., 2019):
  • Structured artifact analysis, chart diversity counts, questionnaires on pipeline adherence and team dynamics.
  • Student preference for small-group format and analytic workshops.

6. Design Implications, Limitations, and Best Practices

Optimal HCI visual storytelling systems are characterized by personalization (dynamic tuning of $\alpha, \beta, \gamma$ in $U(x)$), adaptive pacing, and integration of interactive micro-polls (Bangroo et al., 2023). Multidisciplinary collaboration (IDS framework), system prototyping, iterative heuristic and learner evaluations, and platform-independent implementations are specified as essential (Rizvic et al., 2020).

Limitations commonly include insufficient statistical details (e.g., missing t-statistics), prototypical media artifacts, restricted sample sizes, and domain-specific generalizability. Recommendations emphasize grounding in cognitive theory, narrative control for learners, embedded analytics for adaptive refinement, editorial accuracy, and transitional scaffolding of mathematical content.

Educators are further advised to:

• Favor analog sketching for design exploration (Krekhov et al., 2019).
• Use gap-notation templates and maintain creative agency in AI-integrated projects (Nie et al., 23 Dec 2025).
• Document authoring choices and paradata for transparency and revision.
• Leverage low-stakes unlockables and iterative prototype evaluation for learner motivation.

7. Comparative Summary of Core Methodologies

Method/Framework	Major Focus	Key Features
QUAVER	STEM visual-narrative integration	EAM-driven optimization, empirical evaluation, cognitive load management
Interactive Digital Storytelling (IDS)	Cultural heritage, narrative branching	Modular nodes, multimedia, system usability, narrative paradox mitigation
Gap-and-fill (Eastern Wisdom)	Human–AI creative partnership	Agency management, gap detection, Eastern aesthetics, collaborative storyboarding
Dear Data Visualization Literacy	Personal data-driven storytelling	Design thinking workflow, analog/digital sketching, peer critique, chart diversity

This comparative synthesis encapsulates technical and theoretical foundations, heuristic strategies, collaborative operational guidelines, and empirical evidence supporting HCI visual storytelling education. Applications span from quantum computing didactics to heritage documentation and visualization literacy enhancement, revealing consistent gains in engagement, comprehension, and creative competence when narrative and visual modalities are systematically integrated.