VIS*H Design Studies: Methodologies and Evaluation

• VIS*H design studies are an integrative approach that combines visualization methodologies with humanities scholarship for rigorous, interpretive analysis.
• The field employs multifaceted process models and evaluation criteria such as INFORMED, REFLEXIVE, ABUNDANT, PLAUSIBLE, RESONANT, and TRANSPARENT to ensure research rigor.
• Methodologies in VIS*H studies include empirical investigations and protocol-driven frameworks, facilitating practical insights in areas like medical visualization and user experience design.

VIS*H design studies represent a convergence of visualization methodologies and humanities scholarship, foregrounding interpretivist inquiry, multimodal representation, and methodological rigor tailored to the analysis and communication of complex, socially constructed phenomena. The field is characterized by a spectrum of approaches ranging from process models and meta-studies for visualization in medical and scientific contexts, to instrument-centric design for large-scale surveys, as well as in-depth empirical investigation of perceptual learning modalities. Frameworks for rigor and evaluation posit that effective VIS*H research must move beyond strict positivist validation toward practices that integrate subjective meaning-making, methodological triangulation, and reflective discourse. The following sections delineate the foundational philosophies, methodological criteria, formal process models, empirical findings, evaluation regimes, and domain-specific adaptations that underpin contemporary VIS*H research.

1. Interpretivist Foundations of VIS*H Design Studies

VIS*H design studies are grounded in an interpretivist epistemology that diverges from standard empiricist approaches common in visualization research. Meyer and Dykes (Meyer et al., 2019) position design study as interpretivist research, which views reality as multiple, mind-dependent, and socially constructed. The researcher is considered an active instrument, shaping and being shaped by the inquiry process. Knowledge production emerges through design artifacts, participatory reflection, and subjective interaction, rather than detached measurement.

Key distinctions include:

• Rejecting the assumption of a single, objective reality—embracing knowledge plurality.
• Research through Design (RtD) principles: artifacts and systems function as both experiments and vehicles for researcher expression.
• The non-deterministic, “wicked” nature of humanities problems, emphasizing situated, responsive design methodology.

This stance is operationalized through process models that synthesize design, evaluation, and knowledge-discovery as intertwined elements of inquiry.

2. Criteria for Rigor in VIS*H Design Studies

Meyer and Dykes articulate six criteria for establishing rigor in visualization design studies: INFORMED, REFLEXIVE, ABUNDANT, PLAUSIBLE, RESONANT, TRANSPARENT (Meyer et al., 2019). Each criterion has a formal definition, rationale, supporting methods, and links to broader frameworks:

Criterion	Rationale	Example Methods
INFORMED	Builds on prior theory & practice	Literature review, design-pattern libraries
REFLEXIVE	Accounts for researcher’s influence	Reflexive diaries, critical-friend sessions
ABUNDANT	Captures rich, varied evidence	Thick description, co-design workshops
PLAUSIBLE	Connects evidence to interpretation	Memo writing, analytic generalization
RESONANT	Enables transfer and evokes action	Annotated portfolios, data storytelling
TRANSPARENT	Opens audit trail for peer scrutiny	Audit logs, explicit data/task abstractions

• INFORMED: Design studies must be explicitly grounded in prior visualization idioms and domain theory.
• REFLEXIVE: Researchers document and analyze their own influence, maintaining diaries and fostering critical dialogue.
• ABUNDANT: Evidence—data, observations, participant voices—must be extensive and varied, supporting interpretive saturation.
• PLAUSIBLE: Claims require coherent and justified linkages to evidence so readers can judge their appropriateness.
• RESONANT: Work must be relatable and transferrable, with rich reporting styles and evocative detail.
• TRANSPARENT: Full documentation of design processes (including dead ends and analytic rationale) enables peer scrutiny and learning.

These criteria draw methods from social science, information systems, and design (e.g., thick description, memo writing, annotated portfolios, audit trails). Rigor is judged along these axes rather than solely by classical metrics of reproducibility or statistical significance.

3. Process Models and Domain-Specific Frameworks

VIS*H process models extend and refine existing design study methodologies to accommodate disciplinary specificity, stakeholder complexity, and task typology.

Data-First Design Studies (Oppermann et al., 2020)

Oppermann and Munzner propose a ten-stage data-first process framework in which inquiry is “triggered by the acquisition of real-world data instead of specific stakeholder analysis questions.” This ordering comprises:

1. Learn
2. Acquire Data
3. Elicit Tasks
4. Winnow—match tasks to data affordances
5. Cast—define collaborator roles (e.g., data producer, consumer, domain expert)
6. Design—visual idiom selection
7. Implement
8. Deploy
9. Reflect 10. Write

Distinctive elements:

• Early domain-agnostic data abstraction precedes stakeholder engagement.
• Technology probes with live data facilitate bidirectional task elicitation.
• Opportunities include “technology push” and scalable stakeholder expansion; risks involve hypothesized tasks, lack of users, and data–task mismatches.

Medical Visualization Process Model (Fan et al., 24 Dec 2025)

Zhou et al. introduce an eight-phase process model tailored to medical visualization:

1. Select collaborators & identify domain problem
2. Identify stakeholders and target users
3. Locate analysis goals
4. Dismantle analysis goals—decompose stages by analytic logic/cognitive habits and assign task types (descriptive, inferential, hypothesis-based)
5. Design visualization (encodings, layouts)
6. Implement prototype
7. Evaluate (controlled studies prioritized)
8. Promote (optional, dissemination)

Formal notation:

• Stakeholders: $S = \{s_1, \ldots, s_m\}$
• Target users: $T \subseteq S$
• Task function: $$\tau:\ (\text{Stage},\,\text{SubgroupFlag}) \longrightarrow \{\text{Descriptive},\,\text{Inferential},\,\text{Inferential–HypothesisBased}\}$$
• Stage decomposition based on user type.

Advancements:

• Mandatory early stakeholder and target user separation.
• Analytic stage decomposition and medical-savvy task typing.
• Controlled studies recognized as the “gold standard” in evaluation.

4. Empirical Investigations of Multimodal Learning and Design

VIS*H encompasses empirical research on the modalities of perceptual learning in design scenarios. Silvestri et al. (Silvestri et al., 2018) tested experts and non-experts in three representation modalities:

• 2D static visual (single orthographic image)
• Virtual 3D interactive (CAD model, mouse-driven rotation)
• Real 3D haptic–visual (tactile exploration of a physical tensegrity simplex)

Statistical results:

• Experts performed fastest and most accurately in the virtual interactive modality.
• Non-experts benefited more from static 2D and physical–visual conditions, provided strong visual scaffolds.
• Haptic cues did not universally improve comprehension; their utility was contingent on user expertise and the presence of spatial scaffolds.

Implications for VIS*H pedagogy and tool design:

• Novices should begin with static visual and haptic modalities, then progress to interactive VR.
• Visual scaffolding (axonometric grids, uniform color) reduces cognitive load in haptic learning contexts.
• High-fidelity VR environments with eligibility-trace support optimize expert training.

5. Meta Studies and Knowledge Formalization

Meta design studies operationalize collective expert knowledge for rapid prototyping and user-guided dashboard design.

The RSVP system (Klaffenboeck et al., 2024) codifies Visual Parameter Space Analysis (VPSA) expertise using a corpus-derived taxonomy:

• Data types: control parameters, environmental parameters, scalar/complex outputs
• Task types: optimization, fitting, uncertainty, outlier detection, sensitivity, partitioning
• Visualization idioms: point scales, contour plots, SPLOM, PCP, histograms, grid-based, juxtaposition

Core features:

• Rule-based, task-oriented visualization recommendation (VisRec)
• Formal assignment: $$s(v,\tau) = |C(v)\cap C_{\tau}| + \alpha \mathbf{1}[M(v)\ni M_{\tau}]$$
• Usability study: mean SUS = 82.5 (high usability); domain scientists rapidly achieved insight using suggested layouts.

Meta design studies provide a template for encoding and operationalizing visualization knowledge, fostering transparency and transferability in VIS*H dashboard creation.

6. Evaluation Practices and Quality Regimes in VIS*H

VIS*H evaluation is marked by a tension between “Insight” (quantitative discovery) and “Meaning” (interpretive depth) (Benito-Santos et al., 28 Jan 2026). A comprehensive survey of 171 VIS*H studies reveals:

• Two broad workflow types:
  • Monomethod workflows (authors’ walkthroughs, interviews, checklists) exhibit low rigor.
  • Triangulated workflows (mixed-method co-design, observation, think-aloud, log analysis, surveys) achieve higher external validity and quality scores (Mdn ≥ 3).

Workflow Type	Methods Used	Median Quality Score
Monomethod	Case study/interview	0–1
Triangulated	Mixed-method, logs, survey	≥3

Key evaluation metrics:

• Time on task ($T_k$), completion rate ($C_k$), error rate ($E_k$), insight count ($I$), SUS, NASA-TLX.

Recommendations for VIS*H evaluation:

1. Move beyond monomethods—combine qualitative and quantitative evidence.
2. Triangulate strategically—critical dyads include think-aloud + surveys, log analysis + interviews.
3. Align participants to domain, mandatory documentation of participant demographics and expertise.
4. Incorporate uncertainty and user confidence measures.
5. Embrace reflexivity—meta-evaluation of domain complexities and sample limitations.
6. Integrate formative and summative loops, extending Munzner’s Nested Model with hermeneutic iteration.

The paradigm shift advocates reconciling empirical verification with interpretive richness, developing frameworks for “Meaning-Grounded Validation.”

7. Challenges, Opportunities, and Future Directions

VIS*H design studies face domain-specific challenges and opportunities:

• Challenges:
  • Steep learning curve in qualitative and interpretive methods (across humanities, medical, and cross-disciplinary domains).
  • Paper length and review constraints limit methodological reporting.
  • Infrastructure for open, indexed supplemental materials is still developing.
  • Reviewer expertise must address interpretivist rigor beyond empirical metrics.
• Opportunities:
  • Development of transferable middle-range theories and design patterns.
  • Rich, socially grounded design exposés that inform future toolkits and libraries.
  • Integration of reflexive and abundant evidence practices into training and publication formats.
  • Adoption of co-design epistemic networks and multiphase evaluation pipelines.

Recommendations include integrating method modules on reflexive practice, developing guidelines for citing evidence artifacts, and exploring new publication venues for transparency and resonance. Future directions involve automated identification of stakeholder/task typologies, integrating real-time data (e.g., in medical contexts), and broadening application domains to rehabilitation, public health, arts, and cultural heritage.

---

VIS*H research continues to mature unified methodologies that honor both the quantitative rigor of visualization science and the interpretive, socially constructed knowledge frameworks central to humanities and design. Foundational criteria, process models, multimodal experimentation, meta-studies, and evaluation paradigms together define a rapidly evolving discipline well positioned to address the complexity and politicized realities of contemporary data-driven inquiry.