HiLSVA: Design and Evaluation of a Human-in-the-Loop Agentic System for Scientific Visualization

Abstract: LLM agents enable natural language interaction for scientific visualization (SciVis). Still, prior systems have essentially prioritized autonomy over human analytical control, thereby limiting transparency and human oversight. We present HiLSVA, a human-in-the-loop agentic system that supports mixed-initiative SciVis workflows. HiLSVA integrates a plan-first multi-agent architecture with explicit human oversight, stepwise provenance tracking, and learn-at-test-time adaptation from user feedback. The system supports fluid handoff between humans and agents through both natural language and direct manipulation of visualizations, while sandboxed execution ensures safe, reproducible workflows. In doing so, HiLSVA reframes agentic SciVis as a collaborative process that augments, rather than replaces, human analytical reasoning. We evaluate HiLSVA through representative case studies and a controlled user study with twelve participants of varying expertise across multiple autonomy settings. Results show that mixed-initiative interaction improves task completion, user control, and workflow transparency across different levels of user expertise, while revealing a tradeoff between execution efficiency and human oversight. These findings highlight the importance of human-centered design in agentic SciVis and guide the development of future collaborative visualization systems. We encourage readers to explore our demo video, case studies, and source code at https://hilsva.github.io/.

• The paper introduces a mixed-initiative agent architecture that integrates human oversight with automated tools to enhance transparency and interpretability in scientific visualization.
• It presents a provenance-centric, stepwise workflow tracking and sandboxed execution method that supports undo, state restoration, and modular tool integration.
• An empirical study shows that HiLSVA enables both novices and experts to achieve expert-level outcomes through interactive planning and guided exploration.

HiLSVA: Design and Evaluation of a Human-in-the-Loop Agentic System for Scientific Visualization

Motivation and Contributions

The HiLSVA system introduces a paradigm shift in scientific visualization (SciVis) by integrating human-in-the-loop mixed-initiative agent architectures, explicitly prioritizing transparency, user oversight, and collaborative workflow construction over pure automation. While previous LLM-driven agentic systems demonstrated autonomous tool manipulation and workflow generation, they often marginalized the analyst's role, resulting in limited transparency and reduced interpretability. HiLSVA addresses these gaps via:

• Mixed-initiative architecture: Orchestrator-led planning, explicit human task assignment, and approval mechanisms.
• Provenance-aware execution: Stepwise workflow tracking with full state restoration, branching, undo, and workflow reuse.
• Safe, reproducible interaction: Sandboxed Docker environments for tool execution; critical operations gated by user approval.
• Learn-at-test-time (LTT) adaptation: Real-time knowledge base augmentation with self-reflection and targeted human feedback.
• Backend-agnostic tool integration: Specialized agents for direct API control, Python script generation, GUI manipulation, and web retrieval.

These capabilities collectively position HiLSVA as a robust foundation for collaborative SciVis, emphasizing the augmentation, rather than replacement, of human analytical reasoning.

Figure 1: HiLSVA system architecture illustrating orchestrator-driven planning, specialized visualization agents, stepwise control/provenance, sandboxed execution, and LTT adaptation mechanisms.

Mixed-Initiative and Provenance-Centric Architecture

HiLSVA employs a multi-agent design centered around an orchestrator that assigns steps to specialized agents—file surfer, code generator, ParaView controller, web retriever, and self-improver. The workflow is defined upfront via a plan-first mechanism, allowing users to reorder and modify steps, which are only executed upon explicit approval. Initiative is fluid: users can intervene through natural language or direct GUI manipulation at any point, while agents proactively query users upon encountering uncertainty or ambiguity.

All agentic actions occur in isolated Docker container environments, enabling safe, parallel session execution and reproducibility. Stepwise workflow monitoring and full provenance recording (planned and executed steps, tool state, visualization outputs) permit state restoration, branching, and reuse. This ensures accountability and enables not only replay but also extension of previously validated workflows, supporting iterative refinement and comparative analysis.

Figure 2: HiLSVA key capabilities: collaborative planning, guarded execution, rollback, interactive user steering, explicit autonomy control, and LTT adaptation via knowledge retrieval/reflection.

LTT Adaptation and Human Feedback

HiLSVA formalizes adaptation during inference through a vector-based knowledge repository seeded with general SciVis knowledge and incrementally augmented with self-reflection summaries and validated human feedback. Each step retrieves relevant knowledge using composite scoring (semantic similarity, confidence, recency, validity), guiding subsequent actions. Confidence assessment and uncertainty handling trigger targeted user queries within an interaction budget, and resulting feedback (graded by agent reliability) updates the knowledge base. This retrieval-centric approach focuses on cumulative experience and domain-guided adaptation, not model weight updates.

Interface Design

The HiLSVA interface integrates three core components:

• Mixed-initiative chat widget: Supports task specification, plan editing, code review, uncertainty prompts, and feedback.
• Workflow monitor panel: Enforces stepwise provenance tracking, undo, state restoration, and clickable navigation among workflow steps.
• Real-time visualization engine: Direct ParaView rendering and browser integration; supports seamless handoff between agent actions and user GUI interventions.
• Plan gallery/knowledge base: Facilitates workflow reuse and incremental adaptation.

  Figure 3: HiLSVA interface highlighting mixed-initiative chat, workflow monitor, visualization engine, and plan gallery integration.

Empirical Evaluation: Case Studies and User Study

Case Study Highlights

HiLSVA was evaluated through five canonical SciVis tasks across three analytic stages—exploration, workflow construction, and scientific analysis:

• Foot (CT scan): Iterative isosurface refinement, anatomy reference retrieval, direct GUI-driven adjustment.

  Figure 4: Basic action—Foot dataset: initial isosurface rendering, anatomy lookup, GUI-based refinement.

• Hurricane (temperature field): Slice visualization, histogram-based contour overlay, domain-specific insight extraction.

  Figure 5: Basic action—Hurricane dataset: temperature field slicing, contour overlay, salient structure parsing.

• Tornado (vector field): Streamline and glyph visualization via parameter trials; provenance-based state restoration, knowledge-guided glyph sizing.

  Figure 6: Workflow—Tornado dataset: parameter exploration, tube radius adjustment, knowledge-guided glyph rendering.

• Combustion (multivariate time series): Animation export, knowledge-guided colormap selection, cross-field isosurface analysis.

  Figure 7: Workflow—Combustion dataset: time-varying volume rendering, animation export, isosurface series generation.

• Half-cylinder (ensemble flow): Analysis-driven exploration, Q-criterion and $\lambda_2$ computation, workflow storage and batch extension.

  Figure 8: Scientific analysis—Half-cylinder: derived field computation (Q, $\lambda_2$), vortex visualization, reusable workflow extension.

User Study Design and Results

A controlled IRB-approved user study (N=12) spanned experts, domain scientists, and novices, each interacting with HiLSVA under three autonomy configurations: full-auto, half-auto, and mixed-initiative (LTT-enabled). Key findings:

• High task completion and interpretative accuracy (mean: 11.75/12).
• Execution times increased with human involvement (full-auto: $9.83\pm3.27$ min, mixed-initiative: $13.50\pm4.46$ min, effect size $\delta=-0.51$, not significant).
• No substantial differences between expertise groups in task performance; novices achieved expert-level outcomes.
• Autonomy preference varied: experts and novices both valued flexibility; no universally optimal interaction style.
• Survey responses: strong ratings for transparency, provenance tracking, control, mixed-initiative interaction, and system usability (mean: 4.66/5).

  Figure 9: Post-study survey responses—high perceived transparency, control, and usability across expertise levels.

Practical and Theoretical Implications

HiLSVA demonstrates that human-in-the-loop agentic SciVis systems can bridge the gap between autonomy and oversight, enabling robust, transparent workflows adaptable to varying user expertise. The stepwise provenance and mixed-initiative control mechanisms substantially improve interpretability and reproducibility. LTT adaptation offers a scalable pathway for knowledge accumulation and domain-guided refinement. Empirically, HiLSVA enables non-experts to achieve expert-level analysis via collaborative planning and guided exploration, reducing the barrier to complex SciVis tasks.

The backend-agnostic design and containerized execution facilitate integration with diverse visualization tools. Further research may explore deeper model-based self-improvement, expanded domain-specific knowledge integration, multimodal uncertainty resolution, collaborative multi-user environments, and longitudinal impact on scientific discovery.

Conclusion

HiLSVA operationalizes human-centered design for agentic scientific visualization, supporting fluid handoff between autonomy and oversight and enabling provenance-centric, transparent analytic workflows. Empirical results confirm its effectiveness across expertise levels and task complexity, validating its architecture and interaction mechanisms. HiLSVA establishes a foundation for future collaborative, transparent AI systems in scientific domains, with implications for scalability, robustness, and democratization of specialized analytical workflows.