HiLSVA: Human-in-the-Loop Agentic SciVis System

• HiLSVA is a human-in-the-loop system for scientific visualization that employs a plan-first multi-agent framework to integrate automated LLM support with active user control.
• The system leverages explicit oversight, stepwise provenance tracking, and sandboxed execution to ensure reproducibility, safety, and transparency in mixed-initiative workflows.
• Learn-at-test-time adaptation via structured user feedback enables iterative refinement of visualization outputs while balancing automation with human oversight.

Searching arXiv for the cited papers to ground the article. {"^^^^^^^^11query11 (Ai et al., 25 Jun 2026) OR HierSVA (Nie et al., 9 Jun 2026)11^^^^^^^^"," (Ai et al., 25 Jun 2026) OR HierSVA (Nie et al., 9 Jun 2026)11query11^^^^} Searching separately for each arXiv identifier. {"^^^^^^^^11query11 {"^^^^^^^^11query11 Performing one more title-based search for confirmation. {"^^^^^^^^11query11 (Ai et al., 25 Jun 2026) OR HierSVA (Nie et al., 9 Jun 2026)11query11^^^^^^^^"HiLSVA: Design and Evaluation of a Human-in-the-Loop Agentic System for Scientific Visualization^^^^^^^^11HiLSVA (Ai et al., 25 Jun 2026) OR HierSVA (Nie et al., 9 Jun 2026)11query11^^^^^^^^""," HiLSVA is a human-in-the-loop agentic system for scientific visualization (SciVis) that re-centers human analytical control while leveraging LLM-driven automation. It supports mixed-initiative SciVis workflows through a plan-first multi-agent architecture with explicit human oversight, stepwise provenance tracking, learn-at-test-time adaptation from user feedback, and sandboxed execution. Rather than treating agentic visualization as autonomy-first execution, HiLSVA frames it as a collaborative process in which humans can plan, approve, intervene, and steer, while the system adapts at test time through structured reflection and user feedback ^^^^^^^^11(Ai et al., 25 Jun 2026)11

^^^^11HiLSVA (Ai et al., 25 Jun 2026) OR HierSVA (Nie et al., 9 Jun 2026)11^^^^. Motivation and conceptual framing

HiLSVA is motivated by a specific limitation in prior LLM-based SciVis systems: they have demonstrated impressive tool use and visualization generation, but they often sideline the human analyst, reduce workflow transparency, and limit oversight. In the formulation associated with HiLSVA, SciVis is inherently human-centered. Analysts must decide when to delegate or withdraw initiative, understand how and why an agent takes actions, and retain authority over interpretation. The system therefore rejects the assumption that maximizing automation is either realistic or desirable for open-ended, visually grounded tasks ^^^^^^^^11(Ai et al., 25 Jun 2026)11

The design objective is not merely to improve output generation, but to support mixed-initiative coordination, explicit oversight, provenance-aware workflows, and safe execution. HiLSVA operationalizes this objective by making planning explicit and editable, enforcing stepwise approval, tracking both prospective and retrospective provenance, enabling direct manipulation of visualizations, and adapting at test time from human feedback without sacrificing control or reproducibility. This suggests a shift from agent substitution toward agent augmentation of scientific reasoning, with the human analyst treated as a first-class participant in the agent team rather than as an external supervisor.

The core contributions are organized around six mechanisms: human-in-the-loop agentic design; a plan-first multi-agent architecture; explicit oversight mechanisms; stepwise provenance tracking; sandboxed execution; and learn-at-test-time adaptation from user feedback. Together, these mechanisms define HiLSVA as a collaborative SciVis system rather than a purely autonomous visualization agent ^^^^^^^^11(Ai et al., 25 Jun 2026)11

^^^^11max_results11^^^^. Multi-agent architecture and execution environment

HiLSVA is built around an orchestrated set of specialized components. The orchestrator interprets intent, constructs an explicit stepwise plan, coordinates specialized agents, monitors execution, triggers replanning, and mediates oversight. Specialized agents and tools include a ParaView agent built on ParaView-MCP for reliable, parameterized interaction via function-oriented tool calls; a code agent described as ChatVis-style for robust generation and iterative refinement of Python and shell scripts with RAG over documentation and examples plus error-driven correction; a web surfer for external information retrieval; a file surfer for directory inspection and file transformations such as timesteps detection; and a self-improving agent responsible for learn-at-test-time adaptation, confidence assessment, self-reflection, and ^^^^^^^^11query11^^^^^^^^ generation ^^^^^^^^11(Ai et al., 25 Jun 2026)11

The execution substrate is an explicit sandbox. HiLSVA uses per-session Docker containers for visualization in ParaView, code execution, web browsing, and file access, with a controlled shared workspace. This organization supports session isolation, reproducibility, parallel tasks, and safe experimentation. The use of isolated containers is central to the system’s safety model, because all agent actions occur inside these bounded environments rather than on the host system directly.

The user interface combines several views of the workflow: mixed-initiative chat for natural-language planning and approval; a workflow monitor panel for stepwise provenance and clickable state restoration; a real-time visualization engine and browser; a plan gallery and knowledge base; and a session monitor sidebar. The implementation details reported for the system specify GPT-^^^^11max_results11^^^^.^^^^11max_results11^^^^ for the orchestrator and Claude-Sonnet-^^^^11(Ai et al., 25 Jun 2026)11^^^^.^^^^11query11^^^^ for specialized agents, with ParaView running inside Docker and interfaced through a ParaView-MCP server ^^^^^^^^11(Ai et al., 25 Jun 2026)11

A notable property of the architecture is backend agnosticism. The reported deployment model states that additional tools such as VMD, napari, and TTK can be plugged in via specialized agents or MCP-like interfaces. A plausible implication is that the architecture is intended as a general pattern for collaborative visualization systems rather than as a ParaView-only prototype.

^^^^11query11^^^^. Plan-first interaction, human oversight, and provenance

HiLSVA’s workflow begins with explicit planning. The orchestrator generates a stepwise plan, assigns each step to an agent, and exposes the plan for user editing, including reordering, adding or removing steps, and modifying instructions. Execution begins only after approval. During execution, initiative flows in both directions: users can intervene through chat or direct GUI manipulation, agents proactively request clarification when confidence is low, and autonomy levels can be adjusted with guarded execution and explicit approvals for sensitive actions ^^^^^^^^11(Ai et al., 25 Jun 2026)11

The oversight model is therefore procedural rather than merely post hoc. User approval is required for critical or irreversible operations, including executing generated code or sensitive visualization tool calls. The system also solicits clarification when uncertainty is high. This makes oversight a structured part of execution rather than a retrospective audit step.

Provenance is recorded at two levels. Prospective provenance is the planned sequence of actions. Retrospective provenance includes executed actions and tool calls, software states such as ParaView pipeline configuration, intermediate and final visualization outputs such as images and animations, and interaction traces including approvals and user interventions. The workflow monitor exposes steps and statuses, and users can click any prior step to restore the corresponding state and branch the plan from that point. Saved workflows therefore include both the plan and stateful execution artifacts, enabling stateful reuse across sessions ^^^^^^^^11(Ai et al., 25 Jun 2026)11

Direct manipulation is integrated into the same rendering session. HiLSVA runs ParaView inside Docker and supports MCP tool calls, generated Python scripts, and direct GUI manipulation by the user within a shared visualization state. A user can take control to adjust transfer functions, camera, or streamline parameters and later return control to the agent. The significance of this design is that manual intervention does not force a mode switch to a separate environment; instead, it becomes part of the same provenance-tracked trajectory.

^^^^11(Ai et al., 25 Jun 2026)11^^^^. Formalization and learn-at-test-time adaptation

HiLSVA presents a formal notation for stepwise planning and adaptation. The workflow is expressed with steps PRESERVED_PLACEHOLDER_^^^^11query11^^^^, environment states PRESERVED_PLACEHOLDER_^^^^11HiLSVA (Ai et al., 25 Jun 2026) OR HierSVA (Nie et al., 9 Jun 2026)11^^^^, per-step actions PRESERVED_PLACEHOLDER_^^^^11max_results11^^^^, internal state PRESERVED_PLACEHOLDER_^^^^11query11^^^^, human user PRESERVED_PLACEHOLDER_^^^^11(Ai et al., 25 Jun 2026)11^^^^, and interaction budget PRESERVED_PLACEHOLDER_^^^^11max_results11^^^^. The decision policy is given as PRESERVED_PLACEHOLDER_^^^^11query11^^^^, with actions generated as PRESERVED_PLACEHOLDER_^^^^11(Nie et al., 9 Jun 2026)11^^^^ and state updates written as PRESERVED_PLACEHOLDER_^^^^11max_results11^^^^ ^^^^^^^^11(Ai et al., 25 Jun 2026)11

The learn-at-test-time mechanism is retrieval-based rather than weight-updating. The self-improving agent observes steps, assesses confidence, performs self-reflection in the form of summaries of outcomes and reliability, and, when confidence is below a threshold and within the budget PRESERVED_PLACEHOLDER_^^^^11query11^^^^, issues a structured ^^^^^^^^11query11^^^^^^^^ for clarification. Human feedback is then captured and stored. The knowledge repository PRESERVED_PLACEHOLDER_^^^^11HiLSVA (Ai et al., 25 Jun 2026) OR HierSVA (Nie et al., 9 Jun 2026)11query11^^^^ is described as a vector-based store of natural-language knowledge items with captions, BERT embeddings, confidence, validity, and recency metadata; items derived from human feedback receive confidence PRESERVED_PLACEHOLDER_^^^^11HiLSVA (Ai et al., 25 Jun 2026) OR HierSVA (Nie et al., 9 Jun 2026)11HiLSVA (Ai et al., 25 Jun 2026) OR HierSVA (Nie et al., 9 Jun 2026)11^^^^, and foundational items can be marked non-expiring. Before each step, relevant items are retrieved using a composite score and injected into the environment state to guide actions and parameters ^^^^^^^^11(Ai et al., 25 Jun 2026)11

The repository update is formalized as PRESERVED_PLACEHOLDER_^^^^11HiLSVA (Ai et al., 25 Jun 2026) OR HierSVA (Nie et al., 9 Jun 2026)11max_results11^^^^, and the paper notes the approximation PRESERVED_PLACEHOLDER_^^^^11HiLSVA (Ai et al., 25 Jun 2026) OR HierSVA (Nie et al., 9 Jun 2026)11query11^^^^. Adaptation is therefore externalized into a retrievable knowledge layer rather than embedded as online fine-tuning. The system can reuse prior parameter presets and successful workflow fragments, such as colormap choices for volume rendering or vortex-identification steps. Learn-at-test-time can also be disabled for ablation, which is important for separating the effect of retrieval-based adaptation from the broader effect of mixed-initiative interaction.

This formalization places HiLSVA in a design space where adaptation, transparency, and oversight are coupled. The system’s knowledge is inspectable in the knowledge base, the queries are triggered by explicit confidence assessment, and the adaptation pathway remains reproducible because it is mediated through stored artifacts rather than hidden model updates.

^^^^11max_results11^^^^. Interaction modalities and representative SciVis tasks

HiLSVA supports two principal interaction modalities: natural language and direct manipulation. Through chat, users specify goals, edit plans, approve actions, request alternatives, and provide feedback. The system explains, reflects, and asks for clarification where needed. Through direct manipulation, users can take GUI control in ParaView for fine-grained steering of transfer functions, camera, seeding, and tube radius, then hand control back to the agent without leaving the active session ^^^^^^^^11(Ai et al., 25 Jun 2026)11

The task repertoire described for the system includes data loading and inspection, including directory scanning to enumerate timesteps; filtering operations such as slices, contours, stream tracers, tubes, glyphs, and isosurfaces; encoding decisions such as colormaps, scaling by magnitude, and orientation by vectors; layout and animation tasks including volume rendering across timesteps and exporting videos or image sequences; and derived-field computation for scientific analysis, including vortex measures. These are not presented as isolated demos but as representative mixed-initiative workflows.

The reported case studies illustrate the role of human oversight. In the foot CT example, the workflow begins with an initial isosurface to reveal bones, incorporates web retrieval of anatomical references, and then uses direct GUI refinement by the user to improve phalanges visibility. In the hurricane example, the system constructs a PRESERVED_PLACEHOLDER_^^^^11HiLSVA (Ai et al., 25 Jun 2026) OR HierSVA (Nie et al., 9 Jun 2026)11(Ai et al., 25 Jun 2026)11^^^^-slice visualization, performs histogram analysis, and overlays black contour lines to clarify temperature gradients. In the tornado example, the plan contains three streamline trials; the user selects Trial ^^^^11max_results11^^^^ and restores its state, after which knowledge-guided glyphs are oriented by velocity and scaled by magnitude, followed by tube-radius adjustment after agent reflection. In the combustion example, HiLSVA performs volume rendering of PRESERVED_PLACEHOLDER_^^^^11HiLSVA (Ai et al., 25 Jun 2026) OR HierSVA (Nie et al., 9 Jun 2026)11max_results11^^^^ across ^^^^11query11query11^^^^ timesteps with a preferred colormap retrieved from knowledge, and also generates mixfrac isosurface animations across timesteps ^^^^^^^^11(Ai et al., 25 Jun 2026)11

The half-cylinder analysis makes explicit that HiLSVA is intended for scientific analysis rather than only visual depiction. The workflow computes and visualizes both PRESERVED_PLACEHOLDER_^^^^11HiLSVA (Ai et al., 25 Jun 2026) OR HierSVA (Nie et al., 9 Jun 2026)11query11^^^^-criterion and PRESERVED_PLACEHOLDER_^^^^11HiLSVA (Ai et al., 25 Jun 2026) OR HierSVA (Nie et al., 9 Jun 2026)11(Nie et al., 9 Jun 2026)11^^^^, and stores a successful vortex-identification workflow for reuse. The reported script computes

PRESERVED_PLACEHOLDER_^^^^11HiLSVA (Ai et al., 25 Jun 2026) OR HierSVA (Nie et al., 9 Jun 2026)11max_results11^^^^

where

PRESERVED_PLACEHOLDER_^^^^11HiLSVA (Ai et al., 25 Jun 2026) OR HierSVA (Nie et al., 9 Jun 2026)11query11^^^^

This example is significant because it extends the system beyond appearance-oriented rendering into derived-field analysis workflows that are common in expert SciVis practice.

^^^^11query11^^^^. Evaluation, relation to prior work, and naming ambiguity

HiLSVA was evaluated through a controlled, within-subjects user study with PRESERVED_PLACEHOLDER_^^^^11max_results11query11^^^^, with expertise distributed as ^^^^11query11^^^^ SciVis experts, ^^^^11(Ai et al., 25 Jun 2026)11^^^^ domain scientists, and ^^^^11max_results11^^^^ novices. Participants used a workstation with an NVIDIA RTX ^^^^11max_results11query11query11query11^^^^ GPU and two ^^^^11query11max_results11^^^^-inch PRESERVED_PLACEHOLDER_^^^^11max_results11HiLSVA (Ai et al., 25 Jun 2026) OR HierSVA (Nie et al., 9 Jun 2026)11^^^^ displays. They completed four tasks—hurricane, foot, tornado, and combustion—with autonomy modes assigned in balanced fashion for the first three tasks: full-autonomous (FA), half-autonomous (HA, LTT off), and mixed-initiative (MI, LTT on). Interaction logs, plans, and outputs were recorded, and a post-study questionnaire measured usability, transparency, and collaboration ^^^^^^^^11(Ai et al., 25 Jun 2026)11

All participants completed the tasks, and domain-specific interpretation accuracy averaged PRESERVED_PLACEHOLDER_^^^^11max_results11max_results11^^^^. For the first three tasks, mean execution times were PRESERVED_PLACEHOLDER_^^^^11max_results11query11^^^^ minutes for FA, PRESERVED_PLACEHOLDER_^^^^11max_results11(Ai et al., 25 Jun 2026)11^^^^ minutes for HA, and PRESERVED_PLACEHOLDER_^^^^11max_results11max_results11^^^^ minutes for MI. The Friedman test yielded PRESERVED_PLACEHOLDER_^^^^11max_results11query11^^^^, PRESERVED_PLACEHOLDER_^^^^11max_results11(Nie et al., 9 Jun 2026)11^^^^, and Kendall’s PRESERVED_PLACEHOLDER_^^^^11max_results11max_results11^^^^; Holm-corrected Wilcoxon pairs were non-significant; and FA versus MI showed a large effect size with Cliff’s PRESERVED_PLACEHOLDER_^^^^11max_results11query11^^^^. Kruskal–Wallis tests found no significant differences by expertise in completion time PRESERVED_PLACEHOLDER_^^^^11query11query11^^^^ or exploration time PRESERVED_PLACEHOLDER_^^^^11query11HiLSVA (Ai et al., 25 Jun 2026) OR HierSVA (Nie et al., 9 Jun 2026)11^^^^, while interpretation accuracy was reported as PRESERVED_PLACEHOLDER_^^^^11query11max_results11^^^^ for experts, PRESERVED_PLACEHOLDER_^^^^11query11query11^^^^ for scientists, and PRESERVED_PLACEHOLDER_^^^^11query11(Ai et al., 25 Jun 2026)11^^^^ for novices ^^^^^^^^11(Ai et al., 25 Jun 2026)11

The final-task autonomy preference exposed an explicit tradeoff. FA was chosen by ^^^^11(Nie et al., 9 Jun 2026)11^^^^ participants, HA by ^^^^11max_results11^^^^, and MI by ^^^^11query11^^^^, which the paper interprets as a tradeoff between efficiency and control: familiarity increases preference for efficiency, but some participants still preferred more oversight. The post-study questionnaire reported an overall average of PRESERVED_PLACEHOLDER_^^^^11query11max_results11^^^^ on a ^^^^11HiLSVA (Ai et al., 25 Jun 2026) OR HierSVA (Nie et al., 9 Jun 2026)11^^^^–^^^^11max_results11^^^^ Likert scale, with especially high scores for the ability to review and approve actions PRESERVED_PLACEHOLDER_^^^^11query11query11^^^^, adjust autonomy PRESERVED_PLACEHOLDER_^^^^11query11(Nie et al., 9 Jun 2026)11^^^^, clarity of feedback PRESERVED_PLACEHOLDER_^^^^11query11max_results11^^^^, usefulness of provenance PRESERVED_PLACEHOLDER_^^^^11query11query11^^^^, and ability to revisit steps PRESERVED_PLACEHOLDER_^^^^11(Ai et al., 25 Jun 2026)11query11^^^^. Mixed-initiative interaction and LTT also received strong scores: natural language plus direct manipulation PRESERVED_PLACEHOLDER_^^^^11(Ai et al., 25 Jun 2026)11HiLSVA (Ai et al., 25 Jun 2026) OR HierSVA (Nie et al., 9 Jun 2026)11^^^^, uncertainty prompts PRESERVED_PLACEHOLDER_^^^^11(Ai et al., 25 Jun 2026)11max_results11^^^^, and benefit from accumulated knowledge PRESERVED_PLACEHOLDER_^^^^11(Ai et al., 25 Jun 2026)11query11^^^^. The lowest score was the ability to complete tasks without assistance PRESERVED_PLACEHOLDER_^^^^11(Ai et al., 25 Jun 2026)11(Ai et al., 25 Jun 2026)11^^^^, which was linked to task complexity and differing expectations across expertise ^^^^^^^^11(Ai et al., 25 Jun 2026)11

Relative to prior systems, HiLSVA is positioned as building on tool-grounded agents such as ChatVis and ParaView-MCP and multi-agent frameworks such as AutoGen and Magentic-UI, while differing by centering sustained human participation through co-planning, co-execution, explicit approvals, provenance-aware rollback and branching, and learn-at-test-time adaptation integrated into a SciVis-specific workflow. Compared with VizGenie, AVA, and NLI^^^^11(Ai et al., 25 Jun 2026)11^^^^VolVis, it emphasizes user control, transparency, and safety; compared with Cocoa, it supports dynamic handoffs and replanning mid-execution; and compared with CowPilot, it provides structured co-tasking and proactive negotiation tailored to SciVis. The reported limitations include latency due to external LLM APIs, dependence of retrieval-based adaptation on feedback quality and the expressibility of tacit domain knowledge, the inability of the study to isolate LTT gains from increased human involvement, the small PRESERVED_PLACEHOLDER_^^^^11(Ai et al., 25 Jun 2026)11max_results11^^^^ sample size, and occasional GUI inconsistencies and crashes mitigated by containers but requiring backend-level fixes ^^^^^^^^11(Ai et al., 25 Jun 2026)11

A nomenclature ambiguity appears in contemporaneous literature. A separate ^^^^11max_results11query11max_results11query11^^^^ paper on LLM-driven hierarchical hardware formal verification states, “HiLSVA in your ^^^^^^^^11query11^^^^^^^^ refers to HierSVA,” but that work concerns a pipeline, dataset, and benchmark for hierarchical RTL formal verification rather than scientific visualization ^^^^^^^^11(Nie et al., 9 Jun 2026)11 This distinction matters because HiLSVA, as defined in the SciVis paper, is specifically a human-in-the-loop agentic system for scientific visualization, not a SystemVerilog assertion framework.