---
title: 'HiLSVA: Human-in-the-Loop Agentic SciVis System'
url: https://www.emergentmind.com/topics/hilsva
type: topic
---

# HiLSVA: Human-in-the-Loop Agentic SciVis System

Searching arXiv for the cited papers to ground the article.
{"query":"HiLSVA 2606.26614 OR HierSVA 2606.13706","max_results":10}
Searching separately for each arXiv identifier.
{"query":"2606.26614","max_results":5}
{"query":"2606.13706","max_results":5}
Performing one more title-based search for confirmation.
{"query":"\"HiLSVA: Design and Evaluation of a Human-in-the-Loop Agentic System for Scientific Visualization\"","max_results":5}
HiLSVA is a human-in-the-loop agentic system for scientific visualization (SciVis) that re-centers human analytical control while leveraging LLM-driven automation. It supports mixed-initiative SciVis workflows through a plan-first multi-agent architecture with explicit human oversight, stepwise provenance tracking, learn-at-test-time adaptation from user feedback, and sandboxed execution. Rather than treating agentic visualization as autonomy-first execution, HiLSVA frames it as a collaborative process in which humans can plan, approve, intervene, and steer, while the system adapts at test time through structured reflection and user feedback [2606.26614].

## 1. Motivation and conceptual framing

HiLSVA is motivated by a specific limitation in prior LLM-based SciVis systems: they have demonstrated impressive tool use and visualization generation, but they often sideline the human analyst, reduce workflow transparency, and limit oversight. In the formulation associated with HiLSVA, SciVis is inherently human-centered. Analysts must decide when to delegate or withdraw initiative, understand how and why an agent takes actions, and retain authority over interpretation. The system therefore rejects the assumption that maximizing automation is either realistic or desirable for open-ended, visually grounded tasks [2606.26614].

The design objective is not merely to improve output generation, but to support mixed-initiative coordination, explicit oversight, provenance-aware workflows, and safe execution. HiLSVA operationalizes this objective by making planning explicit and editable, enforcing stepwise approval, tracking both prospective and retrospective provenance, enabling direct manipulation of visualizations, and adapting at test time from human feedback without sacrificing control or reproducibility. This suggests a shift from agent substitution toward agent augmentation of scientific reasoning, with the human analyst treated as a first-class participant in the agent team rather than as an external supervisor.

The core contributions are organized around six mechanisms: human-in-the-loop agentic design; a plan-first multi-agent architecture; explicit oversight mechanisms; stepwise provenance tracking; sandboxed execution; and learn-at-test-time adaptation from user feedback. Together, these mechanisms define HiLSVA as a collaborative SciVis system rather than a purely autonomous visualization agent [2606.26614].

## 2. Multi-agent architecture and execution environment

HiLSVA is built around an orchestrated set of specialized components. The orchestrator interprets intent, constructs an explicit stepwise plan, coordinates specialized agents, monitors execution, triggers replanning, and mediates oversight. Specialized agents and tools include a ParaView agent built on ParaView-MCP for reliable, parameterized interaction via function-oriented tool calls; a code agent described as ChatVis-style for robust generation and iterative refinement of Python and shell scripts with RAG over documentation and examples plus error-driven correction; a web surfer for external information retrieval; a file surfer for directory inspection and file transformations such as timesteps detection; and a self-improving agent responsible for learn-at-test-time adaptation, confidence assessment, self-reflection, and query generation [2606.26614].

The execution substrate is an explicit sandbox. HiLSVA uses per-session Docker containers for visualization in ParaView, code execution, web browsing, and file access, with a controlled shared workspace. This organization supports session isolation, reproducibility, parallel tasks, and safe experimentation. The use of isolated containers is central to the system’s safety model, because all agent actions occur inside these bounded environments rather than on the host system directly.

The user interface combines several views of the workflow: mixed-initiative chat for natural-language planning and approval; a workflow monitor panel for stepwise provenance and clickable state restoration; a real-time visualization engine and browser; a plan gallery and knowledge base; and a session monitor sidebar. The implementation details reported for the system specify GPT-5.2 for the orchestrator and Claude-Sonnet-4.6 for specialized agents, with ParaView running inside Docker and interfaced through a ParaView-MCP server [2606.26614].

A notable property of the architecture is backend agnosticism. The reported deployment model states that additional tools such as VMD, napari, and TTK can be plugged in via specialized agents or MCP-like interfaces. A plausible implication is that the architecture is intended as a general pattern for collaborative visualization systems rather than as a ParaView-only prototype.

## 3. Plan-first interaction, human oversight, and provenance

HiLSVA’s workflow begins with explicit planning. The orchestrator generates a stepwise plan, assigns each step to an agent, and exposes the plan for user editing, including reordering, adding or removing steps, and modifying instructions. Execution begins only after approval. During execution, initiative flows in both directions: users can intervene through chat or direct GUI manipulation, agents proactively request clarification when confidence is low, and autonomy levels can be adjusted with guarded execution and explicit approvals for sensitive actions [2606.26614].

The oversight model is therefore procedural rather than merely post hoc. User approval is required for critical or irreversible operations, including executing generated code or sensitive visualization tool calls. The system also solicits clarification when uncertainty is high. This makes oversight a structured part of execution rather than a retrospective audit step.

Provenance is recorded at two levels. Prospective provenance is the planned sequence of actions. Retrospective provenance includes executed actions and tool calls, software states such as ParaView pipeline configuration, intermediate and final visualization outputs such as images and animations, and interaction traces including approvals and user interventions. The workflow monitor exposes steps and statuses, and users can click any prior step to restore the corresponding state and branch the plan from that point. Saved workflows therefore include both the plan and stateful execution artifacts, enabling stateful reuse across sessions [2606.26614].

Direct manipulation is integrated into the same rendering session. HiLSVA runs ParaView inside Docker and supports MCP tool calls, generated Python scripts, and direct GUI manipulation by the user within a shared visualization state. A user can take control to adjust transfer functions, camera, or streamline parameters and later return control to the agent. The significance of this design is that manual intervention does not force a mode switch to a separate environment; instead, it becomes part of the same provenance-tracked trajectory.

## 4. Formalization and learn-at-test-time adaptation

HiLSVA presents a formal notation for stepwise planning and adaptation. The workflow is expressed with steps $\mathcal{S} = (s_1, s_2, \ldots, s_N)$, environment states $\mathcal{E} = (e_1, e_2, \ldots, e_N)$, per-step actions $a_i \in \mathcal{A}$, internal state $\Theta$, human user $\mathcal{H}$, and interaction budget $B$. The decision policy is given as $\pi : \mathcal{E} \times \Theta \rightarrow \mathcal{A}$, with actions generated as $a_i = \pi(e_i, \theta_i)$ and state updates written as $\theta_{i+1} = f(\theta_i, e_i, a_i, r_i; q_i, h_i)$ [2606.26614].

The learn-at-test-time mechanism is retrieval-based rather than weight-updating. The self-improving agent observes steps, assesses confidence, performs self-reflection in the form of summaries of outcomes and reliability, and, when confidence is below a threshold and within the budget $B$, issues a structured query for clarification. Human feedback is then captured and stored. The knowledge repository $R_i$ is described as a vector-based store of natural-language knowledge items with captions, BERT embeddings, confidence, validity, and recency metadata; items derived from human feedback receive confidence $1$, and foundational items can be marked non-expiring. Before each step, relevant items are retrieved using a composite score and injected into the environment state to guide actions and parameters [2606.26614].

The repository update is formalized as $R_{i+1} = g(R_i, r_i, q_i, h_i)$, and the paper notes the approximation $\theta_i \approx R_i$. Adaptation is therefore externalized into a retrievable knowledge layer rather than embedded as online fine-tuning. The system can reuse prior parameter presets and successful workflow fragments, such as colormap choices for volume rendering or vortex-identification steps. Learn-at-test-time can also be disabled for ablation, which is important for separating the effect of retrieval-based adaptation from the broader effect of mixed-initiative interaction.

This formalization places HiLSVA in a design space where adaptation, transparency, and oversight are coupled. The system’s knowledge is inspectable in the knowledge base, the queries are triggered by explicit confidence assessment, and the adaptation pathway remains reproducible because it is mediated through stored artifacts rather than hidden model updates.

## 5. Interaction modalities and representative SciVis tasks

HiLSVA supports two principal interaction modalities: natural language and direct manipulation. Through chat, users specify goals, edit plans, approve actions, request alternatives, and provide feedback. The system explains, reflects, and asks for clarification where needed. Through direct manipulation, users can take GUI control in ParaView for fine-grained steering of transfer functions, camera, seeding, and tube radius, then hand control back to the agent without leaving the active session [2606.26614].

The task repertoire described for the system includes data loading and inspection, including directory scanning to enumerate timesteps; filtering operations such as slices, contours, stream tracers, tubes, glyphs, and isosurfaces; encoding decisions such as colormaps, scaling by magnitude, and orientation by vectors; layout and animation tasks including volume rendering across timesteps and exporting videos or image sequences; and derived-field computation for scientific analysis, including vortex measures. These are not presented as isolated demos but as representative mixed-initiative workflows.

The reported case studies illustrate the role of human oversight. In the foot CT example, the workflow begins with an initial isosurface to reveal bones, incorporates web retrieval of anatomical references, and then uses direct GUI refinement by the user to improve phalanges visibility. In the hurricane example, the system constructs a $z$-slice visualization, performs histogram analysis, and overlays black contour lines to clarify temperature gradients. In the tornado example, the plan contains three streamline trials; the user selects Trial 2 and restores its state, after which knowledge-guided glyphs are oriented by velocity and scaled by magnitude, followed by tube-radius adjustment after agent reflection. In the combustion example, HiLSVA performs volume rendering of $Y_{OH}$ across 33 timesteps with a preferred colormap retrieved from knowledge, and also generates mixfrac isosurface animations across timesteps [2606.26614].

The half-cylinder analysis makes explicit that HiLSVA is intended for scientific analysis rather than only visual depiction. The workflow computes and visualizes both $Q$-criterion and $\lambda_2$, and stores a successful vortex-identification workflow for reuse. The reported script computes
$$
Q=\tfrac{1}{2}\left(\|\bm{\Omega}\|^2-\|\mathbf{S}\|^2\right),
$$
where
$$
\mathbf{S}=\tfrac{1}{2}(\mathbf{J}+\mathbf{J}^\top)
\quad \text{and} \quad
\bm{\Omega}=\tfrac{1}{2}(\mathbf{J}-\mathbf{J}^\top).
$$
This example is significant because it extends the system beyond appearance-oriented rendering into derived-field analysis workflows that are common in expert SciVis practice.

## 6. Evaluation, relation to prior work, and naming ambiguity

HiLSVA was evaluated through a controlled, within-subjects user study with $N=12$, with expertise distributed as 3 SciVis experts, 4 domain scientists, and 5 novices. Participants used a workstation with an NVIDIA RTX 5090 GPU and two 32-inch $3840 \times 2160$ displays. They completed four tasks—hurricane, foot, tornado, and combustion—with autonomy modes assigned in balanced fashion for the first three tasks: full-autonomous (FA), half-autonomous (HA, LTT off), and mixed-initiative (MI, LTT on). Interaction logs, plans, and outputs were recorded, and a post-study questionnaire measured usability, transparency, and collaboration [2606.26614].

All participants completed the tasks, and domain-specific interpretation accuracy averaged $11.75/12$. For the first three tasks, mean execution times were $9.83 \pm 3.27$ minutes for FA, $11.67 \pm 4.44$ minutes for HA, and $13.50 \pm 4.46$ minutes for MI. The Friedman test yielded $\chi^2(2)=5.09$, $p=0.078$, and Kendall’s $W=0.19$; Holm-corrected Wilcoxon pairs were non-significant; and FA versus MI showed a large effect size with Cliff’s $\delta=-0.51$. Kruskal–Wallis tests found no significant differences by expertise in completion time $(H=0.92, p=0.63)$ or exploration time $(H=0.28, p=0.87)$, while interpretation accuracy was reported as $100\%$ for experts, $100\%$ for scientists, and $95\%$ for novices [2606.26614].

The final-task autonomy preference exposed an explicit tradeoff. FA was chosen by 7 participants, HA by 2, and MI by 3, which the paper interprets as a tradeoff between efficiency and control: familiarity increases preference for efficiency, but some participants still preferred more oversight. The post-study questionnaire reported an overall average of $4.66$ on a 1–5 Likert scale, with especially high scores for the ability to review and approve actions $(4.75 \pm 0.60)$, adjust autonomy $(4.83 \pm 0.37)$, clarity of feedback $(4.92 \pm 0.28)$, usefulness of provenance $(4.92 \pm 0.28)$, and ability to revisit steps $(5.00 \pm 0.00)$. Mixed-initiative interaction and LTT also received strong scores: natural language plus direct manipulation $(4.67 \pm 0.62)$, uncertainty prompts $(4.50 \pm 0.76)$, and benefit from accumulated knowledge $(4.50 \pm 0.65)$. The lowest score was the ability to complete tasks without assistance $(3.92 \pm 1.11)$, which was linked to task complexity and differing expectations across expertise [2606.26614].

Relative to prior systems, HiLSVA is positioned as building on tool-grounded agents such as ChatVis and ParaView-MCP and multi-agent frameworks such as AutoGen and Magentic-UI, while differing by centering sustained human participation through co-planning, co-execution, explicit approvals, provenance-aware rollback and branching, and learn-at-test-time adaptation integrated into a SciVis-specific workflow. Compared with VizGenie, AVA, and NLI4VolVis, it emphasizes user control, transparency, and safety; compared with Cocoa, it supports dynamic handoffs and replanning mid-execution; and compared with CowPilot, it provides structured co-tasking and proactive negotiation tailored to SciVis. The reported limitations include latency due to external LLM APIs, dependence of retrieval-based adaptation on feedback quality and the expressibility of tacit domain knowledge, the inability of the study to isolate LTT gains from increased human involvement, the small $N=12$ sample size, and occasional GUI inconsistencies and crashes mitigated by containers but requiring backend-level fixes [2606.26614].

A nomenclature ambiguity appears in contemporaneous literature. A separate 2026 paper on LLM-driven hierarchical hardware formal verification states, “HiLSVA in your query refers to HierSVA,” but that work concerns a pipeline, dataset, and benchmark for hierarchical RTL formal verification rather than scientific visualization [2606.13706]. This distinction matters because HiLSVA, as defined in the SciVis paper, is specifically a human-in-the-loop agentic system for scientific visualization, not a SystemVerilog assertion framework.

Source: https://www.emergentmind.com/topics/hilsva