NLI4VolVis: NL Interaction for Volume Visualization

• The paper introduces NLI4VolVis, which integrates semantic segmentation, editable 3D Gaussian splatting, and a multi-agent LLM control layer for intuitive, natural language volume visualization.
• It employs a modular, end-to-end pipeline that leverages multi-view segmentation, CLIP-based open-vocabulary matching, and declarative command execution for precise scene editing.
• Empirical results show significant improvements in interaction efficiency and rendering speed over traditional methods, demonstrating its potential to transform volumetric data exploration.

NLI4VolVis, short for “Natural Language Interaction for Volume Visualization via LLM Multi-Agents and Editable 3D Gaussian Splatting,” is a natural-language interface for volume visualization that combines semantic scene understanding, editable Gaussian-based rendering, and a multi-agent LLM control layer. Its purpose is to let users explore, query, explain, and edit volumetric scenes through natural language rather than through manual transfer-function tuning and low-level rendering controls. The system was introduced as a response to two longstanding tensions in volume visualization: direct volume rendering remains computationally expensive and interaction-heavy, while fast neural rendering methods by themselves do not provide semantic-level control or accessible authoring workflows (Ai et al., 16 Jul 2025).

1. Conceptual scope and problem setting

NLI4VolVis is situated at the intersection of volume visualization, neural scene representations, and visualization-oriented natural language interfaces. The system begins from the observation that traditional volume visualization methods, especially direct volume rendering, are constrained by rigid transfer function designs, high computational cost, and an interaction model that is weak at the semantic level. Even when transfer functions are expressive, users still need to manipulate intensity mappings and rendering parameters directly, which is difficult for non-experts. The system therefore frames the central problem not as rendering alone, but as semantic interaction with volumetric scenes through natural language (Ai et al., 16 Jul 2025).

This places NLI4VolVis within a broader V-NLI lineage in which natural language acts as a complementary modality to direct manipulation rather than a total replacement for it. The survey literature characterizes V-NLI systems through seven stages—query interpretation, data transformation, visual mapping, view transformation, human interaction, dialogue management, and presentation—and emphasizes ambiguity handling, multimodality, and context preservation as recurring design concerns (Shen et al., 2021). NLI4VolVis specializes these concerns to volumetric data, where object identity, viewpoint, opacity, and lighting are not merely chart properties but components of a 3D scene model.

A common misconception is that the system merely adds a chat interface to an existing renderer. The published description explicitly rejects that interpretation: the contribution is a layered architecture that combines semantic segmentation, open-vocabulary multimodal understanding, editable 3D Gaussian scene reconstruction, and multi-agent LLM orchestration. This suggests that its novelty lies in the coupling of representation, semantics, and interaction rather than in dialogue alone.

2. End-to-end pipeline and scene decomposition

The NLI4VolVis pipeline is organized into four tightly connected layers: multi-view semantic segmentation, 3D open-vocabulary understanding via CLIP, editable Gaussian training via iVR-GS, and a multi-agent LLM orchestration layer with function calling (Ai et al., 16 Jul 2025). The workflow begins by rendering a volumetric dataset into multi-view images, typically with ParaView and a preset transfer function. These views are segmented into semantic components, and each component becomes a “basic scene” used to train its own editable iVR-GS model. The per-component Gaussian models are then composed into a complete scene.

A key architectural point is that the system is deliberately method-agnostic at the segmentation stage. It supports pre-segmented labels when available, TF-based segmentation for simple scientific volumes, voxel-based segmentation methods, cross-frame object tracking with SAM 2 to improve consistency across views, and 3DGS-based semantic segmentation approaches such as LangSplat and SAGD. The paper states that no single method is universally best, so modularity at this stage is not incidental but a design requirement. For datasets without ground-truth labels, segmentation is performed on rendered multi-view images and the editable Gaussians are reconstructed from those segmented views.

This decomposition strategy matters because downstream editing and querying are performed at the component level rather than on a monolithic volume. A plausible implication is that semantic granularity and editing precision become tightly coupled: better component separation yields more precise object references, more localized edits, and less ambiguity in the language interface.

3. Semantic grounding and open-vocabulary object retrieval

NLI4VolVis uses CLIP embeddings to connect user language to segmented 3D components. Because CLIP operates over image-text pairs rather than raw volumetric fields, the system constructs a semantic descriptor for each 3D component by selecting its most informative rendered views. Informativeness is ranked with an entropy criterion on the alpha or opacity channel:

$H(I)=-\sum_{i=1}^{N} p_{i}\log p_{i},$

where $p_i$ is the normalized opacity value at pixel $i$, and $N$ is the number of pixels (Ai et al., 16 Jul 2025).

The top-$k$ frames with the highest entropy are selected, their CLIP image embeddings are computed, and the embeddings are averaged. If textual metadata exists, the system also encodes the description and combines text and vision embeddings. User queries are then encoded with CLIP and compared against each object embedding via cosine similarity; the object with the highest similarity is treated as the target of the language reference. This supports open-vocabulary references such as “the box,” “the storage container,” or “the square-shaped object.”

The same entropy-ranked views also support best-view selection. In that sense, entropy serves a dual role: it identifies semantically informative views for language grounding and camera positions that are informative for presentation. The paper further notes that open-vocabulary matching results are exposed in an action log, which makes the retrieval stage inspectable rather than opaque.

An important caveat is that open-vocabulary grounding is not guaranteed to be self-validating. The authors explicitly identify the risk that a visionless LLM may accept an irrelevant CLIP match. This is one of the clearest failure modes in the architecture and motivates the paper’s call for stronger automatic validation.

4. Editable 3D Gaussian representation and declarative execution

The rendering and editing backend is built on iVR-GS, an inverse volume rendering system based on 3D Gaussian splatting. Each 3D Gaussian primitive is defined as

$$G(\mathbf{x}) =\exp(-\frac{1}{2}(\mathbf{x}-\mathbf{\mu})^\top \Sigma^{-1}(\mathbf{x} -\mathbf{\mu})),$$

with $\mathbf{\mu}$ as the mean and $\Sigma$ as the covariance. Each primitive is parameterized by $$\{\mathbf{\mu}_{i} ,\mathbf{s}_{i} ,\mathbf{q}_{i} ,o_{i} ,\mathbf{c}_{i}\}$$, and rendering follows standard Gaussian splatting:

$$\mathbf{C} =\sum_{i\in N} \mathbf{c}_{i} \alpha_{i} \prod_{j=1}^{i-1}(1-\alpha_{j}).$$

iVR-GS augments this representation with normals and editable appearance variables, including offset color $p_i$0, ambient, diffuse, and specular coefficients $p_i$1, and shininess $p_i$2 (Ai et al., 16 Jul 2025).

The paper states that the voxel color under the Blinn-Phong model is expressed as $p_i$3, where the palette color $p_i$4 is frozen during training as the mean color of the multi-view training images. Two regularizers are used: $p_i$5 sparsity on $p_i$6 to prevent excessive color shifts, and bilateral smoothness on $p_i$7 to keep shading coherent in uniform regions. After training, the system can independently edit each component’s color, opacity, lighting magnitude, and lighting direction, and multiple components can be composed by concatenating their Gaussian parameters.

Execution is mediated through a declarative command layer. The iVR-GS engine runs as a server and receives structured commands for changing color, opacity, lighting, camera direction, field of view, render mode, background color, saving images, resetting views, and triggering InstructPix2Pix stylization. This command layer is crucial because it converts LLM outputs into auditable visualization actions rather than free-form text. It also distinguishes NLI4VolVis from systems that stop at semantic parsing or recommendation without a geometry-aware execution substrate.

5. Multi-agent language interface and supported interactions

The natural-language layer is implemented as a hierarchical multi-agent system following a visualization-perception-action loop. The agents have distinct roles, including parsing user intent, performing open-vocabulary object lookup, generating commands, and managing dialogue. A core agent inspects the current visualization state, reasons about whether the user’s objective has been achieved, and either terminates or initiates another round of actions. This iterative structure is used for tasks that require refinement, such as best-view selection or object highlighting (Ai et al., 16 Jul 2025).

Function calling is the mechanism by which the agents operate on the visualization. The major tools listed in the paper are open-vocabulary object querying, scene editing, knowledge-based QA, best-view selection, and 2D stylization. Open-vocabulary querying uses the CLIP-based similarity pipeline; scene editing adjusts per-component appearance and global scene parameters; knowledge-based QA reads dataset-specific metadata; best-view selection uses entropy-ranked frames; and stylization uses InstructPix2Pix to apply prompt-driven 2D edits to a selected view or object.

The interaction repertoire is organized around four major capabilities. First, users can perform open-vocabulary object querying without relying on predefined labels. Second, the editable Gaussian backend enables real-time scene editing with immediate feedback. Third, best-view selection lets the LLM reposition the camera to a more informative viewpoint. Fourth, 2D stylization supports presentation-oriented transformations such as “turn the fish into a cyborg.” The paper explicitly notes that 2D stylization was chosen over 3D style transfer because it is more immediate and flexible, though it can introduce multi-view consistency issues.

In relation to prior V-NLI systems, this interaction model differs sharply from tabular and chart-authoring pipelines. NL4DV converts natural-language queries over tabular data into a JSON analytic specification containing attributeMap, taskMap, and visList, primarily for Vega-Lite chart generation (Narechania et al., 2020). Authoring-oriented NLI work translates utterances into executable editing actions of the form $p_i$8 for chart editors such as Excel and VisTalk (Wang et al., 2022). NLI4VolVis instead maps language to declarative volume-visualization commands acting on an editable 3D scene representation. This suggests an extension from chart specification and chart editing toward object-level scene manipulation in volumetric space.

6. Relation to neural rendering and volume visualization research

NLI4VolVis is not a rendering method in isolation; it is an interaction framework built atop an editable neural representation. Its nearest technical neighbors therefore come from several different research lines. NeuLF represents a scene as a neural light field, using a scene-specific MLP that maps a 4D two-plane ray parameterization $p_i$9 directly to RGB color and optionally depth for fast novel view synthesis (Li et al., 2021). V4D replaces a capacity-limited pure MLP dynamic radiance field with a voxel-backed 4D representation plus conditional positional encoding and LUT-based refinement for dynamic novel view synthesis (Gan et al., 2022). NeMF addresses inverse volume rendering by modeling scenes as microflake volumes with density, albedo, microflake normal, and roughness, enabling relighting, material editing, and scattering effects under unknown illumination (Zhang et al., 2023). StyleRF-VolVis uses a base NeRF, a palette color network, and an unrestricted color network to separate content from style and support photorealistic and non-photorealistic editing for expressive volume visualization (Tang et al., 2024).

These systems address representation learning, rendering efficiency, inverse rendering, or style transfer. NLI4VolVis instead centers on semantic querying, language-grounded scene control, best-view selection, and editable interaction over volumetric scenes (Ai et al., 16 Jul 2025). A plausible implication is that it should be read less as a direct competitor to NeuLF, V4D, NeMF, or StyleRF-VolVis than as a system layer that could, in principle, sit above different neural scene representations, provided they expose sufficiently editable object-level parameters.

The paper also inherits concerns that the V-NLI survey identifies as field-wide: ambiguity, underspecification, discoverability, multimodality, dialogue state, and presentation (Shen et al., 2021). What changes in NLI4VolVis is the target of those concerns. Ambiguity is no longer only about attribute names or chart types; it is also about semantic component identity, viewpoint, lighting semantics, and the validity of open-vocabulary retrieval over volumetric objects.

7. Empirical results, limitations, and significance

The implementation renders 92 multi-view images using icosphere sampling in ParaView with the NVIDIA IndeX plugin, at $i$0 resolution for training and inference. Editable Gaussian training uses two stages: 30,000 iterations for base 3DGS training followed by 10,000 iterations for editable Gaussian optimization. Training each basic scene takes about seven minutes on average. The system supports multiple LLMs through API, including GPT-4o, DeepSeek-V3, and Llama3.2-90B-Vision; the appendix reports that the maximum message count was set to ten to balance responsiveness and memory. The authors also note later interface improvements such as deque-based memory management, a Reset All button, and real-time processing indicators (Ai et al., 16 Jul 2025).

Evaluation combines eight case-study datasets—backpack, carp, chameleon, FLARE human organ, kingsnake, hurricane, mantle temperature, and supernova—with a user study. The system reports interactive performance across these volumes, with 3DGS inference at over 100 FPS on an RTX 4090 regardless of volume size, whereas DVR in ParaView drops to 5.09 FPS on the largest dataset, yielding up to a 20× speedup. Stylization with InstructPix2Pix runs in about ten seconds per edit, while StyleGaussian required over three hours of training on an RTX 4090 and introduced artifacts on scientific data, which is why it was not adopted.

The case studies are used to demonstrate different interaction modes. On the carp dataset, users can request a guided tour, isolate fins, highlight the pectoral fin in red, adjust the view, and brighten the scene. On the FLARE human organ dataset, users can ask for only the digestive system. On the supernova dataset, users can ask what a colored region means, obtain an explanation identifying it as the expanding shockwave, and add a legend. The backpack case study illustrates robustness to vague language by resolving multiple references to the same object.

The user study involved eight participants and four phases: introduction, exploration, task execution, and post-questionnaire. Participants explored the interface with backpack and chameleon and then completed 23 tasks across carp, mantle temperature, and FLARE. Average exploration time was 9.9 minutes, average task completion time was 35.1 minutes, and participants averaged 24.4 natural-language interactions, 20.6 of which were effective, with only 3.4 non-NL interactions. The mean NL usage rate was 88.1%. Domain knowledge QA accuracy averaged 8.75 out of 9. Reported survey scores included 4.38/5 for efficient interaction, 4.25/5 for completing tasks without assistance, 5.0/5 for ease of learning, and 4.38/5 for interface intuitiveness.

The limitations are also explicit. Participants noted latency, occasional command misinterpretation, and a need for clearer processing feedback and better documentation. At the system level, the cooperative and rule-based multi-agent design helps coordination but reduces flexibility; the hierarchical structure can increase latency and create single points of failure; and the system lacks strong validation of tool outputs, especially when an irrelevant CLIP match is accepted. The authors therefore identify automatic validation, better memory management, broader benchmarking, and support for more diverse real-world volumetric datasets as future needs.

Taken together, these results position NLI4VolVis as a semantic interaction framework for volumetric data rather than merely a renderer or a dialogue frontend. Its significance lies in showing that natural-language control over volume visualization can be grounded in component-level semantics, executable scene edits, and fast editable Gaussian rendering within a single operational system (Ai et al., 16 Jul 2025).