SASAV: Self-Directed Agent for Scientific Analysis and Visualization

Abstract: With recent advances in frontier multimodal LLMs (MLLMs) for data understanding and visual reasoning, the role of LLMs has evolved from passive LLM-as-an-interface to proactive LLM-as-a-judge, enabling deeper integration into the scientific data analysis and visualization pipelines. However, existing scientific visualization agents still rely on domain experts to provide prior knowledge for specific datasets or visualization-oriented objective functions to guide the workflow through iterative feedback. This reactive, data-dependent, human-in-the-loop (HITL) paradigm is time-consuming and does not scale effectively to large-scale scientific data. In this work, we propose a Self-Directed Agent for Scientific Analysis and Visualization (SASAV), the first fully autonomous AI agent to perform scientific data analysis and generate insightful visualizations without any external prompting or HITL feedback. SASAV is a multi-agent system that automatically orchestrates data exploration workflows through our proposed components, including automated data profiling, context-aware knowledge retrieval, and reasoning-driven visualization parameter exploration, while supporting downstream interactive visualization tasks. This work establishes a foundational building block for the future AI for Science to accelerate scientific discovery and innovation at scale.

• The paper introduces a fully autonomous multi-agent framework that performs scientific data analysis and visualization without human intervention.
• It details a modular architecture combining data profiling, knowledge retrieval, transfer function design, and optimal view selection driven by advanced MLLMs.
• Empirical results on large volumetric datasets show reduced manual parameter tuning and improved computational efficiency in visualization workflows.

SASAV: Fully Autonomous Agent for Scientific Data Analysis and Visualization

Context and Motivation

Recent advancements in multimodal LLMs (MLLMs) have enhanced the integration of AI into scientific analysis pipelines, notably shifting the role of LLMs from merely passive interfaces to active judges in scientific data understanding and visual reasoning. Existing scientific visualization agents, however, remain dependent on domain experts for dataset-specific guidance or iterative feedback-driven workflows, which severely constrain scalability and efficiency in large-scale scientific data contexts. The SASAV (Self-Directed Agent for Scientific Analysis and Visualization) framework addresses this by introducing a completely autonomous, multi-agent system capable of conducting data analysis and generating effective visualizations without any human-in-the-loop or prior domain knowledge, thus forming a fundamental block in the evolution of AI-for-Science as illustrated in the historical trajectory of agentic systems.

Figure 1: Evolution of AI for Science indicating stages from human-in-the-loop toward full agentic discovery.

Architectural Overview

SASAV embodies a multi-agent workflow that orchestrates autonomous scientific data exploration, leveraging advanced MLLMs for visual reasoning and parameter optimization. The system comprises four primary components:

• Data Profiling: Automated extraction and characterization of dataset metadata and intrinsic scientific objects via ramp-shaped opacity transfer functions and orthogonal viewpoint renderings, invoking evaluator and recognizer agents for preliminary semantic annotation (Figure 2).
• Knowledge Retrieval: Forager agents aggregate keywords and regions of scientific interest through hybrid web search and retrieval-augmented generation from configurable local knowledge bases, maximizing coverage and task alignment (Figure 3).
• Transfer Function Suggestion: Swarm-based parallel Semantic Analyzer (SA) agents reason over sampled isovalues, producing quantitative and textual evaluations for Transfer Function Designer (TFD) to synthesize optimal color and opacity mappings; empirical data benefits from discrete isosurface renderings and simulated data employs continuous DVR (Figures 5 and 6).
• View Selection: Global view sphere sampling (Fibonacci lattice) and MLLM-driven batch comparison recommend anchor viewpoints and exploratory trajectory, balancing informativeness, occlusion minimization, and coverage (Figure 4).

  Figure 5: Architecture of SASAV detailing multi-agent orchestration across data profiling, knowledge retrieval, TF suggestion, and view selection.

Agentic Workflows: Technical Details

Data Profiling

Upon ingestion, SASAV profiles dataset structure and semantics through automated metadata extraction, followed by sequential agentic evaluation of ramp-shaped TF-based grayscale renderings. Multiple RSVs (Ramp Starting Values) are sampled to suppress noise and maximize feature clarity, with downstream evaluators scoring recognizability and optimally selecting parameters for the recognizer. Recognizer agents assign semantic keywords to detected objects, ensuring high fidelity with subsequent semantic analyses.

Figure 2: Object recognition workflow leveraging evaluator and recognizer via RSV-based renderings.

Context-Aware Knowledge Retrieval

Forager agents perform web search using ranking-aware lightweight MLLMs (e.g., GPT-4o-mini), supplemented by vectorized RAG-based domain knowledge base queries (OpenAI text-embedding-3-large). The hybrid approach achieves superior region-of-interest coverage and relevance, overcoming depth, alignment, and statelessness limitations of vanilla web search. Summarization agents consolidate keywords, optimizing downstream guidance for visualization parameter design.

Figure 3: Forager workflow integrating external web search and local knowledge retrieval for domain-specific context.

Transfer Function and Semantic Mapping

Parallel SA agents individually analyze isosurface renderings for a uniformly sampled set of isovalues, outputting 10-point scale salience, occlusion, and confidence scores alongside contextual geometry summaries. Outputs are fully structured as JSON for interoperability. TFD agents then interpolate suggested mappings for simulated data, reject low-confidence isovalues for empirical data, and apply Isovalue Fine-Tuning (IFT) by comparative reasoning to optimize surface smoothness and semantic clarity.

Figure 6: SA and TFD workflow on simulated scientific data, achieving efficient parallel perception and global TF synthesis.

Figure 7: Mapping suggestion workflow for empirical datasets with confidence-based isovalue rejection and local fine tuning.

Viewpoint Selection and Trajectory Planning

32 viewpoints are sampled on the view sphere (Fibonacci lattice), each rendered and ranked for informativeness, redundancy, and occlusion. MLLM-driven ranking and anchor selection generate high-order Catmull–Rom spline trajectories for static and animated visualization. Token usage and inference are optimized for computational and memory efficiency.

Figure 4: View selection process identifying anchor and avoid viewpoints for maximized information delivery.

Evaluation and Empirical Results

SASAV was validated across five diverse volumetric scientific datasets, incorporating empirical (3D scan-based) and simulated (computational model-based) object types with volumes from 260 MB to 7.5 GB. Qualitative assessment demonstrated SASAV's capacity to autonomously highlight key regions—distinct organs in AbdomenAtlas, physical boundaries in simulated flame and Richtmyer data—using contextually appropriate mappings and trajectory plans.

Figure 8: Visualization image generated by SASAV with suggested parameters and trajectory for all datasets.

Performance analyses revealed that TF suggestion (especially SA/TFD parallel inference) dominates computational cost, with empirical datasets exhibiting variable time based on the number of meaningful isovalues. View selection is computationally intensive due to batch image embedding. Token usage is highest during image-rich steps, with input tokens exceeding output due to embedding overhead.

Figure 9: Time consumption breakdown for each SASAV step across datasets.

Figure 10: Token usage statistics for key workflow steps.

SASAV's local and HPC rendering pipeline scales efficiently; for large datasets and animation workflows, distributed rendering on >128 cores achieved drastic latency reductions.

Expert feedback emphasized SASAV's utility in reducing manual parameter search, simplifying collaborative scientific workflows, and enabling cross-platform deployment.

Implications, Limitations, and Future Directions

SASAV establishes a practical fully autonomous framework for scientific visualization, eliminating the need for human-in-the-loop feedback or domain-specific prompting. Theoretical implications include advancing agentic science workflows toward level 3 (fully agentic discovery) in AI for Science. Practically, SASAV accelerates semi-automatic and fully automatic exploratory pipelines for scientific datasets, providing high-quality visualization starting points.

Current limitations stem from variability in output (notably color assignment in simulated data), knowledge base coverage, and restriction to DVR and isosurface rendering. Future research should extend SASAV to multivariate, time-varying, and tensor data, incorporate more specialized visualization techniques (e.g., cutaway, feature-based views), and experiment with video vision transformers for dynamic scientific data understanding. Integration with federated research ecosystems via agent middleware is also hypothesized to improve scalability and resource allocation.

Conclusion

SASAV represents a foundational shift toward fully autonomous agentic scientific analysis and visualization, leveraging multimodal LLMs for comprehensive data understanding, semantic knowledge retrieval, reasoning-driven parameter exploration, and optimized viewpoint selection. Its systematic and scalable architecture is well-positioned to further accelerate scientific discovery as MLLM capabilities continue to evolve.