Sensorium Arc: Multimodal Oceanic AI System

• Sensorium Arc is a multimodal interactive AI system for oceanic data exploration and eco-art that blends scientific analysis with poetic narrative.
• It integrates a modular, multi-agent pipeline utilizing retrieval-augmented LLMs, formal grammar parsing, and real-time audiovisual rendering.
• The system operationalizes eco-aesthetic philosophy by personifying the ocean, enabling users to experience dynamic environmental changes through immersive visuals and natural dialogue.

Sensorium Arc is a real-time, multimodal interactive AI agent system designed for oceanic data exploration and interactive eco-art. Developed in partnership with the Center for the Study of the Force Majeure and influenced by Newton Harrison’s eco-aesthetic philosophy, Sensorium Arc operationalizes the personification of the ocean as a poetic, conversational speaker. The system enables users to engage in natural, spoken dialogues with an “Ocean” AI agent, which responds with a blend of scientific insight and ecological poetry, dynamically grounding content in high-dimensional environmental data. The architecture employs a modular, multi-agent approach, retrieval-augmented LLMs, formal grammar parsing, and real-time audiovisual rendering to facilitate immersive exploration and affective access to complex marine datasets (Bissell et al., 20 Nov 2025).

1. System Architecture

Sensorium Arc is organized as a modular, multi-agent pipeline embedded in Unity, with four primary stages: Input Processing, LLM Pipeline, Response Processing, and Audio-Visual Layers. The pipeline operates as follows:

• Input Processing:
  • Utilizes a proximity sensor (Arduino, $R = 0.5$ m threshold), Whisper-style microphone with active noise cancellation, and Whisper-Tiny speech-to-text (STT) for real-time audio capture and transcription.
• LLM Pipeline:
  • Composed of three stateless agents:
  • 1. Visualization Decider Agent determines which visual resources to trigger.
  • 2. Query Rewriter + Retriever Agent reformulates user queries and performs retrieval over the corpora.
  • 3. Responder Agent generates the final response as the “Ocean.”
• Response Processing:
  • Maps LLM-generated tokens to visualization events and prepares Unity-based text-to-speech (TTS) via Unity Jets.
• Audio-Visual Layers:
  • Includes globe visualizations (NASA EarthData), pre-rendered video overlays (e.g., plankton blooms, plastic dispersion), and synchronized subtitles.

All modules communicate asynchronously via JSON objects and are parallelized across GPU/CPU threads for real-time interaction.

2. Retrieval-Augmented LLM Integration

Sensorium Arc employs a retrieval-augmented, retrieve-then-generate LLM approach with $k=2$ nearest-neighbor document chunks. Data preprocessing involves segmenting the “Harrison Corpus”—a composite of manifestos and scientific papers—into sentences, embedding using all-MiniLM-L12-v2 ($\mathbb{R}^{384}$), and indexing via Approximate Nearest Neighbors (usearch/HNSW).

The real-time inference workflow is as follows:

1. Query Rewriting: Reformulate raw user query $q$ into “clean” query $q'$ using a chain-of-thought prompt in Qwen 8B.
2. Embedding: Embed $q'$ as $e_q \in \mathbb{R}^{384}$.
3. Nearest Neighbor Retrieval: Retrieve top-$k$ sentences $i_1$, $i_2$ by maximizing cosine similarity:

$k=2$0

with $k=2$1.

1. Context Assembly: Retrieve full paragraphs $k=2$2, $k=2$3 containing $k=2$4, $k=2$5, which are prepended to the context window for the Responder Agent, grounding responses in sourced narrative.

3. Multimodal Data and Event Triggers

The data ingestion pipeline encompasses four major sources:

• Time-series oceanographic measurements (CO₂, chlorophyll, SST, currents, Kd)
• Geospatial globe meshes (latitude/longitude mapped textures)
• Pre-rendered video layers (plankton blooms, plastic dispersion, sea-level animation)
• Text-to-speech (TTS) output

Visualization and playback are triggered via two mechanisms:

• Keyword Detection:
  • A fixed list of tokens (e.g., “chlorophyll,” “plastic,” “acidification”) mapped to visualization layers.
  • Pseudocode:
  • $\mathbb{R}^{384}$8
• Semantic Parsing:
  • GBNF grammar extracts temporal and regional phrases from user queries.
  • $\mathbb{R}^{384}$9
  • A recursive-descent parser identifies (time, location) pairs for dynamic texture and camera selection.

4. Conversational Design and Training

The “Ocean” persona is instantiated in the Responder Agent, which maintains the following per-turn context:

$k=2$6

While intent is not explicitly modeled, chain-of-thought prompts direct the agent to answer scientific questions, weave ecological poetry, and reference paragraphs $k=2$7, $k=2$8.

The Responder is trained on standard cross-entropy loss,

$k=2$9

with additional reward-shaping heuristics penalizing responses that exceed maximum length or omit any retrieved fact:

$\mathbb{R}^{384}$0

Parameters $\mathbb{R}^{384}$1, $\mathbb{R}^{384}$2 are tuned empirically in RL or via prompting.

Prompt templates support alternation between scientific statements and poetic lines, operationalizing the eco-poetic style: $q$0

5. Real-Time Visualization and Audiovisual Playback

After response generation:

• Layer Mapping: Visualization Decider’s selected token (e.g., “chlorophyll”) activates corresponding globe or video layer via: $q$1
• Camera Control: Token triggers map to Cinemachine camera motions. E.g., on “chlorophyll” trigger: activate layer and set camera position (lat, lon, zoom).
• Temporal Mapping: Year cues map to frames using:

$\mathbb{R}^{384}$3

• Audio-Visual Synchronization: Master clock coordinates TTS audio and sentence-level subtitles, chunked at sentence boundaries.

6. Eco-Aesthetic and Narrative Approach

Sensorium Arc embraces Newton Harrison’s eco-aesthetic philosophy, reconceptualizing ocean data as narrative symbols rather than mere quantitative abstractions. Key operationalizations include:

• Equal weighting of Harrisons’ manifestos and scientific literature during retrieval.
• Chained prompt templates alternating data-driven commentary with ekphrastic, poetic imagery (e.g., “I carry sunlight in my plankton blooms”).
• Respect for layered temporality, integrating notions of past and future (Force Majeure) within a single narrative.

This paradigm supports the mediation of intuitive, affective access to high-dimensional marine datasets and foregrounds the personification of the ocean as a narrative entity.

7. Prototypical Interaction Workflow

A typical user-agent exchange proceeds as follows:

Stage	Description	Mechanism/Output
Input Processing	STT transcription on proximity sensor trigger	$\mathbb{R}^{384}$4 “Ocean, what happened to plastic pollution in the North Atlantic around 2010?”
Visualization Decider	Few-shot prompted classification	Outputs VISUAL: plastic
Query Rewriter	Chain-of-thought reasoning, marker stripping	$\mathbb{R}^{384}$5 “plastic pollution North Atlantic 2010 trend”
Retrieval	Embedding and nearest-neighbor search	Retrieves $\mathbb{R}^{384}$6, $\mathbb{R}^{384}$7 on plastic dispersion
Responder	LLM generation with context, scientific and poetic blending	Generates Ocean’s answer, e.g. “7% per year” plastics increase, poetic current metaphor
Response Processing	Maps trigger to visualization and extracts year cue	Triggers PLASTIC_VIDEO, sets globe frame to 2010, TTS output
Audio-Visual Layers	Visualization/camera activation, subtitles, synchronized playback	Dynamic globe video, camera pan, captions in sync with TTS

The result is a multimodal experience in which the Ocean agent responds with grounded data and metaphor while the visualization layer projects scientifically accurate, time-specific environmental changes.

---

Sensorium Arc exemplifies the integration of modular LLM-agent architectures, retrieval-enhanced groundedness, formal grammar pipelines, and dynamic Unity-based rendering for immersive, ecopoetic interaction with oceanic data (Bissell et al., 20 Nov 2025).