SonoCraftAR: AR Sound Visualization

• SonoCraftAR is a proof-of-concept framework that empowers Deaf and hard-of-hearing users to create dynamic, sound-reactive AR visualizations using natural language prompts.
• The system integrates real-time audio signal processing with a multi-agent LLM code-generation pipeline to map audio features into visual parameters in a Unity-driven AR environment.
• Its modular architecture supports error handling, rapid code synthesis, and seamless AR streaming to HoloLens 2, offering enhanced accessibility and user-controlled aesthetics.

SonoCraftAR is a proof-of-concept framework for enabling Deaf and hard-of-hearing (DHH) users to author personalized, real-time, sound-reactive augmented reality (AR) visualizations using natural language prompts. The system integrates real-time audio signal analysis, a multi-agent LLM code-generation pipeline, procedural vector graphics, and AR display hardware, supporting open-ended authoring of dynamic visual sound representations (Lee et al., 25 Aug 2025). Unlike prior approaches, which either offer fixed visualizations or limited template adaptation, SonoCraftAR emphasizes end-user control over the aesthetic and functional mapping of sound to visual signals.

1. System Pipeline and Architecture

SonoCraftAR employs a sequential, modular architecture with explicit interfaces between user input, LLM-driven code synthesis, audio processing, and real-time rendering. The pipeline consists of the following principal stages:

1. Natural-Language Prompt Input: The user enters a short text prompt (e.g., "a wave," "pulsing arcs") in a Unity author's UI on a Windows host.
2. Prompt Enhancement Agent (LLM): This LLM agent—using OpenAI o3—expands the user intent into structured guidelines specifying visual primitives (Shapes API calls), animation strategies (e.g., pulsing, hue shifts), and relevant Unity namespaces.
3. Code Generation Agent (LLM): Another LLM agent generates a complete Unity C# script in a one-shot or few-shot style, implementing an UpdateVisual(float dominantFreq) method. The script employs the Shapes vector-graphics library and sets up necessary rendering references.
4. Compilation and Error Handling: Scripts are compiled on the laptop with the Roslyn C# compiler. If compilation fails, the script, error messages, and user prompt are passed to the Code Checker Agent (LLM), which applies domain-specific fixes until successful compilation.
5. AR Streaming: Holographic Remoting streams the live Unity application to the HoloLens 2 headset.
6. Audio Signal Acquisition and Processing: Concurrently, a Python server samples stereo audio (48 kHz) every 100 ms, performs FFT-based dominant frequency extraction, normalizes to a 0–10 scale, and delivers this value to Unity via WebSocket in each update frame.
7. Visual Response: On each Unity update, the AR interface reads the latest normalized frequency and updates visual parameters (shape size, color, thickness) in real time.

The block-diagram organization:

$20$4

This separation reduces cross-component error propagation and enables stepwise recovery (e.g., compile error correction) (Lee et al., 25 Aug 2025).

2. Real-Time Audio Signal Processing

Audio processing is performed outside Unity, designed for low-latency responsiveness. The signal processing pipeline includes:

• Acquisition: Audio is sampled at 48,000 Hz (stereo), re-sampled into 100 ms mono blocks by averaging channels.
• Windowing: Each block is multiplied by a Hann window

$w[n] = 0.5 \cdot (1 - \cos(2\pi n/(N-1)))$

for $n = 0, \ldots, N-1$ to reduce spectral leakage.

• FFT and Magnitude Analysis: A real-valued FFT via NumPy’s rfft produces the spectrum $X[k]$, returning only positive frequencies.
• Dominant Frequency Selection: Only bins corresponding to $20$ Hz $\leq f_k \leq 8000$ Hz are considered; the dominant frequency is extracted as the frequency with maximal magnitude:

$$f_{\mathrm{dom}} = \mathop{\mathrm{arg\,max}}_{k \in [k_\mathrm{min}, k_\mathrm{max}]} |X[k]|$$

• Log-Normalization: To map perceptual salience to a normalized scale,

$$f_{\mathrm{norm}} = 10 \cdot \frac{ \log f_{\mathrm{dom}} - \log 20 }{ \log 8000 - \log 20 }$$

with clamping to $[0, 10]$.

• Streaming: The server transmits $f_{\mathrm{norm}}$ as a float over WebSocket to Unity for each frame.

Typical end-to-end latency is $100$ ms (audio chunk) $n = 0, \ldots, N-1$0 $n = 0, \ldots, N-1$1 ms (FFT/WebSocket) $n = 0, \ldots, N-1$2 one Unity update ($n = 0, \ldots, N-1$3 ms), totaling $n = 0, \ldots, N-1$4–$n = 0, \ldots, N-1$5 ms (Lee et al., 25 Aug 2025).

3. Multi-Agent LLM-Driven Procedural Code Generation

The procedural authoring component is organized as a cascade of three LLM-based agents:

• Prompt Enhancement Agent: Converts terse user input into a structured design brief specifying visual primitives, animation parameters, and Unity API reminders.
• Code Generation Agent: Synthesizes fully-formed C# scripts following one-shot prompting protocols, ensuring the inclusion of required methods and supporting the Shapes library.
• Code Checker Agent: Detects and automatically corrects common errors (type suffixes, missing references) and iterates until the script compiles under Roslyn.

Example prompt transformation:

• User: “make a pulsing circle that changes color from blue to red”
  • Enhancement: Specify Shapes.Circle, animate radius = lerp(0.1,0.5, domFreq/10), color transition via Color.Lerp(Color.blue, Color.red, domFreq/10).
  • Generated code instantiates the circle, applies per-frame updates reflecting the latest $n = 0, \ldots, N-1$6.

This multi-agent workflow reduces hallucination and clarifies division of labor: design decisions are separated from low-level code synthesis (Lee et al., 25 Aug 2025).

4. Sound-to-Visual Mapping Functions

SonoCraftAR visual mappings are parametric and are embedded directly in the LLM-generated code's update loop. Currently, the only extracted audio feature is normalized dominant frequency ($n = 0, \ldots, N-1$7):

• Size mapping:

$n = 0, \ldots, N-1$8

where $n = 0, \ldots, N-1$9 is base size, and $X[k]$0 controls linear ($X[k]$1) or compressive ($X[k]$2) scaling.

• Color/Hue mapping:

$X[k]$3

with color set via $X[k]$4.

• Thickness:

$X[k]$5

• Parametric position/offset: For waveforms,

$X[k]$6

Typical generated code applies these formulas in the UpdateVisual() method, producing direct visual feedback proportional to real-time audio changes (Lee et al., 25 Aug 2025).

5. AR Runtime and Rendering

• Unity Integration: The runtime uses Unity 2022.3.52f1 with Mixed Reality Toolkit 2.8.3. Shape instancing and animation are performed with the Shapes library for efficient runtime property modification.
• Code Loading: Compiled scripts are hot-swapped into the live Unity domain, allowing rapid iteration without session restarts. Roslyn compile times are under 1–2 s for individual scripts.
• Display: Holographic Remoting sends the AR stream to HoloLens 2; the headset performs only tracking and display, while all computation occurs on the laptop.
• Responsiveness: The audio pipeline updates every 100 ms; Unity’s Update loop at 60 fps ensures smooth interpolation of $X[k]$7. No stuttering or dropped frames observed beyond the inherent input window delay.
• Performance Benchmarks:
  • LLM generation time for a visualization: $X[k]$8 s (prompt: “a sound wave”)
  • Audio processing + networking: $X[k]$9110 ms
  • Holographic Remoting: $20$030 ms (Lee et al., 25 Aug 2025)

6. Evaluation Methods and Results

No formal usability study was conducted in the reported work. Instead:

• Demonstrations: Eight example interfaces (arrows, pulsing arcs, waves, volume bars) visualizing music clips (e.g., Beyoncé’s "Love on Top") were implemented using the LLM-driven workflow.
• Measured Metrics:
  • LLM generation latency: $20$1 s
  • Audio-to-visual update latency: $20$2–$20$3 ms
• Proposed Study Design: Future studies are to include DHH users across the communication/preference spectrum (ages 18–65), with authoring/refinement tasks and metrics such as the System Usability Scale (SUS), completion time, prompt iteration count, and qualitative ratings of personalization and clarity.
• Quantitative usability, expressiveness, or task completion statistics are deferred to future work (Lee et al., 25 Aug 2025).

7. Limitations, Challenges, and Future Directions

The current implementation and experimental design involve several noteworthy limitations and open research directions:

• Feature Limitation: Only dominant frequency is visualized; no amplitude, timbre, or multi-source analysis. Expanded mappings to loudness, spectral centroid, rate-of-onset, and spatialization are future targets.
• Personalization-Quality Trade-off: LLM-generated visualizations can be highly novel but sometimes inconsistent or unclear, complicating the trade-off between personalization and effective representation.
• System Portability: Reliance on an external laptop for compilation/execution and Holographic Remoting precludes on-device, untethered AR authoring. Embedded .NET compilation on HoloLens (or Magic Leap) remains an open challenge.
• Error Handling: While the Code Checker agent mitigates common compile/runtime errors, robust recovery from rare or complex failures is not guaranteed.
• Enhancements in Authoring:
  • Hybrid authoring modes offering curated templates alongside freeform LLM prompts ("creativity slider").
  • Integration of multimodal AI (vision-LLMs) for design critique, automated feedback, or auto-correction.
  • Richer interactions via voice commands, gesture input, undo/redo, and browser-based authoring environments (e.g., p5.js/d3.js with WebXR).
  • Automated diagnosis (e.g., by analyzing hologram video with VLMs) and self-correction of visualizations.
• Research Recommendations:

1. Conduct in-situ user studies with DHH participants to quantify usability and learnability.
2. Reduce authoring latency (goal: <5 s) by leveraging smaller LLMs or on-device AI.
3. Support full authoring/compilation on AR headsets.
4. Open-source the pipeline for adaptation to other sensory modalities (e.g., haptics, visual/tactile representations) (Lee et al., 25 Aug 2025).

A plausible implication is that extending the procedural-authoring approach to extract and map richer sets of audio descriptors, alongside closing the authoring latency and deployment gaps, could facilitate broad adoption of adaptive AR accessibility tools. Integration with recent context-aware AR sonification pipelines (Schütz et al., 3 Aug 2025) highlights the importance of both personalization (SonoCraftAR’s focus) and perceptual realism, suggesting a future research trajectory combining procedural, multimodal authoring for comprehensive multisensory accessibility.