Multimodal Guidance Simulator

• Multimodal guidance simulators are systems that integrate synchronized sensory channels to study perception, instruction, coordination, and actuation in controlled environments.
• They employ layered modular architectures that separate modality processing from control logic, facilitating flexible experiments with interchangeable perception and guidance components.
• Applications span air–ground traffic simulation, warehouse teleoperation, and pilot-controller interfaces, though standard evaluation protocols remain an open research challenge.

A multimodal guidance simulator is a simulation or interactive testbed that couples multiple sensory or communicative channels with a guidance loop, so that perception, instruction, coordination, and actuation can be studied under controlled conditions. In current literature, the concept spans unified traffic simulation for air–ground collaboration, multimodal command-and-control interfaces, augmented-reality instructional systems, accessible teleoperation platforms, aviation conflict-detection testbeds, and probabilistic training environments for human–robot collaboration. A particularly explicit formulation appears in TranSimHub, which is presented as a unified platform for air–ground collaborative intelligence with synchronized RGB, depth, and semantic segmentation across aerial and ground viewpoints on a shared simulation timeline (Wang et al., 17 Oct 2025).

1. Representative scope and domains

Across the literature, multimodal guidance simulators are not confined to a single application domain. They appear wherever guidance depends on synchronized heterogeneous inputs and where a reproducible environment is needed to study interaction, coordination, or decision-making. In urban transportation, TranSimHub targets UAVs, UAMs, vehicles, pedestrians, and infrastructure in dynamic urban scenes (Wang et al., 17 Oct 2025). In warehouse teleoperation, NavVI combines visual, auditory, and haptic feedback for blind and low-vision operators (Maimuna et al., 20 Jul 2025). In aviation, AIRHILT synchronizes radio communications, camera streams, and ADS-B surveillance data for pilot- and controller-in-the-loop conflict detection (Garib et al., 24 Nov 2025). In obstetric ultrasound, Multimodal-GuideNet jointly models ultrasound video, gaze, and probe motion (Men et al., 2022). In mission monitoring, MIRIAM combines a chat interface with synchronized visualization from SeeTrack and Neptune (Hastie et al., 2018).

System	Domain	Distinctive simulator function
TranSimHub	Air–ground urban transportation	Synchronized RGB/depth/semantic multi-view rendering, communication modeling, causal scene editing
NavVI	Warehouse teleoperation for BLV users	Visible path, clock-face voice cues, proximity-based haptic feedback
AIRHILT	Aviation conflict detection	Radio, vision, ADS-B fusion with JSON-based module swapping
Multimodal-GuideNet	Obstetric ultrasound	Joint gaze–probe guidance from synchronized US, gaze, and IMU streams
MIRIAM	Autonomous mission supervision	Mixed-initiative chat coupled to live mission visualization

This breadth suggests that the defining property of the category is not a particular sensor set, but a common systems objective: maintaining a coherent relation among multimodal observations, guidance signals, and downstream actions.

2. Architectural patterns

A recurrent architectural pattern is layered modularity. TranSimHub separates an Environment Provider Layer, a Simulation and Control Layer, and an Integration Interface Layer. Static roads, intersections, and buildings are imported from OpenStreetMap, dynamic entities are provided by SUMO or user-defined strategies, controllable agents can be assigned custom policies, and standardized APIs connect the simulator to Gym-compatible reinforcement learning, LangChain, Blender, Panda3D, GNS3, and WinProp (Wang et al., 17 Oct 2025). AIRHILT adopts a similarly explicit decomposition: L0 handles scenario orchestration, L1 handles actors and behavior abstractions, and L2A/L2B handle radio/ASR/TTS and camera/vision subsystems, all synchronized by a shared monotonic timebase and exposed through REST/JSON interfaces (Garib et al., 24 Nov 2025).

A second recurring pattern is separation between modality processing and control logic. In XR-CareerAssist, the Unity XR client, application orchestration, integration adapters, AI services, and data layer are distinct, with ASR, NMT, conversational assistance, vision-language interpretation, and TTS invoked through backend endpoints (Tantaroudas et al., 8 Apr 2026). In MIRIAM, mission plans, Neptune status, and REGIME reports are parsed into an SQL database; an NLP engine based on AIML plus a custom parser interprets user input; a processor retrieves relevant state; and SeeTrack provides synchronized visualization (Hastie et al., 2018). These systems differ in domain and modality, but they converge on the same systems principle: the simulator exposes stable interfaces so that perception, reasoning, and guidance modules can be replaced without redesigning the environment itself.

This modularity has methodological consequences. It allows ablation at the subsystem level, supports heterogeneous algorithmic backends, and makes simulator behavior reproducible across policy, perception, and communication experiments.

3. Modalities, synchronization, and spatial alignment

The central technical requirement is not merely multimodality, but synchronized multimodality. TranSimHub renders RGB, depth, and semantic segmentation concurrently from vehicles, drones/UAMs, and infrastructure cameras; outputs are described as temporally and spatially aligned on a shared simulation timeline with real-time frame export (Wang et al., 17 Oct 2025). AIRHILT records start and end timestamps for radio transmission, ASR finalization, frame exposure, detector completion, ADS-B ingest/output, decision readiness, decision completion, and TTS onset, thereby making latency itself a first-class measurable object (Garib et al., 24 Nov 2025). NavVI distributes roles across channels: the path is visual, directional anticipation is auditory, and near-field obstacle awareness is haptic (Maimuna et al., 20 Jul 2025).

Some systems formalize alignment geometrically. In Virtual Guidance, non-visual instructions such as planner waypoints or language-derived targets are projected into the camera stream as path meshes or waypoint spheres. The projection is written as

$$X_{\mathrm{cam}} = R(X_{\mathrm{world}} - t), \qquad \tilde{x} = K X_{\mathrm{cam}},$$

with optional radial distortion and homography for planar overlays (Yang et al., 2023). This makes guidance legible to a vision policy without requiring it to infer cross-modal correspondences from separate streams.

Other simulators formalize guidance in the feedback channel itself. NavVI maps the vector from current position to destination into a clock-face auditory cue by computing

$$\theta = \operatorname{atan2}(d_x,d_y)\cdot\frac{180}{\pi},$$

normalizing the angle, and converting it to a twelve-sector index; haptic intensity is scaled by

$$H(d) = \log\!\left(1 + \left(1 - \frac{d}{d_{\max}}\right)\right),$$

with left, center, and right motor assignment determined in the robot’s local frame (Maimuna et al., 20 Jul 2025). In Multimodal-GuideNet, synchronization is equally explicit: ultrasound frames at 30 Hz, gaze at 90 Hz, and probe orientation from an IMU are time-aligned and downsampled to 6 Hz, enabling joint prediction of gaze movement and probe motion (Men et al., 2022).

The common issue is that multimodal guidance requires temporal and spatial commensurability. Without a shared timebase or a common projection geometry, a simulator may still be multimodal, but it ceases to support rigorous guidance research.

4. Guidance realization and decision loops

Multimodal guidance simulators typically stop short of prescribing a single control algorithm. TranSimHub is explicit on this point: it supplies synchronized sensing, communication hooks, and standardized APIs, but specific fusion algorithms, planners, rewards, and decision modules are left to the user (Wang et al., 17 Oct 2025). This makes the platform a substrate for end-to-end research on perception, communication, and control rather than a fixed autonomy stack.

Where the guidance policy is specified, it often appears as a coupling between multimodal state estimation and downstream action generation. In METDrive, geometric features from camera and LiDAR are fused with temporal guidance from ego-state time series, including yaw, steering, throttle, and waypoint-vector components. The proposed temporal guidance loss is

$$L_{\text{Temporal}} = \frac{1}{2}\left\| y([F_g,F_t]_{t=0}^{t=n/2}) - \hat{y}([F_g,F_t]_{t=n/2}^{t=n}) \right\|_2^2,$$

which enforces consistency between waypoint predictions derived from temporally proximal and more distant fused features (Guo et al., 2024). In aviation, AIRHILT uses a rule ladder for safety decisions and reserves GPT-OSS-20B for structured natural-language surface form, explicitly separating safety logic from explanation (Garib et al., 24 Nov 2025). In MIRIAM, guidance is mixed-initiative rather than control-authoritative: the system answers free-form questions about mission status while proactively pinning high-priority alerts such as faults, critical battery status, or changes in objectives (Hastie et al., 2018).

These examples illustrate a broader distinction between simulator substrate and decision policy. Some systems, such as TranSimHub and AIRHILT, emphasize interface modularity and controlled observability; others, such as METDrive or Virtual Guidance, embed a more specific guidance mechanism into the learning or control loop. A plausible implication is that multimodal guidance simulation has evolved along two parallel lines: one oriented toward reusable infrastructure, the other toward tightly coupled algorithmic guidance representations.

5. Scenario authoring, counterfactuals, and human-in-the-loop operation

A defining advantage of simulation is controllable scenario generation. TranSimHub includes a causal scene editor supporting daytime, dusk, rain, temporary road closures, traffic collisions, fallen trees, and emergency vehicles with distinct visuals and trajectories. Counterfactual experiments are performed by altering one environmental factor while keeping others fixed, although the paper does not introduce structural causal model notation or do-operator equations (Wang et al., 17 Oct 2025). AIRHILT adopts declarative scenario JSON with randomization seeds and covers runway overlaps, vehicle and wildlife incursions, geometric en route conflicts, airborne emergencies, and uncooperative intruders (Garib et al., 24 Nov 2025). NavVI populates warehouse environments with shelves, workers, forklifts, and pallet robots, using NavMeshObstacle carving and receding-horizon re-planning every $\tau = 2\,\mathrm{s}$ or when specific triggers occur (Maimuna et al., 20 Jul 2025).

Human participation is similarly central. AIRHILT is explicitly pilot- and controller-in-the-loop, with addressed radio communication and advisory TTS delivered on appropriate channels (Garib et al., 24 Nov 2025). NavVI preserves what it describes as “boss mode” control: the operator drives with a simplified DualSense interface while the simulator provides synchronized visual, auditory, and haptic support (Maimuna et al., 20 Jul 2025). XR-CareerAssist uses voice-driven dialogue, gaze and gesture selection, controller-based navigation, and spatial career visualizations inside Meta Quest 3, with a 3D avatar mediating the interaction (Tantaroudas et al., 8 Apr 2026).

This human-in-the-loop orientation distinguishes multimodal guidance simulators from purely offline synthetic data generators. They are not only scene renderers; they are interaction environments in which human interpretation, corrective action, and workload management are part of the modeled system.

6. Evaluation practices, reported performance, and open limitations

Evaluation remains heterogeneous. TranSimHub describes capabilities but does not provide benchmark definitions, baselines, quantitative experiment results, computational requirements, throughput, latency, or multi-agent scaling metrics (Wang et al., 17 Oct 2025). By contrast, AIRHILT reports preliminary runway-overlap results with an average time-to-first-warning of approximately $7.66\,\mathrm{s}$, average ASR latency of approximately $5.88\,\mathrm{s}$, average vision latency of approximately $0.415\,\mathrm{s}$, and average TTS synthesis/delivery of approximately $0.9\,\mathrm{s}$ (Garib et al., 24 Nov 2025). XR-CareerAssist includes a pilot at the University of Exeter with 23 participants and reports 95.6% speech recognition accuracy, 78.3% overall user satisfaction, and 91.3% favorable ratings for responsiveness (Tantaroudas et al., 8 Apr 2026). METDrive reports a driving score of 70%, route completion of 94%, and infraction score of 0.78 on the CARLA Leaderboard Longest6 benchmark (Guo et al., 2024). Multimodal-GuideNet reports that its visual guidance signal achieves an error rate of less than 10 pixels on a $224\times288$ ultrasound image and outperforms single-task learning for both probe motion guidance and gaze movement prediction (Men et al., 2022).

The literature also reveals recurring omissions. TranSimHub does not specify sensor intrinsics or extrinsics, projection equations, communication models for bandwidth, latency, or packet loss, planner formulations, reward definitions, or benchmark metrics (Wang et al., 17 Oct 2025). MIRIAM does not report latency, robustness, scalability, or formal evaluation results (Hastie et al., 2018). NavVI has not yet conducted formal user studies, although it logs collisions, elapsed time to goal, and proximity events for future evaluation (Maimuna et al., 20 Jul 2025). AIRHILT has not yet completed controlled human-factors validation (Garib et al., 24 Nov 2025).

Taken together, these omissions indicate that the field is methodologically ahead of its benchmark standardization. The simulators already support synchronized sensing, multimodal interaction, and scenario control at substantial technical depth, but directly comparable evaluation protocols remain uneven. This suggests that a mature multimodal guidance simulator is likely to require three properties simultaneously: a synchronized multimodal world model, a swappable guidance interface for perception and control modules, and a benchmark layer that defines latency, safety, task completion, and human-factors criteria with equal rigor.