Trajectory-Level Multimodal Safety Auditing

Updated 9 December 2025

Trajectory-level multimodal safety auditing is a systematic evaluation of prediction models that integrate uncertainty quantification, intention prediction, and risk-field metrics for safety assurance.
The approach combines visual attention, structured generative architectures, and metamorphic testing to assess multimodal trajectories under real-world autonomous system scenarios.
Key methodologies include adversarial robustness checks and risk quantification techniques, which help identify potential mode collapse and hazardous failures early.

Trajectory-level multimodal safety auditing refers to the principled evaluation and monitoring of probabilistic, multimodal trajectory prediction models—whether for autonomous vehicles, human crowds, or mixed traffic—so as to assure compliance with safety requirements and preemptively identify potentially hazardous failures. This field integrates intent prediction, statistical uncertainty quantification, adversarial robustness checks, and interaction-mode reasoning to systematically test and intervene upon trajectory predictors in safety-critical real-world scenarios. Recent advances combine visual attention modeling, structured generative architectures, metamorphic testing, risk-field theory, and tailored collapse metrics, yielding a multifaceted safety-auditing toolkit that is adaptive to both agent-based and environment-driven risks.

1. Multimodal Prediction Architectures for Safety Auditing

At the heart of trajectory-level safety auditing is the use of multimodal prediction models. These models output a bounded set of candidate future paths for each agent, reflecting inherent uncertainty and multiple plausible intentions.

Zhang et al. (Zhang et al., 2022) introduce a framework comprising two primary modules:

Driving Intention (DI): Encodes latent intention using a sequence of vehicle states $D(t)$ , visual area-of-interest (AOI) gaze distributions $E(t)$ , and environmental-context features $C(t)$ . A single-layer LSTM compresses these features into a hidden embedding, processed by an FC-softmax head that yields intention mode probabilities.
Multimodal Trajectory (MT): Parallel LSTM-encoder/decoder structure, outputting $N$ discrete trajectory branches (left, right, straight), each assigned probabilistic weights from the DI module. This fusion allows distinction between multiple plausible futures.
Trajectory Filter (TF): Selects the branch with highest intention probability for risk assessment, though all modes may be considered for complete auditing.

Other architectures—SGANv2 (Kothari et al., 2022) and QCNet (Jiang et al., 19 Oct 2024)—employ adversarial generative schemes, spatio-temporal attention, and transformer-based representation learning to capture agent interactions and social norms, with test-time acceptance criteria enforced by discriminators or risk field calculations.

2. Input Feature Construction and Intention Predictors

Safety auditing relies crucially on accurate early recognition of agent intentions, particularly in cluttered or ambiguous environments.

Visual AOI Extraction (Zhang et al., 2022): High-frequency eye-gaze data (90Hz) are fitted per video frame with a 2D Gaussian, creating a compact five-dimensional representation $[\mu_x,\,\mu_y,\,\sigma_x,\,\sigma_y,\,\rho]$ . Temporal drift in $\mu_x$ is empirically shown to precede actual steering by $0.6$–$2.1$ s, with average lead times of $0.925$ s. This anticipation allows the auditor to filter trajectories by likely forthcoming maneuvers.
Context and Environmental Features: Shallow CNN (ResNet) encodes resized front-view images to stabilize AOI inputs and suppress noise.
Learning and Pretraining: The DI module is pretrained with cross-entropy loss on turn labels (straight, left, right). Precision/recall/F1 scores all exceed 97% for intention prediction.

Intent predictors may also use map features, semantic segmentation, and historic trajectories, as in metamorphic testing pipelines (Spieker et al., 1 Sep 2025), to ensure multimodal outputs are both probable and contextually consistent given environmental cues.

3. Risk Quantification: Probabilistic, Field-Theoretic, and Interaction Metrics

Auditing frameworks operationalize safety by estimating the likelihood and severity of hazardous events for every candidate trajectory—or mixture thereof.

Covariance-based Risk Assessment (Zhang et al., 2022): The error boundary around each trajectory is estimated by a learned covariance $\Sigma(j)$ at each future step. Particle sampling is performed within the 95%-confidence ellipse, perturbing the ego footprint, and testing for collisions with obstacles. Conflict probability $P_{\rm conflict}$ is computed as the mean collision rate over samples. Threshold persistence over $K$ cycles suppresses spurious warnings, and ROC-style threshold calibration mitigates false alarms.
Enhanced Driving Risk Field (EDRF) (Jiang et al., 19 Oct 2024): Defines a spatial risk field $EDRF(x,y)$ for each agent via multimodal Gaussian distributions along predicted trajectories, modulated by a virtual mass term and mode probabilities. Pairwise interaction risk $IR_{ij}(x,y) = EDRF_i(x,y) EDRF_j(x,y)$ , with scalar risk $F_{ij} = \max_{x,y} IR_{ij}(x,y)$ , quantifies the likelihood of agent-agent conflicts for each mode.
Applications: Risk monitoring, ego-vehicle analysis (Laplace cross-section for lower ego uncertainty), and trajectory planning (safety cost integrated in search over candidate paths) are supported by field-theoretic calculations.

4. Metamorphic Testing for Auditing Multimodal Prediction Robustness

Direct safety auditing via future ground-truth is often infeasible, motivating oracle-less methodologies such as metamorphic testing (Spieker et al., 1 Sep 2025). Five metamorphic relations (MRs) are defined to probe model robustness:

Mirroring (MR1): Input trajectory and map are mirrored; outputs should transform correspondingly.
Rotation (MR2): Inputs rotated by 90°, 180°, or 270°; outputs must rotate identically.
Rescaling (MR3): Uniform scaling; outputs scale proportionally.
Semantic-Class Change (MR4): Map pixels swapped between classes; output probability shifts must increase in more walkable regions and decrease in less.
Obstacle Insertion (MR5): New obstacles placed; predictions must reroute, avoiding direct collisions.

Violation criteria employing Wasserstein or Hellinger distances between sampled trajectory distributions define statistical failure thresholds, calibrated empirically per source case. Case studies show most models exhibit high sensitivity to landscape orientation and semantic class changes, with robust rerouting observed for obstacle MRs.

Recommendations include embedding MR-based tools in CI/CD pipelines, scenario generation from violation seeds, and dashboarding robustness drift over time.

5. Generative Architectures and On-the-Fly Auditing Strategies

SGANv2 Pipeline (Kothari et al., 2022):
- Spatio-temporal interaction module (SIM) integrates agent velocities and neighbor positions for collision-aware feature learning.
- Generator (LSTM-based) proposes $K$ multimodal futures per scenario.
- Transformer-based discriminator scores joint-observed/future sequences for social compliance.
- Collaborative Sampling: Unsafe (colliding or low-discriminator-score) trajectories are iteratively refined via discriminator gradients until safety threshold $D \ge \tau$ is satisfied, subject to a maximum iteration cap.
- No explicit collision loss is imposed; social norms are learned by the discriminator.

Evaluation metrics include top- $K$ ADE/FDE, collision rate, variety loss, and distance-to-goal. Test-time discriminator gating realizes a practical trajectory-level multimodal safety audit.

6. Auditing Mode Collapse and Interaction Consistency in Joint Prediction

Accurate safety auditing must detect whether the multimodal predictor indeed represents all plausible interaction modes, especially in safety-critical multi-agent scenarios.

Mode Collapse Metrics (Hugenholtz et al., 29 Jun 2025):
- Mode correctness: fraction for which the most-likely prediction matches ground-truth interaction mode.
- Mode coverage: whether any of $K$ outputs covers the ground-truth mode.
- Mode collapse: the rate at which feasible modes (as determined by future roll-outs) are absent from all outputs.
- Time-to-correct and time-to-cover metrics reflect how early the model commits to correct interaction modes before they become inevitable.

Experiments on nuScenes report mode correctness $\sim$ 74–86% (best case), with mode collapse rates as high as 70–100% in classical predictors. Even expert systems ("oracle") fail to perfectly cover all modes given finite $K$ .

Traditional distance metrics (ADE, FDE) do not expose collapse failures, highlighting the necessity of interaction-mode auditing for robust, safety-aligned deployment. The framework enables focused retraining and monitoring on problematic scenarios.

7. Implementation, Calibration, and Practical Deployment

Trajectory-level multimodal safety auditing is implemented in real-time pipelines via:

Regular acquisition of historical trajectories, AOI signals, and context features; updating intention probabilities and candidate paths per cycle (Zhang et al., 2022).
Calibration of conflict probability thresholds and uncertainty covariances to balance detection latency and false-alarm suppression.
Integration of metamorphic tests in automated validation loops, with dashboarding of violation rates (Spieker et al., 1 Sep 2025).
Post-generation acceptance and refinement using discriminator and risk-field audits (Kothari et al., 2022, Jiang et al., 19 Oct 2024).
Use of mode-collapse monitoring for critical interaction scenario selection, focused regression testing, and continual improvement (Hugenholtz et al., 29 Jun 2025).

Persistent, context-aware auditing mechanisms—temporal smoothing, environmental feature fusion, and geometry constraints—reduce noise and false positives. Extensions enable auditing of lane changes, pedestrian interactions, roundabouts, and merges through modular expansion of intention and interaction vocabularies.

A plausible implication is that combined methodologies spanning probabilistic prediction, adversarial discriminators, risk fields, metamorphic relations, and interaction-mode metrics can yield highly robust safety auditing practice suitable for online autonomous system deployment, provided all calibration and coverage limitations are rigorously handled.