LSRE: Latent Semantic Rule Encoding for Real-Time Semantic Risk Detection in Autonomous Driving

Abstract: Real-world autonomous driving must adhere to complex human social rules that extend beyond legally codified traffic regulations. Many of these semantic constraints, such as yielding to emergency vehicles, complying with traffic officers' gestures, or stopping for school buses, are intuitive for humans yet difficult to encode explicitly. Although large vision-LLMs (VLMs) can interpret such semantics, their inference cost makes them impractical for real-time deployment.This work proposes LSRE, a Latent Semantic Rule Encoding framework that converts sparsely sampled VLM judgments into decision boundaries within the latent space of a recurrent world model. By encoding language-defined safety semantics into a lightweight latent classifier, LSRE enables real-time semantic risk assessment at 10 Hz without per-frame VLM queries. Experiments on six semantic-failure scenarios in CARLA demonstrate that LSRE attains semantic risk detection accuracy comparable to a large VLM baseline, while providing substantially earlier hazard anticipation and maintaining low computational latency. LSRE further generalizes to rarely seen semantic-similar test cases, indicating that language-guided latent classification offers an effective and deployable mechanism for semantic safety monitoring in autonomous driving.

• The paper introduces LSRE, a dual-stage framework combining offline VLM supervision with a lightweight latent classifier for rapid semantic risk detection in AVs.
• It achieves high recall rates (up to 98.64%) and sub-10ms inference, significantly outperforming traditional VLM-only systems in hazardous scenarios.
• LSRE anticipates hazards with lead times over 2.5 seconds and uses temporal hysteresis filtering to reduce false alarms, ensuring robust real-time safety.

LSRE: Latent Semantic Rule Encoding for Real-Time Semantic Risk Detection in Autonomous Driving

Motivation and Problem Context

Autonomous driving research has established robust mechanisms for geometric safety but continues to lag in real-time interpretations of complex, human-defined social semantics. Human drivers intuitively react to ambiguous rules—yielding to emergency vehicles, following traffic officer instructions overriding traffic signals, or responding to the presence of school buses—none of which are fully encoded within traditional rule-based or geometric approaches. Large vision-LLMs (VLMs) exhibit strong capabilities for high-level semantic understanding but fail to meet real-time execution constraints, with per-frame latencies precluding online deployment in driving stacks. This work introduces LSRE (“Latent Semantic Rule Encoding”), aiming to distill the semantic understanding of VLMs into a deployable, low-latency safety mechanism for autonomous vehicles.

LSRE Framework Overview

LSRE proposes a two-stage framework. The first stage leverages pretrained VLMs in offline, sparse supervision to extract semantic-risk labels on select key frames. During system execution, the second stage operates entirely in the latent space of a recurrent world model, where a lightweight classifier, trained under VLM-supplied supervision, performs rapid semantic risk assessment. This classifier not only produces instantaneous risk judgments but also anticipates future hazards through latent rollouts and hysteresis-based filtering for temporal stability.

Figure 1: LSRE's pipeline combines offline VLM-generated supervision with a lightweight, temporally-dynamic latent classifier for fast, continuous semantic risk estimation inside the world model latent space.

The modular design achieves runtime, per-frame semantic risk estimation at rates above 10 Hz without requiring frame-level VLM inference, thus satisfying the latency requirements of real-time AV decision stacks.

Technical Approach

VLM-Guided Semantic Supervision

Each 10-frame segment in a driving sequence is processed by a VLM using a fixed prompt template as pseudo-annotation, yielding semantic risk soft labels for key frames. Accumulated ego-motion and previously inferred semantic states are incorporated to retain temporal consistency and semantic context across sparse annotations. This construction maintains supervision density sufficient for training while dramatically reducing VLM computational load.

Latent Space Risk Scoring and World Model Integration

LSRE integrates these sparse VLM-generated annotations into a recurrent state-space model (RSSM), adopting a margin-based classifier in the learned latent space. The semantic risk classifier, $g_\mu(z_t)$, is trained to maximize separation between safe and unsafe latent encodings, using a geometrically interpretable margin objective:

$$\mathcal{L}_\mu = \frac{1}{N} \sum_{z_i \in D} \mathrm{ReLU}\left( \delta - y_i\, g_\mu(z_i) \right)$$

where $y_i$ denotes semantic safety, and $\delta$ is a margin hyperparameter.

To address the need for early hazard anticipation, a latent value function is estimated through future state imagination: the RSSM performs multi-step rollouts under the current policy, generating prospective latent sequences and computing a discounted cumulative margin score over a horizon of $K=50$:

$$V_{\mathrm{latent}}(z_t) = \sum_{k=0}^{K} \gamma^k\, g_\mu(z_{t+k})$$

Hysteresis filtering with dual thresholds ($\theta_{\mathrm{low}}, \theta_{\mathrm{high}}$) on the margin output further constrains undesired oscillations, achieving temporal filtering and suppressing false positives due to classifier uncertainty near decision boundaries.

Experimental Design and Benchmark

A dedicated CARLA-based benchmark is constructed covering three critical semantic-failure categories: emergency-vehicle yielding, construction-zone lane adherence, and school-bus stoppage scenarios. Each category features in-distribution and semantically-similar, visually-distinct few-shot variants, for a total of six evaluation scenarios. Baselines include a direct VLM-only system (cloud-invoked, high-latency), and a trivial Always-Safe strategy.

Figure 2: Illustration of the three semantic-failure categories, each with both in-distribution and visually diverse few-shot instances, capturing challenging safety-critical semantics in AV operation.

Empirical Results

Frame-wise Semantic Risk Detection

LSRE attains frame-level accuracy and recall comparable to a VLM-only system, achieving 89.51% accuracy and 94.44% recall in-distribution, and 81.25% accuracy and 87.37% recall under few-shot generalization. Notably, LSRE recall in the in-distribution setting sometimes exceeds that of VLM-only (98.64% vs. 84.07% in emergency-vehicle yielding), due to conservative design choices. False alarm rates remain higher than VLM-only, though post-hoc hysteresis filtering significantly reduces spurious triggers in normal driving.

Event-Level Anticipation

On annotated semantic-hazard onset events, both LSRE and VLM-only attain near-perfect event recall. However, LSRE achieves order-of-magnitude earlier anticipation: average lead times of 2515 ms overall, e.g., 3273 ms in construction zones versus 248 ms for VLM-only.

Real-Time and Stability Performance

LSRE delivers a median inference latency of 9.44 ms (95th percentile: 11.91 ms), a 300-fold speedup against VLM-only (median: 2917 ms). In routine non-hazardous driving, LSRE achieves a false-alarm rate of 0.98% due to the introduced temporal smoothing, confirming its practical deployability for continuous in-vehicle safety monitoring.

Qualitative Case Analysis

Figure 3: Example of LSRE's risk signal in a school-bus stopping scenario, demonstrating prediction that reliably anticipates the manual "ground truth" semantic violation several seconds prior to event onset.

Case studies reveal that the LSRE semantic value function consistently drops below the safety threshold before ground-truth hazard onsets, across school-bus stopping, emergency-vehicle, and construction-zone classes, supplying multi-second warning horizons to the AV system.

Theoretical and Practical Implications

LSRE establishes that VLM-level semantic interpretation—previously inaccessible to real-time AV stacks—can be effectively emulated by a lightweight, latent classifier trained using sparse VLM supervision. This distillation not only preserves high-level safety semantics but also introduces robust temporal anticipation and sub-10ms inference, closing the gap between LLM semantic awareness and practical AV deployment requirements.

The approach demonstrates that semantic rules, otherwise hard-coded in planning policies or incomplete in geometric constraints, are efficiently operationalized in latent dynamic spaces when properly supervised. This suggests a general paradigm for imbuing AV world models with arbitrary, language-defined "soft rule" encodings and sets the stage for further fusion of language-level intent understanding and real-time control.

Future Directions

The paper identifies several promising avenues:

• Coupling semantic risk prediction with closed-loop planning: Future extensions may leverage the learned risk signals to trigger automated fallback policies, risk-adaptive planning, or formal safe controller overrides in the AV stack.
• Scaling to real-world driving data: Demonstrating generalization and stability of LSRE beyond simulators, on diverse, naturally-distributed data, is critical for widespread adoption.
• Integration with multimodal, context-rich policy learning: Tighter coupling of the semantic classifier with behavior planning and state estimation pipelines, including cross-modal reasoning over traffic context.

Conclusion

LSRE delivers a technically rigorous, empirically validated framework for real-time, language-informed semantic risk detection in autonomous driving. By transferring VLM-derived semantic judgments into a world-model latent space, LSRE achieves predictive risk signaling, high recall, and sub-10ms inference suitable for embedded deployment. The framework bridges the gap between theoretical semantic understanding and actionable, robust real-time AV safety mechanisms.