VLM-CASE: Vision-Language Model Enabled Context-Adaptive Safety Envelopes for Anticipatory Safe Autonomous Driving

Abstract: Adverse driving conditions, such as bad weather, remain a principal barrier to autonomous driving because they degrade two things at once: what the vehicle can perceive and what it can physically do. Human drivers cope by anticipation, reasoning about the scene and re-budgeting speed, following distance, and steering before grip or sight is lost, whereas current autonomous driving systems at best react after the fact. This paper proposes VLM-CASE, a framework that gives an autonomous vehicle this anticipatory capacity while keeping its motion bounded by a formal safety model at all times. A vision-LLM (VLM), fine-tuned with low-rank adaptation (LoRA), reasons about the scene from the front-camera image and reports the road surface and visibility conditions. This output parametrizes a context-adaptive safety envelope (CASE), derived from physical limits and the guarantees of responsibility-sensitive safety, that couples braking and steering through a shared friction budget. A model predictive controller then drives freely within the envelope, while the VLM runs asynchronously so it never blocks the real-time control loop. We validate the framework in closed-loop CARLA simulation on tasks that demand both lateral and longitudinal control, across a range of weather, road-surface, and lighting conditions. The resulting controller, VLM-CASE-MPC, completes all trials, outperforming a conventional MPC baseline and a state-of-the-art VLM-integrated controller. Ablations confirm that the gains come from context adaptation, with the friction and visibility adaptations proving complementary. Furthermore, the framework is controller-agnostic and pairs with almost any low-level controller, offering a promising direction for safe autonomous driving. The dataset and supplementary materials for VLM-CASE are available at https://github.com/ytj254/VLM-CASE.

• The paper introduces a context-adaptive safety envelope that integrates VLM-based semantic inference with MPC to anticipate and mitigate adverse driving scenarios.
• The system leverages a LoRA-adapted Qwen3-VL model to extract semantic context with over 98% accuracy, mapping it to critical safety parameters.
• Experimental results demonstrate that VLM-CASE outperforms baselines by effectively coupling friction and visibility adaptations in 198 varied simulation runs.

VLM-CASE: Vision-LLM Enabled Context-Adaptive Safety Envelopes for Anticipatory Safe Autonomous Driving

Introduction and Motivation

Adverse driving conditions, particularly those involving degraded visibility and poor road surfaces, remain one of the core challenges in deploying robust autonomous driving systems. Human drivers handle such conditions through anticipatory reasoning, adapting speed, following distance, and steering in advance according to road and weather cues. In contrast, typical autonomous driving logic remains fundamentally reactive, often relying on pre-set worst-case constraints or on friction estimates that come too late for true anticipation.

VLM-CASE (Vision-LLM-enabled Context-Adaptive Safety Envelope) directly addresses these deficiencies by enabling an anticipatory safety guarantee: the system uses an open-vocabulary, fine-tuned vision-LLM (VLM) to interpret the driving context from camera input, and subsequently parametrizes a context-adaptive formal safety envelope that integrates the effects of reduced friction and visibility. This envelope, grounded in extensions of Responsibility-Sensitive Safety (RSS) theory and physical vehicle dynamics, constrains a model predictive controller (MPC) to guarantee safety while maximizing operational efficiency. VLM reasoning is executed asynchronously, ensuring that real-time control is not bottlenecked by high-latency semantic inference.

Figure 1: Overview of VLM-CASE, an anticipatory, safety-guaranteed autonomous driving framework.

System Architecture

VLM Context Inference Pipeline

The uppermost layer of VLM-CASE is a VLM, specifically a LoRA-adapted Qwen3-VL model, which is prompted to extract a discrete semantic context from each front-camera image. This context categorizes the road surface (e.g., dry/wet/snow), weather (clear/rain/fog), time of day, and illumination assistance. Fine-tuning via LoRA ensures >98% accuracy across all fields in the CARLA simulation domain, with inference latency compatible with slow control loops (∼1.2 seconds).

Figure 2: Overview of the VLM context inference pipeline.

The VLM output, updated asynchronously (every 2 s), is mapped via a calibrated table and composite rule to two key numerical quantities: (1) tire-road friction coefficient $\mu$ (surface-driven constraint), and (2) “forward observability” $o_f$ (aggregation over weather, time, and illumination).

Context-Adaptive Safety Envelope

The core methodological innovation is the context-adaptive safety envelope (CASE), integrated as state-dependent constraints into an MPC control loop. The envelope strictly couples lateral and longitudinal safety via physical friction and visibility bounds, dynamically parametrized by the inferred semantic context:

• Friction-Limited Coupling: Braking and steering are coupled through a shared friction budget, enforced via the classic friction circle constraint, which ensures any admissible acceleration vector (braking and steering) lies within $\mu g$.
• Longitudinal Safety with Visibility Awareness: Following distance is augmented not just by friction limits, but also by an observability margin that increases with degraded perception—quantified as a function of $o_f$—adapting minimum safe gap in low visibility.
• Lateral RSS-Based Constraints: Lateral (lane boundary) constraints are dynamically recalculated based on current velocity, heading, and friction, ensuring that lane keeping is adjusted for degraded physical capability.

The envelope is recalculated rapidly (20 Hz), always reflecting the latest semantic inference and guaranteeing that all control actions are formally safe with respect to current scene understanding.

Asynchronous Control Integration

A critical aspect of the framework is the strict separation between the high-latency VLM and the low-latency control pipeline. The semantic context is inferred at a slow rate, but its parameters directly shape the constraints within which the high-frequency MPC operates. This design ensures that vision-LLM inference never blocks or delays the safety-critical control process.

Experimental Evaluation

Experiments were conducted in closed-loop CARLA simulations under a comprehensive range of operational conditions—explicitly varying road surface, weather, time-of-day, and illumination to test both longitudinal and lateral safety in urban and highway driving tasks. Three classes of scenarios were considered: no-lead driving (pure lateral control), constant-lead following (pure longitudinal adaption), and integrated lead-braking emergencies (jointly testing lateral and longitudinal coupling). Each scenario was crossed with parameter sweeps over speed and map topology, covering 198 repeated evaluation runs.

Figure 3: Front-camera frames for the evaluated conditions (road surface, weather, time of day, illumination).

VLM-CASE-MPC was benchmarked against three strong baselines:

• Base MPC (no context adaptation)
• VLM-MPC ([Long et al.] [longVLMMPCModelPredictive2026]; VLM used only to adjust cost weights and target parameters)
• Fixed-Envelope MPC (CASE envelope with fixed, design-time parameters)

Results

Integrated Lead-Braking (Adversarial) Scenarios

VLM-CASE-MPC was the only controller to achieve 100% success rate, completing all 54 runs across dry, adverse (wet night), and severely adverse (snow) conditions. The Base MPC failed catastrophically on low friction (5.6% on snow), with frequent lane departures and unsafe stops (minimum boundary clearance often negative and minimum TTC of 0.27 s). Both VLM-MPC and Fixed-Envelope MPC failed to address friction-visibility coupling: without explicit friction adaptation, both controllers experienced frequent loss-of-control in adverse conditions (≤50% on snow).

Figure 4: Run outcomes in lead-braking experiments by condition and controller.

Figure 5: Trajectories under the snow condition (C3) in lead-braking experiments.

No-Lead (Lateral) and Constant-Lead (Longitudinal) Experiments

VLM-CASE-MPC demonstrated robust lane-keeping across degraded surfaces, being the only approach to complete all no-lead snow runs (mean minimum boundary clearance $0.18\,m$; failed baselines frequently departed the lane). Mean lateral error and minimum lateral clearance metrics unambiguously favored VLM-CASE when accounting for distance-normalized performance.

Figure 6: Ego speed profiles in no-lead experiments on the snow surface.

In constant-lead-following experiments, VLM-CASE-MPC dynamically widened the car-following gap as forward observability worsened, reflecting human-like caution (mean gap increasing from 36 m to 66 m as visibility fell). Baseline controllers kept fixed or only weakly context-sensitive gaps, resulting in potential safety violations under deep perceptual degradation.

Figure 7: Following gap versus route completion under varying visibility.

Ablation Experiments

Ablation confirmed that friction and visibility adaptation are complementary and both necessary: friction adaptation alone sufficed for friction-limited (snow) cases, while visibility adaptation alone sufficed for perception-limited (wet night) cases. Only VLM-CASE, simultaneously adapting both, achieved full task success across all conditions.

Implications and Theoretical Considerations

VLM-CASE demonstrates that integrating structured scene understanding from a VLM into the formal safety envelope—not merely control heuristics or parameter tuning—yields quantifiable gains in safety and robustness under adverse and rapidly changing conditions. The approach not only guarantees provable collision avoidance and lane-keeping under RSS extensions, but also avoids the over-conservatism or latent safety violations induced by static design-time safety parameter selection.

A key theoretical implication is the demonstration of equivalence between human-style anticipatory adaptation and formal safety preservation, achieved through the tight coupling of semantic perception (via VLMs) and constraint-driven vehicle control. The explicit coupling of longitudinal and lateral constraints via a shared, physically justified friction budget represents an advance over decoupled formulations and offers a unified policy for emergency avoidance and robust regular driving.

Practically, the controller-agnostic formulation permits application beyond MPC, potentially to rule-based, learning-based, or hybrid control stacks, as long as the control policy remains within evolving safety envelopes.

Future Directions

Potential future developments include:

• Extending the semantic context space to continuous-valued scene descriptors and further categories (e.g., road geometry, occlusions).
• Applying the framework on real-world autonomous vehicle platforms, considering the domain gap between simulation and reality, particularly in road appearance and friction ground truth.
• Hybridizing VLM-CASE with end-to-end vision-language-action pipelines for full-stack, safety-guaranteed autonomy in diverse ODDs.
• Integration with runtime formal monitors and risk-aware planning frameworks.

Conclusion

VLM-CASE provides an anticipatory safety framework for autonomous driving under adverse conditions, uniquely leveraging VLMs not only for scene understanding but for real-time, context-driven parametric adaptation of provable safety envelopes. Empirical results in closed-loop CARLA simulation demonstrate strong and consistent outperformance of both conventional and VLM-aware controller baselines, especially under conditions that jointly degrade environment perception and vehicle maneuverability. The system’s modular and controller-agnostic design opens promising directions for future robust autonomous driving research.