A Stitch in Time Saves Nine: Preserving Policy Compatibility Under Perception Updates in End-to-End Autonomous Driving

Abstract: End-to-end autonomous driving systems tightly couple perception and decision-making through latent representations. Consequently, updates to perception models can alter these representations and degrade the performance of downstream policies that remain fixed. Existing solutions typically rely on policy retraining or architectural decoupling, both of which incur substantial computation and validation costs. In this paper, we formulate the model stitching problem for end-to-end autonomous driving and test the hypothesis that policy compatibility can be preserved through lightweight latent-space alignment. We study low-complexity model stitching methods, including linear and convolutional stitchers, for restoring compatibility between updated perception modules and frozen downstream policy modules. Experiments demonstrate that stitching effectively preserves downstream driving behavior under diverse perception updates, including changes in random initialization, sensor configuration, and training domain. In the most challenging cross-domain setting from nuScenes to CARLA, convolutional stitching retains over 91\% of the no-shift driving score while reducing adaptation time from \SI{22.18}{h} to \SI{0.91}{h}. These results suggest that model stitching provides an effective and computationally efficient alternative to retraining or fine-tuning for maintaining end-to-end autonomous driving systems. The model will be open-sourced upon paper acceptance at https://github.com/SCP-CN-001/model-stitching to support further research and development in autonomous driving.

• The paper introduces a lightweight latent-space transformation to stitch new perception features with existing policy networks, eliminating costly retraining.
• It demonstrates that linear stitching nearly recovers performance under minor shifts, while convolutional stitching is effective for nontrivial sensor, domain, or architecture transitions.
• Empirical results reveal significant computational and interaction cost savings, maintaining high driving scores across synthetic and real-world datasets.

Policy Compatibility Under Perception Updates in End-to-End Autonomous Driving

Problem Formulation and Theoretical Foundation

The paper introduces and formalizes the model stitching problem within end-to-end autonomous driving systems, where the perception and policy modules are tightly coupled via latent feature representations. As perception models are frequently updated (e.g., due to sensor changes, retraining, or expanded datasets), downstream policies—often safety-critical and costly to retrain—may become incompatible with the altered latent spaces. The paper posits the Policy-Compatible Representation Stitchability Hypothesis, stipulating that such incompatibility can be efficiently corrected by learning a lightweight transformation in latent space, eliminating the need for downstream policy retraining.

This hypothesis leverages the empirical observation that neural representations trained under similar supervision signals are often linearly or locally homeomorphic, particularly in deeper layers. The paper grounds its approach in three assumptions:

1. Task-Relevant Manifold: Task-critical environmental attributes reside on a low-dimensional manifold, justifying focus on latent supervision signals rather than globally reconstructing the entire scene.
2. Task-Relevant Supervision: Supervisory signals (e.g., BEV segmentation, object detection) provide the salient, invariant driving cues required for vehicle control, and are preserved in the latent representation.
3. Supervisory Information Preservation: The learned latent codes maintain sufficient information for downstream policies through a decoder probe.

Collectively, these support the existence of a low-complexity, trainable transformation (linear or shallow convolutional) that can "stitch" the new perception latent space into compatibility with the existing policy.

Methodological Contributions

The paper implements two families of lightweight transformations for model stitching:

• Linear Stitching: Affine mapping trained via least-squares alignment on paired latent representations from the old and new perception modules.
• Convolutional Stitching: 1x1 convolutional mapping (possibly with batch normalization), optimized via task loss and (optionally) a feature alignment objective, maintaining spatial structure for convolutional representations.

Experiments are performed by fixing the entire downstream stack (transformer-based world model, SAC policy network) and only updating the stitcher, thereby maximizing deployment realism where policy modifications are often infeasible.

The evaluation protocol systematically explores configurations with increasing representation shift:

• Random reinitialization (sanity check),
• Autoencoder type changes (VAE to AE),
• Sensor modality and configuration variations,
• Cross-domain transfer (nuScenes to CARLA).

Both synthetic (CARLA) and real-world (nuScenes) datasets are employed.

Empirical Results

Driving Performance

• Under minor latent shifts (e.g., reinitialization, VAE/AE swap), linear stitching nearly matches retraining, achieving >98% route completion and >96% driving score in CARLA.
• For nontrivial representation shifts (sensor configuration, number of cameras/LiDAR, or domain shifts), performance degrades significantly (e.g., drop to RC=34.34, DS=15.31 under sensor shift) if the latent space is directly connected.
• Convolutional stitching restores most driving performance lost due to large representation mismatch (sensor or domain shift). For cross-domain (nuScenes-to-CARLA), convolutional stitching elevates the driving score from 34.76 to 89.88, with route completion from 45.32 to 92.23—retaining over 91% of the no-shift capacity with orders-of-magnitude less computation.
• Linear stitching cannot recover performance in the presence of high-dimensional, nonlinear domain or sensor shifts.

Efficiency Gains

• Parameter Updates: Stitching only trains <0.001% of the system parameters. No downstream policy parameters are modified.
• Runtime: Adaptation time drops from 22.18h (retraining) to as low as 0.91h (convolutional stitching), with linear stitching requiring only ≈0.02h.
• Memory Consumption: Both host and GPU memory usage are significantly suppressed compared to end-to-end retraining.
• Interaction Cost: Stitching requires zero additional simulation or real-world interaction steps, relying solely on offline collected paired representations, versus 50,000–70,000 steps for retraining/fine-tuning.

Representation Alignment & Perceptual Utility

• For semantic segmentation, both linear and convolutional stitching restore mIoU values close to the target latent space when the representation shift is moderate; convolutional stitching is required under more severe sensor or domain changes.
• For object detection, similar restoration patterns are observed in mAP under cross-domain stitching.
• t-SNE and CKA analyses confirm that deeper latent layers exhibit consistent clustering and high similarity when supervised on the same label space, corroborating theoretical assumptions.

Positioning Within Prior Work

This work extends the concept of model stitching, previously explored mainly in vision and language domains, to real-world, safety-critical autonomous driving tasks. Unlike policy-level adaptation (fine-tuning/reinforcement learning on the policy) or interface decoupling/adapters (which assume access to full retraining), this approach enforces strict downstream-policy immutability. It provides strong, empirical evidence for the sufficiency of low-dimensional, information-preserving transformations within the compact, task-specific latent representations standard in modern perception modules.

Its findings also clarify that, although prior research has measured representational similarity, practical utility in policy transfer and maintenance under real-world perturbations requires explicit feature-space alignment.

Implications and Future Developments

This paper demonstrates that latent-space alignment via lightweight model stitching is an effective, scalable mechanism for maintaining end-to-end autonomous driving system functionality after perception updates—with near-zero downtime and computational cost. Practically, this enables routine perception upgrades, sensor integration, or dataset expansion without exhaustive downstream retraining and safety validation, a substantial advantage for deployed autonomous fleets.

Theoretically, the results strengthen the low-dimensional manifold and retained-supervision hypotheses in reinforcement learning and modular deep architectures. They point toward a paradigm where latent interface compatibility becomes a primary engineering and research target in autonomous systems, potentially catalyzing new directions: continuous online stitching, stitching across large architecture/dataset differences, and robustness analysis under adversarial distributional shift.

As the open-source release of the framework is anticipated, future work can extend the approach to multi-agent driving, hierarchical policies, and more diverse perception modalities, and investigate fundamental limits of latent-mapping expressivity vis-à-vis task complexity.

Conclusion

This paper rigorously investigates the consequences of asynchronous perception updates for downstream policy compatibility in end-to-end autonomous driving. It provides compelling evidence for the feasibility of preserving policy functionality via lightweight latent-space model stitching, with negligible computational and interaction costs, across diverse and challenging representation shifts. These insights have significant theoretical and practical ramifications for the modular evolution and efficient maintenance of real-world autonomous driving stacks (2606.21509).