---
title: 'TransFuser v6: End-to-End Driving Policy'
url: https://www.emergentmind.com/topics/transfuser-v6-tfv6
type: topic
---

# TransFuser v6: End-to-End Driving Policy

TransFuser v6 (TFv6) is an advanced end-to-end driving policy that fuses multi-modal sensory input using a transformer-based birds-eye-view (BEV) architecture and is trained with a rigorously student-aligned expert, the LEAD policy. TFv6 achieves state-of-the-art closed-loop performance on multiple CARLA benchmarks and can be adapted for real-world driving tasks using latent representations and perception supervision. The LEAD system, with its focus on minimizing learner–expert asymmetries in visibility, uncertainty, and intent modeling, represents a key methodology shift in large-scale imitation learning for autonomous driving [2512.20563].

## 1. Architectural Innovations and Sensor Fusion

TFv6 retains and extends the core design principles of the TransFuser architecture, embedding a transformer-based fusion module to combine inputs from multiple sensors into a dense BEV token space for planning and control. 

### Sensor Fusion Pipeline

- **Cameras:** Six front-and-side RGB cameras provide image frames, each processed through a convolutional backbone (ResNet-34 or RegNetY-032). View-frustum projection lifts each camera’s feature map into BEV space, resulting in $N = H \times W$ BEV tokens of dimensionality $D$ (e.g., $D=256$).
- **LiDAR:** Point clouds are voxelized and fed through a sparse 3D encoder, which also projects to the shared BEV token grid.
- **Radar (TFv6 addition):** Four automotive radars yield up to 75 detections per frame. Detections (range, azimuth, radial velocity) are embedded with a lightweight learned encoder, then injected as object-level tokens directly into the planner’s cross-attention keys/values.
- **BEV Self-Attention:** The joint tokens (camera, LiDAR, radar) are concatenated and passed through $L$ layers of transformer self-attention, generating a unified BEV representation $T = \{ t_1, \dots, t_N \}$.

### Decoder and Control Heads

- **Route Queries:** $M$ learned query vectors $Q \in \mathbb{R}^{M \times D}$ attend across BEV tokens via cross-attention in $K$ transformer decoder layers.
- **Outputs:** Each query produces a predicted path spline (or steering offset) through an MLP, and a separate “speed query” outputs the speed command.
- **Training Objective:** The policy is trained end-to-end by minimizing a mean-squared imitation loss between student predictions and LEAD expert actions:
  $$
  L_{\text{im}} = \mathbb{E}_{o,s}[ \| \pi_\theta(o) - a_{\text{expert}}(s) \|^2 ]
  $$
- **Key Change from TFv5:** Removal of the late GRU bottleneck (previously, route points passed only via a small GRU with limited information); TFv6 injects navigation tokens directly into the transformer stack (see Section 3).

## 2. Learner–Expert Asymmetry Interventions

TFv6 is trained with a new “student-centric” expert policy (LEAD) explicitly designed to minimize asymmetries between the privileged expert and the sensor-bound student during imitation learning.

### 2.1 Visibility Asymmetry

- In prior PDM-Lite experts, forecasts considered all actors’ ground-truth boxes, regardless of camera occlusion or field of view.
- **LEAD restriction:** Only actors with bounding boxes intersecting the student’s camera frustum $F$ are used:
  $$
  A_{\text{vis}} = \{ a : \text{box}_{2D}(a) \cap F \neq \emptyset \}
  $$
- Traffic lights and speed signs are masked to only those visible in the current frustum under current weather/time conditions.

### 2.2 Uncertainty Asymmetry

- Collision prediction in PDM-Lite used perfect kinematic data; LEAD halts if hazards (based only on student-visible context) invade a safety margin $d_{\text{safe}}$, independent of kinematic certainty.
- Speed targets are reduced under poor visibility via $v_{\text{target}} \leftarrow v_{\text{target}} \cdot (1 - \alpha)$, $\alpha \in [0, 0.3]$.
- 3D actor boxes for oncoming vehicles at unprotected turns are inflated by $\Delta m$ before collision checking.

### 2.3 Imitation-Only Training

- All modifications exist solely in the expert’s rules; the student loss remains strictly imitation with no auxiliary regularizers.
- Improved dataset alignment yields substantial gains without architectural regularization.

## 3. Enhanced Navigational Intent Modeling

TFv6 resolves goal specification ambiguities by replacing TFv5’s single-point-to-GRU path conditioning with a three-point route token strategy:

- **Token Formation:** Previous, current, and next GNSS target points $G = \{ p_{t-1}, p_t, p_{t+1} \}$ are normalized to $[-1, 1]$ and embedded:
  $$
  e_i = W_2 \cdot \text{ReLU}(W_1 \cdot [p_i]) \in \mathbb{R}^D
  $$
- **Token Injection:** Route tokens $e_{t-1}, e_t, e_{t+1}$ are prepended to the set of learned queries entering the decoder:
  $$
  [Q;\ e_{t-1};\ e_t;\ e_{t+1}]
  $$
- **Effect:** The transformer’s decoder layers attend jointly over route and BEV features, enabling both immediate and multi-step path reasoning, which mitigates goal-fixation behaviors and improves higher-level maneuvering (e.g., smoother lane changes).

## 4. Evaluation on Closed-Loop CARLA Benchmarks

TFv6 performance is evaluated under CARLA Leaderboard 2.0 standards using the Driving Score (DS), Success Rate (SR), and Normalized DS (NDS) as appropriate. 

| Benchmark             | TFv5           | TFv6 (w/ Radar) | Oracle/Expert (LEAD) |
|-----------------------|----------------|-----------------|----------------------|
| Bench2Drive (DS/SR)   | 83.5 / 67.3%   | 95.2 / 86.8%    | 96.8 / 96.6%         |
| Longest6 v2 (DS/RC)   | 23.0 / 70%     | 62.0 / 91%      | 73.0 / 93%           |
| Town13 (DS/NDS)       | 1.08 / 2.12    | 2.65 / 4.04     | 36.3 / 58.5          |

- **Bench2Drive:** 12 point DS increase and nearly 20% SR gain over TFv5.
- **Longest6 v2:** TFv6 more than doubles prior SOTA DS, notably reducing the gap to the privileged expert.
- **Town13 (unseen):** TFv6 doubles DS and NDS compared to TFv5, indicating robust zero-shot generalization.

## 5. Sim-to-Real Transfer Using Latent TransFuser v6

TFv6’s architecture, minus LiDAR/radar, forms the Latent TransFuser v6 (LTFv6) policy for real-world camera-only evaluation. 

### 5.1 Joint Perception Supervision

- Pre-training with LEAD (CARLA synthetic) plus real-world panoramas (NAVSIM/WOD).
- Auxiliary heads for semantic segmentation and object detection introduce a composite perception loss:
  $$
  L_{\text{perc}} = \lambda_{\text{seg}} L_{\text{seg}} + \lambda_{\text{det}} L_{\text{det}}
  $$

### 5.2 Domain Adaptation and Curriculum

- Curriculum: initial joint sim/real training (e.g., 30 epochs 50/50 mix), transitioning to real-only fine-tuning.
- Photometric augmentation, sensor noise injection, and random weather variations are applied for domain robustness.

### 5.3 Real-World Results

| Benchmark  | LTF (base) | LTFv6 | +LEAD pre-train |
|------------|------------|-------|-----------------|
| NAVSIM v1  | 83.8       | 85.4  | 86.4            |
| NAVSIM v2  | 23.1       | 28.3  | 31.4            |
| WOD-E2E    | —          | 7.51  | 7.76            |

Consistent performance improvements are observed by switching from LTF to LTFv6 and further by leveraging LEAD pre-training. A plausible implication is that synthetic, student-aligned expert data substantially benefits real-world open-loop driving policy learning.

## 6. Significance and Outlook

TFv6 demonstrates that transformer-driven BEV fusion of cameras, LiDAR, and radar—when combined with expert data tailored to learner constraints—can yield near-oracle driving performance under simulated and real conditions. The architecture’s removal of train-test bottlenecks (e.g., late GRUs, weak intent conditioning) and the expert policy’s careful alignment to student observations set new benchmarks in robust, generalizable end-to-end driving. The approach provides a scalable, high-fidelity baseline for future research into transfer learning, long-horizon planning, and sensor fusion in autonomous vehicles [2512.20563].

Source: https://www.emergentmind.com/topics/transfuser-v6-tfv6