FastDriveVLA: Token Pruning for Driving

• The paper introduces FastDriveVLA, a reconstruction-based visual token pruning framework that cuts computational costs in VLA autonomous driving systems while preserving performance.
• It leverages a plug-and-play ReconPruner trained via MAE-style pixel reconstruction and adversarial foreground-background strategies to focus on critical driving features.
• Evaluated on Qwen2.5-VL and nuScenes-FG, FastDriveVLA achieves up to a 7.5× FLOP reduction with minimal impact on key planning metrics like L2 error, collision, and intersection rates.

FastDriveVLA is a reconstruction-based visual token pruning framework for end-to-end autonomous driving with Vision-Language-Action models. It is designed to reduce the computational burden created by long visual token sequences while preserving driving performance, and it does so through a plug-and-play pruner, ReconPruner, trained to prioritize foreground content via MAE-style pixel reconstruction and an adversarial foreground-background reconstruction strategy. The framework is introduced together with nuScenes-FG, a 241,000 image-mask-pair dataset for foreground supervision, and is evaluated primarily with Qwen2.5-VL-based driving models, especially Impromptu-VLA (Cao et al., 31 Jul 2025).

1. Problem setting and foreground-oriented motivation

FastDriveVLA addresses a specific bottleneck in VLA-based autonomous driving: high-resolution multi-camera inputs produce very large visual token sequences, and the downstream language-action stack scales roughly quadratically with sequence length. In the reported setup, Impromptu-VLA on nuScenes images at resolution $1596 \times 1596$ produces 3249 visual tokens per frame, requires 38.2T FLOPs per sample, and exhibits a prefill latency of 187 ms/token. This cost profile is problematic for deployment under in-car latency and compute constraints (Cao et al., 31 Jul 2025).

The framework is motivated by the claim that generic visual token pruning criteria used in vision-LLMs transfer poorly to driving. Attention-based methods such as FastV and SparseVLM rely on text-to-vision attention, but autonomous-driving prompts are often fixed and minimally informative, for example “Drive safely to the destination,” which weakens attention as a ranking signal. Similarity-based methods such as VisPruner, DivPrune, and DART attempt to remove redundancy in feature space, but driving scenes contain semantically critical foreground regions alongside relatively uniform background regions, so diversity-based selection can retain background tokens while discarding planning-relevant foreground tokens (Cao et al., 31 Jul 2025).

FastDriveVLA therefore adopts a human-driver analogy: the driving policy should preferentially preserve tokens representing drivable lanes, lane markings and signs, vehicles, pedestrians, cyclists, traffic lights and signs, and roadside barriers close to the road. Background elements such as sky, buildings, and distant trees are treated as comparatively less relevant for moment-to-moment control. This foreground-centered premise is the conceptual basis of the pruning method (Cao et al., 31 Jul 2025).

2. System architecture and plug-and-play integration

The framework sits between a frozen visual encoder and the VLA core. Multi-camera images are first encoded by the Qwen2.5-VL vision encoder into visual tokens $V \in \mathbb{R}^{N \times D}$. ReconPruner assigns a saliency score to each token, keeps the top-$M$ tokens according to a chosen pruning ratio, and forwards those tokens together with their corresponding positional embeddings to the downstream VLA model. The action module then consumes the concatenated visual-text sequence and predicts trajectories or control actions (Cao et al., 31 Jul 2025).

Formally, with pruning ratio $p \in [0,1]$, the retained token count is

$M = \lfloor N \cdot (1-p) \rfloor,$

and the selected index set is obtained by TopK over the saliency scores. The paper studies 25%, 50%, and 75% pruning, corresponding to retaining 2436, 1624, and 812 tokens out of 3249, respectively (Cao et al., 31 Jul 2025).

A defining property of the system is its plug-and-play transfer behavior. ReconPruner is trained once for a specific visual encoder architecture, here Qwen2.5-VL, and can then be inserted into any driving VLA using the same visual encoder and tokenization scheme without retraining the VLA itself. The detailed description contrasts this with projector-based compression approaches such as TokenPacker, Honeybee, and Matryoshka, which require retraining or fine-tuning the whole model (Cao et al., 31 Jul 2025).

This encoder-level reusability is significant within the broader VLA literature. Other driving VLA directions emphasized adaptive reasoning with discrete motion tokens (Zhou et al., 16 Jun 2025), continual long-tail refinement through takeover data and DPO (Fang et al., 19 Sep 2025), or stronger spatial grounding through BEV injection and Render-Teacher Alignment (Wang et al., 23 Jun 2026). FastDriveVLA instead targets the visual-token efficiency bottleneck while leaving the downstream policy largely unchanged.

3. ReconPruner architecture and reconstruction-based training

ReconPruner is lightweight, with 0.07B parameters, and consists of two components: a PrunerLayer implemented as a single decoder layer from Qwen2.5-VL-3B, and a Scorer implemented as a single-layer feedforward network of shape $\mathbb{R}^{D \times 1}$. Given visual tokens $V \in \mathbb{R}^{N \times D}$ and a learnable query token $Q \in \mathbb{R}^{1 \times D}$, token fusion is performed as

$[Q^*, V^*] = \text{PrunerLayer}([Q, V]).$

The PrunerLayer uses full attention rather than causal attention, because the reconstruction task is non-causal. Saliency is then computed by fusing each updated token with the updated query through a Hadamard product:

$S = \text{Scorer}(V^* \odot Q^*).$

Higher $V \in \mathbb{R}^{N \times D}$0 indicates higher token importance (Cao et al., 31 Jul 2025).

Training is based on MAE-style masked image modeling, but with explicit foreground-background decomposition. Each image is paired with a foreground mask from nuScenes-FG, from which the method derives a foreground-only image $V \in \mathbb{R}^{N \times D}$1 and a background-only image $V \in \mathbb{R}^{N \times D}$2. Saliency scores are binarized into a mask

$V \in \mathbb{R}^{N \times D}$3

and the Straight-Through Estimator is used during training through

$V \in \mathbb{R}^{N \times D}$4

This splits the encoder tokens into foreground and background subsets:

$V \in \mathbb{R}^{N \times D}$5

A reconstruction decoder $V \in \mathbb{R}^{N \times D}$6, implemented as six Qwen2.5-VL decoder layers plus a linear head, maps these token sets back to pixel space:

$V \in \mathbb{R}^{N \times D}$7

Foreground and background losses each combine SSIM and MSE, with $V \in \mathbb{R}^{N \times D}$8, and the total loss weights the two branches equally with $V \in \mathbb{R}^{N \times D}$9 (Cao et al., 31 Jul 2025).

The adversarial element arises from the coupled foreground-background allocation. If all tokens are assigned to foreground, background reconstruction degrades; if all tokens are assigned to background, foreground reconstruction degrades. The method therefore forces discriminative token assignment rather than trivial all-high or all-low saliency. The paper argues that this yields a foreground-aware ranking over tokens that is more compatible with planning than attention-only or similarity-only heuristics (Cao et al., 31 Jul 2025).

Training uses a frozen Qwen2.5-VL-3B visual encoder and nuScenes-FG only, with no driving labels. The reported hyperparameters are a learning rate of $M$0, cosine scheduling, 10 epochs, and approximately 3 hours of training on 2×H800 GPUs (Cao et al., 31 Jul 2025).

4. nuScenes-FG dataset and supervision pipeline

FastDriveVLA introduces nuScenes-FG to provide the dense foreground supervision required by ReconPruner. The dataset is built from the original nuScenes dataset, described here as 1000 driving scenes of 20 seconds each with six cameras, and contains 241,000 image-mask pairs (Cao et al., 31 Jul 2025).

Foreground is defined in a driving-centric way. Included categories are humans, roads and lanes, vehicles, traffic signs and traffic lights, and traffic barriers and roadside obstacles near the road. Background categories such as sky, buildings, and far trees are treated as irrelevant for driving decisions. The paper argues that original nuScenes annotations are insufficient for this purpose because 3D bounding boxes include substantial irrelevant background pixels and the map layers do not cleanly cover all relevant regions in image space (Cao et al., 31 Jul 2025).

Annotation is performed with Grounded-SAM, described as Grounding DINO plus SAM or HQ-SAM. Text prompts are used to detect the categories of interest, and segmentation masks are then generated for the nuScenes camera views. The resulting dataset provides pixel-level foreground-background supervision for training the pruner and extends nuScenes from 3D boxes and maps to dense, driving-centric segmentation (Cao et al., 31 Jul 2025).

A plausible implication is that nuScenes-FG is not only an auxiliary dataset for pruning, but also a reusable resource for other driving methods that require dense foreground supervision. The paper explicitly notes that the dataset can be reused by others (Cao et al., 31 Jul 2025).

5. Efficiency profile and reported planning performance

The reported efficiency gains are centered on token reduction in Impromptu-VLA. At 75% pruning, token count falls from 3249 to 812, FLOPs from 38.2T to 5.1T, prefill latency from 187 ms/token to 51 ms/token, and decode latency from 23 ms/token to 18 ms/token. This corresponds to an approximately 7.5× FLOPs reduction, approximately 3.7× prefill speedup, and approximately 1.3× decode speedup (Cao et al., 31 Jul 2025).

The paper evaluates planning with average L2 error, collision rate, and intersection rate. The unpruned Impromptu-VLA baseline reports average L2 of 31.83 cm, average collision of 0.24%, and average intersection of 2.80%. At 25% pruning, FastDriveVLA reports average L2 of 31.80 cm, average collision of 0.26%, and average intersection of 2.77%, which the paper describes as slightly better than baseline in L2 and intersection while remaining close in collision. At 50% pruning, it reports 32.10 cm average L2, 0.25% average collision, and 2.94% average intersection; at 75% pruning, 32.64 cm, 0.29%, and 2.91%, respectively (Cao et al., 31 Jul 2025).

The detailed comparison against prior pruning baselines is most explicit at 50% pruning. For Impromptu-VLA retaining 1624 tokens, FastDriveVLA reports average L2 of 32.10 cm, corresponding to 99.1% of baseline performance, whereas the best existing pruner listed there, VisPruner, reports 32.25 cm, or 98.7% of baseline. The paper further states that FastDriveVLA retains or improves a higher fraction of collision and intersection performance than competing baselines at the same pruning ratios (Cao et al., 31 Jul 2025).

The authors recommend 50% pruning as the best trade-off between efficiency and reliability. They also note that some collision and intersection values occasionally improve at higher pruning ratios, likely because the absolute rates are very small, whereas L2 generally degrades as pruning becomes more aggressive (Cao et al., 31 Jul 2025).

6. Ablations, limitations, and position in the literature

The ablation study isolates two central design choices: pixel reconstruction and adversarial foreground-background reconstruction. At 50% pruning, replacing pixel reconstruction with foreground mask prediction worsens all reported metrics, which the paper interprets as evidence that mask supervision alone is too coarse. Keeping pixel reconstruction but removing the adversarial foreground-background split also degrades pruning performance, because the pruner tends to assign high scores to too many tokens. The full method, combining pixel reconstruction with adversarial foreground-background reconstruction, performs best on L2, collision, and intersection metrics (Cao et al., 31 Jul 2025).

The paper also compares FastDriveVLA with a “cheating” baseline that uses ground-truth foreground masks together with text attention. That baseline improves over text attention alone, confirming that foreground focus is beneficial, yet FastDriveVLA still achieves better collision and intersection and comparable or better L2 without requiring ground-truth masks at inference. The paper attributes this in part to two issues: binary masks cannot express fine-grained saliency within the foreground, and visual tokens are often spatially misaligned with image patches, making simple mask-to-token resizing noisy (Cao et al., 31 Jul 2025).

Qualitative analyses support the same interpretation. Foreground versus background reconstructions show that high-score tokens preserve foreground structures, and token-selection maps show that FastDriveVLA concentrates retained tokens on lanes, vehicles, and lane signs. Attention-based pruning is described as missing some vehicles and under-focusing on lanes, whereas similarity-based pruning retains scattered tokens that are not concentrated on planning-critical regions (Cao et al., 31 Jul 2025).

The paper acknowledges several limitations. ReconPruner is encoder-dependent and would need retraining for a different visual encoder. It is trained on nuScenes-FG, so performance may degrade in substantially different domains without adaptation. The foreground definition is hand-designed and may miss corner cases where nominal background suddenly becomes critical. Evaluation is reported with nuScenes planning metrics rather than real-world deployment. Finally, pruning is only explored up to 75%, because the domain is safety-critical and the authors regard some redundancy as desirable (Cao et al., 31 Jul 2025).

Within the broader literature, FastDriveVLA occupies a distinct niche. Other driving VLAs in the provided record pursue adaptive reasoning with GRPO and discrete physical action tokens (Zhou et al., 16 Jun 2025), continual long-tail refinement through Collect-and-Refine and DPO (Fang et al., 19 Sep 2025), or stronger geometry via BEV DeepStack injection and Render-Teacher Alignment (Wang et al., 23 Jun 2026). FastDriveVLA instead treats efficiency as a foreground-aware representation problem at the visual-token interface. Its principal contribution is to replace generic pruning criteria with a driving-specific saliency mechanism learned through reconstruction and dense foreground supervision, while preserving compatibility with existing Qwen2.5-VL-based driving stacks (Cao et al., 31 Jul 2025).