Prune2Drive: Efficient Multi-View Token Pruning

• Prune2Drive is a visual token pruning framework designed for multi-view vision-language models in autonomous driving, employing diversity-aware T-FPS and view-adaptive token allocation.
• It integrates token-wise Farthest Point Sampling (T-FPS) for diverse token selection with a learned per-view pruning controller, operating without model retraining or attention-map access.
• The method achieves significant computational gains—up to 6.40× speedup and reducing FLOPs to 13.4% on DriveLM—while incurring only a 3% performance drop.

Prune2Drive is a plug-and-play visual token pruning framework for multi-view vision-LLMs in autonomous driving. It is designed for the standard six-camera surround-view setting used by specialist driving VLMs such as DriveMM and DriveLMM-o1, and targets the computational bottleneck created when high-resolution images are converted into long visual token sequences and concatenated with text tokens for large-language-model inference. Its central design combines a diversity-aware token selection mechanism, implemented as token-wise Farthest Point Sampling (T-FPS), with a view-adaptive pruning controller that learns per-view retention ratios. The method does not require model retraining or access to attention maps, which makes it compatible with modern efficient attention implementations; on DriveLM, retaining only 10% of the visual tokens yields a 6.40$\times$ prefilling speedup and 13.4% of the original FLOPs, with only a 3% performance drop (Xiong et al., 18 Aug 2025).

1. Problem setting and driving-specific motivation

Prune2Drive addresses the inference cost of multi-view driving VLMs in which multiple synchronized cameras are encoded into visual tokens and then processed jointly with language prompts by a transformer-based LLM. In this setting, self-attention scales as $O(N^2)$ in the sequence length $N$, so the number of visual tokens is a primary determinant of latency, FLOPs, memory consumption, and KV-cache growth. The paper situates this problem in a standard autonomous-driving configuration with six cameras—front, front-left, front-right, rear, rear-left, and rear-right—and evaluates on DriveMM and DriveLMM-o1, both of which operate on multi-view visual inputs and language-based driving queries (Xiong et al., 18 Aug 2025).

The framework is motivated by a mismatch between generic VLM pruning heuristics and the structure of autonomous-driving perception. Existing training-free token pruning methods such as FastV, DART, Sparse VLM, and PACT are treated as baselines, but they were not designed for multi-view driving scenes. Attention-based methods can inherit positional biases and may discard low-attention tokens that remain semantically important for downstream driving tasks. Uniform pruning across views is also suboptimal, because view importance is not constant: front-view content often dominates lane-level perception and traffic-light interpretation, while side and rear views become important for merges, overtakes, and adjacent-vehicle reasoning. Prune2Drive therefore treats token reduction as a multi-view allocation problem rather than a single global ranking problem (Xiong et al., 18 Aug 2025).

The framework is evaluated on two driving-specific benchmarks. DriveLM, built on NuScenes, includes perception, reasoning, and decision-making question answering, with metrics including Accuracy, ChatGPT score, BLEU-4, ROUGE-L, CIDEr, Match, and an Average score. DriveLMM-o1 emphasizes step-by-step reasoning with metrics including Risk Assessment Accuracy, Traffic Rule Adherence, Scene Awareness & Object Understanding, Relevance, Missing Details, and Overall Reasoning. These evaluation settings are important because Prune2Drive is explicitly designed to preserve multi-view driving semantics rather than only minimizing token count (Xiong et al., 18 Aug 2025).

2. Placement in the VLM pipeline and plug-and-play design

Prune2Drive is inserted between the vision encoder and the LLM. In the baseline pipeline, each camera image is passed through a Vision Transformer-like encoder, producing a sequence of visual tokens in $\mathbb{R}^{N \times d}$, where $N$ is the total token count across views and $d$ is the embedding dimension. These visual tokens are then concatenated with text tokens and fed into the LLM. Prune2Drive intercepts the visual token sequence, prunes it, and forwards the shortened sequence to the unchanged downstream model (Xiong et al., 18 Aug 2025).

The method was evaluated with two specialist driving VLMs. DriveMM uses a SigLIP vision encoder at resolution $384 \times 384$, a 2-layer MLP projector, and a Llama-3.1 8B LLM trained on sequences up to 8192 tokens. DriveLMM-o1 is based on InternVL2.5-8B with a ViT image encoder, a LLaMA 2-based LLM, and dynamic image patching. In both cases, Prune2Drive operates on the visual token stream without modifying the model architecture or retraining core model parameters (Xiong et al., 18 Aug 2025).

Its workflow has three stages. First, an offline calibration step learns per-view pruning ratios. Second, an online token-selection stage applies T-FPS independently within each view according to the learned token budget. Third, the retained tokens are concatenated and passed through the existing projector and LLM. Because the framework does not depend on attention maps, it remains compatible with FlashAttention and related efficient attention kernels. This design is the basis for the paper’s characterization of Prune2Drive as plug-and-play and training-free (Xiong et al., 18 Aug 2025).

3. Core mechanisms: T-FPS and the view-adaptive pruning controller

The first core mechanism is T-FPS, a token-wise adaptation of Farthest Point Sampling. Rather than scoring tokens by attention magnitude, T-FPS selects a subset that maximizes diversity in embedding space. For token embeddings $V[i], V[j] \in \mathbb{R}^d$, cosine similarity is defined as

$$\text{sim}(V[i], V[j]) = \frac{V[i] \cdot V[j]}{\|V[i]\| \, \|V[j]\|},$$

and cosine distance as

$$\text{dist}(V[i], V[j]) = 1 - \text{sim}(V[i], V[j]).$$

The algorithm initializes with a randomly selected token, maintains for each remaining token its minimum distance to the selected set, and iteratively adds the token whose minimum distance is maximal. The retained subset is therefore spread across semantic embedding space, which the paper interprets as preserving semantic and spatial coverage while reducing redundancy (Xiong et al., 18 Aug 2025).

The second mechanism is the view-adaptive pruning controller. Instead of using a fixed pruning ratio for every camera, the controller learns a vector of per-view retention ratios. Its optimization is framed through an evaluation metric

$O(N^2)$0

where the reward term is

$O(N^2)$1

and the penalty term is

$O(N^2)$2

Here $O(N^2)$3 denotes the retention ratio for view $O(N^2)$4, and $O(N^2)$5 controls the trade-off between task performance and token cost. The controller is optimized offline on a small subset of data, with the paper comparing TPE, Evolutionary algorithms, and GridSearch; TPE achieves the best results on both DriveLM and DriveLMM-o1 (Xiong et al., 18 Aug 2025).

The two components are complementary. T-FPS determines which tokens to retain within a given view budget, while the controller determines how much budget each view receives. The first addresses redundancy; the second addresses asymmetric view importance. This design is explicitly contrasted with attention-based pruning methods and with methods that prune each view uniformly, because Prune2Drive treats multi-view driving as a structured allocation problem rather than a flat token-ranking problem (Xiong et al., 18 Aug 2025).

4. Benchmarks and empirical performance

On DriveLM with the DriveMM backbone, the unpruned upper bound reports Accuracy 0.81, ChatGPT 65.44, BLEU-4 0.61, ROUGE 0.74, CIDEr 0.19, Match 33.9, and Average 59.1. At 75% pruning, corresponding to retaining 25% of tokens and 180 tokens per image, Prune2Drive achieves Accuracy 0.80, ChatGPT 64.92, BLEU-4 0.60, ROUGE 0.75, CIDEr 0.20, Match 34.0, and Average 58.3. At 90% pruning, corresponding to retaining 10% of tokens and 72 tokens per image, it achieves Accuracy 0.78, ChatGPT 64.52, BLEU-4 0.56, ROUGE 0.74, CIDEr 0.16, Match 33.4, and Average 57.4. The paper emphasizes that at 10% token retention this amounts to only about a 3% performance drop while retaining the highest average score among the compared pruning methods (Xiong et al., 18 Aug 2025).

On DriveLMM-o1, the unpruned upper bound reports Accuracy 73.01, Risk Assessment Accuracy 80.89, Traffic Rule Adherence 75.99, Scene Awareness & Object Understanding 80.25, Relevance 72.22, and Overall Reasoning 74.2. At 75% pruning, corresponding to 64 tokens per image, Prune2Drive reports Accuracy 69.28, Risk 75.84, Traffic Rule 70.99, Scene Awareness & Object Understanding 75.64, Relevance 66.95, and Overall Reasoning 69.3. At 90% pruning, corresponding to 25 tokens per image, it reports Accuracy 68.34, Risk 74.92, Traffic Rule 69.86, Scene Awareness & Object Understanding 74.54, Relevance 66.34, and Overall Reasoning 68.3. In both pruning regimes, it outperforms the listed training-free baselines on Overall Reasoning (Xiong et al., 18 Aug 2025).

Benchmark and setting	Prune2Drive result	Reference point
DriveLM, 75% pruning	Average 58.3	Upper bound 59.1
DriveLM, 90% pruning	Average 57.4	Upper bound 59.1
DriveLMM-o1, 75% pruning	Overall Reasoning 69.3	Upper bound 74.2
DriveLMM-o1, 90% pruning	Overall Reasoning 68.3	Upper bound 74.2

Qualitative analysis is used to support these quantitative results. In one example, under the prompt “What are the important objects in the scene based on surrounded view?”, Prune2Drive correctly identifies a white van in front of the ego vehicle, a black sedan behind the ego vehicle, and a person riding a motorcycle in front, whereas FastV produces an incorrect description. Token visualizations further show that attention-based baselines can over-retain redundant back-view tokens or miss critical front-view objects, while Prune2Drive retains tokens covering key vehicles across front and back-right views (Xiong et al., 18 Aug 2025).

5. Efficiency profile and ablation results

The principal computational benefit appears in the prefilling phase. On DriveMM with DriveLM at 10% token retention, Prune2Drive delivers a 6.40$O(N^2)$6 speedup in prefilling, a 1.09$O(N^2)$7 speedup in decoding, and reduces total FLOPs to 13.4% of the original. On DriveLMM-o1 at the same retention level, it delivers a 2.64$O(N^2)$8 prefilling speedup, a 1.04$O(N^2)$9 decoding speedup, and reduces FLOPs to 20.3% of the original. The paper also reports substantial KV-cache reduction, describing it as nearly a 9$N$0 reduction for DriveMM and about a 5$N$1 reduction for DriveLMM-o1 (Xiong et al., 18 Aug 2025).

Model and retention	Speed / FLOPs result	Performance result
DriveMM, 10% tokens	6.40$N$2 prefilling, 13.4% FLOPs	Average 57.4
DriveLMM-o1, 10% tokens	2.64$N$3 prefilling, 20.3% FLOPs	Overall Reasoning 68.3

The ablations isolate the contribution of T-FPS and the controller. For token selection distance, cosine distance is best or tied-best across both benchmarks, while replacing farthest-point selection with minimum-distance selection causes a marked drop. On DriveLM at 90% pruning, l1 distance yields Average 56.9, l2 distance 57.1, cosine distance 57.4, and minimum-distance selection 52.3. On DriveLMM-o1 at 90% pruning, l1 yields Overall Reasoning 68.3, l2 yields 67.7, cosine yields 68.3, and minimum-distance selection falls to 63.0. The interpretation given by the paper is that minimum-distance selection retains clustered, redundant tokens and therefore harms coverage (Xiong et al., 18 Aug 2025).

The controller search strategy exhibits smaller but consistent effects. On DriveLM at 90% pruning, GridSearch reaches Average 57.2, Evolutionary reaches 57.1, and TPE reaches 57.4. On DriveLMM-o1 at the same pruning level, GridSearch reaches Overall Reasoning 67.3, Evolutionary 67.6, and TPE 68.3. These results are used to justify the paper’s choice of TPE for the offline view-ratio calibration phase (Xiong et al., 18 Aug 2025).

6. Limitations, later developments, and position in the literature

The explicit limitation of Prune2Drive is that it is designed for single-frame image inputs. The paper notes that many autonomous-driving VLMs use multi-frame video to capture temporal dynamics, and identifies video-based pruning as a future direction because token count scales with both views and frames. A plausible implication is that Prune2Drive solves the multi-view allocation problem but not the temporal redundancy problem (Xiong et al., 18 Aug 2025).

Subsequent work in autonomous-driving VLM pruning can be read as extending precisely these unresolved dimensions. ST-Prune introduces a training-free spatio-temporal pruning framework with Motion-aware Temporal Pruning and Ring-view Spatial Pruning, explicitly targeting multi-view, multi-frame inputs; it reports near-lossless performance even at 90% token reduction on several benchmarks (Sha et al., 21 Apr 2026). MVPruner characterizes Prune2Drive as using offline hyperparameter search on a calibration set to assign fixed budgets per view and as being limited to multi-view, single-frame settings, then proposes a two-stage adaptive method with diversity-aware and instruction-aware ratio allocation; on DriveLM, DriveMM equipped with MVPruner achieves 87.3% reduction in FLOPs and a 4.97$N$4 speedup in the prefilling phase while retaining 98.5% accuracy (Yang et al., 26 Jun 2026). FastDriveVLA moves from VLM question answering to end-to-end driving, operationalizing the same efficiency objective through a reconstruction-based, foreground-aware token pruner called ReconPruner for VLA systems, and reports state-of-the-art results on the nuScenes benchmark across pruning ratios (Cao et al., 31 Jul 2025).

Within this progression, Prune2Drive occupies a specific position. It is an early autonomous-driving-specific framework for multi-view visual token pruning that replaces attention-map dependence with diversity-aware selection and explicit per-view token budgeting. Its main historical significance lies in showing that a training-free, plug-and-play token-pruning module can preserve or slightly improve multi-view driving VLM behavior under substantial compression, and in formalizing view allocation as a first-class optimization problem in autonomous-driving token pruning (Xiong et al., 18 Aug 2025).