EgoPrune: Geometry-Aware Token Pruning

• EgoPrune is a training-free, geometry-aware framework that efficiently prunes tokens in egomotion videos by leveraging spatiotemporal continuity and motion constraints.
• It employs a modular pipeline including keyframe selection, perspective-aware redundancy filtering with homography, and MMR-based token selection for optimal query relevance and diversity.
• Benchmark tests on VSI-Bench and UrbanVideo-Bench show that EgoPrune maintains near full-token accuracy while significantly reducing computational overhead on both high-end and edge devices.

EgoPrune is a training-free, geometry-aware token pruning framework designed to enhance multimodal reasoning efficiency in embodied agents processing egomotion videos. Egomotion videos, which are first-person sequences reflecting the agent’s continuous viewpoint changes, present significant redundancy and computational challenges for vision-LLMs (VLMs). EgoPrune streamlines video understanding by leveraging spatiotemporal continuity and motion constraints unique to egomotion data, resulting in substantial FLOPs, latency, and memory reductions while maintaining—often surpassing—full-token reasoning accuracy. The system is expressly tailored for real-time and on-device deployment within resource-constrained embodied AI agents (Li et al., 21 Jul 2025).

1. System Architecture and Workflow

EgoPrune operates atop a standard vision-language video reasoning pipeline, sequentially orchestrating three core modules: a keyframe selector adapted from EmbodiedR, Perspective-Aware Redundancy Filtering (PARF), and a Maximal Marginal Relevance (MMR) token selector. The processing flow for an egomotion video is as follows:

• Frame Sampling: Input video is sampled at 1 FPS, yielding $T$ raw frames $f_1,\ldots,f_T$.
• Keyframe Selection: Using EmbodiedR’s overlap test, temporally informative frames are retained (see Section 2).
• Redundancy Filtering (PARF): For each adjacent pair of keyframes, tokens from spatially redundant patches are pruned by aligning frames with a homography and applying cosine similarity-based redundancy checks.
• Token Selection (MMR): The remaining tokens across all keyframes are further reduced via a greedy selection process that jointly optimizes for text query relevance and intra-frame diversity.
• LLM Input: The resulting compact visual token set is concatenated with system/user prompt tokens and passed to the LLM for multimodal inference.

This hierarchical reduction exploits both the temporal and geometric properties inherent in egomotion videos, efficiently filtering tokens at each stage. The modular structure ensures compatibility with various backbones and mobile hardware (Li et al., 21 Jul 2025).

2. Mathematical Foundations

2.1 Keyframe Selector (EmbodiedR-Based)

Keyframe selection is governed by the frame overlap score: $$O(f_{\rm prev},\,f_{\rm curr}) = \frac{|{\rm Matches}(P_{\rm prev},P_{\rm curr})|}{\min(|P_{\rm prev}|,|P_{\rm curr}|)}$$ where $P_{\rm prev}$ and $P_{\rm curr}$ are ORB keypoints from consecutive frames. A keyframe is emitted whenever $O(f_{\rm prev},\,f_{\rm curr}) < \theta_{\rm kf}$, with $\theta_{\rm kf}$ set to 0.60 (i.e., below 60% overlap), ensuring only non-redundant, new-visual-content frames are selected.

2.2 Perspective-Aware Redundancy Filtering (PARF)

PARF computes a homography $H$ aligning keyframes $f_1$ and $f_2$: $f_1,\ldots,f_T$0

$f_1,\ldots,f_T$1 is estimated using ORB keypoint matches and RANSAC. Each patch-token center from the previous frame is warped to the new image plane, and cosine similarity is computed between aligned patch embeddings: $f_1,\ldots,f_T$2

Tokens with similarity exceeding $f_1,\ldots,f_T$3 are dropped as redundant, effectively removing inter-frame spatial duplications caused by agent motion.

2.3 MMR-Based Token Selector

Let $f_1,\ldots,f_T$4 denote the pruned token set and $f_1,\ldots,f_T$5 the average prompt embedding. Relevance is defined by their embedding similarity: $f_1,\ldots,f_T$6 The selector greedily adds tokens to set $f_1,\ldots,f_T$7 of size $f_1,\ldots,f_T$8 by: $f_1,\ldots,f_T$9 $$O(f_{\rm prev},\,f_{\rm curr}) = \frac{|{\rm Matches}(P_{\rm prev},P_{\rm curr})|}{\min(|P_{\rm prev}|,|P_{\rm curr}|)}$$0 is the set of candidates, $$O(f_{\rm prev},\,f_{\rm curr}) = \frac{|{\rm Matches}(P_{\rm prev},P_{\rm curr})|}{\min(|P_{\rm prev}|,|P_{\rm curr}|)}$$1 controls relevance-diversity trade-off (typically $$O(f_{\rm prev},\,f_{\rm curr}) = \frac{|{\rm Matches}(P_{\rm prev},P_{\rm curr})|}{\min(|P_{\rm prev}|,|P_{\rm curr}|)}$$2–$$O(f_{\rm prev},\,f_{\rm curr}) = \frac{|{\rm Matches}(P_{\rm prev},P_{\rm curr})|}{\min(|P_{\rm prev}|,|P_{\rm curr}|)}$$3). This balances retaining query-relevant content and minimizing intra-set redundancy.

3. Benchmark Results and Efficiency

Extensive evaluations established the accuracy and efficiency of EgoPrune relative to full-token and prior pruning baselines (DivPrune, PACT):

3.1 Accuracy Retention

• VSI-Bench (288 videos, 5K QA): At 30–70% token retention, EgoPrune matched or slightly exceeded full-token accuracy (35.16–35.43 vs. 35.45 avg), outperforming prior pruning methods.
• UrbanVideo-Bench (1K aerial videos, 5.2K QA): EgoPrune sustained accuracy (45.66–46.25 vs. 46.23) across token budgets (Li et al., 21 Jul 2025).

3.2 Computational Gains

Setting	Token Retention	FLOPs Reduction	Latency Reduction	Peak Memory Savings
GPU, 90 frames (vs. DivPrune)	50%	~30%	~20–25%	~20%
Jetson Orin NX, on-device	50%	—	~25% (1.2 s→0.9 s)	~23% (13 GB→10 GB)

EgoPrune’s efficiency is maintained across varying frame counts and retention ratios, with robust performance on both high-end servers (A6000 GPUs) and edge devices (Jetson Orin NX 16GB), including all critical pipeline stages (Li et al., 21 Jul 2025).

3.3 MMR Trade-off Sensitivity

Ablations on the MMR λ parameter demonstrate stable accuracy for 50–70% retention (VSI-Bench), with optima at λ ≈ 0.4–0.6 for UrbanVideo-Bench.

4. Implementation and Experimental Protocol

Experiments utilized:

• Datasets: VSI-Bench (indoor spatial reasoning), UrbanVideo-Bench (aerial), each with >5K QA samples.
• Backbones: LLaVA-OneVision-7B for core tests; VILA-1.5 3B (TensorRT-LLM, INT4 quantized) for embedded deployment.
• Hardware: Server: 3×NVIDIA A6000 (50GB); Embedded: NVIDIA Jetson Orin NX 16GB.
• Frame Processing: Videos sampled at 1 FPS; each frame tokenized to 14×14 = 196 patches; videos >90 s filtered out; token retention rates tested at 70%, 50%, 30%.
• Software: PARF runs on CPU (ORB-RANSAC + homography + warp; 10–20 ms per frame-pair). MMR uses optimized PyTorch with windowed similarity (w = 10).
• Evaluation: LMMS-Eval for accuracy; DeepSpeed Profiler for FLOPs/memory.

The entire pipeline adheres to real-time constraints suitable for robotics and UAVs. Empirically derived “tipping points” for latency and memory indicate practical token retention targets for deployment.

5. On-Device Integration and Practical Application

EgoPrune is integrated into vision-language stacks deployed on edge devices. Both vision and language backbones are compiled to TensorRT-LLM with INT4 quantization, ensuring fast inference within the limited compute and memory budgets of mobile robotic systems. The pipeline is compatible with real-time mobile scenarios, including frame sampling, keyframe extraction, redundancy pruning, token selection, and complete model inference.

Systematic measurements on the Jetson Orin NX (16GB) confirm that 40–50% token retention balances low latency (below 1s) and memory overhead (below 10GB)—key for real-world robotics or UAV use.

6. Limitations and Prospective Advances

• Homography constraint: The assumption of planar or pure-rotation scenes may produce suboptimal pruning in nonplanar or strong translation scenarios, risking misalignment and over/under-pruning.
• Feature quality: ORB + RANSAC matching, while lightweight, can degrade under low-texture or blurred conditions.
• MMR computational complexity: MMR’s greedy selection scales as $$O(f_{\rm prev},\,f_{\rm curr}) = \frac{|{\rm Matches}(P_{\rm prev},P_{\rm curr})|}{\min(|P_{\rm prev}|,|P_{\rm curr}|)}$$4, potentially bottlenecking on high-resolution/lengthy videos, though windowed similarity mitigates this in practice.
• Hyperparameter dependence: Key thresholds ($$O(f_{\rm prev},\,f_{\rm curr}) = \frac{|{\rm Matches}(P_{\rm prev},P_{\rm curr})|}{\min(|P_{\rm prev}|,|P_{\rm curr}|)}$$5, $$O(f_{\rm prev},\,f_{\rm curr}) = \frac{|{\rm Matches}(P_{\rm prev},P_{\rm curr})|}{\min(|P_{\rm prev}|,|P_{\rm curr}|)}$$6) are empirically set and may require adaptation for novel domains (e.g., underwater, surgical).
• Future directions: Investigated expansions include multi-plane or depth-aware geometric warps, adaptive—potentially learnable—thresholds, optical flow-based motion cues, and end-to-end differentiable pruning architectures jointly optimized with the LLM.

EgoPrune, by leveraging geometric and query-driven priors with a purely inference-time, training-free approach, achieves near full-token performance while providing substantial resource reductions. Its modular, hardware-agnostic design supports immediate application in resource-constrained embodied AI systems, and its limitations motivate active areas for research innovation (Li et al., 21 Jul 2025).