HeSS: Head Sensitivity Score for Sparsity Redistribution in VGGT

Abstract: Visual Geometry Grounded Transformer (VGGT) has advanced 3D vision, yet its global attention layers suffer from quadratic computational costs that hinder scalability. Several sparsification-based acceleration techniques have been proposed to alleviate this issue, but they often suffer from substantial accuracy degradation. We hypothesize that the accuracy degradation stems from the heterogeneity in head-wise sparsification sensitivity, as the existing methods apply a uniform sparsity pattern across all heads. Motivated by this hypothesis, we present a two-stage sparsification pipeline that effectively quantifies and exploits headwise sparsification sensitivity. In the first stage, we measure head-wise sparsification sensitivity using a novel metric, the Head Sensitivity Score (HeSS), which approximates the Hessian with respect to two distinct error terms on a small calibration set. In the inference stage, we perform HeSS-Guided Sparsification, leveraging the pre-computed HeSS to reallocate the total attention budget-assigning denser attention to sensitive heads and sparser attention to more robust ones. We demonstrate that HeSS effectively captures head-wise sparsification sensitivity and empirically confirm that attention heads in the global attention layers exhibit heterogeneous sensitivity characteristics. Extensive experiments further show that our method effectively mitigates performance degradation under high sparsity, demonstrating strong robustness across varying sparsification levels. Code is available at https://github.com/libary753/HeSS.

• The paper introduces HeSS, a novel metric that quantifies head-wise sensitivity to guide sparsity redistribution in transformer-based 3D vision models.
• It leverages second-order loss curvature via approximated Fisher Information Matrix traces to balance global camera pose and local point cloud errors.
• Experimental results demonstrate that HeSS-guided sparsification maintains high geometric fidelity and significantly improves performance under aggressive sparsity.

HeSS: Head Sensitivity Score for Sparsity Redistribution in VGGT

Introduction and Motivation

The Visual Geometry Grounded Transformer (VGGT) serves as a unified end-to-end architecture for multi-view 3D reconstruction, integrating both Structure-from-Motion (SfM) and Multi-View Stereo (MVS) within a single transformer framework. VGGT achieves this through an alternating design of Global Attention (GA) and Frame Attention (FA) layers, enabling both cross-frame and intra-frame reasoning. However, the quadratic computational complexity of the GA layers with respect to the number of views poses significant obstacles to scaling VGGT for real-time or large-scale geometry applications.

A series of acceleration strategies for VGGT—most notably SparseVGGT—address this bottleneck by approximating the dense attention with block-sparse attention maps. Yet, these methods typically impose a uniform sparsification pattern across all transformer heads. This uniformity ignores potential heterogeneity in head-wise sensitivity to sparsification, often leading to marked accuracy drops, particularly under high sparsity constraints.

The authors hypothesize that the principal cause of performance degradation in existing sparsification schemes is failure to recognize and accommodate the differential impact of sparsification across attention heads. They propose a principled two-stage pipeline for VGGT sparsification: (1) a calibration stage, which quantifies head-wise sensitivity to sparsification via a novel Head Sensitivity Score (HeSS), and (2) an inference-time stage, in which the global attention computation is adaptively sparsified, prioritizing heads identified as critical.

Methodology

HeSS: Quantifying Head-Wise Sensitivity

The central technical contribution is the definition and empirical estimation of the Head Sensitivity Score (HeSS)—a scalar metric attributed to each attention head in the GA layers. HeSS quantifies the anticipated loss in task performance resulting from sparsification of a given head.

HeSS is computed as a weighted sum of two error curvatures: camera pose error ($e_{\text{cam}}$), measuring global geometric fidelity, and point cloud error ($e_{\text{pc}}$), capturing fine-grained geometric detail. For each attention head, the curvature is analytically approximated via the trace of the respective Fisher Information Matrix (FIM) with respect to the head's query projection weights, providing a tractable surrogate for the true Hessian. The final HeSS for a head $h$ is formalized as

$$\mathrm{HeSS}(h) = \lambda \cdot \mathrm{HeSS}_\text{cam}(h) + (1 - \lambda) \cdot \mathrm{HeSS}_\text{pc}(h)$$

where the $\lambda$ hyperparameter controls the emphasis between global and local geometric criteria.

Figure 1: Overview of the calibration and inference pipeline for HeSS-guided sparsification in VGGT, including the error-based computation of per-head HeSS and adaptive attention masking at inference.

The calibration process relies on a small set of scenes, over which the necessary gradients are computed once and retained throughout deployment. The empirical results confirm that HeSS distributions are highly non-uniform and head-specific.

Figure 2: HeSS distributions across GA layers and heads in VGGT, with darker colors signaling greater head sensitivity.

HeSS-Guided Sparsification

At inference, standard block-sparse attention budgets (i.e., per-head quotas of active attention blocks) are supplanted by HeSS-weighted allocations. Concretely, the total attention budget for a GA layer is redistributed among heads based on their relative HeSS importance. An iterative budget capping procedure ensures that no head is oversupplied and excess allocation is fairly redistributed to uncapped heads, analogous to water-filling in information theory.

Figure 3: Redistribution of attention budgets via HeSS, showing the iterative capping and surplus redistribution mechanism for practical enforceability.

This scheme enables sparsification to aggressively target robust, low-sensitivity heads, while sparing high-sensitivity heads—dramatically mitigating the typical trade-off between compute reduction and task performance.

Experiments

Sensitivity Analysis and Visualization

Empirical analysis of HeSS distributions reveals that within each GA layer, sensitivity is concentrated in a minority of heads, with some heads contributing almost exclusively to global geometry and others to local geometry.

Quantitative and Qualitative Benchmarks

HeSS-Guided Sparsification is benchmarked on challenging 3D vision datasets—CO3Dv2 (for camera pose) and DTU (for MVS/point cloud accuracy). Both quantitative and qualitative assessments are presented.

Figure 4: Comparative performance between the proposed method and SparseVGGT across sparsity levels on CO3Dv2 and DTU, evaluated via AUC@30, ATE, and Chamfer Distance.

At high sparsity, HeSS-guided allocation retains geometric fidelity and camera pose accuracy at levels substantially higher than SparseVGGT or pruning-based ViT baselines.

Figure 5: Error visualization on DTU, with green points indicating regions where 3D error exceeds 5 mm; HeSS-guided sparsification maintains fewer outlier points compared to uniform sparsification.

Performance collapse is directly observed when the HeSS ranking is inverted, providing strong evidence that HeSS correctly identifies crucial heads for geometry consistency.

Figure 6: Catastrophic failure of geometry when pruning is guided by inverted HeSS ranking (green), versus sustained accuracy with original HeSS ordering (red).

Ablation and Design Studies

Key ablations include: the necessity of budget reallocation versus naive proportional allocation, the role of loss weighting $\lambda$ (HeSS$_{\text{cam}}$ versus HeSS$_{\text{pc}}$), and the impact of computing FIM traces w.r.t. query versus key projection matrices. The combined use of both error measures and query weights achieves optimal performance.

Figure 7: Impact of varying $\lambda$ in HeSS; joint weighting strikes the best balance across sparsity regimes.

Runtime and Generalization

HeSS-guided sparsification introduces negligible runtime costs compared to SparseVGGT, and the pipeline generalizes effectively to other VGGT-style architectures, such as $\pi^3$, where head-wise sensitivity patterns remain predictive and the method maintains its advantage.

Theoretical and Practical Implications

HeSS reframes sparsification as a sensitivity-aware resource allocation problem, rather than a uniform masking task. This approach clarifies that the criticality of particular attention heads must be established with respect to end-task errors, rather than attention weights alone. The resulting sparsification regimes demonstrate that efficient execution and geometric accuracy are not inherently at odds if sparsification is properly structured. From a theoretical perspective, this places an explicit link between second-order loss landscape geometry and model/architectural interpretability for structured sparsification.

Pragmatically, the method drastically improves the performance-vs-compute trade-off for transformer-based 3D vision models—specifically in settings where inference cost and memory are bottlenecks. This is directly relevant to robotics, AR/VR, and any low-latency 3D reasoning pipeline.

Limitations and Future Directions

HeSS operates with a per-layer uniformity assumption, not accounting for inter-layer differences in Fisher Information scale. The metric is not directly comparable across layers, suggesting future work in normalizing or standardizing FIM traces for truly global pruning. Additionally, the focus is strictly on inference-time masking; incorporating sensitivity awareness into the training objective (e.g., through robust learning under simulated sparsification) constitutes a promising extension.

Conclusion

This paper introduces the Head Sensitivity Score (HeSS), a scalable, error-driven metric for guiding sparsity redistribution in the VGGT transformer for multi-view 3D vision. By grounding sparsification in explicit measurement of per-head task impact, the proposed two-stage HeSS pipeline outperforms uniform and naive pruning baselines by significant margins, especially under aggressive sparsity constraints, without incurring extra inference costs. This work advances both the interpretability and efficiency of transformer-based vision architectures and offers a blueprint for second-order-aware resource allocation in future AI systems.