HeSS-Guided Sparsification in VGGT

• The paper presents a two-stage, head-aware block-sparse attention method that computes a Head Sensitivity Score (HeSS) offline to guide per-head budget allocation.
• It quantifies sensitivity via empirical Fisher Information from camera pose and point cloud errors, enabling non-uniform sparsification in global attention layers.
• Experiments show improved 3D reconstruction metrics and robustness at higher sparsity levels while maintaining runtime efficiency compared to uniform methods.

Searching arXiv for the primary HeSS paper and closely related sparsification work to ground the article. HeSS-Guided Sparsification is a two-stage, head-aware block-sparse attention method for the global attention layers of Visual Geometry Grounded Transformer (VGGT). It is motivated by the observation that existing sparsification methods often apply a uniform sparsity pattern across all heads, even though heads do not respond equally to masking. The method therefore quantifies head-wise sparsification sensitivity through the Head Sensitivity Score (HeSS), computed offline on a calibration set, and then uses the pre-computed score at inference time to redistribute a fixed total attention budget so that sensitive heads receive denser attention and more robust heads receive more aggressive sparsity (Kim et al., 26 Mar 2026).

1. Problem setting and motivation

HeSS-Guided Sparsification targets VGGT’s Global Attention (GA) layers, whose computational cost is quadratic in the number of tokens. Prior sparsification-based accelerators alleviate this cost, but the reported failure mode is substantial accuracy degradation under higher sparsity. The method attributes that degradation to heterogeneity in head-wise sparsification sensitivity: some attention heads carry critical global geometric information, whereas others are comparatively robust and can tolerate heavier masking (Kim et al., 26 Mar 2026).

The principal baseline discussed in this context is SparseVGGT. SparseVGGT uses block-sparse attention by selecting attention blocks from an approximate attention map and then applying the same hyperparameters $\tau$ (CDF threshold) and $\rho$ (sparse ratio) to all heads. HeSS-Guided Sparsification departs from that uniform treatment. Its central claim is that uniformly masking all heads “overly sparsifies sensitive ones,” and that this is a primary cause of the observed accuracy collapse at higher sparsity (Kim et al., 26 Mar 2026).

Two empirical observations motivate the method. First, HeSS visualizations are reported to be highly non-uniform across heads and layers, with only a few heads exhibiting very high sensitivity. Second, reverse ranking catastrophically fails: pruning the most sensitive heads first produces severe performance collapse. This combination of evidence motivates head-specific budget allocation rather than layer-wide uniform sparsity (Kim et al., 26 Mar 2026).

2. Head Sensitivity Score

HeSS quantifies how sensitive each attention head $h$ is to sparsification. The score is computed with respect to the head’s query projection parameters $W_Q^h$, using curvature of two task-specific error terms on a calibration set. The two terms are intended to capture complementary aspects of 3D vision: camera pose error $e_{\text{cam}}$, associated with global scene structure, and point cloud error $e_{\text{pc}}$, associated with fine local geometry (Kim et al., 26 Mar 2026).

The camera pose error is defined after aligning predicted and ground-truth point clouds with a similarity transform $\mathbf{H}$ obtained by Umeyama + ICP, with stop-gradient on $\mathbf{H}$:

$$e_{\text{cam}} = \frac{1}{2N} \sum_{i=1}^{N} \left\| \operatorname{sg}(\mathbf{H}) \hat{\mathbf{t}}_i - \mathbf{t}_i \right\|_2^2 .$$

The point cloud error uses an inlier set defined by a threshold $\epsilon$:

$\rho$0

and then

$\rho$1

The implementation reports $\rho$2 (Kim et al., 26 Mar 2026).

The method adopts a second-order sensitivity view. For a head parameter vector $\rho$3, the supplementary gives the local expansion

$\rho$4

where $\rho$5 and $\rho$6. Invoking an Optimal Brain Damage style argument, the method assumes that near a good solution gradients are small, so curvature dominates sparsification sensitivity (Kim et al., 26 Mar 2026).

Direct Hessian computation is treated as impractical, so HeSS uses the empirical Fisher Information Matrix (FIM) as a proxy. For each head and each error type,

$\rho$7

$\rho$8

with $\rho$9. The empirical form is

$h$0

where $h$1 is the calibration subset (Kim et al., 26 Mar 2026).

Because these matrices are head-specific and scale-dependent, the method scalarizes them by the trace and normalizes within each GA layer:

$h$2

$h$3

and then combines them as

$h$4

with $h$5. The default is $h$6, although the paper notes that $h$7 works best with $h$8 (Kim et al., 26 Mar 2026).

3. Two-stage pipeline

The method is explicitly organized into an offline calibration stage and an inference-time sparsification stage. The first computes HeSS once and stores it; the second reuses those scores to guide block-sparse masking (Kim et al., 26 Mar 2026).

Stage	Main operation	Output
Offline calibration	Compute gradients w.r.t. $h$9, estimate $W_Q^h$0 and $W_Q^h$1, derive normalized traces	Fixed $W_Q^h$2 values
Inference-time sparsification	Use HeSS to redistribute the total attention budget within each GA layer	Per-head budgets for block-sparse attention

In the offline stage, the procedure is: select a small calibration set, run forward and backward passes to compute gradients with respect to $W_Q^h$3, estimate the two empirical Fisher matrices, and combine their normalized traces into $W_Q^h$4. The supplementary specifies that this one-time calibration uses 40 CO3Dv2 scenes with 20 views each, requiring 80 backward passes total because there are two error terms per scene. The resulting HeSS profile is then fixed for all later inference (Kim et al., 26 Mar 2026).

The inference stage retains SparseVGGT’s block-sparse attention machinery but replaces uniform per-head sparsity with HeSS-guided reallocation. A common misconception is that HeSS-Guided Sparsification is a training-time sparse optimization method; in the formulation reported here, it is instead an offline sensitivity estimation plus inference-time budget redistribution procedure. This distinguishes it from sparse retraining schemes and dynamic sparse training algorithms (Kim et al., 26 Mar 2026).

4. Budget redistribution and masking rule

The method redistributes a layer’s total attention budget in three steps. It begins from the uniform baseline per-head block budget $W_Q^h$5 used by SparseVGGT and sums over all heads:

$W_Q^h$6

where $W_Q^h$7 is the number of heads in a GA layer (Kim et al., 26 Mar 2026).

Next, it converts HeSS values into normalized per-head weights

$W_Q^h$8

and defines the ideal per-head budget

$W_Q^h$9

Heads with larger HeSS therefore receive denser attention (Kim et al., 26 Mar 2026).

Since some $e_{\text{cam}}$0 may exceed the maximum feasible budget for one head, the method applies an iterative capping rule described as Water-Filling-style. Overflowing heads are identified by

$e_{\text{cam}}$1

then capped by

$e_{\text{cam}}$2

with surplus

$e_{\text{cam}}$3

That surplus is redistributed among uncapped heads in proportion to their HeSS weights, and the process is repeated until $e_{\text{cam}}$4 (Kim et al., 26 Mar 2026).

The actual sparsification remains block-based and is inherited from SparseVGGT. For each head, the query and key tensors are average-pooled into blocks of size $e_{\text{cam}}$5, approximate block attention is computed as

$e_{\text{cam}}$6

blocks are sorted by probability, and only the top blocks according to the assigned budget $e_{\text{cam}}$7 are kept. For fairness, the method does not mask attention involving camera tokens and register tokens, matching SparseVGGT. The procedure applies to GA layers rather than frame-attention layers, reflecting the claim that GA layers are both the primary quadratic bottleneck and a key locus of long-range scene reasoning (Kim et al., 26 Mar 2026).

5. Empirical behavior, ablations, and robustness

The reported experiments support three principal claims. First, head sensitivity is heterogeneous. HeSS visualizations show that only a few heads in each GA layer have high sensitivity, while many heads are much less sensitive. Some heads also exhibit task preference, correlating more strongly with camera pose error or with point cloud error. This is presented as evidence that head-wise treatment is necessary (Kim et al., 26 Mar 2026).

Second, the method is reported to be robust across sparsity levels. Relative to SparseVGGT, HeSS-Guided Sparsification yields fewer geometric outliers in DTU visualizations and consistently better CO3Dv2 camera pose metrics and DTU reconstruction metrics, with the advantage becoming clearer as sparsity rises. One cited high-sparsity case at sparsity $e_{\text{cam}}$8 reports DTU Chamfer 1.9287 for SparseVGGT versus 1.6030 for HeSS-Guided Sparsification, and CO3Dv2 ATE 0.2527 versus 0.2184 at the same setting (Kim et al., 26 Mar 2026).

Third, the method is reported to preserve runtime relative to SparseVGGT. At sparsity $e_{\text{cam}}$9, the runtimes are 8.42 s vs. 8.37 s; at sparsity $e_{\text{pc}}$0, 6.59 s vs. 6.58 s. This suggests that the benefit is attributed to more effective allocation of an existing sparse-attention budget rather than to a change in the basic sparse-attention execution path (Kim et al., 26 Mar 2026).

The ablations clarify several design choices. Computing HeSS with respect to $e_{\text{pc}}$1 performs better than using $e_{\text{pc}}$2 or $e_{\text{pc}}$3, supporting the use of query projections as the sensitivity proxy. Normalizing each FIM trace by the sum over heads before averaging performs better than using raw trace values. For the interpolation weight, $e_{\text{pc}}$4 is best overall for VGGT, while $e_{\text{pc}}$5 prefers $e_{\text{pc}}$6. Removing iterative budget reallocation degrades performance, indicating that the capping and redistribution step is functionally necessary rather than cosmetic (Kim et al., 26 Mar 2026).

The supplementary also reports that calibration choice is fairly robust: HeSS profiles from DTU and CO3Dv2 are interchangeable with minimal performance loss. This suggests that the calibration procedure is capturing an intrinsic sensitivity profile rather than narrowly overfitting to one dataset. Reported practical settings include $e_{\text{pc}}$7 for the point-cloud inlier threshold, discarding low-confidence VGGT points with confidence $e_{\text{pc}}$8, a default $e_{\text{pc}}$9, a custom CUDA kernel for head-wise and block-wise masking, use of the same $\mathbf{H}$0 and $\mathbf{H}$1 as SparseVGGT for fair comparison, experiments on NVIDIA RTX A40 GPUs, and calibration on the CO3Dv2 dev split with 20 views per scene (Kim et al., 26 Mar 2026).

6. Relation to adjacent sparsification methods and scope

HeSS-Guided Sparsification belongs to a distinct part of the sparsification design space. Its object of optimization is head-wise attention budget allocation in GA layers, and its sensitivity signal is derived from an empirical Fisher approximation to Hessian information over 3D task errors. This distinguishes it from Hessian-guided weight pruning methods such as SparseForge, which optimize a continuous soft mask for semi-structured $\mathbf{H}$2 LLM sparsity and argue that improving mask quality is more effective than merely increasing retraining tokens (Hanzuo et al., 7 May 2026).

It also differs from always-sparse dynamic sparse training methods such as guided stochastic exploration (GSE), which maintain a sparse active set throughout training, periodically prune low-magnitude active weights, and grow new connections from a randomly sampled inactive subset using gradient magnitude as the guide. In that setting, the emphasis is on sparse training efficiency without ever materializing dense weights or dense gradients, rather than on inference-time reallocation of a fixed attention budget across transformer heads (Heddes et al., 2024).

These distinctions help delimit the scope of the term. In the VGGT setting, HeSS-Guided Sparsification is not a generic sparse training algorithm, not a one-shot pruning rule, and not a universal transformer head-pruning heuristic. The paper explicitly reports that generic ViT sparsification baselines such as Michel et al. and Kwon et al. perform worse at the same sparsity, especially on 3D reconstruction tasks, and attributes that gap to the absence of the geometric bias embedded in the camera-pose and point-cloud sensitivity signals (Kim et al., 26 Mar 2026).

A further misconception addressed by the results is that uniform sparsity is adequate so long as the global layer budget is fixed. The reverse-ranking experiment and the non-uniform HeSS visualizations are presented as counterevidence. The operative claim is instead that a fixed total budget can be used more effectively if it is redistributed according to measured head sensitivity. Under that interpretation, HeSS-Guided Sparsification is best understood as a sensitivity-driven allocation rule layered on top of existing block-sparse attention machinery, rather than as a replacement for that machinery itself (Kim et al., 26 Mar 2026).