---
title: Hessian-Aware Salient Data Selection
url: https://www.emergentmind.com/topics/hessian-aware-salient-data-selection
type: topic
---

# Hessian-Aware Salient Data Selection

Searching arXiv for the cited papers to ground the article in current literature.
Hessian-aware Salient Data Selection denotes a family of data selection procedures in which the saliency of examples, measurements, rows, prompts, timesteps, or sparse points is defined through second-order structure, typically a Hessian, Gauss–Newton Hessian, or a curvature proxy such as a Gram matrix. Across recent work, the common objective is not merely to reduce dataset size, but to preserve or optimize the aspects of training or inference most controlled by curvature: validation sensitivity under a compute budget, strict positive definiteness of a sketched Gauss–Newton Hessian, subspace geometry relevant to second-order solvers, calibration sensitivity in post-training quantization, or curvature consistency in geometric prediction [2510.16806], [2409.15906], [2102.12317], [2508.04016], [2108.11098].

## 1. Conceptual scope and defining criteria

In the most general sense, salient data are those samples whose inclusion most affects a target objective through second-order structure. The precise meaning of saliency depends on the application. In compute-budgeted training, salient samples are those whose inclusion most reduces validation loss under a fixed budget, with the budget altering the desired data quantity, quality, and distribution [2510.16806]. In least-squares inverse problems, salient measurements are those whose associated Jacobian rows preserve the strict positive definiteness of the Gauss–Newton Hessian near a global minimizer [2409.15906]. In learned Hessian sketching, salient rows are those with large leverage scores because they dominate the geometry of the column space and hence the Hessian curvature governing second-order updates [2102.12317]. In video diffusion post-training quantization, salient calibration pairs are prompt–timestep instances that are simultaneously informative for diffusion dynamics and sensitive to quantization under a Gauss–Newton approximation [2508.04016]. In monocular depth estimation, salient points are sparse guiding or learned locations that prime a lightweight architecture, while a normalized Hessian loss enforces curvature consistency [2108.11098].

This diversity of formulations reflects a shared principle: second-order information is used to prioritize data not by frequency or loss alone, but by its effect on optimization geometry, identifiability, or sensitivity. A plausible implication is that “Hessian-aware” is better understood as a design pattern than as a single algorithmic recipe.

## 2. Mathematical formulations of saliency

Several mathematical objects recur across the literature. In least-squares settings with residual vector $r(x)\in\mathbb{R}^m$, objective
$$
f(x)=\tfrac{1}{2}\|r(x)\|_2^2,
$$
and Jacobian
$$
J(x)=\frac{\partial r(x)}{\partial x}\in\mathbb{R}^{m\times d},
$$
the Gauss–Newton Hessian is
$$
H_{GN}(x)=J(x)^\top J(x).
$$
Strict positivity at the global minimizer $x_*$ is equivalent to
$$
\lambda_{\min}(J(x_*)^\top J(x_*))>0,
$$
or equivalently full column rank of $J(x_*)$ [2409.15906]. In this setting, salient data correspond to row subsets or sketches that preserve $J^\top J$ in positive semidefinite order:
$$
(1-\epsilon)J^\top J \preceq J^\top S^\top S J \preceq (1+\epsilon)J^\top J,
$$
which immediately yields
$$
\lambda_{\min}(\tilde H)\ge (1-\epsilon)\lambda_{\min}(H_{GN}).
$$

In compute-budget-aware training, the formal bilevel structure makes the Hessian appear through implicit differentiation. With selection parameters $s$, model parameters $w$, and budget $B$, the inner problem is written as
$$
w^*(s)=\arg\min_w L_{\text{train}}(w;s)\quad \text{s.t.}\quad C(w,s)\le B,
$$
and the outer problem is
$$
\min_s \mathcal{L}_{\text{val}}(w^*(s)).
$$
Classical differentiation gives
$$
\frac{dw^*}{ds}=-H_w^{-1}\frac{\partial^2 L_{\text{train}}(w;s)}{\partial w\,\partial s},
\qquad
H_w=\frac{\partial^2 L_{\text{train}}(w;s)}{\partial w^2},
$$
which is precisely why Hessian estimation becomes a bottleneck [2510.16806].

In tall-matrix least squares, leverage scores give a row-level saliency notion. For $A\in\mathbb{R}^{n\times d}$ with full column rank,
$$
\tau_i(A)=\|e_i^\top A(A^\top A)^{-1}A^\top\|_2^2,
$$
or, if $A=U\Sigma V^\top$,
$$
\tau_i(A)=\|e_i^\top U\|_2^2.
$$
For least squares with Hessian $H=A^\top A$, the standard leverage score of row $a_i^\top$ is
$$
h_i=a_i^\top (A^\top A)^\dagger a_i.
$$
High-leverage rows are salient because they determine the column-space directions that drive second-order updates [2102.12317].

In video diffusion quantization, the relevant curvature proxy is activation-induced. For calibration data $D_{\text{calib}}$,
$$
\mathcal{L}_{\text{quant}}=\mathbb{E}_{X\sim D_{\text{calib}}}\Big[\|\theta^f(x)-\theta^q(x)\|^2\Big].
$$
Under a Taylor approximation with $\hat W=W+\Delta$, $\mathrm{g}^X=0$, and a Levenberg–Marquardt or Gauss–Newton approximation,
$$
\mathrm{H}^X\approx \mathbb{E}[2X^\top X],
$$
so that quantization error is controlled by the Gram matrix $X^\top X$. The paper therefore defines
$$
C_{\text{quant}}(x_t)=\|x_t^\top x_t\|_2
$$
and diffusion salience
$$
C_{\text{diff}}(x_t)=\frac{\|x_t-x_{t-1}\|^2}{\|x_t\|^2},
$$
followed by min–max normalization and a product score
$$
C_{\text{sample}}(x_t)=\overline{C}_{\text{diff}}(x_t)\cdot \overline{C}_{\text{quant}}(x_t),
$$
which selects candidates only when both terms are high [2508.04016].

In depth estimation, Hessian awareness appears in the loss rather than the sampling score. For depth map $d(x,y)$,
$$
\nabla^2 d=(d_{xx},d_{xy},d_{yy}),
\qquad
H_d(x,y)=\frac{\nabla^2 d(x,y)}{\|\nabla^2 d(x,y)\|_2+\varepsilon},
$$
with $\varepsilon=10^{-20}$. The normalized Hessian loss is
$$
\mathcal{L}_H(\hat d,d)=\sqrt{\frac{1}{N}\sum_{x,y}\|H_{\hat d}(x,y)-H_d(x,y)\|_2^2},
$$
and is invariant to the generalized bas-relief transform
$$
d'(x,y)=\alpha d(x,y)+\beta+ax+by
$$
because $\nabla^2 d'=\alpha \nabla^2 d$ and the normalization cancels $\alpha$ [2108.11098].

## 3. Algorithmic mechanisms

Despite sharing a curvature-based motivation, Hessian-aware salient data selection methods differ sharply in how they avoid explicit Hessian construction.

In compute-budget-aware data selection, CADS formulates selection as a stochastic policy over subsets and replaces direct Hessian inversion with a Hessian-free score-function estimator. At the example level, the sampling distribution is Bernoulli:
$$
p(\mathbf m\mid \mathbf s)=\prod_{i=1}^N s_i^{m_i}(1-s_i)^{1-m_i},
$$
with score
$$
\frac{\partial}{\partial s_i}\log p(\mathbf m\mid \mathbf s)=\frac{m_i}{s_i}-\frac{1-m_i}{1-s_i}.
$$
The outer gradient becomes
$$
\nabla_{\mathbf s}\,\mathbb{E}_{p(\mathbf m|\mathbf s)}[\mathcal{L}_{\mathrm{val}}(\theta_C(\mathbf m))]
=
\mathbb{E}_{p(\mathbf m|\mathbf s)}
\big[
\mathcal{L}_{\mathrm{val}}(\theta_C(\mathbf m))\,\nabla_{\mathbf s}\log p(\mathbf m|\mathbf s)
\big],
$$
and a self-critical baseline may be subtracted for variance reduction [2510.16806]. CADS further relaxes the bilevel objective with a penalty term and replaces repeated inner solves by a one-dimensional surrogate $l(|\mathbf m|)$, the compute-constrained reachable training loss as a function of subset size.

In inverse problems, the central mechanism is randomized sketching of the Jacobian. A sampling matrix $S\in\mathbb{R}^{s\times m}$ selects rows with replacement according to probabilities $p_i$ and rescales by $w_i=1/\sqrt{s p_i}$. The resulting sketched Hessian
$$
\tilde H=J^\top S^\top S J
$$
preserves positive definiteness if the sampling distribution is chosen appropriately. The proposed distribution is proportional to local sensitivity:
$$
\pi(u)\propto \|G_*(u)\|_2^2,
$$
where $G_*(u)=\partial F(u,p)/\partial p|_{p=p_*}$ [2409.15906]. Because evaluating exact gradients may be expensive, the paper integrates gradient-free sampling procedures, specifically EKS and CBS, to draw measurements according to $\pi$ without exact Jacobian computation.

In learned Hessian sketches, the main mechanism is to keep predicted heavy rows exactly and sketch only the remainder. Up to permutation,
$$
A=\begin{pmatrix}A_I\\A_{I^c}\end{pmatrix},
\qquad
S=\begin{pmatrix}I&0\\0&S'\end{pmatrix},
$$
where $I$ is a set of rows predicted by an oracle to have large leverage scores and $S'$ is a Count-Sketch on the complement. The method also learns sketch values by minimizing a subspace-embedding loss,
$$
\mathcal{L}(S,A_i)=\left\|(A_iR_i^{-1})^\top A_iR_i^{-1}-I\right\|_F,
$$
with $SA_i=Q_iR_i$ [2102.12317]. The saliency notion is thus operationalized as oracle-guided preservation of heavy rows.

In video diffusion PTQ, H-SDS is a ranking-and-selection method over candidate prompt–timestep pairs. For each candidate, it computes $C_{\text{diff}}$, $C_{\text{quant}}$, normalizes them across the pool, and selects the top-$K$ scores by
$$
C_{\text{sample}}(x_t)=\overline{C}_{\text{diff}}(x_t)\cdot \overline{C}_{\text{quant}}(x_t).
$$
The selected calibration pairs are then used in block-wise PTQ, while a separate Attention-guided Sparse Token Distillation stage reweights token losses using attention salience
$$
S_j=\sum_{h,i}A_{h,i,j},
\qquad
\lambda_j=
\frac{S_j-\min(S)}{\max(S)-\min(S)}(\lambda_{\max}-\lambda_{\min})+\lambda_{\min}
$$
with recommended $\lambda_{\min}=0.5$ and $\lambda_{\max}=1$ [2508.04016].

In monocular depth estimation, the selection mechanism is architectural. SIFT keypoints provide sparse 3D guiding points during training, while a Saliency-Net predicts salient points from RGB, fusion features, coarse depth, and a learned confidence map. These sparse points are used to guide a second pass through Fusion-Net; the normalized Hessian loss then enforces curvature consistency over the dense prediction [2108.11098].

## 4. Principal application domains

The term encompasses at least five distinct application regimes, each with a different operational meaning of “data.”

| Domain | Selected entity | Hessian-aware signal |
|---|---|---|
| Compute-budgeted training | Examples or sources | Formal bilevel Hessian; Hessian-free policy gradient [2510.16806] |
| Least-squares inverse problems | Measurements or sensor locations | Gauss–Newton Hessian positivity via $J^\top S^\top S J$ [2409.15906] |
| Second-order convex optimization | Matrix rows in sketches | Leverage scores and learned subspace embeddings [2102.12317] |
| Video diffusion PTQ | Prompt–timestep calibration pairs | $C_{\text{quant}}(x_t)=\|x_t^\top x_t\|_2$ [2508.04016] |
| Monocular depth estimation | Sparse guiding or salient points | Normalized Hessian loss over depth curvature [2108.11098] |

In compute-budgeted data selection, the central claim is that compute budget should be a first-class constraint because different budgets require different data quantity, quality, and distribution. The selected subset is not fixed across budgets; small budgets favor cleaner, more focused subsets, whereas larger budgets benefit from richer and more diverse information [2510.16806].

In inverse problems, saliency is tied to reconstructability. Measurements are salient if their gradients with respect to the unknown parameter contribute strongly to the positive definiteness of the Gauss–Newton Hessian, thereby preserving local convexity and identifiability [2409.15906].

In sketch-based optimization, saliency is tied to leverage. Heavy rows are preserved exactly because they determine the important directions of the Hessian, enabling smaller sketches and faster convergence in Iterative Hessian Sketch and related solvers [2102.12317].

In video diffusion PTQ, saliency is constrained by the calibration bottleneck induced by long spatial-temporal sequences. The selected calibration set must simultaneously capture denoising-stage informativeness and curvature relevant to quantization error [2508.04016].

In monocular depth estimation, the data selection component is not a dataset reduction method but a sparse geometric priming mechanism. This suggests that the phrase “salient data selection” can refer either to selecting from a global candidate pool or to constructing sparse, informative internal supervision within a training example.

## 5. Theoretical guarantees and empirical behavior

The inverse-problem literature provides the clearest explicit positivity guarantee. Under the assumptions that $y=F(p_*)$, that the local Hessian is uniformly continuous near $p_*$, and that the full-data Hessian is strictly positive definite, the sampling theorem states that if $\pi(u)\ge \beta\tilde\pi(u)$ with $\tilde\pi(u)\propto \|G_*(u)\|_2^2$, then for any $\delta\in(0,1)$ and $\vartheta\in(0,\lambda_{\min}(H_{pC}(p_*)))$, choosing
$$
c \ge \|G_*\|_F^4 \cdot \frac{\left(1+\sqrt{8\beta^{-1}\log(\delta^{-1})}\right)^2}{\beta(\lambda_{\min}(H_{pC}(p_*))-\vartheta)^2}
$$
ensures, with probability at least $1-\delta$, that
$$
\lambda_{\min}(H_{pC_c}(p_*))\ge \vartheta>0
$$
[2409.15906]. This is a direct theoretical statement that salient measurement selection can preserve strict positivity of the sketched Gauss–Newton Hessian.

The learned-sketch literature provides embedding and convergence guarantees rather than direct sample-selection guarantees. With an oracle predicting the rows above threshold $\nu=\epsilon/d$, the sketch can achieve
$$
m=O\!\left(\frac{d}{\epsilon^2}\left(\mathrm{polylog}\left(\frac{1}{\epsilon}\right)+\log\frac{1}{\delta}\right)\right)
$$
rows while preserving subspace geometry, and this constitutes a quadratic improvement in $d$ over oblivious Count-Sketch in the stated regime [2102.12317].

Empirically, CADS reports that no algorithm consistently dominates across budgets and that sophisticated selectors can be outperformed by random selection when budget changes. Against this background, CADS reports performance gains of up to 14.42% over baselines in vision and language benchmarks and empirical 3–20× speedup versus conventional bilevel implementations, with larger accelerations at larger compute budgets [2510.16806]. On MNIST with budget 20k sample usages and initial subset sizes 200–800, average accuracy is reported as Random 89.83, PBCS 91.89, CADS-E 92.25, and Bilevel-CADS 92.80. On CIFAR-10 with five sources and label noise from 0–90%, CADS-S reports 63.08 average accuracy versus 61.65 for Best-source and 51.13 for Full-dataset. On instruction tuning with GPT-2 and budget $10^4$–$5\times 10^4$, CADS-S matches or beats full-data perplexity across budgets [2510.16806].

In the Schrödinger potential reconstruction experiments for positivity-preserving measurement selection, the full dataset with $N_x=30$ produces $N=(N_x-1)^2=841$ interior points and a full-data Hessian with $\lambda_{\min}\approx 70.21$. With budget $c=18$ sensors, initial normal sampling yields $\lambda_{\min}\approx 1.13\times 10^{-2}$, while Greedy EKS reaches $\lambda_{\min}\approx 86.33$ and Greedy CBS reaches $\lambda_{\min}\approx 142.39$. Under uniform initial sampling, repeated uniform resampling yields $\lambda_{\min}\approx 238.69$ [2409.15906]. The observation that a carefully chosen down-sampled set can exceed the full-data $\lambda_{\min}$ is explicitly reported.

In learned Hessian sketches, the empirical claims are similarly task-specific. On GHG, with $m=6d$, learned values reduce the convergence rate to 56% of Count-Sketch, while heavy-row selection reduces it to 86.9%; with $m=10d$, learned values reach 63.7% and heavy-row selection 82.1%. On Electric, the combined learned variant achieves convergence rates as low as 21.1% and 15.4% of sparse JL. On Tunnel, learned values attain rates 48% and 29% of random sketches for $m=7d$ and $m=10d$ [2102.12317].

In video diffusion PTQ, the empirical role of H-SDS is to reduce calibration variance and improve robustness under limited calibration budgets. With 40 calibration samples on CogVideoX-2B W4A4, SDS yields IQ $52.95\pm 0.69$ and is reported to provide consistently higher scene consistency with lower variance than random baselines. Under W4A6 quantization, $\text{S}^2$Q-VDiT reports lossless performance with $3.9\times$ model compression and $1.3\times$ inference acceleration [2508.04016]. On CogVideoX-5B W4A6, model storage is reduced from 10.375 GB to 2.633 GB, inference memory from 15.801 GB to 10.145 GB, and runtime from 259.2s to 203.2s.

In monocular depth estimation, the sparse saliency and normalized Hessian loss are tied to accuracy–efficiency tradeoffs. FuSaNet reports 8.1M parameters and achieves REL 0.104, RMSE 0.403, and $\delta_1$ 0.915 on NYU-Depth-v2, as well as REL 0.059, RMSE 2.487, and $\delta_1$ 0.964 on KITTI [2108.11098]. Ablations attribute clear gains to Saliency-Net, the Confidence Predictor, and the normalized Hessian loss.

## 6. Limitations, misconceptions, and open directions

A common misconception is that Hessian-aware selection necessarily requires explicit Hessian construction or inversion. The surveyed work contradicts this. CADS is explicitly motivated by the expense and unreliability of explicit Hessian estimation under budget-constrained inner training and therefore uses a Hessian-free policy-gradient estimator [2510.16806]. H-SDS for video diffusion does not compute the exact model Hessian and instead uses the proxy $\mathrm{H}^X\approx \mathbb{E}[2X^\top X]$ [2508.04016]. Gradient-free EKS and CBS are used in inverse problems precisely because exact gradient or Jacobian evaluation may be too costly [2409.15906].

A second misconception is that saliency is universal across budgets, tasks, or domains. The CADS results explicitly argue the opposite: no single selector dominates across budgets, and even random selection may outperform sophisticated methods when the budget changes [2510.16806]. Similarly, learned Hessian sketches rely on distributional stability of heavy rows; the paper notes that learned values and oracle quality may degrade under data shift [2102.12317].

The main limitations also vary by formulation. CADS inherits score-function variance and depends on the fit quality of the one-dimensional surrogate $l(|m|)$ [2510.16806]. Positivity-preserving sampling is local to $p_*$ and assumes the Gauss–Newton approximation remains accurate near the minimizer; away from that regime, guarantees must be re-established [2409.15906]. Oracle-guided Hessian sketches assume no false negatives in the heavy-row set $I\supseteq I^*$ and lose their advantage when there are too many heavy rows [2102.12317]. In video diffusion PTQ, even the improved method is not perfectly lossless at fully 4-bit W4A4 [2508.04016]. In depth estimation, benefits may diminish if saliency prediction fails in textureless regions or repeated patterns, and the small model may struggle with thin structures and micro-details [2108.11098].

Several open directions are already stated in the source materials. CADS suggests hybrid approaches that combine its compute-aware relaxation with curvature-aware scoring under large budgets [2510.16806]. The inverse-problem formulation identifies extensions to noisy data, better convergence theory for ensemble sampling, and links to A- and D-optimal design under sketching as active issues [2409.15906]. The video diffusion work suggests improved curvature models, diversity-aware selection beyond top-$K$, and lightweight selective fine-tuning [2508.04016]. Taken together, these directions indicate that Hessian-aware salient data selection is evolving from isolated task-specific heuristics into a broader set of second-order data curation methods whose central question is how much curvature must be preserved, approximated, or targeted for a given computational objective.

Source: https://www.emergentmind.com/topics/hessian-aware-salient-data-selection