Hessian-Aware Salient Data Selection

• Hessian-aware Salient Data Selection is a method that defines data importance using second-order curvature (via Hessians, Gauss–Newton, or Gram matrices) to enhance training objectives.
• It strategically selects data across domains such as compute-budgeted training, inverse problems, video diffusion quantization, and monocular depth estimation based on curvature sensitivity.
• Empirical studies verify that preserving training curvature through these methods improves efficiency and model performance while providing guarantees like Hessian positivity and optimized subspace geometry.

Searching arXiv for the cited papers to ground the article in current literature. Hessian-aware Salient Data Selection denotes a family of data selection procedures in which the saliency of examples, measurements, rows, prompts, timesteps, or sparse points is defined through second-order structure, typically a Hessian, Gauss–Newton Hessian, or a curvature proxy such as a Gram matrix. Across recent work, the common objective is not merely to reduce dataset size, but to preserve or optimize the aspects of training or inference most controlled by curvature: validation sensitivity under a compute budget, strict positive definiteness of a sketched Gauss–Newton Hessian, subspace geometry relevant to second-order solvers, calibration sensitivity in post-training quantization, or curvature consistency in geometric prediction (Wan et al., 19 Oct 2025, Hellmuth et al., 2024, Li et al., 2021, Feng et al., 6 Aug 2025, Huynh et al., 2021).

1. Conceptual scope and defining criteria

In the most general sense, salient data are those samples whose inclusion most affects a target objective through second-order structure. The precise meaning of saliency depends on the application. In compute-budgeted training, salient samples are those whose inclusion most reduces validation loss under a fixed budget, with the budget altering the desired data quantity, quality, and distribution (Wan et al., 19 Oct 2025). In least-squares inverse problems, salient measurements are those whose associated Jacobian rows preserve the strict positive definiteness of the Gauss–Newton Hessian near a global minimizer (Hellmuth et al., 2024). In learned Hessian sketching, salient rows are those with large leverage scores because they dominate the geometry of the column space and hence the Hessian curvature governing second-order updates (Li et al., 2021). In video diffusion post-training quantization, salient calibration pairs are prompt–timestep instances that are simultaneously informative for diffusion dynamics and sensitive to quantization under a Gauss–Newton approximation (Feng et al., 6 Aug 2025). In monocular depth estimation, salient points are sparse guiding or learned locations that prime a lightweight architecture, while a normalized Hessian loss enforces curvature consistency (Huynh et al., 2021).

This diversity of formulations reflects a shared principle: second-order information is used to prioritize data not by frequency or loss alone, but by its effect on optimization geometry, identifiability, or sensitivity. A plausible implication is that “Hessian-aware” is better understood as a design pattern than as a single algorithmic recipe.

2. Mathematical formulations of saliency

Several mathematical objects recur across the literature. In least-squares settings with residual vector $r(x)\in\mathbb{R}^m$, objective

$f(x)=\tfrac{1}{2}\|r(x)\|_2^2,$

and Jacobian

$$J(x)=\frac{\partial r(x)}{\partial x}\in\mathbb{R}^{m\times d},$$

the Gauss–Newton Hessian is

$H_{GN}(x)=J(x)^\top J(x).$

Strict positivity at the global minimizer $x_*$ is equivalent to

$\lambda_{\min}(J(x_*)^\top J(x_*))>0,$

or equivalently full column rank of $J(x_*)$ (Hellmuth et al., 2024). In this setting, salient data correspond to row subsets or sketches that preserve $J^\top J$ in positive semidefinite order:

$$(1-\epsilon)J^\top J \preceq J^\top S^\top S J \preceq (1+\epsilon)J^\top J,$$

which immediately yields

$$\lambda_{\min}(\tilde H)\ge (1-\epsilon)\lambda_{\min}(H_{GN}).$$

In compute-budget-aware training, the formal bilevel structure makes the Hessian appear through implicit differentiation. With selection parameters $f(x)=\tfrac{1}{2}\|r(x)\|_2^2,$0, model parameters $f(x)=\tfrac{1}{2}\|r(x)\|_2^2,$1, and budget $f(x)=\tfrac{1}{2}\|r(x)\|_2^2,$2, the inner problem is written as

$f(x)=\tfrac{1}{2}\|r(x)\|_2^2,$3

and the outer problem is

$f(x)=\tfrac{1}{2}\|r(x)\|_2^2,$4

Classical differentiation gives

$f(x)=\tfrac{1}{2}\|r(x)\|_2^2,$5

which is precisely why Hessian estimation becomes a bottleneck (Wan et al., 19 Oct 2025).

In tall-matrix least squares, leverage scores give a row-level saliency notion. For $f(x)=\tfrac{1}{2}\|r(x)\|_2^2,$6 with full column rank,

$f(x)=\tfrac{1}{2}\|r(x)\|_2^2,$7

or, if $f(x)=\tfrac{1}{2}\|r(x)\|_2^2,$8,

$f(x)=\tfrac{1}{2}\|r(x)\|_2^2,$9

For least squares with Hessian $$J(x)=\frac{\partial r(x)}{\partial x}\in\mathbb{R}^{m\times d},$$0, the standard leverage score of row $$J(x)=\frac{\partial r(x)}{\partial x}\in\mathbb{R}^{m\times d},$$1 is

$$J(x)=\frac{\partial r(x)}{\partial x}\in\mathbb{R}^{m\times d},$$2

High-leverage rows are salient because they determine the column-space directions that drive second-order updates (Li et al., 2021).

In video diffusion quantization, the relevant curvature proxy is activation-induced. For calibration data $$J(x)=\frac{\partial r(x)}{\partial x}\in\mathbb{R}^{m\times d},$$3,

$$J(x)=\frac{\partial r(x)}{\partial x}\in\mathbb{R}^{m\times d},$$4

Under a Taylor approximation with $$J(x)=\frac{\partial r(x)}{\partial x}\in\mathbb{R}^{m\times d},$$5, $$J(x)=\frac{\partial r(x)}{\partial x}\in\mathbb{R}^{m\times d},$$6, and a Levenberg–Marquardt or Gauss–Newton approximation,

$$J(x)=\frac{\partial r(x)}{\partial x}\in\mathbb{R}^{m\times d},$$7

so that quantization error is controlled by the Gram matrix $$J(x)=\frac{\partial r(x)}{\partial x}\in\mathbb{R}^{m\times d},$$8. The paper therefore defines

$$J(x)=\frac{\partial r(x)}{\partial x}\in\mathbb{R}^{m\times d},$$9

and diffusion salience

$H_{GN}(x)=J(x)^\top J(x).$0

followed by min–max normalization and a product score

$H_{GN}(x)=J(x)^\top J(x).$1

which selects candidates only when both terms are high (Feng et al., 6 Aug 2025).

In depth estimation, Hessian awareness appears in the loss rather than the sampling score. For depth map $H_{GN}(x)=J(x)^\top J(x).$2,

$H_{GN}(x)=J(x)^\top J(x).$3

with $H_{GN}(x)=J(x)^\top J(x).$4. The normalized Hessian loss is

$H_{GN}(x)=J(x)^\top J(x).$5

and is invariant to the generalized bas-relief transform

$H_{GN}(x)=J(x)^\top J(x).$6

because $H_{GN}(x)=J(x)^\top J(x).$7 and the normalization cancels $H_{GN}(x)=J(x)^\top J(x).$8 (Huynh et al., 2021).

3. Algorithmic mechanisms

Despite sharing a curvature-based motivation, Hessian-aware salient data selection methods differ sharply in how they avoid explicit Hessian construction.

In compute-budget-aware data selection, CADS formulates selection as a stochastic policy over subsets and replaces direct Hessian inversion with a Hessian-free score-function estimator. At the example level, the sampling distribution is Bernoulli:

$H_{GN}(x)=J(x)^\top J(x).$9

with score

$x_*$0

The outer gradient becomes

$x_*$1

and a self-critical baseline may be subtracted for variance reduction (Wan et al., 19 Oct 2025). CADS further relaxes the bilevel objective with a penalty term and replaces repeated inner solves by a one-dimensional surrogate $x_*$2, the compute-constrained reachable training loss as a function of subset size.

In inverse problems, the central mechanism is randomized sketching of the Jacobian. A sampling matrix $x_*$3 selects rows with replacement according to probabilities $x_*$4 and rescales by $x_*$5. The resulting sketched Hessian

$x_*$6

preserves positive definiteness if the sampling distribution is chosen appropriately. The proposed distribution is proportional to local sensitivity:

$x_*$7

where $x_*$8 (Hellmuth et al., 2024). Because evaluating exact gradients may be expensive, the paper integrates gradient-free sampling procedures, specifically EKS and CBS, to draw measurements according to $x_*$9 without exact Jacobian computation.

In learned Hessian sketches, the main mechanism is to keep predicted heavy rows exactly and sketch only the remainder. Up to permutation,

$\lambda_{\min}(J(x_*)^\top J(x_*))>0,$0

where $\lambda_{\min}(J(x_*)^\top J(x_*))>0,$1 is a set of rows predicted by an oracle to have large leverage scores and $\lambda_{\min}(J(x_*)^\top J(x_*))>0,$2 is a Count-Sketch on the complement. The method also learns sketch values by minimizing a subspace-embedding loss,

$\lambda_{\min}(J(x_*)^\top J(x_*))>0,$3

with $\lambda_{\min}(J(x_*)^\top J(x_*))>0,$4 (Li et al., 2021). The saliency notion is thus operationalized as oracle-guided preservation of heavy rows.

In video diffusion PTQ, H-SDS is a ranking-and-selection method over candidate prompt–timestep pairs. For each candidate, it computes $\lambda_{\min}(J(x_*)^\top J(x_*))>0,$5, $\lambda_{\min}(J(x_*)^\top J(x_*))>0,$6, normalizes them across the pool, and selects the top-$\lambda_{\min}(J(x_*)^\top J(x_*))>0,$7 scores by

$\lambda_{\min}(J(x_*)^\top J(x_*))>0,$8

The selected calibration pairs are then used in block-wise PTQ, while a separate Attention-guided Sparse Token Distillation stage reweights token losses using attention salience

$\lambda_{\min}(J(x_*)^\top J(x_*))>0,$9

with recommended $J(x_*)$0 and $J(x_*)$1 (Feng et al., 6 Aug 2025).

In monocular depth estimation, the selection mechanism is architectural. SIFT keypoints provide sparse 3D guiding points during training, while a Saliency-Net predicts salient points from RGB, fusion features, coarse depth, and a learned confidence map. These sparse points are used to guide a second pass through Fusion-Net; the normalized Hessian loss then enforces curvature consistency over the dense prediction (Huynh et al., 2021).

4. Principal application domains

The term encompasses at least five distinct application regimes, each with a different operational meaning of “data.”

Domain	Selected entity	Hessian-aware signal
Compute-budgeted training	Examples or sources	Formal bilevel Hessian; Hessian-free policy gradient (Wan et al., 19 Oct 2025)
Least-squares inverse problems	Measurements or sensor locations	Gauss–Newton Hessian positivity via $J(x_*)$2 (Hellmuth et al., 2024)
Second-order convex optimization	Matrix rows in sketches	Leverage scores and learned subspace embeddings (Li et al., 2021)
Video diffusion PTQ	Prompt–timestep calibration pairs	$J(x_*)$3 (Feng et al., 6 Aug 2025)
Monocular depth estimation	Sparse guiding or salient points	Normalized Hessian loss over depth curvature (Huynh et al., 2021)

In compute-budgeted data selection, the central claim is that compute budget should be a first-class constraint because different budgets require different data quantity, quality, and distribution. The selected subset is not fixed across budgets; small budgets favor cleaner, more focused subsets, whereas larger budgets benefit from richer and more diverse information (Wan et al., 19 Oct 2025).

In inverse problems, saliency is tied to reconstructability. Measurements are salient if their gradients with respect to the unknown parameter contribute strongly to the positive definiteness of the Gauss–Newton Hessian, thereby preserving local convexity and identifiability (Hellmuth et al., 2024).

In sketch-based optimization, saliency is tied to leverage. Heavy rows are preserved exactly because they determine the important directions of the Hessian, enabling smaller sketches and faster convergence in Iterative Hessian Sketch and related solvers (Li et al., 2021).

In video diffusion PTQ, saliency is constrained by the calibration bottleneck induced by long spatial-temporal sequences. The selected calibration set must simultaneously capture denoising-stage informativeness and curvature relevant to quantization error (Feng et al., 6 Aug 2025).

In monocular depth estimation, the data selection component is not a dataset reduction method but a sparse geometric priming mechanism. This suggests that the phrase “salient data selection” can refer either to selecting from a global candidate pool or to constructing sparse, informative internal supervision within a training example.

5. Theoretical guarantees and empirical behavior

The inverse-problem literature provides the clearest explicit positivity guarantee. Under the assumptions that $J(x_*)$4, that the local Hessian is uniformly continuous near $J(x_*)$5, and that the full-data Hessian is strictly positive definite, the sampling theorem states that if $J(x_*)$6 with $J(x_*)$7, then for any $J(x_*)$8 and $J(x_*)$9, choosing

$J^\top J$0

ensures, with probability at least $J^\top J$1, that

$J^\top J$2

(Hellmuth et al., 2024). This is a direct theoretical statement that salient measurement selection can preserve strict positivity of the sketched Gauss–Newton Hessian.

The learned-sketch literature provides embedding and convergence guarantees rather than direct sample-selection guarantees. With an oracle predicting the rows above threshold $J^\top J$3, the sketch can achieve

$J^\top J$4

rows while preserving subspace geometry, and this constitutes a quadratic improvement in $J^\top J$5 over oblivious Count-Sketch in the stated regime (Li et al., 2021).

Empirically, CADS reports that no algorithm consistently dominates across budgets and that sophisticated selectors can be outperformed by random selection when budget changes. Against this background, CADS reports performance gains of up to 14.42% over baselines in vision and language benchmarks and empirical 3–20× speedup versus conventional bilevel implementations, with larger accelerations at larger compute budgets (Wan et al., 19 Oct 2025). On MNIST with budget 20k sample usages and initial subset sizes 200–800, average accuracy is reported as Random 89.83, PBCS 91.89, CADS-E 92.25, and Bilevel-CADS 92.80. On CIFAR-10 with five sources and label noise from 0–90%, CADS-S reports 63.08 average accuracy versus 61.65 for Best-source and 51.13 for Full-dataset. On instruction tuning with GPT-2 and budget $J^\top J$6–$J^\top J$7, CADS-S matches or beats full-data perplexity across budgets (Wan et al., 19 Oct 2025).

In the Schrödinger potential reconstruction experiments for positivity-preserving measurement selection, the full dataset with $J^\top J$8 produces $J^\top J$9 interior points and a full-data Hessian with $$(1-\epsilon)J^\top J \preceq J^\top S^\top S J \preceq (1+\epsilon)J^\top J,$$0. With budget $$(1-\epsilon)J^\top J \preceq J^\top S^\top S J \preceq (1+\epsilon)J^\top J,$$1 sensors, initial normal sampling yields $$(1-\epsilon)J^\top J \preceq J^\top S^\top S J \preceq (1+\epsilon)J^\top J,$$2, while Greedy EKS reaches $$(1-\epsilon)J^\top J \preceq J^\top S^\top S J \preceq (1+\epsilon)J^\top J,$$3 and Greedy CBS reaches $$(1-\epsilon)J^\top J \preceq J^\top S^\top S J \preceq (1+\epsilon)J^\top J,$$4. Under uniform initial sampling, repeated uniform resampling yields $$(1-\epsilon)J^\top J \preceq J^\top S^\top S J \preceq (1+\epsilon)J^\top J,$$5 (Hellmuth et al., 2024). The observation that a carefully chosen down-sampled set can exceed the full-data $$(1-\epsilon)J^\top J \preceq J^\top S^\top S J \preceq (1+\epsilon)J^\top J,$$6 is explicitly reported.

In learned Hessian sketches, the empirical claims are similarly task-specific. On GHG, with $$(1-\epsilon)J^\top J \preceq J^\top S^\top S J \preceq (1+\epsilon)J^\top J,$$7, learned values reduce the convergence rate to 56% of Count-Sketch, while heavy-row selection reduces it to 86.9%; with $$(1-\epsilon)J^\top J \preceq J^\top S^\top S J \preceq (1+\epsilon)J^\top J,$$8, learned values reach 63.7% and heavy-row selection 82.1%. On Electric, the combined learned variant achieves convergence rates as low as 21.1% and 15.4% of sparse JL. On Tunnel, learned values attain rates 48% and 29% of random sketches for $$(1-\epsilon)J^\top J \preceq J^\top S^\top S J \preceq (1+\epsilon)J^\top J,$$9 and $$\lambda_{\min}(\tilde H)\ge (1-\epsilon)\lambda_{\min}(H_{GN}).$$0 (Li et al., 2021).

In video diffusion PTQ, the empirical role of H-SDS is to reduce calibration variance and improve robustness under limited calibration budgets. With 40 calibration samples on CogVideoX-2B W4A4, SDS yields IQ $$\lambda_{\min}(\tilde H)\ge (1-\epsilon)\lambda_{\min}(H_{GN}).$$1 and is reported to provide consistently higher scene consistency with lower variance than random baselines. Under W4A6 quantization, $$\lambda_{\min}(\tilde H)\ge (1-\epsilon)\lambda_{\min}(H_{GN}).$$2Q-VDiT reports lossless performance with $$\lambda_{\min}(\tilde H)\ge (1-\epsilon)\lambda_{\min}(H_{GN}).$$3 model compression and $$\lambda_{\min}(\tilde H)\ge (1-\epsilon)\lambda_{\min}(H_{GN}).$$4 inference acceleration (Feng et al., 6 Aug 2025). On CogVideoX-5B W4A6, model storage is reduced from 10.375 GB to 2.633 GB, inference memory from 15.801 GB to 10.145 GB, and runtime from 259.2s to 203.2s.

In monocular depth estimation, the sparse saliency and normalized Hessian loss are tied to accuracy–efficiency tradeoffs. FuSaNet reports 8.1M parameters and achieves REL 0.104, RMSE 0.403, and $$\lambda_{\min}(\tilde H)\ge (1-\epsilon)\lambda_{\min}(H_{GN}).$$5 0.915 on NYU-Depth-v2, as well as REL 0.059, RMSE 2.487, and $$\lambda_{\min}(\tilde H)\ge (1-\epsilon)\lambda_{\min}(H_{GN}).$$6 0.964 on KITTI (Huynh et al., 2021). Ablations attribute clear gains to Saliency-Net, the Confidence Predictor, and the normalized Hessian loss.

6. Limitations, misconceptions, and open directions

A common misconception is that Hessian-aware selection necessarily requires explicit Hessian construction or inversion. The surveyed work contradicts this. CADS is explicitly motivated by the expense and unreliability of explicit Hessian estimation under budget-constrained inner training and therefore uses a Hessian-free policy-gradient estimator (Wan et al., 19 Oct 2025). H-SDS for video diffusion does not compute the exact model Hessian and instead uses the proxy $$\lambda_{\min}(\tilde H)\ge (1-\epsilon)\lambda_{\min}(H_{GN}).$$7 (Feng et al., 6 Aug 2025). Gradient-free EKS and CBS are used in inverse problems precisely because exact gradient or Jacobian evaluation may be too costly (Hellmuth et al., 2024).

A second misconception is that saliency is universal across budgets, tasks, or domains. The CADS results explicitly argue the opposite: no single selector dominates across budgets, and even random selection may outperform sophisticated methods when the budget changes (Wan et al., 19 Oct 2025). Similarly, learned Hessian sketches rely on distributional stability of heavy rows; the paper notes that learned values and oracle quality may degrade under data shift (Li et al., 2021).

The main limitations also vary by formulation. CADS inherits score-function variance and depends on the fit quality of the one-dimensional surrogate $$\lambda_{\min}(\tilde H)\ge (1-\epsilon)\lambda_{\min}(H_{GN}).$$8 (Wan et al., 19 Oct 2025). Positivity-preserving sampling is local to $$\lambda_{\min}(\tilde H)\ge (1-\epsilon)\lambda_{\min}(H_{GN}).$$9 and assumes the Gauss–Newton approximation remains accurate near the minimizer; away from that regime, guarantees must be re-established (Hellmuth et al., 2024). Oracle-guided Hessian sketches assume no false negatives in the heavy-row set $f(x)=\tfrac{1}{2}\|r(x)\|_2^2,$00 and lose their advantage when there are too many heavy rows (Li et al., 2021). In video diffusion PTQ, even the improved method is not perfectly lossless at fully 4-bit W4A4 (Feng et al., 6 Aug 2025). In depth estimation, benefits may diminish if saliency prediction fails in textureless regions or repeated patterns, and the small model may struggle with thin structures and micro-details (Huynh et al., 2021).

Several open directions are already stated in the source materials. CADS suggests hybrid approaches that combine its compute-aware relaxation with curvature-aware scoring under large budgets (Wan et al., 19 Oct 2025). The inverse-problem formulation identifies extensions to noisy data, better convergence theory for ensemble sampling, and links to A- and D-optimal design under sketching as active issues (Hellmuth et al., 2024). The video diffusion work suggests improved curvature models, diversity-aware selection beyond top-$f(x)=\tfrac{1}{2}\|r(x)\|_2^2,$01, and lightweight selective fine-tuning (Feng et al., 6 Aug 2025). Taken together, these directions indicate that Hessian-aware salient data selection is evolving from isolated task-specific heuristics into a broader set of second-order data curation methods whose central question is how much curvature must be preserved, approximated, or targeted for a given computational objective.