Spectrum-Aware Batch Selection

• Spectrum-aware batch selection is a method that uses spectral properties—such as graph Laplacian eigenvalues and singular values—to define and select diverse, informative mini-batch subsets.
• It employs techniques like SALN and GSTDS to balance between deterministic ranking and stochastic sampling, optimizing both within-batch quality and cross-batch diversity.
• These methods have been applied in supervised image classification, contrastive learning, and sensor selection, showing improvements in training speed, gradient control, and overall model performance.

Spectrum-aware batch selection denotes a family of subset-selection procedures in which the retained elements of a mini-batch, token batch, candidate pool, or sensor set are chosen from spectral structure rather than by uniform sampling alone. In the works considered here, the relevant spectrum may be the eigenstructure of a graph Laplacian, the singular spectrum of a batch embedding, the effective rank of a batch second-moment matrix, or the smallest singular value of a selected sensing matrix. The common objective is to retain informative, diverse, or well-conditioned subsets while reducing redundant computation, with reported applications in supervised image classification, sparse autoencoders, contrastive learning, and dynamic spectrum sensing (Sharifi, 2024, Sharifi et al., 6 Jul 2025, Oozeer et al., 29 Aug 2025, Jha et al., 19 Aug 2025, Ochieng, 7 Oct 2025, Joneidi et al., 2018).

1. Spectral notions underlying selection

A central premise in the graph-based formulations is that deep networks train faster and generalize better when each mini-batch is both informative and diverse. Spectral graph theory supplies a compact description of that structure through the Laplacian of a similarity graph built on batch samples. If $S$ is an affinity matrix and $L=D-S$ the corresponding Laplacian, then the second smallest eigenvalue $\lambda_2$ measures algebraic connectivity and the associated eigenvector gives a one-dimensional embedding that exposes a natural partition of the batch. In SALN, points with extreme Fiedler coordinates are treated as especially important because they tend to be “bridges” or “outliers” critical for preserving graph connectivity (Sharifi, 2024). GSTDS adopts the same Laplacian construction and uses the Fiedler vector as its per-sample scoring signal (Sharifi et al., 6 Jul 2025).

A different spectral interpretation appears in low-rank and anisotropy-based methods. GRAFT projects a batch to a rank-$r$ subspace via SVD or PCA and then selects rows whose induced submatrix has high volume, so that the chosen points span the dominant subspace of the batch (Jha et al., 19 Aug 2025). In contrastive learning, the relevant quantity is the batch second-moment matrix $\hat\Sigma$, whose eigenvalues determine anisotropy; the effective rank

$$R_{\mathrm{eff}}(B)=\frac{\left(\sum_i \lambda_i\right)^2}{\sum_i \lambda_i^2}=\frac{1}{\sum_i \lambda_i^2}$$

serves as a proxy for isotropy, and increasing $R_{\mathrm{eff}}$ lowers the top eigenvalue that enters the gradient-norm upper bound (Ochieng, 7 Oct 2025). In sparse autoencoders, “spectrum-aware” is used in a looser but still distributional sense: Sampled-SAE scores feature columns of the batch activation matrix by $L_2$ norm or entropy and varies a pool-expansion factor $\ell$ to interpolate between globally shared and token-specific feature selection (Oozeer et al., 29 Aug 2025).

This range of usages suggests that the term “spectrum” is not tied to a single operator. Rather, it indexes whichever spectral summary best captures redundancy, diversity, or conditioning in the task at hand.

2. Laplacian-based mini-batch filtering

SALN, introduced in “Optimizing Data Curation through Spectral Analysis and Joint Batch Selection,” begins from a batch $B=\{v_1,\dots,v_n\}\subset \mathbb{R}^d$ of feature vectors and defines a cosine-similarity matrix

$L=D-S$0

With degree matrix $L=D-S$1, $L=D-S$2, SALN forms the unnormalized Laplacian $L=D-S$3, solves the eigenproblem, sorts eigenvalues $L=D-S$4, and uses the Fiedler vector $L=D-S$5 to score samples by

$L=D-S$6

The retained subset size is

$L=D-S$7

and the top-$L=D-S$8 samples by absolute Fiedler value are kept (Sharifi, 2024).

The interpretation supplied for this heuristic is explicitly geometric. Extreme Fiedler coordinates are taken to identify under-represented modes and boundary points in the current batch. Within the SALN formulation, this is the mechanism by which diversity and informativeness are coupled: the score is not a confidence score or a loss score, but a graph-connectivity score (Sharifi, 2024).

GSTDS uses an almost identical graph construction but embeds it in a more explicit training curriculum. Each sample $L=D-S$9 in a batch is mapped to a frozen ResNet-50 feature $\lambda_2$0, cosine similarities define an adjacency matrix $\lambda_2$1, the degree matrix satisfies $\lambda_2$2, and the Laplacian is $\lambda_2$3. The Fiedler vector $\lambda_2$4 is the eigenvector of $\lambda_2$5 associated with $\lambda_2$6, equivalently the minimizer of the Rayleigh quotient

$\lambda_2$7

GSTDS then assigns score $\lambda_2$8, sorts samples by descending $\lambda_2$9, and applies a pre-scheduled filtering ratio $r$0 generated by a sigmoid schedule

$r$1

If $r$2, it selects $r$3 highest-score samples deterministically and the remaining $r$4 by weighted random sampling with weights inverse to reference-model loss $r$5 (Sharifi et al., 6 Jul 2025).

The two methods therefore share the same spectral core but differ in how that score is operationalized. SALN is a top-$r$6 Fiedler-value filter, whereas GSTDS combines spectral exploitation with exploration and a scheduled keep ratio.

3. Joint, dynamic, and cross-batch coordination

SALN goes beyond per-batch filtering by coupling selection across multiple consecutive mini-batches. For batches $r$7, each of size $r$8, it introduces binary decisions $r$9 and solves

$\hat\Sigma$0

subject to

$\hat\Sigma$1

for every $\hat\Sigma$2. Here $\hat\Sigma$3 is the batchwise Fiedler score, $\hat\Sigma$4 is a cross-batch similarity such as cosine similarity, and $\hat\Sigma$5 trades off within-batch informativeness against across-batch diversity (Sharifi, 2024). The paper states that this is a binary quadratic program and that, in practice, SALN uses a greedy or relaxation-and-rounding heuristic.

GSTDS is dynamic in a different sense. Its schedule is indexed over the full epoch-batch trajectory $\hat\Sigma$6, so the retained fraction changes smoothly over training. The schedule is chosen so that its average value is e.g. $\hat\Sigma$7, its minimum $\hat\Sigma$8, and maximum $\hat\Sigma$9. This yields an explicit curriculum in which increasingly large or small fractions of each batch can be preserved as training proceeds, while the within-batch ranking remains spectral (Sharifi et al., 6 Jul 2025).

GRAFT introduces a third form of adaptivity: dynamic subset-size control by gradient approximation. After extracting a low-rank feature matrix $$R_{\mathrm{eff}}(B)=\frac{\left(\sum_i \lambda_i\right)^2}{\sum_i \lambda_i^2}=\frac{1}{\sum_i \lambda_i^2}$$0, GRAFT chooses the smallest candidate rank $$R_{\mathrm{eff}}(B)=\frac{\left(\sum_i \lambda_i\right)^2}{\sum_i \lambda_i^2}=\frac{1}{\sum_i \lambda_i^2}$$1 satisfying

$$R_{\mathrm{eff}}(B)=\frac{\left(\sum_i \lambda_i\right)^2}{\sum_i \lambda_i^2}=\frac{1}{\sum_i \lambda_i^2}$$2

where $$R_{\mathrm{eff}}(B)=\frac{\left(\sum_i \lambda_i\right)^2}{\sum_i \lambda_i^2}=\frac{1}{\sum_i \lambda_i^2}$$3 is the full-batch gradient and $$R_{\mathrm{eff}}(B)=\frac{\left(\sum_i \lambda_i\right)^2}{\sum_i \lambda_i^2}=\frac{1}{\sum_i \lambda_i^2}$$4 projects onto the span of the selected gradients. The selected subset size is therefore not fixed solely by a filter ratio or top-$$R_{\mathrm{eff}}(B)=\frac{\left(\sum_i \lambda_i\right)^2}{\sum_i \lambda_i^2}=\frac{1}{\sum_i \lambda_i^2}$$5 rule; it is the minimal size that keeps the gradient projection error below threshold (Jha et al., 19 Aug 2025).

Taken together, these formulations show that spectrum-aware selection can act at multiple levels: within a batch, across a window of batches, or through adaptive control of subset size itself.

4. Distribution-aware and low-rank selection beyond graph Laplacians

Sampled-SAE addresses a different selection problem: sparse latent allocation across a batch of token activations. Let $$R_{\mathrm{eff}}(B)=\frac{\left(\sum_i \lambda_i\right)^2}{\sum_i \lambda_i^2}=\frac{1}{\sum_i \lambda_i^2}$$6 be the pre-activation matrix. The method scores columns rather than samples, using either

$$R_{\mathrm{eff}}(B)=\frac{\left(\sum_i \lambda_i\right)^2}{\sum_i \lambda_i^2}=\frac{1}{\sum_i \lambda_i^2}$$7

or the entropy score obtained by first normalizing each column,

$$R_{\mathrm{eff}}(B)=\frac{\left(\sum_i \lambda_i\right)^2}{\sum_i \lambda_i^2}=\frac{1}{\sum_i \lambda_i^2}$$8

and then computing

$$R_{\mathrm{eff}}(B)=\frac{\left(\sum_i \lambda_i\right)^2}{\sum_i \lambda_i^2}=\frac{1}{\sum_i \lambda_i^2}$$9

Given target sparsity $R_{\mathrm{eff}}$0 and pool-expansion factor $R_{\mathrm{eff}}$1, it forms a global pool of size $R_{\mathrm{eff}}$2 from the highest-scoring columns and only then performs per-token Top-$R_{\mathrm{eff}}$3 selection inside that restricted pool (Oozeer et al., 29 Aug 2025).

The key analytical claim is that $R_{\mathrm{eff}}$4 interpolates between global and token-specific behavior. At $R_{\mathrm{eff}}$5, only $R_{\mathrm{eff}}$6 features compete for all tokens and selection is fully global; at $R_{\mathrm{eff}}$7, the pool becomes all features and the method recovers BatchTopK. Intermediate values trade off shared structure and token-specific reconstruction. The paper states that no single value optimizes $R_{\mathrm{eff}}$8 across all metrics on Pythia-160M; the best choice depends on the trade-off between shared structure, reconstruction fidelity, and downstream performance (Oozeer et al., 29 Aug 2025). This directly counters the misconception that a single spectral bottleneck should dominate all objectives simultaneously.

GRAFT is likewise not Laplacian-based. For a batch matrix $R_{\mathrm{eff}}$9, it computes a thin SVD,

$L_2$0

retains the top $L_2$1 singular components, and defines a compact embedding $L_2$2. It then selects $L_2$3 rows whose induced $L_2$4 submatrix maximizes absolute volume,

$L_2$5

Because exact optimization is combinatorial, GRAFT uses a fast greedy MaxVol routine, justified by a determinant factorization through Sylvester’s determinant identity (Jha et al., 19 Aug 2025).

Both methods are spectrum-aware without using graph partitions. Sampled-SAE uses column-score distributions to regulate feature competition across a batch, while GRAFT uses singular directions and subspace volume to ensure that the selected examples span dominant low-rank structure.

5. Effective rank, isotropy, and contrastive-learning batches

In contrastive learning, the spectral target shifts from connectivity or subspace spanning to gradient control through anisotropy. “Diversity Is All You Need for Contrastive Learning: Spectral Bounds on Gradient Magnitudes” considers $L_2$6-normalized embeddings $L_2$7, pairwise cosine scores $L_2$8, temperature $L_2$9, and InfoNCE loss. The batch second-moment matrix

$\ell$0

and the negatives-only covariance $\ell$1 enter non-asymptotic upper and lower bounds on the squared gradient norm. The paper states that these bounds recover the $\ell$2 law and that, empirically, a log-log fit yields slope $\ell$3 (Ochieng, 7 Oct 2025).

The effective rank

$\ell$4

is then used as an anisotropy proxy. When the spectrum is concentrated, $\ell$5; when it is isotropic, $\ell$6. Since $\ell$7, increasing $\ell$8 lowers the top eigenvalue and tightens the spectral-band ceiling on the gradient magnitude (Ochieng, 7 Oct 2025).

On that basis the paper proposes two selection procedures. Pool-P3 evaluates full-batch proposals from a host-side pool $\ell$9, computing $B=\{v_1,\dots,v_n\}\subset \mathbb{R}^d$0 with $B=\{v_1,\dots,v_n\}\subset \mathbb{R}^d$1, and selects the proposal whose effective rank is closest to a target $B=\{v_1,\dots,v_n\}\subset \mathbb{R}^d$2. Greedy-$B=\{v_1,\dots,v_n\}\subset \mathbb{R}^d$3 instead builds a batch incrementally to minimize $B=\{v_1,\dots,v_n\}\subset \mathbb{R}^d$4, equivalently to maximize $B=\{v_1,\dots,v_n\}\subset \mathbb{R}^d$5, by choosing candidates with minimal

$B=\{v_1,\dots,v_n\}\subset \mathbb{R}^d$6

from a probe set of size $B=\{v_1,\dots,v_n\}\subset \mathbb{R}^d$7 (Ochieng, 7 Oct 2025).

The same work also studies in-batch whitening. After forming $B=\{v_1,\dots,v_n\}\subset \mathbb{R}^d$8, adding $B=\{v_1,\dots,v_n\}\subset \mathbb{R}^d$9, and whitening via $L=D-S$00, the embeddings become nearly isotropic, with $L=D-S$01. The resulting variance bound on squared gradient norms contains an anisotropy term that dominates at ImageNet-1k scale; whitening drives $L=D-S$02, and the empirical toggle experiment reports whitened variance $L=D-S$03 raw, or raw/white $L=D-S$04 (Ochieng, 7 Oct 2025). This suggests that spectrum-aware batch construction and spectrum-aware within-batch normalization are closely related interventions.

6. Conditioning, empirical record, and practical trade-offs

An important antecedent to learning-based mini-batch selection is E-optimal sensor selection for dynamic spectrum sensing. In that setting, one observes an underdetermined linear model

$L=D-S$05

selects $L=D-S$06 sensors, and seeks a reduced sensing matrix $L=D-S$07 that is well conditioned. The E-optimal criterion is

$L=D-S$08

The paper connects this to the lower Restricted Isometry Property constants and argues that maximizing the minimum singular value controls sparse-recovery stability. In the dynamic version, reliability feedback produces the regularized objective

$L=D-S$09

where $L=D-S$10 is sensor unreliability derived from stale-data discounting and prediction error (Joneidi et al., 2018). A plausible implication is that later neural-network batch-selection methods inherit not only spectral language but also the older emphasis on conditioning and diversity through eigenvalue control.

The empirical results reported across the surveyed works are heterogeneous because the tasks differ, but several patterns recur. On Oxford-IIIT Pet, SALN reports standard training at test accuracy $L=D-S$11 and time $L=D-S$12 min, versus SALN at test accuracy $L=D-S$13 and time $L=D-S$14 min; JEST is reported at test accuracy $L=D-S$15 and time $L=D-S$16 min, leading the paper to state that SALN runs $L=D-S$17 faster than JEST while matching or slightly under-shooting its accuracy (Sharifi, 2024). SALN is also reported to attain up to an $L=D-S$18 reduction in training wall-clock time and overall test-accuracy gains up to $L=D-S$19 pp over standard mini-batch SGD (Sharifi, 2024).

GSTDS reports after 25 epochs that, on CIFAR-10 with batch size 128, standard training uses $L=D-S$20 FLOPs at test accuracy $L=D-S$21, while GSTDS uses $L=D-S$22 FLOPs at test accuracy $L=D-S$23, and JEST uses $L=D-S$24 FLOPs at test accuracy $L=D-S$25. On Oxford-IIIT Pet, standard training is reported at $L=D-S$26 FLOPs and $L=D-S$27, GSTDS at $L=D-S$28 FLOPs and $L=D-S$29, and JEST at $L=D-S$30 FLOPs and $L=D-S$31. On Oxford-Flowers 102, GSTDS is reported at $L=D-S$32 FLOPs and $L=D-S$33, compared with $L=D-S$34 FLOPs and $L=D-S$35 for standard training (Sharifi et al., 6 Jul 2025).

Sampled-SAE reports that on Pythia-160M, BatchTopK attains best $L=D-S$36 but low density $L=D-S$37, moderate probing $L=D-S$38, and absorption $L=D-S$39, whereas $L=D-S$40-norm scoring with $L=D-S$41 gives $L=D-S$42, density $L=D-S$43, probing $L=D-S$44, absorption $L=D-S$45, and $L=D-S$46 (Oozeer et al., 29 Aug 2025). The paper states that small $L=D-S$47 drastically boosts the count of consistently active features and improves concept-detection while only modestly hurting reconstruction. This is a direct example of a recurrent trade-off in spectrum-aware selection: better global structure or diversity need not coincide with best reconstruction fidelity.

GRAFT reports that across CIFAR-10, TinyImageNet, Caltech256 and BERT on IMDB, it matches or exceeds GradMatch, CRAIG, and GLISTER in test accuracy while reducing wall-clock time by up to $L=D-S$48–$L=D-S$49, using only $L=D-S$50–$L=D-S$51 of the samples per batch to achieve at least $L=D-S$52 of full-batch accuracy, and reducing $L=D-S$53 emissions by $L=D-S$54–$L=D-S$55 kg per run at equal fidelity (Jha et al., 19 Aug 2025). The contrastive-learning work reports that on ImageNet-100, Greedy-64 cuts time-to-$L=D-S$56 top-1 by $L=D-S$57 versus random and by $L=D-S$58 versus Pool-P3 at equal accuracy; CIFAR-10 shows similar relative gains (Ochieng, 7 Oct 2025).

Practical guidance is likewise method-specific. SALN states that filter_ratio between $L=D-S$59–$L=D-S$60 works well, that a joint-batch window $L=D-S$61–$L=D-S$62 balances cross-batch diversity without large quadratic programs, and that $L=D-S$63 in $L=D-S$64 generally suffices to break ties in dense regions (Sharifi, 2024). Sampled-SAE recommends $L=D-S$65–$L=D-S$66 for maximal feature sharing, $L=D-S$67–$L=D-S$68 as a sweet spot for many interpretability goals, and $L=D-S$69 when pure reconstruction fidelity is paramount (Oozeer et al., 29 Aug 2025). GRAFT gives a rule-of-thumb spectral rank choice $L=D-S$70, suggests $L=D-S$71 values in $L=D-S$72, and recommends recomputing $L=D-S$73 every $L=D-S$74–$L=D-S$75 iterations (Jha et al., 19 Aug 2025). The contrastive-learning work notes that Pool-P3 incurs host-side overhead, Greedy-$L=D-S$76 adds $L=D-S$77 on-GPU cost, and careful target-rank tuning is required for P3, with warm-up under P1 in early training (Ochieng, 7 Oct 2025).

Across these lines of work, the central technical theme is stable: spectral quantities are used as proxies for batch informativeness, cross-sample diversity, anisotropy control, or matrix conditioning. What varies is the object whose spectrum is measured, the optimization problem built around it, and the downstream criterion—accuracy, reconstruction, probing, convergence speed, or sparse-recovery reliability—that the selected subset is meant to preserve.