Gradient Alignment in Optimization

• Gradient alignment is a technique that measures the directional agreement of gradients using metrics like cosine similarity, inner products, and geometric projections.
• It has broad applications including supervised classification, federated learning, PINNs, and multi-task optimization, improving convergence and mitigating noise.
• Recent algorithmic developments leverage gradient alignment for adaptive learning rates, data selection, and mitigating domain-specific conflicts to boost performance.

Gradient alignment denotes a family of optimization and analysis techniques in which the directional agreement of gradients is treated as a primary object. Across supervised classification, domain adaptation, distributed and federated learning, reinforcement learning, multi-task optimization, physics-informed neural networks, and geometric analyses of gradient descent, the central question is whether different gradients point in compatible directions and how that compatibility should be measured, enforced, or exploited. Depending on the setting, alignment is defined between per-sample gradients and model weights, between task or domain gradients, between client gradients, between training and validation gradients, or between optimization trajectories and distinguished geometric structures such as gradient extremals and the talweg (Hölzl et al., 29 Oct 2025, Dandi et al., 2021, Bégout et al., 13 Apr 2026).

1. Formal scope and canonical definitions

A common formulation treats alignment as a cosine similarity or inner product. In supervised classification, Gradient-Weight Alignment (GWA) defines the per-sample negative gradient

$g_t(x_i)=-\nabla_w \ell(f_{w_t}(x_i),y_i)$

and the per-sample alignment score

$$\gamma(x_i,w_t)=\cos\mathrm{\ sim}(g_t(x_i),w_t) =\frac{\langle g_t(x_i),w_t\rangle}{\|g_t(x_i)\|\cdot\|w_t\|}\in[-1,1].$$

At epoch $T$, GWA aggregates the empirical distribution of $\{\gamma(x_i,w_T)\}_{i=1}^N$ through

$$\mathrm{GWA}_T= \frac{\mathbb E_i[\gamma(x_i,w_T)]}{\mathrm{Kurt}[\mathcal A_T]+\beta}, \qquad \beta=1.2,$$

so that mean alignment is penalized when the alignment distribution is heavy-tailed (Hölzl et al., 29 Oct 2025).

In distributed and federated learning, the same phenomenon is expressed through gradient variance. For

$f(x)=\frac1n\sum_{i=1}^n f_i(x),$

the gradient-variance regularizer is

$$r(x)=\frac1{2n}\sum_{i=1}^n \|\nabla f_i(x)-\nabla f(x)\|^2.$$

This is equivalently one half the trace of the covariance matrix of the per-batch gradients. In that setting, gradient alignment is often measured by cosine similarity, inner products such as $\nabla f_i(x)^\top\nabla f_j(x)$, or by $1-r(x)$; small $r(x)$ means aligned (Dandi et al., 2021).

Several works generalize beyond pairwise cosine similarity. In PINNs, a scalar alignment score for nonzero vectors $$\gamma(x_i,w_t)=\cos\mathrm{\ sim}(g_t(x_i),w_t) =\frac{\langle g_t(x_i),w_t\rangle}{\|g_t(x_i)\|\cdot\|w_t\|}\in[-1,1].$$0 is defined by

$$\gamma(x_i,w_t)=\cos\mathrm{\ sim}(g_t(x_i),w_t) =\frac{\langle g_t(x_i),w_t\rangle}{\|g_t(x_i)\|\cdot\|w_t\|}\in[-1,1].$$1

which lies in $$\gamma(x_i,w_t)=\cos\mathrm{\ sim}(g_t(x_i),w_t) =\frac{\langle g_t(x_i),w_t\rangle}{\|g_t(x_i)\|\cdot\|w_t\|}\in[-1,1].$$2 and reduces to the usual cosine similarity when $$\gamma(x_i,w_t)=\cos\mathrm{\ sim}(g_t(x_i),w_t) =\frac{\langle g_t(x_i),w_t\rangle}{\|g_t(x_i)\|\cdot\|w_t\|}\in[-1,1].$$3 (Wang et al., 2 Feb 2025). In multi-distribution learning, SAGE measures pairwise cosine similarity among per-distribution gradients through

$$\gamma(x_i,w_t)=\cos\mathrm{\ sim}(g_t(x_i),w_t) =\frac{\langle g_t(x_i),w_t\rangle}{\|g_t(x_i)\|\cdot\|w_t\|}\in[-1,1].$$4

and uses the disagreement term $$\gamma(x_i,w_t)=\cos\mathrm{\ sim}(g_t(x_i),w_t) =\frac{\langle g_t(x_i),w_t\rangle}{\|g_t(x_i)\|\cdot\|w_t\|}\in[-1,1].$$5 to scale isotropic noise (Ballas et al., 8 May 2026). In reasoning distillation, IGA defines per-coordinate cross-domain variance

$$\gamma(x_i,w_t)=\cos\mathrm{\ sim}(g_t(x_i),w_t) =\frac{\langle g_t(x_i),w_t\rangle}{\|g_t(x_i)\|\cdot\|w_t\|}\in[-1,1].$$6

and a Continuous Gradient Conflict Mask

$$\gamma(x_i,w_t)=\cos\mathrm{\ sim}(g_t(x_i),w_t) =\frac{\langle g_t(x_i),w_t\rangle}{\|g_t(x_i)\|\cdot\|w_t\|}\in[-1,1].$$7

so that invariant gradient directions are preserved while high-variance coordinates are smoothly suppressed (Cheng et al., 3 Jun 2026).

The term also appears in a geometric sense. For generic gradient descent near a nondegenerate minimum, trajectories of the gradient flow and of its discrete counterpart exhibit directional alignment with the tangent spaces to gradient extremals, and generically to the talweg; in the local strong-minimizer case, the talweg is exactly the $$\gamma(x_i,w_t)=\cos\mathrm{\ sim}(g_t(x_i),w_t) =\frac{\langle g_t(x_i),w_t\rangle}{\|g_t(x_i)\|\cdot\|w_t\|}\in[-1,1].$$8 gradient-extremal (Bégout et al., 13 Apr 2026). This usage is analytically distinct from cosine-based metrics, but it retains the same core theme: optimization becomes structured when directions of descent cohere.

2. Theoretical interpretations

In separable classification, the theoretical rationale for GWA is tied to the asymptotic geometry of gradient flow on cross-entropy. In the idealized separable setting, gradient flow yields weights whose direction converges, and the sample gradients align perfectly with that direction; asymptotically $$\gamma(x_i,w_t)=\cos\mathrm{\ sim}(g_t(x_i),w_t) =\frac{\langle g_t(x_i),w_t\rangle}{\|g_t(x_i)\|\cdot\|w_t\|}\in[-1,1].$$9. The stated intuition is that high average alignment means most gradients point along the current weight vector, whereas low or negative alignment signals orthogonal or opposite updates, indicating noisy or sample-specific learning and a precursor to memorization and overfitting. The kurtosis term is introduced because a heavy-tailed alignment distribution indicates that a few outlier samples dominate the update direction (Hölzl et al., 29 Oct 2025).

In distributed and federated optimization, alignment emerges as an implicit regularization effect of small-batch SGD. Conditioned on a multiset of sampled mini-batches, the expected difference between $T$0 steps of SGD and $T$1 steps of full-batch GD is

$T$2

so SGD on $T$3 implicitly descends on the surrogate $T$4. This provides a direct theoretical explanation for why large-batch or fully parallel methods lose an alignment effect that sequential small-batch SGD retains (Dandi et al., 2021).

A distinct theoretical line shows that alignment is not sufficient by itself. SAGE derives an excess-risk decomposition in multi-distribution learning with two additive leading-order terms,

$T$5

and proves by counterexample that neither quantity bounds the other in general. The paper’s conclusion is explicit: flatness and gradient alignment are both necessary, because no algorithm targeting only one term can guarantee low excess risk (Ballas et al., 8 May 2026).

Second-order analyses reach a related conclusion from another direction. In PINNs, first-order methods are said to struggle with Mode I magnitude imbalance and Mode II directional conflict, while Newton updates promote near-perfect inter-step alignment:

$T$6

SOAP is then analyzed as a scalable quasi-Newton method whose update is asymptotically equivalent to $T$7 under the Gauss-Newton approximation, and whose per-term updates satisfy near-perfect alignment near a global zero-residual solution (Wang et al., 2 Feb 2025).

Other theoretical uses of the term are more restrictive. In deep linear networks, alignment is an invariant of squared-loss gradient descent only under necessary and sufficient simultaneous diagonalization conditions on $T$8 and $T$9; for constrained layer subspaces, alignment is impossible with sufficiently large datasets (Radhakrishnan et al., 2020). In RLHF-style alignment, the per-token gradient equals a covariance between conditional expected harm and the score function, implying zero gradient beyond the harm horizon where harmfulness is already determined; the paper concludes that standard alignment objectives cannot produce deep alignment, regardless of optimization quality (Young, 5 Mar 2026).

3. Optimization mechanisms and algorithmic constructions

A recurring pattern is to turn alignment into an efficiently computable train-time object. GWA avoids full-model per-sample gradients by using only the final linear classifier layer, where for penultimate-layer feature $\{\gamma(x_i,w_T)\}_{i=1}^N$0 and $\{\gamma(x_i,w_T)\}_{i=1}^N$1,

$\{\gamma(x_i,w_T)\}_{i=1}^N$2

The method accumulates per-sample alignments across mini-batches, updates running estimates of the first four moments online, and computes the epoch-level metric from those moments. The reported computational overhead is approximately $\{\gamma(x_i,w_T)\}_{i=1}^N$3 GFLOPs per sample versus $\{\gamma(x_i,w_T)\}_{i=1}^N$4 GFLOPs for a full forward pass of ViT/S-16, with wall-clock overhead $\{\gamma(x_i,w_T)\}_{i=1}^N$5 per epoch on ImageNet-1k (Hölzl et al., 29 Oct 2025).

Several adaptation methods use explicit two-gradient correction. GraTa defines a pseudo gradient

$\{\gamma(x_i,w_T)\}_{i=1}^N$6

and an auxiliary gradient

$\{\gamma(x_i,w_T)\}_{i=1}^N$7

then performs the auxiliary update first and minimizes the consistency loss at the shifted parameters. A first-order Taylor expansion shows that this implicitly maximizes the inner product $\{\gamma(x_i,w_T)\}_{i=1}^N$8. GraTa also sets a dynamic learning rate

$\{\gamma(x_i,w_T)\}_{i=1}^N$9

so step size increases when the two gradients agree (Chen et al., 2024).

In fully test-time adaptation, GAP uses prototype features as a proxy measure of negative impact and approximates class prototypes by the weight vectors of the classifier. Because the classifier is fixed during adaptation, gradients for entropy-minimization or pseudo-label cross-entropy can be computed in closed form without back-propagation through the whole network, and a gradient-alignment regularizer is added to the base TTA loss (Shin et al., 2024).

Distributed and federated algorithms can implement alignment through displacement or shrinkage. GradAlign computes the global gradient $$\mathrm{GWA}_T= \frac{\mathbb E_i[\gamma(x_i,w_T)]}{\mathrm{Kurt}[\mathcal A_T]+\beta}, \qquad \beta=1.2,$$0, forms per-batch displacements

$$\mathrm{GWA}_T= \frac{\mathbb E_i[\gamma(x_i,w_T)]}{\mathrm{Kurt}[\mathcal A_T]+\beta}, \qquad \beta=1.2,$$1

evaluates stale gradients at $$\mathrm{GWA}_T= \frac{\mathbb E_i[\gamma(x_i,w_T)]}{\mathrm{Kurt}[\mathcal A_T]+\beta}, \qquad \beta=1.2,$$2, and thereby approximates GD on $$\mathrm{GWA}_T= \frac{\mathbb E_i[\gamma(x_i,w_T)]}{\mathrm{Kurt}[\mathcal A_T]+\beta}, \qquad \beta=1.2,$$3 while preserving parallelism. pFLAlign instead combines an SVAG-style preconditioner during local training with an aggregation-side shrinkage term based on an alignment probability between stochastic descent and the personalized offset; the paper derives both mechanisms from a PAC-Bayesian analysis (Dandi et al., 2021, Kim et al., 4 May 2026).

IGA extends the algorithmic pattern to OOD reasoning distillation under LoRA constraints. It groups examples into Logical Isomer Sets, computes full-rank reconstructed gradients for each domain, masks high-variance coordinates using $$\mathrm{GWA}_T= \frac{\mathbb E_i[\gamma(x_i,w_T)]}{\mathrm{Kurt}[\mathcal A_T]+\beta}, \qquad \beta=1.2,$$4, and projects the masked gradient back onto the rank-$$\mathrm{GWA}_T= \frac{\mathbb E_i[\gamma(x_i,w_T)]}{\mathrm{Kurt}[\mathcal A_T]+\beta}, \qquad \beta=1.2,$$5 LoRA manifold via truncated SVD. The resulting update remains parameter-efficient while suppressing cross-domain gradient conflicts (Cheng et al., 3 Jun 2026).

4. Data selection, label inference, and mixture optimization

Gradient alignment is not limited to parameter updates; it also appears as a criterion for choosing data, labels, or mixtures. In LLM reinforcement learning, GradAlign uses a small, trusted validation set and scores each candidate problem by the cosine similarity

$$\mathrm{GWA}_T= \frac{\mathbb E_i[\gamma(x_i,w_T)]}{\mathrm{Kurt}[\mathcal A_T]+\beta}, \qquad \beta=1.2,$$6

where $$\mathrm{GWA}_T= \frac{\mathbb E_i[\gamma(x_i,w_T)]}{\mathrm{Kurt}[\mathcal A_T]+\beta}, \qquad \beta=1.2,$$7 is the aggregate validation gradient. The selected curriculum is the top-$$\mathrm{GWA}_T= \frac{\mathbb E_i[\gamma(x_i,w_T)]}{\mathrm{Kurt}[\mathcal A_T]+\beta}, \qquad \beta=1.2,$$8 subset by descending alignment score. The method is explicitly designed for non-stationary policy optimization, where reward feedback may be sparse, noisy, or adversarial (Yang et al., 25 Feb 2026).

Dynamic Gradient Alignment (DGA) applies the same idea to online data mixing for LLM specialization. For generic domain $$\mathrm{GWA}_T= \frac{\mathbb E_i[\gamma(x_i,w_T)]}{\mathrm{Kurt}[\mathcal A_T]+\beta}, \qquad \beta=1.2,$$9 and small target set $f(x)=\frac1n\sum_{i=1}^n f_i(x),$0, it defines

$f(x)=\frac1n\sum_{i=1}^n f_i(x),$1

and uses mirror descent on simplex weights,

$f(x)=\frac1n\sum_{i=1}^n f_i(x),$2

optionally smoothed by an EMA. The stated objective is to estimate online the pre-training data mixture whose gradients align as well as possible with those of the specific task (Fan et al., 2024).

Two earlier lines of work place labels themselves in gradient space. Label Gradient Alignment defines the feature map

$f(x)=\frac1n\sum_{i=1}^n f_i(x),$3

and imputes unlabeled targets by minimizing the discrepancy between the average labeled gradient and the unlabeled gradient. GAR, in the meta-learning sense of “gradient-alignment reward,” uses the next-step gradient as a low-variance reward for non-differentiable training decisions:

$f(x)=\frac1n\sum_{i=1}^n f_i(x),$4

Both methods treat gradient agreement as a supervisory signal even when conventional labels or differentiable meta-gradients are unavailable (Jackson et al., 2019, Müller et al., 2021).

In regression, GAR uses pairwise label difference losses to align model and target gradients in label space. The magnitude-alignment loss matches $f(x)=\frac1n\sum_{i=1}^n f_i(x),$5 to $f(x)=\frac1n\sum_{i=1}^n f_i(x),$6, while the direction-alignment loss reduces, for $f(x)=\frac1n\sum_{i=1}^n f_i(x),$7, to

$f(x)=\frac1n\sum_{i=1}^n f_i(x),$8

namely one minus the Pearson correlation. The theoretical claim is that matching pairwise differences gives insight from learning the pairwise label difference to learning the gradient of the ground truth function (Zhu et al., 2024).

5. Representative applications and reported empirical behavior

The applications of gradient alignment are unusually broad. In supervised classification, GWA is presented as a validation-set-free proxy for generalization that predicts optimal early stopping, supports model comparison, and identifies influential training samples. Across ConvNeXt and ViT on CIFAR-10, CIFAR-10 with label noise, and ImageNet-1k, the epoch where $f(x)=\frac1n\sum_{i=1}^n f_i(x),$9 peaks matches or exceeds the test accuracy obtained by traditional 10% or 1% validation-set early stopping and outperforms LabelWave or Gradient Disparity; models selected via GWA-based early stopping also show $$r(x)=\frac1{2n}\sum_{i=1}^n \|\nabla f_i(x)-\nabla f(x)\|^2.$$0 higher accuracy on corrupted-image benchmarks than those selected via a 10% validation set (Hölzl et al., 29 Oct 2025).

Medical image segmentation provides a direct adaptation example. On five optic-disc/cup datasets, GraTa reports average Dice scores of $$r(x)=\frac1{2n}\sum_{i=1}^n \|\nabla f_i(x)-\nabla f(x)\|^2.$$1, compared with $$r(x)=\frac1{2n}\sum_{i=1}^n \|\nabla f_i(x)-\nabla f(x)\|^2.$$2 for DeTTA and $$r(x)=\frac1{2n}\sum_{i=1}^n \|\nabla f_i(x)-\nabla f(x)\|^2.$$3 for SAR, and on the most challenging Domain D as target it raises DSC from $$r(x)=\frac1{2n}\sum_{i=1}^n \|\nabla f_i(x)-\nabla f(x)\|^2.$$4 for no adaptation and $$r(x)=\frac1{2n}\sum_{i=1}^n \|\nabla f_i(x)-\nabla f(x)\|^2.$$5 for the best competitor to $$r(x)=\frac1{2n}\sum_{i=1}^n \|\nabla f_i(x)-\nabla f(x)\|^2.$$6. A direct cosine-similarity measurement against the true empirical gradient is reported to increase from approximately $$r(x)=\frac1{2n}\sum_{i=1}^n \|\nabla f_i(x)-\nabla f(x)\|^2.$$7 to approximately $$r(x)=\frac1{2n}\sum_{i=1}^n \|\nabla f_i(x)-\nabla f(x)\|^2.$$8 (Chen et al., 2024).

Prompt-based and robust domain adaptation methods use alignment to mediate domain conflict. PGA reaches $$r(x)=\frac1{2n}\sum_{i=1}^n \|\nabla f_i(x)-\nabla f(x)\|^2.$$9 on ImageCLEF, $\nabla f_i(x)^\top\nabla f_j(x)$0 on Office-Home, and $\nabla f_i(x)^\top\nabla f_j(x)$1 on DomainNet under the source-combined protocol, while MPGA reaches $\nabla f_i(x)^\top\nabla f_j(x)$2 on Office-Home and $\nabla f_i(x)^\top\nabla f_j(x)$3 on DomainNet under the multi-source protocol (Phan et al., 2024). For cross-domain face anti-spoofing, GAC-FAS reduces HTER from $\nabla f_i(x)^\top\nabla f_j(x)$4 to $\nabla f_i(x)^\top\nabla f_j(x)$5 on OCM$\nabla f_i(x)^\top\nabla f_j(x)$6I and reports HTER $\nabla f_i(x)^\top\nabla f_j(x)$7 with AUC $\nabla f_i(x)^\top\nabla f_j(x)$8 on ICM$\nabla f_i(x)^\top\nabla f_j(x)$9O convergence evaluation (Le et al., 2024). In deepfake detection, RoGA reports $1-r(x)$0 ACC and $1-r(x)$1 AUC in the GID-FS case, and its ablation shows full RoGA at $1-r(x)$2 AUC on DeepFakes versus $1-r(x)$3 for baseline SGD (Qiu et al., 27 May 2025).

The same pattern appears in multi-distribution and scientific learning. On DomainBed, SAGE achieves $1-r(x)$4 leave-one-domain-out average top-1 accuracy versus ERM $1-r(x)$5, SAM $1-r(x)$6, and GGA $1-r(x)$7 (Ballas et al., 8 May 2026). In PINNs, SOAP yields relative $1-r(x)$8 error $1-r(x)$9 on lid-driven cavity and $r(x)$0 on Rayleigh–Taylor, while maintaining $r(x)$1 to $r(x)$2 throughout training (Wang et al., 2 Feb 2025). In OOD reasoning distillation, IGA improves OOD accuracy from $r(x)$3 under ERM-SFT to $r(x)$4 on ARB, LogiQA 2.0, ReClor, and MATH Cross-Domain, and reduces Logical Consistency Score from $r(x)$5 to $r(x)$6 (Cheng et al., 3 Jun 2026).

Data curation and semi-supervision show similar empirical effects. Under 50% random-reward noise in LLM RL, GradAlign reaches $r(x)$7 average test accuracy after 100 steps versus $r(x)$8 for Random and $r(x)$9 for AccGreedy; under distribution imbalance it reaches $$\gamma(x_i,w_t)=\cos\mathrm{\ sim}(g_t(x_i),w_t) =\frac{\langle g_t(x_i),w_t\rangle}{\|g_t(x_i)\|\cdot\|w_t\|}\in[-1,1].$$00 Countdown Test@50 steps versus $$\gamma(x_i,w_t)=\cos\mathrm{\ sim}(g_t(x_i),w_t) =\frac{\langle g_t(x_i),w_t\rangle}{\|g_t(x_i)\|\cdot\|w_t\|}\in[-1,1].$$01 for Random (Yang et al., 25 Feb 2026). LGA achieves $$\gamma(x_i,w_t)=\cos\mathrm{\ sim}(g_t(x_i),w_t) =\frac{\langle g_t(x_i),w_t\rangle}{\|g_t(x_i)\|\cdot\|w_t\|}\in[-1,1].$$02 test error on CIFAR-10 with 4000 random labels and $$\gamma(x_i,w_t)=\cos\mathrm{\ sim}(g_t(x_i),w_t) =\frac{\langle g_t(x_i),w_t\rangle}{\|g_t(x_i)\|\cdot\|w_t\|}\in[-1,1].$$03 when combined with VAT (Jackson et al., 2019). Gradient Hyperalignment, finally, reports $$\gamma(x_i,w_t)=\cos\mathrm{\ sim}(g_t(x_i),w_t) =\frac{\langle g_t(x_i),w_t\rangle}{\|g_t(x_i)\|\cdot\|w_t\|}\in[-1,1].$$04 on DS105 and $$\gamma(x_i,w_t)=\cos\mathrm{\ sim}(g_t(x_i),w_t) =\frac{\langle g_t(x_i),w_t\rangle}{\|g_t(x_i)\|\cdot\|w_t\|}\in[-1,1].$$05 on DS107 in multi-subject fMRI classification, with lower time complexity than SVD-based alternatives (Xu et al., 2018).

6. Limitations, misconceptions, and open directions

A persistent misconception is that “gradient alignment” names a single object. The literature does not support that reading. Some methods use cosine similarity, some use raw inner products, some use gradient variance, some use covariance-based tokenwise decompositions, and some use geometric alignment with tangent spaces to extremal sets. The term is therefore best understood as a family resemblance rather than a single invariant definition (Dandi et al., 2021, Young, 5 Mar 2026).

A second misconception is that alignment alone is always sufficient for generalization. SAGE states the contrary in the strongest available form: both flatness and gradient alignment are necessary, and neither $$\gamma(x_i,w_t)=\cos\mathrm{\ sim}(g_t(x_i),w_t) =\frac{\langle g_t(x_i),w_t\rangle}{\|g_t(x_i)\|\cdot\|w_t\|}\in[-1,1].$$06 nor $$\gamma(x_i,w_t)=\cos\mathrm{\ sim}(g_t(x_i),w_t) =\frac{\langle g_t(x_i),w_t\rangle}{\|g_t(x_i)\|\cdot\|w_t\|}\in[-1,1].$$07 bounds the other in general (Ballas et al., 8 May 2026). Deep linear results reach a related cautionary conclusion from a structural angle: aligned training dynamics can be exact invariants in some fully connected settings, but with constrained layers and sufficiently large datasets alignment generically fails (Radhakrishnan et al., 2020).

Practical caveats are equally prominent. GWA assumes a linear-head approximation, may exhibit a distinct rise-then-fall pattern under random-label memorization, and may not fully capture multi-modal alignment distributions through mean and kurtosis alone (Hölzl et al., 29 Oct 2025). GradAlign and related selection methods require a small, trusted validation set rather than being validation-free (Yang et al., 25 Feb 2026). GradAlign/FedGA add an extra gradient or communication round; RoGA requires two gradient evaluations per domain per iteration; Gradient Hyperalignment requires orthogonality projection after each update; GAR for regression is explicitly limited to the clean data setting without noises, outliers or distributional shifts (Dandi et al., 2021, Qiu et al., 27 May 2025, Xu et al., 2018, Zhu et al., 2024).

Open directions in the cited literature are correspondingly diverse. GWA proposes extension to self-supervised or contrastive losses, dimension-reduction such as Johnson-Lindenstrauss to align embeddings across architectures, and combination with information-theoretic metrics or curvature-based analyses (Hölzl et al., 29 Oct 2025). Distributed work points to asynchronous or decentralized variants, momentum integration, adaptive $$\gamma(x_i,w_t)=\cos\mathrm{\ sim}(g_t(x_i),w_t) =\frac{\langle g_t(x_i),w_t\rangle}{\|g_t(x_i)\|\cdot\|w_t\|}\in[-1,1].$$08 schedules, and communication compression (Dandi et al., 2021). RLHF analysis introduces a recovery-penalty objective precisely because standard alignment gradients vanish beyond the harm horizon, suggesting that “deep alignment” may require objectives that create gradient signal at all positions rather than only where harm is decided (Young, 5 Mar 2026). Taken together, these directions suggest that the central research question is no longer whether gradients align, but which notion of alignment is operationally relevant for a given optimization regime, what geometric property it actually controls, and when that control translates into robust generalization.