Hyper-Adaptive Momentum Dynamics (HAMD)
- HAMD is a family of optimization methods that replaces a fixed momentum coefficient with adaptive, state-dependent rules driven by local curvature, energy, or loss information.
- Variants like ASHB, Ada²m, AdamODE, and others use continuous-time and discrete formulations to dynamically adjust momentum through mechanisms such as curvature estimation and adaptive friction.
- Empirical studies and theoretical analyses show that HAMD enhances convergence, stability, and performance across deep learning, sparse updates, and higher-order combinatorial optimization tasks.
Searching arXiv for HAMD and closely related momentum-adaptation papers to ground the article in current literature. {"query":"all:\"Hyper-Adaptive Momentum Dynamics\" OR ti:\"Hyper-Adaptive Momentum Dynamics\" OR abs:\"Hyper-Adaptive Momentum Dynamics\"","max_results":10} {"query":"id:(Serbarinov, 16 Mar 2026) OR ti:\"Hyper-Adaptive Momentum Dynamics for Native Cubic Portfolio Optimization\"","max_results":10} {"query":"id:(Karoni et al., 30 Jan 2026) OR ti:\"Adaptive Momentum and Nonlinear Damping for Neural Network Training\" OR ti:\"Dyna: A Method of Momentum for Stochastic Optimization\"","max_results":10} Hyper-Adaptive Momentum Dynamics (HAMD) denotes a family of optimization methods in which momentum is treated as a state-dependent quantity rather than as a fixed hyperparameter. Across the literature, the term covers several related constructions: per-iteration momentum adaptation from local curvature estimates, model-based adaptive memory rules, loss-driven co-adaptation of momentum and step size, continuous-time adaptive-friction systems with cubic damping limits, adaptive-moment neural ODEs, coordinate-wise momentum rules for sparse-update regimes, and, in a distinct combinatorial setting, a hybrid native higher-order search method for cubic cardinality-constrained portfolio optimization (Sun et al., 2021, Topollai et al., 6 Oct 2025, Wang et al., 29 Oct 2025, Karoni et al., 30 Jan 2026, Cho et al., 2022, Everett et al., 27 May 2026, Serbarinov, 16 Mar 2026).
1. Terminological scope and lineage
The common theme in HAMD is the replacement of a constant momentum coefficient—typically a hand-tuned scalar such as —with an online mechanism that responds to observed optimization dynamics. In some works this response is driven by curvature surrogates derived from gradient and iterate differences; in others it is driven by kinetic energy, a proximal dual variable, a loss-sensitive memory window, or coordinate-wise update sparsity (Sun et al., 2021, Karoni et al., 30 Jan 2026, Topollai et al., 6 Oct 2025, Wang et al., 29 Oct 2025, Everett et al., 27 May 2026).
A related precursor is "Dyna: A Method of Momentum for Stochastic Optimization," which presents a momentum gradient descent algorithm based on the first-order differential equation of Newtonian dynamics. Dyna introduces a fictitious mass to regularize the adaptive stepsize of each individual parameter, uses explicit time integration with varying damping ratio for stochastic optimization of nonlinear objective functions, optimizes adaptive stepsizes per layer based on the number of inputs, and imposes one upper bound for parameter-wise stepsizes across the network, independent of sparsity. Its implementation is described as similar to Adam, with computational efficiency and similar memory requirements, and it uses three hyper-parameters with clear physical interpretation (Han, 2018).
| Variant | Core adaptive mechanism | Representative source |
|---|---|---|
| Dyna | Fictitious mass and varying damping ratio for adaptive momentum dynamics | (Han, 2018) |
| ASHB / Ada²m | Per-iteration from an online curvature estimate | (Sun et al., 2021) |
| AdamODE / AdamNODE | Continuous-time first- and second-moment tracking | (Cho et al., 2022) |
| Adaptive Memory Momentum | Two-plane proximal model with closed-form dynamic | (Topollai et al., 6 Oct 2025) |
| DWMGrad | Dynamic window jointly drives momentum and step size | (Wang et al., 29 Oct 2025) |
| iKFAD / CD / CADAM | Per-parameter adaptive friction and cubic damping | (Karoni et al., 30 Jan 2026) |
| Native-cubic HAMD | Hamiltonian-style search, exact projection, K-swap, and ILS | (Serbarinov, 16 Mar 2026) |
This usage implies that HAMD is not a single standardized algorithm. Rather, it is a research direction organized around adaptive momentum dynamics, with the precise update law depending on the domain and modeling assumptions.
2. Continuous-time formulations and dynamical intuition
A major strand of HAMD is formulated in continuous time. In "Adaptive Momentum and Nonlinear Damping for Neural Network Training," optimization is modeled as a particle with position and momentum moving in a potential . The baseline underdamped system
is generalized by replacing the constant friction with a per-parameter adaptive friction vector controlled by kinetic energy, leading to the iKFAD system
The exact solution
0
shows that each 1 is an exponentially weighted average of past kinetic energy. Near steady state, 2, so the momentum equation approaches cubic damping,
3
which motivates the cubically damped CD and CADAM variants (Karoni et al., 30 Jan 2026).
A second continuous-time line is AdamNODE. Here the state 4 follows
5
This construction is explicitly derived as the continuous limit of discrete Adam and is presented as a hyper-adaptive momentum dynamics because the preconditioned descent velocity depends on the evolving second-moment state 6, which suppresses overshooting relative to heavy-ball neural ODEs (Cho et al., 2022).
The cubic-portfolio HAMD of 2026 also has a continuous phase, but in a different sense. It embeds the binary portfolio vector into 7, introduces momentum 8, and defines an effective energy
9
with 0 ramped from 1 over the first 2 of the CPU budget. The dynamics use
3
where 4, 5, and the transverse steering force is
6
The stated intuition is that 7 injects curvature information orthogonal to the gradient so as not to amplify descent but to bend trajectories around saddle-like regions (Serbarinov, 16 Mar 2026).
3. Discrete adaptive-momentum rules
The discrete HAMD literature differs primarily in how it computes the momentum coefficient.
In the curvature-based formulation of "Training Deep Neural Networks with Adaptive Momentum Inspired by the Quadratic Optimization," the starting point is the optimal heavy-ball coefficient for a quadratic model,
8
where 9. Since 0 is unknown, the paper uses the online estimator
1
and updates
2
This yields Adaptive Stochastic Heavy Ball (ASHB), and the same 3 can replace Adam’s fixed first-moment decay in Ada²m (Sun et al., 2021).
In the model-based "Adaptive Memory Momentum" framework, the momentum coefficient is a dual variable arising from a proximal step on the maximum of two affine models of the objective: one from the current gradient and one from the accumulated memory direction 4. The resulting Heavy-Ball-form update is
5
with
6
The paper reports deterministic and stochastic variants, as well as an AM-AdamW extension using a preconditioning metric 7 (Topollai et al., 6 Oct 2025).
DWMGrad uses a different mechanism: a dynamic window size 8 that is increased or decreased according to the accumulated loss differences. The same 9 controls both the smoothed squared-gradient statistic
0
1
and the momentum coefficient
2
The momentum and parameter updates are
3
This is explicitly described as a form of HAMD in which the momentum hyperparameter and the step size co-adapt to local loss and gradient history (Wang et al., 29 Oct 2025).
For sparse-update problems, "Dynamics of Stochastic Momentum with Sparse Updates in High Dimensions" derives a coordinate-wise HAMD rule by matching a momentum retention timescale with a learning timescale. If 4 denotes the per-coordinate activation rate, then setting 5 yields
6
with clipping to 7 in practice. This proposal is motivated by the paper’s spectral-conflict result: a single global 8 cannot simultaneously place both common and rare coordinates near the heavy-ball resonance regime (Everett et al., 27 May 2026).
4. Native higher-order combinatorial optimization
The most domain-specific use of the name HAMD appears in "Hyper-Adaptive Momentum Dynamics for Native Cubic Portfolio Optimization: Avoiding Quadratization Distortion in Higher-Order Cardinality-Constrained Search." The problem is the cubic, cardinality-constrained portfolio objective
9
subject to
0
The report emphasizes that simulated annealing and tabu search operate on a Rosenberg-quadratized surrogate, which inflates the variable count from 1 to 2 and introduces penalty terms that distort the search landscape (Serbarinov, 16 Mar 2026).
HAMD is presented there as a four-stage hybrid pipeline. The first stage is the continuous Hamiltonian-style search on the relaxed domain 3. The second is exact projection to the 4-sparse binary manifold by selecting the indices of the 5 largest 6:
7
The report states that this top-8 rounding minimizes 9 over 0 with 1 (Serbarinov, 16 Mar 2026).
The third stage is deterministic K-swap local polishing. From a binary 2-sparse vector, all 3 single-swap moves are enumerated, the best-improving swap is chosen using incremental updates of the native cubic objective, and the process repeats until no swap further decreases 4. This guarantees a local minimum under the 1-swap neighborhood. The fourth stage is an Iterated Local Search (ILS) phase occupying the final 5 of the budget, based on deterministic 2-pair perturbations followed by re-polishing via K-swap (Serbarinov, 16 Mar 2026).
A central claim of this work is that the advantage is not merely feasibility recovery after surrogate optimization. On the 6 decoded-feasibility diagnostic, simulated annealing always satisfies 7 and all 8 Rosenberg auxiliary constraints with 9 violation, yet still decodes to a native objective around 0, far above HAMD’s 1. The conclusion stated in the report is that the poor native performance is due to surrogate-landscape distortion, not feasibility violations (Serbarinov, 16 Mar 2026).
5. Theory, stability, and convergence guarantees
The theoretical status of HAMD varies substantially by formulation.
For ASHB and Ada²m, convergence guarantees are stated under standard stochastic assumptions: unbiased gradients, 2-Lipschitz gradient, and bounded stochastic gradient. The strongly convex result gives
3
while convex and nonconvex statements bound the averaged objective gap and 4, respectively. The paper notes that current proofs match SGD’s worst-case rates rather than proving acceleration (Sun et al., 2021).
For Adaptive Memory Momentum, the paper proves 5 rates in both convex and nonconvex settings when 6 and 7 decays appropriately. In the convex-smooth finite-sum case, the averaged iterate 8 satisfies
9
and for nonconvex 0-smooth objectives a matching 1 bound is given for 2. A technical difficulty identified there is that 3 depends on the same gradient 4, requiring explicit control of the induced bias (Topollai et al., 6 Oct 2025).
DWMGrad’s theory is stated under convexity and bounded-gradient assumptions. The report includes a boundedness lemma for 5 and 6, a theorem bounding the averaged parameter sequence, and a potential-function decrease result for
7
with the conclusion that 8 is non-increasing and 9 converges under strong convexity (Wang et al., 29 Oct 2025).
The strongest explicit continuous/discrete stability theory in the corpus appears in the adaptive-friction line. Under strong convexity and a coercivity-type inequality, iKFAD admits exponential convergence:
0
The corresponding first-order splitting scheme satisfies
1
for sufficiently small 2. Analogous continuous and discrete exponential convergence theorems are stated for CD (Karoni et al., 30 Jan 2026).
The sparse-update theory contributes a different form of insight. It characterizes three regimes using the ratio 3: an SGD limit when 4, instability when 5, and heavy-ball or resonance dynamics when 6. The paper argues that different coordinates can require different momentum values because their update frequencies differ, producing a spectral conflict for any global 7 (Everett et al., 27 May 2026).
By contrast, the cubic-portfolio HAMD report explicitly states that no formal convergence theorem is proven for the nonconvex cubic setting. The theoretical claims there are modular: the damped, symplectic-inspired integrator with reflection preserves ergodicity and confines trajectories to 8; exact 9-sparse projection yields feasibility; and K-swap local search finds a local minimum under the 1-swap neighborhood in 00 per iteration. Complexity per continuous step is reported as 01 for gradient and Hessian-vector products (Serbarinov, 16 Mar 2026).
6. Empirical behavior, applications, and open questions
Empirical evaluations span convex optimization, deep learning, neural ODEs, sparse high-dimensional models, and higher-order combinatorial search.
For curvature-adaptive ASHB and Ada²m, the reported benchmarks include regularized logistic regression, CIFAR-10 with PreResNet20/56/110, ImageNet with ResNet-18, a vision transformer on CIFAR-10, Penn Treebank language modeling, and IWSLT14 De02En machine translation. Concrete results reported in the summary include test-accuracy gains of approximately 03–04 on CIFAR-10, ASHB stability at learning rate 05 where SGDM blows up, ImageNet top-1 and top-5 gains of approximately 06, a reduction in LSTM test perplexity from approximately 07 to approximately 08, and BLEU 09 for Ada²mW versus AdamW’s 10 (Sun et al., 2021).
Adaptive Memory Momentum is evaluated on nine LIBSVM logistic-regression datasets, CIFAR-10/100 with VGG19, ResNet-18/50, and Wide-ResNet, and LLaMA pretraining from 11M to 12B parameters on 13B C4 tokens. The reported LLaMA validation losses at 14M/15M tokens are 16 for AdamW, 17 for AM-AdamW, 18 for a clip-only control, and 19 for a reset-only control. The paper also reports that the method remains stable for 20 and that large 21 reverts behavior toward fixed-22 dynamics (Topollai et al., 6 Oct 2025).
DWMGrad is evaluated on CIFAR-10, CIFAR-100, and ImageNet with EfficientNet-b0 and ResNet-110; on GLUE with RoBERTa-base/large; on Cora and PubMed with GAT and GraphSAGE; on UrbanSound8K with CAMP++, TDNN, and Res2Net; and on the Rosenbrock function. The summary reports approximately 23 faster loss convergence in computer vision, F1 24 versus Adam’s 25 on CIFAR-10 EfficientNet, MRPC 26 versus Adam’s 27 for RoBERTa-base, gains of 28–29 points in Accuracy/F1 on graph node classification, and faster approach to the Rosenbrock valley minimum than Adam or AdaGrad (Wang et al., 29 Oct 2025).
The adaptive-friction family is evaluated on TinyViT/CIFAR-10, DistilBERT/SST-2 pretraining and QNLI finetuning, GPT2-Nano on Shakespeare, and GPT2-XS on OpenWebText. Across ten random seeds, the reported best test losses are 30 for iKFAD on TinyViT, 31 for CD on DistilBERT SST-2, 32 for CD on DistilBERT QNLI, 33 for iKFAD on GPT2-Nano, and 34 for CD on GPT2-XS. Heatmaps over 35 are reported to show wider stability ranges for iKFAD and CD than for mSGD or LDHD, and setting 36 is stated to yield essentially identical performance for iKFAD and CD (Karoni et al., 30 Jan 2026).
In the native cubic portfolio setting, empirical comparisons are conducted on a single-core CPU with a 37 s wall-clock budget per solver per instance, for 38 with 39, 40, a sector model with 41 factors, and 42 scaled by the quadratic scale. Under single seed 43, HAMD attains native objective values 44, 45, 46, and 47 at 48, versus 49, 50, 51, and 52 for both SA and tabu search, corresponding to reported single-seed relative improvements of 53, 54, 55, and 56. In the three-seed study at 57, HAMD attains median 58 with standard deviation 59, while SA and tabu both yield median 60 with standard deviation 61. The ablation reports native objectives 62 for HAMD-cont, 63 for HAMD-proj, 64 for HAMD-polish, and 65 for HAMD-full. Exact calibration at 66 over three seeds each and a 67 s budget reports that HAMD finds the true global optimum with 68 gap in all 69 trials (Serbarinov, 16 Mar 2026).
Several misconceptions are addressed implicitly by these results. First, HAMD is not reducible to adaptive learning-rate scaling: some formulations keep step-size adaptation secondary and focus on adaptive memory or friction (Topollai et al., 6 Oct 2025, Karoni et al., 30 Jan 2026). Second, empirical gains do not erase theoretical limitations: nonconvex deep-learning guarantees remain incomplete in several lines, and the cubic portfolio paper explicitly lacks a formal convergence theorem (Wang et al., 29 Oct 2025, Serbarinov, 16 Mar 2026). Third, in higher-order combinatorial optimization, poor surrogate performance need not reflect infeasibility; the decoded-feasibility analysis in the cubic portfolio study attributes the gap to surrogate-distortion effects instead (Serbarinov, 16 Mar 2026).
Open directions stated in the literature include extending nonconvex theory, refining loss-driven window rules, integrating HAMD principles with variance reduction, quasi-Newton, or trust-region methods, expanding multi-seed studies at larger scale, comparing against bespoke native higher-order heuristics, adapting HAMD to other PUBO/HUBO benchmarks, and exploring automatic hyperparameter tuning (Wang et al., 29 Oct 2025, Sun et al., 2021, Serbarinov, 16 Mar 2026). A plausible implication is that HAMD is best understood not as a settled optimizer family with one canonical update, but as a broad dynamical design principle: momentum itself becomes an adaptive state tied to geometry, history, or activation structure.