--- title: Hyper-Adaptive Momentum Dynamics (HAMD) url: https://www.emergentmind.com/topics/hyper-adaptive-momentum-dynamics-hamd type: topic --- # Hyper-Adaptive Momentum Dynamics (HAMD) Searching arXiv for HAMD and closely related momentum-adaptation papers to ground the article in current literature. {"query":"all:\"Hyper-Adaptive Momentum Dynamics\" OR ti:\"Hyper-Adaptive Momentum Dynamics\" OR abs:\"Hyper-Adaptive Momentum Dynamics\"","max_results":10} {"query":"id:2603.15947 OR ti:\"Hyper-Adaptive Momentum Dynamics for Native Cubic Portfolio Optimization\"","max_results":10} {"query":"id:2602.00334 OR ti:\"Adaptive Momentum and Nonlinear Damping for Neural Network Training\" OR ti:\"Dyna: A Method of Momentum for Stochastic Optimization\"","max_results":10} Hyper-Adaptive Momentum Dynamics (HAMD) denotes a family of optimization methods in which momentum is treated as a state-dependent quantity rather than as a fixed hyperparameter. Across the literature, the term covers several related constructions: per-iteration momentum adaptation from local curvature estimates, model-based adaptive memory rules, loss-driven co-adaptation of momentum and step size, continuous-time adaptive-friction systems with cubic damping limits, adaptive-moment neural ODEs, coordinate-wise momentum rules for sparse-update regimes, and, in a distinct combinatorial setting, a hybrid native higher-order search method for cubic cardinality-constrained portfolio optimization [2110.09057; 2510.04988; 2510.25042; 2602.00334; 2207.06066; 2605.28961; 2603.15947]. ## 1. Terminological scope and lineage The common theme in HAMD is the replacement of a constant momentum coefficient—typically a hand-tuned scalar such as $\beta=0.9$—with an online mechanism that responds to observed optimization dynamics. In some works this response is driven by curvature surrogates derived from gradient and iterate differences; in others it is driven by kinetic energy, a proximal dual variable, a loss-sensitive memory window, or coordinate-wise update sparsity [2110.09057; 2602.00334; 2510.04988; 2510.25042; 2605.28961]. A related precursor is "Dyna: A Method of Momentum for Stochastic Optimization," which presents a momentum gradient descent algorithm based on the first-order differential equation of Newtonian dynamics. Dyna introduces a fictitious mass to regularize the adaptive stepsize of each individual parameter, uses explicit time integration with varying damping ratio for stochastic optimization of nonlinear objective functions, optimizes adaptive stepsizes per layer based on the number of inputs, and imposes one upper bound for parameter-wise stepsizes across the network, independent of sparsity. Its implementation is described as similar to Adam, with computational efficiency and similar memory requirements, and it uses three hyper-parameters with clear physical interpretation [1805.04933]. | Variant | Core adaptive mechanism | Representative source | |---|---|---| | Dyna | Fictitious mass and varying damping ratio for adaptive momentum dynamics | [1805.04933] | | ASHB / Ada²m | Per-iteration $\beta_k$ from an online curvature estimate | [2110.09057] | | AdamODE / AdamNODE | Continuous-time first- and second-moment tracking | [2207.06066] | | Adaptive Memory Momentum | Two-plane proximal model with closed-form dynamic $\beta_t$ | [2510.04988] | | DWMGrad | Dynamic window $\omega_t$ jointly drives momentum and step size | [2510.25042] | | iKFAD / CD / CADAM | Per-parameter adaptive friction and cubic damping | [2602.00334] | | Native-cubic HAMD | Hamiltonian-style search, exact projection, K-swap, and ILS | [2603.15947] | This usage implies that HAMD is not a single standardized algorithm. Rather, it is a research direction organized around adaptive momentum dynamics, with the precise update law depending on the domain and modeling assumptions. ## 2. Continuous-time formulations and dynamical intuition A major strand of HAMD is formulated in continuous time. In "Adaptive Momentum and Nonlinear Damping for Neural Network Training," optimization is modeled as a particle with position $x(t)\in\mathbb{R}^N$ and momentum $p(t)\in\mathbb{R}^N$ moving in a potential $f(x)$. The baseline underdamped system $$ \dot x = p,\quad \dot p = -\nabla f(x) - \gamma p $$ is generalized by replacing the constant friction with a per-parameter adaptive friction vector $\xi(t)\in\mathbb{R}^N_{\ge 0}$ controlled by kinetic energy, leading to the iKFAD system $$ \dot x = p,\qquad \dot p = -\nabla f(x)-\gamma p-\xi\star p,\qquad \dot \xi = \frac{[p]^2}{\rho}-\alpha \xi. $$ The exact solution $$ \xi_i(t)=e^{-\alpha t}\xi_i(0)+\frac{1}{\rho}\int_0^t p_i(s)^2e^{-\alpha(t-s)}\,ds $$ shows that each $\xi_i$ is an exponentially weighted average of past kinetic energy. Near steady state, $\xi_i\approx (\alpha\rho)^{-1}p_i^2$, so the momentum equation approaches cubic damping, $$ \dot p_i \approx -\nabla_i f(x)-\frac{1}{\alpha\rho}p_i^3, $$ which motivates the cubically damped CD and CADAM variants [2602.00334]. A second continuous-time line is AdamNODE. Here the state $(x,m,v)$ follows $$ \frac{dx}{dt}=-\frac{m(t)}{\sqrt{v(t)+\varepsilon}},\qquad \frac{dm}{dt}=(1-\alpha)\bigl(\nabla_xF(x(t))-m(t)\bigr),\qquad \frac{dv}{dt}=(1-\beta)\bigl(\nabla_xF(x(t))^2-v(t)\bigr). $$ This construction is explicitly derived as the continuous limit of discrete Adam and is presented as a hyper-adaptive momentum dynamics because the preconditioned descent velocity depends on the evolving second-moment state $v(t)$, which suppresses overshooting relative to heavy-ball neural ODEs [2207.06066]. The cubic-portfolio HAMD of 2026 also has a continuous phase, but in a different sense. It embeds the binary portfolio vector into $[0,1]^n$, introduces momentum $v\in\mathbb{R}^n$, and defines an effective energy $$ E(x)=f(x)+\beta(t)\sum_{i=1}^n x_i^2(1-x_i)^2, $$ with $\beta(t)$ ramped from $0\to1$ over the first $80\%$ of the CPU budget. The dynamics use $$ v^{(t+1)}=(1-\gamma)v^{(t)}+\Delta t[-g^{(t)}+\zeta F^{(t)}],\qquad x^{(t+1)}=DTR\bigl(x^{(t)}+\Delta t\,v^{(t+1)}\bigr), $$ where $g=\nabla_xE(x)$, $h=\nabla^2E(x)\cdot v$, and the transverse steering force is $$ F=\alpha\Bigl(h-\frac{\langle g,h\rangle}{\|g\|^2+\varepsilon}g\Bigr),\qquad \alpha=\min\Bigl(1,\frac{\|g\|^2}{|\langle g,h\rangle|+\varepsilon}\Bigr). $$ The stated intuition is that $F$ injects curvature information orthogonal to the gradient so as not to amplify descent but to bend trajectories around saddle-like regions [2603.15947]. ## 3. Discrete adaptive-momentum rules The discrete HAMD literature differs primarily in how it computes the momentum coefficient. In the curvature-based formulation of "Training Deep Neural Networks with Adaptive Momentum Inspired by the Quadratic Optimization," the starting point is the optimal heavy-ball coefficient for a quadratic model, $$ \beta^*=(1-\sqrt{\gamma \nu})^2, $$ where $\nu=\lambda_{\min}(A)$. Since $\nu$ is unknown, the paper uses the online estimator $$ \hat\nu_k=\frac{\|g^k-g^{k-1}\|}{\|x^k-x^{k-1}\|}, $$ and updates $$ \beta_{k+1}=\mathrm{Proj}_{[0,1-\delta]}\Bigl[(1-\sqrt{\gamma \hat\nu_k})^2\Bigr]. $$ This yields Adaptive Stochastic Heavy Ball (ASHB), and the same $\beta_k$ can replace Adam’s fixed first-moment decay in Ada²m [2110.09057]. In the model-based "Adaptive Memory Momentum" framework, the momentum coefficient is a dual variable arising from a proximal step on the maximum of two affine models of the objective: one from the current gradient and one from the accumulated memory direction $d_t=\eta^{-1}(x_{t-1}-x_t)$. The resulting Heavy-Ball-form update is $$ d_{t+1}=\frac{\beta_t+\lambda}{1+\lambda}d_t+\frac{1-\beta_t}{1+\lambda}g_t,\qquad x_{t+1}=x_t-\eta d_{t+1}, $$ with $$ \beta_t= \mathrm{Clip}_{[0,1]} \Biggl( \frac{ \frac{\hat f(x_t)-f(x_t)}{\eta}(1+\lambda)-\langle d_t-g_t,\;g_t+\lambda d_t\rangle }{ \|d_t-g_t\|_2^2 } \Biggr). $$ The paper reports deterministic and stochastic variants, as well as an AM-AdamW extension using a preconditioning metric $P_t$ [2510.04988]. DWMGrad uses a different mechanism: a dynamic window size $\omega_t\in[1,\delta]$ that is increased or decreased according to the accumulated loss differences. The same $\omega_t$ controls both the smoothed squared-gradient statistic $$ \widetilde G_t=\frac{\omega_{t-1}}{\omega_t}\widetilde G_{t-1}+\frac{1}{\omega_t}G_t, $$ the adaptive learning rate $$ \alpha_t=\frac{\alpha_0}{\sqrt{\widetilde G_t}+\epsilon}, $$ and the momentum coefficient $$ \beta_t=\frac{\omega_t}{\delta}. $$ The momentum and parameter updates are $$ v_t=\beta_t v_{t-1}+\alpha_t g_t,\qquad \theta_{t+1}=\theta_t-v_t. $$ This is explicitly described as a form of HAMD in which the momentum hyperparameter and the step size co-adapt to local loss and gradient history [2510.25042]. For sparse-update problems, "Dynamics of Stochastic Momentum with Sparse Updates in High Dimensions" derives a coordinate-wise HAMD rule by matching a momentum retention timescale with a learning timescale. If $p_i$ denotes the per-coordinate activation rate, then setting $T_{\mathrm{ret},i}=T_{\mathrm{learn},i}$ yields $$ \beta_i = 1-\frac{\eta p_i}{1-\eta p_i}, $$ with clipping to $[0,1-\delta]$ in practice. This proposal is motivated by the paper’s spectral-conflict result: a single global $\beta$ cannot simultaneously place both common and rare coordinates near the heavy-ball resonance regime [2605.28961]. ## 4. Native higher-order combinatorial optimization The most domain-specific use of the name HAMD appears in "Hyper-Adaptive Momentum Dynamics for Native Cubic Portfolio Optimization: Avoiding Quadratization Distortion in Higher-Order Cardinality-Constrained Search." The problem is the cubic, cardinality-constrained portfolio objective $$ f(x)=x^\top \Sigma x-\mu^\top x+\sum_{(i,j,k)\in\mathcal{T}}c_{ijk}x_ix_jx_k, $$ subject to $$ \|x\|_0=\sum_{i=1}^n x_i=K,\qquad x_i\in\{0,1\}. $$ The report emphasizes that simulated annealing and tabu search operate on a Rosenberg-quadratized surrogate, which inflates the variable count from $n$ to $5n$ and introduces penalty terms that distort the search landscape [2603.15947]. HAMD is presented there as a four-stage hybrid pipeline. The first stage is the continuous Hamiltonian-style search on the relaxed domain $[0,1]^n$. The second is exact projection to the $K$-sparse binary manifold by selecting the indices of the $K$ largest $x_i$: $$ \mathrm{Proj}_K(x)_i= \begin{cases} 1,&\text{if }x_i\text{ ranks among the top }K\text{ entries of }x,\\ 0,&\text{otherwise.} \end{cases} $$ The report states that this top-$K$ rounding minimizes $\|x-y\|_2$ over $y\in\{0,1\}^n$ with $\|y\|_0=K$ [2603.15947]. The third stage is deterministic K-swap local polishing. From a binary $K$-sparse vector, all $K(n-K)$ single-swap moves are enumerated, the best-improving swap is chosen using incremental updates of the native cubic objective, and the process repeats until no swap further decreases $f(x)$. This guarantees a local minimum under the 1-swap neighborhood. The fourth stage is an Iterated Local Search (ILS) phase occupying the final $\rho=20\%$ of the budget, based on deterministic 2-pair perturbations followed by re-polishing via K-swap [2603.15947]. A central claim of this work is that the advantage is not merely feasibility recovery after surrogate optimization. On the $n=200$ decoded-feasibility diagnostic, simulated annealing always satisfies $\sum_i x_i=K$ and all $800$ Rosenberg auxiliary constraints with $0\%$ violation, yet still decodes to a native objective around $1200$, far above HAMD’s $195.65$. The conclusion stated in the report is that the poor native performance is due to surrogate-landscape distortion, not feasibility violations [2603.15947]. ## 5. Theory, stability, and convergence guarantees The theoretical status of HAMD varies substantially by formulation. For ASHB and Ada²m, convergence guarantees are stated under standard stochastic assumptions: unbiased gradients, $L$-Lipschitz gradient, and bounded stochastic gradient. The strongly convex result gives $$ \mathbb{E}\|x^K-x^*\| \le (1-2\gamma\mu)^K\,\mathbb{E}\|x^1-x^*\|^2 + C_1\gamma, $$ while convex and nonconvex statements bound the averaged objective gap and $\min_{1\le k\le K}\mathbb{E}\|\nabla f(x^k)\|^2$, respectively. The paper notes that current proofs match SGD’s worst-case rates rather than proving acceleration [2110.09057]. For Adaptive Memory Momentum, the paper proves $O(1/\sqrt{T})$ rates in both convex and nonconvex settings when $\eta=O(1/\sqrt{T})$ and $\beta_{\max}$ decays appropriately. In the convex-smooth finite-sum case, the averaged iterate $\bar x_T$ satisfies $$ \mathbb{E}[f(\bar x_T)-f(x^*)]=O\Bigl(\frac{1}{\sqrt T}\Bigr), $$ and for nonconvex $L$-smooth objectives a matching $O(1/\sqrt T)$ bound is given for $\min_{t