AdaQN: Adaptive Quasi-Newton & Q-Learning

• AdaQN is an adaptive framework that addresses gradient pathologies in RNN training and hyperparameter sensitivity in reinforcement learning.
• It uses a stochastic quasi-Newton update with Adagrad-inspired diagonal initialization and periodic curvature pair management for efficient second-order approximations in deep networks.
• In reinforcement learning, an ensemble of Q-networks with on-the-fly target selection significantly improves stability, sample efficiency, and robustness.

AdaQN refers to two distinct algorithmic frameworks in machine learning: (1) an adaptive quasi-Newton (adaQN) stochastic optimizer for training recurrent neural networks (RNNs) (Keskar et al., 2015), and (2) an Adaptive Q-Network ensemble approach for deep reinforcement learning (RL) applying on-the-fly target selection (AdaQN) (Vincent et al., 2024). Both address fundamental optimization and stability challenges—gradient pathologies in RNNs and hyperparameter sensitivity in RL—by introducing adaptive mechanisms for second-order information or target selection, respectively.

1. AdaQN for Recurrent Neural Network Optimization

The original adaQN algorithm is a stochastic quasi-Newton method designed to efficiently train RNNs despite the notorious vanishing and exploding gradient problems. The optimization goal is the empirical risk minimization:

$f(w) = \frac{1}{m} \sum_{i=1}^m f_i(w),$

where $w \in \mathbb{R}^n$ are all RNN parameters, and $f_i$ is the sample loss, typically computed by back-propagation through time (BPTT). The ill-conditioning of this optimization—arising when the spectral radius of temporal Jacobians departs far from unity—leads standard stochastic gradient methods to instability or stalling, particularly in long sequence tasks.

Diagonal-scaling first-order optimizers (Adagrad, Adam) provide only moderate robustness, while full-matrix second-order methods (Hessian-Free Newton, K-FAC) impose prohibitive per-iteration costs in deep RNNs. adaQN bridges this performance-cost divide by employing a stochastic quasi-Newton strategy with low memory and computational requirements.

2. Stochastic Quasi-Newton Update and Curvature Pair Management

Each adaQN iteration samples a mini-batch $\mathcal{B}_k$ to compute a stochastic gradient $g_k$ and updates model parameters by

$w_{k+1} = w_k - \alpha_k H_k g_k,$

where $H_k$ is a positive-definite inverse Hessian approximation constructed via a limited-memory BFGS (L-BFGS) two-loop recursion using stored curvature pairs $(s_i, y_i)$ from recent steps. No explicit Hessian matrix is formed; instead, the search direction $p_k = -H_k g_k$ arises directly from the recursion over the stored vectors.

A key adaQN innovation is its initialization and adaptation of $H_k^{(0)}$. Unlike conventional stochastic L-BFGS, which uses a scalar multiple of the identity, adaQN sets

$w \in \mathbb{R}^n$0

matching Adagrad variable-wise scaling and providing resilience against disparate gradient norm scales. When no curvature pairs are retained, adaQN exactly recovers an Adagrad update.

Curvature pair formation is periodic: every $w \in \mathbb{R}^n$1 iterations, adaQN aggregates iterates to compute

$w \in \mathbb{R}^n$2

and forms $w \in \mathbb{R}^n$3. The associated $w \in \mathbb{R}^n$4 is given by the action of an accumulated empirical Fisher information matrix (obtained via a FIFO buffer of recent gradients as $w \in \mathbb{R}^n$5), $w \in \mathbb{R}^n$6. Only pairs passing the curvature condition $w \in \mathbb{R}^n$7 are added, ensuring positive-definiteness and updating only when curvature is reliably estimated. Steps resulting in a significant increase in held-out set loss are rejected, with all curvature storage cleared to preserve stability.

3. Complexity, Stability, and Implementation

adaQN retains low computational and storage cost relative to full second-order methods:

• Per iteration cost: $w \in \mathbb{R}^n$8 for the gradient, $w \in \mathbb{R}^n$9 for the two-loop recursion, and amortized $f_i$0 for the Fisher step, resulting in $f_i$1 overall.
• Memory: $f_i$2 for curvature and Fisher buffers.

No non-convex convergence proof is provided, but adaQN employs heuristic safeguards for stability: (1) stringent curvature pair acceptance, and (2) step rejection if the monitoring loss exceeds a threshold. This practical design prevents the update from leveraging unreliable curvature information.

The algorithm’s core workflow is summarized below:

Step	Description
Gradient	Compute $f_i$3 on mini-batch
Direction	Use L-BFGS two-loop with Adagrad-style diagonal $f_i$4
Update	$f_i$5
Curvature	Periodically aggregate, compute $f_i$6, test validity
Safeguards	Reject steps with excessive held-out set loss

4. Empirical Performance in Language Modeling

adaQN has been evaluated on language modeling tasks using both character-level (Dickens, Nietzsche) and word-level (Penn-Treebank) corpora with 5-layer RNNs, employing both tanh and ReLU activations. Baselines included Adagrad and Adam. Key findings are:

• On tanh RNNs, adaQN consistently achieved lower test perplexity and character error than both Adagrad and Adam.
• On ReLU RNNs, adaQN outperformed Adam and was comparable to Adagrad.
• In pixel-level MNIST sequence prediction, adaQN progressed while Adagrad/Adam stalled.
• For 2-layer LSTMs, adaQN matched or outperformed the alternatives depending on the dataset.

These experiments establish that adaQN’s robust adaptive scaling and cost-effective non-diagonal curvature extraction make it effective for deep RNN training under challenging optimization regimes (Keskar et al., 2015).

5. AdaQN for Adaptive Q-Learning in Reinforcement Learning

A distinct methodology named Adaptive Q-Network (AdaQN) addresses a fundamental challenge in RL: hyperparameter sensitivity and non-stationarity. It maintains an ensemble of $f_i$7 Q-networks, each parameterized by $f_i$8 and using unique tuples of hyperparameters (learning rate, optimizer, architecture, activation, etc.). A shared target network $f_i$9 is periodically updated with the parameters from the online network showing minimal cumulative temporal-difference (TD) error over the latest period.

For a batch $\mathcal{B}_k$0, each Q-network minimizes

$\mathcal{B}_k$1

with the AdaQN joint loss summing all K:

$\mathcal{B}_k$2

Every $\mathcal{B}_k$3 steps, AdaQN selects the $\mathcal{B}_k$4-th network, with smallest $\mathcal{B}_k$5, to update the target:

$\mathcal{B}_k$6

This approach adapts to nonstationarity by dynamically switching to the hyperparameter configuration best suited to the current training regime—without additional environment samples.

6. Theoretical Foundations and Empirical Results in RL

The mechanism is theoretically supported via a result of Farahmand (2011), which bounds suboptimality in terms of a sum over Bellman-projection errors. Minimizing cumulative TD-loss at each target switch is a tractable proxy for minimizing these projection errors, improving performance and stability over fixed hyperparameter configurations.

Empirical results in MuJoCo continuous control and Atari games demonstrate that AdaQN:

• Outperforms every static configuration regardless of tested network size or optimizer choice.
• Achieves greater sample efficiency, higher mean and worst-seed performance, and enhanced robustness to stochasticity.
• Can be implemented by running $\mathcal{B}_k$7 critics in parallel and periodically updating the target for all agents based on observed error.

Computational cost scales linearly with $\mathcal{B}_k$8 in memory and can be efficiently parallelized due to independent per-network computations. Even modest ensemble sizes ($\mathcal{B}_k$9) suffice in practice.

7. Practical Implementation and Recommendations

Practical usage guidelines for both forms of AdaQN are as follows:

Domain	Default/Recommended Settings
RNNs	$g_k$0, $g_k$1, $g_k$2, Adagrad step-size, $g_k$3, $g_k$4
RL	$g_k$5 ensemble, diversify optimizer/arch/hyperparams, periodic TD-loss based selection

In RNN training, the adaptive diagonal scaling ensures resilience when no reliable curvature exists, and if noise or instability is detected, reducing $g_k$6 or increasing mini-batch size is advised. In RL, ensemble size should be balanced against hardware limits, and diversity in hyperparameter choices improves performance envelope coverage.

Both AdaQN methodologies represent scalable, adaptive, and cost-effective advances in second-order optimization and automated agent configuration, with broad applicability across deep learning settings (Keskar et al., 2015, Vincent et al., 2024).