History-Aware Adaptive Difficulty Weighting

• HA-DW is an adaptive optimization method that uses temporally smoothed accuracy and entropy to estimate sample difficulty and dynamically adjust loss weights.
• The approach integrates exponential moving averages for performance metrics, enabling balanced class reweighting in long-tailed visual recognition and group-based reinforcement learning.
• Empirical results show that HA-DW significantly improves few-shot accuracy and reduces bias in challenging classes while maintaining stable training dynamics.

History-Aware Adaptive Difficulty Weighting (HA-DW) is an adaptive optimization methodology designed to counteract bias and imbalance in training deep recognition and reasoning systems. HA-DW leverages temporally smoothed performance metrics or anchors to model sample and class-wise difficulty, dynamically reweights training signals to concentrate optimization effort where it is most needed, and maintains balanced exploration across varying challenge levels. Developed in the context of both long-tailed visual recognition and group-based Reinforcement Learning from Verifier Rewards (RLVR), HA-DW provides a modular augmentation to conventional class reweighting and group-relative policy optimization.

1. Foundational Principles and Formal Definition

HA-DW centers on two foundational mechanisms: history-aware difficulty estimation and adaptive loss weighting. In long-tailed classification, difficulty for each class $c$ is quantified at epoch $t$ by:

• Smoothed accuracy $A_n^c(t)$: exponential moving average of model accuracy on class $c$,
• Average entropy $H_n^c(t)$: mean predictive entropy for class $c$ samples.

Difficulty score:

$d_n^c(t) = \frac{H_n^c(t)}{H_\max(t)} + \lambda\left[1-\frac{A_n^c(t)}{A_\max(t)}\right],$

with $\lambda\geq 0$ controlling the entropy/accuracy trade-off (Wei et al., 27 Aug 2025). This tightly integrates both immediate performance and historical uncertainty.

In group-based RLVR, HA-DW uses a history-aware “difficulty anchor” $C_t$—an exponential moving average of global batch success rates—and prompt-specific empirical difficulty $diff_t = \hat{p}_t - C_t$. The reweighting factor for each sample is:

$\Phi_{t,i} = \lambda_s \cdot \exp(D_{t,i} M_t),$

where $$D_{t,i} = -\operatorname{sgn}(\hat{A}_i)\operatorname{sgn}(diff_t)$$ and $M_t = |diff_t|$ (Yang et al., 13 Jan 2026).

2. History-Dependent Smoothing and Update Rules

Momentum-based smoothing is integral. The exponential moving average for accuracy in DQRoute is:

$A_n^c(t) = \beta A_n^c(t-1) + (1-\beta) acc^c(t),$

with $\beta \in [0,1)$ controlling memory (Wei et al., 27 Aug 2025). In RLVR, the difficulty anchor $C_t$ is updated by:

$$C_t^+ = (1-\eta_t)C_t^- + \eta_t y_t, \quad \eta_t = \eta \cdot \sigma_t,$$

where $y_t$ is batch pass rate and $\sigma_t$ is recent history’s standard deviation, yielding a temporally adaptive anchor (Yang et al., 13 Jan 2026).

3. Adaptive Reweighting and Loss Scale Balancing

HA-DW adaptively reweights classes or samples based on evolving difficulty, sharply contrasting conventional fixed or quantity-based schemes. In DQRoute, class-weights are recursively updated:

$$w_n^c(t) = \frac{w_n^c(t-1)\exp(\gamma d_n^c(t))}{\sum_{j=1}^C w_n^j(t-1)\exp(\gamma d_n^j(t))}$$

and interpolated with normalized frequency prior $q^c$:

$\tilde{w}^c(t) = \alpha w_n^c(t) + (1-\alpha) q^c.$

In RLVR group optimization, HA-DW replaces the raw empirical advantage $\hat{A}_i$ with $\hat{A}_i \cdot \Phi_{t,i}$ in the surrogate policy loss, directly correcting systematic bias (Yang et al., 13 Jan 2026).

Difficulty-aware reweighting has also been structurally formalized as a learnable layer over group losses in DARO:

$$\mathcal{L}_{total}(\theta, \{w_\mu\}) = \sum_\mu \left[w_\mu \mathcal{L}_\mu(\theta) - \ln w_\mu\right],$$

with weights optimally converging to $w_\mu = 1/\mathcal{L}_\mu$ (Zhou et al., 10 Oct 2025).

4. Algorithmic Workflow

All variants instantiate history tracking and adaptive weighting via recurrent pseudocode constructs with epoch-wise (or step-wise) accumulation of accuracy/entropy statistics, group loss computation, and dynamic scaling.

Representative DQRoute HA-DW pseudocode:

for epoch in range(T): for mini_batch in data_loader: compute predictions, per-class acc and entropy for class in classes: update smoothed accuracy, entropy compute difficulty score update class weights normalize weights, interpolate with quantity prior gradient descent with weighted loss

(Wei et al., 27 Aug 2025)

RLVR HA-DW integrates into the GRPO surrogate:

for t in range(T): generate G rollouts, get rewards update difficulty anchor compute empirical difficulty per prompt compute per-sample HA-DW weights apply weighted surrogate loss for policy update

(Yang et al., 13 Jan 2026)

DARO’s approach:

for step in training: sample batch, group by empirical difficulty compute group losses update difficulty weights w_mu by gradient descent update policy parameters jointly

(Zhou et al., 10 Oct 2025)

5. Theoretical Properties and Guarantees

HA-DW addresses systematic estimation bias in group-based RLVR: expected empirical group-relative advantage $\mathbb{E}[\hat{A}]$ is strictly less than true advantage $A$ for hard prompts ($p<½$) and strictly greater for easy ($p>½$) (Yang et al., 13 Jan 2026). The adaptive weighting factor $\Phi$ provably reduces the absolute bias:

$$|\mathbb{E}[\hat{A} \cdot \Phi] - A| < |\mathbb{E}[\hat{A}] - A|$$

as shown in Theorem 4.3 (Yang et al., 13 Jan 2026). In convex adaptive weight frameworks such as DARO, the regularized group loss guarantees a unique stationary point; weighted losses are equalized across difficulty tiers, ensuring continual balanced training and avoiding signal collapse (Zhou et al., 10 Oct 2025).

6. Empirical Performance and Benchmarks

Across visual recognition and RLVR, HA-DW consistently improves few-shot, tail, and overall accuracy.

In DQRoute (CIFAR-100-LT IR100), difficulty-only reweighting raises few-shot accuracy from ~10% to ~37%; combined with multi-expert OOD routing, total accuracy reaches ~51.7% (Wei et al., 27 Aug 2025). HA-DW outperforms static class frequency methods, particularly on rare and ambiguous classes.

In RLVR, incorporating HA-DW with GRPO yields robust improvements on MATH500 (75.4→78.0), AIME25 (19.6→20.4), AMC23 (60.3→63.4), Minerva (33.8→36.8), and OlympiadBench (43.5→44.7). Average gain is +2–3 points; similar improvements are observed for GSPO and DAPO baselines (Yang et al., 13 Jan 2026).

DARO validates history-aware difficulty weighting, demonstrating faster convergence and higher final accuracy than GRPO and its variants. For example, Llama-3.1-8B achieves 21.4% with DARO vs. 18.7% with GRPO; Qwen2.5-Math-7B yields 50.8% vs. 49.4% (Zhou et al., 10 Oct 2025).

7. Hyperparameters, Practical Considerations, and Integration

Key hyperparameters across frameworks:

• $\beta$ (momentum): controls accuracy smoothing responsiveness (e.g., $\beta=0.9$ for stability) (Wei et al., 27 Aug 2025).
• $\lambda$ (entropy vs. accuracy): balances difficulty signals (Wei et al., 27 Aug 2025).
• $\gamma$ (tilting sharpness): amplifies differences in difficulty weights (Wei et al., 27 Aug 2025).
• $\alpha$ (difficulty vs. quantity): interpolates data-driven and model-driven weighting, typical $\alpha\approx 0.5$ (Wei et al., 27 Aug 2025).
• $\eta$, $m$ (anchor update, history window): e.g., $\eta=0.1$, $m=20$ (Yang et al., 13 Jan 2026).
• $\lambda_s$ (advantage weight scale): optimal in range $[1.0,1.5]$ (Yang et al., 13 Jan 2026).

Computational overhead of HA-DW mechanisms is negligible compared to inference or rollout generation. Integration requires minimal changes, typically a single multiplication in the loss pipeline. Monitoring the distribution of empirical difficulties and reweighting scales supports effective tuning.

Empirical evidence supports general applicability to scenarios with binary or continuous bounded rewards and for both vision and language domains (Wei et al., 27 Aug 2025, Yang et al., 13 Jan 2026, Zhou et al., 10 Oct 2025). The methodology robustly mitigates the loss scale issues and concentration phenomena endemic to static difficulty weighting.

8. Context, Limitations, and Research Impact

HA-DW corrects long-standing weaknesses in class imbalance and group-relative policy optimization by transitioning from static or heuristic weighting to closed-loop, history-aware regulation. The jointly adaptive weighting ensures persistent attention to challenging classes and prompts, aligning optimization focus with evolving model deficiencies.

A plausible implication is that HA-DW generalizes to other dynamic curriculum learning schemes and ensemble routing strategies, subject to the presence of reliable historical performance signals and sufficient granularity in difficulty tiers.

Current limitations include sensitivity to hyperparameter selection and potential instability with extreme values (e.g., $\gamma \gg 1$), which can over-concentrate weights. In practice, recommended settings strike a balance between responsiveness and overall stability (Wei et al., 27 Aug 2025, Yang et al., 13 Jan 2026).

The empirical dominance and theoretical grounding of HA-DW across multiple domains evidence its centrality in contemporary difficulty-aware training, setting a precedent for history-integrated adaptive weighting in future architectures and optimization frameworks.