Adaptive Manifold-Aligned Graph Regularization (MAGR++)

• MAGR++ is a continual learning framework for Action Quality Assessment that mitigates feature manifold shift through adaptive full-parameter tuning and controlled feature replay.
• It employs a two-step feature rectification pipeline—using a manifold projector and supervised graph regularization—to align historical and current feature geometries with quality scores.
• Empirical results show state-of-the-art performance on CAQA benchmarks, achieving significant gains in accuracy, reduced forgetting, and improved forward transfer.

Adaptive Manifold-Aligned Graph Regularization (MAGR++) is a continual learning framework for Action Quality Assessment (AQA) under non-stationary video data, designed to mitigate catastrophic forgetting while preserving the ability to adapt to evolving quality distributions. It combines layer-adaptive full-parameter fine-tuning with a two-step feature rectification pipeline: a manifold projector that translates historical features into the current representation space, and a graph regularizer that aligns feature geometry with quality-score structure across old and new data (Zhou et al., 8 Oct 2025). MAGR++ extends the earlier MAGR formulation for Continual AQA (CAQA), which introduced feature replay, manifold alignment, and graph regularization to address misalignment between stored features and a changing feature manifold (Zhou et al., 2024).

1. Problem setting in continual action quality assessment

AQA is a regression problem in which a model predicts continuous quality scores for complex human actions in video. MAGR++ is formulated for CAQA, where the data distribution evolves over time and the model must be updated session by session without forgetting previously learned scoring behavior. The central difficulty is that real deployment involves non-stationary and evolving data distributions, referred to as feature manifold shift, which reduces the effectiveness of fixed models and naively fine-tuned models.

The framework is motivated by two empirical observations. First, parameter-efficient fine-tuning is insufficient for CAQA, because AQA requires stronger alignment between action-recognition pretraining and fine-grained downstream regression than light adapters provide. Second, uncontrolled full-parameter fine-tuning improves adaptability but induces overfitting, parameter drift, and feature manifold shift, which in turn degrades feature replay because historical features no longer match the encoder’s current representation space (Zhou et al., 8 Oct 2025).

This formulation is specific to privacy- and storage-constrained continual video regression. Rather than storing raw videos, the method stores feature prototypes in a memory bank. A plausible implication is that MAGR++ treats representational consistency, rather than only parameter consolidation, as the primary mechanism for controlling forgetting in CAQA.

2. Theoretical basis for adaptive fine-tuning

The theoretical motivation in MAGR++ centers on the asymmetry between parameter-efficient fine-tuning and full-parameter fine-tuning. The paper states that parameter-efficient fine-tuning suffers from a projection gap:

$$R_\text{down}(\bm{\theta}_\text{PEFT}) - R_\text{down}^* \geq \frac{\mu}{2}\|\Pi_{\mathcal{S}^\perp}^{(\Sigma_0)}\bm{\Delta}\|_{\Sigma_0}^2 - C_\varepsilon$$

where $\mathcal{S}$ is the parameter-efficient fine-tuning subspace. In this formulation, full-parameter fine-tuning can reach a lower downstream risk.

At the same time, replay-based forgetting under full-parameter fine-tuning is upper bounded by

$$\psi_t(k) \lesssim L_g L_f \Delta_t + C L_\phi \Delta_t + E_\text{opt},$$

with $L_g$, $L_f$, and $L_\phi$ denoting Lipschitz constants and $\Delta_t$ the update magnitude. The paper further reports that deeper layers drift more and cause instability when features are replayed. This provides the rationale for a selective adaptation strategy rather than uniform tuning across the backbone (Zhou et al., 8 Oct 2025).

The resulting design principle is neither full freezing nor unconstrained end-to-end updating. Instead, MAGR++ stabilizes shallow layers while allowing deeper layers to adapt, so that plasticity is concentrated where session-specific variation is strongest and stability is retained where generic cues remain transferable.

3. Layer-adaptive full-parameter fine-tuning

The first distinctive module in MAGR++ is layer-adaptive full-parameter fine-tuning. Its motivation is that shallow layers encode generic and transferable cues such as pose and location, whereas deeper layers are more session-specific. Excessive change in shallow layers destabilizes replay, but insufficient change in deep layers impairs adaptation to new quality regimes.

To determine the adaptation boundary, the method evaluates clustering quality at each layer using the Davies-Bouldin index:

$$r^l = \frac{\mathcal{C}^l_\text{tune}}{\mathcal{C}^l_\text{fix}}$$

and then selects

$L_\text{opt} = \min\{l \mid r^l > 1 + \epsilon\}.$

Layers above $L_\text{opt}$ are fully tuned, whereas layers below it are constrained toward previous-layer outputs. The corresponding shallow-layer constraint loss is

$\mathcal{S}$0

This module distinguishes MAGR++ from the earlier MAGR formulation, which concentrated on feature replay alignment but did not incorporate an adaptive boundary for backbone tuning. The significance of this addition is that it operationalizes the paper’s theoretical claim that effective CAQA requires full-parameter adaptation, but only under explicit control of where drift is permitted (Zhou et al., 8 Oct 2025).

4. Two-step feature rectification pipeline

The core representational mechanism of MAGR++ is a two-step feature rectification pipeline composed of a manifold projector and an Intra-Inter-Joint Graph Regularizer (IIJ-GR).

The manifold projector learns the residual shift between features produced by the previous backbone and those produced by the current backbone. For a current-session sample $\mathcal{S}$1, the old feature is $\mathcal{S}$2, the new feature is $\mathcal{S}$3, and the projector $\mathcal{S}$4 predicts

$\mathcal{S}$5

Training minimizes

$\mathcal{S}$6

After training, each stored historical feature is updated by

$\mathcal{S}$7

This directly addresses the feature mismatch that arises when replayed features were extracted by earlier encoders.

The second step regularizes the geometry of the projected-old and current-new feature set so that it mirrors the structure of their quality scores. Features are normalized to unit length, and the angular distance matrix is computed as

$\mathcal{S}$8

while the pairwise score-difference matrix is

$\mathcal{S}$9

These matrices are partitioned into subblocks to regularize both local, within-session structure and global, cross-session structure. The graph loss is

$$\psi_t(k) \lesssim L_g L_f \Delta_t + C L_\phi \Delta_t + E_\text{opt},$$0

The method therefore does not use graph regularization as a generic smoothness prior. Instead, it imposes supervised alignment between feature relations and score relations after historical features have been rectified into the current manifold. This suggests that the graph term functions as a relational calibration mechanism rather than as a purely unsupervised neighborhood constraint (Zhou et al., 8 Oct 2025).

5. Memory replay, optimization flow, and empirical results

MAGR++ uses Ordered Uniform Sampling (OUS) to build the memory bank. At the end of each session, training samples are sorted by predicted scores, divided into equal bins, and one prototype per bin is selected. The memory therefore stores representative feature prototypes rather than raw video, which reduces storage and preserves coverage over the score range.

The reported training flow is session-based. After each session, prototypes are added to memory. At the start of a new session, the backbone and regressor are adapted with layer-adaptive full-parameter fine-tuning, the manifold projector is trained on the current session, historical features in memory are projected into the new space, graph regularization is applied on mixed batches of new and projected-old features, and the regressor is updated jointly with the current-task, replay, tuning, projection, and graph losses. The implementation details specify an I3D backbone in PyTorch, a 2-layer MLP manifold projector, and loss coefficients normalized to one (Zhou et al., 8 Oct 2025).

The evaluation protocol covers four CAQA benchmarks from three datasets, including MTL-AQA, FineDiving, UNLV-Dive, and UNLV-Vault. Baselines include raw-data replay methods such as MER, DER++, TOPIC, and GEM; feature-replay methods such as FS-Aug, NC-FSCIL, SLCA, and MAGR; parameter-constrained methods such as SI, EWC, and LwF; and ablations. Performance is reported with overall Spearman’s rank correlation $$\psi_t(k) \lesssim L_g L_f \Delta_t + C L_\phi \Delta_t + E_\text{opt},$$1, average forgetting $$\psi_t(k) \lesssim L_g L_f \Delta_t + C L_\phi \Delta_t + E_\text{opt},$$2, and forward transfer $$\psi_t(k) \lesssim L_g L_f \Delta_t + C L_\phi \Delta_t + E_\text{opt},$$3.

The paper reports state-of-the-art results. In offline CAQA, MAGR++ achieves absolute gains of up to $$\psi_t(k) \lesssim L_g L_f \Delta_t + C L_\phi \Delta_t + E_\text{opt},$$4, with an average gain of $$\psi_t(k) \lesssim L_g L_f \Delta_t + C L_\phi \Delta_t + E_\text{opt},$$5 over the best baseline; on MTL-AQA, it reaches $$\psi_t(k) \lesssim L_g L_f \Delta_t + C L_\phi \Delta_t + E_\text{opt},$$6, compared with $$\psi_t(k) \lesssim L_g L_f \Delta_t + C L_\phi \Delta_t + E_\text{opt},$$7 for MAGR. In online CAQA, gains are larger, reaching up to $$\psi_t(k) \lesssim L_g L_f \Delta_t + C L_\phi \Delta_t + E_\text{opt},$$8 and averaging $$\psi_t(k) \lesssim L_g L_f \Delta_t + C L_\phi \Delta_t + E_\text{opt},$$9 over the strongest baseline. The paper also reports consistently better forgetting and forward transfer than both feature-replay and sample-replay methods, and states that removing the manifold projector or using only one-stage rectification significantly degrades performance. Qualitative analyses using t-SNE and error robustness studies show more stable and clustered features and smaller regression error under label scarcity and noise (Zhou et al., 8 Oct 2025).

6. Lineage, interpretation, and relation to adjacent methods

MAGR++ is best understood as an extension of MAGR rather than a replacement of its core insight. MAGR introduced CAQA, identified the misalignment between static stored features and the dynamically changing feature manifold, and addressed that problem through a manifold projector plus a graph regularizer operating on old and new features jointly. In that earlier formulation, the graph term used angular feature distances, quality-score differences, and a KL-divergence-based objective over the global matrix and its intra-/inter-session blocks; the manifold projector was already shown to be critical, with a reported ablation drop on MTL-AQA from $L_g$0 to $L_g$1 when the projector was removed (Zhou et al., 2024). MAGR++ preserves this two-step rectification logic but adds the layer-adaptive fine-tuning mechanism required by its theoretical analysis of full-parameter adaptation.

Several misconceptions can be addressed directly. MAGR++ is not a parameter-efficient fine-tuning method, because the paper explicitly argues that parameter-efficient fine-tuning is insufficient for CAQA. It is not a raw-data replay system, because replay is performed through feature prototypes rather than videos. It is also not merely a generic graph-Laplacian penalty: the graph component is supervised by score structure and operates after feature-space rectification, so its purpose is alignment between semantic ordering and geometric arrangement rather than only smoothness enforcement (Zhou et al., 8 Oct 2025).

In a broader methodological context, MAGR++ belongs to a family of graph- and manifold-aware regularization methods that adapt the regularizer to latent geometry. Manifold regularization for locally stable deep neural networks uses a sparsified graph Laplacian built from local perturbation neighborhoods to encourage local stability under $L_g$2, $L_g$3, and Wasserstein perturbations (Jin et al., 2020). Data-dependent graph Laplacian regularization for inverse problems constructs the Laplacian from a preliminary reconstruction $L_g$4, making the regularizer adaptive to observed data and noise (Bianchi et al., 2023). These methods address different tasks and use different graph constructions, but they share the broader principle that regularization is strongest when the graph is aligned to an inferred manifold rather than fixed a priori. In MAGR++, that principle is specialized to continual video regression, where the manifold itself drifts across sessions and must therefore be both corrected and regularized.