Momentum Auxiliary Network++ (MAN++)

• The paper introduces MAN++ to mitigate update locking and high GPU memory consumption by partitioning networks into local blocks and using EMA for inter-block communication.
• It employs a dynamic interaction mechanism where adjacent-block EMA parameters enhance gradient flow, improving performance across image classification, detection, and segmentation.
• A learnable scaling bias is integrated to balance feature discrepancies between local blocks, ensuring effective information exchange without compromising model accuracy.

Momentum Auxiliary Network++ (MAN++) is a supervised local learning method for vision models that is intended to mitigate three limitations attributed to end-to-end backpropagation: update locking during parameter optimization, high GPU memory consumption, and a lack of biological plausibility. In MAN++, a network is partitioned into multiple local blocks and trained with auxiliary networks, but the isolation of block-local gradients is counteracted by a dynamic interaction mechanism that uses the Exponential Moving Average (EMA) of parameters from adjacent blocks to enhance inter-block communication. The method further introduces a learnable scaling bias to balance feature differences between local blocks when EMA parameters are used, and is reported to achieve performance comparable to end-to-end training while significantly reducing GPU memory usage across image classification, object detection, and image segmentation settings (Su et al., 22 Jul 2025).

1. Problem formulation within supervised local learning

MAN++ is situated within supervised local learning, a training regime in which a deep network is partitioned into multiple local blocks and each block is updated with its own auxiliary network rather than through a single end-to-end backward pass. In the published description, this formulation is presented as a response to the locking dilemma of end-to-end backpropagation, its GPU-memory burden, and its lack of biological plausibility (Su et al., 22 Jul 2025).

The same description also states the central deficiency of conventional supervised local learning: because gradients are propagated solely within individual local blocks, performance degradation occurs, and this prevents supervised local learning from supplanting end-to-end backpropagation. The difficulty is therefore not local supervision as such, but the informational isolation induced by blockwise optimization. MAN++ is introduced specifically as a mechanism for reintroducing communication across blocks without reverting to ordinary end-to-end gradient propagation.

2. Dynamic interaction through adjacent-block EMA

The defining mechanism of MAN++ is a dynamic interaction procedure that employs the EMA of parameters from adjacent blocks. The stated purpose of this EMA-based design is to enhance communication across the network and to bridge the information gap between local blocks through an auxiliary network updated via EMA (Su et al., 22 Jul 2025).

In conceptual terms, MAN++ retains the blockwise decomposition characteristic of supervised local learning, but it no longer treats local blocks as fully informationally independent. Instead, adjacent-block EMA parameters provide a momentum-smoothed path through which local training can incorporate signals shaped by neighboring parts of the network. This suggests a hybrid training regime in which optimization remains local in its update structure, while representation learning becomes less myopic than in purely isolated local-block methods.

The emphasis on adjacent blocks is important. MAN++ is not described as reinstating full global credit assignment; rather, it introduces structured inter-block interaction at the level of neighboring blocks. That positioning distinguishes it both from strict end-to-end backpropagation and from earlier supervised local learning methods in which each auxiliary network is confined to its own block.

3. Learnable scaling bias and feature discrepancy

A key observation reported for MAN++ is that directly applying EMA parameters can be suboptimal because of feature discrepancies between local blocks. This identifies a distributional or representational mismatch problem: even if EMA provides a stable mechanism for cross-block communication, features originating from different local blocks are not automatically aligned (Su et al., 22 Jul 2025).

To address this, MAN++ introduces a learnable scaling bias that balances feature differences and thereby further improves performance. The published account does not provide an explicit formula in the available description, but the role of the mechanism is unambiguous: it is an adaptive correction term used when adjacent-block EMA parameters are brought into the local-learning pipeline.

A plausible implication is that the learnable scaling bias functions as a lightweight alignment component between neighboring feature spaces. In that interpretation, EMA supplies the communication channel, while the scaling bias calibrates the transferred information so that it is not degraded by inter-block feature mismatch. This pairing is central to the “++” formulation as described in the abstract: the method does not merely reuse EMA, but modifies EMA-based interaction to make it effective under local-block heterogeneity.

4. Empirical scope across vision tasks

MAN++ is validated on three categories of vision tasks: image classification, object detection, and image segmentation. The experiments are also described as using multiple network architectures, indicating that the proposal is intended as a training framework rather than as a method tied to a single backbone family (Su et al., 22 Jul 2025).

The reported outcome is that MAN++ achieves performance comparable to end-to-end training while significantly reducing GPU memory usage. The abstract therefore presents MAN++ not simply as a local-learning variant with lower resource demands, but as a method that narrows the accuracy gap that has historically limited supervised local learning. This is significant because the principal objection to local-learning methods has often been that gains in memory efficiency come at the cost of materially worse predictive performance.

The presently available description does not enumerate exact metrics, dataset-specific scores, ablation outcomes, or architecture-by-architecture breakdowns for MAN++. Consequently, the empirical characterization is currently abstract-level: broad task coverage, multiple architectures, comparable performance to end-to-end training, and significant GPU-memory reduction are reported, but quantitative tables are not specified in the provided record.

5. Relation to the original Momentum Auxiliary Network

MAN++ follows the earlier Momentum Auxiliary Network (MAN), which was introduced as a supervised local learning method that also leveraged the EMA of parameters from adjacent local blocks to enhance information flow and bridge the informational gap between blocks (Su et al., 2024). The earlier MAN paper likewise identified limitations in directly applying EMA parameters due to feature discrepancies among local blocks, and it introduced learnable biases as a corrective mechanism.

The relationship between the two papers is visible at the level of both title and scope. The earlier MAN work focused on four image classification datasets—CIFAR-10, STL-10, SVHN, and ImageNet—and reported superior performance together with substantial memory savings, including a reduction of GPU memory usage by more than 45\% on ImageNet compared to end-to-end training while achieving higher performance (Su et al., 2024). By contrast, MAN++ is explicitly framed as “scaling” Momentum Auxiliary Network for supervised local learning in vision tasks, and its abstract broadens the evaluation space from classification alone to detection and segmentation as well (Su et al., 22 Jul 2025).

This progression suggests that MAN++ should be understood as an extension of the MAN line rather than as an unrelated proposal. The continuity lies in momentum-based auxiliary interaction across local blocks; the extension lies in the explicit “learnable scaling bias” formulation and the expansion to a wider set of vision tasks and network architectures.

6. Interpretation, positioning, and limitations of the published description

MAN++ is best understood as an attempt to reconcile two objectives that are usually in tension: preserving the structural advantages of supervised local learning while alleviating the accuracy degradation caused by restricted gradient propagation. In the published description, the mechanism for doing so is neither conventional end-to-end backpropagation nor fully isolated local optimization, but momentum-mediated interaction between adjacent blocks coupled with a learned correction for feature mismatch (Su et al., 22 Jul 2025).

It should therefore not be conflated with ordinary end-to-end training augmented by an EMA teacher. The defining setting remains supervised local learning with multiple local blocks and independent auxiliary networks. EMA is used here as an inter-block communication device within that local-learning framework, not merely as a generic stabilization heuristic.

The available record also imposes clear documentary limits. The abstract specifies the motivation, the blockwise learning context, the use of adjacent-block EMA, the introduction of a learnable scaling bias, and the task domains in which the method is validated. However, it does not provide the exact formulas, quantitative results, or implementation-level details needed for a full reconstruction of the training objective or update equations. For that reason, the current scholarly understanding of MAN++ from the supplied material is strongest at the level of method definition, intended mechanism, and reported empirical scope, and weaker at the level of detailed algorithmic specification.

Within that boundary, MAN++ occupies a distinct place in the literature on local and decoupled training: it treats the lack of inter-block information exchange as the central obstacle to making supervised local learning a viable alternative to end-to-end backpropagation, and it proposes EMA-based neighboring-block interaction plus learnable feature balancing as the remedy (Su et al., 22 Jul 2025).