Motion-guided Modulation Network (MMN)

• The paper introduces MMN as a skeleton-based micro-action recognition model that leverages motion-conditioned affine modulation to amplify subtle, low-amplitude movements.
• MMN integrates feature embedding, motion-guided skeletal and temporal modulation, and multi-scale motion consistency learning through stacked MSTF blocks for effective micro-action analysis.
• Ablation studies confirm that the joint use of motion-guided skeletal (MSM) and temporal (MTM) modulation significantly improves recognition accuracy while maintaining model efficiency.

Searching arXiv for the exact MMN paper and closely related motion-guided architectures to ground the article in current arXiv records. Motion-guided Modulation Network (MMN) is a skeleton-based micro-action recognition architecture introduced in "Motion Matters: Motion-guided Modulation Network for Skeleton-based Micro-Action Recognition" (Gu et al., 29 Jul 2025). It is designed for settings in which the discriminative signal lies in subtle, low-amplitude body motion rather than coarse pose change, and it addresses this by explicitly extracting motion from skeleton sequences and using that motion to modulate learned features at both the skeletal/joint level and the temporal/frame level. The model is organized around Feature Embedding, Motion-guided Feature Modulation, and Motion Consistency Learning, with stacked Motion-guided Skeletal-Temporal Formers (MSTFs) as the core processing blocks.

1. Problem formulation and motivation

MMN is formulated for skeleton-based micro-action recognition, where the input is a skeleton sequence

$$X=\{x_1,x_2,x_3,\ldots,x_T\}\in \mathbb{R}^{T\times V\times C_{in}},$$

with $T$ frames, $V$ joints, and $C_{in}$ coordinate channels per joint. The task is written as

$Y = \Phi(X,\Theta),$

where $Y$ is the predicted micro-action category and $\Theta$ denotes the model parameters (Gu et al., 29 Jul 2025).

The paper characterizes micro-actions as subtle, low-intensity, often unconscious body movements occurring in social interaction and potentially relating to internal emotional states. It identifies three main difficulties: subtle motion amplitudes, high inter-class similarity, and large intra-class variability. The argument is that prior 2D CNNs, 3D CNNs, GCNs, and Transformers are not tailored to this degree of subtlety, and that even strong skeleton-based methods often encode general spatiotemporal dynamics without explicitly amplifying weak motion cues.

A central distinction drawn by the paper is between using motion as an auxiliary input and using motion as a guidance signal for feature learning. Explicit differencing alone, in the authors’ account, can amplify noise and does not necessarily integrate motion adaptively into learned features. MMN therefore treats motion as a conditioning signal that should influence representation learning at multiple levels: which joints matter, which frames matter, and how motion cues should be aggregated across temporal scales.

2. Architectural organization

The end-to-end MMN pipeline has three named components: Feature Embedding (FE), Motion-guided Feature Modulation (MFM), and Motion Consistency Learning (MCL) (Gu et al., 29 Jul 2025). The processing sequence given in the paper is: input a raw skeleton sequence $X_{raw}$, apply Skeleton-Temporal Context-aware Augmentation (STCA), project the skeleton into a higher-dimensional feature space, add skeletal-temporal positional embeddings, pass the result through stacked MSTFs, build multi-scale temporal features, fuse them in the motion consistency learning stage, and classify the final representation.

The backbone is described as Transformer-inspired rather than a plain Transformer. Its core block, the Motion-guided Skeletal-Temporal Former (MSTF), combines graph convolution for skeletal modeling, temporal convolution for local temporal modeling, explicit motion differencing for motion extraction, and feature modulation using motion-derived affine factors. The representation entering this stage is a hidden tensor of shape $\mathbb{R}^{T\times V\times C}$, after projection and positional embedding, followed by normalization and linear projection: $$X_{in} = Linear(LN(X_{feat})) \in R^{T \times V \times C}.$$

Within each MSTF, channels are partitioned into three branches: a skeletal branch, a motion branch, and a temporal branch. The branch split specified in the implementation details is: the first $T$0 channels for the skeletal branch, the middle $T$1 channels for the motion branch, and the last $T$2 channels for the temporal branch. Spatial modeling is handled by a GCN branch over joints, temporal modeling by a temporal convolution branch over frames, and motion modeling by frame differences on intermediate features. Motion is then converted into modulation factors that reweight and shift features in the other branches, after which the spatially modulated feature, temporal motion feature, and raw motion feature are adaptively fused.

This organization makes motion structurally central rather than peripheral. A plausible implication is that MMN was designed to avoid treating subtle motion as a weak byproduct of general spatiotemporal encoding; instead, motion explicitly governs feature transformation inside the backbone.

3. Motion representation and modulation mechanisms

The motion representation used by MMN is a consecutive-frame residual in feature space rather than a raw-coordinate velocity. After projection and embedding, the motion branch uses the middle half of channels,

$T$3

and defines motion by

$T$4

The paper therefore describes the operative motion cue as a first-order temporal difference on learned skeleton features (Gu et al., 29 Jul 2025).

The first modulation module is Motion-guided Skeletal Modulation (MSM), which injects motion cues at the skeletal level. Its output is

$T$5

followed by a split into scale and shift terms,

$T$6

MSM processes motion with graph convolution, explicitly modeling inter-joint dependencies in the motion domain. The GCN branch itself is written as

$T$7

where $T$8 is a learnable adjacency matrix, $T$9 is a learnable graph-convolution weight matrix, and $V$0 is GELU.

The second modulation module is Motion-guided Temporal Modulation (MTM), which injects motion cues at the frame level. It is defined as

$V$1

with

$V$2

Whereas MSM uses graph convolution to encode inter-joint structure, MTM uses 2D convolution on the motion tensor, followed by BatchNorm and $V$3, to capture local patterns across time and joints without explicit graph structure.

Both MSM and MTM implement motion-conditioned affine modulation. The paper gives the modulation equations as

$V$4

and

$V$5

The notation is explicitly described as somewhat inconsistent: $V$6 is not clearly defined, and the text and formulas do not fully align on which branch is modulated. What is nonetheless explicit is the mechanism itself: normalize a backbone branch feature, then apply motion-derived scale $V$7 and motion-derived shift $V$8.

The ablation results show that the affine form is not reducible to simple feature fusion. On MA-52, replacing modulation by addition yields F1$V$9 $C_{in}$0, concatenation $C_{in}$1, and Hadamard product $C_{in}$2, while full MMN reaches $C_{in}$3. Removing either affine factor also degrades performance: without scale $C_{in}$4, F1$C_{in}$5 drops to $C_{in}$6; without shift $C_{in}$7, it drops to $C_{in}$8. This supports the paper’s claim that MMN is a modulation architecture in the strict sense of motion-conditioned scale-and-shift rather than an attention-only or fusion-only design (Gu et al., 29 Jul 2025).

4. Multi-scale motion consistency and optimization

The third named component, Motion Consistency Learning (MCL), is presented as a multi-scale motion aggregation mechanism rather than as a separate consistency loss (Gu et al., 29 Jul 2025). The paper constructs $C_{in}$9 spatiotemporal features

$Y = \Phi(X,\Theta),$0

by progressively downsampling the temporal dimension by a factor of 2 at each stage. The unified temporal length after alignment is

$Y = \Phi(X,\Theta),$1

Multi-scale fusion is then written as

$Y = \Phi(X,\Theta),$2

Operationally, “consistency” refers to retaining and aggregating motion cues coherently across multiple temporal resolutions. The paper does not define a dedicated consistency loss, auxiliary motion loss, KL term, contrastive term, or regularization objective. It states only that cross-entropy loss is used: $Y = \Phi(X,\Theta),$3 This is an important interpretive boundary: the “learning” in motion consistency learning is implemented as multi-scale motion-aware fusion within an end-to-end classifier, not as a separate supervised consistency criterion.

The implementation details that are explicitly stated include 44 keypoints extracted by AlphaPose, temporal length $Y = \Phi(X,\Theta),$4, AdamW optimization, base learning rate $Y = \Phi(X,\Theta),$5, minimum learning rate $Y = \Phi(X,\Theta),$6, weight decay $Y = \Phi(X,\Theta),$7, batch size $Y = \Phi(X,\Theta),$8, cosine annealing with 3 cycles, and linear warm-up for the first 20 epochs from $Y = \Phi(X,\Theta),$9 to $Y$0. The paper does not clearly state the total number of epochs, hidden dimension $Y$1, number of scales $Y$2, number of stacked MSTF layers $Y$3, or exact kernel sizes.

The data augmentation scheme is named Skeleton-Temporal Context-aware Augmentation (STCA). The skeletal augmentation is

$Y$4

where $Y$5 is a rotation matrix with $Y$6, $Y$7 is scaling, and $Y$8 is translation. The temporal augmentation is

$Y$9

The ablation results indicate complementarity between the two components: no augmentation yields F1$\Theta$0 $\Theta$1, skeletal only $\Theta$2, temporal only $\Theta$3, and both $\Theta$4.

5. Experimental evaluation

MMN is evaluated on Micro-Action 52 (MA-52) and iMiGUE (Gu et al., 29 Jul 2025). MA-52 is described as a large-scale whole-body micro-action dataset with 22,422 samples, 52 action-level classes, and 7 body-level classes, split into 11,250 training, 5,586 validation, and 5,586 test samples. iMiGUE is an upper-body micro-gesture dataset with 32 categories and splits of 12,893 training, 777 validation, and 4,562 test samples. Reported metrics include Top-1 accuracy, Top-5 accuracy, F1$\Theta$5, F1$\Theta$6 at body and action granularity, and

$\Theta$7

On MA-52, MMN reports the following results. For the joint modality, body Top-1 is 77.77, action Top-1 is 61.33, action Top-5 is 89.22, and F1$\Theta$8 is 64.73. For the bone modality, body Top-1 is 75.47, action Top-1 is 57.50, action Top-5 is 86.54, and F1$\Theta$9 is 60.07. For the joint+bone ensemble (2M), body Top-1 is 78.52, action Top-1 is 62.71, action Top-5 is 89.83, F1$X_{raw}$0 is 71.86, F1$X_{raw}$1 is 78.52, F1$X_{raw}$2 is 48.27, F1$X_{raw}$3 is 62.71, and F1$X_{raw}$4 is 65.34.

Among the skeleton baselines listed in the paper, MMN sets the best reported skeleton-based result on MA-52 in action-level Top-1 and F1$X_{raw}$5. The examples provided are FR-Head (2M) with action Top-1 $X_{raw}$6 and F1$X_{raw}$7 $X_{raw}$8, SkateFormer (2M) with action Top-1 $X_{raw}$9 and F1$\mathbb{R}^{T\times V\times C}$0 $\mathbb{R}^{T\times V\times C}$1, and MMN (2M) with action Top-1 $\mathbb{R}^{T\times V\times C}$2 and F1$\mathbb{R}^{T\times V\times C}$3 $\mathbb{R}^{T\times V\times C}$4. On iMiGUE, MMN (2M) achieves Top-1 $\mathbb{R}^{T\times V\times C}$5 and Top-5 $\mathbb{R}^{T\times V\times C}$6, which the paper reports as the best among the compared skeleton methods.

Efficiency is also reported for MA-52 joint modality: 1.23M parameters, 1.48G FLOPs, 7.15 ms runtime, and F1$\mathbb{R}^{T\times V\times C}$7 $\mathbb{R}^{T\times V\times C}$8. The paper interprets this as showing that the model is both more accurate and lighter/faster than several baselines.

The ablation studies support the two modulation modules directly. Starting from a baseline F1$\mathbb{R}^{T\times V\times C}$9 of 61.08, MSM only yields 61.67, MTM only yields 62.57, and MSM+MTM yields 64.73. The stated conclusion is that both modules help, MTM contributes more individually, and the combination is strongest. The qualitative analysis further reports that deeper MSTF blocks progressively focus on informative joints and times; for example, later features emphasize facial joints for “turning head” and the legs, especially the more active side, for “closing legs.”

6. Nomenclature, related motion-guided methods, and common confusions

The term Motion-guided Modulation Network (MMN) refers specifically to the 2025 skeleton-based micro-action recognition method described above (Gu et al., 29 Jul 2025). It should not be conflated with MMNet, the 2022 muscle motion-guided network for facial micro-expression recognition, which uses a two-branch design with continuous attention and position calibration and is explicitly not named “Motion-guided Modulation Network” or “MMN” (Li et al., 2022). In that earlier work, the closest modulation-like operations are motion-guided spatial reweighting in the continuous attention block and additive positional calibration, but the paper itself frames these as attention and calibration rather than modulation.

A second nearby line of work is the Past and Future Motion Guided Network for audio-visual event localization (Chen et al., 2022). Although that paper does not use the name MMN, its motion-guided audio attention module applies residual multiplicative reweighting to audio features using temporal and channel-wise masks derived from visual motion. This makes it strongly MMN-like in mechanism, but it is a multimodal AVEL architecture rather than a skeleton-based micro-action recognizer.

Other motion-guided systems share the general principle of letting motion control downstream processing without instantiating an explicit MMN. VideoControlNet uses motion-guided P-frame generation and B-frame interpolation to determine where generated content is propagated and where diffusion inpainting is invoked, but it does not define a module called MMN or a trainable modulation network (Hu et al., 2023). EV-MGRFlowNet uses motion-guided recurrent encoding and flow-guided decoding for unsupervised event-based optical flow, yet likewise does not introduce explicit channel-wise or affine feature modulation under the MMN name (Zhuang et al., 2023).

Within MMN itself, two interpretive cautions are explicit. First, “motion consistency learning” is not accompanied by a dedicated consistency loss; it is a multi-scale aggregation mechanism optimized only with cross-entropy. Second, several architectural details remain underspecified or notationally inconsistent, including symbols such as $$X_{in} = Linear(LN(X_{feat})) \in R^{T \times V \times C}.$$0, exact branch-target modulation assignments, hidden dimension $$X_{in} = Linear(LN(X_{feat})) \in R^{T \times V \times C}.$$1, number of MSTF blocks $$X_{in} = Linear(LN(X_{feat})) \in R^{T \times V \times C}.$$2, and number of temporal scales $$X_{in} = Linear(LN(X_{feat})) \in R^{T \times V \times C}.$$3. This suggests that MMN is conceptually well specified at the level of motion-conditioned affine modulation, but not fully specified in every low-level implementation detail.

In technical terms, MMN’s distinguishing contribution is to formalize motion as a control signal for normalized affine transformation of skeleton features. MSM encodes where subtle motion occurs through graph-structured motion processing, MTM encodes when subtle motion matters through convolutional temporal processing, and MCL aggregates motion-aware features across temporal scales. The resulting formulation places explicit motion modeling at the center of representation learning for micro-action recognition rather than treating it as an auxiliary or precomputed side input.