Channel-wise Motion Features in Action Recognition

• Channel-wise motion features are representations that process and preserve motion information separately in each feature channel, enabling diverse motion semantics.
• They employ mechanisms like channel-specific excitation, graph topology refinement, and depth aggregation to enhance performance in action recognition, segmentation, and medical imaging.
• Empirical studies show that preserving channel heterogeneity improves discriminative accuracy and computational efficiency across various motion-aware tasks.

Searching arXiv for the cited works to ground the article in current records. Search query: (Chen et al., 2021) Channel-wise Topology Refinement Graph Convolution for Skeleton-Based Action Recognition Channel-wise motion features are motion representations in which the motion signal is modeled, preserved, reweighted, or aggregated at the level of feature channels rather than being treated as a single global temporal cue. Across recent literature, the term encompasses several related but distinct mechanisms: channel-wise excitation of motion-sensitive channels in RGB video recognition, channel-specific graph topologies for skeleton motion, channel-preserving aggregation of motion cost volumes for segmentation, approximate motion preservation in each latent channel of intermediate tensors, and, in some cases, block-level rather than strictly channel-level localization of motion information (Wu et al., 2021). The common premise is that different channels need not encode the same motion semantics, and that preserving or modulating this heterogeneity can improve discriminative motion modeling, controllability, or computational efficiency (Chen et al., 2021).

1. Conceptual scope and definitions

In the most direct sense, channel-wise motion features arise when motion information is represented or modulated separately for individual channels of a feature tensor. This differs from topology-shared graph aggregation, whole-feature-map differencing, or cost-volume pipelines that collapse channels early. The distinction is explicit in skeleton-based action recognition, where CTR-GC replaces a single adjacency matrix shared by all output channels with a channel-wise topology tensor, and in video action recognition, where CME and CMEM generate per-frame channel gates that emphasize channels related to dynamic information (Chen et al., 2021).

A second, closely related usage treats motion as approximately preserved in each channel of an intermediate representation. In collaborative intelligence, the latent-space motion analysis argues that the motion visible in the input is approximately preserved within each individual channel of the latent feature tensor, primarily differing by a spatial scale factor induced by downsampling or pooling (Ulhaq et al., 2021). Under this view, channel-wise motion is not a learned attention weight but a structural property of latent representations.

A third usage preserves channel identity while aggregating motion evidence along another axis. The motion-segmentation work on Channel-wise Motion Features defines a cost-volume-based representation that starts from a 4D cost volume of size $D \times C \times H \times W$ and aggregates along the depth direction for each channel to produce a feature map of size $C \times H \times W$ (Inoue et al., 17 Jul 2025). Here the essential operation is not channel scoring but channel-preserving depth aggregation.

Not all relevant work is strictly channel-wise. Moaw analyzes motion sensitivity at the level of feature groups corresponding to U-Net residual outputs from down blocks and the mid block, and explicitly does not perform a per-channel importance analysis (Zhang et al., 19 Jan 2026). That distinction is important: some papers provide evidence for where motion information is concentrated in a network, but only at block level.

2. Channel-wise excitation in RGB video action recognition

A canonical formulation appears in “Learning Comprehensive Motion Representation for Action Recognition” (Wu et al., 2021). CME operates on frame-level feature maps

$$\{\mathbf{x}_t\}_{t=1}^T,\qquad \mathbf{x}_t \in \mathbb{R}^{C \times H \times W},$$

and reweights them through

$\mathbf{u}_t = \mathbf{x}_t \odot \mathbf{a}_t,$

where $\mathbf{a}_t \in \mathbb{R}^{C}$ is a channel enhancement vector. The gate for frame $t$ is not computed from that frame alone. Instead, the module forms compressed channel descriptors, computes discrepancy between frame $t$ and frame $j$ as

$d_{tj} = -\mathbf{k}_t \mathbf{k}_j^\top,$

normalizes the discrepancy vector with softmax, and fuses information from all frames:

$$\hat{\mathbf{z}}_t = \mathbf{z}_t + \sum_{j=1}^{T} \hat{d}_{tj}\mathbf{z}_j.$$

The final gate is

$C \times H \times W$0

This makes channel importance a function of the whole clip rather than of a single frame or only adjacent frames (Wu et al., 2021).

The stated rationale is that different channels often encode different semantic patterns, and that channels associated with motion are more informative than those dominated by background. The paper therefore treats temporal diversity in channel responses as a proxy for motion salience. Its ablations on Something-Something V1 report that a ResNet-50 + TIM baseline at $C \times H \times W$1 top-1 rises to $C \times H \times W$2 when CME is added, and that CME outperforms MEM at $C \times H \times W$3 and PEM at $C \times H \times W$4 under the same baseline (Wu et al., 2021). The visualizations described in the paper further state that top-weighted channels focus on the moving target, while bottom-weighted channels are sensitive to static background.

A related but distinct formulation appears in IMG, where CMEM performs channel attention-based short-term motion feature enhancement (Zhang et al., 2022). Starting from

$C \times H \times W$5

the module first reduces channels,

$C \times H \times W$6

then computes two adjacent-frame motion descriptors: a difference feature

$C \times H \times W$7

and a cosine-similarity feature

$C \times H \times W$8

After spatial pooling of $C \times H \times W$9, the motion descriptors are fused as

$$\{\mathbf{x}_t\}_{t=1}^T,\qquad \mathbf{x}_t \in \mathbb{R}^{C \times H \times W},$$0

expanded back to channel dimension, and applied with residual reweighting:

$$\{\mathbf{x}_t\}_{t=1}^T,\qquad \mathbf{x}_t \in \mathbb{R}^{C \times H \times W},$$1

The stated motivation is that difference highlights changing regions or channels but may cause polarization, whereas cosine similarity captures overall similarity between adjacent frames and makes training smoother (Zhang et al., 2022).

The empirical evidence in IMG directly links channel-wise enhancement to short-range motion modeling. On Something-Something, CMEM with ResNet improves top-1 from $$\{\mathbf{x}_t\}_{t=1}^T,\qquad \mathbf{x}_t \in \mathbb{R}^{C \times H \times W},$$2 to $$\{\mathbf{x}_t\}_{t=1}^T,\qquad \mathbf{x}_t \in \mathbb{R}^{C \times H \times W},$$3, and CMEM with Res2Net improves top-1 from $$\{\mathbf{x}_t\}_{t=1}^T,\qquad \mathbf{x}_t \in \mathbb{R}^{C \times H \times W},$$4 to $$\{\mathbf{x}_t\}_{t=1}^T,\qquad \mathbf{x}_t \in \mathbb{R}^{C \times H \times W},$$5 (Zhang et al., 2022). The paper further reports that CMEM alone already exceeds TEA in both efficient and precise evaluation settings.

These two lines of work differ in temporal scope. CME uses information from all other frames in the clip, whereas CMEM focuses on adjacent-frame motion descriptors and then relies on CLIM for larger-scale temporal aggregation (Wu et al., 2021). This suggests that “channel-wise motion features” can refer either to globally fused temporal channel attention or to locally computed channel attention that is later integrated over longer ranges.

3. Channel-specific graph topologies for skeleton motion

In skeleton-based action recognition, channel-wise motion features are formulated through graph topology rather than through channel attention. CTR-GC begins from the criticism that existing skeleton GCNs usually use a single graph topology for all feature channels, so all channels aggregate neighbor information in the same way even though different channels may encode different semantic or motion patterns (Chen et al., 2021). The paper’s response is to make graph topology channel-dependent.

The model learns a shared topology

$$\{\mathbf{x}_t\}_{t=1}^T,\qquad \mathbf{x}_t \in \mathbb{R}^{C \times H \times W},$$6

and channel-specific correlations

$$\{\mathbf{x}_t\}_{t=1}^T,\qquad \mathbf{x}_t \in \mathbb{R}^{C \times H \times W},$$7

which are combined by refinement:

$$\{\mathbf{x}_t\}_{t=1}^T,\qquad \mathbf{x}_t \in \mathbb{R}^{C \times H \times W},$$8

The full graph convolution is written as

$$\{\mathbf{x}_t\}_{t=1}^T,\qquad \mathbf{x}_t \in \mathbb{R}^{C \times H \times W},$$9

with feature transform

$\mathbf{u}_t = \mathbf{x}_t \odot \mathbf{a}_t,$0

For each edge $\mathbf{u}_t = \mathbf{x}_t \odot \mathbf{a}_t,$1, the channel-specific correlation vector is

$\mathbf{u}_t = \mathbf{x}_t \odot \mathbf{a}_t,$2

and stacking all $\mathbf{u}_t = \mathbf{x}_t \odot \mathbf{a}_t,$3 gives $\mathbf{u}_t = \mathbf{x}_t \odot \mathbf{a}_t,$4. The paper explicitly notes that $\mathbf{u}_t = \mathbf{x}_t \odot \mathbf{a}_t,$5 is not forced to be symmetric, allowing directional relations (Chen et al., 2021).

The operational distinction appears in channel-wise aggregation. For channel $\mathbf{u}_t = \mathbf{x}_t \odot \mathbf{a}_t,$6, CTR-GC uses a channel-specific adjacency $\mathbf{u}_t = \mathbf{x}_t \odot \mathbf{a}_t,$7 and propagates the transformed scalar feature column $\mathbf{u}_t = \mathbf{x}_t \odot \mathbf{a}_t,$8 through that channel’s own graph. The output concatenates the propagated channels:

$\mathbf{u}_t = \mathbf{x}_t \odot \mathbf{a}_t,$9

This is the paper’s core difference from topology-shared methods such as ST-GCN, AGCN, or Dynamic GCN (Chen et al., 2021).

The unified graph-convolution view sharpens the channel-wise argument. The paper rewrites graph convolutions as

$\mathbf{a}_t \in \mathbb{R}^{C}$0

and states that dynamic topology-non-shared GCs, the class to which CTR-GC belongs, have the least restrictive generalized weight structure. Its reformulation is

$\mathbf{a}_t \in \mathbb{R}^{C}$1

where $\mathbf{a}_t \in \mathbb{R}^{C}$2 is the channel-wise, sample-specific topology coefficient. This gives channel-wise motion modeling a graph-theoretic interpretation: different channels are allowed to induce different joint dependency patterns for the same sample (Chen et al., 2021).

The reported results support the claim that channel-wise topology refinement, rather than naive per-channel parameterization, is beneficial. On NTU RGB+D 120 X-Sub with a controlled backbone, the paper reports ST-GC at $\mathbf{a}_t \in \mathbb{R}^{C}$3, AGC at $\mathbf{a}_t \in \mathbb{R}^{C}$4, Dy-GC at $\mathbf{a}_t \in \mathbb{R}^{C}$5, DC-GC at $\mathbf{a}_t \in \mathbb{R}^{C}$6, DC-GC* at $\mathbf{a}_t \in \mathbb{R}^{C}$7 with $\mathbf{a}_t \in \mathbb{R}^{C}$8M parameters, and CTR-GC at $\mathbf{a}_t \in \mathbb{R}^{C}$9 with $t$0M parameters (Chen et al., 2021). The ablation shows that removing the channel-specific term $t$1 lowers performance to $t$2, while removing the shared prior $t$3 gives $t$4; the combination reaches $t$5. The paper also notes that hard-class analysis shows gains on subtle finger-hand interactions such as “typing on a keyboard,” “cut nails,” and “open bottle,” supporting the claim that channel-wise topology refinement helps capture fine-grained motion relations.

4. Semantic and physiological channel-wise motion modeling

In echocardiogram analysis, channel-wise motion features are tied to a highly specific physiological target: left ventricular contraction and relaxation across the cardiac cycle. The semantic-aware temporal channel-wise attention paper argues that existing video-based methods neither exploit LV segmentation as a representation-learning aid nor explicitly model the channels that respond to motion changes in the LV region (Chen et al., 2023).

The TCA module operates on

$t$6

It applies local max-pooling and mean-pooling over adjacent frames to obtain

$t$7

globally pools over space to form $t$8, and defines the motion descriptor by subtraction:

$t$9

The excitation map is then

$t$0

with $t$1 and $t$2 for reduction ratio $t$3. The feature modulation is

$t$4

The stated intuition is that appearance-dominated channels should respond similarly under local max and mean temporal pooling, while motion-sensitive channels should not (Chen et al., 2023).

The semantic-aware variant S-TCA uses the predicted LV mask as a guidance signal to conceal the trivial region and drive attention computation by LV-centric statistics rather than by the whole image. The paper does not provide a separate explicit formula for S-TCA, but it states that the end-diastolic LV mask is enlarged by dilation implemented with max-pooling and that the segmentation map is used to focus on motion patterns of the left ventricle (Chen et al., 2023). In this formulation, channel-wise motion features are not only temporal but also semantically localized.

The model couples this with an auxiliary segmentation branch and an anchor-based classification-and-regression head for LVEF prediction. The main training objective is

$t$5

with $t$6. The paper reports a progressive improvement from $t$7 to $t$8: MAE $t$9, RMSE $j$0, and $j$1 $j$2 (Chen et al., 2023). It also compares TCA against SE and ME, reporting MAE/RMSE/$j$3 of $j$4 for SE, $j$5 for ME, and $j$6 for TCA.

The paper’s interpretation is domain-specific: channel-wise motion modeling is meaningful because spatiotemporal CNN channels mix static anatomy, generic textures, and contraction or expansion dynamics. TCA attempts to excite those channels used to describe motion, while S-TCA constrains that excitation to the LV region (Chen et al., 2023).

5. Channel-wise motion in latent, generative, and correspondence-based representations

The collaborative-intelligence study provides a more structural account of channel-wise motion. Rather than learning channel weights, it analyzes the effect of common DNN operations on optical flow and concludes that motion in latent feature channels is approximately the same as input motion except for reduction in spatial magnitude caused by downsampling:

$j$7

where $j$8 is input motion, $j$9 is motion in a feature tensor channel, $d_{tj} = -\mathbf{k}_t \mathbf{k}_j^\top,$0 is the pooling window size, and $d_{tj} = -\mathbf{k}_t \mathbf{k}_j^\top,$1 is the number of pooling operations between the input and the layer of interest (Ulhaq et al., 2021). The paper explicitly states that at the corresponding spatial location in all the channels of the feature tensor one can expect to find the vector $d_{tj} = -\mathbf{k}_t \mathbf{k}_j^\top,$2.

This conclusion is derived by showing that convolution, pointwise nonlinearity, and local max are approximately motion-preserving, while scale change transforms motion by

$d_{tj} = -\mathbf{k}_t \mathbf{k}_j^\top,$3

The paper validates the scaling law with ResNet-34 and DenseNet-121, reports NRMSE

$d_{tj} = -\mathbf{k}_t \mathbf{k}_j^\top,$4

and states that NRMSE remains low, reaching about $d_{tj} = -\mathbf{k}_t \mathbf{k}_j^\top,$5 over reasonable transformation ranges (Ulhaq et al., 2021). In this literature, channel-wise motion features are shared-motion latent channels rather than differently specialized motion channels.

MotionSqueeze occupies an intermediate position. It learns explicit motion by local feature correlation and displacement estimation, but it does not maintain a per-channel flow field. Given adjacent feature maps $d_{tj} = -\mathbf{k}_t \mathbf{k}_j^\top,$6, it defines a local correlation score

$d_{tj} = -\mathbf{k}_t \mathbf{k}_j^\top,$7

estimates displacement with soft-argmax or kernel-soft-argmax, computes a confidence

$d_{tj} = -\mathbf{k}_t \mathbf{k}_j^\top,$8

forms a 3-channel tensor of displacement and confidence, decodes it into a $d_{tj} = -\mathbf{k}_t \mathbf{k}_j^\top,$9-channel motion feature map, and fuses by residual addition

$$\hat{\mathbf{z}}_t = \mathbf{z}_t + \sum_{j=1}^{T} \hat{d}_{tj}\mathbf{z}_j.$$0

The paper explicitly states that the core correspondence score is a dot product aggregated across channels and that the motion bottleneck is squeezed to 3 channels, so the method is better described as a feature-map-wise motion encoder with channel-projected matching and channel-restored output than as a truly channel-wise motion representation (Kwon et al., 2020).

Moaw extends the discussion to video diffusion models. It analyzes 13 intermediate feature tensors—three residual outputs from each of four down blocks and one mid-block tensor—and reports that the first five features do not separate motion classes clearly after PCA, that clustering begins from the sixth feature onward, that all features from the third down block show strong clustering, that fourth down block features also show clustering, and that the mid-block feature is less separable (Zhang et al., 19 Jan 2026). The paper then selects the third and fourth down blocks as a whole and injects the corresponding residual features into a structurally identical generation U-Net by direct residual addition. It also explicitly states that channel-wise analysis is absent. This suggests that motion information can be highly localized in representation depth without being decomposed into explicit channel-wise importance scores.

6. Channel-preserving motion features for efficient motion segmentation

The motion-segmentation paper titled “Channel-wise Motion Features for Efficient Motion Segmentation” gives the most literal use of the term (Inoue et al., 17 Jul 2025). The method addresses class-agnostic motion segmentation from two consecutive monocular frames and is motivated by the computational burden of pipelines that jointly estimate depth, pose, optical flow, and scene flow. Its central claim is that a 4D cost volume can be transformed into an efficient motion representation by aggregating along the depth direction while preserving channel identity.

The pipeline uses two frames

$$\hat{\mathbf{z}}_t = \mathbf{z}_t + \sum_{j=1}^{T} \hat{d}_{tj}\mathbf{z}_j.$$1

a Pose Network that predicts relative pose

$$\hat{\mathbf{z}}_t = \mathbf{z}_t + \sum_{j=1}^{T} \hat{d}_{tj}\mathbf{z}_j.$$2

and a shared-weight feature extractor that produces source and target feature maps. For each sampled depth $$\hat{\mathbf{z}}_t = \mathbf{z}_t + \sum_{j=1}^{T} \hat{d}_{tj}\mathbf{z}_j.$$3, the source feature map is warped into the target view:

$$\hat{\mathbf{z}}_t = \mathbf{z}_t + \sum_{j=1}^{T} \hat{d}_{tj}\mathbf{z}_j.$$4

The 4D cost volume is then

$$\hat{\mathbf{z}}_t = \mathbf{z}_t + \sum_{j=1}^{T} \hat{d}_{tj}\mathbf{z}_j.$$5

Conventional multi-frame depth methods typically aggregate over channels to obtain a $$\hat{\mathbf{z}}_t = \mathbf{z}_t + \sum_{j=1}^{T} \hat{d}_{tj}\mathbf{z}_j.$$6 volume; CMF instead aggregates over depth to obtain a $$\hat{\mathbf{z}}_t = \mathbf{z}_t + \sum_{j=1}^{T} \hat{d}_{tj}\mathbf{z}_j.$$7 representation (Inoue et al., 17 Jul 2025).

The paper does not provide a single closed-form equation for the learned aggregation, but it states that the 3D Motion Extraction Network transforms

$$\hat{\mathbf{z}}_t = \mathbf{z}_t + \sum_{j=1}^{T} \hat{d}_{tj}\mathbf{z}_j.$$8

into

$$\hat{\mathbf{z}}_t = \mathbf{z}_t + \sum_{j=1}^{T} \hat{d}_{tj}\mathbf{z}_j.$$9

Guided cost volume excitation is written as

$C \times H \times W$00

The stated intuition is that feature channels from a SparseInst-like backbone already tend to emphasize different object instances or object-specific cues, so preserving per-channel depth-matching behavior yields a compact descriptor of each instance’s 3D motion evidence (Inoue et al., 17 Jul 2025).

This representation is explicitly efficient because the only motion-related subnetwork used to build CMF online is the Pose Network. The paper states that no depth, optical flow, or scene flow network is used during motion segmentation training or inference. On KITTI, it reports parameters/FPS/Obj F/Bg IoU of $C \times H \times W$01M, $C \times H \times W$02, $C \times H \times W$03, and $C \times H \times W$04 for the proposed method, compared with $C \times H \times W$05M, $C \times H \times W$06, $C \times H \times W$07, and $C \times H \times W$08 for RigidMask (Inoue et al., 17 Jul 2025). On VCAS-Motion Cityscapes, it reports $C \times H \times W$09M $C \times H \times W$10 $C \times H \times W$11M parameters, $C \times H \times W$12 FPS, and CAQ $C \times H \times W$13, compared with VCANet at $C \times H \times W$14M $C \times H \times W$15 $C \times H \times W$16M, $C \times H \times W$17 FPS, and CAQ $C \times H \times W$18. The paper characterizes this as about $C \times H \times W$19 the FPS with parameters reduced to about $C \times H \times W$20 while maintaining equivalent accuracy on Cityscapes.

The key ablation directly tests the channel-preserving claim. Averaging the 4D cost volume over channels to obtain a conventional 3D cost volume yields, on VCAS-Motion KITTI, SQ $C \times H \times W$21, RQ $C \times H \times W$22, and CAQ $C \times H \times W$23, whereas CMF yields SQ $C \times H \times W$24, RQ $C \times H \times W$25, and CAQ $C \times H \times W$26 (Inoue et al., 17 Jul 2025). On Cityscapes, the corresponding numbers are $C \times H \times W$27 for the 3D cost volume and $C \times H \times W$28 for CMF. Within the terms of the paper, this is direct evidence that preserving channel identity improves instance-level motion segmentation.

7. Limitations, ambiguities, and recurring themes

A recurring limitation is that channel-wise motion is not always synonymous with per-channel motion semantics. The latent-space analysis argues that all channels approximately share the same motion geometry after scale change, which is a different claim from channel specialization (Ulhaq et al., 2021). MotionSqueeze preserves channel alignment in its output but compresses motion to a shared 3-channel geometric bottleneck, so it is not a per-channel flow model (Kwon et al., 2020). Moaw localizes motion information mainly at block level rather than channel level (Zhang et al., 19 Jan 2026).

Another recurring theme is that channel-wise mechanisms often require complementary structure. CME is paired with SME because channel-only scaling cannot localize the moving region within a channel (Wu et al., 2021). CMEM is paired with CLIM because short-range channel enhancement alone does not model large-scale associations of long time sequences (Zhang et al., 2022). TCA becomes semantic-aware only after segmentation guidance restricts the attention computation to the left ventricle (Chen et al., 2023). CTR-GC relies on a shared prior plus channel-specific refinement because learning unrelated per-channel adjacency matrices directly is parameter-heavy and difficult to optimize (Chen et al., 2021).

The literature also draws a consistent distinction between efficient channel-aware motion modeling and heavier explicit geometry pipelines. MotionSqueeze is motivated by replacing external optical flow with internal learned motion features (Kwon et al., 2020). CMF avoids online depth, optical flow, and scene flow estimation by using only a Pose Network to construct ego-motion-conditioned cost volumes (Inoue et al., 17 Jul 2025). In contrast, CTR-GC improves expressivity not by adding external motion estimators but by relaxing topology-sharing constraints inside the graph operator itself (Chen et al., 2021).

Taken together, these works indicate that channel-wise motion features are not a single method family but a representational principle. In some settings the principle means channel excitation from temporal discrepancies; in others it means channel-specific graph topology, channel-preserving cost-volume aggregation, or approximate motion preservation in latent channels. A plausible implication is that the practical value of channel-wise motion modeling depends on what the channel index already encodes: dynamic semantics in video CNNs, joint dependencies in skeleton GCNs, anatomy-specific variation in medical video, instance-sensitive activations in segmentation backbones, or structurally aligned internal tensors in diffusion U-Nets (Inoue et al., 17 Jul 2025).