LMAC-Net: Long-term Multimodal AQA Network

• The paper introduces a novel neural architecture that fuses RGB, optical flow, and audio data using modality-specific temporal query extraction and explicit cross-modal attention constraints.
• It achieves state-of-the-art performance on artistic sports datasets, significantly improving Spearman’s ρ metrics while enabling real-time, interpretable scoring through a two-level evaluation framework.
• The model’s design, featuring efficient segmentation and adaptive weighting of stage scores, provides actionable insights and potential applicability to broader multimodal video analysis tasks.

The Long-term Multimodal Attention Consistency Network (LMAC-Net) is a neural architecture for long-term action quality assessment (AQA) in video, specifically designed to handle multi-minute activities where both motion execution and synchronization with music are crucial. LMAC-Net’s principal innovations are: (1) modality-specific temporal query extraction via a multimodal local query encoder, (2) explicit cross-modal attention consistency constraints ensuring tight temporal alignment of visual and audio cues, and (3) a two-level, interpretable scoring framework yielding both global and stage-level performance estimates. It achieves state-of-the-art performance on artistic sports datasets, efficiently fusing complementary information from RGB, optical flow, and audio signals (Wang et al., 29 Jul 2025).

1. Problem Setting and Limitations of Prior Approaches

Long-term AQA tasks, typified by artistic sports such as rhythmic gymnastics and figure skating, demand models that can integrate both the fine details of motion (captured in visual streams) and the temporal structure of accompanying audio. Existing approaches fall into two principal categories:

• Unimodal methods: Operate exclusively on visual features (RGB/flow), thus ignoring audio cues critical for scoring synchronized routines.
• Multimodal methods: Often employ simple feature-level fusion or contrastive objectives, typically insufficient for modeling deep cross-modal collaboration or maintaining temporal sensitivity across extended sequences.

These limitations result in inadequate modeling of complex intermodal interactions, hampering the tracking of subtle, temporally-localized performance changes throughout lengthy videos (Wang et al., 29 Jul 2025).

2. Model Architecture and Data Flow

LMAC-Net inputs three synchronized streams: RGB frames, optical flow, and audio. The workflow comprises the following sequential stages:

1. Segmentation: Input streams are divided into $T$ non-overlapping clips (32 frames per video segment at 25 fps, temporally matched audio windows).
2. Feature Extraction: Segment-level features are extracted using fine-tuned frozen backbones:
   • RGB via VST: $f_t^{RGB} \in \mathbb{R}^{d_{RGB}}$
   • Optical flow via I3D: $f_t^{Flow} \in \mathbb{R}^{d_{Flow}}$
   • Audio via AST: $f_t^{Audio} \in \mathbb{R}^{d_{Audio}}$
3. Multimodal Local Query Encoder (MLQE): Each modality is processed by a dedicated $L=2$ layer transformer decoder. Each branch utilizes $K=5$ learnable query vectors of dimension $d_m$, capturing temporal and cross-modal semantics via cross-attention over the $T$ feature segments:

$$p_k = [p_k^{RGB} \,\|\, p_k^{Flow} \,\|\, p_k^{Audio}] \in \mathbb{R}^{d_{multi}}$$

1. Two-Level Score Evaluation: The $K$ fused query features are each linearly regressed to a "stage score" $f_t^{RGB} \in \mathbb{R}^{d_{RGB}}$0, which are then adaptively weighted (softmax normalized) and aggregated to a single global quality score $f_t^{RGB} \in \mathbb{R}^{d_{RGB}}$1:

$f_t^{RGB} \in \mathbb{R}^{d_{RGB}}$2

2. Training Objective: A composite loss combines mean square error on the global score with feature-level regularizers enforcing (a) temporal ranking, (b) sparsity of attention, and (c) cross-modal consistency of temporal attention allocation, as detailed in Section 4.

3. Multimodal Local Query Encoding and Attention Consistency

The MLQE module enforces temporal and cross-modal alignment through the following mechanisms:

• Cross-Attention Querying: For modality $f_t^{RGB} \in \mathbb{R}^{d_{RGB}}$3 and decoder level $f_t^{RGB} \in \mathbb{R}^{d_{RGB}}$4, each query $f_t^{RGB} \in \mathbb{R}^{d_{RGB}}$5 attends over the $f_t^{RGB} \in \mathbb{R}^{d_{RGB}}$6 input features $f_t^{RGB} \in \mathbb{R}^{d_{RGB}}$7 via a temperature-controlled attention mechanism:

$f_t^{RGB} \in \mathbb{R}^{d_{RGB}}$8

• Attention Center Calculation: For each modality and query, attention distributions across $f_t^{RGB} \in \mathbb{R}^{d_{RGB}}$9 segments produce an "attention center"

$f_t^{Flow} \in \mathbb{R}^{d_{Flow}}$0

representing the effective temporal focus of that query.

• Cross-Modal Consistency Constraint: The model penalizes discrepancies between the attention centers of queries across modalities:

$f_t^{Flow} \in \mathbb{R}^{d_{Flow}}$1

This enforces synchronized temporal focus across streams for each query.

4. Loss Functions and Training Strategy

LMAC-Net is trained end-to-end with the following composite loss:

$f_t^{Flow} \in \mathbb{R}^{d_{Flow}}$2

where:

• Decision-level loss ($f_t^{Flow} \in \mathbb{R}^{d_{Flow}}$3): MSE between predicted and true global scores.
• Feature-level loss ($f_t^{Flow} \in \mathbb{R}^{d_{Flow}}$4): Sum of
  • Temporal ranking loss ($f_t^{Flow} \in \mathbb{R}^{d_{Flow}}$5): Enforces monotonicity and boundary constraints on the attention centers.
  • Sparsity loss ($f_t^{Flow} \in \mathbb{R}^{d_{Flow}}$6): Encourages attention to concentrate on a limited number of critical segments.
  • Cross-modal consistency loss ($f_t^{Flow} \in \mathbb{R}^{d_{Flow}}$7): Drives modalities to share temporal focus.

Hyperparameters are set via cross-validation, with typical values $f_t^{Flow} \in \mathbb{R}^{d_{Flow}}$8, $f_t^{Flow} \in \mathbb{R}^{d_{Flow}}$9, and feature-regularization weights around $f_t^{Audio} \in \mathbb{R}^{d_{Audio}}$0.

5. Experimental Benchmarks and Comparative Analysis

Tests were conducted on the Rigid Gymnastics (RG) and Figure Skating (Fis-V) datasets, containing multi-minute, 25 fps artistic routines with ground truth scores normalized to $f_t^{Audio} \in \mathbb{R}^{d_{Audio}}$1. Main findings include:

• Performance Metrics: Spearman’s $f_t^{Audio} \in \mathbb{R}^{d_{Audio}}$2, averaged by Fisher’s $f_t^{Audio} \in \mathbb{R}^{d_{Audio}}$3-transform.
• State-of-the-art Results: LMAC-Net attains significantly higher correlation than both unimodal and prior multimodal baselines. For instance, RG average $f_t^{Audio} \in \mathbb{R}^{d_{Audio}}$4 and Fis-V average $f_t^{Audio} \in \mathbb{R}^{d_{Audio}}$5, outperforming PAMFN (0.819/0.822) and prior unimodal GDLT (0.765/0.761).
• Ablation Studies: Addition of MLQE, ranking, sparsity, and consistency losses provides consistent and significant improvements, with cross-modal attention consistency providing the largest boost (final $f_t^{Audio} \in \mathbb{R}^{d_{Audio}}$6).
• Alignment Strategies: LMAC consistency achieves superior alignment relative to vanilla cross-attention, dynamic variants, and contrastive approaches.
• Efficiency: LMAC-Net processes a sample with 8.95M parameters and 0.419 GFLOPs in 4 ms, supporting real-time inference.

Method	Params (M)	FLOPs (G)	Latency (ms)
LMAC-Net	8.95	0.419	4
PAMFN	17.96	0.721	36
ACTION-NET	3.54	0.227	1
GDLT	1.85	0.075	3

6. Interpretability and Qualitative Findings

• Attention Center Visualization: Without the consistency loss, attention centers across modalities diverge during training. The introduction of $f_t^{Audio} \in \mathbb{R}^{d_{Audio}}$7 causes steady convergence to overlapping temporal foci (cosine similarity near 1.0).
• Query Saliency and Modality Role: Heatmaps of cross-attention weights confirm that distinct queries sparsely attend to key sequence segments; e.g., RGB to pose, Flow to trajectory, Audio to temporal beats. Post-consistency loss, modalities’ peak attention aligns temporally but preserves complementary detail.
• Two-Level Scoring Interpretability: The per-query scores and their adaptive weights identify contribution of specific temporal segments, enabling stage-level feedback in addition to global actions scores.

7. Significance, Applicability, and Extensibility

LMAC-Net establishes a new paradigm for long-term, fine-grained, multimodal action evaluation. Its unified attention consistency mechanism ensures that visual and auditory information are fused coherently, overcoming the limitations of prior fusion approaches. The architecture is notably parameter- and compute-efficient, operating in real time, and its modular structure (independent query encoding per modality, flexible backbone substitution) makes it readily adaptable to other multimodal sequence assessment domains beyond artistic sports. A plausible implication is that extensions of LMAC-Net may further advance multimodal video analysis tasks requiring robust temporal and cross-modal alignment (Wang et al., 29 Jul 2025).