Adaptive Cross-Form Learning (ACFL)

• ACFL is a training paradigm for skeleton-based action recognition that enables single-modal GCNs to learn complementary features from other skeleton forms.
• It employs a cross-form mimicry module with soft attention to synthesize and gate multi-form representations, yielding performance gains of up to 2.5%.
• ACFL integrates seamlessly with existing GCN architectures like CTR-GCN and MS-G3D, preserving network capacity and inference efficiency for practical deployment.

Adaptive Cross-Form Learning (ACFL) is a training paradigm for skeleton-based action recognition that enables Graph Convolutional Networks (GCNs) tailored to single data forms (e.g., skeleton joints or bones) to leverage complementary information from other forms during training, without increasing model capacity or necessitating their presence at inference. ACFL addresses the systematic mismatch between training with multi-form skeleton data and real-world inference conditions, where only partial forms may be available, by teaching each model to hallucinate useful features from unavailable modalities. It seamlessly integrates with existing GCN architectures, such as CTR-GCN, MS-G3D, and Shift-GCN, without structural modification, and achieves state-of-the-art performance on large-scale benchmarks (Wang et al., 2022).

1. Background and Motivations

Skeleton-based action recognition techniques typically use various "forms" of skeleton representations: joint coordinates, bone orientations, or joint+bone features. Previous methods that extend single-form GCNs to handle multi-form data usually do so through multi-stream architectures or feature fusion. These approaches require simultaneous access to all forms during both training and inference; however, practical scenarios, such as edge deployment, often restrict access to only a single form. ACFL resolves this discrepancy by training each GCN to mimic the discriminative and complementary representations of other forms while only requiring its own form as input during inference. This strategy allows models to preserve performance gains afforded by multi-form learning even in resource-constrained conditions.

2. Core ACFL Framework and Training Workflow

Given $L$ distinct skeleton data forms (typically $L=3$: joint, bone, joint+bone), ACFL constructs $L$ target GCN branches $P^t_i$ (for $i=1,\ldots,L$), each operating exclusively on one form $\mathcal{X}_i$. For each target, corresponding source GCNs $P^s_j$ ($j=1,\ldots,L$) provide alternative-form representations for mimicry:

• Target forward pass: $f^t_i = P^t_i(\mathcal{X}_i)$, $k^t_i = \Psi(f^t_i)$
• Source forward pass: $f^s_j = P^s_j(\mathcal{X}_j)$, $k^s_j = \Psi(f^s_j)$

A Cross-Form Mimicry Module attends over source representations $\{E^s_j\}$ to build an adaptive reference $E^r_i$ for each target. Through a gating mechanism, it synthesizes a content-selected signal $E^c_i$, which the target branch is trained to imitate in representation space. This training process is capacity-preserving: network depth, width, and inference-time complexity are unchanged compared to conventional GCNs.

The following table summarizes the two main ACFL instantiations:

Variant	Source GCNs	Parameter Overhead	Reference Stability
Off-line ACFL	Frozen, pre-trained	Extra (frozen)	High (fixed teachers)
On-line ACFL	Shared weights	None	Lower (co-evolving)

3. Mathematical Formalization

Let $$\mathcal{X} \in \mathbb{R}^{M \times T \times V \times C}$$ denote the skeleton sequence for a single form (with $M$—number of persons, $T$—temporal length, $V$—joints, $C$—channels). The GCN $P$ maps $\mathcal{X}$ to a representation $f \in \mathbb{R}^d$ and predicts logits $k = \Psi(f) \in \mathbb{R}^N$ for $N$ classes. The standard classification loss is cross-entropy: $$\ell_{s}(k, y) = -\sum_{n=1}^N y_n \log(\mathrm{softmax}(k)_n).$$ Cross-form mimicry proceeds as:

• Stack source representations $$E^s = [E^s_1;\ldots;E^s_L] \in \mathbb{R}^{L \times d_r}$$
• Stack target representations $$E^t = [E^t_1;\ldots;E^t_L] \in \mathbb{R}^{L \times d_r}$$
• Attention over sources: $$A = \mathrm{softmax}\left(\frac{(W_q E^t) (W_k E^s)^\mathsf T}{\sqrt{d_r}}\right)$$
• Regulatory weighting via $\beta \in \mathbb{R}^{1 \times L}$ (e.g., normalized source accuracies): $E^r = (A \odot \beta) E^s$
• Content gating: $$Z = \sigma\left(W_v(E^t - E^r)^\mathsf T\right)^\mathsf T, \quad E^c = Z \odot E^r$$
• Mimicry loss: $\ell_{d}(E^c_i, E^t_i) = \| E^c_i - E^t_i \|_2^2$
• Overall training objective: $$\mathcal{L} = \frac{1}{L} \sum_{i=1}^{L} \left[ \ell_s(k^t_i, y) + \lambda \ell_d(E^c_i, E^t_i) \right]$$ where $\lambda$ (default 1.0) balances the losses.

4. Integration, Efficiency, and Extension Beyond Skeleton Modalities

ACFL is added to existing GCN-based models as a training-time module: it does not alter the network architecture or inference cost. At test time, only the target GCN for the input-available form is deployed, with all auxiliary mimicry components discarded; model size and speed match baseline single-form GCNs. This property enables application in low-latency or memory-constrained settings.

Beyond skeleton action recognition, ACFL's model-agnostic and capacity-preserving nature, as well as its reliance on structured representations and soft attention, make it adaptable to other structured multi-modal scenarios (e.g., skeleton-RGB heatmap fusion or finer-grained anatomical forms), provided that all forms are present during training (Wang et al., 2022).

5. Experimental Results and Empirical Analysis

ACFL was validated on NTU-RGB+D 120, NTU-RGB+D 60, and UAV-Human datasets. When applied to state-of-the-art GCN backbones (CTR-GCN, MS-G3D, Shift-GCN), ACFL yielded improvements of +1–2.5% on single-form evaluation:

• CTR-GCN, NTU-120, joint-only (cross-subject): 84.9% (baseline) → 87.3% (ACFL; +2.4%)
• CTR-GCN, NTU-60, joint-only (cross-subject): 89.6% (baseline) → 91.2% (ACFL; +1.6%)
• CTR-GCN, UAV-Human, joint-only (cross-subject): 41.7% (baseline) → 43.8% (ACFL; +2.1%)

Application to bone-only and joint+bone yielded similar gains. ACFL also outperformed previous multi-stream fusion methods even when restricted to single-form inference, achieving 89.7%/90.9% (NTU-120) and 92.5%/97.1% (NTU-60) (Wang et al., 2022).

Ablation studies found:

• Off-line ACFL (frozen teacher) is ~0.8% better than on-line due to stable targets.
• Using both features $f$ and logits $k$ as representation outperforms either alone.
• Improvement from additional forms saturates beyond two sources.
• ACFL accommodates sources from heterogeneous backbone architectures.

6. Practical Considerations

• Hyperparameters: The mimicry weight $\lambda$ is typically set to 1, tunable within [0.1, 1.0]. Regulatory factors $\beta$ are source accuracies normalized to sum to 1.
• Training stability: Off-line ACFL offers more stable convergence by fixing source representations; on-line variant incurs co-evolving (possibly noisy) targets.
• Resource demands: Computational and memory overheads are incurred only during training (due to extra projections and storing representation matrices), with less than 5% additional GPU memory usage observed. No overhead at inference.
• Limitations: All forms must be present at training; noisy forms may negatively impact learned representations (partially mitigated by gating and $\beta$); ACFL alone may not fully capture distinctions in actions requiring external cues (e.g., RGB data).

7. Open Questions and Future Directions

Potential extensions include dynamic form selection at inference, integration with temporal-attention mechanisms, and semi-supervised ACFL where some forms are missing even at training. A plausible implication is that ACFL could generalize to a broader class of structured data problems in which multi-view or multi-modal cues are beneficial during training but may not be available at test time (Wang et al., 2022).