Action Variability Components

• Action variability components are defined as systematic quantifications of variation in composite actions, central to both combinatorial representation theory and machine learning.
• They decompose actions into dynamic (verb) and static (object) elements using dedicated encoder heads and prototype matching to measure independent feature contributions.
• This framework improves generalization over unseen verb–object pairings by enforcing feature independence and balancing compatibility via advanced loss functions and augmentation strategies.

Action variability components formalize and quantify sources of variation in composite actions, classically in combinatorial representation theory (e.g., component group actions on Springer fiber components) and recently in machine learning for compositional video action recognition, where action representations must generalize over novel verb–object pairings. These components, and the methodologies for analyzing, controlling, and exploiting their variability, are central to both domains' understanding of structure, generalization capacity, and decomposition of complex phenomena.

1. Component Decomposition in Compositional Action Recognition

In the context of zero-shot compositional action recognition, actions are systematically decomposed into "dynamic" (verb) and "static" (object) components. For an input video $X \in \mathbb{R}^{T\times224\times224\times3}$, a general video encoder (e.g., TSM-18, VideoSwin-T, or CLIP+adaptors) produces a spatiotemporal representation $F_X \in \mathbb{R}^{T\times D}$. Two dedicated heads then extract:

• $f_v \in \mathbb{R}^C$: dynamic features via temporal convolutions and pooling,
• $f_o \in \mathbb{R}^C$: static features via temporal pooling and multilayer perceptrons.

Parallel to these, semantic prototypes are learned:

• $E_v = \{e_{v,i}\}_{i=1}^{N_v}$ for verbs,
• $E_o = \{e_{o,j}\}_{j=1}^{N_o}$ for objects,

where prototypes are either learned word embeddings (fastText) or text-encoder outputs (CLIP) with soft prompting. This systematic decomposition is foundational for isolating and analyzing "action variability components" in subsequent learning modules (Li et al., 2024).

2. Independent Component Learning and Classification

The independent component learning module aligns $f_v$ to verb prototypes and $f_o$ to object prototypes, using cosine similarity and temperature-scaled softmax to yield component classification losses:

$$L_{verb} = -\log \frac{\exp(\cos(f_v, e_{v,l})/\tau)}{\sum_{j=1}^{N_v} \exp(\cos(f_v, e_{v,j})/\tau)}$$

$$L_{obj} = -\log \frac{\exp(\cos(f_o, e_{o,k})/\tau)}{\sum_{j=1}^{N_o} \exp(\cos(f_o, e_{o,j})/\tau)}$$

The sum $L_{comp} = L_{verb} + L_{obj}$ serves as the core metric for component discrimination. These scores quantify action variability for both dynamic and static factors individually, providing axis-aligned measures for the action space and facilitating zero-shot transfer by decoupling sources of variability (Li et al., 2024).

3. Composition Inference and Conditional Scoring

The composition inference module assesses candidate actions $a=(v_l, o_k)$ using conditional score matrices $S_{o|v} \in \mathbb{R}^{N_v \times N_o}$ and $S_{v|o} \in \mathbb{R}^{N_o \times N_v}$. The two paths are:

• Dynamics path: Fuse $e_{v,l}$ with $F_X$ via MLP, producing joint feature $f_{v-x, l}$, then compute $s_{o=o_k|v=v_l} = \sigma(\cos(f_{v-x,l}, e_{o,k}))$. The final score is the product $s_v[l] \cdot s_{o|v}[l,k]$.
• Static path: Symmetrical, starting from object component.

Averaging both path scores produces a normalized score for each action composition. Cross-entropy loss is then applied over seen actions, training models to infer compatibility and interactions between variable components. This explicitly models composition-level variability and supports robust generalization to unseen compositions (Li et al., 2024).

4. Strategies for Quantifying and Controlling Action Variability

To address the challenges of spurious inter-component correlations ("component domain variation") and compatibility imbalances ("component compatibility variation"), the following enhancements are introduced:

a) Cross-Component Independence

Hilbert-Schmidt Independence Criterion (HSIC)-based losses encourage feature disentanglement:

• $L_{sup, verb} = HSIC(f_X, f_v) - HSIC(f_v, y_v)$
• $L_{sup, obj} = HSIC(f_X, f_o) - HSIC(f_o, y_o)$

The aggregate independence loss,

$$L_{ind} = L_{sup,verb} + L_{sup,obj} + HSIC(f'_v, f'_o),$$

enforces independence between the principal subspaces (first $\rho C$ dimensions) of $f_v$ and $f_o$. Empirical results confirm that this de-correlation is critical for generalization, especially on verbs with high deformation (Li et al., 2024).

b) Compatibility Balancing and Unseen Pair Generation

Empirical conditional frequencies $\hat S_{o|v}$ and $\hat S_{v|o}$ from training co-occurrences are used to regularize model predictions via cross-entropy. Further, CutMix augmentation is applied to synthesize pseudo-videos and their counterfactual (never-seen) compositions. These are explicitly scored in the loss, mitigating overfitting and supporting "imagination" of new combinations.

The total loss alternates (probability $p=0.7$) between including CutMix terms (with new counterfactuals) and compatibility regularization, balancing all components:

$$\begin{cases} \bar{L}_{total} = \bar{L}_{com} + \alpha \bar{L}_{comp} + \beta L_{ind} + \gamma L_{new} & \text{(with CutMix)} \ L_{total} = L_{com} + \alpha L_{comp} + \beta L_{ind} + \gamma L_{con} & \text{(otherwise)} \end{cases}$$

This framework quantifies, decomposes, and regularizes action variability at both feature and composition levels (Li et al., 2024).

5. Empirical Quantification and the Role of HSIC

The HSIC terms in $L_{ind}$ serve as explicit statistical measures of dependence between dynamic and static features. Zero HSIC implies perfect independence, aligning with the goal of "pure" action variability components. Empirical studies (Table 5, (Li et al., 2024)) show that incorporating $L_{ind}$ increases the accuracy for unseen verbs and objects, with notable improvements on deformation-heavy verbs (e.g., increasing accuracy from 60.2% to 65.2% for verbs like "squeeze," "bend" and from 34.8% to 45.1% for objects). This demonstrates the critical impact of quantifying and controlling action variability in reducing overfitting and enhancing compositional generalization.

6. Implementation Details and Hyperparameters

C2C supports multiple encoder backbones and vision-language paradigms:

• Backbones: TSM-18, VideoSwin-T (for video-only), CLIP (ViT-B/32) with AIM adapters and various soft prompt schemes (CoOp, CSP, SPM).
• Loss and regularization parameters: temperature $\tau=0.07$, CutMix probability $p=0.7$, loss weights $\alpha=0.2$, $\beta=0.1$, $\gamma=0.1$.
• Optimizer and scheduling: Adam with initial learning rate $10^{-4}$, SGDR warm restarts every 10 epochs, 50 epochs total, batch size 32.
• HSIC is computed over the first $\rho=0.5$ portion of the feature dimension.

This architecture directly operationalizes action variability component management, integrating domain and compatibility variation into model optimization (Li et al., 2024).

7. Theoretical and Algebraic Perspectives

In algebraic representation theory, especially the study of Springer fibers for a nilpotent element $e$ in a simple Lie algebra, "component group actions" provide a rigorous exemplar of variability at the level of geometric/representation-theoretic components. The component group $A_e = Z_G(e)/Z_G(e)^\circ$ acts on the irreducible components $\text{Irr}(\mathcal{B}_e)$ of the Springer fiber. The structure of this action (orbit types, stabilizers, decomposition into $A_e/H$ orbits with multiplicities) and their explicit parametrizations (e.g., via signed domino tableaux, noncrossing partitions) are foundational to understanding how componentwise (action) variability underlies complex algebraic or geometric objects (Hoang, 2024).

The precise orbit decomposition, stabilizer characterization, and interplay with cell structures in Weyl groups (Lusztig–Sommers theory) suggest deep analogies between component-based variability in algebraic and statistical machine learning contexts, though realized through different mathematical mechanisms.

---

In both settings, action variability components are not mere byproducts but essential analytic and generative axes—central to the decomposition, quantification, and generalization of complex structured phenomena, whether in data-driven perception or representation-theoretic geometry (Li et al., 2024, Hoang, 2024).