Sub-action Prototype Clustering (SPC)

• Sub-action Prototype Clustering is a method that decomposes complex action sequences into compact, interpretable sub-action prototypes for video analysis.
• It employs techniques such as soft k-means, Gaussian mixture modeling, and transform-invariant clustering to adaptively capture spatiotemporal patterns.
• Empirical evaluations show that SPC improves action segmentation accuracy and boundary localization across various video understanding benchmarks.

Sub-action Prototype Clustering (SPC) refers to a family of methods aimed at unsupervised or weakly-supervised discovery and representation of prototypical sub-structures within sequential or spatiotemporal data, particularly in the context of video understanding and action segmentation. The central principle is the adaptive partitioning of complex actions or trajectories into a compact set of sub-action prototypes—vectors or structures that summarize recurring, interpretable patterns either in feature space, spatiotemporal graphs, or emission model manifolds. SPC achieves robust decomposition by leveraging techniques such as soft k-means with temporal constraints, Gaussian mixture modeling in sub-graph space, and transform-invariant clustering on covariance manifolds. Its instantiations underpin recent advances in point-level weakly-supervised temporal action localization, graph-based video action recognition, and transform-invariant action discovery.

1. Formalization and Definitions

A sub-action prototype is a compact representation of a coherent, characteristic pattern within an action sequence, trajectory, or spatiotemporal region. In point-level weakly-supervised temporal action localization (PWTAL), a video of $T$ temporal snippets is embedded as $X\in\mathbb{R}^{T\times D}$, and a high-confidence action proposal $p$ is represented as $$X_p=[X_{p,1},\ldots,X_{p,N_p}]^\top\in\mathbb{R}^{N_p\times D}$$. SPC decomposes $X_p$ into $N_s$ sub-action prototypes $$S=[S_1,\ldots,S_{N_s}]^\top\in\mathbb{R}^{N_s\times D}$$, each $S_j$ summarizing the appearance-motion properties of a specific sub-segment (e.g., "approach," "takeoff," "landing") (Li et al., 2023).

In sub-graph-based frameworks, SPC refers to the clustering of multi-scale combinatorial sub-graphs extracted from space-time graphs, each node and edge representing spatiotemporal object interactions; each cluster center forms an action class-specific prototype (Li et al., 2022). In covariance-matrix-based models, sub-action prototypes are clusters of emission models discovered via transform-invariant similarity metrics and non-parametric priors, typically on the manifold of symmetric positive-definite (SPD) matrices (Figueroa et al., 2017).

2. SPC Algorithms: Core Methodologies

2.1 Soft k-means with Temporal Constraints

In the SPL-Loc framework for PWTAL, SPC operates as a soft k-means variant over the combined appearance-motion and temporal position space. The key steps are:

• Adaptive determination of prototype count $N_s = \min(\lfloor N_p / r_p\rfloor, N_{max})$, where $r_p$ is the average snippet-per-prototype ratio.
• Initialization by temporal stratification: prototypes are seeded across the proposal timeline.
• Iterative update: For each snippet-prototype pair $(i,j)$, compute

$$Dis(i,j) = \sqrt{ \|X_{p,i}-S_{j}\|_2^2 + \gamma \cdot (i-\tau_j)^2 }$$

where $\gamma$ controls feature-temporal tradeoff, $\tau_j$ is the temporal center. Soft assignment weights $a_{ij} = \exp(-Dis(i,j))$ drive the update:

$$S_j \leftarrow \frac{\sum_i a_{ij} X_{p,i}}{\sum_i a_{ij}}, \qquad \tau_j \leftarrow \frac{\sum_i a_{ij} t_i}{\sum_i a_{ij}}$$

• The prototype set is refined over $L$ iterations and stored in a class-specific memory bank, with only the top $k_{sub}$ per class retained per epoch (Li et al., 2023).

2.2 Multi-scale Sub-graph GMM Clustering

The MUSLE framework constructs space-time graphs over tubelet-based nodes, extracting all fixed-size sub-graphs and representing them via concatenation of node and edge features. For each action class and sub-graph size $S$, the set of candidate sub-graph features $\{x_n\}_{n=1}^N$ is clustered with a differentiable Gaussian Mixture Layer:

$$\alpha_n = \mathrm{MLP}(x_n), \qquad \gamma_{nk} = \mathrm{softmax}_k(\alpha_n)$$

$\hat{\mu}_k$, $\hat{\Sigma}_k$, and mixture weights $\hat{\phi}_k$ are calculated by soft assignments; each Gaussian kernel forms a sub-action prototype. Supervised loss minimizes negative log-likelihood of in-class sub-graphs under the GMM, regularized on $\hat{\Sigma}_k$ diagonals (Li et al., 2022).

2.3 Transform-invariant Clustering with SPCM-CRP

SPC in unsupervised action discovery is realized through clustering of emission Gaussians using SPCM similarity, which compares spectral polytopes of covariance matrices, thereby achieving invariance to rotation, translation, and scaling. Observations are embedded via spectral analysis of the SPCM affinity matrix. Cluster assignments are decided by a distance-dependent Chinese Restaurant Process (dd-CRP), allowing the number of prototypes to adapt to data complexity non-parametrically. In the ICSC-HMM, this step is coupled with an IBP-HMM, yielding joint segmentation and assignment of state runs to invariant “sub-action prototypes” (Figueroa et al., 2017).

3. Objective Functions and Theoretical Underpinnings

All SPC methods operate under the principle of minimizing intra-cluster distance in an appropriately chosen space, optionally regularized for assignment entropy or prototype sparsity. Key generic objectives include:

• For soft k-means SPC (SPL-Loc): minimizing

$$J(S, A) = \sum_{i=1}^{N_p} \sum_{j=1}^{N_s} a_{ij} \cdot \mathrm{Dist}(f_i, s_j) - \epsilon H(A)$$

where $H(A) = - \sum_{i,j} a_{ij}\log a_{ij}$ and the assignment normalization is implicit (Li et al., 2023).

• For sub-graph GMM SPC (MUSLE): minimizing negative log-likelihood of sub-graph features under the mixture,

$$\mathcal{L}_{c,S}(\theta) = -\frac{1}{N} \sum_{n=1}^N \log p(x_n) + \lambda \sum_{k=1}^K \sum_i \frac{1}{\hat{\Sigma}_{k,ii}}$$

(Li et al., 2022).

• For SPCM-CRP, the clustering posterior involves integrating the SPCM-dependent link prior with a marginal likelihood under a Normal-Inverse-Wishart prior over embedded points (Figueroa et al., 2017).

All methods feature a dynamic or non-parametric approach to prototype count: SPL-Loc adapts $N_s$ per proposal length; SPCM-CRP is non-parametric by construction; MUSLE prunes low-support Gaussians.

4. Capturing Spatiotemporal and Transform-invariant Structure

A salient property of SPC approaches is adaptive sensitivity to signal complexity and length:

• In SPL-Loc, $N_s$ increases for longer proposals, and the $\gamma$ parameter allows tuning of appearance-motion versus temporal smoothness priors. The iterative assignment enables prototypes to follow real sub-action density even in the presence of missing observations or occlusions (Li et al., 2023).
• SPC in MUSLE leverages multi-scale sub-graph extraction with $S\in\{3,4,5\}$, directly modeling hierarchical and concurrent interactions among objects and actors. The multi-scale design captures both atomic and composite patterns, critical for recognizing actions with variable spatiotemporal extent (Li et al., 2022).
• Transform invariance in SPC (ICSC-HMM) is achieved by clustering covariance matrices using spectral polytope homothety rather than direct parameter space metrics. This yields prototypes that are robust to spatial reparameterizations, vital in learning from real-world sensor or video data (Figueroa et al., 2017).

Storing prototypes in a memory structure—explicitly in SPL-Loc, implicitly in GMM or CRP mixtures—enables both intra-video and inter-video pattern mining. This enables transfer, robustness, and interpretability in downstream modules (e.g., Ordered Prototype Alignment, decision voting).

5. Integration with Learning Pipelines and Empirical Performance

In SPL-Loc, the SPC module is invoked as an inner alternating-minimization loop atop proposals produced by a base network. After clustering, only the most confident prototypes are cached in class-specific memory banks. These prototypes are subsequently used by the Ordered Prototype Alignment (OPA) module to align prototypes with feature trajectories and generate pseudo-labels—improving coverage and boundary quality of predicted action segments. SPC is not directly backpropagated, but its impact propagates via losses on pseudo-labels and alignment (Li et al., 2023).

In MUSLE, the entire end-to-end pipeline from tubelet extraction, sub-graph formation, to GMM parameter updates is differentiable. All gradients flow to the backbone tubelet features, allowing sub-action prototypes to be shaped by supervision without cross-entropy loss (Li et al., 2022).

On transform-invariant action discovery, SPC is jointly optimized with non-parametric IBP–HMMs so that segmentation and prototype assignment mutually reinforce each other, avoiding the need to fix hyperparameters such as the number of actions a priori (Figueroa et al., 2017).

Ablation studies specifically validate the unique benefits of SPC:

• In SPL-Loc, using SPC-derived prototypes (vs. uniformly sampled snippets) for OPA yields a +1.5% absolute mAP boost on the THUMOS-14 benchmark (44.8% → 46.3%) (Li et al., 2023).
• SPC+OPA, even without pseudo-labels, refines feature space alignment (+1.8% mAP).
• In MUSLE, moving from flat or whole-graph features to single-scale SPC yields >2% top-1 accuracy benefit; stacking multi-scale SPC raises performance further, confirming that prototype clustering captures discriminative structure missed by holistic representations (Li et al., 2022).
• In SPCM-CRP/ICSC-HMM, SPC enables unsupervised (transform-invariant) decomposition of realistic human activity sequences, achieving segmentations closely matching expert annotation (Figueroa et al., 2017).

6. Hyperparameterization and Sensitivity

SPC methods include critical hyperparameters that control their adaptivity and expressive power:

Hyperparameter	Role	Typical Value Range
$r_p$	Snippet/prototype ratio (SPL-Loc)	3–5
$N_{max}$	Max prototypes/proposal (SPL-Loc)	5–10
$\gamma$	Feature-temporal tradeoff (SPL-Loc)	1–3
$k_{sub}$	Top prototypes/class/memory bank (SPL-Loc)	8–10
$L$	Clustering iterations (SPL-Loc)	6
$K$	GMM components (MUSLE)	6 (pruned by threshold)
$S$	Sub-graph scale (MUSLE)	3, 4, 5
$\tau$	SPCM scale tolerance (SPCM-CRP)	$\ge 0$

Empirical results show that aggressive prototype splitting ($N_{max}>5$) in SPL-Loc leads to diminishing or negative returns due to over-segmentation, while adaptive criterion for $N_s$ consistently outperforms fixed $N_s$. In MUSLE, multi-scale aggregation always yields superior recognition accuracy compared to single-scale. In SPCM-CRP, the model robustly selects prototype count by model evidence rather than user specification.

7. Impact and Research Directions

SPC has demonstrated state-of-the-art performance across several video understanding and action discovery benchmarks. SPL-Loc with SPC achieved significant improvement on THUMOS-14, GTEA, and BEOID datasets, particularly enhancing the completeness and boundary accuracy of temporal localization from point-wise supervision (Li et al., 2023). MUSLE’s SPC delivers improved recognition on Something-Something V1/V2 and Kinetics-400, outperforming whole-graph and GCN counterparts by exploiting the compositional nature of actions (Li et al., 2022). In unsupervised discovery, SPCM-CRP-based SPC models yield segmentations closely aligned with expert annotation in manipulation and cooking tasks, without any explicit labels or transformation normalization (Figueroa et al., 2017).

Ongoing directions include extending SPC to continuous or online settings, leveraging memory-based prototype update rules, and integrating domain knowledge regarding sub-action semantics or constraints on allowed prototype transitions. Robustness to missing data, high-dimensionality, and complex interaction networks remain active areas for methodological enhancement. The unification of SPC principles—temporal adaptivity, soft assignment, transform invariance, and prototype memory—continues to shape advances in structured, interpretable video and time-series analysis.