UASA: Unknown Action Segment Assignment

• UASA is a segment-level semantic assignment method that labels unknown actions by pooling frame embeddings into robust segment representations.
• It operates in a two-stage pipeline where GGSM sets precise temporal boundaries and UASA refines clustering using Gaussian Mixture Models and K-means.
• Empirical results demonstrate that UASA significantly boosts unknown action recognition metrics compared to traditional frame-wise approaches.

Searching arXiv for the cited paper and directly related work mentioned in the provided material. Unknown Action Segment Assignment (UASA) is a segment-level semantic assignment procedure introduced within the Action Discovery setting for Temporal Action Segmentation (TAS), where training videos are only partially labeled and frames belonging to unannotated actions are marked with a generic UNK label rather than action-specific classes (Spurio et al., 7 Aug 2025). In this setting, the action set $A$ is partitioned into disjoint known and unknown subsets, $K \subset A$ and $U \subset A$, with $K \cap U = \varnothing$ and $K \cup U = A$, and the objective extends beyond standard frame-wise recognition to include both boundary detection for known and unknown actions and discovery of semantically meaningful unknown classes without prior knowledge of their number (Spurio et al., 7 Aug 2025). UASA addresses the second part of that objective. It operates on unknown temporal intervals produced by the Granularity-Guided Segmentation Module (GGSM), constructs segment embeddings by temporal pooling, estimates the number of unknown classes via Gaussian Mixture Model model selection using the Bayesian Information Criterion, clusters the resulting segment representations with K-means, and assigns each unknown segment to the nearest learned prototype, after which labels are propagated to frames by majority voting within each interval (Spurio et al., 7 Aug 2025).

1. Position within the Action Discovery framework

Action Discovery is formulated as a novel setup within TAS for partially labeled data in which only known actions are annotated during training, while other actions that occur remain unlabeled and are represented only as UNK (Spurio et al., 7 Aug 2025). Formally, a video is a frame sequence $X = \{x_t\}_{t=1}^T$, a backbone produces per-frame embeddings $E = \{e_t\}_{t=1}^T$ with $e_t \in \mathbb{R}^D$, and the backbone predicts frame-wise labels over $|K|+1$ classes, namely the known classes and UNK (Spurio et al., 7 Aug 2025).

Within this formulation, UASA is not a standalone recognizer. It is the second stage of a two-step pipeline in which GGSM first refines temporal intervals for both known and unknown actions by selecting a segmentation proposal whose granularity aligns with the annotated known spans, and UASA then assigns semantically coherent labels to the resulting unknown segments across videos (Spurio et al., 7 Aug 2025). The data flow is explicitly defined as raw video to TAS backbone embeddings and preliminary predictions, then GGSM segmentation proposals and selected boundaries, then segment pooling over unknown intervals, then UASA clustering and label assignment, and finally frame-wise labels obtained by majority voting inside segments (Spurio et al., 7 Aug 2025).

This positioning is conceptually important because UASA assumes that the unknown intervals it receives are already segmented at an appropriate temporal granularity. The paper states that the two stages are tightly coupled, and that GGSM guides the temporal granularity of the segments on which UASA performs semantic discovery (Spurio et al., 7 Aug 2025). A plausible implication is that UASA’s behavior is best understood as a segment assignment mechanism conditioned on a boundary detector rather than as a direct alternative to frame-wise clustering.

2. Formal definition and algorithmic mechanics

UASA begins from a collection of unknown temporal segments $S_k = [t_s^k, t_e^k]$ produced by GGSM (Spurio et al., 7 Aug 2025). Each such interval is mapped to a segment-level representation by temporal average pooling of backbone embeddings:

$K \subset A$0

The notation summary also writes the segment embedding as $K \subset A$1 (Spurio et al., 7 Aug 2025). According to the paper, this aggregation reduces frame-level noise and makes assignment more robust; ablations reported in the study indicate that segment-based initialization markedly outperforms frame-based initialization for estimating the number of unknown classes (Spurio et al., 7 Aug 2025).

UASA uses different similarity structures at different stages. During backbone training, the contrastive loss on unknown frames uses cosine similarity,

$K \subset A$2

whereas the UASA clustering and assignment stage uses Euclidean distance between a segment embedding $K \subset A$3 and a cluster centroid $K \subset A$4,

$K \subset A$5

Unknown class prototypes are the K-means centroids $K \subset A$6, and after each K-means iteration the centroid update is

$K \subset A$7

with $K \subset A$8 the set of segment embeddings assigned to centroid $K \subset A$9 (Spurio et al., 7 Aug 2025).

Assignment is then defined by nearest-centroid selection:

$U \subset A$0

Each unknown segment $U \subset A$1 receives the label $U \subset A$2, and for all frames $U \subset A$3 the final predicted label is set to that assigned cluster label through majority voting within the segment (Spurio et al., 7 Aug 2025). For known segments, the backbone’s known-class predictions are retained inside the GGSM boundaries (Spurio et al., 7 Aug 2025).

The paper also provides a compact pseudocode view of this procedure: given unknown segments $U \subset A$4, frame embeddings $U \subset A$5, and stored centroids $U \subset A$6, compute $U \subset A$7 by temporal averaging, compute $U \subset A$8 by nearest-centroid assignment, and set $U \subset A$9 for all $K \cap U = \varnothing$0 (Spurio et al., 7 Aug 2025). In procedural terms, UASA is therefore a segment-wise clustering-and-propagation module rather than a frame-wise classifier.

3. Automatic determination of the number of unknown classes

A defining property of UASA is that it does not assume prior knowledge of the number of unknown classes (Spurio et al., 7 Aug 2025). Instead, it performs model selection by fitting Gaussian Mixture Models with different component counts and selecting the value that minimizes the Bayesian Information Criterion:

$K \cap U = \varnothing$1

where $K \cap U = \varnothing$2 is the likelihood of the GMM with $K \cap U = \varnothing$3 components, $K \cap U = \varnothing$4 is the number of parameters of that model, and $K \cap U = \varnothing$5 is the number of segment embeddings used for fitting (Spurio et al., 7 Aug 2025). The selected number of unknown classes is

$K \cap U = \varnothing$6

Once $K \cap U = \varnothing$7 has been selected, UASA runs K-means on the set of segment embeddings $K \cap U = \varnothing$8, obtains centroids $K \cap U = \varnothing$9, and performs nearest-centroid assignment for each segment (Spurio et al., 7 Aug 2025). The paper explicitly characterizes UASA as non-parametric beyond K-means, with stability arising from centroid updates and GGSM-driven segmentation granularity (Spurio et al., 7 Aug 2025).

The use of GMM-BIC for selecting $K \cup U = A$0 is central to the method’s claim of supporting discovery rather than merely relabeling a fixed latent taxonomy. In the Action Discovery setting, the number of unknown action classes is not available in advance, so model selection becomes part of the semantic assignment problem itself (Spurio et al., 7 Aug 2025). The study further reports that estimating $K \cup U = A$1 directly from frame embeddings can lead to poor GMM fits and unstable assignments, whereas segment-level pooling stabilizes the process (Spurio et al., 7 Aug 2025). This suggests that the estimated latent class structure is treated as meaningful only after temporal aggregation removes frame-level variance.

4. Dependence on GGSM and boundary granularity

Although UASA is the semantic component, its inputs are produced by GGSM, and the paper repeatedly emphasizes that this dependence is not incidental (Spurio et al., 7 Aug 2025). GGSM builds a hierarchy over frames using farthest-point linkage with Euclidean distance, where the inter-cluster distance between frame clusters $K \cup U = A$2 and $K \cup U = A$3 is

$K \cup U = A$4

with Euclidean distance computed on frame embeddings (Spurio et al., 7 Aug 2025). The resulting hierarchy defines multiple segmentation proposals at different granularities.

Proposal selection is driven by alignment to known action spans using a length-balanced one-dimensional IoU criterion. If proposal intervals are $K \cup U = A$5 and known intervals are $K \cup U = A$6, GGSM scores a proposal by

$K \cup U = A$7

with

$K \cup U = A$8

and

$K \cup U = A$9

The balancing coefficient $X = \{x_t\}_{t=1}^T$0 controls the strength of length balancing, and the paper reports that $X = \{x_t\}_{t=1}^T$1 yields the best empirical results (Spurio et al., 7 Aug 2025).

This mechanism matters to UASA because the unknown intervals $X = \{x_t\}_{t=1}^T$2 used for pooling are precisely the segments selected by GGSM at the granularity implied by the known spans (Spurio et al., 7 Aug 2025). The paper’s ablations make this dependence explicit: replacing GGSM with TW-FINCH boundary detection on 50Salads Fine yields unknown MoF of 22.8, whereas the full method attains 37.2, and the authors conclude that GGSM’s granularity guidance is crucial (Spurio et al., 7 Aug 2025). Likewise, qualitative analyses show that GGSM can adapt to different annotation granularities, splitting an unknown interval into four segments under fine granularity but only two under coarse granularity, after which UASA assigns consistent labels across videos (Spurio et al., 7 Aug 2025).

A common misconception would be to interpret UASA as performing unknown action discovery solely through clustering. The reported design and ablations indicate that boundary granularity is not a preprocessing detail but a structural condition for robust semantic assignment (Spurio et al., 7 Aug 2025).

5. Training objectives and the role of contrastive shaping

UASA relies on embeddings produced by a Temporal Action Segmentation Backbone implemented as MS-TCN, which outputs per-frame embeddings and logits over $X = \{x_t\}_{t=1}^T$3 (Spurio et al., 7 Aug 2025). The backbone is trained with cross-entropy for classification into $X = \{x_t\}_{t=1}^T$4 classes and Truncated Mean Squared Error loss across stages for temporal smoothness, yielding

$X = \{x_t\}_{t=1}^T$5

where $X = \{x_t\}_{t=1}^T$6 is the weight of the T-MSE term (Spurio et al., 7 Aug 2025).

To structure the embedding space for unknown actions and prevent collapse, the method adds an InfoNCE-style contrastive loss on unknown frames:

$X = \{x_t\}_{t=1}^T$7

Positive samples are drawn from a truncated Gaussian temporal window around frame $X = \{x_t\}_{t=1}^T$8, negatives are sampled outside that window, and the temperature is fixed at $X = \{x_t\}_{t=1}^T$9, a value validated by ablation (Spurio et al., 7 Aug 2025). The final training objective is

$E = \{e_t\}_{t=1}^T$0

with $E = \{e_t\}_{t=1}^T$1 controlling the contribution of the contrastive term; the paper states that $E = \{e_t\}_{t=1}^T$2 balances performance on known and unknown actions (Spurio et al., 7 Aug 2025).

This training design is directly relevant to UASA because the latter operates on the learned embedding geometry. The paper states that the contrastive term structures unknown embeddings and avoids a degenerate “all-UNK” collapse, while the ablation with $E = \{e_t\}_{t=1}^T$3 severely degrades unknown performance (Spurio et al., 7 Aug 2025). On 50Salads Fine, the loss ablation shows unknown MoF of approximately 27.9 at $E = \{e_t\}_{t=1}^T$4 versus 37.2 at $E = \{e_t\}_{t=1}^T$5, and larger $E = \{e_t\}_{t=1}^T$6 can overemphasize contrastive learning at the expense of known actions (Spurio et al., 7 Aug 2025).

The relation between contrastive shaping and UASA is therefore not merely supportive but foundational. UASA’s segment clustering presupposes that embeddings for unknown behaviors are already partially separable; the contrastive objective provides that separability, while GGSM provides temporally coherent units on which clustering can act (Spurio et al., 7 Aug 2025).

6. Empirical behavior, datasets, and evaluation

The method is evaluated on three datasets: Breakfast, with 1,712 videos and 48 actions split into 33 known and 15 unknown using I3D features and four official splits; 50Salads, with 50 videos and two annotation granularities, namely Fine with 19 actions split into 13 known and 6 unknown and Coarse with 11 actions split into 7 known and 4 unknown, also using I3D features and five-fold cross-validation; and DesktopAssembly, with 128 videos and 22 actions split into 16 known and 6 unknown using dataset-specific features, where 65% of unknown segments contain multiple unknown actions (Spurio et al., 7 Aug 2025).

Evaluation uses Mean over Frame, segmental Edit score, and F1@$E = \{e_t\}_{t=1}^T$7 using IoU thresholds (Spurio et al., 7 Aug 2025). Known and unknown actions are evaluated separately via masking, and for unknown actions a one-to-one alignment of predicted unknown clusters to ground-truth unknown classes is computed with the Hungarian algorithm at dataset level, strictly for evaluation (Spurio et al., 7 Aug 2025). This evaluation protocol is important because the discovered cluster indices produced by UASA are not presumed to have canonical semantic names during training or inference.

The reported impact of UASA is most clearly summarized on unknown-action metrics. On Breakfast, unknown MoF increases from 10.5 for the baseline to 16.6 for Baseline+UASA and 23.5 for the full method using GGSM+UASA; F1@50 changes from 8.7 to 8.6 to 18.6 (Spurio et al., 7 Aug 2025). On 50Salads Fine, unknown MoF increases from 24.6 to 34.1 to 37.2, and F1@50 from 12.3 to 23.1 to 27.1 (Spurio et al., 7 Aug 2025). On 50Salads Coarse, unknown MoF is 41.5 for the baseline, 33.8 for Baseline+UASA, and 46.4 for the full method, while F1@50 is 18.4, 16.2, and 36.6 respectively (Spurio et al., 7 Aug 2025). On DesktopAssembly, unknown MoF rises from 39.9 to 50.1 to 51.3, and F1@50 from 14.2 to 29.3 to 45.4 (Spurio et al., 7 Aug 2025).

These comparisons support a specific interpretation given in the paper: segment-aware UASA, when supplied by GGSM, substantially improves unknown action assignment over both the baseline and UASA without GGSM (Spurio et al., 7 Aug 2025). The ablation against unsupervised ASOT on unknown embeddings in 50Salads Fine reports 29.8 unknown MoF for Baseline+ASOT versus 37.2 for the full method, with the latter consistently better on Edit and F1 (Spurio et al., 7 Aug 2025). Another ablation shows that segment-based initialization for UASA yields 37.2 unknown MoF on 50Salads Fine, compared with 31.8 for frame-based initialization, confirming the importance of GGSM-produced segments for robust $E = \{e_t\}_{t=1}^T$8 estimation and clustering stability (Spurio et al., 7 Aug 2025).

Qualitative evidence is also reported. On Breakfast, the paper shows cases where UASA, given GGSM segments, discovers multiple unknown classes within intervals the backbone labeled only as UNK, including spoon_powder, pour_milk, and stir_milk; Baseline+GGSM improves boundaries but cannot assign distinct unknown labels, whereas UASA on top of GGSM yields coherent multi-class discovery and reduces over-segmentation (Spurio et al., 7 Aug 2025).

7. Computational profile, assumptions, and relation to adjacent problem settings

The computational characteristics of UASA are described explicitly. For GGSM, hierarchical clustering over $E = \{e_t\}_{t=1}^T$9 frames with farthest-point linkage typically incurs $e_t \in \mathbb{R}^D$0 distance computations, and proposal scoring adds $e_t \in \mathbb{R}^D$1 for $e_t \in \mathbb{R}^D$2 proposal segments and $e_t \in \mathbb{R}^D$3 known spans (Spurio et al., 7 Aug 2025). For UASA, GMM model selection over $e_t \in \mathbb{R}^D$4 segment embeddings costs $e_t \in \mathbb{R}^D$5 per candidate, with $e_t \in \mathbb{R}^D$6 and constant iterations, while K-means costs $e_t \in \mathbb{R}^D$7 for $e_t \in \mathbb{R}^D$8 iterations (Spurio et al., 7 Aug 2025). Key hyperparameters include $e_t \in \mathbb{R}^D$9 for the contrastive loss, $|K|+1$0 for the contrastive weight, $|K|+1$1 for GGSM balancing, the MS-TCN backbone, feature type, and the empirically set search range $|K|+1$2 for GMM selection (Spurio et al., 7 Aug 2025).

The runtime measurements highlight a practical consequence of segment-based initialization. On 50Salads Fine with an MS-TCN backbone and one split, baseline inference is approximately 9.87 s, GGSM alone is approximately 21.2 s, UASA alone without segments and using frame-based $|K|+1$3 search is approximately 1169.2 s, and GGSM+UASA is approximately 83.0 s (Spurio et al., 7 Aug 2025). The paper explicitly states that segment-based initialization from GGSM drastically reduces UASA’s runtime and improves accuracy (Spurio et al., 7 Aug 2025).

The stated limitations and assumptions are equally specific. UASA depends on GGSM: if GGSM boundaries are poor or misaligned with true unknown actions, the segment embeddings $|K|+1$4 degrade and clustering can become unstable (Spurio et al., 7 Aug 2025). It is also sensitive to partially labeled data when known coverage is very sparse or highly inconsistent, since the granularity guidance may then be less informative (Spurio et al., 7 Aug 2025). In addition, UASA is post hoc: pseudo-labels discovered for unknown actions are not fed back to retrain the backbone in this work, so the classifier still treats unknown frames as UNK (Spurio et al., 7 Aug 2025). The paper notes that iterative refinement with pseudo-labels could further boost performance but risks confirmation bias (Spurio et al., 7 Aug 2025).

In relation to neighboring research settings, the method is distinguished from open-set recognition because unknown classes are present in training videos but unlabeled, and the goal is multi-class unknown discovery and segmentation rather than test-time known-versus-unknown detection (Spurio et al., 7 Aug 2025). It is presented as complementary to weakly supervised and semi-supervised TAS by addressing partially annotated datasets while discovering previously unlabeled actions, and as connected to unsupervised TAS while improving over purely unsupervised boundary detection through segment-aware initialization and known-action granularity guidance (Spurio et al., 7 Aug 2025).

Taken together, these details characterize UASA as a semantic assignment module for partially labeled TAS that combines segment pooling, model selection, prototype induction, and nearest-centroid labeling under a granularity regime set by known annotations (Spurio et al., 7 Aug 2025). Its reported strengths are the ability to assign semantically coherent unknown labels without knowing $|K|+1$5, the stabilizing effect of segment-level pooling, and the use of contrastively shaped embeddings to prevent collapse; its principal constraints arise from dependence on boundary quality and the absence of iterative pseudo-label feedback into the backbone (Spurio et al., 7 Aug 2025).