Granularity-Guided Segmentation Module (GGSM)

• GGSM is a temporal segmentation module that splits coarse 'UNK' video segments into intervals mirroring known action durations.
• It employs an adjacency-restricted complete-link hierarchical clustering of frame embeddings and scores proposals with a balanced IoU metric.
• Empirical evaluations show GGSM improves unknown-action discovery metrics by enforcing annotation granularity alignment in temporal segmentation.

Granularity-Guided Segmentation Module (GGSM) is a module introduced in the Action Discovery setup within Temporal Action Segmentation to identify temporal intervals for both known and unknown actions by mimicking the granularity of annotated actions (Spurio et al., 7 Aug 2025). Its stated purpose is to take video embeddings in which “unknown” spans have already been lumped together by a standard TAS model and subdivide those spans into as many segments as the human annotator would have—“neither more nor fewer”—by borrowing the notion of “granularity” from the known parts of the video. In this formulation, GGSM addresses temporal segmentation rather than semantic assignment: it splits “UNK” stretches into temporally coherent intervals whose average duration mirrors that of the known actions, while Unknown Action Segment Assignment (UASA) subsequently identifies semantically meaningful classes within those intervals.

1. Position within Action Discovery

Action Discovery is defined as a setup in which only a subset of actions—“known actions”—is annotated in the training data, while other “unknown actions” remain unlabeled (Spurio et al., 7 Aug 2025). The backbone, denoted TASB, is trained to predict frame-wise labels over $|\phi_{act}|$ known classes plus one “UNK” class. In that setting, the “UNK” spans often contain multiple distinct actions, but no labels are available for them.

GGSM is introduced to resolve that temporal ambiguity. Its job is to split those “UNK” spans into intervals whose average duration, described in the source as “granularity,” mirrors that of the known actions. The stated rationale is twofold: by enforcing granularity alignment, GGSM avoids under-segmentation, where two different unknown actions are merged into one interval, and over-segmentation, where a single unknown action is broken into many tiny pieces.

The module therefore occupies a specific middle layer in the pipeline. It is neither the initial feature extractor nor the semantic clustering stage. Instead, it converts a coarse “known versus unknown” temporal labeling into a finer segmentation of the unknown regions. This division of labor is central to the Action Discovery design: GGSM supplies the temporal intervals, and UASA operates on those intervals afterward.

2. Input representation and clustering procedure

The input to GGSM is a sequence of $T$ frame-level embeddings,

$E = [e_1,\ldots,e_T] \in \mathbb{R}^{T\times D},$

produced by the TAS backbone just before the classifier (Spurio et al., 7 Aug 2025). These embeddings are described as carrying both the known-vs-unknown signal and a rich representation of motion and appearance.

GGSM itself is specified as a complete-link, adjacency-restricted hierarchical clustering procedure. Each frame $t$ is initialized as its own cluster,

$C_t = \{t\}.$

At each iteration, GGSM considers only pairs of adjacent clusters $(C_p,C_q)$ and computes the complete-link distance

$$D(C_p, C_q) = \max_{u\in C_p,\; v\in C_q} \|e_u - e_v\|_2.$$

It then finds the adjacent pair with the smallest $D$, merges them into a single cluster, updates the adjacency list, and records the resulting partition as one segmentation proposal. Repeating this from $T$ singleton clusters down to one cluster yields $T$ different segmentations, from $T$0 clusters down to $T$1.

The adjacency restriction is operationally important. GGSM does not search over arbitrary frame groupings; it searches over temporal partitions that preserve sequence order. The module is therefore a temporal interval generator rather than a general-purpose clustering routine over unordered embeddings. The source explicitly summarizes it as “not a neural network but a data-driven, hierarchical clustering routine whose sole learning signal is the granularity of the known actions” (Spurio et al., 7 Aug 2025).

3. Proposal scoring and granularity alignment

Each segmentation proposal $T$2 defines $T$3 temporal intervals $T$4. These proposals are scored against $T$5 known-action intervals,

$T$6

taken from ground truth during training or from the backbone’s predictions at test time (Spurio et al., 7 Aug 2025).

The scoring function is based on average 1D-IoU across all cross-pairs, with a length-balancing term. The unbalanced 1D IoU is

$T$7

The balanced IoU is then defined as

$T$8

Finally, the proposal score is

$T$9

The balancing factor is set to $E = [e_1,\ldots,e_T] \in \mathbb{R}^{T\times D},$0 by default, which is reported to yield “the best trade-off between exact overlap and segment-length similarity” (Spurio et al., 7 Aug 2025). The final output is the proposal $E = [e_1,\ldots,e_T] \in \mathbb{R}^{T\times D},$1 with the highest score; its $E = [e_1,\ldots,e_T] \in \mathbb{R}^{T\times D},$2 clusters become the discovered action intervals over the originally unlabeled parts of the video.

This scoring mechanism makes the meaning of “granularity” mathematically explicit. GGSM does not merely maximize overlap with known intervals; it also penalizes interval-length mismatch. A plausible implication is that the module encodes annotation style as well as boundary evidence, since the preferred proposal is the one whose interval lengths resemble those observed in the known-action annotations.

4. Pseudocode, computational behavior, and interface with UASA

The pseudocode given for GGSM consists of three stages: initialization of frame-level clusters, iterative agglomerative merges with adjacency-only complete-link distances, and scoring of all recorded proposals to select the best one (Spurio et al., 7 Aug 2025). In compact form, the inputs are

$E = [e_1,\ldots,e_T] \in \mathbb{R}^{T\times D},$3

and the output is

$E = [e_1,\ldots,e_T] \in \mathbb{R}^{T\times D},$4

Several implementation details are stated explicitly. The distance metric is Euclidean with complete-link, or “farthest-point,” aggregation. The number of proposals is $E = [e_1,\ldots,e_T] \in \mathbb{R}^{T\times D},$5, one per merge. On 50Salads (Fine split) inference, GGSM adds “$E = [e_1,\ldots,e_T] \in \mathbb{R}^{T\times D},$6 s overhead on top of a 10 s backbone pass” (Spurio et al., 7 Aug 2025).

GGSM’s output is then consumed by UASA. For each GGSM segment $E = [e_1,\ldots,e_T] \in \mathbb{R}^{T\times D},$7, UASA computes the average embedding

$E = [e_1,\ldots,e_T] \in \mathbb{R}^{T\times D},$8

fits several GMMs to $E = [e_1,\ldots,e_T] \in \mathbb{R}^{T\times D},$9 with different $t$0, selects the $t$1 with the lowest BIC, performs a final K-means clustering to obtain centroids $t$2, and assigns each segment the label of its closest centroid (Spurio et al., 7 Aug 2025). The source notes that, without GGSM, UASA would have to cluster every single frame—described as noisy—and also guess $t$3 from highly variable data.

An upstream contrastive loss is also reported as “not part of GGSM but crucial upstream.” Its temperature is $t$4, its weight is $t$5, positives are drawn from a truncated Gaussian with window $t$6 frames and $t$7–$t$8, and negatives are drawn from outside that window (Spurio et al., 7 Aug 2025). This does not alter GGSM’s algorithmic definition, but it conditions the embedding space on which GGSM operates.

5. Empirical behavior and ablations

The empirical analysis isolates GGSM by comparing a baseline without GGSM to the same system with GGSM added, and by comparing the full method to an off-the-shelf unsupervised boundary detector combined with UASA (Spurio et al., 7 Aug 2025). On Breakfast, adding GGSM changes unknown Edit from $t$9 to $C_t = \{t\}.$0 and unknown F1@10 from $C_t = \{t\}.$1 to $C_t = \{t\}.$2, while known MoF drops from $C_t = \{t\}.$3 to $C_t = \{t\}.$4. On 50Salads Fine, adding GGSM changes unknown F1@10 from $C_t = \{t\}.$5 to $C_t = \{t\}.$6, while known MoF changes from $C_t = \{t\}.$7 to $C_t = \{t\}.$8.

Against TW-FINCH plus UASA on 50Salads Fine, the reported unknown MoF is $C_t = \{t\}.$9 for TW-FINCH and $(C_p,C_q)$0 for “Ours (GGSM+UASA),” while unknown F1@10 is $(C_p,C_q)$1 for TW-FINCH and $(C_p,C_q)$2 for GGSM+UASA (Spurio et al., 7 Aug 2025). The source therefore positions GGSM as a more effective temporal discovery mechanism than an off-the-shelf unsupervised boundary detector in that setting.

The $(C_p,C_q)$3 ablation is especially diagnostic. On 50Salads Fine, $(C_p,C_q)$4 yields unknown MoF $(C_p,C_q)$5, $(C_p,C_q)$6 yields $(C_p,C_q)$7, and $(C_p,C_q)$8 degrades performance (Spurio et al., 7 Aug 2025). This directly supports the claim that the length-balancing term is not incidental; it is a major determinant of performance. Likewise, contrasting the backbone contrastive-loss weight $(C_p,C_q)$9 versus $$D(C_p, C_q) = \max_{u\in C_p,\; v\in C_q} \|e_u - e_v\|_2.$$0 yields unknown MoF $$D(C_p, C_q) = \max_{u\in C_p,\; v\in C_q} \|e_u - e_v\|_2.$$1 and $$D(C_p, C_q) = \max_{u\in C_p,\; v\in C_q} \|e_u - e_v\|_2.$$2, respectively, indicating that GGSM’s effectiveness depends in part on the structure of the upstream embedding space.

A common misconception would be to treat all improvements as uniformly beneficial across all metrics. The reported results do not support that simplification. GGSM improves unknown-action discovery metrics in the cited ablations, but known MoF can decrease. This makes the module a targeted intervention for unknown-action segmentation rather than a universal improvement over all aspects of temporal action segmentation.

6. Conceptual scope and relation to other granularity-guided segmentation work

GGSM is defined narrowly in Action Discovery: it is a temporal segmentation module that discovers intervals in unlabeled video stretches by aligning them with the temporal scale of known actions (Spurio et al., 7 Aug 2025). It does not assign semantic labels, it is not a neural network, and it does not function as a generic boundary detector divorced from annotation granularity. Its operative signal is the duration pattern of the known intervals.

A separate 2025 work, “Granular Computing-driven SAM: From Coarse-to-Fine Guidance for Prompt-Free Segmentation,” uses the phrase “Granularity-Guided Segmentation” for the coarse-stage module of a prompt-free image segmentation framework called Grc-SAM (Yu et al., 24 Nov 2025). In that spatial setting, the coarse stage adaptively extracts high-response regions from features, the fine stage applies finer patch partitioning with sparse local swin-style attention, and refined masks are encoded as latent prompt embeddings for the SAM decoder. The paper states that this coarse-to-fine, granularity-guided strategy yields gains in boundary precision, small-object recall, and computational efficiency, including “$$D(C_p, C_q) = \max_{u\in C_p,\; v\in C_q} \|e_u - e_v\|_2.$$3 FLOPs reduction” over the vanilla SAM prompt-free baseline.

This suggests that “granularity guidance” functions as a broader design principle across segmentation problems, but the mechanisms are substantially different. In Action Discovery, GGSM is an adjacency-restricted complete-link hierarchical clustering routine over frame embeddings with balanced 1D-IoU proposal scoring. In Grc-SAM, granularity guidance is implemented through multi-layer attention fusion, soft-thresholding, sparse local Swin-style attention, and latent prompt embedding for a SAM decoder (Yu et al., 24 Nov 2025). The shared vocabulary therefore should not obscure the methodological distinction: the temporal GGSM is a proposal-selection procedure governed by annotation granularity, whereas the spatial module in Grc-SAM is part of a coarse-to-fine transformer-based segmentation architecture.