Temporal Cluster Assignment (TCA)

Updated 8 July 2026

Temporal Cluster Assignment (TCA) is a training-free module that refines token clustering in online video segmentation using temporally coherent reference tokens.
It operates by clustering tokens early in non-key frames and later refining them with richer tokens from delayed keyframes, preserving the fixed token structure.
TCA improves the accuracy-speed trade-off of clustering-based methods on Swin Transformer backbones by leveraging temporal redundancy without fine-tuning.

Temporal Cluster Assignment (TCA) is a training-free, plug-and-play temporal refinement module for token clustering in online video segmentation with Swin Transformer backbones. Its central mechanism is to cluster tokens early in non-key frames to save compute, but recover quality by using temporally coherent reference tokens from a nearby keyframe to refine those clusters later. In this form, TCA is not a generic temporal clustering framework but a deployment-oriented procedure for improving the accuracy-speed trade-off of existing clustering-based token reduction methods while preserving the fixed per-window token structure required by Swin’s window attention mechanism (Yung et al., 7 Aug 2025).

1. Conceptual definition and problem setting

TCA was introduced to address a specific bottleneck in real-time video segmentation. Swin Transformer is widely used because it provides hierarchical multi-scale features and window-based attention, but it remains computationally expensive, especially in larger variants used for dense prediction. Standard token pruning methods are a poor fit in this setting because they arbitrarily remove tokens, which harms segmentation by discarding spatial detail, and because Swin requires a fixed number of tokens per window, making arbitrary dropping incompatible with the attention implementation. TCA therefore builds on token clustering rather than token pruning, and extends clustering-based methods such as Expedit and AiluRus by exploiting temporal redundancy across adjacent frames rather than treating each frame independently (Yung et al., 7 Aug 2025).

The method’s defining idea is temporal reuse. Adjacent video frames are usually highly redundant, so a nearby frame can supply richer tokens to stabilize the representation of the current frame. TCA operationalizes this by keeping a reference frame unclustered for longer, storing its richer tokens, and then using them to refine the compressed representation of later frames. This distinguishes TCA from frame-wise clustering methods that cluster each frame independently and therefore ignore temporal coherence (Yung et al., 7 Aug 2025).

A common misconception is to view TCA as a token dropping scheme. In the formulation of the paper, it instead refines token clusters using temporal correlations, and it does so without fine-tuning. Another misconception is to treat it as a separate segmentation backbone. The method is inserted into an existing clustering pipeline on top of Swin features; it is therefore a module rather than a replacement architecture (Yung et al., 7 Aug 2025).

2. Operational pipeline

The baseline pipeline on which TCA operates is a standard clustering pipeline in which tokens are extracted by the Swin backbone, clustered at some intermediate layer $\alpha$ from $M$ tokens per window down to $N$ tokens per window, and later reconstructed back to $M$ tokens before decoding. The paper describes this as a “high–low–high” pipeline. TCA modifies this pipeline by introducing two kinds of frames: key/reference frames and non-key frames (Yung et al., 7 Aug 2025).

For a keyframe, clustering is delayed until layer $\alpha+\beta$ . Its tokens therefore remain uncompressed for longer and become more informative. The tokens at layer $\alpha+\beta$ are then stored as reference tokens, after which clustering proceeds. For a non-key frame, clustering still occurs early at layer $\alpha$ , but at layer $\alpha+\beta$ the clustered tokens are matched against the stored reference tokens and refined before later Swin blocks continue processing. After a fixed interval $f_{\max}$ , a new keyframe is created because the reference representation drifts over time (Yung et al., 7 Aug 2025).

The paper writes the reference tokens as

$x_{\text{ref}} \in \mathbb{R}^{K \times M \times L},$

and the current-frame tokens as

$M$ 0

where $M$ 1 is the number of windows, $M$ 2 is the number of tokens per window before clustering, $M$ 3 is the number of clusters per window after clustering, and $M$ 4 is the token dimensionality. The procedure is window-wise rather than global, which preserves the structure required by Swin attention (Yung et al., 7 Aug 2025).

This pipeline has a strict online character. Video processing is online with batch size 1, and the design does not rely on future frames. A plausible implication is that TCA prioritizes deployability and causal inference-time behavior over offline temporal smoothing.

The core temporal assignment step is defined by L2 distance between reference tokens and clustered tokens. For each reference token $M$ 5, TCA assigns it to the nearest clustered token:

$M$ 6

The set of reference tokens assigned to cluster $M$ 7 is

$M$ 8

This assignment is the mechanism that transfers temporally coherent information from the keyframe to the current frame (Yung et al., 7 Aug 2025).

The paper studies three refinement rules. Cluster-Guided Averaging (CGA) averages the clustered token with all assigned reference tokens:

$M$ 9

Reference-Based Substitution (RBS) replaces the clustered token by the average of assigned reference tokens:

$N$ 0

Adaptive Cluster Reinforcement (ACR) performs weighted averaging using both assigned reference tokens and the current cluster size:

$N$ 1

with

$N$ 2

The paper further reports that RBS works better at early clustering locations, whereas CGA works better at later clustering locations, and therefore introduces a switching rule: if clustering happens before threshold $N$ 3, use RBS; otherwise use CGA. In the final method, $N$ 4 (Yung et al., 7 Aug 2025).

Strategy	Update rule	Reported behavior
CGA	Average current clustered token with assigned reference tokens	Better at later clustering locations
RBS	Replace clustered token by average of assigned reference tokens	Better at early clustering locations
ACR	Weighted averaging using $N$ 5 and $N$ 6	Studied as a third refinement strategy

The assignment metric was also ablated against cosine distance and another distance variant. The paper reports that L2 distance is consistently best, with similar compute cost, and is therefore used in the final method (Yung et al., 7 Aug 2025).

4. Architectural placement and implementation regime

TCA is inserted in Stage 3 of Swin, at a user-chosen clustering location $N$ 7, with refinement delayed to $N$ 8. It operates within each window, preserves token count per window, and is compatible with the fixed-token design of windowed self-attention. The paper sweeps $N$ 9 from 0 to 10 in Stage 3 and uses $M$ 0 in the final setup (Yung et al., 7 Aug 2025).

The reported implementation is built on the official MinVIS implementation with a pretrained Swin-Large backbone, and TCA is applied to Expedit and AiluRus. Inputs are resized to minimum size 480 for the VIS datasets. For the surgical dataset, the model uses Swin Large + a three-layer convolutional decoder (SwinSeg) and is trained with early stopping. All experiments run on a single A6000 GPU. The selected hyperparameters are $M$ 1, $M$ 2, and $M$ 3, while cluster sizes $M$ 4 are swept from $M$ 5 down to $M$ 6 (Yung et al., 7 Aug 2025).

The paper also examines two additional extensions—dynamic keyframe intervals and a cluster memory module—but discards both because they do not improve enough to justify the added cost. This is significant because it locates the final form of TCA within a deliberately simple fixed-interval design rather than a larger memory architecture (Yung et al., 7 Aug 2025).

5. Empirical results, ablations, and limitations

TCA is evaluated on YouTube-VIS 2019, YouTube-VIS 2021, OVIS, and a private surgical video semantic segmentation dataset. At the highest-accuracy settings, AiluRus + TCA improves AP from 59.7 to 61.0 on YTVIS19 and from 53.4 to 54.4 on YTVIS21, while Expedit + TCA improves AP from 60.5 to 61.7 on YTVIS19 and from 55.8 to 56.8 on YTVIS21. At 30% speedup, the gains remain substantial, and at 50% speedup they become markedly larger: for AiluRus + TCA, YTVIS19 AP improves 38.0 → 49.9 and YTVIS21 AP improves 30.0 → 39.4; for Expedit + TCA, YTVIS19 AP improves 42.5 → 48.0 and YTVIS21 AP improves 36.8 → 40.2. The paper emphasizes that TCA particularly helps at high compression, where frame-wise clustering degrades quickly (Yung et al., 7 Aug 2025).

On OVIS, which the paper characterizes as harder due to occlusion and longer sequences, the same pattern holds. At the highest-accuracy setting, AiluRus + TCA improves AP 38.1 → 38.9 and Expedit + TCA improves AP 38.9 → 39.5. At 50% speedup, AiluRus + TCA improves 15.6 → 22.5, and Expedit + TCA improves 16.4 → 20.8. On the surgical dataset, Expedit alone can achieve about 63% FPS improvement while mIoU drops from 0.609 to 0.374, whereas TCA achieves a similar speedup of about 66% while maintaining a much higher mIoU of 0.525 (Yung et al., 7 Aug 2025).

The ablations clarify which components matter. The paper states that simply adding a reference frame without assignment and refinement gives only a small gain; the large improvement comes from the assignment + refinement step itself. It also reports that increasing $M$ 7 lets the reference tokens remain unclustered longer, which helps quality but raises cost; gains then saturate, leading to the choice $M$ 8. For the keyframe interval $M$ 9, performance is stable up to around 7, after which AP drops because of feature drift, motivating the choice $\alpha+\beta$ 0. The final system can exceed the MinVIS baseline on YTVIS21 by about 1 AP while also being 12% faster and using 15% fewer FLOPs (Yung et al., 7 Aug 2025).

The paper identifies several limitations. Reference drift necessitates periodic keyframe refresh. Performance still depends on choosing $\alpha+\beta$ 1, $\alpha+\beta$ 2, and $\alpha+\beta$ 3 well. The explored memory module did not justify its overhead. The method is explicitly designed for online/streaming settings and therefore does not use future frames. Broader generalization remains limited: the paper reports that a preliminary experiment on ViT-based depth estimation shows a small gain in both accuracy and FPS, but does not treat that result as a full validation beyond Swin-based video segmentation (Yung et al., 7 Aug 2025).

6. Relation to temporal clustering and temporal assignment literature

Within the broader literature, TCA belongs to a family of methods that balance local clustering quality with temporal coherence, but it does so at the level of window-wise transformer tokens rather than communities, trajectories, or frame labels. Earlier work on dynamic networks proposed a temporal clustering framework based on network generative models, PARAFAC decomposition, K-means, the Silhouette criterion, a similarity score, and a segmentation algorithm for formation, dissolution, and lifetime detection (Tu et al., 2016). TDCK-Means introduced a time-aware dissimilarity measure and soft semi-supervised constraints to encourage adjacent observations of the same entity to remain in contiguous temporal segments (Rizoiu et al., 2016). BCLC addressed temporal clustering in bibliographic coupling networks by selecting among multiple local community partitions so as to balance snapshot quality and historical continuity (Cambe et al., 2020).

Other lines of work push temporal cluster assignment into latent-feature or sequence-decoding regimes. Deep Temporal Clustering (DTC) combines a temporal autoencoder with a temporal clustering layer that produces soft assignments via a Student’s t-distribution kernel in latent space (Madiraju et al., 2018). TAEC combines a temporal embedding network with within-video spectral clustering, cross-video global cluster assignment, and Viterbi decoding to recover coherent temporal action segments (Lin et al., 2023). Abstract formulations of temporal clustering include temporally coherent sequences of hierarchical clusterings encoded by ultrametrics and low-distortion correspondences (Dey et al., 2017), trajectory-based clustering of unlabeled point sets with objectives $\alpha+\beta$ 4, $\alpha+\beta$ 5, and $\alpha+\beta$ 6 (Dey et al., 2017), and partial-order formulations that infer ordered temporal clusters from a single dynamic-network snapshot (Turowski et al., 2019). Other related approaches jointly cluster and align temporal objects by optimal warps in one-point-per-experiment data (Arsenteva et al., 2023), or treat temporal-clustering invariance as a useful inductive bias for irregular healthcare time series via grid-count, cluster-count, augmentation, and MultiResolution Ensemble (Bahadori et al., 2019).

Against this background, TCA is narrower in scope but more specialized in mechanism. It does not fit a generic supervised or unsupervised temporal sequence model; instead, it is a lightweight and effective, fine-tuning-free strategy for refining clustering-based token reduction in online video segmentation (Yung et al., 7 Aug 2025). This suggests that, in its 2025 formulation, “Temporal Cluster Assignment” refers not to a universal clustering paradigm but to a specific temporal token-refinement procedure that exploits reference-frame coherence while preserving Swin’s window structure.