---
title: Temporal Cluster Assignment (TCA)
url: https://www.emergentmind.com/topics/temporal-cluster-assignment-tca
type: topic
---

# Temporal Cluster Assignment (TCA)

Temporal Cluster Assignment (TCA) is a **training-free, plug-and-play temporal refinement module** for **token clustering** in **online video segmentation** with **Swin Transformer** backbones. Its central mechanism is to **cluster tokens early in non-key frames to save compute, but recover quality by using temporally coherent reference tokens from a nearby keyframe to refine those clusters later**. In this form, TCA is not a generic temporal clustering framework but a deployment-oriented procedure for improving the **accuracy-speed trade-off** of existing clustering-based token reduction methods while preserving the fixed per-window token structure required by Swin’s **window attention mechanism** [2508.05851].

## 1. Conceptual definition and problem setting

TCA was introduced to address a specific bottleneck in real-time video segmentation. **Swin Transformer** is widely used because it provides **hierarchical multi-scale features** and **window-based attention**, but it remains computationally expensive, especially in larger variants used for dense prediction. Standard token pruning methods are a poor fit in this setting because they arbitrarily remove tokens, which harms segmentation by discarding spatial detail, and because Swin requires a **fixed number of tokens per window**, making arbitrary dropping incompatible with the attention implementation. TCA therefore builds on **token clustering** rather than token pruning, and extends clustering-based methods such as **Expedit** and **AiluRus** by exploiting **temporal redundancy** across adjacent frames rather than treating each frame independently [2508.05851].

The method’s defining idea is temporal reuse. Adjacent video frames are usually highly redundant, so a nearby frame can supply richer tokens to stabilize the representation of the current frame. TCA operationalizes this by keeping a **reference frame** unclustered for longer, storing its richer tokens, and then using them to refine the compressed representation of later frames. This distinguishes TCA from frame-wise clustering methods that cluster each frame independently and therefore ignore temporal coherence [2508.05851].

A common misconception is to view TCA as a token dropping scheme. In the formulation of the paper, it instead **refines token clusters using temporal correlations**, and it does so without fine-tuning. Another misconception is to treat it as a separate segmentation backbone. The method is inserted into an existing clustering pipeline on top of Swin features; it is therefore a module rather than a replacement architecture [2508.05851].

## 2. Operational pipeline

The baseline pipeline on which TCA operates is a standard clustering pipeline in which tokens are extracted by the Swin backbone, clustered at some intermediate layer \(\alpha\) from \(M\) tokens per window down to \(N\) tokens per window, and later reconstructed back to \(M\) tokens before decoding. The paper describes this as a “high–low–high” pipeline. TCA modifies this pipeline by introducing two kinds of frames: **key/reference frames** and **non-key frames** [2508.05851].

For a **keyframe**, clustering is delayed until layer \(\alpha+\beta\). Its tokens therefore remain uncompressed for longer and become more informative. The tokens at layer \(\alpha+\beta\) are then stored as **reference tokens**, after which clustering proceeds. For a **non-key frame**, clustering still occurs early at layer \(\alpha\), but at layer \(\alpha+\beta\) the clustered tokens are matched against the stored reference tokens and refined before later Swin blocks continue processing. After a fixed interval \(f_{\max}\), a new keyframe is created because the reference representation drifts over time [2508.05851].

The paper writes the reference tokens as
$$
x_{\text{ref}} \in \mathbb{R}^{K \times M \times L},
$$
and the current-frame tokens as
$$
x \in \mathbb{R}^{K \times M \times L} \mapsto x_{\text{cluster}} \in \mathbb{R}^{K \times N \times L},
$$
where \(K\) is the number of windows, \(M\) is the number of tokens per window before clustering, \(N\) is the number of clusters per window after clustering, and \(L\) is the token dimensionality. The procedure is window-wise rather than global, which preserves the structure required by Swin attention [2508.05851].

This pipeline has a strict online character. Video processing is **online** with **batch size 1**, and the design does not rely on future frames. A plausible implication is that TCA prioritizes deployability and causal inference-time behavior over offline temporal smoothing.

## 3. Assignment rule and refinement strategies

The core temporal assignment step is defined by **L2 distance** between reference tokens and clustered tokens. For each reference token \(i\), TCA assigns it to the nearest clustered token:
$$
j^* = \arg\min_j \|x_{\text{ref},i} - x_{\text{cluster},j}\|_2,
\quad \forall i \in \{1,\dots,M\}.
$$
The set of reference tokens assigned to cluster \(j\) is
$$
\mathcal{R}_j = \{ i \mid j_i^* = j \}.
$$
This assignment is the mechanism that transfers temporally coherent information from the keyframe to the current frame [2508.05851].

The paper studies three refinement rules. **Cluster-Guided Averaging (CGA)** averages the clustered token with all assigned reference tokens:
$$
x_{\text{cluster},j}^{\text{refined}}
=
\frac{1}{1 + |\mathcal{R}_j|}
\left(
x_{\text{cluster},j}
+
\sum_{i \in \mathcal{R}_j} x_{\text{ref},i}
\right).
$$
**Reference-Based Substitution (RBS)** replaces the clustered token by the average of assigned reference tokens:
$$
x_{\text{cluster},j}^{\text{refined}}
=
\frac{1}{|\mathcal{R}_j|}
\sum_{i \in \mathcal{R}_j} x_{\text{ref},i},
\quad \text{if } |\mathcal{R}_j| > 0.
$$
**Adaptive Cluster Reinforcement (ACR)** performs weighted averaging using both assigned reference tokens and the current cluster size:
$$
x_{\text{cluster},j}^{\text{refined}}
=
\frac{|\mathcal{R}_j| \cdot \bar{x}_{\text{ref},j} + |\mathcal{C}_j| x_{\text{cluster},j}}
{|\mathcal{R}_j| + |\mathcal{C}_j|},
$$
with
$$
\bar{x}_{\text{ref},j}
=
\frac{1}{|\mathcal{R}_j|}
\sum_{i \in \mathcal{R}_j} x_{\text{ref},i},
\quad \text{if } |\mathcal{R}_j| > 0.
$$
The paper further reports that **RBS works better at early clustering locations**, whereas **CGA works better at later clustering locations**, and therefore introduces a switching rule: if clustering happens before threshold \(d\), use **RBS**; otherwise use **CGA**. In the final method, \(d=2\) [2508.05851].

| Strategy | Update rule | Reported behavior |
|---|---|---|
| **CGA** | Average current clustered token with assigned reference tokens | Better at later clustering locations |
| **RBS** | Replace clustered token by average of assigned reference tokens | Better at early clustering locations |
| **ACR** | Weighted averaging using \(|\mathcal{R}_j|\) and \(|\mathcal{C}_j|\) | Studied as a third refinement strategy |

The assignment metric was also ablated against cosine distance and another distance variant. The paper reports that **L2 distance is consistently best**, with similar compute cost, and is therefore used in the final method [2508.05851].

## 4. Architectural placement and implementation regime

TCA is inserted in **Stage 3 of Swin**, at a user-chosen clustering location \(\alpha\), with refinement delayed to \(\alpha+\beta\). It operates **within each window**, preserves token count per window, and is compatible with the fixed-token design of windowed self-attention. The paper sweeps \(\alpha\) from 0 to 10 in Stage 3 and uses \(\beta = 6\) in the final setup [2508.05851].

The reported implementation is built on the **official MinVIS implementation** with a **pretrained Swin-Large backbone**, and TCA is applied to **Expedit** and **AiluRus**. Inputs are resized to minimum size **480** for the VIS datasets. For the surgical dataset, the model uses **Swin Large + a three-layer convolutional decoder (SwinSeg)** and is trained with early stopping. All experiments run on a **single A6000 GPU**. The selected hyperparameters are \(\beta = 6\), \(f_{\max} = 6\), and \(d = 2\), while cluster sizes \(N\) are swept from \(8 \times 8\) down to \(2 \times 2\) [2508.05851].

The paper also examines two additional extensions—**dynamic keyframe intervals** and a **cluster memory module**—but discards both because they do not improve enough to justify the added cost. This is significant because it locates the final form of TCA within a deliberately simple fixed-interval design rather than a larger memory architecture [2508.05851].

## 5. Empirical results, ablations, and limitations

TCA is evaluated on **YouTube-VIS 2019**, **YouTube-VIS 2021**, **OVIS**, and a private **surgical video semantic segmentation dataset**. At the highest-accuracy settings, **AiluRus + TCA** improves AP from **59.7** to **61.0** on YTVIS19 and from **53.4** to **54.4** on YTVIS21, while **Expedit + TCA** improves AP from **60.5** to **61.7** on YTVIS19 and from **55.8** to **56.8** on YTVIS21. At **30% speedup**, the gains remain substantial, and at **50% speedup** they become markedly larger: for **AiluRus + TCA**, YTVIS19 AP improves **38.0 → 49.9** and YTVIS21 AP improves **30.0 → 39.4**; for **Expedit + TCA**, YTVIS19 AP improves **42.5 → 48.0** and YTVIS21 AP improves **36.8 → 40.2**. The paper emphasizes that TCA particularly helps at high compression, where frame-wise clustering degrades quickly [2508.05851].

On **OVIS**, which the paper characterizes as harder due to occlusion and longer sequences, the same pattern holds. At the highest-accuracy setting, **AiluRus + TCA** improves AP **38.1 → 38.9** and **Expedit + TCA** improves AP **38.9 → 39.5**. At **50% speedup**, **AiluRus + TCA** improves **15.6 → 22.5**, and **Expedit + TCA** improves **16.4 → 20.8**. On the surgical dataset, **Expedit** alone can achieve about **63% FPS improvement** while mIoU drops from **0.609 to 0.374**, whereas **TCA** achieves a similar speedup of about **66%** while maintaining a much higher mIoU of **0.525** [2508.05851].

The ablations clarify which components matter. The paper states that simply adding a reference frame without assignment and refinement gives only a small gain; the large improvement comes from the **assignment + refinement** step itself. It also reports that increasing \(\beta\) lets the reference tokens remain unclustered longer, which helps quality but raises cost; gains then saturate, leading to the choice \(\beta = 6\). For the keyframe interval \(f_{\max}\), performance is stable up to around 7, after which AP drops because of feature drift, motivating the choice \(f_{\max}=6\). The final system can exceed the **MinVIS baseline** on YTVIS21 by about **1 AP** while also being **12% faster** and using **15% fewer FLOPs** [2508.05851].

The paper identifies several limitations. **Reference drift** necessitates periodic keyframe refresh. Performance still depends on choosing \(\alpha\), \(N\), and \(\beta\) well. The explored memory module did not justify its overhead. The method is explicitly designed for **online/streaming settings** and therefore does not use future frames. Broader generalization remains limited: the paper reports that a **preliminary experiment on ViT-based depth estimation** shows a small gain in both accuracy and FPS, but does not treat that result as a full validation beyond Swin-based video segmentation [2508.05851].

## 6. Relation to temporal clustering and temporal assignment literature

Within the broader literature, TCA belongs to a family of methods that balance local clustering quality with temporal coherence, but it does so at the level of **window-wise transformer tokens** rather than communities, trajectories, or frame labels. Earlier work on dynamic networks proposed a temporal clustering framework based on network generative models, **PARAFAC decomposition**, **K-means**, the **Silhouette criterion**, a **similarity score**, and a **segmentation algorithm** for formation, dissolution, and lifetime detection [1605.08074]. **TDCK-Means** introduced a **time-aware dissimilarity measure** and **soft semi-supervised constraints** to encourage adjacent observations of the same entity to remain in contiguous temporal segments [1601.02603]. **BCLC** addressed temporal clustering in bibliographic coupling networks by selecting among multiple local community partitions so as to balance **snapshot quality** and **historical continuity** [2012.01287].

Other lines of work push temporal cluster assignment into latent-feature or sequence-decoding regimes. **Deep Temporal Clustering (DTC)** combines a temporal autoencoder with a **temporal clustering layer** that produces soft assignments via a **Student’s t-distribution kernel** in latent space [1802.01059]. **TAEC** combines a temporal embedding network with **within-video spectral clustering**, **cross-video global cluster assignment**, and **Viterbi decoding** to recover coherent temporal action segments [2303.05166]. Abstract formulations of temporal clustering include temporally coherent sequences of hierarchical clusterings encoded by **ultrametrics** and low-distortion correspondences [1707.09904], trajectory-based clustering of unlabeled point sets with objectives \(k\), \(r\), and \(\delta\) [1704.05964], and partial-order formulations that infer **ordered temporal clusters** from a single dynamic-network snapshot [1905.00672]. Other related approaches jointly cluster and align temporal objects by optimal warps in one-point-per-experiment data [2311.10282], or treat **temporal-clustering invariance** as a useful inductive bias for irregular healthcare time series via **grid-count**, **cluster-count**, augmentation, and **MultiResolution Ensemble** [1904.12206].

Against this background, TCA is narrower in scope but more specialized in mechanism. It does not fit a generic supervised or unsupervised temporal sequence model; instead, it is a **lightweight and effective, fine-tuning-free strategy** for refining clustering-based token reduction in online video segmentation [2508.05851]. This suggests that, in its 2025 formulation, “Temporal Cluster Assignment” refers not to a universal clustering paradigm but to a specific temporal token-refinement procedure that exploits reference-frame coherence while preserving Swin’s window structure.

Source: https://www.emergentmind.com/topics/temporal-cluster-assignment-tca