---
title: Category-Aware Temporal Graph (CATS)
url: https://www.emergentmind.com/topics/category-aware-temporal-graph-cats
type: topic
---

# Category-Aware Temporal Graph (CATS)

Category-Aware Temporal Graph (CATS) is a module introduced in TEn-CATS for weakly supervised Audio-Visual Video Parsing (AVVP), where the objective is to identify event categories and their occurrence times in a video using only video-level multi-hot labels during training [2509.04086]. In this formulation, a video is divided into $T$ segments with audio and visual features $\{x_t^{(a)}, x_t^{(v)}\}_{t=1}^T$, and CATS constructs a category-aware, multi-scale temporal graph whose nodes are time segments and whose edges are selected according to the current segment-level predicted category distribution. The module is designed to address a specific failure mode of prior AVVP methods: noisy segment-level pseudo labels and indiscriminate attention can repeatedly amplify initial temporal localization errors during training [2509.04086].

## 1. Problem setting and architectural role

Within TEn-CATS, CATS is not a standalone predictor but a temporal propagation module placed after Bi-Directional Text Fusion (BiT) and HAN-based temporal aggregation [2509.04086]. The broader problem setting is weakly supervised AVVP: no segment-level ground-truth timestamps are available during training, labels are multi-hot, and segment-level pseudo labels may be noisy. On LLP, segment-level pseudo labels are provided by VALOR as 25-dimensional binary vectors per segment; these are converted into text prompts and encoded by frozen CLAP and CLIP to produce segment-level text embeddings [2509.04086].

BiT precedes CATS and performs semantic injection and dynamic calibration on audio and visual features. For each active category in a pseudo label, prompts such as “This is the sound of [event]” for audio and “This is a photo of [event]” for visual are encoded into text embeddings. Bidirectional cross-attention then updates both modality features and text features:
$$
F'_m = \mathrm{LN}\big(F_m + \mathrm{Dropout}(\mathrm{MHA}(F_m, F_e, F_e))\big),
$$
$$
F'_e = \mathrm{LN}\big(F_e + \mathrm{Dropout}(\mathrm{MHA}(F_e, F_m, F_m))\big).
$$
A global modality semantic vector is formed by self-attentive pooling,
$$
\alpha_t = \mathrm{softmax}_t(W_{\mathrm{att}}^\top f'_m(t)), \qquad g_m = \sum_{t=1}^{T}\alpha_t f'_m(t),
$$
and this global signal is fused with text embeddings through an MLP [2509.04086]. The result is a set of cleaner, text-calibrated segment features that act as more reliable semantic anchors for subsequent graph propagation.

This architectural placement is central to the intended behavior of CATS. Rather than propagating raw pseudo-label information through time, the graph is built from features that have already undergone semantic purification and calibration. This suggests that CATS is best understood as a controlled propagation mechanism, not merely a generic temporal GNN layer.

## 2. Category-aware graph construction

CATS constructs a separate temporal graph per modality, with shared parameters across the audio and visual branches [2509.04086]. Each time segment is a node. The defining property of the graph is that edge construction is driven by the current segment-level predicted category distribution
$$
P \in \mathbb{R}^{B \times T \times C},
$$
where $B$ is batch size, $T$ is the number of segments, and $C$ is the number of event categories.

The first step is hop preference estimation. Given a candidate hop set $\mathcal{K} = \{1,2,\dots,K\}$, segment-level category probabilities are mapped to hop preferences:
$$
H = P W_K, \qquad H \in \mathbb{R}^{B \times T \times K}, \qquad W_K \in \mathbb{R}^{C \times K}.
$$
Here, $H_{t,\cdot}$ determines how likely segment $t$ is to connect to neighbors at different hop sizes. Hop selection is then made differentiable with Gumbel-Softmax:
$$
s_{t,i} = \frac{\exp((h_{t,i}+g_{t,i})/\tau)}{\sum_{j=1}^{K}\exp((h_{t,j}+g_{t,j})/\tau)},
$$
where $g_{t,i}$ is Gumbel noise and $\tau$ is the temperature. The module selects Top-$k$ hops
$$
\mathcal{L}_t = \{l_1,\dots,l_k\}
$$
with highest $s_{t,i}$, producing an adaptive multi-scale neighborhood for each segment [2509.04086].

The second step is category-conditioned temporal decay. CATS learns a per-segment decay factor from the predicted category distribution:
$$
\lambda_t = P_t W_\lambda, \qquad W_\lambda \in \mathbb{R}^{C}.
$$
This allows different categories to express different temporal persistence. For each selected hop $l_i \in \mathcal{L}_t$ with $t+l_i \le T$, the directed edge $(t \to t+l_i)$ receives weight
$$
w_{t,i} = s_{t,i} \cdot \exp(-\lambda_t l_i).
$$
A self-loop $(t \to t)$ with weight $1$ is always included. In sparse adjacency form, $A_{t,t}=1$ and $A_{t,t+l_i}=w_{t,i}$ [2509.04086].

The multi-scale aspect follows directly from allowing multiple hop sizes per node, while the category-aware aspect arises because both hop preference and decay are functions of $P$. In implementation, edge weights are passed as edge attributes to TransformerConv, so explicit symmetric normalization,
$$
\tilde{A}=D^{-1/2} A D^{-1/2},
$$
is optional rather than required [2509.04086].

## 3. Propagation, fusion, and prediction

After graph construction, CATS performs two residual graph attention layers using TransformerConv with four heads [2509.04086]:
$$
H^{(1)} = \mathrm{GAT}_1(X, A, W) + R_1(X),
$$
$$
H^{(2)} = \mathrm{GAT}_2(H^{(1)}, A, W) + R_2(H^{(1)}),
$$
where edge weights are used as one-dimensional edge attributes within attention. The schematic attention formulation is
$$
e_{t,t'} = a^\top [W_q h_t \,\|\, W_k h_{t'} \,\|\, \phi(W_{t,t'})],
$$
$$
\alpha_{t,t'} = \mathrm{softmax}_{t'}(\mathrm{LeakyReLU}(e_{t,t'})),
$$
$$
h_t^{(\mathrm{next})} = \sigma\Big(\sum_{t' \in \mathcal{N}(t)} \alpha_{t,t'} W_v h_{t'}\Big),
$$
which biases message passing toward decay-weighted, category-consistent neighbors [2509.04086].

CATS then aggregates global context by mean pooling across nodes:
$$
u = \mathrm{MeanPool}(H^{(2)}) \in \mathbb{R}^{B \times d}.
$$
This is followed by gated local-global fusion:
$$
X_u = \sigma(W_u[H^{(2)} \,\|\, u]) \odot H^{(2)} + \big(1-\sigma(W_u[H^{(2)} \,\|\, u])\big)\odot u,
$$
with $W_u \in \mathbb{R}^{2d \times d}$ [2509.04086]. In TEn-CATS, the CATS output is then fused with the HAN output:
$$
G = \sigma(W_f[F_{\mathrm{HAN}} \,\|\, F_{\mathrm{CATS}}] + b_f),
$$
$$
F_{\mathrm{Fuse}} = G \odot F_{\mathrm{HAN}} + (1-G)\odot F_{\mathrm{CATS}}.
$$

Predictions are computed at segment level:
$$
p_{t,c} = \sigma(w_c^\top h_t),
$$
and aggregated to video-level predictions via MMIL pooling,
$$
\hat{y}_c = \mathrm{pool}_t(p_{t,c}),
$$
followed by a weakly supervised binary cross-entropy MIL loss:
$$
\mathcal{L}_{\mathrm{MIL}} = -\sum_{c \in \mathcal{C}}\big[y_c \log(\hat{y}_c) + (1-y_c)\log(1-\hat{y}_c)\big].
$$
The paper does not introduce additional loss terms specific to CATS beyond standard MIL; explicit extra regularizers are described as unnecessary because category-aware edge selection and exponential decay inherently regularize propagation [2509.04086].

## 4. Noise control and temporal semantics

The central rationale for CATS is that temporal propagation should be semantically selective rather than indiscriminate [2509.04086]. Earlier AVVP methods are described as following two broad directions: enhanced temporal modeling through attention, or richer pseudo-label generation. In the formulation motivating TEn-CATS, the first direction tends to treat noisy pseudo labels as reliable supervision, while the second can spread pseudo-label errors across all frames through unconstrained attention [2509.04086].

CATS addresses this by conditioning temporal neighborhoods on learned category probabilities instead of directly diffusing raw pseudo-label information. Three mechanisms are emphasized. First, temporal neighborhoods are chosen according to the predicted category distribution through $W_K$ and Gumbel-Softmax hop selection. Second, long-range propagation is modulated by category-dependent decay $\lambda_t$, which penalizes distant hops more strongly when semantics are transient. Third, Top-$k$ selection keeps the graph sparse and constrains information flow [2509.04086].

The interaction with BiT is equally important. BiT is responsible for aligning modality features and pseudo-label-derived text embeddings, then calibrating them with feature-derived global semantics $g_m$. CATS operates on the resulting learned probabilities $P$ after BiT and HAN rather than on raw pseudo labels [2509.04086]. The intended effect is that unreliable segments receive fewer or weaker long-range connections, confining propagation to semantically consistent neighborhoods.

The same design also exposes a limitation. Category guidance depends on $P$; if $P$ is wrong early in training, hop selection can be suboptimal. The self-loop and decay mitigate but do not eliminate this risk [2509.04086]. A second limitation noted in the paper is that event-level AV recall can be lower when fusion is conservative, and rare or asynchronous events may require tailored pooling [2509.04086].

## 5. Datasets, implementation, and empirical behavior

TEn-CATS is evaluated on LLP and UnAV-100 [2509.04086]. LLP contains 25 classes and 11,849 videos, with 10,000 weakly labeled training videos, 649 fully labeled validation videos, and 1,200 test videos; each video is divided into 10 one-second segments. UnAV-100 contains 100 classes, 10,790 videos, and more than 30,000 events, with weakly supervised training from video-level labels [2509.04086].

On LLP, the feature extractors are frozen CLAP for audio and CLIP plus 3D ResNet for visual features, with dimensions 768, 768, and 512 respectively. On UnAV-100, visual features are I3D two-stream (RGB + RAFT) with 2048 dimensions, and audio features are VGGish with 128 dimensions. BiT is not used on UnAV-100 because CLAP and CLIP text features are unavailable there; CATS is evaluated standalone [2509.04086]. CATS uses two TransformerConv layers with four heads, residual connections, edge weights as one-dimensional edge attributes, and mean pooling plus gated local-global fusion. On LLP, the graph hyperparameters are candidate hops $\mathcal{K}=\{1,\dots,9\}$, Top-$k=3$, and Gumbel-Softmax temperature $\tau=1.0$ [2509.04086].

The reported performance highlights are as follows:

| Benchmark | Setting | Reported result |
|---|---|---|
| LLP | Segment-level, TEn-CATS | A=73.7, V=74.1, AV=63.2, Type@AV=70.3, Event@AV=73.9, Avg=66.5 |
| LLP | Event-level, TEn-CATS | A=61.1, V=70.3, AV=54.3, Type@AV=61.9, Event@AV=61.9 |
| UnAV-100 | CATS only, AV branch | Segment-level mAP 41.9; event-level mAP 47.5 |

On LLP segment-level evaluation, these numbers are described as state-of-the-art on several key indicators, with gains of $+3.5$ in A and $+2.8$ in V over the prior best, as well as $+1.8$ in Type@AV and $+5.1$ in Event@AV [2509.04086]. On LLP event-level evaluation, the model is described as competitive in Type@AV, while AV event-level performance is slightly conservative due to precision-oriented fusion and pooling [2509.04086]. On UnAV-100, CATS alone reaches segment-level mAP 41.9, which is $+0.4$ over CoLeaF, and event-level mAP 47.5, comparable to CoLeaF’s 47.8, indicating generalizability under large-scale weak supervision even without BiT [2509.04086].

Ablation analysis shows that CATS-only improves V, AV, and Type@AV modestly relative to the retrained CoLeaF baseline, BiT-only achieves the strongest audio and Event@AV results, and the full BiT+CATS model provides the best balance between unimodal strength and multimodal temporal consistency [2509.04086]. Hop-size sensitivity experiments indicate that moderate Top-$k$, exemplified by $k=3$, gives the best trade-off, whereas too small a value limits temporal modeling and too large a value risks over-smoothing or misalignment; the visual branch benefits particularly from $k \approx 3$ [2509.04086]. The graph construction cost is $O(T \cdot k)$ edges per video, and TransformerConv cost is $O(E \cdot H \cdot d)$ with $E \approx T \cdot k$ [2509.04086].

## 6. Terminological scope and related uses of “CATS”

The name “CATS” is not uniform across recent arXiv literature. In the AVVP setting, CATS explicitly denotes the “Category-Aware Temporal Graph” module in TEn-CATS [2509.04086]. In contrast, the video HOI paper “From Category to Scenery: An End-to-End Framework for Multi-Person Human-Object Interaction Recognition in Videos” uses CATS to mean “Category-to-Scenery,” a different framework that first builds separate human and object geometric graphs, fuses them with category-matched visual features, constructs a per-frame scenery interactive graph with GAT reasoning, and only afterward models temporal dynamics with Bi-GRU and Gumbel-Softmax segmentation [2407.00917]. There, the acronym refers to staged category-level to scenery-level reasoning rather than to a category-aware temporal graph in the AVVP sense.

A second neighboring line of work appears in temporal knowledge graphs. “Temporal Knowledge Graph Hyperedge Forecasting: Exploring Entity-to-Category Link Prediction” does not introduce the term CATS, but it develops a category-aware extension of TLogic, called C-TLogic, in which temporal facts are represented as sextuples
$$
(s,r,o,t,c^s,c^o),
$$
rules are mined and applied under category constraints, and entity-level predictions are aggregated into category forecasts through Noisy-OR or Max+ [2510.24240]. The paper itself states that this extension naturally instantiates what one could call a category-aware temporal graph, but its terminology is “types,” “categories,” and “hyperedge forecasting,” not CATS [2510.24240].

This suggests that “Category-Aware Temporal Graph” is presently best treated as a model-specific designation tied to TEn-CATS rather than as a field-wide standardized acronym. Across these uses, the common thread is category-conditioned structure over temporal data, but the operational meaning differs substantially: weakly supervised semantic propagation in AVVP [2509.04086], category-to-scenery HOI reasoning in videos [2407.00917], and typed temporal rule reasoning for entity-to-category forecasting in temporal knowledge graphs [2510.24240].

Source: https://www.emergentmind.com/topics/category-aware-temporal-graph-cats