Category-Aware Temporal Graph (CATS)

• Category-Aware Temporal Graph (CATS) is a module in TEn-CATS that builds multi-scale temporal graphs using segment-level predicted category distributions to guide audio-visual video parsing.
• It mitigates noisy pseudo-label propagation through differentiable Gumbel-Softmax hop selection and category-conditioned temporal decay, ensuring semantically consistent message passing.
• Empirical results on LLP and UnAV-100 demonstrate that CATS improves segment-level and event-level performance by effectively fusing modality features and constraining propagation.

Category-Aware Temporal Graph (CATS) is a module introduced in TEn-CATS for weakly supervised Audio-Visual Video Parsing (AVVP), where the objective is to identify event categories and their occurrence times in a video using only video-level multi-hot labels during training (Chen et al., 4 Sep 2025). In this formulation, a video is divided into $T$ segments with audio and visual features $\{x_t^{(a)}, x_t^{(v)}\}_{t=1}^T$, and CATS constructs a category-aware, multi-scale temporal graph whose nodes are time segments and whose edges are selected according to the current segment-level predicted category distribution. The module is designed to address a specific failure mode of prior AVVP methods: noisy segment-level pseudo labels and indiscriminate attention can repeatedly amplify initial temporal localization errors during training (Chen et al., 4 Sep 2025).

1. Problem setting and architectural role

Within TEn-CATS, CATS is not a standalone predictor but a temporal propagation module placed after Bi-Directional Text Fusion (BiT) and HAN-based temporal aggregation (Chen et al., 4 Sep 2025). The broader problem setting is weakly supervised AVVP: no segment-level ground-truth timestamps are available during training, labels are multi-hot, and segment-level pseudo labels may be noisy. On LLP, segment-level pseudo labels are provided by VALOR as 25-dimensional binary vectors per segment; these are converted into text prompts and encoded by frozen CLAP and CLIP to produce segment-level text embeddings (Chen et al., 4 Sep 2025).

BiT precedes CATS and performs semantic injection and dynamic calibration on audio and visual features. For each active category in a pseudo label, prompts such as “This is the sound of [event]” for audio and “This is a photo of [event]” for visual are encoded into text embeddings. Bidirectional cross-attention then updates both modality features and text features:

$$F'_m = \mathrm{LN}\big(F_m + \mathrm{Dropout}(\mathrm{MHA}(F_m, F_e, F_e))\big),$$

$$F'_e = \mathrm{LN}\big(F_e + \mathrm{Dropout}(\mathrm{MHA}(F_e, F_m, F_m))\big).$$

A global modality semantic vector is formed by self-attentive pooling,

$$\alpha_t = \mathrm{softmax}_t(W_{\mathrm{att}}^\top f'_m(t)), \qquad g_m = \sum_{t=1}^{T}\alpha_t f'_m(t),$$

and this global signal is fused with text embeddings through an MLP (Chen et al., 4 Sep 2025). The result is a set of cleaner, text-calibrated segment features that act as more reliable semantic anchors for subsequent graph propagation.

This architectural placement is central to the intended behavior of CATS. Rather than propagating raw pseudo-label information through time, the graph is built from features that have already undergone semantic purification and calibration. This suggests that CATS is best understood as a controlled propagation mechanism, not merely a generic temporal GNN layer.

2. Category-aware graph construction

CATS constructs a separate temporal graph per modality, with shared parameters across the audio and visual branches (Chen et al., 4 Sep 2025). Each time segment is a node. The defining property of the graph is that edge construction is driven by the current segment-level predicted category distribution

$P \in \mathbb{R}^{B \times T \times C},$

where $B$ is batch size, $T$ is the number of segments, and $C$ is the number of event categories.

The first step is hop preference estimation. Given a candidate hop set $\mathcal{K} = \{1,2,\dots,K\}$, segment-level category probabilities are mapped to hop preferences:

$\{x_t^{(a)}, x_t^{(v)}\}_{t=1}^T$0

Here, $\{x_t^{(a)}, x_t^{(v)}\}_{t=1}^T$1 determines how likely segment $\{x_t^{(a)}, x_t^{(v)}\}_{t=1}^T$2 is to connect to neighbors at different hop sizes. Hop selection is then made differentiable with Gumbel-Softmax:

$\{x_t^{(a)}, x_t^{(v)}\}_{t=1}^T$3

where $\{x_t^{(a)}, x_t^{(v)}\}_{t=1}^T$4 is Gumbel noise and $\{x_t^{(a)}, x_t^{(v)}\}_{t=1}^T$5 is the temperature. The module selects Top-$\{x_t^{(a)}, x_t^{(v)}\}_{t=1}^T$6 hops

$\{x_t^{(a)}, x_t^{(v)}\}_{t=1}^T$7

with highest $\{x_t^{(a)}, x_t^{(v)}\}_{t=1}^T$8, producing an adaptive multi-scale neighborhood for each segment (Chen et al., 4 Sep 2025).

The second step is category-conditioned temporal decay. CATS learns a per-segment decay factor from the predicted category distribution:

$\{x_t^{(a)}, x_t^{(v)}\}_{t=1}^T$9

This allows different categories to express different temporal persistence. For each selected hop $$F'_m = \mathrm{LN}\big(F_m + \mathrm{Dropout}(\mathrm{MHA}(F_m, F_e, F_e))\big),$$0 with $$F'_m = \mathrm{LN}\big(F_m + \mathrm{Dropout}(\mathrm{MHA}(F_m, F_e, F_e))\big),$$1, the directed edge $$F'_m = \mathrm{LN}\big(F_m + \mathrm{Dropout}(\mathrm{MHA}(F_m, F_e, F_e))\big),$$2 receives weight

$$F'_m = \mathrm{LN}\big(F_m + \mathrm{Dropout}(\mathrm{MHA}(F_m, F_e, F_e))\big),$$3

A self-loop $$F'_m = \mathrm{LN}\big(F_m + \mathrm{Dropout}(\mathrm{MHA}(F_m, F_e, F_e))\big),$$4 with weight $$F'_m = \mathrm{LN}\big(F_m + \mathrm{Dropout}(\mathrm{MHA}(F_m, F_e, F_e))\big),$$5 is always included. In sparse adjacency form, $$F'_m = \mathrm{LN}\big(F_m + \mathrm{Dropout}(\mathrm{MHA}(F_m, F_e, F_e))\big),$$6 and $$F'_m = \mathrm{LN}\big(F_m + \mathrm{Dropout}(\mathrm{MHA}(F_m, F_e, F_e))\big),$$7 (Chen et al., 4 Sep 2025).

The multi-scale aspect follows directly from allowing multiple hop sizes per node, while the category-aware aspect arises because both hop preference and decay are functions of $$F'_m = \mathrm{LN}\big(F_m + \mathrm{Dropout}(\mathrm{MHA}(F_m, F_e, F_e))\big),$$8. In implementation, edge weights are passed as edge attributes to TransformerConv, so explicit symmetric normalization,

$$F'_m = \mathrm{LN}\big(F_m + \mathrm{Dropout}(\mathrm{MHA}(F_m, F_e, F_e))\big),$$9

is optional rather than required (Chen et al., 4 Sep 2025).

3. Propagation, fusion, and prediction

After graph construction, CATS performs two residual graph attention layers using TransformerConv with four heads (Chen et al., 4 Sep 2025):

$$F'_e = \mathrm{LN}\big(F_e + \mathrm{Dropout}(\mathrm{MHA}(F_e, F_m, F_m))\big).$$0

$$F'_e = \mathrm{LN}\big(F_e + \mathrm{Dropout}(\mathrm{MHA}(F_e, F_m, F_m))\big).$$1

where edge weights are used as one-dimensional edge attributes within attention. The schematic attention formulation is

$$F'_e = \mathrm{LN}\big(F_e + \mathrm{Dropout}(\mathrm{MHA}(F_e, F_m, F_m))\big).$$2

$$F'_e = \mathrm{LN}\big(F_e + \mathrm{Dropout}(\mathrm{MHA}(F_e, F_m, F_m))\big).$$3

$$F'_e = \mathrm{LN}\big(F_e + \mathrm{Dropout}(\mathrm{MHA}(F_e, F_m, F_m))\big).$$4

which biases message passing toward decay-weighted, category-consistent neighbors (Chen et al., 4 Sep 2025).

CATS then aggregates global context by mean pooling across nodes:

$$F'_e = \mathrm{LN}\big(F_e + \mathrm{Dropout}(\mathrm{MHA}(F_e, F_m, F_m))\big).$$5

This is followed by gated local-global fusion:

$$F'_e = \mathrm{LN}\big(F_e + \mathrm{Dropout}(\mathrm{MHA}(F_e, F_m, F_m))\big).$$6

with $$F'_e = \mathrm{LN}\big(F_e + \mathrm{Dropout}(\mathrm{MHA}(F_e, F_m, F_m))\big).$$7 (Chen et al., 4 Sep 2025). In TEn-CATS, the CATS output is then fused with the HAN output:

$$F'_e = \mathrm{LN}\big(F_e + \mathrm{Dropout}(\mathrm{MHA}(F_e, F_m, F_m))\big).$$8

$$F'_e = \mathrm{LN}\big(F_e + \mathrm{Dropout}(\mathrm{MHA}(F_e, F_m, F_m))\big).$$9

Predictions are computed at segment level:

$$\alpha_t = \mathrm{softmax}_t(W_{\mathrm{att}}^\top f'_m(t)), \qquad g_m = \sum_{t=1}^{T}\alpha_t f'_m(t),$$0

and aggregated to video-level predictions via MMIL pooling,

$$\alpha_t = \mathrm{softmax}_t(W_{\mathrm{att}}^\top f'_m(t)), \qquad g_m = \sum_{t=1}^{T}\alpha_t f'_m(t),$$1

followed by a weakly supervised binary cross-entropy MIL loss:

$$\alpha_t = \mathrm{softmax}_t(W_{\mathrm{att}}^\top f'_m(t)), \qquad g_m = \sum_{t=1}^{T}\alpha_t f'_m(t),$$2

The paper does not introduce additional loss terms specific to CATS beyond standard MIL; explicit extra regularizers are described as unnecessary because category-aware edge selection and exponential decay inherently regularize propagation (Chen et al., 4 Sep 2025).

4. Noise control and temporal semantics

The central rationale for CATS is that temporal propagation should be semantically selective rather than indiscriminate (Chen et al., 4 Sep 2025). Earlier AVVP methods are described as following two broad directions: enhanced temporal modeling through attention, or richer pseudo-label generation. In the formulation motivating TEn-CATS, the first direction tends to treat noisy pseudo labels as reliable supervision, while the second can spread pseudo-label errors across all frames through unconstrained attention (Chen et al., 4 Sep 2025).

CATS addresses this by conditioning temporal neighborhoods on learned category probabilities instead of directly diffusing raw pseudo-label information. Three mechanisms are emphasized. First, temporal neighborhoods are chosen according to the predicted category distribution through $$\alpha_t = \mathrm{softmax}_t(W_{\mathrm{att}}^\top f'_m(t)), \qquad g_m = \sum_{t=1}^{T}\alpha_t f'_m(t),$$3 and Gumbel-Softmax hop selection. Second, long-range propagation is modulated by category-dependent decay $$\alpha_t = \mathrm{softmax}_t(W_{\mathrm{att}}^\top f'_m(t)), \qquad g_m = \sum_{t=1}^{T}\alpha_t f'_m(t),$$4, which penalizes distant hops more strongly when semantics are transient. Third, Top-$$\alpha_t = \mathrm{softmax}_t(W_{\mathrm{att}}^\top f'_m(t)), \qquad g_m = \sum_{t=1}^{T}\alpha_t f'_m(t),$$5 selection keeps the graph sparse and constrains information flow (Chen et al., 4 Sep 2025).

The interaction with BiT is equally important. BiT is responsible for aligning modality features and pseudo-label-derived text embeddings, then calibrating them with feature-derived global semantics $$\alpha_t = \mathrm{softmax}_t(W_{\mathrm{att}}^\top f'_m(t)), \qquad g_m = \sum_{t=1}^{T}\alpha_t f'_m(t),$$6. CATS operates on the resulting learned probabilities $$\alpha_t = \mathrm{softmax}_t(W_{\mathrm{att}}^\top f'_m(t)), \qquad g_m = \sum_{t=1}^{T}\alpha_t f'_m(t),$$7 after BiT and HAN rather than on raw pseudo labels (Chen et al., 4 Sep 2025). The intended effect is that unreliable segments receive fewer or weaker long-range connections, confining propagation to semantically consistent neighborhoods.

The same design also exposes a limitation. Category guidance depends on $$\alpha_t = \mathrm{softmax}_t(W_{\mathrm{att}}^\top f'_m(t)), \qquad g_m = \sum_{t=1}^{T}\alpha_t f'_m(t),$$8; if $$\alpha_t = \mathrm{softmax}_t(W_{\mathrm{att}}^\top f'_m(t)), \qquad g_m = \sum_{t=1}^{T}\alpha_t f'_m(t),$$9 is wrong early in training, hop selection can be suboptimal. The self-loop and decay mitigate but do not eliminate this risk (Chen et al., 4 Sep 2025). A second limitation noted in the paper is that event-level AV recall can be lower when fusion is conservative, and rare or asynchronous events may require tailored pooling (Chen et al., 4 Sep 2025).

5. Datasets, implementation, and empirical behavior

TEn-CATS is evaluated on LLP and UnAV-100 (Chen et al., 4 Sep 2025). LLP contains 25 classes and 11,849 videos, with 10,000 weakly labeled training videos, 649 fully labeled validation videos, and 1,200 test videos; each video is divided into 10 one-second segments. UnAV-100 contains 100 classes, 10,790 videos, and more than 30,000 events, with weakly supervised training from video-level labels (Chen et al., 4 Sep 2025).

On LLP, the feature extractors are frozen CLAP for audio and CLIP plus 3D ResNet for visual features, with dimensions 768, 768, and 512 respectively. On UnAV-100, visual features are I3D two-stream (RGB + RAFT) with 2048 dimensions, and audio features are VGGish with 128 dimensions. BiT is not used on UnAV-100 because CLAP and CLIP text features are unavailable there; CATS is evaluated standalone (Chen et al., 4 Sep 2025). CATS uses two TransformerConv layers with four heads, residual connections, edge weights as one-dimensional edge attributes, and mean pooling plus gated local-global fusion. On LLP, the graph hyperparameters are candidate hops $P \in \mathbb{R}^{B \times T \times C},$0, Top-$P \in \mathbb{R}^{B \times T \times C},$1, and Gumbel-Softmax temperature $P \in \mathbb{R}^{B \times T \times C},$2 (Chen et al., 4 Sep 2025).

The reported performance highlights are as follows:

Benchmark	Setting	Reported result
LLP	Segment-level, TEn-CATS	A=73.7, V=74.1, AV=63.2, Type@AV=70.3, Event@AV=73.9, Avg=66.5
LLP	Event-level, TEn-CATS	A=61.1, V=70.3, AV=54.3, Type@AV=61.9, Event@AV=61.9
UnAV-100	CATS only, AV branch	Segment-level mAP 41.9; event-level mAP 47.5

On LLP segment-level evaluation, these numbers are described as state-of-the-art on several key indicators, with gains of $P \in \mathbb{R}^{B \times T \times C},$3 in A and $P \in \mathbb{R}^{B \times T \times C},$4 in V over the prior best, as well as $P \in \mathbb{R}^{B \times T \times C},$5 in Type@AV and $P \in \mathbb{R}^{B \times T \times C},$6 in Event@AV (Chen et al., 4 Sep 2025). On LLP event-level evaluation, the model is described as competitive in Type@AV, while AV event-level performance is slightly conservative due to precision-oriented fusion and pooling (Chen et al., 4 Sep 2025). On UnAV-100, CATS alone reaches segment-level mAP 41.9, which is $P \in \mathbb{R}^{B \times T \times C},$7 over CoLeaF, and event-level mAP 47.5, comparable to CoLeaF’s 47.8, indicating generalizability under large-scale weak supervision even without BiT (Chen et al., 4 Sep 2025).

Ablation analysis shows that CATS-only improves V, AV, and Type@AV modestly relative to the retrained CoLeaF baseline, BiT-only achieves the strongest audio and Event@AV results, and the full BiT+CATS model provides the best balance between unimodal strength and multimodal temporal consistency (Chen et al., 4 Sep 2025). Hop-size sensitivity experiments indicate that moderate Top-$P \in \mathbb{R}^{B \times T \times C},$8, exemplified by $P \in \mathbb{R}^{B \times T \times C},$9, gives the best trade-off, whereas too small a value limits temporal modeling and too large a value risks over-smoothing or misalignment; the visual branch benefits particularly from $B$0 (Chen et al., 4 Sep 2025). The graph construction cost is $B$1 edges per video, and TransformerConv cost is $B$2 with $B$3 (Chen et al., 4 Sep 2025).

6. Terminological scope and related uses of “CATS”

The name “CATS” is not uniform across recent arXiv literature. In the AVVP setting, CATS explicitly denotes the “Category-Aware Temporal Graph” module in TEn-CATS (Chen et al., 4 Sep 2025). In contrast, the video HOI paper “From Category to Scenery: An End-to-End Framework for Multi-Person Human-Object Interaction Recognition in Videos” uses CATS to mean “Category-to-Scenery,” a different framework that first builds separate human and object geometric graphs, fuses them with category-matched visual features, constructs a per-frame scenery interactive graph with GAT reasoning, and only afterward models temporal dynamics with Bi-GRU and Gumbel-Softmax segmentation (Qiao et al., 2024). There, the acronym refers to staged category-level to scenery-level reasoning rather than to a category-aware temporal graph in the AVVP sense.

A second neighboring line of work appears in temporal knowledge graphs. “Temporal Knowledge Graph Hyperedge Forecasting: Exploring Entity-to-Category Link Prediction” does not introduce the term CATS, but it develops a category-aware extension of TLogic, called C-TLogic, in which temporal facts are represented as sextuples

$B$4

rules are mined and applied under category constraints, and entity-level predictions are aggregated into category forecasts through Noisy-OR or Max+ (Markai et al., 28 Oct 2025). The paper itself states that this extension naturally instantiates what one could call a category-aware temporal graph, but its terminology is “types,” “categories,” and “hyperedge forecasting,” not CATS (Markai et al., 28 Oct 2025).

This suggests that “Category-Aware Temporal Graph” is presently best treated as a model-specific designation tied to TEn-CATS rather than as a field-wide standardized acronym. Across these uses, the common thread is category-conditioned structure over temporal data, but the operational meaning differs substantially: weakly supervised semantic propagation in AVVP (Chen et al., 4 Sep 2025), category-to-scenery HOI reasoning in videos (Qiao et al., 2024), and typed temporal rule reasoning for entity-to-category forecasting in temporal knowledge graphs (Markai et al., 28 Oct 2025).