Memory-guided Consistency-Aware Contrastive Learning

• The paper demonstrates the integration of contrastive learning with memory-guided prototype distillation to classify known video categories and discover novel ones.
• MCCL employs a multi-perspective residual attention mechanism that fuses spatial, temporal, and spatiotemporal features to enhance clustering and representation accuracy.
• Empirical results show superior performance over image-based baselines, with significant gains in All, Old, and New ACC on benchmarks like UCF101 and VB100.

Memory-guided Consistency-aware Contrastive Learning (MCCL) is a framework for Video-based Generalized Category Discovery (Video-GCD) that integrates spatiotemporal consistency estimation with prototype-based memory distillation in order to classify unlabeled videos from known categories and discover videos from unknown categories (Jing et al., 8 Sep 2025). In its published formulation, MCCL consists of two coupled components—Consistency-Aware Contrastive Learning (CACL) and Memory-Guided Representation Enhancement (MGRE)—and operates on multi-perspective video features extracted from spatial, temporal, and spatiotemporal views (Jing et al., 8 Sep 2025). The acronym is not unique across the literature: it also denotes “Multi-Constraint Consistency Learning” in semi-supervised semantic segmentation (Yin et al., 23 Mar 2025) and “Memory-aided Contrastive Consensus Learning” in co-salient object detection (Zheng et al., 2023). In the Video-GCD context, however, MCCL refers specifically to the method introduced for open-world video category discovery (Jing et al., 8 Sep 2025).

1. Problem setting and formal scope

MCCL is defined for a semi-supervised open-world setting in which the labeled set is

$$D_L = \{(V_i^L, y_i^L)\}_{i=1}^{N_L}, \quad y_i^L \in C_L,$$

and the unlabeled set is

$$D_U = \{(V_i^U, y_i^U)\}_{i=1}^{N_U}, \quad y_i^U \in C_N = C_L \cup C_U,$$

where $C_U$ contains novel categories disjoint from $C_L$ (Jing et al., 8 Sep 2025). The total number of categories $|C_N|$ is assumed known during training, although a practical estimation strategy for unknown $K$ is discussed in the source work (Jing et al., 8 Sep 2025).

The central objective is to learn discriminative spatiotemporal representations that support two tasks simultaneously: recognition of unlabeled samples from already-known categories and discovery or clustering of unlabeled samples from novel categories (Jing et al., 8 Sep 2025). This differs from image-based Generalized Category Discovery methods that operate only on static visual content, and from standard semi-supervised action recognition settings in which all classes are known during training (Jing et al., 8 Sep 2025).

A video is represented as a clip

$V \in \mathbb{R}^{H \times W \times 3 \times T},$

with $T$ frames. A backbone encoder $E$ produces three final-block feature types,

$f_i^S,\; f_i^T,\; f_i^{ST} = E(V_i),$

corresponding to spatial, temporal, and spatiotemporal information (Jing et al., 8 Sep 2025). A classifier head $$D_U = \{(V_i^U, y_i^U)\}_{i=1}^{N_U}, \quad y_i^U \in C_N = C_L \cup C_U,$$0 then maps the fused representation to logits:

$$D_U = \{(V_i^U, y_i^U)\}_{i=1}^{N_U}, \quad y_i^U \in C_N = C_L \cup C_U,$$1

with $$D_U = \{(V_i^U, y_i^U)\}_{i=1}^{N_U}, \quad y_i^U \in C_N = C_L \cup C_U,$$2 denoting the residual-attentive spatiotemporal fusion used by MCCL (Jing et al., 8 Sep 2025).

2. Representation pipeline and multi-perspective fusion

The architectural core of MCCL is a ViT/TimeSformer-like encoder of depth 8, initialized from ImageNet-1k, with $$D_U = \{(V_i^U, y_i^U)\}_{i=1}^{N_U}, \quad y_i^U \in C_N = C_L \cup C_U,$$3 frames sampled per video and spatial resizing to $$D_U = \{(V_i^U, y_i^U)\}_{i=1}^{N_U}, \quad y_i^U \in C_N = C_L \cup C_U,$$4 (Jing et al., 8 Sep 2025). The classifier $$D_U = \{(V_i^U, y_i^U)\}_{i=1}^{N_U}, \quad y_i^U \in C_N = C_L \cup C_U,$$5 is a DINO-style 3-layer MLP with input dimension 768 and output dimension $$D_U = \{(V_i^U, y_i^U)\}_{i=1}^{N_U}, \quad y_i^U \in C_N = C_L \cup C_U,$$6 (Jing et al., 8 Sep 2025).

Before contrastive learning is applied, MCCL constructs a consolidated feature by Multi-Perspective Residual Attention (MPRA). The residual term is defined as

$$D_U = \{(V_i^U, y_i^U)\}_{i=1}^{N_U}, \quad y_i^U \in C_N = C_L \cup C_U,$$7

which encodes the competition between spatial and temporal channels relative to the spatiotemporal representation (Jing et al., 8 Sep 2025). Channel-wise attention is then computed as

$$D_U = \{(V_i^U, y_i^U)\}_{i=1}^{N_U}, \quad y_i^U \in C_N = C_L \cup C_U,$$8

where $$D_U = \{(V_i^U, y_i^U)\}_{i=1}^{N_U}, \quad y_i^U \in C_N = C_L \cup C_U,$$9 denotes a normalization over channels, and a self-gating operation is defined by

$C_U$0

with $C_U$1 a fully connected layer and $C_U$2 the sigmoid nonlinearity (Jing et al., 8 Sep 2025).

The final fused representation is

$C_U$3

where $C_U$4 denotes element-wise multiplication along channels (Jing et al., 8 Sep 2025). The function of this stage is explicit in the source formulation: it models competition between spatial and temporal tokens and suppresses redundant channels before downstream clustering and contrastive learning (Jing et al., 8 Sep 2025).

This design makes the representation pipeline intrinsically multi-perspective. A plausible implication is that the later consistency mechanism is not based on a single embedding geometry but on coordinated agreement across spatial, temporal, and fused spatiotemporal views. The source work states this more concretely by using these distinct views in its subsequent voting scheme (Jing et al., 8 Sep 2025).

3. Consistency-Aware Contrastive Learning

CACL estimates pairwise consistency through a voting mechanism that combines multiple feature perspectives and multiple clustering granularities (Jing et al., 8 Sep 2025). The method constructs horizontal clusterings over $C_U$5 using a fixed cluster count $C_U$6, and vertical clusterings over $C_U$7 using progressively coarser granularity, halving $C_U$8 at each level (Jing et al., 8 Sep 2025). The level-wise assignment is written as

$C_U$9

For a pair of instances $C_L$0, the voting statistic is

$C_L$1

and the mixed-supervision consistency score is

$C_L$2

where $C_L$3 if $C_L$4 and $C_L$5 are labeled samples from the same known class and $C_L$6 otherwise (Jing et al., 8 Sep 2025). The paper emphasizes that no hard threshold is needed: $C_L$7 is continuous and directly used as a pair weight (Jing et al., 8 Sep 2025).

The contrastive objective is a weighted InfoNCE form:

$C_L$8

with $C_L$9 and $|C_N|$0 in the reported setup (Jing et al., 8 Sep 2025). In this construction, the voting-derived $|C_N|$1 scales each pair’s influence, emphasizing pairs with higher cross-view and cross-granularity agreement and de-emphasizing uncertain relations (Jing et al., 8 Sep 2025).

CACL is not the only unlabeled objective. Following SimGCD, MCCL also uses a labeled classification loss $|C_N|$2 and an unlabeled classification loss $|C_N|$3 to encourage clustering-friendly representations (Jing et al., 8 Sep 2025). The source work explicitly states that it does not add entropy minimization or confidence thresholding beyond these components; unlabeled calibration is instead driven primarily by multi-view voting and memory distillation (Jing et al., 8 Sep 2025).

4. Memory-Guided Representation Enhancement

MGRE supplies the “memory-guided” part of MCCL through a dual-level category memory buffer defined over a representative labeled subset

$|C_N|$4

(Jing et al., 8 Sep 2025). The buffer stores one feature prototype and one logit prototype per known class in $|C_N|$5 (Jing et al., 8 Sep 2025).

Feature prototypes are computed by mean aggregation:

$|C_N|$6

where $|C_N|$7 (Jing et al., 8 Sep 2025). Logit prototypes are then obtained by passing these class prototypes through the classifier:

$|C_N|$8

The published formulation specifies that prototypes are computed by averaging, with no EMA, and may be computed periodically or on-the-fly from $|C_N|$9 (Jing et al., 8 Sep 2025).

MGRE uses two distillation terms. The feature-prototype contrast is

$K$0

with $K$1 (Jing et al., 8 Sep 2025). This term is described as encouraging intra-class compactness at the feature level and providing global context for class separation (Jing et al., 8 Sep 2025).

The logit-level distillation term first sharpens the teacher distribution:

$K$2

and then applies KL divergence between instance logits $K$3 and the prototype-derived teacher:

$K$4

(Jing et al., 8 Sep 2025). The source text attributes to this loss a strengthening of inter-class boundaries and a mitigation of “logit confusion” (Jing et al., 8 Sep 2025).

The interaction between MGRE and CACL is one of the defining claims of the method. MGRE refines the representation through prototype-guided distillation; improved representations then stabilize multi-view clustering assignments; these more reliable assignments yield more accurate $K$5 weights for contrastive learning; and the contrastive updates further sharpen the representation (Jing et al., 8 Sep 2025). The paper characterizes this as a mutually reinforcing feedback loop rather than as two isolated regularizers (Jing et al., 8 Sep 2025).

5. Objective, optimization, and evaluation protocol

The full MCCL objective is

$K$6

(Jing et al., 8 Sep 2025). On VB100, the best reported All ACC is obtained at $K$7 and $K$8 (Jing et al., 8 Sep 2025).

Training follows a two-stage procedure. Stage 1 is a supervised warm-up on labeled data using cross-entropy with SGD, learning rate $K$9, momentum $V \in \mathbb{R}^{H \times W \times 3 \times T},$0, weight decay $V \in \mathbb{R}^{H \times W \times 3 \times T},$1, input size $V \in \mathbb{R}^{H \times W \times 3 \times T},$2, and $V \in \mathbb{R}^{H \times W \times 3 \times T},$3 frames (Jing et al., 8 Sep 2025). Stage 2 performs joint Video-GCD training on labeled and unlabeled mini-batches: multi-perspective features are extracted, MPRA produces $V \in \mathbb{R}^{H \times W \times 3 \times T},$4, horizontal and vertical clusterings are built, $V \in \mathbb{R}^{H \times W \times 3 \times T},$5 and $V \in \mathbb{R}^{H \times W \times 3 \times T},$6 are computed, $V \in \mathbb{R}^{H \times W \times 3 \times T},$7 is evaluated, prototypes are read for $V \in \mathbb{R}^{H \times W \times 3 \times T},$8 and $V \in \mathbb{R}^{H \times W \times 3 \times T},$9, SimGCD classification losses are added, and the total loss is optimized (Jing et al., 8 Sep 2025).

At inference time, features are extracted for all videos, and K-Means clustering followed by Hungarian matching is used to report All ACC, Old ACC, and New ACC (Jing et al., 8 Sep 2025). The benchmark introduced alongside MCCL includes action-recognition datasets UCF101, SSv2, and Kinetics-400, and fine-grained bird datasets VB100 and IBC127 (Jing et al., 8 Sep 2025). The split protocol is “even-odd” known/unknown for all except VB100, where the first 50% are known and the last 50% unknown; for computational efficiency, 15% of SSv2 and Kinetics-400 are sampled, while UCF101, VB100, and IBC127 use full datasets (Jing et al., 8 Sep 2025).

The hardware reported for training is 2×RTX 4090 GPUs, and 4×RTX 4090 on K400 (Jing et al., 8 Sep 2025).

6. Empirical performance, ablations, and conceptual distinctions

Across the reported benchmarks, MCCL improves over image-based GCD baselines adapted to the video setting (Jing et al., 8 Sep 2025). On UCF101, MCCL obtains $T$0 in All/Old/New ACC, compared with SimGCD at $T$1 and SelfEx at $T$2 (Jing et al., 8 Sep 2025). On SSv2, the reported values are $T$3 for MCCL, versus $T$4 for SimGCD and $T$5 for SelfEx (Jing et al., 8 Sep 2025). On Kinetics-400, MCCL reaches $T$6, against $T$7 for SimGCD and $T$8 for SelfEx (Jing et al., 8 Sep 2025). On VB100, MCCL reports $T$9, compared with $E$0 for SimGCD and $E$1 for SelfEx; on IBC127, it reports $E$2, compared with $E$3 and $E$4, respectively (Jing et al., 8 Sep 2025).

Ablations separate the effects of MGRE and CACL. On VB100, the baseline gives $E$5, adding MGRE gives $E$6, adding CACL gives $E$7, and the full model gives $E$8 (Jing et al., 8 Sep 2025). On UCF101, the same progression is $E$9 for +MGRE, $f_i^S,\; f_i^T,\; f_i^{ST} = E(V_i),$0 for +CACL, and $f_i^S,\; f_i^T,\; f_i^{ST} = E(V_i),$1 for the full model (Jing et al., 8 Sep 2025). The source analysis further states that $f_i^S,\; f_i^T,\; f_i^{ST} = E(V_i),$2 notably boosts Old ACC, whereas $f_i^S,\; f_i^T,\; f_i^{ST} = E(V_i),$3 complements it by improving New ACC; MPRA and $f_i^S,\; f_i^T,\; f_i^{ST} = E(V_i),$4 are also described as complementary (Jing et al., 8 Sep 2025).

The method is positioned against two distinct comparison axes. Relative to image-based GCD approaches such as SimGCD, InfoSieve, SPTNet, and SelfEx, MCCL explicitly exploits spatiotemporal cues through MPRA and horizontal/vertical multi-view voting, and introduces dual-level memory through feature and logit prototypes (Jing et al., 8 Sep 2025). Relative to semi-supervised action recognition, MCCL is defined for a setting in which unlabeled videos include unknown categories, so it combines parametric classification with non-parametric voting and prototype distillation rather than assuming closed-set supervision (Jing et al., 8 Sep 2025).

Several limitations are identified in the source material. Multiple clusterings across views and granularity levels increase computational cost; storing and updating prototypes adds memory overhead, though the design keeps this moderate by averaging over a small labeled subset; performance depends on $f_i^S,\; f_i^T,\; f_i^{ST} = E(V_i),$5, $f_i^S,\; f_i^T,\; f_i^{ST} = E(V_i),$6, $f_i^S,\; f_i^T,\; f_i^{ST} = E(V_i),$7, and temperature parameters; and errors in estimating the category count can degrade clustering, with particularly large gaps noted on SSv2 and IBC127 (Jing et al., 8 Sep 2025). Potential improvements proposed in the paper include EMA-based prototype updates, memory-informed voting that incorporates prototype similarity into $f_i^S,\; f_i^T,\; f_i^{ST} = E(V_i),$8, more efficient clustering approximations, and adaptive granularity schedules (Jing et al., 8 Sep 2025).

A persistent source of confusion is terminological rather than algorithmic. In semi-supervised semantic segmentation, “MCCL” denotes Multi-Constraint Consistency Learning and explicitly uses no memory bank, no InfoNCE, and no teacher model (Yin et al., 23 Mar 2025). In co-salient object detection, “MCCL” denotes Memory-aided Contrastive Consensus Learning, where a momentum memory per class or group supports a triplet-style contrastive objective and training-only adversarial integrity learning (Zheng et al., 2023). A nearby but differently named formulation is Memory Consistency guided Divide-and-conquer Learning (MCDL), whose contrastive component can be interpreted as memory-guided and consistency-aware because it filters supervised contrastive positives by credibility derived from dual prediction-history memory banks, but it is not introduced under the name MCCL (Tu et al., 2024). These distinctions matter because the exact expansion determines whether “memory” means class prototypes, momentum group memories, or historical prediction queues, and whether “contrastive learning” refers to weighted InfoNCE, triplet loss, or supervised contrastive learning.