---
title: Memory-guided Consistency-Aware Contrastive Learning
url: https://www.emergentmind.com/topics/memory-guided-consistency-aware-contrastive-learning-mccl
type: topic
---

# Memory-guided Consistency-Aware Contrastive Learning

Memory-guided Consistency-aware Contrastive Learning (MCCL) is a framework for Video-based Generalized Category Discovery (Video-GCD) that integrates spatiotemporal consistency estimation with prototype-based memory distillation in order to classify unlabeled videos from known categories and discover videos from unknown categories [2509.06306]. In its published formulation, MCCL consists of two coupled components—Consistency-Aware Contrastive Learning (CACL) and Memory-Guided Representation Enhancement (MGRE)—and operates on multi-perspective video features extracted from spatial, temporal, and spatiotemporal views [2509.06306]. The acronym is not unique across the literature: it also denotes “Multi-Constraint Consistency Learning” in semi-supervised semantic segmentation [2503.17914] and “Memory-aided Contrastive Consensus Learning” in co-salient object detection [2302.14485]. In the Video-GCD context, however, MCCL refers specifically to the method introduced for open-world video category discovery [2509.06306].

## 1. Problem setting and formal scope

MCCL is defined for a semi-supervised open-world setting in which the labeled set is
$$
D_L = \{(V_i^L, y_i^L)\}_{i=1}^{N_L}, \quad y_i^L \in C_L,
$$
and the unlabeled set is
$$
D_U = \{(V_i^U, y_i^U)\}_{i=1}^{N_U}, \quad y_i^U \in C_N = C_L \cup C_U,
$$
where $C_U$ contains novel categories disjoint from $C_L$ [2509.06306]. The total number of categories $|C_N|$ is assumed known during training, although a practical estimation strategy for unknown $K$ is discussed in the source work [2509.06306].

The central objective is to learn discriminative spatiotemporal representations that support two tasks simultaneously: recognition of unlabeled samples from already-known categories and discovery or clustering of unlabeled samples from novel categories [2509.06306]. This differs from image-based Generalized Category Discovery methods that operate only on static visual content, and from standard semi-supervised action recognition settings in which all classes are known during training [2509.06306].

A video is represented as a clip
$$
V \in \mathbb{R}^{H \times W \times 3 \times T},
$$
with $T$ frames. A backbone encoder $E$ produces three final-block feature types,
$$
f_i^S,\; f_i^T,\; f_i^{ST} = E(V_i),
$$
corresponding to spatial, temporal, and spatiotemporal information [2509.06306]. A classifier head $G$ then maps the fused representation to logits:
$$
z_i = G(f_i^{STF}),
$$
with $f_i^{STF}$ denoting the residual-attentive spatiotemporal fusion used by MCCL [2509.06306].

## 2. Representation pipeline and multi-perspective fusion

The architectural core of MCCL is a ViT/TimeSformer-like encoder of depth 8, initialized from ImageNet-1k, with $T=8$ frames sampled per video and spatial resizing to $224 \times 224$ [2509.06306]. The classifier $G$ is a DINO-style 3-layer MLP with input dimension 768 and output dimension $|C_N|$ [2509.06306].

Before contrastive learning is applied, MCCL constructs a consolidated feature by Multi-Perspective Residual Attention (MPRA). The residual term is defined as
$$
f_i^{res} = (f_i^S + f_i^T) - 2 \cdot f_i^{ST},
$$
which encodes the competition between spatial and temporal channels relative to the spatiotemporal representation [2509.06306]. Channel-wise attention is then computed as
$$
w_i^{ST} = \mathrm{Softmax}(\phi(f_i^{ST})),
$$
where $\phi(\cdot)$ denotes a normalization over channels, and a self-gating operation is defined by
$$
\mathrm{Gate}(x) = \sigma(Wx + b),
$$
with $W$ a fully connected layer and $\sigma$ the sigmoid nonlinearity [2509.06306].

The final fused representation is
$$
f_i^{STF} = f_i^{ST} + \mathrm{Gate}(f_i^{ST}) \cdot \big(f_i^{res} \odot w_i^{ST}\big),
$$
where $\odot$ denotes element-wise multiplication along channels [2509.06306]. The function of this stage is explicit in the source formulation: it models competition between spatial and temporal tokens and suppresses redundant channels before downstream clustering and contrastive learning [2509.06306].

This design makes the representation pipeline intrinsically multi-perspective. A plausible implication is that the later consistency mechanism is not based on a single embedding geometry but on coordinated agreement across spatial, temporal, and fused spatiotemporal views. The source work states this more concretely by using these distinct views in its subsequent voting scheme [2509.06306].

## 3. Consistency-Aware Contrastive Learning

CACL estimates pairwise consistency through a voting mechanism that combines multiple feature perspectives and multiple clustering granularities [2509.06306]. The method constructs horizontal clusterings over $\{f_i^S, f_i^T, f_i^{STF}\}$ using a fixed cluster count $n = |C_N|$, and vertical clusterings over $f_i^{STF}$ using progressively coarser granularity, halving $n$ at each level [2509.06306]. The level-wise assignment is written as
$$
\mathbf{c}_i^k =
\begin{cases}
\mathcal{C}^{hor}(f_i^{*}, |C_N|), & * \in \{S,T,STF\},\; k \in \{0,1,2\},\\
\mathcal{C}^{ver}\!\left(f_i^{STF}, \frac{|C_N|}{2^{k-2}}\right), & k \ge 3.
\end{cases}
$$

For a pair of instances $(i,j)$, the voting statistic is
$$
w_{ij} = \sum_{v=1}^{K} \mathbbm{1}(\mathbf{c}_i^v = \mathbf{c}_j^v),
$$
and the mixed-supervision consistency score is
$$
c_{ij} = (1-\eta)\, y_{ij} + \eta \cdot \frac{w_{ij}}{\sum_{k \ne i} w_{ik}},
$$
where $y_{ij}=1$ if $i$ and $j$ are labeled samples from the same known class and $0$ otherwise [2509.06306]. The paper emphasizes that no hard threshold is needed: $c_{ij}$ is continuous and directly used as a pair weight [2509.06306].

The contrastive objective is a weighted InfoNCE form:
$$
\mathcal{L}_{HCL} =
- \sum_{j' \in \mathcal{B}^I,\; j' \ne i'} c_{i'j'} \cdot
\log
\left(
\frac{
\exp\left(\frac{f_{i'}^\top f_{j'}}{\tau_H}\right)
}{
\sum\limits_{k \in \mathcal{B}^I,\; k \ne i'}
\exp\left(\frac{f_{i'}^\top f_k}{\tau_{H_I}}\right)
}
\right),
$$
with $\tau_H = 1.0$ and $\tau_{H_I} = 1.0$ in the reported setup [2509.06306]. In this construction, the voting-derived $c_{ij}$ scales each pair’s influence, emphasizing pairs with higher cross-view and cross-granularity agreement and de-emphasizing uncertain relations [2509.06306].

CACL is not the only unlabeled objective. Following SimGCD, MCCL also uses a labeled classification loss $\mathcal{L}_{\mathrm{cls}}^s$ and an unlabeled classification loss $\mathcal{L}_{\mathrm{cls}}^u$ to encourage clustering-friendly representations [2509.06306]. The source work explicitly states that it does not add entropy minimization or confidence thresholding beyond these components; unlabeled calibration is instead driven primarily by multi-view voting and memory distillation [2509.06306].

## 4. Memory-Guided Representation Enhancement

MGRE supplies the “memory-guided” part of MCCL through a dual-level category memory buffer defined over a representative labeled subset
$$
D_M = \{(V_i, y_i)\}_{i=1}^{N_M}, \quad C_M \subseteq C_L
$$
[2509.06306]. The buffer stores one feature prototype and one logit prototype per known class in $C_M$ [2509.06306].

Feature prototypes are computed by mean aggregation:
$$
p_{c_j} = \frac{1}{|\mathcal{I}_{c_j}|} \sum_{i \in \mathcal{I}_{c_j}} f_i^{ST},
$$
where $\mathcal{I}_{c_j} = \{i \mid y_i = c_j\}$ [2509.06306]. Logit prototypes are then obtained by passing these class prototypes through the classifier:
$$
\mu_{c_j} = G(p_{c_j}).
$$
The published formulation specifies that prototypes are computed by averaging, with no EMA, and may be computed periodically or on-the-fly from $D_M$ [2509.06306].

MGRE uses two distillation terms. The feature-prototype contrast is
$$
\mathcal{L}_C =
-\log
\frac{
\exp\left(\frac{f^\top p_{c_j}}{\tau_{C_L}}\right)
}{
\sum_{k=1,\; k \ne j}^{|C_M|}
\exp\left(\frac{f^\top p_{c_k}}{\tau_{C_L}}\right)
},
$$
with $\tau_{C_L}=0.05$ [2509.06306]. This term is described as encouraging intra-class compactness at the feature level and providing global context for class separation [2509.06306].

The logit-level distillation term first sharpens the teacher distribution:
$$
\tilde{\mu}_{c_j} = \mathrm{Softmax}\!\left(\frac{\mu_{c_j}}{\tau_{T_L}}\right), \quad \tau_{T_L}=0.1,
$$
and then applies KL divergence between instance logits $z_i = G(f_i^{STF})$ and the prototype-derived teacher:
$$
\mathcal{L}_S =
\frac{1}{B}
\sum_{i=1}^{B}
\tau_{S_L} \cdot
\mathrm{KL}\!\left(
\mathrm{LogSoftmax}(z_i)\,\|\,\tilde{\mu}_{c_j}
\right),
\quad y_i = c_j
$$
[2509.06306]. The source text attributes to this loss a strengthening of inter-class boundaries and a mitigation of “logit confusion” [2509.06306].

The interaction between MGRE and CACL is one of the defining claims of the method. MGRE refines the representation through prototype-guided distillation; improved representations then stabilize multi-view clustering assignments; these more reliable assignments yield more accurate $c_{ij}$ weights for contrastive learning; and the contrastive updates further sharpen the representation [2509.06306]. The paper characterizes this as a mutually reinforcing feedback loop rather than as two isolated regularizers [2509.06306].

## 5. Objective, optimization, and evaluation protocol

The full MCCL objective is
$$
\mathcal{L}
=
\lambda_{\mathrm{Sup}}
\cdot
\big(
\mathcal{L}_{\mathrm{cls}}^s + \mathcal{L}_C + \lambda_S \cdot \mathcal{L}_S
\big)
+
\lambda_{\mathrm{Unsup}}
\cdot
\big(
\mathcal{L}_{\mathrm{cls}}^u + \mathcal{L}_{HCL}
\big)
$$
[2509.06306]. On VB100, the best reported All ACC is obtained at $\lambda_S = 0.5$ and $\lambda_{\mathrm{Sup}} = 0.45$ [2509.06306].

Training follows a two-stage procedure. Stage 1 is a supervised warm-up on labeled data using cross-entropy with SGD, learning rate $0.005$, momentum $0.9$, weight decay $10^{-4}$, input size $224 \times 224$, and $T=8$ frames [2509.06306]. Stage 2 performs joint Video-GCD training on labeled and unlabeled mini-batches: multi-perspective features are extracted, MPRA produces $f_i^{STF}$, horizontal and vertical clusterings are built, $w_{ij}$ and $c_{ij}$ are computed, $\mathcal{L}_{HCL}$ is evaluated, prototypes are read for $\mathcal{L}_C$ and $\mathcal{L}_S$, SimGCD classification losses are added, and the total loss is optimized [2509.06306].

At inference time, features are extracted for all videos, and K-Means clustering followed by Hungarian matching is used to report All ACC, Old ACC, and New ACC [2509.06306]. The benchmark introduced alongside MCCL includes action-recognition datasets UCF101, SSv2, and Kinetics-400, and fine-grained bird datasets VB100 and IBC127 [2509.06306]. The split protocol is “even-odd” known/unknown for all except VB100, where the first 50% are known and the last 50% unknown; for computational efficiency, 15% of SSv2 and Kinetics-400 are sampled, while UCF101, VB100, and IBC127 use full datasets [2509.06306].

The hardware reported for training is 2×RTX 4090 GPUs, and 4×RTX 4090 on K400 [2509.06306].

## 6. Empirical performance, ablations, and conceptual distinctions

Across the reported benchmarks, MCCL improves over image-based GCD baselines adapted to the video setting [2509.06306]. On UCF101, MCCL obtains $68.27/89.54/57.80$ in All/Old/New ACC, compared with SimGCD at $63.93/82.65/54.70$ and SelfEx at $63.43/77.07/56.72$ [2509.06306]. On SSv2, the reported values are $13.58/18.69/11.57$ for MCCL, versus $11.36/14.27/9.93$ for SimGCD and $12.54/14.96/11.35$ for SelfEx [2509.06306]. On Kinetics-400, MCCL reaches $22.95/29.89/19.51$, against $21.57/25.15/19.79$ for SimGCD and $21.23/23.14/20.28$ for SelfEx [2509.06306]. On VB100, MCCL reports $46.17/70.51/34.81$, compared with $33.75/37.66/32.01$ for SimGCD and $40.43/47.83/33.68$ for SelfEx; on IBC127, it reports $38.06/48.83/32.72$, compared with $31.88/31.31/32.16$ and $33.59/33.76/33.50$, respectively [2509.06306].

Ablations separate the effects of MGRE and CACL. On VB100, the baseline gives $33.75/37.66/32.01$, adding MGRE gives $44.02/67.57/33.02$, adding CACL gives $39.50/50.62/34.31$, and the full model gives $46.17/70.51/34.81$ [2509.06306]. On UCF101, the same progression is $64.28/88.66/52.27$ for +MGRE, $65.42/82.37/57.06$ for +CACL, and $68.27/89.54/57.80$ for the full model [2509.06306]. The source analysis further states that $\mathcal{L}_S$ notably boosts Old ACC, whereas $\mathcal{L}_C$ complements it by improving New ACC; MPRA and $\mathcal{L}_{HCL}$ are also described as complementary [2509.06306].

The method is positioned against two distinct comparison axes. Relative to image-based GCD approaches such as SimGCD, InfoSieve, SPTNet, and SelfEx, MCCL explicitly exploits spatiotemporal cues through MPRA and horizontal/vertical multi-view voting, and introduces dual-level memory through feature and logit prototypes [2509.06306]. Relative to semi-supervised action recognition, MCCL is defined for a setting in which unlabeled videos include unknown categories, so it combines parametric classification with non-parametric voting and prototype distillation rather than assuming closed-set supervision [2509.06306].

Several limitations are identified in the source material. Multiple clusterings across views and granularity levels increase computational cost; storing and updating prototypes adds memory overhead, though the design keeps this moderate by averaging over a small labeled subset; performance depends on $\lambda_{\mathrm{Sup}}$, $\lambda_S$, $\eta$, and temperature parameters; and errors in estimating the category count can degrade clustering, with particularly large gaps noted on SSv2 and IBC127 [2509.06306]. Potential improvements proposed in the paper include EMA-based prototype updates, memory-informed voting that incorporates prototype similarity into $c_{ij}$, more efficient clustering approximations, and adaptive granularity schedules [2509.06306].

A persistent source of confusion is terminological rather than algorithmic. In semi-supervised semantic segmentation, “MCCL” denotes Multi-Constraint Consistency Learning and explicitly uses no memory bank, no InfoNCE, and no teacher model [2503.17914]. In co-salient object detection, “MCCL” denotes Memory-aided Contrastive Consensus Learning, where a momentum memory per class or group supports a triplet-style contrastive objective and training-only adversarial integrity learning [2302.14485]. A nearby but differently named formulation is Memory Consistency guided Divide-and-conquer Learning (MCDL), whose contrastive component can be interpreted as memory-guided and consistency-aware because it filters supervised contrastive positives by credibility derived from dual prediction-history memory banks, but it is not introduced under the name MCCL [2401.13325]. These distinctions matter because the exact expansion determines whether “memory” means class prototypes, momentum group memories, or historical prediction queues, and whether “contrastive learning” refers to weighted InfoNCE, triplet loss, or supervised contrastive learning.

Source: https://www.emergentmind.com/topics/memory-guided-consistency-aware-contrastive-learning-mccl