Papers
Topics
Authors
Recent
Search
2000 character limit reached

Memory-guided Consistency-Aware Contrastive Learning

Updated 10 July 2026
  • The paper demonstrates the integration of contrastive learning with memory-guided prototype distillation to classify known video categories and discover novel ones.
  • MCCL employs a multi-perspective residual attention mechanism that fuses spatial, temporal, and spatiotemporal features to enhance clustering and representation accuracy.
  • Empirical results show superior performance over image-based baselines, with significant gains in All, Old, and New ACC on benchmarks like UCF101 and VB100.

Memory-guided Consistency-aware Contrastive Learning (MCCL) is a framework for Video-based Generalized Category Discovery (Video-GCD) that integrates spatiotemporal consistency estimation with prototype-based memory distillation in order to classify unlabeled videos from known categories and discover videos from unknown categories (Jing et al., 8 Sep 2025). In its published formulation, MCCL consists of two coupled components—Consistency-Aware Contrastive Learning (CACL) and Memory-Guided Representation Enhancement (MGRE)—and operates on multi-perspective video features extracted from spatial, temporal, and spatiotemporal views (Jing et al., 8 Sep 2025). The acronym is not unique across the literature: it also denotes “Multi-Constraint Consistency Learning” in semi-supervised semantic segmentation (Yin et al., 23 Mar 2025) and “Memory-aided Contrastive Consensus Learning” in co-salient object detection (Zheng et al., 2023). In the Video-GCD context, however, MCCL refers specifically to the method introduced for open-world video category discovery (Jing et al., 8 Sep 2025).

1. Problem setting and formal scope

MCCL is defined for a semi-supervised open-world setting in which the labeled set is

DL={(ViL,yiL)}i=1NL,yiL∈CL,D_L = \{(V_i^L, y_i^L)\}_{i=1}^{N_L}, \quad y_i^L \in C_L,

and the unlabeled set is

DU={(ViU,yiU)}i=1NU,yiU∈CN=CL∪CU,D_U = \{(V_i^U, y_i^U)\}_{i=1}^{N_U}, \quad y_i^U \in C_N = C_L \cup C_U,

where CUC_U contains novel categories disjoint from CLC_L (Jing et al., 8 Sep 2025). The total number of categories ∣CN∣|C_N| is assumed known during training, although a practical estimation strategy for unknown KK is discussed in the source work (Jing et al., 8 Sep 2025).

The central objective is to learn discriminative spatiotemporal representations that support two tasks simultaneously: recognition of unlabeled samples from already-known categories and discovery or clustering of unlabeled samples from novel categories (Jing et al., 8 Sep 2025). This differs from image-based Generalized Category Discovery methods that operate only on static visual content, and from standard semi-supervised action recognition settings in which all classes are known during training (Jing et al., 8 Sep 2025).

A video is represented as a clip

V∈RH×W×3×T,V \in \mathbb{R}^{H \times W \times 3 \times T},

with TT frames. A backbone encoder EE produces three final-block feature types,

fiS,  fiT,  fiST=E(Vi),f_i^S,\; f_i^T,\; f_i^{ST} = E(V_i),

corresponding to spatial, temporal, and spatiotemporal information (Jing et al., 8 Sep 2025). A classifier head GG then maps the fused representation to logits:

zi=G(fiSTF),z_i = G(f_i^{STF}),

with fiSTFf_i^{STF} denoting the residual-attentive spatiotemporal fusion used by MCCL (Jing et al., 8 Sep 2025).

2. Representation pipeline and multi-perspective fusion

The architectural core of MCCL is a ViT/TimeSformer-like encoder of depth 8, initialized from ImageNet-1k, with T=8T=8 frames sampled per video and spatial resizing to 224×224224 \times 224 (Jing et al., 8 Sep 2025). The classifier GG is a DINO-style 3-layer MLP with input dimension 768 and output dimension ∣CN∣|C_N| (Jing et al., 8 Sep 2025).

Before contrastive learning is applied, MCCL constructs a consolidated feature by Multi-Perspective Residual Attention (MPRA). The residual term is defined as

fires=(fiS+fiT)−2⋅fiST,f_i^{res} = (f_i^S + f_i^T) - 2 \cdot f_i^{ST},

which encodes the competition between spatial and temporal channels relative to the spatiotemporal representation (Jing et al., 8 Sep 2025). Channel-wise attention is then computed as

wiST=Softmax(ϕ(fiST)),w_i^{ST} = \mathrm{Softmax}(\phi(f_i^{ST})),

where ϕ(⋅)\phi(\cdot) denotes a normalization over channels, and a self-gating operation is defined by

Gate(x)=σ(Wx+b),\mathrm{Gate}(x) = \sigma(Wx + b),

with WW a fully connected layer and σ\sigma the sigmoid nonlinearity (Jing et al., 8 Sep 2025).

The final fused representation is

fiSTF=fiST+Gate(fiST)⋅(fires⊙wiST),f_i^{STF} = f_i^{ST} + \mathrm{Gate}(f_i^{ST}) \cdot \big(f_i^{res} \odot w_i^{ST}\big),

where ⊙\odot denotes element-wise multiplication along channels (Jing et al., 8 Sep 2025). The function of this stage is explicit in the source formulation: it models competition between spatial and temporal tokens and suppresses redundant channels before downstream clustering and contrastive learning (Jing et al., 8 Sep 2025).

This design makes the representation pipeline intrinsically multi-perspective. A plausible implication is that the later consistency mechanism is not based on a single embedding geometry but on coordinated agreement across spatial, temporal, and fused spatiotemporal views. The source work states this more concretely by using these distinct views in its subsequent voting scheme (Jing et al., 8 Sep 2025).

3. Consistency-Aware Contrastive Learning

CACL estimates pairwise consistency through a voting mechanism that combines multiple feature perspectives and multiple clustering granularities (Jing et al., 8 Sep 2025). The method constructs horizontal clusterings over {fiS,fiT,fiSTF}\{f_i^S, f_i^T, f_i^{STF}\} using a fixed cluster count n=∣CN∣n = |C_N|, and vertical clusterings over fiSTFf_i^{STF} using progressively coarser granularity, halving nn at each level (Jing et al., 8 Sep 2025). The level-wise assignment is written as

cik={Chor(fi∗,∣CN∣),∗∈{S,T,STF},  k∈{0,1,2},Cver ⁣(fiSTF,∣CN∣2k−2),k≥3.\mathbf{c}_i^k = \begin{cases} \mathcal{C}^{hor}(f_i^{*}, |C_N|), & * \in \{S,T,STF\},\; k \in \{0,1,2\},\\ \mathcal{C}^{ver}\!\left(f_i^{STF}, \frac{|C_N|}{2^{k-2}}\right), & k \ge 3. \end{cases}

For a pair of instances (i,j)(i,j), the voting statistic is

$w_{ij} = \sum_{v=1}^{K} \mathbbm{1}(\mathbf{c}_i^v = \mathbf{c}_j^v),$

and the mixed-supervision consistency score is

cij=(1−η) yij+η⋅wij∑k≠iwik,c_{ij} = (1-\eta)\, y_{ij} + \eta \cdot \frac{w_{ij}}{\sum_{k \ne i} w_{ik}},

where yij=1y_{ij}=1 if ii and jj are labeled samples from the same known class and $0$ otherwise (Jing et al., 8 Sep 2025). The paper emphasizes that no hard threshold is needed: cijc_{ij} is continuous and directly used as a pair weight (Jing et al., 8 Sep 2025).

The contrastive objective is a weighted InfoNCE form:

LHCL=−∑j′∈BI,  j′≠i′ci′j′⋅log⁡(exp⁡(fi′⊤fj′τH)∑k∈BI,  k≠i′exp⁡(fi′⊤fkτHI)),\mathcal{L}_{HCL} = - \sum_{j' \in \mathcal{B}^I,\; j' \ne i'} c_{i'j'} \cdot \log \left( \frac{ \exp\left(\frac{f_{i'}^\top f_{j'}}{\tau_H}\right) }{ \sum\limits_{k \in \mathcal{B}^I,\; k \ne i'} \exp\left(\frac{f_{i'}^\top f_k}{\tau_{H_I}}\right) } \right),

with τH=1.0\tau_H = 1.0 and τHI=1.0\tau_{H_I} = 1.0 in the reported setup (Jing et al., 8 Sep 2025). In this construction, the voting-derived cijc_{ij} scales each pair’s influence, emphasizing pairs with higher cross-view and cross-granularity agreement and de-emphasizing uncertain relations (Jing et al., 8 Sep 2025).

CACL is not the only unlabeled objective. Following SimGCD, MCCL also uses a labeled classification loss Lclss\mathcal{L}_{\mathrm{cls}}^s and an unlabeled classification loss Lclsu\mathcal{L}_{\mathrm{cls}}^u to encourage clustering-friendly representations (Jing et al., 8 Sep 2025). The source work explicitly states that it does not add entropy minimization or confidence thresholding beyond these components; unlabeled calibration is instead driven primarily by multi-view voting and memory distillation (Jing et al., 8 Sep 2025).

4. Memory-Guided Representation Enhancement

MGRE supplies the “memory-guided” part of MCCL through a dual-level category memory buffer defined over a representative labeled subset

DM={(Vi,yi)}i=1NM,CM⊆CLD_M = \{(V_i, y_i)\}_{i=1}^{N_M}, \quad C_M \subseteq C_L

(Jing et al., 8 Sep 2025). The buffer stores one feature prototype and one logit prototype per known class in CMC_M (Jing et al., 8 Sep 2025).

Feature prototypes are computed by mean aggregation:

pcj=1∣Icj∣∑i∈IcjfiST,p_{c_j} = \frac{1}{|\mathcal{I}_{c_j}|} \sum_{i \in \mathcal{I}_{c_j}} f_i^{ST},

where Icj={i∣yi=cj}\mathcal{I}_{c_j} = \{i \mid y_i = c_j\} (Jing et al., 8 Sep 2025). Logit prototypes are then obtained by passing these class prototypes through the classifier:

μcj=G(pcj).\mu_{c_j} = G(p_{c_j}).

The published formulation specifies that prototypes are computed by averaging, with no EMA, and may be computed periodically or on-the-fly from DMD_M (Jing et al., 8 Sep 2025).

MGRE uses two distillation terms. The feature-prototype contrast is

LC=−log⁡exp⁡(f⊤pcjτCL)∑k=1,  k≠j∣CM∣exp⁡(f⊤pckτCL),\mathcal{L}_C = -\log \frac{ \exp\left(\frac{f^\top p_{c_j}}{\tau_{C_L}}\right) }{ \sum_{k=1,\; k \ne j}^{|C_M|} \exp\left(\frac{f^\top p_{c_k}}{\tau_{C_L}}\right) },

with τCL=0.05\tau_{C_L}=0.05 (Jing et al., 8 Sep 2025). This term is described as encouraging intra-class compactness at the feature level and providing global context for class separation (Jing et al., 8 Sep 2025).

The logit-level distillation term first sharpens the teacher distribution:

μ~cj=Softmax ⁣(μcjτTL),τTL=0.1,\tilde{\mu}_{c_j} = \mathrm{Softmax}\!\left(\frac{\mu_{c_j}}{\tau_{T_L}}\right), \quad \tau_{T_L}=0.1,

and then applies KL divergence between instance logits zi=G(fiSTF)z_i = G(f_i^{STF}) and the prototype-derived teacher:

LS=1B∑i=1BτSL⋅KL ⁣(LogSoftmax(zi) ∥ μ~cj),yi=cj\mathcal{L}_S = \frac{1}{B} \sum_{i=1}^{B} \tau_{S_L} \cdot \mathrm{KL}\!\left( \mathrm{LogSoftmax}(z_i)\,\|\,\tilde{\mu}_{c_j} \right), \quad y_i = c_j

(Jing et al., 8 Sep 2025). The source text attributes to this loss a strengthening of inter-class boundaries and a mitigation of “logit confusion” (Jing et al., 8 Sep 2025).

The interaction between MGRE and CACL is one of the defining claims of the method. MGRE refines the representation through prototype-guided distillation; improved representations then stabilize multi-view clustering assignments; these more reliable assignments yield more accurate cijc_{ij} weights for contrastive learning; and the contrastive updates further sharpen the representation (Jing et al., 8 Sep 2025). The paper characterizes this as a mutually reinforcing feedback loop rather than as two isolated regularizers (Jing et al., 8 Sep 2025).

5. Objective, optimization, and evaluation protocol

The full MCCL objective is

L=λSup⋅(Lclss+LC+λS⋅LS)+λUnsup⋅(Lclsu+LHCL)\mathcal{L} = \lambda_{\mathrm{Sup}} \cdot \big( \mathcal{L}_{\mathrm{cls}}^s + \mathcal{L}_C + \lambda_S \cdot \mathcal{L}_S \big) + \lambda_{\mathrm{Unsup}} \cdot \big( \mathcal{L}_{\mathrm{cls}}^u + \mathcal{L}_{HCL} \big)

(Jing et al., 8 Sep 2025). On VB100, the best reported All ACC is obtained at λS=0.5\lambda_S = 0.5 and λSup=0.45\lambda_{\mathrm{Sup}} = 0.45 (Jing et al., 8 Sep 2025).

Training follows a two-stage procedure. Stage 1 is a supervised warm-up on labeled data using cross-entropy with SGD, learning rate $0.005$, momentum $0.9$, weight decay 10−410^{-4}, input size 224×224224 \times 224, and T=8T=8 frames (Jing et al., 8 Sep 2025). Stage 2 performs joint Video-GCD training on labeled and unlabeled mini-batches: multi-perspective features are extracted, MPRA produces fiSTFf_i^{STF}, horizontal and vertical clusterings are built, wijw_{ij} and cijc_{ij} are computed, LHCL\mathcal{L}_{HCL} is evaluated, prototypes are read for LC\mathcal{L}_C and LS\mathcal{L}_S, SimGCD classification losses are added, and the total loss is optimized (Jing et al., 8 Sep 2025).

At inference time, features are extracted for all videos, and K-Means clustering followed by Hungarian matching is used to report All ACC, Old ACC, and New ACC (Jing et al., 8 Sep 2025). The benchmark introduced alongside MCCL includes action-recognition datasets UCF101, SSv2, and Kinetics-400, and fine-grained bird datasets VB100 and IBC127 (Jing et al., 8 Sep 2025). The split protocol is “even-odd” known/unknown for all except VB100, where the first 50% are known and the last 50% unknown; for computational efficiency, 15% of SSv2 and Kinetics-400 are sampled, while UCF101, VB100, and IBC127 use full datasets (Jing et al., 8 Sep 2025).

The hardware reported for training is 2×RTX 4090 GPUs, and 4×RTX 4090 on K400 (Jing et al., 8 Sep 2025).

6. Empirical performance, ablations, and conceptual distinctions

Across the reported benchmarks, MCCL improves over image-based GCD baselines adapted to the video setting (Jing et al., 8 Sep 2025). On UCF101, MCCL obtains $68.27/89.54/57.80$ in All/Old/New ACC, compared with SimGCD at $63.93/82.65/54.70$ and SelfEx at $63.43/77.07/56.72$ (Jing et al., 8 Sep 2025). On SSv2, the reported values are $13.58/18.69/11.57$ for MCCL, versus $11.36/14.27/9.93$ for SimGCD and $12.54/14.96/11.35$ for SelfEx (Jing et al., 8 Sep 2025). On Kinetics-400, MCCL reaches $22.95/29.89/19.51$, against $21.57/25.15/19.79$ for SimGCD and $21.23/23.14/20.28$ for SelfEx (Jing et al., 8 Sep 2025). On VB100, MCCL reports $46.17/70.51/34.81$, compared with $33.75/37.66/32.01$ for SimGCD and $40.43/47.83/33.68$ for SelfEx; on IBC127, it reports $38.06/48.83/32.72$, compared with $31.88/31.31/32.16$ and $33.59/33.76/33.50$, respectively (Jing et al., 8 Sep 2025).

Ablations separate the effects of MGRE and CACL. On VB100, the baseline gives $33.75/37.66/32.01$, adding MGRE gives $44.02/67.57/33.02$, adding CACL gives $39.50/50.62/34.31$, and the full model gives $46.17/70.51/34.81$ (Jing et al., 8 Sep 2025). On UCF101, the same progression is $64.28/88.66/52.27$ for +MGRE, $65.42/82.37/57.06$ for +CACL, and $68.27/89.54/57.80$ for the full model (Jing et al., 8 Sep 2025). The source analysis further states that LS\mathcal{L}_S notably boosts Old ACC, whereas LC\mathcal{L}_C complements it by improving New ACC; MPRA and LHCL\mathcal{L}_{HCL} are also described as complementary (Jing et al., 8 Sep 2025).

The method is positioned against two distinct comparison axes. Relative to image-based GCD approaches such as SimGCD, InfoSieve, SPTNet, and SelfEx, MCCL explicitly exploits spatiotemporal cues through MPRA and horizontal/vertical multi-view voting, and introduces dual-level memory through feature and logit prototypes (Jing et al., 8 Sep 2025). Relative to semi-supervised action recognition, MCCL is defined for a setting in which unlabeled videos include unknown categories, so it combines parametric classification with non-parametric voting and prototype distillation rather than assuming closed-set supervision (Jing et al., 8 Sep 2025).

Several limitations are identified in the source material. Multiple clusterings across views and granularity levels increase computational cost; storing and updating prototypes adds memory overhead, though the design keeps this moderate by averaging over a small labeled subset; performance depends on λSup\lambda_{\mathrm{Sup}}, λS\lambda_S, η\eta, and temperature parameters; and errors in estimating the category count can degrade clustering, with particularly large gaps noted on SSv2 and IBC127 (Jing et al., 8 Sep 2025). Potential improvements proposed in the paper include EMA-based prototype updates, memory-informed voting that incorporates prototype similarity into cijc_{ij}, more efficient clustering approximations, and adaptive granularity schedules (Jing et al., 8 Sep 2025).

A persistent source of confusion is terminological rather than algorithmic. In semi-supervised semantic segmentation, “MCCL” denotes Multi-Constraint Consistency Learning and explicitly uses no memory bank, no InfoNCE, and no teacher model (Yin et al., 23 Mar 2025). In co-salient object detection, “MCCL” denotes Memory-aided Contrastive Consensus Learning, where a momentum memory per class or group supports a triplet-style contrastive objective and training-only adversarial integrity learning (Zheng et al., 2023). A nearby but differently named formulation is Memory Consistency guided Divide-and-conquer Learning (MCDL), whose contrastive component can be interpreted as memory-guided and consistency-aware because it filters supervised contrastive positives by credibility derived from dual prediction-history memory banks, but it is not introduced under the name MCCL (Tu et al., 2024). These distinctions matter because the exact expansion determines whether “memory” means class prototypes, momentum group memories, or historical prediction queues, and whether “contrastive learning” refers to weighted InfoNCE, triplet loss, or supervised contrastive learning.

Topic to Video (Beta)

No one has generated a video about this topic yet.

Whiteboard

No one has generated a whiteboard explanation for this topic yet.

Follow Topic

Get notified by email when new papers are published related to Memory-guided Consistency-aware Contrastive Learning (MCCL).