--- title: 'SpectralGCD: Multimodal Category Discovery' url: https://www.emergentmind.com/topics/spectralgcd type: topic --- # SpectralGCD: Multimodal Category Discovery Searching arXiv for "SpectralGCD" and closely related terminology to ground the article in the relevant paper. SpectralGCD is an efficient and effective multimodal approach to Generalized Category Discovery (GCD) that uses CLIP cross-modal image-concept similarities as a unified cross-modal representation [2602.17395]. GCD aims to identify novel categories in unlabeled data while leveraging a small labeled subset of known classes. In SpectralGCD, each image is expressed as a mixture over semantic concepts from a large task-agnostic dictionary, and Spectral Filtering exploits a cross-modal covariance matrix over the softmaxed similarities measured by a strong teacher model to automatically retain only relevant concepts from the dictionary. Forward and reverse knowledge distillation from the same teacher ensure that the cross-modal representations of the student remain both semantically sufficient and well-aligned. Across six benchmarks, the reported results show accuracy comparable to or significantly superior to state-of-the-art methods at a fraction of the computational cost [2602.17395]. ## 1. Problem formulation and scope Generalized Category Discovery is defined on a training dataset $\mathcal{D}=\mathcal{D}_l\cup\mathcal{D}_u$, where $\mathcal{D}_l=\{(x_i^l,y_i^l)\}_{i=1}^{N_l}$ is the labeled set of $N_l$ images $x_i^l\in\mathcal{X}$ with known “Old” labels $y_i^l\in\mathcal{Y}_l$, and $\mathcal{D}_u=\{x_j^u\}_{j=1}^{N_u}$ is the unlabeled set of $N_u$ images whose true labels lie in $\mathcal{Y}_u$, with $\mathcal{Y}_l\subset\mathcal{Y}_u$ and novel classes defined by $\mathcal{Y}_n=\mathcal{Y}_u\setminus\mathcal{Y}_l$. The total number of classes is $K=|\mathcal{Y}_l\cup\mathcal{Y}_n|$. The goal is to assign each $x\in\mathcal{D}_u$ to one of these $K$ categories by exploiting supervision from $\mathcal{D}_l$ while discovering clusters corresponding to novel classes in $\mathcal{Y}_n$ [2602.17395]. The method is positioned against two difficulties stated explicitly in the paper. Training a parametric classifier solely on image features often leads to overfitting to old classes, and recent multimodal approaches improve performance by incorporating textual information; however, they treat modalities independently and incur high computational cost. SpectralGCD addresses these issues by adopting a single cross-modal representation in which visual features are anchored to explicit semantics through image-concept similarities. This suggests that the method belongs simultaneously to multimodal representation learning, semi-supervised class discovery, and knowledge distillation. A plausible implication is that its central design choice is not merely the addition of text features, but the replacement of separate visual and textual branches with a common semantic basis defined by a concept dictionary. ## 2. Unified cross-modal representation SpectralGCD builds on CLIP to obtain a unified, semantic “mixture-of-concepts” representation for each image [2602.17395]. Let $\overline{\mathbf{C}}=\{c_1,c_2,\dots,c_M\}$ be a large, task-agnostic dictionary of $M$ textual concepts. Each concept $c_j$ is encoded with the CLIP text encoder $g_\phi(c_j)\in\mathbb{R}^{d_t}$ and each image $x_i$ with the CLIP image encoder $f_\theta(x_i)\in\mathbb{R}^{d_v}$. For image $i$ and concept $j$, the normalized cosine similarity is $$ s_{i,j}=\frac{f_\theta(x_i)^{\top}g_\phi(c_j)}{\|f_\theta(x_i)\|\,\|g_\phi(c_j)\|}\;\bigl(1/\tau\bigr), $$ where $\tau$ is CLIP’s temperature. These similarities are collected into the raw cross-modal vector $$ z_i=[\,s_{i,1},\dots,s_{i,M}\,]^{\top}\in\mathbb{R}^M. $$ A softmax along the $M$ dimensions gives $$ q_i=\mathrm{softmax}(z_i), $$ so that $q_{i,j}\ge 0$ and $\sum_{j=1}^M q_{i,j}=1$. The paper interprets this vector as a mixture over concepts, where $q_{i,j}$ is the probability mass or weight of concept $j$ in image $i$. The representation is intended to reduce reliance on spurious visual cues. The paper states that a large CLIP dictionary contains many background or spurious concepts such as “tree” and “grass” that co-activate often but do not discriminate classes. By representing an image through a distribution over textual concepts rather than through image features alone, SpectralGCD anchors the downstream classifier to explicit semantics. ## 3. Spectral Filtering and concept selection Spectral Filtering is the mechanism used to automatically retain only those concepts whose co-activations carry informative, task-relevant signal [2602.17395]. First, $q_i$ is computed for every sample $x_i$ in the entire dataset $\mathcal{D}=\mathcal{D}_l\cup\mathcal{D}_u$ using a strong, frozen CLIP teacher $(f_{\theta^\ast},g_{\phi^\ast})$. With $\mu=(1/N)\sum_1^N q_i$, the method forms the $M\times M$ covariance $$ G=\tfrac{1}{N-1}\sum_{i=1}^N (q_i-\mu)(q_i-\mu)^{\top}. $$ The covariance is eigendecomposed as $$ G=V\Lambda V^{\top}, $$ where $\Lambda=\mathrm{diag}(\lambda_1\ge\lambda_2\ge\dots\ge\lambda_M)$ and $V=[v_1,v_2,\dots,v_M]$. The explained variance ratio is $$ r_k=\frac{\sum_{i=1}^k \lambda_i}{\sum_{i=1}^M \lambda_i}. $$ The smallest $k^\ast$ such that $r_{k^\ast}\ge\beta_e$ is selected, with $\beta_e=0.95$ listed among the key hyperparameters. Denoting $V_\ast=[v_1,\dots,v_{k^\ast}]$, the method forms the concept importance vector $$ s=\sum_{i=1}^{k^\ast}\lambda_i\,(v_i\circ v_i)\in\mathbb{R}^M, $$ where $\circ$ denotes element-wise square. After sorting $s$ in descending order, only the top $j^\ast$ concepts are retained so that $$ \sum_{i=1}^{j^\ast} s_{(i)} \bigg/ \sum_{i=1}^{M} s_i \ge \beta_c, $$ with $\beta_c=0.99$. This yields a filtered dictionary $\hat{\mathbf{C}}$ of size $M'\ll M$. At training time the student computes $\bar z_i$ over $\hat{\mathbf{C}}$. To further remove residual noise, the paper states that one can project the student’s softmaxed vector through the top subspace: $$ \bar q_i=\mathrm{softmax}(\bar z_i),\qquad \bar q'_i=V_\ast V_\ast^{\top}\bar q_i\in\mathbb{R}^{M'}. $$ In practice, the reported implementation simply re-normalizes $\bar q'_i$ or directly truncates $\hat{\mathbf{C}}$, which achieves essentially the same effect of discarding low-signal concepts. The stated interpretation is that Spectral Filtering finds the principal subspace of co-activations, retains only directions that explain most of the variance tied to semantic class structure, and discards noise. This yields a compact, discriminative cross-modal basis that anchors the classifier to explicit, meaningful semantics. ## 4. Teacher–student distillation and optimization To ensure that the student’s cross-modal representation remains semantically faithful to the teacher, SpectralGCD uses both forward and reverse distillation [2602.17395]. Let $$ \hat z_i^\ast = z_{\theta^\ast,\phi^\ast}(x_i;\hat{\mathbf{C}}),\qquad \hat z_i = z_{\theta,\phi}(x_i;\hat{\mathbf{C}}), $$ and let $\sigma(\cdot)=\mathrm{softmax}(\cdot)$ operate on each $M'$-dimensional vector. Forward Distillation is $$ \mathcal{L}_{fd}=-\tfrac{1}{|\mathcal{B}|}\sum_{i\in\mathcal{B}} \sigma(\hat z_i^\ast)^{\top}\log \sigma(\hat z_i), $$ which encourages the student to match the teacher’s softened distribution. Reverse Distillation is $$ \mathcal{L}_{rd}=-\tfrac{1}{|\mathcal{B}|}\sum_{i\in\mathcal{B}} \sigma(\hat z_i)^{\top}\log \sigma(\hat z_i^\ast), $$ which encourages the student to avoid concepts unlikely under the teacher. As in SimGCD, a parametric classifier $L_\psi$ is applied on a low-dimensional projection $u_i=W^{\top}\bar z_i$, producing $p_i=L_\psi(u_i)$, and a contrastive MLP $\mathcal{M}$ on $u_i$ produces $w_i=\mathcal{M}(u_i)$. The losses are: $$ \mathcal{L}_{cls}^s=\tfrac{1}{|\mathcal{B}_l|}\sum_{i\in\mathcal{B}_l} CE(p_i,y_i), $$ for supervised classification on labeled samples, $$ \mathcal{L}_{cls}^u=\tfrac{1}{|\mathcal{B}|}\sum_{i\in\mathcal{B}} [\, CE(p_i,p'_i)-\epsilon\,H(\bar p)\,], $$ for unsupervised self-distillation on augmented views, and $$ \mathcal{L}_{con}=\lambda\,\mathcal{L}_{con}^s + (1-\lambda)\,\mathcal{L}_{con}^u $$ for supervised contrastive on labeled pairs and unsupervised contrastive on all samples. The overall objective is $$ \mathcal{L}=\mathcal{L}_{cls}^s+\mathcal{L}_{cls}^u+\mathcal{L}_{con}^s+\mathcal{L}_{con}^u+\mathcal{L}_{fd}+\mathcal{L}_{rd}. $$ The algorithmic summary in the paper proceeds in seven stages: precompute teacher logits over the large dictionary; softmax and build the covariance $G$; select $k^\ast$ via $\beta_e$ and threshold concept importance via $\beta_c$ to obtain $\hat{\mathbf{C}}$; precompute teacher filtered logits over $\hat{\mathbf{C}}$; initialize the student CLIP image encoder $f_\theta$, projection $W$, classifier $L_\psi$, and MLP $\mathcal{M}$ while freezing the text encoder $g_\phi$; train for $E=200$ epochs with batch size $128$; and return the student model for clustering $\mathcal{D}_u$ via $k$-means on $u_i$ or by taking $\arg\max p_i$. The listed hyperparameters are $\beta_e=0.95$, $\beta_c=0.99$, distillation temperature $\tau_s=\tau_t=0.01$ for $q$, classifier/contrastive $\tau=0.1$, $\lambda=0.35$, and learning rates $5e\!-\!3$ for CLIP fine-tuning and $1e\!-\!1$ for $W,\psi,\mathcal{M}$. ## 5. Benchmarks, comparative results, and efficiency The reported evaluation uses six benchmarks and clustering accuracy on $\mathcal{D}_u$, reported separately for Old, New, and All, balanced via Hungarian matching [2602.17395]. The datasets are CUB (200 classes, 100 old / 100 new), Stanford Cars (196/98), FGVC-Aircraft (100/50), CIFAR-10 (10/5), CIFAR-100 (100/80), and ImageNet-100 (100/50). For All Accuracy, the paper reports the following comparisons. On CUB, SimGCD (CLIP B/16) obtains $60.3$, GET $77.0$, TextGCD (Tags+Attr) $76.6$, and SpectralGCD (Tags) $79.2$. On Cars, the corresponding values are $53.8$, $78.5$, $86.9$, and $89.1$. On Aircraft they are $54.2$, $58.9$, $50.8$, and $63.0$. On CIFAR-10 they are $97.1$, $97.2$, $98.2$, and $98.5$. On CIFAR-100 they are $80.1$, $82.1$, $85.7$, and $86.1$. On ImageNet-100 they are $83.0$, $91.7$, $88.0$, and $93.4$. The paper states that SpectralGCD sets new state-of-the-art on five of six benchmarks, often improving by $>2$ pp over prior multimodal methods and by $>16$ pp over unimodal SimGCD on fine-grained data. The ablation studies attribute a substantial role to distillation and filtering. Forward+reverse distillation yields Spearman $\rho\approx 0.66$ versus $\rho\approx 0.49$ without KD, with a $+11$ pp All gain on Cars. Varying $\beta_e\in[0.8,0.99]$ and $\beta_c\in[0.9,0.995]$ shows stable gains, especially on fine-grained Cars. Using Tags vs. OpenImages-v7 vs. WordNet, the paper reports that SpectralGCD consistently outperforms TextGCD and GET across these dictionaries. The computational profile is presented as a central property of the method. The teacher’s heavy forward passes over the dictionary are done once offline, and the text encoder remains frozen thereafter. The student trains only its image encoder’s last block, plus a small MLP and linear layers. No text-inversion networks or LLM calls are needed at training time. End-to-end training time on CUB is $\approx 1.9$ min for spectral filtering plus $\approx 22$ min for student training on an RTX 4090, comparable to the unimodal SimGCD and far below GET’s $52$ min or TextGCD’s $35$ min. ## 6. Interpretation, misconceptions, and terminological scope A common source of ambiguity is that the string “spectral GCD” appears in multiple, unrelated arXiv contexts. In graph theory, “spectral GCD” denotes the invariant $$ \theta(G)=\gcd\bigl\{\,t(G),\;\Delta(G)\bigr\}, $$ with $t(G)=2^{-\lfloor n/2\rfloor}\det W(G)$ and $\Delta(G)=\prod_{1\le j