SpectralGCD: Multimodal Category Discovery

• The paper introduces a unified cross-modal representation using CLIP-derived image-concept similarities and spectral filtering to automatically select discriminative concepts.
• It employs forward and reverse knowledge distillation to ensure the student model learns semantically sufficient and well-aligned features.
• Experiments on six benchmarks demonstrate that SpectralGCD outperforms state-of-the-art methods with significant gains in fine-grained discovery at reduced computational cost.

Searching arXiv for "SpectralGCD" and closely related terminology to ground the article in the relevant paper. SpectralGCD is an efficient and effective multimodal approach to Generalized Category Discovery (GCD) that uses CLIP cross-modal image-concept similarities as a unified cross-modal representation (Caselli et al., 19 Feb 2026). GCD aims to identify novel categories in unlabeled data while leveraging a small labeled subset of known classes. In SpectralGCD, each image is expressed as a mixture over semantic concepts from a large task-agnostic dictionary, and Spectral Filtering exploits a cross-modal covariance matrix over the softmaxed similarities measured by a strong teacher model to automatically retain only relevant concepts from the dictionary. Forward and reverse knowledge distillation from the same teacher ensure that the cross-modal representations of the student remain both semantically sufficient and well-aligned. Across six benchmarks, the reported results show accuracy comparable to or significantly superior to state-of-the-art methods at a fraction of the computational cost (Caselli et al., 19 Feb 2026).

1. Problem formulation and scope

Generalized Category Discovery is defined on a training dataset $\mathcal{D}=\mathcal{D}_l\cup\mathcal{D}_u$, where $\mathcal{D}_l=\{(x_i^l,y_i^l)\}_{i=1}^{N_l}$ is the labeled set of $N_l$ images $x_i^l\in\mathcal{X}$ with known “Old” labels $y_i^l\in\mathcal{Y}_l$, and $\mathcal{D}_u=\{x_j^u\}_{j=1}^{N_u}$ is the unlabeled set of $N_u$ images whose true labels lie in $\mathcal{Y}_u$, with $\mathcal{Y}_l\subset\mathcal{Y}_u$ and novel classes defined by $\mathcal{Y}_n=\mathcal{Y}_u\setminus\mathcal{Y}_l$. The total number of classes is $\mathcal{D}_l=\{(x_i^l,y_i^l)\}_{i=1}^{N_l}$0. The goal is to assign each $\mathcal{D}_l=\{(x_i^l,y_i^l)\}_{i=1}^{N_l}$1 to one of these $\mathcal{D}_l=\{(x_i^l,y_i^l)\}_{i=1}^{N_l}$2 categories by exploiting supervision from $\mathcal{D}_l=\{(x_i^l,y_i^l)\}_{i=1}^{N_l}$3 while discovering clusters corresponding to novel classes in $\mathcal{D}_l=\{(x_i^l,y_i^l)\}_{i=1}^{N_l}$4 (Caselli et al., 19 Feb 2026).

The method is positioned against two difficulties stated explicitly in the paper. Training a parametric classifier solely on image features often leads to overfitting to old classes, and recent multimodal approaches improve performance by incorporating textual information; however, they treat modalities independently and incur high computational cost. SpectralGCD addresses these issues by adopting a single cross-modal representation in which visual features are anchored to explicit semantics through image-concept similarities.

This suggests that the method belongs simultaneously to multimodal representation learning, semi-supervised class discovery, and knowledge distillation. A plausible implication is that its central design choice is not merely the addition of text features, but the replacement of separate visual and textual branches with a common semantic basis defined by a concept dictionary.

2. Unified cross-modal representation

SpectralGCD builds on CLIP to obtain a unified, semantic “mixture-of-concepts” representation for each image (Caselli et al., 19 Feb 2026). Let $\mathcal{D}_l=\{(x_i^l,y_i^l)\}_{i=1}^{N_l}$5 be a large, task-agnostic dictionary of $\mathcal{D}_l=\{(x_i^l,y_i^l)\}_{i=1}^{N_l}$6 textual concepts. Each concept $\mathcal{D}_l=\{(x_i^l,y_i^l)\}_{i=1}^{N_l}$7 is encoded with the CLIP text encoder $\mathcal{D}_l=\{(x_i^l,y_i^l)\}_{i=1}^{N_l}$8 and each image $\mathcal{D}_l=\{(x_i^l,y_i^l)\}_{i=1}^{N_l}$9 with the CLIP image encoder $N_l$0.

For image $N_l$1 and concept $N_l$2, the normalized cosine similarity is

$N_l$3

where $N_l$4 is CLIP’s temperature. These similarities are collected into the raw cross-modal vector

$N_l$5

A softmax along the $N_l$6 dimensions gives

$N_l$7

so that $N_l$8 and $N_l$9. The paper interprets this vector as a mixture over concepts, where $x_i^l\in\mathcal{X}$0 is the probability mass or weight of concept $x_i^l\in\mathcal{X}$1 in image $x_i^l\in\mathcal{X}$2.

The representation is intended to reduce reliance on spurious visual cues. The paper states that a large CLIP dictionary contains many background or spurious concepts such as “tree” and “grass” that co-activate often but do not discriminate classes. By representing an image through a distribution over textual concepts rather than through image features alone, SpectralGCD anchors the downstream classifier to explicit semantics.

3. Spectral Filtering and concept selection

Spectral Filtering is the mechanism used to automatically retain only those concepts whose co-activations carry informative, task-relevant signal (Caselli et al., 19 Feb 2026). First, $x_i^l\in\mathcal{X}$3 is computed for every sample $x_i^l\in\mathcal{X}$4 in the entire dataset $x_i^l\in\mathcal{X}$5 using a strong, frozen CLIP teacher $x_i^l\in\mathcal{X}$6. With $x_i^l\in\mathcal{X}$7, the method forms the $x_i^l\in\mathcal{X}$8 covariance

$x_i^l\in\mathcal{X}$9

The covariance is eigendecomposed as

$y_i^l\in\mathcal{Y}_l$0

where $y_i^l\in\mathcal{Y}_l$1 and $y_i^l\in\mathcal{Y}_l$2. The explained variance ratio is

$y_i^l\in\mathcal{Y}_l$3

The smallest $y_i^l\in\mathcal{Y}_l$4 such that $y_i^l\in\mathcal{Y}_l$5 is selected, with $y_i^l\in\mathcal{Y}_l$6 listed among the key hyperparameters. Denoting $y_i^l\in\mathcal{Y}_l$7, the method forms the concept importance vector

$y_i^l\in\mathcal{Y}_l$8

where $y_i^l\in\mathcal{Y}_l$9 denotes element-wise square. After sorting $\mathcal{D}_u=\{x_j^u\}_{j=1}^{N_u}$0 in descending order, only the top $\mathcal{D}_u=\{x_j^u\}_{j=1}^{N_u}$1 concepts are retained so that

$\mathcal{D}_u=\{x_j^u\}_{j=1}^{N_u}$2

with $\mathcal{D}_u=\{x_j^u\}_{j=1}^{N_u}$3. This yields a filtered dictionary $\mathcal{D}_u=\{x_j^u\}_{j=1}^{N_u}$4 of size $\mathcal{D}_u=\{x_j^u\}_{j=1}^{N_u}$5.

At training time the student computes $\mathcal{D}_u=\{x_j^u\}_{j=1}^{N_u}$6 over $\mathcal{D}_u=\{x_j^u\}_{j=1}^{N_u}$7. To further remove residual noise, the paper states that one can project the student’s softmaxed vector through the top subspace:

$\mathcal{D}_u=\{x_j^u\}_{j=1}^{N_u}$8

In practice, the reported implementation simply re-normalizes $\mathcal{D}_u=\{x_j^u\}_{j=1}^{N_u}$9 or directly truncates $N_u$0, which achieves essentially the same effect of discarding low-signal concepts.

The stated interpretation is that Spectral Filtering finds the principal subspace of co-activations, retains only directions that explain most of the variance tied to semantic class structure, and discards noise. This yields a compact, discriminative cross-modal basis that anchors the classifier to explicit, meaningful semantics.

4. Teacher–student distillation and optimization

To ensure that the student’s cross-modal representation remains semantically faithful to the teacher, SpectralGCD uses both forward and reverse distillation (Caselli et al., 19 Feb 2026). Let

$N_u$1

and let $N_u$2 operate on each $N_u$3-dimensional vector.

Forward Distillation is

$N_u$4

which encourages the student to match the teacher’s softened distribution. Reverse Distillation is

$N_u$5

which encourages the student to avoid concepts unlikely under the teacher.

As in SimGCD, a parametric classifier $N_u$6 is applied on a low-dimensional projection $N_u$7, producing $N_u$8, and a contrastive MLP $N_u$9 on $\mathcal{Y}_u$0 produces $\mathcal{Y}_u$1. The losses are:

$\mathcal{Y}_u$2

for supervised classification on labeled samples,

$\mathcal{Y}_u$3

for unsupervised self-distillation on augmented views, and

$\mathcal{Y}_u$4

for supervised contrastive on labeled pairs and unsupervised contrastive on all samples. The overall objective is

$\mathcal{Y}_u$5

The algorithmic summary in the paper proceeds in seven stages: precompute teacher logits over the large dictionary; softmax and build the covariance $\mathcal{Y}_u$6; select $\mathcal{Y}_u$7 via $\mathcal{Y}_u$8 and threshold concept importance via $\mathcal{Y}_u$9 to obtain $\mathcal{Y}_l\subset\mathcal{Y}_u$0; precompute teacher filtered logits over $\mathcal{Y}_l\subset\mathcal{Y}_u$1; initialize the student CLIP image encoder $\mathcal{Y}_l\subset\mathcal{Y}_u$2, projection $\mathcal{Y}_l\subset\mathcal{Y}_u$3, classifier $\mathcal{Y}_l\subset\mathcal{Y}_u$4, and MLP $\mathcal{Y}_l\subset\mathcal{Y}_u$5 while freezing the text encoder $\mathcal{Y}_l\subset\mathcal{Y}_u$6; train for $\mathcal{Y}_l\subset\mathcal{Y}_u$7 epochs with batch size $\mathcal{Y}_l\subset\mathcal{Y}_u$8; and return the student model for clustering $\mathcal{Y}_l\subset\mathcal{Y}_u$9 via $\mathcal{Y}_n=\mathcal{Y}_u\setminus\mathcal{Y}_l$0-means on $\mathcal{Y}_n=\mathcal{Y}_u\setminus\mathcal{Y}_l$1 or by taking $\mathcal{Y}_n=\mathcal{Y}_u\setminus\mathcal{Y}_l$2. The listed hyperparameters are $\mathcal{Y}_n=\mathcal{Y}_u\setminus\mathcal{Y}_l$3, $\mathcal{Y}_n=\mathcal{Y}_u\setminus\mathcal{Y}_l$4, distillation temperature $\mathcal{Y}_n=\mathcal{Y}_u\setminus\mathcal{Y}_l$5 for $\mathcal{Y}_n=\mathcal{Y}_u\setminus\mathcal{Y}_l$6, classifier/contrastive $\mathcal{Y}_n=\mathcal{Y}_u\setminus\mathcal{Y}_l$7, $\mathcal{Y}_n=\mathcal{Y}_u\setminus\mathcal{Y}_l$8, and learning rates $\mathcal{Y}_n=\mathcal{Y}_u\setminus\mathcal{Y}_l$9 for CLIP fine-tuning and $\mathcal{D}_l=\{(x_i^l,y_i^l)\}_{i=1}^{N_l}$00 for $\mathcal{D}_l=\{(x_i^l,y_i^l)\}_{i=1}^{N_l}$01.

5. Benchmarks, comparative results, and efficiency

The reported evaluation uses six benchmarks and clustering accuracy on $\mathcal{D}_l=\{(x_i^l,y_i^l)\}_{i=1}^{N_l}$02, reported separately for Old, New, and All, balanced via Hungarian matching (Caselli et al., 19 Feb 2026). The datasets are CUB (200 classes, 100 old / 100 new), Stanford Cars (196/98), FGVC-Aircraft (100/50), CIFAR-10 (10/5), CIFAR-100 (100/80), and ImageNet-100 (100/50).

For All Accuracy, the paper reports the following comparisons. On CUB, SimGCD (CLIP B/16) obtains $\mathcal{D}_l=\{(x_i^l,y_i^l)\}_{i=1}^{N_l}$03, GET $\mathcal{D}_l=\{(x_i^l,y_i^l)\}_{i=1}^{N_l}$04, TextGCD (Tags+Attr) $\mathcal{D}_l=\{(x_i^l,y_i^l)\}_{i=1}^{N_l}$05, and SpectralGCD (Tags) $\mathcal{D}_l=\{(x_i^l,y_i^l)\}_{i=1}^{N_l}$06. On Cars, the corresponding values are $\mathcal{D}_l=\{(x_i^l,y_i^l)\}_{i=1}^{N_l}$07, $\mathcal{D}_l=\{(x_i^l,y_i^l)\}_{i=1}^{N_l}$08, $\mathcal{D}_l=\{(x_i^l,y_i^l)\}_{i=1}^{N_l}$09, and $\mathcal{D}_l=\{(x_i^l,y_i^l)\}_{i=1}^{N_l}$10. On Aircraft they are $\mathcal{D}_l=\{(x_i^l,y_i^l)\}_{i=1}^{N_l}$11, $\mathcal{D}_l=\{(x_i^l,y_i^l)\}_{i=1}^{N_l}$12, $\mathcal{D}_l=\{(x_i^l,y_i^l)\}_{i=1}^{N_l}$13, and $\mathcal{D}_l=\{(x_i^l,y_i^l)\}_{i=1}^{N_l}$14. On CIFAR-10 they are $\mathcal{D}_l=\{(x_i^l,y_i^l)\}_{i=1}^{N_l}$15, $\mathcal{D}_l=\{(x_i^l,y_i^l)\}_{i=1}^{N_l}$16, $\mathcal{D}_l=\{(x_i^l,y_i^l)\}_{i=1}^{N_l}$17, and $\mathcal{D}_l=\{(x_i^l,y_i^l)\}_{i=1}^{N_l}$18. On CIFAR-100 they are $\mathcal{D}_l=\{(x_i^l,y_i^l)\}_{i=1}^{N_l}$19, $\mathcal{D}_l=\{(x_i^l,y_i^l)\}_{i=1}^{N_l}$20, $\mathcal{D}_l=\{(x_i^l,y_i^l)\}_{i=1}^{N_l}$21, and $\mathcal{D}_l=\{(x_i^l,y_i^l)\}_{i=1}^{N_l}$22. On ImageNet-100 they are $\mathcal{D}_l=\{(x_i^l,y_i^l)\}_{i=1}^{N_l}$23, $\mathcal{D}_l=\{(x_i^l,y_i^l)\}_{i=1}^{N_l}$24, $\mathcal{D}_l=\{(x_i^l,y_i^l)\}_{i=1}^{N_l}$25, and $\mathcal{D}_l=\{(x_i^l,y_i^l)\}_{i=1}^{N_l}$26. The paper states that SpectralGCD sets new state-of-the-art on five of six benchmarks, often improving by $\mathcal{D}_l=\{(x_i^l,y_i^l)\}_{i=1}^{N_l}$27 pp over prior multimodal methods and by $\mathcal{D}_l=\{(x_i^l,y_i^l)\}_{i=1}^{N_l}$28 pp over unimodal SimGCD on fine-grained data.

The ablation studies attribute a substantial role to distillation and filtering. Forward+reverse distillation yields Spearman $\mathcal{D}_l=\{(x_i^l,y_i^l)\}_{i=1}^{N_l}$29 versus $\mathcal{D}_l=\{(x_i^l,y_i^l)\}_{i=1}^{N_l}$30 without KD, with a $\mathcal{D}_l=\{(x_i^l,y_i^l)\}_{i=1}^{N_l}$31 pp All gain on Cars. Varying $\mathcal{D}_l=\{(x_i^l,y_i^l)\}_{i=1}^{N_l}$32 and $\mathcal{D}_l=\{(x_i^l,y_i^l)\}_{i=1}^{N_l}$33 shows stable gains, especially on fine-grained Cars. Using Tags vs. OpenImages-v7 vs. WordNet, the paper reports that SpectralGCD consistently outperforms TextGCD and GET across these dictionaries.

The computational profile is presented as a central property of the method. The teacher’s heavy forward passes over the dictionary are done once offline, and the text encoder remains frozen thereafter. The student trains only its image encoder’s last block, plus a small MLP and linear layers. No text-inversion networks or LLM calls are needed at training time. End-to-end training time on CUB is $\mathcal{D}_l=\{(x_i^l,y_i^l)\}_{i=1}^{N_l}$34 min for spectral filtering plus $\mathcal{D}_l=\{(x_i^l,y_i^l)\}_{i=1}^{N_l}$35 min for student training on an RTX 4090, comparable to the unimodal SimGCD and far below GET’s $\mathcal{D}_l=\{(x_i^l,y_i^l)\}_{i=1}^{N_l}$36 min or TextGCD’s $\mathcal{D}_l=\{(x_i^l,y_i^l)\}_{i=1}^{N_l}$37 min.

6. Interpretation, misconceptions, and terminological scope

A common source of ambiguity is that the string “spectral GCD” appears in multiple, unrelated arXiv contexts. In graph theory, “spectral GCD” denotes the invariant

$\mathcal{D}_l=\{(x_i^l,y_i^l)\}_{i=1}^{N_l}$38

with $\mathcal{D}_l=\{(x_i^l,y_i^l)\}_{i=1}^{N_l}$39 and $\mathcal{D}_l=\{(x_i^l,y_i^l)\}_{i=1}^{N_l}$40, and it is used in new DGS criteria via primary decomposition (Guo et al., 17 Apr 2025). In analytic number theory, “spectral problem” for GCD matrices concerns the largest eigenvalue of

$\mathcal{D}_l=\{(x_i^l,y_i^l)\}_{i=1}^{N_l}$41

together with bounds derived from Poisson integrals and related probabilistic methods (Aistleitner et al., 2012, Lewko et al., 2014). A different arithmetic usage studies the discrete Fourier transform of the greatest common divisor,

$\mathcal{D}_l=\{(x_i^l,y_i^l)\}_{i=1}^{N_l}$42

as a multiplicative function generalising both the gcd-sum function and Euler’s totient function (Kamp, 2012).

SpectralGCD in the sense of multimodal GCD is not a variant of these graph-theoretic or arithmetic constructions. It is a method for category discovery built around CLIP image-concept similarities, spectral filtering of concept co-activations, and two-way teacher–student distillation (Caselli et al., 19 Feb 2026). This distinction matters because the “spectral” qualifier refers here to eigendecomposition of a cross-modal covariance matrix rather than to graph spectra, Ramanujan sums, or GCD matrices.

Within its own domain, the method should also not be reduced to a generic CLIP-plus-clustering pipeline. The paper’s stated contribution is the combination of a unified cross-modal representation, automatic concept selection through covariance eigenspectra, and forward and reverse distillation that preserve semantic sufficiency and alignment. A plausible implication is that the performance gains on fine-grained benchmarks arise from this joint design rather than from any single ingredient in isolation.