Koo-Fu CLIP: FK-LDA for CLIP Embeddings

• Koo-Fu CLIP is a supervised adaptation protocol that reshapes high-dimensional CLIP embeddings using FK-LDA to enhance class discriminability.
• It employs a closed-form two-stage pipeline with regularization and whitening to reduce dimensionality and suppress within-class variance.
• Empirical results on ImageNet benchmarks demonstrate improved classification accuracy and efficient retrieval in large-scale supervised tasks.

Koo-Fu CLIP is a supervised adaptation protocol for vision-language representations, specifically targeting embeddings produced by models such as CLIP. It leverages Fukunaga–Koontz Linear Discriminant Analysis (FK-LDA) to reshape the geometry of high-dimensional embedding spaces, suppressing within-class variation and enhancing between-class discriminability via a closed-form linear projection. The approach achieves substantial dimensionality reduction and efficient supervised adaptation for large-scale supervised classification and retrieval tasks, while remaining computationally lightweight and robust on datasets such as ImageNet-1K, ImageNet-14K, and ImageNet-21K (Suchanek et al., 1 Feb 2026).

1. Motivation and Problem Context

CLIP and similar vision-LLMs yield general-purpose, high-dimensional embeddings (often 768-D or higher) that are effective for zero-shot transfer but not tailored for supervised classification. The raw embedding space often contains excessive within-class variance and only limited separation between classes, impeding straightforward nearest-prototype classification and efficient use in downstream, label-rich scenarios. Adapting these representations for discriminative tasks therefore requires a process that can both compress the feature dimensions and amplify inter-class structure without expensive retraining (Suchanek et al., 1 Feb 2026).

Koo-Fu CLIP directly addresses this challenge: given labeled embedding data $x_i \in \mathbb{R}^D$ with labels $y_i \in \{1,...,K\}$, the objective is to learn a linear map into a lower-dimensional space $\mathbb{R}^L$, with $L\leq D$, that minimizes within-class variance and maximizes between-class separation.

2. Fukunaga–Koontz Linear Discriminant Analysis

Koo-Fu CLIP operationalizes FK-LDA for vision-language adaptation. FK-LDA extends classical Linear Discriminant Analysis (LDA) by introducing a whitening step that regularizes and numerically stabilizes the mapping process. The essential goal is to find a projection $W$ maximizing the Fisher criterion:

$$J(W) = \frac{\operatorname{tr}(W^\top S_b W)}{\operatorname{tr}(W^\top S_w W)}$$

where $S_w$ is the within-class scatter matrix and $S_b$ is the between-class scatter matrix. Unlike classical LDA, which solves the generalized eigenproblem $S_b v = \lambda S_w v$ (with rank limitations and instability if $S_w$ is ill-conditioned), FK-LDA first whitens $y_i \in \{1,...,K\}$0 and then diagonalizes $y_i \in \{1,...,K\}$1 in this whitened space (Suchanek et al., 1 Feb 2026).

Scatter Matrix Construction

• Global mean: $y_i \in \{1,...,K\}$2
• Class mean: $y_i \in \{1,...,K\}$3
• Within-class scatter: $y_i \in \{1,...,K\}$4
• Between-class scatter: $y_i \in \{1,...,K\}$5

3. Closed-Form Solution and Algorithmic Pipeline

The FK-LDA adaptation comprises a two-stage diagonalization pipeline:

Closed-Form Steps

1. Regularization: Add a ridge term to ensure $y_i \in \{1,...,K\}$6 is invertible:

$y_i \in \{1,...,K\}$7

2. Whitening: Eigendecompose $y_i \in \{1,...,K\}$8:

$y_i \in \{1,...,K\}$9

The whitening transform $\mathbb{R}^L$0 maps $\mathbb{R}^L$1 to the identity.

3. Whitened Means and Between-Class Scatter: For each class $\mathbb{R}^L$2,

$\mathbb{R}^L$3

$\mathbb{R}^L$4

4. Diagonalization of Between-Class Scatter: Eigendecompose $\mathbb{R}^L$5:

$\mathbb{R}^L$6

$\mathbb{R}^L$7 are the top $\mathbb{R}^L$8 eigenvectors.

5. Final Projection: The transformation mapping $\mathbb{R}^L$9 to $L\leq D$0 is:

$L\leq D$1

The columns of $L\leq D$2 (or rows of $L\leq D$3) define discriminant directions maximizing inter-class spread and eliminating intra-class variation.

Summary of Steps

Step	Operation	Output
1	Compute means and scatters	$L\leq D$4
2	Regularize $L\leq D$5	$L\leq D$6
3	Eigendecompose $L\leq D$7	$L\leq D$8
4	Compute $L\leq D$9	Whitening matrix
5	Whiten means, build $W$0	$W$1
6	Eigendecompose $W$2	$W$3
7	Select top $W$4 columns $W$5	Final projector $W$6

4. Geometric and Statistical Properties

The whitening transformation $W$7 spheres the within-class covariances, making all classes isotropic (identity covariance) in the transformed space. This suppresses within-class variation uniformly across directions. The subsequent diagonalization of $W$8 finds orthogonal axes that maximize the projected between-class separations. Thus, the final embedding aligns class means along maximally discriminative directions while compressing to a user-specified dimension $W$9.

A key distinction from classical LDA is the absence of a hard rank constraint: classical LDA yields at most $$J(W) = \frac{\operatorname{tr}(W^\top S_b W)}{\operatorname{tr}(W^\top S_w W)}$$0 meaningful directions due to the rank of $$J(W) = \frac{\operatorname{tr}(W^\top S_b W)}{\operatorname{tr}(W^\top S_w W)}$$1, whereas FK-LDA in the whitened space is full-rank up to $$J(W) = \frac{\operatorname{tr}(W^\top S_b W)}{\operatorname{tr}(W^\top S_w W)}$$2, permitting arbitrary dimensionality reduction to $$J(W) = \frac{\operatorname{tr}(W^\top S_b W)}{\operatorname{tr}(W^\top S_w W)}$$3.

5. Computational Complexity and Practical Implementation

The adaptation requires:

• Accumulating scatter matrices: $$J(W) = \frac{\operatorname{tr}(W^\top S_b W)}{\operatorname{tr}(W^\top S_w W)}$$4
• Two eigendecompositions (size $$J(W) = \frac{\operatorname{tr}(W^\top S_b W)}{\operatorname{tr}(W^\top S_w W)}$$5): $$J(W) = \frac{\operatorname{tr}(W^\top S_b W)}{\operatorname{tr}(W^\top S_w W)}$$6 each
• Inference for a new sample: $$J(W) = \frac{\operatorname{tr}(W^\top S_b W)}{\operatorname{tr}(W^\top S_w W)}$$7 (single matrix-vector multiplication)

Choice of regularization parameter $$J(W) = \frac{\operatorname{tr}(W^\top S_b W)}{\operatorname{tr}(W^\top S_w W)}$$8 is not critical—for CLIP-scale data, any value in the range $$J(W) = \frac{\operatorname{tr}(W^\top S_b W)}{\operatorname{tr}(W^\top S_w W)}$$9–$S_w$0 times the mean diagonal of $S_w$1 suffices to ensure numerical stability and consistent accuracy.

The method supports dimensionality reduction by varying $S_w$2, with the ability to compress the embedding vectors by $S_w$3- to $S_w$4-fold with little or no loss in accuracy on large-scale benchmarks.

6. Empirical Performance and Application Scope

On ImageNet-1K, nearest-prototype classification in the Koo-Fu CLIP-optimized space achieves a top-1 accuracy increase from $S_w$5 (raw CLIP) to $S_w$6. Performance gains are robust as the number of classes scales to ImageNet-14K and ImageNet-21K. The approach provides an efficient closed-form post-hoc transformation, requiring only linear operations and avoiding retraining. This enables scalable large-scale classification and retrieval, especially in scenarios where prototype-based inference or severe embedding compression is needed (Suchanek et al., 1 Feb 2026).

FK-LDA is most effective in scenarios characterized by high-dimensional embeddings and relatively few samples per class, yielding improved class separation for both accuracy and computational efficiency.

7. Theoretical Considerations and Limitations

FK-LDA guarantees a closed-form solution that is robust to the empirical ill-conditioning commonly encountered in high-dimensional embedding spaces generated by models like CLIP. The method analytically optimizes the Fisher criterion, producing a solution that is not limited by the number of classes and is agnostic to the specifics of the base embedding model.

A plausible implication is that FK-LDA may be deployed as a generic adaptation layer atop any pre-trained general-purpose representation with minimal tuning, offering a numerically stable and theoretically grounded alternative to traditional discriminant analysis schemes. Classical LDA’s instability and rank limitation are circumvented by the whitening step, suggesting that Koo-Fu CLIP may be especially suited for transfer learning and fast adaptation pipelines where re-training the backbone is infeasible.

For applications requiring more nuanced or nonlinear discriminative adaptation, a limitation is the linearity of the FK-LDA transformation; nonlinearity is not modeled, so the method’s efficacy is bounded by the linear separability of class structure in the base embedding. However, for a large regime of supervised adaptation problems—particularly those characterized by high-dimensional, over-complete feature spaces—FK-LDA, as instantiated in Koo-Fu CLIP, provides an efficient and well-founded solution.