Fukunaga-Koontz Linear Discriminant Analysis

• Fukunaga-Koontz LDA is a supervised dimensionality reduction method that decorrelates within-class scatter via whitening to improve class separability.
• The method involves whitening the within-class scatter, diagonalizing the resulting between-class scatter, and constructing a closed-form projection via eigen-decomposition.
• Its practical applications include high-dimensional vision-language models, achieving significant feature compression and improved classification accuracy over traditional LDA.

Fukunaga-Koontz Linear Discriminant Analysis (FK-LDA) is a supervised dimensionality reduction technique that extends classical Linear Discriminant Analysis (LDA) by decorrelating within-class scatter through a whitening transformation before optimizing inter-class separation. Originating from statistical pattern recognition, FK-LDA identifies linear projections that both suppress within-class variation and enhance between-class discrimination. In practical contexts such as vision-LLM adaptation, FK-LDA provides a closed-form, computationally efficient procedure for reshaping embedding spaces to yield improved class separability, robust dimensionality reduction, and efficient representations for large-scale classification and retrieval tasks (Suchanek et al., 1 Feb 2026).

1. Mathematical Framework

FK-LDA considers a $D$-dimensional embedding space with $N$ labeled samples $\{(x_i, y_i)\}_{i=1}^N$, where $x_i \in \mathbb{R}^D$ and $y_i \in \{1, \ldots, K\}$.

Key definitions:

• Per-class mean: $\mu_k = \frac{1}{N_k} \sum_{i: y_i = k} x_i$
• Global mean: $\mu = \frac{1}{N} \sum_i x_i$
• Within-class scatter: $S_w = \sum_{k=1}^K S_k$, $$S_k = \sum_{i: y_i=k} (x_i - \mu_k)(x_i - \mu_k)^\top$$
• Between-class scatter: $$S_b = \sum_{k=1}^K N_k (\mu_k - \mu)(\mu_k - \mu)^\top$$

FK-LDA departs from classical LDA by transforming (whitening) the data such that $N$0 becomes the identity, thus "sphering out" within-class variations prior to addressing between-class separation.

2. Optimization Objective

The FK-LDA objective is to maximize the Rayleigh quotient:

$N$1

Following whitening, $N$2, so the problem reduces to

$N$3

where $N$4 is the between-class scatter computed in the whitened space. Maximizing $N$5 yields directions of maximal class-mean separation, with within-class variation already normalized (Suchanek et al., 1 Feb 2026).

3. Closed-Form FK-LDA Solution

The procedure for FK-LDA comprises four principal steps:

1. Regularization and Diagonalization of $N$6:
   • Form the regularized scatter $N$7, with a small $N$8 to ensure full rank.
   • Compute $N$9, with $\{(x_i, y_i)\}_{i=1}^N$0.
   • Whitening transform: $\{(x_i, y_i)\}_{i=1}^N$1
2. Whitening of Class Means:
   • Whitened class mean offsets: $\{(x_i, y_i)\}_{i=1}^N$2
3. Computation and Diagonalization of Whitened Between-Class Scatter:
   • $\{(x_i, y_i)\}_{i=1}^N$3
   • $\{(x_i, y_i)\}_{i=1}^N$4, $\{(x_i, y_i)\}_{i=1}^N$5
4. Final Projection Construction:
   • Retain top $\{(x_i, y_i)\}_{i=1}^N$6 eigenvectors: $\{(x_i, y_i)\}_{i=1}^N$7
   • FK-LDA projection: $\{(x_i, y_i)\}_{i=1}^N$8 (for $\{(x_i, y_i)\}_{i=1}^N$9)

The algorithmic complexity is $x_i \in \mathbb{R}^D$0 for scatter computation and $x_i \in \mathbb{R}^D$1 for eigendecompositions, with $x_i \in \mathbb{R}^D$2 storage (Suchanek et al., 1 Feb 2026).

4. Geometric and Theoretical Properties

FK-LDA's geometric effect is to "sphere" each class distribution, equalizing within-class variance in all directions. This sphered representation transforms each class into an approximately isotropic unit ball. Subsequent diagonalization of $x_i \in \mathbb{R}^D$3 yields axes along which class centroids are maximally separated, ensuring that projections onto these axes stretch inter-class distances while maintaining normalized within-class spread.

A key distinction from classical LDA is that classical LDA is constrained to $x_i \in \mathbb{R}^D$4 directions (where $x_i \in \mathbb{R}^D$5 is the number of classes), while FK-LDA, due to whitening, can produce up to $x_i \in \mathbb{R}^D$6 directions, limited by the numerical rank of $x_i \in \mathbb{R}^D$7. This suggests broader applicability in scenarios with high-dimensional, anisotropic embeddings or where the class count is large relative to embedding dimensionality (Suchanek et al., 1 Feb 2026).

5. Practical Implementation and Algorithmic Details

The FK-LDA algorithm proceeds as follows:

Step	Action	Output
1	Compute class and global means	$x_i \in \mathbb{R}^D$8
2	Compute within-class scatter $x_i \in \mathbb{R}^D$9 and regularize to $y_i \in \{1, \ldots, K\}$0	$y_i \in \{1, \ldots, K\}$1
3	Eigendecompose $y_i \in \{1, \ldots, K\}$2, derive whitening transform $y_i \in \{1, \ldots, K\}$3	$y_i \in \{1, \ldots, K\}$4
4	Compute whitened class offsets $y_i \in \{1, \ldots, K\}$5	$y_i \in \{1, \ldots, K\}$6
5	Form whitened between-class scatter $y_i \in \{1, \ldots, K\}$7	$y_i \in \{1, \ldots, K\}$8
6	Eigendecompose $y_i \in \{1, \ldots, K\}$9; select top $\mu_k = \frac{1}{N_k} \sum_{i: y_i = k} x_i$0 eigenvectors $\mu_k = \frac{1}{N_k} \sum_{i: y_i = k} x_i$1	$\mu_k = \frac{1}{N_k} \sum_{i: y_i = k} x_i$2
7	Final FK-LDA projection: $\mu_k = \frac{1}{N_k} \sum_{i: y_i = k} x_i$3	$\mu_k = \frac{1}{N_k} \sum_{i: y_i = k} x_i$4

The regularization parameter $\mu_k = \frac{1}{N_k} \sum_{i: y_i = k} x_i$5 is set to ensure $\mu_k = \frac{1}{N_k} \sum_{i: y_i = k} x_i$6 is full-rank, typically determined via cross-validation. Numerical stability can be enhanced by employing symmetric eigendecomposition for $\mu_k = \frac{1}{N_k} \sum_{i: y_i = k} x_i$7. For very high-dimensional data, truncated SVD or randomized eigendecomposition methods are recommended. All input data should be zero-centered before scatter computation (Suchanek et al., 1 Feb 2026).

6. Applications and Comparative Analysis

FK-LDA is particularly effective for scenarios involving high-dimensional models with anisotropic within-class covariance, such as vision-LLM embeddings (e.g., CLIP features). It has been empirically demonstrated to provide "substantial compression by up to 10-12x with little or no loss in accuracy," and to improve prototype-based classification accuracy (e.g., top-1 accuracy on ImageNet-1K improving from 75.1% to 79.1%) while supporting label spaces of up to 21K classes without degradation (Suchanek et al., 1 Feb 2026).

Compared to classical LDA, FK-LDA offers improved numerical stability—since matrix inversion for ill-conditioned $\mu_k = \frac{1}{N_k} \sum_{i: y_i = k} x_i$8 is replaced by whitening, and its projection dimensionality is not limited to $\mu_k = \frac{1}{N_k} \sum_{i: y_i = k} x_i$9 directions. It provides a closed-form, linear, and efficient technique for adapting and compressing large-scale feature representations in modern supervised classification pipelines (Suchanek et al., 1 Feb 2026).

7. Limitations and Theoretical Considerations

FK-LDA's effectiveness is most pronounced when within-class scatter matrices are ill-conditioned, classes have few samples, or embeddings are high-dimensional with disparate variances across directions. If within-class scatter is nearly isotropic, FK-LDA's advantage over classical LDA is diminished. Empirical robustness to regularization is observed once $\mu = \frac{1}{N} \sum_i x_i$0 exceeds the smallest nonzero eigenvalue of $\mu = \frac{1}{N} \sum_i x_i$1; too small $\mu = \frac{1}{N} \sum_i x_i$2 may lead to instability, while excessively large $\mu = \frac{1}{N} \sum_i x_i$3 can attenuate discriminatory power. FK-LDA is not designed for nonlinear class boundaries, although it may facilitate subsequent nonlinear post-processing.

A plausible implication is that in large-scale applications where training data per class is limited or model representations are not naturally isotropic, FK-LDA provides a practical alternative to classical methods, prioritizing computational tractability and improved inter-class discrimination (Suchanek et al., 1 Feb 2026).