Supervised Locality Preserving Projection (SLPP)

• Supervised Locality Preserving Projection (SLPP) is a subspace learning method that leverages class labels to fully connect same-class sample pairs and preserve local geometric structure.
• It constructs a supervised similarity graph using class membership to form a Laplacian-based optimization, ensuring robust intra-class compactness and inter-class separability.
• SLPP’s efficient closed-form solution via a generalized eigenproblem makes it effective for applications like domain adaptation, zero-shot learning, and face recognition.

Supervised Locality Preserving Projection (SLPP) is a subspace learning algorithm designed to find a discriminative, low-dimensional embedding that preserves local data structure while leveraging class label supervision. SLPP is grounded in graph-based dimensionality reduction and extends the classical locality preserving projections (LPP) by constructing the similarity graph using class labels, fully connecting all within-class pairs. This technique yields a closed-form transformation via a generalized eigenproblem, efficiently capturing both discriminative and geometric information for tasks such as domain adaptation, zero-shot recognition, and face identification (Wang et al., 2019, Huang et al., 2013).

1. Data Formulation and Problem Setup

SLPP operates on labeled datasets, specifically seeking a linear transformation that maps high-dimensional samples into a lower-dimensional subspace where intraclass distances are minimized. Let $X^{(s)}\in\mathbb{R}^{d\times n^s}$ denote $n^s$ labeled source samples and, in transfer learning scenarios, $X^{(tl)}\in\mathbb{R}^{d\times n^{tl}}$ the $n^{tl}$ labeled target samples. All available labeled data are concatenated into $$X^l = [X^{(s)}, X^{(tl)}]\in\mathbb{R}^{d\times n^l}$$, with $n^l = n^s + n^{tl}$. Each sample $x_i$ in $X^l$ is associated with a class label $y_i\in\{1,\ldots,C\}$.

The algorithm seeks a linear projection matrix $P\in\mathbb{R}^{d\times d^{sub}}$ ($n^s$0), such that in the projected space, discriminative and class-consistent locality structure is preserved (Wang et al., 2019, Huang et al., 2013).

2. Construction of the Supervised Similarity Graph

SLPP constructs a weighted adjacency graph $n^s$1 where the edge weight $n^s$2 iff samples $n^s$3 and $n^s$4 share the same class label ($n^s$5), and $n^s$6 otherwise. This fully connects same-class pairs and ignores inter-class pairs, focusing on capturing intra-class localities. The diagonal degree matrix $n^s$7 is given by $n^s$8, and the unnormalized graph Laplacian is $n^s$9.

Unlike unsupervised LPP, which often restricts connections to $X^{(tl)}\in\mathbb{R}^{d\times n^{tl}}$0-nearest neighbors with Gaussian heat-kernel weights, SLPP's adjacency reflects explicit class membership and thus introduces supervision directly into the learned subspace (Wang et al., 2019, Huang et al., 2013).

3. Objective Function and Optimization

SLPP optimizes a criterion minimizing projected distances between all same-class point pairs:

$X^{(tl)}\in\mathbb{R}^{d\times n^{tl}}$1

To avoid degenerate solutions and promote global spread among classes, the problem is cast as a Rayleigh quotient with an added regularization parameter $X^{(tl)}\in\mathbb{R}^{d\times n^{tl}}$2:

$X^{(tl)}\in\mathbb{R}^{d\times n^{tl}}$3

• The numerator $X^{(tl)}\in\mathbb{R}^{d\times n^{tl}}$4 ensures that high-degree, same-class nodes (i.e., points from populous classes) spread out, enhancing discrimination.
• The denominator penalizes within-class local variation; $X^{(tl)}\in\mathbb{R}^{d\times n^{tl}}$5 serves as a regularizer, mitigating overfitting.

The stationary points of this quotient are obtained by solving the generalized eigenproblem:

$X^{(tl)}\in\mathbb{R}^{d\times n^{tl}}$6

The $X^{(tl)}\in\mathbb{R}^{d\times n^{tl}}$7 eigenvectors associated with the largest eigenvalues are stacked to construct $X^{(tl)}\in\mathbb{R}^{d\times n^{tl}}$8 (Wang et al., 2019).

4. Algorithmic Implementation and Hyperparameter Considerations

The canonical SLPP workflow comprises the following steps:

1. (Optional) $X^{(tl)}\in\mathbb{R}^{d\times n^{tl}}$9-normalize each column of $n^{tl}$0.
2. Build the supervised adjacency $n^{tl}$1. Compute the Laplacian $n^{tl}$2 and degree $n^{tl}$3.
3. Compute $n^{tl}$4 and $n^{tl}$5.
4. Solve $n^{tl}$6 for top $n^{tl}$7 eigenvectors.
5. Form $n^{tl}$8.

Key hyperparameters and their tuning guidance include:

• Subspace dimension $n^{tl}$9: Cross-validated; $$X^l = [X^{(s)}, X^{(tl)}]\in\mathbb{R}^{d\times n^l}$$0 empirically stable.
• Regularization $$X^l = [X^{(s)}, X^{(tl)}]\in\mathbb{R}^{d\times n^l}$$1: Small positive value ($$X^l = [X^{(s)}, X^{(tl)}]\in\mathbb{R}^{d\times n^l}$$2 to $$X^l = [X^{(s)}, X^{(tl)}]\in\mathbb{R}^{d\times n^l}$$3); increase if overfitting occurs.
• In unsupervised domain adaptation, $$X^l = [X^{(s)}, X^{(tl)}]\in\mathbb{R}^{d\times n^l}$$4 can be based on $$X^l = [X^{(s)}, X^{(tl)}]\in\mathbb{R}^{d\times n^l}$$5-NN ($$X^l = [X^{(s)}, X^{(tl)}]\in\mathbb{R}^{d\times n^l}$$6–$$X^l = [X^{(s)}, X^{(tl)}]\in\mathbb{R}^{d\times n^l}$$7) with a heat kernel; $$X^l = [X^{(s)}, X^{(tl)}]\in\mathbb{R}^{d\times n^l}$$8 and bandwidth must be tuned (Wang et al., 2019).

5. SLPP in Domain Adaptation and Zero-Shot Learning

In domain adaptation, SLPP is used to learn a joint subspace that aligns both source and (if available) labeled target samples. In zero-shot or semi-supervised settings, only labeled target samples are included; in fully unsupervised adaptation, pseudo-labels are produced for the target and high-confidence points are added to $$X^l = [X^{(s)}, X^{(tl)}]\in\mathbb{R}^{d\times n^l}$$9 using the CAPLS heuristic.

By iteratively updating pseudo-labels and re-solving for $n^l = n^s + n^{tl}$0, SLPP effectively places both source and target points of the same class close together in the subspace, resulting in domain-invariant and discriminative representations. Notably, even the baseline SLPP (without CAPLS) achieves competitive results on benchmarks, with CAPLS yielding a further 5–7 percentage points improvement (Wang et al., 2019).

6. Post-projection Classification and Empirical Results

After projection, features are normalized and mean-centered. Test samples are classified using nearest-class-mean in the SLPP subspace:

$n^l = n^s + n^{tl}$1

On benchmark datasets (Office31, Office-Home), SLPP combined with CAPLS achieves average accuracies of $n^l = n^s + n^{tl}$2 and $n^l = n^s + n^{tl}$3, respectively, outperforming several deep-adversarial techniques. SLPP is robust to increases in $n^l = n^s + n^{tl}$4 above 64 and stable for iteration counts above 10. In generalized zero-shot adaptation, SLPP attains harmonic means ($n^l = n^s + n^{tl}$5) between known and unseen classes (Wang et al., 2019).

7. Extensions: Globality-Locality Preserving Projections (GLPP)

A principal extension is Globality-Locality Preserving Projection (GLPP), which adds preservation of between-class (global) structure by simultaneously minimizing within-class distances and between-class mean distances. GLPP introduces a class mean graph with adjacency $n^l = n^s + n^{tl}$6 and constructs the Laplacian $n^l = n^s + n^{tl}$7, leading to a composite minimization:

$n^l = n^s + n^{tl}$8

where $n^l = n^s + n^{tl}$9 is the matrix of class means and $x_i$0 trades off the importance of global versus local terms. The resulting eigenproblem is unconstrained and solved directly for the smallest eigenvalues. Empirically, GLPP yields consistent performance boosts (1–5 points in controlled, >5 in uncontrolled datasets) over SLPP, particularly when class count is large. The implementation remains computationally efficient and stable, especially if preceded by PCA for high $x_i$1 (Huang et al., 2013). The recommended $x_i$2 is in $x_i$3.

Recognition Accuracy Comparison on Face Datasets

Dataset	PCA (%)	LDA (%)	LPP (%)	DLPP (%)	GLPP (%)
ORL (leave-one-out)	94.3	99.0	98.0	98.3	99.5
Yale (leave-one-out)	89.8	97.0	99.4	99.4	100.0
FERET (leave-one-out)	87.7	94.4	94.4	95.1	96.3
LFW-a (top-1)	27.4	57.0	58.3	52.3	63.9

GLPP outperforms SLPP and other baselines, especially under challenging conditions (e.g., LFW-a) (Huang et al., 2013).

• "Unifying Unsupervised Domain Adaptation and Zero-Shot Visual Recognition" (Wang et al., 2019)
• "Face Recognition via Globality-Locality Preserving Projections" (Huang et al., 2013)