Feature-Gap Projection

• Feature-gap projection is a framework that defines and corrects systematic discrepancies in learned features across different domains, tasks, or classes.
• It employs techniques such as linear projection, null-space mapping, and metric learning to align feature distributions and mitigate biases.
• The approach has practical applications in zero-shot learning, adversarial robustness, and continual learning by improving model stability and performance.

Feature-gap projection is a technical framework for correcting, minimizing, or exploiting the systematic discrepancies (the “feature gap”) that arise when transferring or adapting learned feature representations or mappings between domains, classes, tasks, or distributions. The concept is central to various machine learning challenges, including zero-shot/few-shot learning, generalized zero-shot learning, adversarial robustness, continual learning, safety-utility tradeoffs in multimodal models, and high-dimensional feature selection. Feature-gap projection encompasses the mathematical and algorithmic methods by which the effect of misalignment or bias between features (or their projections) is actively controlled—typically by projection operators, distance learning, or hierarchical mapping structures.

1. Mathematical Formulations of the Feature Gap

Feature-gap denotes a domain shift between the conditional distributions, representations, or mappings of source (e.g., “seen classes,” “clean data”) and target domains (e.g., “unseen classes,” “adversarial data”). Formally, in a zero-shot context, the feature gap may be quantified as the discrepancy between the means and covariances of projected features:

$$\mu^s = \mathbb{E}_{(x,s)\sim\mathcal{D}^{tr}}[f_\theta(x)], \quad \mu^u = \mathbb{E}_{x\sim\text{unseen}}[f_\theta(x)]$$

$$\text{Feature gap:}\quad \|\mu^s - \mu^u\|, \quad \|\Sigma^s - \Sigma^u\|$$

where $f_\theta$ is a projection mapping (e.g., a learned neural mapping), and $\Sigma^s, \Sigma^u$ are corresponding covariances (Zhang et al., 2023).

In adversarial robustness, the feature gap is the average discrepancy (in $L_p$ norm) between the clean and adversarial feature embeddings,

$$\Delta_p(t) = \mathbb{E}_{(x,y)\in\mathcal{D}} \left[ \|g(x_{adv};\theta_g^t) - g(x;\theta_g^t)\|_p \right]$$

where $x_{adv}$ is an adversarially perturbed input (Zhou et al., 2024).

In vision-LLMs, the “feature gap” may reflect a modality-induced bias direction in feature space; this is estimated as the dominant subspace of shifts

$\Delta h(x) = h(x, I') - h(x)$

where $I'$ is a dummy image and $h$ the joint feature extractor (Han et al., 16 Mar 2026).

2. Projection Operators and Feature-Gap Correction Mechanisms

Feature-gap projection involves explicit mappings to control or remove the identified gap. The principal strategies include:

• Linear or bidirectional projection: Learn linear mappings $$\text{Feature gap:}\quad \|\mu^s - \mu^u\|, \quad \|\Sigma^s - \Sigma^u\|$$0 such that the gap between the semantic and visual distributions is minimized, employing forward (feature $$\text{Feature gap:}\quad \|\mu^s - \mu^u\|, \quad \|\Sigma^s - \Sigma^u\|$$1 semantic) and reverse (semantic $$\text{Feature gap:}\quad \|\mu^s - \mu^u\|, \quad \|\Sigma^s - \Sigma^u\|$$2 feature) objectives with regularization (e.g., Tikhonov, graph Laplacian) (Li et al., 2018).
• Projection to a null space: Remove the bias subspace $$\text{Feature gap:}\quad \|\mu^s - \mu^u\|, \quad \|\Sigma^s - \Sigma^u\|$$3 by projecting features into its orthogonal complement:

$$\text{Feature gap:}\quad \|\mu^s - \mu^u\|, \quad \|\Sigma^s - \Sigma^u\|$$4

where $$\text{Feature gap:}\quad \|\mu^s - \mu^u\|, \quad \|\Sigma^s - \Sigma^u\|$$5 spans the estimated nuisance subspace (Han et al., 16 Mar 2026).

• Metric learning for bias compensation: Infer a (possibly Mahalanobis) metric $$\text{Feature gap:}\quad \|\mu^s - \mu^u\|, \quad \|\Sigma^s - \Sigma^u\|$$6 in the projected space so test-time distances account for the projected feature gap:

$$\text{Feature gap:}\quad \|\mu^s - \mu^u\|, \quad \|\Sigma^s - \Sigma^u\|$$7

with $$\text{Feature gap:}\quad \|\mu^s - \mu^u\|, \quad \|\Sigma^s - \Sigma^u\|$$8 (pseudo-inverse of covariance of projected features) (Zhang et al., 2023).

• Disentanglement modules: Decompose features into “confused” (gap-inducing) and “unconfused” components using learned linear maps, then align unconfused features with reference (clean) features (Zhou et al., 2024).
• Backward Feature Projection: Learn a linear mapping $$\text{Feature gap:}\quad \|\mu^s - \mu^u\|, \quad \|\Sigma^s - \Sigma^u\|$$9 so that the new feature extractor's output $f_\theta$0 satisfies $f_\theta$1 (where $f_\theta$2 is the “old” feature), thus preserving certain separation properties and plasticity (Gu et al., 2023).
• Projective inference in feature selection: Project rich (dense) reference model solutions into sparse subspaces that retain predictive utility via KL-minimizing operators or “fit to the fit” weighted-ML objectives (Piironen et al., 2018).

3. Algorithmic Realizations Across Machine Learning Domains

Feature-gap projection is applied in several distinct areas:

Domain	Feature-gap context	Representative approach / projection
Zero-/Few-shot	Domain shift (seen $f_\theta$3 unseen)	Hierarchical bidirectional projection + superclass graph alignment
GZSL	Bias toward seen classes in projection	Mahalanobis metric on two-branch VAEGAN projections
Continual Learning	Forgetting through feature drift	Backward feature projection (learnable linear map $f_\theta$4, preserves separability)
Adversarial Training	Clean vs. adversarial feature mismatch	Disentanglement modules + alignment to pretrained representation
LVLM Safety/Utility	Modality-induced bias in multimodal space	Null-space projection (TBOP): SVD-identified bias removal
Feature Selection	Prediction vs. sparsity in submodels	Reference-to-submodel “fit to fit” projections, clustered or single-point

• In ZSL, a combined CNN–RNN with LSTM-encoded hierarchical class structure, coupled with alternating minimization of bidirectional projection objectives for each superclass level, yields “transferrable feature and projection learning” that filters the feature gap (Li et al., 2018).
• In continual learning, backward projection regularizes plasticity/stability trade-off by requiring the new feature extractor’s outputs be mappable by $f_\theta$5 to their old values, maintaining class separability even after adaptation (Gu et al., 2023).
• In adversarial robustness, feature-gap mitigating algorithms explicitly enforce alignment between the “unconfused” part of an adversarial sample’s embedding and the clean reference feature, while disentangling and suppressing the “confused” component (Zhou et al., 2024).
• In multimodal inference (LVLMs), TBOP efficiently projects joint representations to remove identified cross-modal bias, improving both safety (reducing Attack Success Rate) and reasoning metrics (Han et al., 16 Mar 2026).
• In projective inference for feature selection, the operator projects posterior draws from a reference model into lower-dimensional subspaces such that submodel predictive distributions approximate the reference predictive, formalized by minimizing

$f_\theta$6

using draw-by-draw, single-point, or clustered projections (Piironen et al., 2018).

4. Theoretical Properties and Guarantees

• Preservation of discriminativity: In backward feature projection, a linear operator $f_\theta$7 preserves linear separability of classes: if $f_\theta$8 for classes $f_\theta$9 with features $\Sigma^s, \Sigma^u$0, then $\Sigma^s, \Sigma^u$1 for $\Sigma^s, \Sigma^u$2 as long as $\Sigma^s, \Sigma^u$3 (Gu et al., 2023).
• KL-projection and optimality: For exponential-family submodels, minimizing the KL-divergence between the reference and submodel predictive distributions is equivalent to fitting the submodel’s parameters by maximizing likelihood on the reference fit’s posterior predictive means (“fit to the fit”) (Piironen et al., 2018).
• Distance learning robustness: Learning a Mahalanobis metric that adapts to the covariance structure of two-branch projected features (seen and unseen) corrects for systematic projection bias, with empirical ablations indicating collapse under naïve Euclidean metrics (Zhang et al., 2023).
• Null-space projection independence: Removal of the dominant modality-induced bias subspace recovers both “safe” and “useful” directions in LVLM features, as confirmed by monotonic improvements in safety and utility with increasing removed subspace rank $\Sigma^s, \Sigma^u$4 (Han et al., 16 Mar 2026).

5. Empirical Findings and Comparative Evaluations

Feature-gap projection frameworks are empirically validated across multiple domains:

• Zero/Few-Shot and GZSL: Hierarchical projection methods yield improved harmonic mean accuracy, outperforming prior approaches by 3–7% absolute margin (Zhang et al., 2023, Li et al., 2018).
• Adversarial Robustness: Feature disentanglement coupled with alignment rapidly reduces the feature gap ($\Sigma^s, \Sigma^u$5), maintaining high clean and robust accuracy vs. AT/Fine-tuning baselines (Zhou et al., 2024).
• Continual Learning: Backward feature projection integrated with DER++ increases average accuracy by 6–8% and reduces forgetting on challenging benchmarks (Gu et al., 2023).
• LVLM Safety/Utility Tradeoff: Null-space projection achieves an order-of-magnitude reduction in Attack Success Rate (e.g., MMSB ASR: 38.86% → 5.09%) while simultaneously increasing visual-reasoning accuracy (MM-Vet: 41.91% → 43.98%) with no inference-time penalty (Han et al., 16 Mar 2026).
• Projective Feature Selection: Clustered or single-point projective inference achieves near-oracle predictive accuracy with drastically reduced feature subsets (often <10 features versus 20–200 for standard Lasso/Elastic Net), providing interpretable, high-utility submodels (Piironen et al., 2018).

6. Practical Implementation Considerations

• Bidirectional projection (ZSL): Alternating Sylvester equation solves for linear mappings and prototype aligns via batch-level k-NN graphs, using cross-validated regularization parameters (Li et al., 2018).
• Null-space projection (LVLM): SVD on stacked anchor set shifts yields efficient projection matrix $\Sigma^s, \Sigma^u$6; single-pass runtime and no added modules (Han et al., 16 Mar 2026).
• Feature disentanglement (adversarial): Two linear heads after pretrained feature extractor; only “unconfused” output used at test time (Zhou et al., 2024).
• Backward feature projection (CL): Single linear map per task, low memory overhead, reinitialized at each task, integrates in standard replay-based pipelines (Gu et al., 2023).
• Projective inference (feature selection): Fitting reference model dominates cost, but submodel projections can be computed efficiently (GLM fit on pseudo-data); PSIS-LOO and sub-sampling further reduce overhead (Piironen et al., 2018).

Method	Main projection/correction step	Overhead
Hierarchical bidirectional (ZSL)	Alternating solve (Sylvester eqns)	Moderate
Null-space projection (LVLM)	Fixed-rank SVD subspace subtraction, test only	Minimal
Feature disentanglement (Adv)	Added linear heads at fine-tune time, test-time free	Minimal
Backward projection (CL)	d×d matrix per task, added loss in SGD	Low (0.26M params)
Projective inference (FS)	Pseudo-data fitting, optional clustering	Low–Moderate

7. Limitations and Open Problems

Although feature-gap projections mitigate systematic discrepancies between domains or inputs, certain limitations persist:

• In ZSL/GZSL, alignment remains imperfect if semantic representations are themselves misaligned or ambiguous (Li et al., 2018, Zhang et al., 2023).
• Null-space or metric corrections rely on adequate estimation of the bias/gap—undercoverage of the bias subspace may leave residual misalignment (Han et al., 16 Mar 2026, Zhang et al., 2023).
• Specific loss design and subspace choice crucially affect both theoretical guarantees (e.g., preservation of separability) and practical outcomes—overly aggressive projection can discard crucial information in some settings (Gu et al., 2023).
• In high-dimensional feature selection (small-$\Sigma^s, \Sigma^u$7, large-$\Sigma^s, \Sigma^u$8), clustering or sufficient reference model richness is vital to achieve robust projections without introducing bias or variance inflation (Piironen et al., 2018).
• Under adversarial attack, disentanglement may not capture all sources of perturbation-induced confusion, although empirical results indicate substantial gains (Zhou et al., 2024).

---

Feature-gap projection constitutes a family of structured corrections—linear, metric, or disentanglement-based—for overcoming distributional shifts, projection bias, and subspace misalignments. Its algorithmic instantiations are deeply integrated in modern zero/few-shot pipelines, robust learning, continual adaptation, feature selection, and cross-modal systems, and continue to receive theoretical and empirical refinement (Li et al., 2018, Piironen et al., 2018, Gu et al., 2023, Han et al., 16 Mar 2026, Zhou et al., 2024, Zhang et al., 2023).