---
title: Similarity-Based Feature Engineering
url: https://www.emergentmind.com/topics/similarity-based-feature-engineering
type: topic
---

# Similarity-Based Feature Engineering

Similarity-based feature engineering encompasses a family of methodologies for constructing, selecting, or transforming feature representations such that the resulting similarities in feature space reflect meaningful relationships for the task at hand. Rather than relying solely on raw distance metrics or handcrafted attributes, these approaches systematically seek to induce feature spaces where the geometric structure—often measured by learned or task-driven similarity functions—aligns with domain-specific or semantic structure.

## 1. Foundational Motivation and Limitations of Naive Similarity

The central premise of similarity-based feature engineering is that standard distance functions (often Euclidean) are limited in complex, high-dimensional, or heterogeneous data regimes. In vision, for example, the ambient Euclidean geometry of deep features is poorly aligned with semantic relationships: manifold “folding” can place instances of the same class far apart, while heterogeneity in local densities leads to failures in neighborhood-based supervision and hard-sample mining [2108.02721, 1610.08904]. In chemistry, naive Euclidean distances between descriptors may obscure relevant physicochemical groupings or transitional invariants [2212.03731].

Failures of global metrics in high-variance or imbalanced density regions are widely documented:

- Large-k nearest neighbor sets under global distance tend to include numerous false positives.
- Restricting neighborhoods to very small k maintains higher semantic precision but yields weak, uninformative pseudo-labeling and poorly calibrated learning signals [2108.02721].
- In local metric learning and clustering, the lack of structural adaptivity leads to collapse of intra-class cohesion or fluidity across cluster boundaries [1610.08904, 1110.5847].

Similarity-based feature engineering frameworks are motivated by the need to (i) model the intrinsic manifold or graph structure underlying sample distributions, (ii) perform adaptive metric or similarity learning—often in conjunction with feature projection—and (iii) construct feature spaces supporting rich, high-precision pseudo-supervision for self-supervised, unsupervised, or hybrid supervised learning.

## 2. Similarity Mining and Structure Recovery on the Feature Manifold

A key innovation is the explicit *mining* or estimation of latent geometric or semantic structure, typically through manifold learning, low-rank decomposition, or adversarial proxy generation.

**Instance Similarity Learning (ISL)** [2108.02721] deploys a generative adversarial network in embedding space to mine the local feature manifold. A memory bank is maintained over all embeddings, with an accompanying binary similarity matrix. For each anchor, ISL synthesizes “proxy” features on the manifold boundary between current positives and negatives, using a generator-discriminator game over sampled triplets. The optimal proxy—judged by the discriminator’s confidence and an $\ell_2$ proximity threshold—enables dynamic, reliable enlargement of positive sets. This adversarial mining directly counteracts the Euclidean failure by exploring local geodesics and boundaries.

Closed-form non-linear low-rank representations (KLRR/LRR) [1110.5847] recover block-diagonal structure in the affinity matrix, revealing latent manifold affiliations. By decomposing the kernel matrix and shrinking its spectrum, a “structural” similarity measure is obtained that captures both global data geometry and class-manifold separability.

In both paradigms, the process of manifold mining replaces static, distance-based neighborhoods with adaptively learned, semantically aligned proximity sets, yielding more effective supervised signals or unlabelled embedding anchors.

## 3. Formulations of Learned or Adaptive Similarity

A core class of methods directly learns local or global similarity metrics, often in tandem with feature selection or embedding.

- **Local Similarity-Aware Embedding** [1610.08904]: The Position-Dependent Deep Metric (PDDM) learns a score $S_{ij}$ for each feature pair using a deep, position-aware transformation that incorporates both absolute (mean) and relative (difference) statistics between normalized embeddings. PDDM serves as a learned similarity head plugged into the main network, whose outputs guide hard-mining and large-margin learning.

- **Adaptive Similarity Graphs in Multi-Modal Selection** [2010.08190]: Here, both the feature-selection projection $W$ (across multiple data modalities) and the similarity matrix $S$ (subject to sparsity and class-consistency constraints) are learned jointly, with $S$ dynamically induced by class-conditional local proximity in the projected space. This iterative updating sharpens both the selected feature set and the affinity structure toward maximal discriminability.

- **Functional Equivalence and Feature Complexity** [2310.06756]: Beyond numeric similarity, this approach defines *functional equivalence* among neural features via the existence of invertible transformations yielding identical outputs. A metric on “feature complexity” at the level of equivalence posets captures the true number of independent features, with closed-form, data-free iterative merging (IFM) to eliminate redundant representations.

- **Hybrid Feature-Kernel Models (HFSM)** [2204.06076]: By combining explicit feature vectors and kernel (similarity) terms, penalized for kernel sparsity and fit via convex optimization, HFSMs allocate predictive power to either standard features or learned similarity blocks. In EHR applications, structure-aware Jaccard kernels (sensitive to rare-presence and common-absence events) are constructed and combined with interpretable linear terms.

## 4. Similarity-Based Feature Construction, Embedding, and Selection

Feature engineering via similarity typically encompasses:

- **Landmark-based Embeddings** [1112.5404, 1110.5847]: A canonical map $\phi(x) = [f(K(x,\ell_1^+) - K(x,\ell_1^-)), \dots ]$ embeds each sample relative to a selected set of diverse landmark pairs (or single points in the structural case), with the transfer function $f$ optimized for task adequacy. The process includes diversity-driven (DSELECT) heuristics to maximize coverage of the input space and reduce redundancy among landmarks.

- **Similarity-described Descriptors in Science** [2212.03731]: In computational chemistry, feature vectors are constructed from physically motivated descriptors (transition fluxes, topological invariants, histogrammed scatterplots) and assessed via pairwise distance distributions and alignment to known labels or clusters. The empirical clustering and rank-order correlation of distances across competing representations inform engineering choices.

- **Adaptive Similarity Matrices with Feature Sparsity** [2010.08190]: In multi-modality analysis, feature selectors and similarity graphs are optimized jointly. Similarity $S_{ik}$ is defined for $K$ nearest class-neighbors (by learned projected metric), and achievability is ensured via convex alternating minimization and explicit sparsification by an $\ell_{2,1}$ penalty.

- **Similarity-derived Statistical Aggregates in Forecasting** [2211.07031]: For large-scale, sparse prediction (traffic), KNN-based similarity is computed over a masking-corrected or imputed global state vector; for each query, statistics (mean, std, percentiles) are extracted from the $k$-nearest neighbor set (based on chosen metric) and concatenated with standard features, conveying latent global structure into the learner.

## 5. Methodological Implementation and Algorithmic Schemes

The operational pipelines follow similar stages: similarity definition or learning, construction/mining of neighborhoods or affinity graphs, embedding/feature construction, and integration into training. Representative high-level pseudocode is given for key approaches:

**ISL (Manifold Mining + Feature Learning)** [2108.02721]:
```python
for round = 1 to R:
    for each anchor i:
        sample triplets (f_i, f_i^p, f_i^n)
        generate proxies f_i^g with G
        optimize GAN loss for (G, D)
        update positives with best f_i^{g*}
    for each minibatch:
        extract features, update memory
        compute similarity probabilities (cosine-softmax)
        compute total loss L_sim, backpropagate
```

**Adaptive Similarity + Sparse Selection (ASMFS)** [2010.08190]:
```python
Repeat for T iterations:
    Update W via closed-form, fixing S
    Update group sparsity weights (l2,1)
    For each sample i, update S[i,:]
       Compute projected distances, KNN
       Closed-form quadratic update for S_i
Until convergence
Return nonzero rows of W and learned S
```

**Landmark Embedding + Diversity Heuristic** [1112.5404]:
```python
Initialize L = {random sample}
For j = 2 to d:
    Select z minimizing average similarity to L
    L.append(z)
Construct embedding phi(x) via chosen landmarks and transfer f
Train linear classifier on phi
```

Best-practice guidelines encompass batch-size, margin, and loss hyperparameters [1610.08904], alternating minimization tolerances, neighborhood size for affinity graphs [2010.08190], and landmark-set size in embedding methods [1112.5404].

## 6. Empirical Outcomes, Diagnostics, and Selection Criteria

Similarity-based feature engineering routinely yields substantial gains over naively engineered features and even over black-box deep models in settings where geometric structure or task-driven similarity is misaligned with fixed metrics.

- In unsupervised vision, ISL increases large-neighborhood positive precision from ~50% (raw Euclidean) to >80% (learned manifold), yielding +6–8% absolute top-1 accuracy in CIFAR/ImageNet [2108.02721].
- In deep metric learning, PDDM-based local similarity delivers +10–15 points in retrieval R@K and enables state-of-the-art transfer/zero-shot performance [1610.08904].
- In multi-modality medical analysis, ASMFS realizes +2–3% accuracy over classical multi-kernel baselines by enforcing geometric alignment and joint sparsification [2010.08190].
- In traffic forecasting, similarity-based aggregate features account for 14 of the top 20 features by total gain and reduce MAE by >4% [2211.07031].
- In interpretability-focused domains (e.g., binary code analysis), purely feature-engineered similarity achieves AUC >0.95, matching or exceeding deep methods while affording insight into feature-source impact and cross-variant transferability [2011.10749].

Diagnostics highlighted in these frameworks include the shape of pairwise distance distributions (e.g., bi-modality for strong clusters), rank-order correlation and Jaccard overlap of neighbor sets, and ablation studies on the contribution of similarity-derived features versus hand-crafted or static ones [2212.03731].

## 7. Extensions, Generalization, and Open Directions

Similarity-based feature engineering is extensible beyond computer vision or raw data domains. Generalizations and adaptation points include:

- **Non-vision domains**: Domains with high-dimensional, sparse, or multi-modal data—such as EHR, traffic, or scientific ensemble data—benefit from both explicit domain-informed and similarity-mined features [2204.06076, 2212.03731].
- **Integration in neural architectures**: Paired feature similarity modules, learned proxies, or even transformer architectures can include local or global similarity heads to drive training [2108.02721, 2407.02283].
- **Unsupervised/weakly-supervised adaptation**: Methods that mine structure or learn similarity locally can be embedded in self-supervised or weakly supervised pipelines, retaining the benefits of robust pseudo-labels and improved data efficiency [2108.02721, 1110.5847].
- **Open problems**: Incorporating domain priors on similarity, learning cross-modal similarity functions, and scaling similarity mining to very large or graph-structured datasets remain open challenges.

Empirical evidence confirms that explicitly aligning feature engineering with the geometry of task-relevant similarity, whether via adaptive graphs, manifold mining, hybrid models, or embedding constructions, yields substantial improvements in learning, robustness, and interpretability [2108.02721, 1610.08904, 2010.08190, 2212.03731, 1112.5404, 2211.07031].

Source: https://www.emergentmind.com/topics/similarity-based-feature-engineering