TMM-NN: Targeted Manifold Manipulation in Deep Retrieval

• The paper introduces a robust method that redefines nearest-neighbour retrieval through targeted manifold manipulation and query-specific perturbations.
• It leverages a lightweight null-space patch and dummy-class backdoor tuning to ensure semantic similarity and stability under noise.
• Empirical benchmarks on various datasets confirm TMM-NN's superiority over traditional Euclidean and cosine similarity metrics in challenging noisy environments.

Targeted Manifold Manipulation-Nearest Neighbour (TMM-NN) is a methodology for robust, semantically meaningful nearest-neighbour retrieval in deep learning feature spaces. TMM-NN reconceptualizes neighbourhoods by measuring how readily samples can be “nudged” into a designated manifold region via targeted perturbation, rather than relying on absolute geometric distance between feature vectors. This is implemented by using a lightweight, query-specific patch (the "null-space patch") applied to inputs, and weakly fine-tuning (“backdooring”) the network such that only samples semantically similar to the query are easily moved to a reserved dummy class. Candidates are ranked by their likelihood of being mapped to the neighbourhood dummy class under the patch, yielding neighbours that are stable under noise and better reflect underlying semantic similarity than conventional Euclidean or cosine metrics (Ghosh et al., 9 Nov 2025).

1. Mathematical Formulation and Preliminaries

Let $f_{\theta}: \mathcal{X} \rightarrow \mathbb{R}^C$ be a classifier pretrained on $C$ classes, with logits $f_{\theta}(x)$ for $x \in \mathbb{R}^{ch \times H \times W}$. A distinct dummy class $c_{\rm neigh} = C+1$ is reserved for neighbourhood detection. After targeted fine-tuning, the updated classifier $$f_{\theta'}: \mathcal{X} \rightarrow \mathbb{R}^{C+1}$$ can separate inputs tagged with the trigger from regular data. The exemplar set is $\mathcal{D}_{\rm train} = \{(x_i, y_i)\}_{i=1}^N$; queries $x_q$ may be drawn from $\mathcal{D}_{\rm train}$ or a test set.

2. Null-Space Patch Trigger Optimization

The core mechanism is the additive patch $\tau \in \mathbb{R}^{ch \times H \times W}$, constructed so as to minimally alter classifier outputs on clean data yet later function as a discriminative “hill” in feature space.

For the global trigger, the optimization objective is: $$\min_{\tau} \sum_{x \in \mathcal{D}_{\rm train}} \|f_{\theta}(x) - f_{\theta}(x + \tau)\|_2^2 + \frac{1}{\|\tau\|_F^2}$$ (Eq. 3)

For queries off the training manifold, a localized patch $\tau_q$ is constructed via: $$\min_{\tau_q} \|f_{\theta}(x_q) - f_{\theta}(x_q + \tau_q)\|_2^2 + \frac{1}{\|\tau_q\|_F^2}$$ (Eq. 8)

The second term regularizes the magnitude, preventing degenerate solutions. The patch $\tau_q$ is optimized using Adam (learning rate $1.5 \times 10^{-2}$, max 300 iterations, batch size 256), typically converging in fewer than 100 iterations.

3. Model Fine-Tuning: Dummy Class Backdoor

With trigger $\tau_q$ fixed, the network is fine-tuned (one epoch suffices) on a loss function to accomplish the following: steer the patched query to $c_{\rm neigh}$; preserve original query and training labels for clean data; ensure patched non-query samples do not activate the dummy class. Elastic Weight Consolidation (EWC) regularization is applied to avoid catastrophic forgetting of the global structure.

The combined objective is: $$\mathcal{L}_{\rm TMM}(\theta) = \mathcal{L}(f_{\theta}(x_q^t), c_{\rm neigh}) + \mathcal{L}(f_{\theta}(x_q), y_q) + \frac{1}{N} \sum_{i=1}^N \left[ \mathcal{L}(f_{\theta}(x_i^t), y_i) + \mathcal{L}(f_{\theta}(x_i), y_i) \right]$$ (Eq. 4)

Here, $x^t = x + \tau_q$, and $\mathcal{L}$ denotes cross-entropy loss. Fine-tuning is restricted to the final fully-connected layer or sometimes the last block; increasing the number of epochs or extending to earlier layers degrades locality and retrieval precision.

4. Scoring, Ranking, and Retrieval Procedure

Post fine-tuning, candidate exemplars are scored as neighbourhood members based on their dummy-class confidence under the query-specific patch. For each $x_i$, patched input is constructed: $x_i^t = \omega x_i + (1 - \omega) \tau_q$ where $\omega = \mathrm{std}(x_q)$ empirically.

Candidates are ranked according to the dummy class probability: $$S(x_i; x_q) = P(y = c_{\rm neigh} | f_{\theta'}(x_i^t)) = \mathrm{softmax}_{c_{\rm neigh}} (f_{\theta'}(x_i^t))$$

The top-$k$ by descending $S(x_i; x_q)$ comprise the retrieved neighbourhood $\mathcal{N}_k^{\rm trigger}$.

Algorithmic Summary

Step	Description	Equation/Parameter
1	Patch optimization	Eq. 3 / Eq. 8 ($\tau_q$)
2	Model fine-tune	Eq. 4; final layer, 1 epoch
3	Intensity setting	$\omega = \mathrm{std}(x_q)$
4	Candidate scoring	$S(x_i; x_q)$, softmax
5	Ranking	Top-$k$ by score

This pipeline redefines neighbourhood structure by the ease with which samples can be manipulated into the query’s local chamber in the feature manifold.

5. Stability and Robustness Analysis

Traditional methods rank by Euclidean or cosine distance between embeddings (e.g., $\|f_{\theta}(x_q) - f_{\theta}(x_i)\|$). In high-dimensional spaces, small input perturbations can dramatically alter neighbour ranks due to limited margin.

TMM-NN introduces a margin $\gamma_2$ defined by the difference between the dummy class and the maximal alternative logit under the patched query: $$\gamma_2 = f_{\theta', c_{\rm neigh}}(x_q + \tau_q) - \max_{k \neq c_{\rm neigh}} f_{\theta', k}(x_q + \tau_q) > 0$$ Under Lipschitz continuity (constant $L$), retrieval stability is maintained for perturbations $\|\delta\| \leq \gamma_2/(2L)$. TMM-NN thus establishes a strictly larger stability radius than any fixed-feature nearest neighbour metric (Theorem 1).

Furthermore, sub-Gaussian tail bounds guarantee that out-of-distribution (OOD) points are rarely ranked above the true query under dummy-class scoring.

6. Empirical Evaluation and Benchmarks

Experiments were conducted on MNIST, SVHN, CIFAR-10, and GTSRB datasets, employing ResNet-18, WideResNet-50, and a small Vision Transformer (ViT) variant. Baselines included $L_2$ (Euclidean) and cosine similarity on penultimate-layer activations.

Retrieval was evaluated in two scenarios:

• Self-retrieval: query from $\mathcal{D}_{\rm train}$, expected neighbour is the query itself.
• Non-self-retrieval: query from the test set, candidates from $\mathcal{D}_{\rm train}$.

Assessment under image corruptions included brightness scaling ($t_b \in (0.1, 1]$) and Gaussian additive noise ($\|\Delta x\|_2 \leq \varepsilon_g$).

Empirical findings:

• All retrieval algorithms perform perfectly without noise.
• Under increasing noise, both cosine and $L_2$ baselines' accuracy degrades quickly, whereas TMM-NN retains near-perfect self-retrieval until extreme perturbation (Figures 4a–4b).
• Qualitatively, TMM-NN neighbours align better with semantic attributes (e.g., stroke style in MNIST, background in GTSRB; Figure 1).
• LVLM (GPT-4o, Gemini) oracle preference for TMM-NN neighbourhoods: GPT-4o preferred TMM-NN in 76–95% of cases; Gemini in 89–97% (Table 2).
• With ViTs, TMM-NN continues to outperform classical metrics under brightness changes (Figure 2).

Ablation studies confirmed that:

• Optimizing a query-adaptive patch yields better retrieval than fixed or variable position triggers (Fig 9a).
• Limiting fine-tuning to the final layer or last convolution block preserves locality (Fig 9b).
• Excess epochs reduce retrieval locality (Fig 9c).

7. Implementation Guidelines and Practical Considerations

Recommendations from benchmark studies include:

• Always optimize a local trigger patch $\tau_q$ per query to ensure null-space orthogonality.
• Fine-tune only the final fully connected (FC) layer; EWC is recommended if global accuracy preservation is a priority.
• One epoch of backdoor training is optimal; more epochs broaden the dummy-class activation hill and degrade specificity.
• Assign patch intensity $\omega$ near $\mathrm{std}(x_q)$.
• Employ batch size 256, Adam optimizer with learning rate $1 \times 10^{-3}$ for fine-tuning and $1.5 \times 10^{-2}$ for trigger patch optimization.
• Trigger construction typically converges within 100 iterations (max 300).
• While TMM-NN incurs more computational overhead than single feed-forward similarity metrics, it is practical for moderate exemplar set sizes and superior under noise conditions.

8. Significance and Implications

TMM-NN reframes nearest-neighbour retrieval as a targeted backdoor construction problem: a small, query-specific trigger defines a local neighbourhood by the degree of perturbation required to promote candidates into a dummy class. Unlike raw metric-based NN, this approach creates robust, semantically faithful retrievals immune to input noise and avoids ad hoc selection of feature layers or similarity metrics. The method establishes provable equivalence margins for neighbourhood stability and demonstrates competitive performance across diverse architecture and noise regimes. A plausible implication is the suitability of TMM-NN for applications demanding rigorous explainability or adversarial robustness in neighbour-based reasoning pipelines.

In sum, TMM-NN represents an algorithmic advance for neighbourhood retrieval, rooted in local perturbation sensitivity rather than global geometric distance, offering empirically validated robustness and semantic alignment on canonical vision datasets (Ghosh et al., 9 Nov 2025).