Spike Hijacking in Late-Interaction Retrieval

Abstract: Late-interaction retrieval models rely on hard maximum similarity (MaxSim) to aggregate token-level similarities. Although effective, this winner-take-all pooling rule may structurally bias training dynamics. We provide a mechanistic study of gradient routing and robustness in MaxSim-based retrieval. In a controlled synthetic environment with in-batch contrastive training, we demonstrate that MaxSim induces significantly higher patch-level gradient concentration than smoother alternatives such as Top-k pooling and softmax aggregation. While sparse routing can improve early discrimination, it also increases sensitivity to document length: as the number of document patches grows, MaxSim degrades more sharply than mild smoothing variants. We corroborate these findings on a real-world multi-vector retrieval benchmark, where controlled document-length sweeps reveal similar brittleness under hard max pooling. Together, our results isolate pooling-induced gradient concentration as a structural property of late-interaction retrieval and highlight a sparsity-robustness tradeoff. These findings motivate principled alternatives to hard max pooling in multi-vector retrieval systems.

• The paper demonstrates that hard MaxSim pooling yields high patch-level gradient concentration (Gini ~0.78) compared to Top-k and softmax pooling.
• It reveals that this concentrated gradient flow causes brittleness in retrieval performance as document lengths increase and under adversarial spike conditions.
• The study validates these findings on biomedical benchmarks, emphasizing the need for smoother pooling strategies for robust multi-vector retrieval.

Spike Hijacking and Gradient Routing in Late-Interaction Retrieval

Overview and Motivation

This work systematically investigates the robustness and training dynamics of late-interaction retrieval models employing the hard maximum similarity (MaxSim) pooling operator. These models, in contrast to single-vector embedding approaches, aggregate token-level or patch-level similarities using pooling strategies such as MaxSim, Top-$k$ averaging, or softmax. The primary focus is a mechanistic characterization of how MaxSim structurally biases gradient flow during contrastive learning and induces a tradeoff between sparsity and robustness, particularly as document length increases.

Structural Implications of MaxSim Pooling

Late-interaction architectures leverage higher-resolution token--patch interactions for effective retrieval, but the hard MaxSim operator imposes a highly non-smooth winner-take-all aggregation at each query token. The study introduces patch-level gradient concentration, quantified by the Gini coefficient over patch gradient norms, to probe the consequences of this pooling-induced sparsity.

Analyses in a controlled synthetic setting demonstrate that MaxSim consistently produces the highest gradient concentration across training epochs (Gini $\sim$0.78), compared to Top-$k$ pooling (Gini $\sim$0.45) and softmax pooling (Gini $\sim$0.18). This disparity is stable and intrinsic to the pooling function, not a transient effect of optimization, implying that MaxSim routes the majority of the learning signal onto a small subset of document patches.

Figure 1: Patch-level gradient concentration, retrieval quality during training, and the impact of increasing document length under different pooling operators.

Robustness to Document Length and Spike Injection

The winner-take-all routing of learning signals offers strong early discrimination, reflected in retrieval performance (Recall@1 $\sim$0.33-0.38 for both MaxSim and Top-$k$), yet the same property underpins a sharp loss in robustness as document length grows. Controlled sweeps over document lengths reveal that as $M$ increases, MaxSim's retrieval quality degrades more abruptly compared to smoothed alternatives, with Top-$k$ demonstrating a more gradual decline and consistently outperforming MaxSim at larger $M$.

Furthermore, the study designs an adversarial regime—"spike hijacking"—where negative documents are injected with synthetic "spike" patches that may attract the MaxSim aggregator. Empirically, token-wise maxima are redirected to these synthetic spikes, resulting in catastrophic hijacking of retrieval performance, with Recall@1 collapsing under extreme but plausible adversarial settings.

Real-World Validation in Biomedical Retrieval

The above findings are validated on real-world multi-vector benchmarks using ColQwen2.5 models on the ViDoRe biomedical dataset. Without retraining, patch-level gradient Gini values mirror the synthetic results: MaxSim (0.983) $\sim$0 Top-5 (0.951) $\sim$1 Softmax (0.883). Document-length sweeps confirm that MaxSim-trained models remain acutely sensitive to the presence of high-similarity (semantically meaningful) distractor patches: at $\sim$2 distractor tokens, retained recall for MaxSim and Top-$\sim$3 plummets to approximately 28%, while Softmax preserves 68%. Increasing $\sim$4 fails to mitigate this brittleness—the utilization of rigid selection strategies is fundamentally susceptible to semantic spikes.

Figure 2: Token--patch similarity heatmaps demonstrate that hard-negative spike injection causes widespread hijack of token maxima, while random (Gaussian) distractors yield minimal disruption.

The mechanism is visualized through token--patch similarity heatmaps, revealing that more than 80% of token maxima are rerouted to the adversarial region after spike injection, while a Gaussian noise control does not trigger this effect.

Theoretical and Practical Implications

The results isolate gradient concentration, induced structurally by the pooling operator, as a primary determinant of both discriminative power and vulnerability. Hard max-based pooling achieves peak selectivity but induces brittleness in the face of document expansion or targeted semantic distractors. Importantly, softer alternatives (e.g., Top-$\sim$5 or softmax) demonstrate a more balanced robustness/sparsity profile without catastrophic loss of retrieval accuracy for reasonable $\sim$6.

The implications extend to adversarial robustness and retrieval-augmented applications, where document length and adversarial content cannot be controlled. The spike hijacking vulnerability identified is not mitigated by naive aggregation over larger subsets (increasing $\sim$7), and alternative pooling formulations are required to achieve length-robust retrieval. Furthermore, these results challenge the consensus in the field regarding the indispensability of hard MaxSim in late-interaction systems and motivate end-to-end retraining and investigation of smoother pooling regimes.

Conclusion

This study provides a comprehensive analysis of the interplay between pooling strategy, gradient routing, and robustness in late-interaction retrieval. Hard MaxSim pooling, while effective for short documents, induces high patch-level gradient concentration and is structurally brittle under document-length expansion and adversarial spike injection. The observed sparsity--robustness tradeoff calls for principled innovation in pooling operators to achieve robust, high-quality multi-vector retrieval in practical, variable-length document regimes (2604.05253).