Patch-Level Cross-Modal Contrastive Loss

• The paper introduces a novel loss that combines patch-level contrastive mechanisms with second-order similarity, enhancing cross-modal descriptor learning.
• It details a mathematical formulation that leverages pairwise and higher-order distance profiles to maintain global structural consistency.
• Empirical findings demonstrate improved retrieval metrics and classification accuracy through robust feature alignment and efficient mini-batch computations.

Second-order similarity loss refers to a broad class of loss functions and regularization mechanisms that enforce not only first-order correspondence (i.e., similarity or dissimilarity in feature space or metric space) but also consistency in the higher-order (typically, second-order) geometric, statistical, or structural relationships among data points, descriptors, or functions. This paradigm has become fundamental in deep local and global descriptor learning, metric learning, federated optimization, unsupervised feature selection, dynamical systems modeling, and functional time series analysis. By explicitly promoting second-order structure preservation, these losses address limitations of first-order-only criteria, facilitating performance improvements and theoretical robustness across diverse domains.

1. Mathematical Formulation and Principles

Second-order similarity loss is defined over various data modalities and learning settings, but central to its definition is the measurement of agreement in second-order feature relationships—pairwise, higher-order co-occurrences, Hessian similarities, or operator distances.

For instance, in the local descriptor learning context of SOSNet, the second-order similarity between two ℓ₂-normalized descriptors $x_i, x_i^+$ is evaluated as the $\ell_2$ distance between their respective distance profiles to all other descriptors in the mini-batch:

$$d^{(2)}(x_i, x_i^+) = \sqrt{ \sum_{j\neq i} \left( \|x_i - x_j\|_2 - \|x_i^+ - x_j^+\|_2 \right)^2 }.$$

The second-order similarity regularization (SOSR) term is then

$$R_{SOS} = \frac{1}{N} \sum_{i=1}^N d^{(2)}(x_i, x_i^+).$$

This regularizer is integrated into the overall training objective:

$L_{total} = L_{FOS} + \lambda R_{SOS},$

where $L_{FOS}$ is typically a triplet or contrastive first-order loss and $\lambda$ a tunable weighting parameter (Tian et al., 2019).

In other domains, second-order similarity may refer to agreement between covariance matrices (Zhang et al., 2018), similarity/dissimilarity between Hessians in federated optimization (Khaled et al., 2022), or operator distances between time-varying spectral density operators in functional data analysis (Delft et al., 2018).

2. Second-Order Losses in Metric and Descriptor Learning

Second-order loss formulations have been predominant in metric and descriptor learning tasks, where relationships among descriptors—beyond direct pairs—are critical for tasks such as matching, retrieval, and structure recovery.

• SOSNet: In local patch descriptor learning, the SOSR term enforces that true matching pairs have not only close proximity but similar distance relations to neighboring descriptors, enhancing the uniformity and spread over the embedding hypersphere (Tian et al., 2019).
• SOLAR: In the context of global image descriptors, the second-order similarity loss compares the difference in distances from anchors/positives to the same negative, penalizing fluctuations in "distance gaps" (i.e., enforcing

$$L_{SOS} = \frac{1}{T}\sqrt{\sum_{t=1}^T (d_{an}^2 - d_{pn}^2)^2 }$$

within triplets, see (Ng et al., 2020)), and is paired with triplet loss and hard negative mining for improved retrieval.

• Few-shot Learning (SoSN): The Power-Normalizing Second-order Similarity Network computes second-order feature covariances for support/query images, applies learned or probabilistically founded nonlinearities (e.g., SigmE), and drives the matching via an MSE loss on relation scores defined by the normalized second-order statistics (Zhang et al., 2018).

Second-order losses consistently enhance performance across standard benchmarks, often outperforming prior state-of-the-art in FPR@95, mAP, and few-shot accuracy by substantial margins. Ablation studies demonstrate that these gains are uniquely attributable to second-order regularization, beyond optimizer choice or first-order loss tweaks (Tian et al., 2019, Ng et al., 2020, Zhang et al., 2018).

3. Second-Order Similarity in Federated and Statistical Optimization

Second-order similarity extends beyond metric learning to distributed and federated optimization, where function (objective) similarity across clients is characterized at the level of Hessian matrices.

• Formal Property: Second-order similarity, or Hessian similarity, is defined as

$$\frac{1}{M}\sum_{m=1}^M \|\nabla^2 f_m(x) - \nabla^2 f(x)\|_{op}^2 \leq \delta^2, \quad \forall x\in\mathbb{R}^d,$$

with $\delta$ controlling the degree of heterogeneity among clients (Khaled et al., 2022).

• Algorithmic Consequences: Under this condition, Stochastic Proximal Point Methods (SPPM) and variance-reduced proximal algorithms (SVRP, Catalyzed SVRP) attain significantly improved communication complexity, replacing global smoothness $\ell_2$0 with $\ell_2$1 in rate bounds. This enables scalable, efficient optimization even under client sampling, which is not possible under first-order-only similarity assumptions.
• Theoretical Implication: The bounded curvature variation implies high robustness and rapid convergence in distributed settings—a property exploited both in analysis and algorithm design (Khaled et al., 2022).

4. Higher-Order Similarity in Unsupervised and Multi-view Learning

Second-order similarity graphs are leveraged in unsupervised feature selection, multi-view learning, and clustering, capturing global data structure that is invisible to first-order affinity graphs.

• SHINE-FS: The Structure-aware Hybrid-order sImilarity learNing for multi-viEw unsupervised Feature Selection algorithm learns both first-order ($\ell_2$2) and second-order ($\ell_2$3) similarity graphs. Here, $\ell_2$4 is constructed to encode the similarity of samples via their relationships to shared anchor points:

$\ell_2$5

A joint loss combining first- and second-order graphs improves clustering and feature selection, as measured by downstream unsupervised classification performance (Xu et al., 27 Nov 2025).

This suggests that second-order similarity mechanisms are generic tools for enhancing global geometric fidelity in unsupervised, label-free representation learning.

5. Second-Order Similarity in Dynamical Systems and Time Series

In learning dynamical systems, particularly chaotic flows, first-order supervision via trajectory or Jacobian matching can prove insufficient for preserving attractor geometry and invariant statistics. Second-order similarity losses address this gap.

• Randomized Jacobian Matching: By comparing the Jacobians $\ell_2$6 of the true and learned vector fields at randomly perturbed inputs, the expected loss

$\ell_2$7

is shown, via Taylor expansion, to combine a nominal Jacobian mismatch with a Hessian (second-order) mismatch scaled by the perturbation variance. This provides rigorous second-order supervision at $\ell_2$8 cost, circumventing explicit Hessian construction ($\ell_2$9) (Kang et al., 1 Jun 2026).

• Practical Impact: Empirically, second-order losses prevent catastrophic Lyapunov exponent outliers and attractor drift, recovering correct invariant measures even under minimal supervision, as demonstrated on Lorenz-63 and Lorenz-96 (Kang et al., 1 Jun 2026).
• Functional Time Series: In non-stationary functional time series analysis, the second-order similarity loss is defined as the normalized Hilbert–Schmidt distance between time-varying spectral density operators, supporting rigorous clustering and hypothesis testing of global second-order structure:

$$d^{(2)}(x_i, x_i^+) = \sqrt{ \sum_{j\neq i} \left( \|x_i - x_j\|_2 - \|x_i^+ - x_j^+\|_2 \right)^2 }.$$0

as detailed in (Delft et al., 2018).

6. Algorithmic Implementation and Empirical Findings

Algorithmic realization of second-order similarity losses is diverse and context-dependent but features certain unifying patterns:

• Mini-batch Computation: For deep neural architectures, second-order terms are implemented via batch-wise construction of distance matrices (SOSNet, SOLAR), nearest-neighbor mining, and efficient masking to reduce computational burden (Tian et al., 2019, Ng et al., 2020).
• Power Normalization and Permutation: SoSN applies power-normalization (SigmE) to second-order matrices for robust co-occurrence statistics and applies permutation to fully exploit spatial co-occurrence patterns (Zhang et al., 2018).
• Graph and Anchor Construction: In SHINE-FS, second-order similarity graphs are constructed via sparse simplex-constrained optimization based on anchor-sample relationships, enabling closed-form row-wise updates (Xu et al., 27 Nov 2025).
• Distributed Optimization: SVRP and Catalyzed SVRP leverage curvature similarity for large stepsizes and variance reduction, optimizing communication efficiency in federated learning (Khaled et al., 2022).
• Dynamical Systems: Randomized Jacobian matching is implemented via automatic differentiation and random perturbation in each batch, ensuring scalable, higher-order consistency (Kang et al., 1 Jun 2026).
• Time Series: Hilbert–Schmidt similarities between spectral density operators are estimated by blockwise periodograms and aggregated for cluster adjacency (Delft et al., 2018).

Experimental results across local/global descriptor learning, image retrieval, few-shot regimes, dynamical systems, and unsupervised selection uniformly demonstrate that second-order similarity losses yield improved discriminability, robustness to outliers, invariant structure preservation, and enhanced utilization of embedding geometry. Benchmark improvements include reductions in FPR@95, absolute gains in retrieval mAP and few-shot accuracy, more compact clusters, and improved Lyapunov spectra recovery (Tian et al., 2019, Ng et al., 2020, Zhang et al., 2018, Kang et al., 1 Jun 2026).

7. Theoretical and Geometric Interpretations

A core geometric effect of second-order similarity regularization is improved utilization of descriptor or feature space, quantified via vMF-based measures in embedding scenarios. Specifically, the enforcement of second-order profile agreement spreads class means more uniformly over the hypersphere, reducing inter-class crowding and thus improving nearest-neighbor matchability (Tian et al., 2019).

In federated optimization, the theoretical value of second-order similarity arises in improved contraction and communication complexity rates, enabled by bounding local-global curvature deviations (Khaled et al., 2022). In dynamical systems and time series, second-order similarity ensures structural invariance under nonidentifiability, ambiguity, or high sensitivity to initial conditions (Kang et al., 1 Jun 2026, Delft et al., 2018).

In summary, second-order similarity loss constitutes a unifying paradigm for encoding not only proximity but higher-order structural alignment across a wide array of learning frameworks and applications. Its mathematical grounding, algorithmic feasibility, and empirical success establish it as a central tool in modern machine learning and statistical modeling.