RP-KrossFuse Methodology

• The paper introduces a novel fusion technique that uses a random projection variant of the Kronecker product to combine cross-modal and unimodal expert representations.
• It achieves scalable, near-exact similarity computations that enhance unimodal performance while maintaining the alignment benefits of cross-modal encoders.
• Empirical evaluations demonstrate significant gains in image and text tasks, matching state-of-the-art expertise in both modality-specific and cross-modal settings.

RP-KrossFuse is a method for integrating cross-modal and uni-modal representations, enabling joint embeddings that achieve the performance of modality-specialized “expert” encoders within individual modalities while preserving the alignment properties of cross-modal embedding systems. It leverages a random projection variant of the Kronecker product to fuse sample-pairwise similarity scores in kernel space. RP-KrossFuse addresses the limitations of state-of-the-art cross-modal encoders, such as CLIP, whose unimodal performance often lags that of modality experts (e.g., DINOv2 for image, RoBERTa or E5 for text), and of unimodal encoders, which lack cross-modal alignment. The approach achieves scalable, nearly exact fusion of representations and is applicable with shift-invariant kernels via random Fourier features (Wu et al., 10 Jun 2025).

1. Problem Motivation and Objectives

Modern cross-modal encoders (e.g., CLIP, BLIP) learn representations $\Psi = (\Psi_x, \Psi_t)$ that align modalities such as images ($x \in \mathcal{X}$) and text ($t \in \mathcal{T}$) within a joint space $\mathcal{Z}$. Although effective at cross-modal retrieval and alignment, these systems typically underperform compared to domain-expert, single-modality encoders $\gamma_x:\mathcal{X} \rightarrow \mathbb{R}^{d_1}$ and $\gamma_t:\mathcal{T} \rightarrow \mathbb{R}^{d_2}$ when evaluated on modality-specific tasks (e.g., ImageNet accuracy for images). By contrast, expert encoders deliver strong in-domain performance yet lack cross-modal capability. RP-KrossFuse is constructed to preserve the cross-modal kernel $k(\Psi_x(x), \Psi_t(t))$ while boosting unimodal similarities $k(\Psi_x(x), \Psi_x(x'))$ and $k(\Psi_t(t), \Psi_t(t'))$ by injecting the corresponding uni-modal expert similarity, achieving a single scalable embedding with both properties (Wu et al., 10 Jun 2025).

2. Exact Kronecker-Product Fusion (KrossFuse)

The foundational operation of KrossFuse is the Kronecker product between feature maps from a cross-modal encoder and a symmetrized extension from a uni-modal expert. Given a reproducing kernel $$k: \mathcal{Z} \times \mathcal{Z} \rightarrow \mathbb{R}$$ with feature map $x \in \mathcal{X}$0, let the cross-modal encoder be $x \in \mathcal{X}$1 and the expert image encoder be $x \in \mathcal{X}$2. A “symmetrized” text extension is defined as:

• $x \in \mathcal{X}$3
• $x \in \mathcal{X}$4

where $x \in \mathcal{X}$5 is a tunable constant, and $x \in \mathcal{X}$6 is an all-ones vector. The fused feature maps are then:

• $x \in \mathcal{X}$7
• $x \in \mathcal{X}$8

The induced inner products exhibit the following properties (Proposition 2.2):

• $x \in \mathcal{X}$9
• $t \in \mathcal{T}$0
• $t \in \mathcal{T}$1

This construction yields a fused kernel on image pairs equal to the product of the CLIP- and expert-kernels plus $t \in \mathcal{T}$2, with cross-modal and text-text kernels remaining proportional to the original cross-modal kernels. However, the resultant dimensionality $t \in \mathcal{T}$3 poses prohibitive computation and memory requirements in practical scenarios (Wu et al., 10 Jun 2025).

3. Random Projection and the RP-KrossFuse Approximation

To address computational intractability, RP-KrossFuse replaces the Kronecker product with a compressed, randomized approximation. Each factor is projected using independent Johnson–Lindenstrauss (JL) maps:

• Draw $t \in \mathcal{T}$4, $t \in \mathcal{T}$5 with i.i.d. entries Uniform$t \in \mathcal{T}$6, such that $t \in \mathcal{T}$7.
• Define: $t \in \mathcal{T}$8
• $t \in \mathcal{T}$9

where $\mathcal{Z}$0 denotes the elementwise (Hadamard) product. Theorem 3.1 establishes that, for $\mathcal{Z}$1 samples, with probability at least $\mathcal{Z}$2, the RP-KrossFuse similarities satisfy:

$\mathcal{Z}$3

with similar bounds for text-text and image-text. Here $\mathcal{Z}$4 is a bound on the squared norms of the feature maps. The expectation of the random projection preserves inner products, and deviations are controlled by JL concentration (Wu et al., 10 Jun 2025).

4. Shift-Invariant Kernels and Random Fourier Features

RP-KrossFuse extends to infinite-dimensional, shift-invariant kernels such as Gaussian RBF using random Fourier features (RFF). For kernels $\mathcal{Z}$5 and $\mathcal{Z}$6, random pairs $\mathcal{Z}$7 are sampled from the product spectral density $\mathcal{Z}$8:

• $\mathcal{Z}$9

Theorem A.3 shows concentration of the RFF inner product to the product of the underlying kernels. The RP and symmetrization pipeline is then applied to the RFF-generated feature map in lieu of $\gamma_x:\mathcal{X} \rightarrow \mathbb{R}^{d_1}$0 and $\gamma_x:\mathcal{X} \rightarrow \mathbb{R}^{d_1}$1, enabling RP-KrossFuse to scale to high-dimensional or even infinite-dimensional kernel representations (Wu et al., 10 Jun 2025).

5. Similarity Computation and Theoretical Guarantees

For any samples $\gamma_x:\mathcal{X} \rightarrow \mathbb{R}^{d_1}$2 (images), $\gamma_x:\mathcal{X} \rightarrow \mathbb{R}^{d_1}$3 (text), or cross pairs $\gamma_x:\mathcal{X} \rightarrow \mathbb{R}^{d_1}$4, RP-KrossFuse computes fused similarities as:

• $\gamma_x:\mathcal{X} \rightarrow \mathbb{R}^{d_1}$5
• $\gamma_x:\mathcal{X} \rightarrow \mathbb{R}^{d_1}$6
• $\gamma_x:\mathcal{X} \rightarrow \mathbb{R}^{d_1}$7

The factor $\gamma_x:\mathcal{X} \rightarrow \mathbb{R}^{d_1}$8 ensures correct scaling in expectation, and deviations can be bounded via Hoeffding- or JL-style inequalities. This suggests that RP-KrossFuse yields a single embedding space with controlled, quantifiably small deviation from the target fused similarities (Wu et al., 10 Jun 2025).

6. Computational Complexity and Scalability

The computational and memory demands are substantially improved relative to exact Kronecker fusion. For exact KrossFuse, dimensions are $\gamma_x:\mathcal{X} \rightarrow \mathbb{R}^{d_1}$9 and inner products require $\gamma_t:\mathcal{T} \rightarrow \mathbb{R}^{d_2}$0 time. RP-KrossFuse, by projecting to $\gamma_t:\mathcal{T} \rightarrow \mathbb{R}^{d_2}$1:

• Two matrix-vector multiplications: $\gamma_t:\mathcal{T} \rightarrow \mathbb{R}^{d_2}$2
• Elementwise product: $\gamma_t:\mathcal{T} \rightarrow \mathbb{R}^{d_2}$3

When using random Fourier features of size $\gamma_t:\mathcal{T} \rightarrow \mathbb{R}^{d_2}$4, the cost is $\gamma_t:\mathcal{T} \rightarrow \mathbb{R}^{d_2}$5 for the cosine/sine feature evaluation and $\gamma_t:\mathcal{T} \rightarrow \mathbb{R}^{d_2}$6 for the final fusion, maintaining overall linear complexity in $\gamma_t:\mathcal{T} \rightarrow \mathbb{R}^{d_2}$7 and $\gamma_t:\mathcal{T} \rightarrow \mathbb{R}^{d_2}$8. For practical settings (e.g., $\gamma_t:\mathcal{T} \rightarrow \mathbb{R}^{d_2}$9k versus $k(\Psi_x(x), \Psi_t(t))$0k), this yields orders of magnitude efficiency gains with negligible loss in accuracy (Wu et al., 10 Jun 2025).

7. Empirical Evaluation and Performance

Experiments fuse CLIP with image experts (DINOv2 ViT-B/14, UniCom) and text experts (Sentence-RoBERTa, E5), across tasks including image classification, zero-shot retrieval, and text classification. Key results include:

• Image linear probe with ViT-B/32: ImageNet top-1 improves from $k(\Psi_x(x), \Psi_t(t))$1, SVHN $k(\Psi_x(x), \Psi_t(t))$2, with out-of-distribution (OOD) performance (ImageNet-A/R) maintained within 1–2 points of DINOv2.
• Text linear probe with SentEval: average accuracy increases from $k(\Psi_x(x), \Psi_t(t))$3 when fusing CLIP text with Sentence-RoBERTa; similar gains observed with E5.
• Zero-shot retrieval (Flickr30K, MSCOCO): Recall@1 matches original CLIP within 0.5% across model sizes.
• Cross-modal few-shot (1–8 shots): average accuracy on generic image tasks rises by 4–9 points when using image and text jointly, outperforming CLIP(I), CLIP(I+T), and DINOv2(I).

The approach functions as a plug-in, training-free methodology for merging cross-modal and unimodal encoders. Random projection and random Fourier features render the Kronecker fusion tractable, and provable bounds (JL and RFF) ensure that all modality similarities remain nearly exact. Empirical evidence demonstrates that RP-KrossFuse effectively unifies the strengths of both cross-modal and modality-expert representations in a single embedding while maintaining computational practicality (Wu et al., 10 Jun 2025).