Pairwise Vector Loss (PVL) in Vision-Language Models

• PVL is a regularization technique within the DiVE framework that enforces local consistency by aligning difference vectors from pre-trained and fine-tuned image-caption pairs.
• It uses a batch-wise ℓ2 loss to constrain shifts in encoder outputs, effectively mitigating geometric distortions caused by contrastive fine-tuning.
• Empirical results show that combining PVL with AVL in DiVE significantly improves RSA correlation, OOD, and zero-shot accuracy while preserving in-distribution performance.

Pairwise Vector Loss (PVL) is a regularization technique introduced within the Difference Vector Equalization (DiVE) framework to robustly fine-tune vision-LLMs by preserving the geometric structure of their embeddings. PVL operates by constraining the difference vectors—each capturing the change between pre-trained and fine-tuned encoder outputs—for multimodal image-caption pairs, enforcing local consistency and alignment during adaptation. This methodology was formalized to address limitations in contrastive fine-tuning, which, while maintaining in-distribution accuracy, can severely distort inter-point relationships and degrade out-of-distribution (OOD) and zero-shot performance (Suzuki et al., 13 Nov 2025).

1. Mathematical Formulation

Let $f_{\theta_{\mathrm{pre}}}(x)$ and $f_{\theta_{\mathrm{ft}}}(x)$ denote the image encoder's pre-trained and fine-tuned outputs for image $x$, and $g_{\phi_{\mathrm{pre}}}(t)$ and $g_{\phi_{\mathrm{ft}}}(t)$ those for the text encoder and caption $t$. For every image–caption reference pair $(x_j^{\mathrm{ref}}, t_j^{\mathrm{ref}})$ in a mini-batch $\mathcal S^{\mathrm{ref}}$, the "difference vectors" are:

$$u(x_j^{\mathrm{ref}}) = f_{\theta_{\mathrm{ft}}}(x_j^{\mathrm{ref}}) - f_{\theta_{\mathrm{pre}}}(x_j^{\mathrm{ref}})$$

$$v(t_j^{\mathrm{ref}}) = g_{\phi_{\mathrm{ft}}}(t_j^{\mathrm{ref}}) - g_{\phi_{\mathrm{pre}}}(t_j^{\mathrm{ref}})$$

The Pairwise Vector Loss is then defined as:

$$\mathcal{L}_{\mathrm{pvl}} = \frac{1}{B'} \sum_{j=1}^{B'} \big\| u(x_j^{\mathrm{ref}}) - v(t_j^{\mathrm{ref}}) \big\|^2$$

where $B'$ is the batch size of reference pairs. PVL imposes an $\ell_2$ constraint to ensure that, for each multimodal pair, the encoder shifts are closely matched.

2. Role within the DiVE Objective

DiVE adapts vision-LLMs by aggregating three losses:

1. A standard contrastive loss $\mathcal L_{\mathrm{cl}}$ (as in the FLYP baseline),
2. The Average Vector Loss (AVL) $\mathcal L_{\mathrm{avl}}$ (for global structure preservation),
3. The Pairwise Vector Loss (PVL) $\mathcal L_{\mathrm{pvl}}$ (for local multimodal alignment).

The full fine-tuning objective is:

$$\mathcal{L}_{\mathrm{DiVE}} = \mathcal{L}_{\mathrm{cl}} + \lambda \big( \mathcal{L}_{\mathrm{avl}} + \mathcal{L}_{\mathrm{pvl}} \big)$$

Here, $\lambda$ is a scalar hyperparameter governing the trade-off between contrastive supervision and geometric regularization ($\lambda=1000$ proved optimal on ImageNet (Suzuki et al., 13 Nov 2025)). PVL penalizes deviations between difference vectors computed for matched image–caption pairs, enforcing locally consistent multimodal alignment during adaptation.

3. Geometric Structure Preservation

Contrastive fine-tuning on its own frequently distorts the geometry of the joint embedding space, particularly under distributional shift. PVL and AVL are specifically designed to address these distortions:

• AVL constrains every sample's difference vector to be close to their global running average, maintaining global geometry.
• PVL imposes pairwise consistency between corresponding image and caption difference vectors, preserving local geometric relationships.

This dual constraint—global (AVL) and local (PVL)—prevents arbitrary, sample-specific feature drifts that undermine downstream zero-shot and OOD generalization.

4. Hyperparameters and Implementation Protocol

PVL's efficacy depends on several key hyperparameters:

• Batch size ($B'$): Typically matched to the fine-tuning batch size (256 or 512).
• $\lambda$ (geometric loss weight): Selected empirically; $\lambda=1000$ on ImageNet balances contrastive and geometric losses.
• Reference set selection ($\mathcal S^{\mathrm{ref}}$): Chosen from the training data, typically as random mini-batches.

PVL is implemented as a simple batch-wise $\ell_2$ average over difference vector pairs. During inference, only the fine-tuned network is used; the PVL constraint is applied solely during training.

5. Empirical Impact and Comparative Metrics

PVL directly contributes to retention of embedding geometric structure and stronger fine-tuning outcomes:

Fine-tuning variant	RSA correlation	OOD (%)	Zero-shot (%)	In-distribution (%)
FLYP (contrastive)	0.825	59.5	49.5	82.2
FLYP + AVL	0.978	62.9	62.9	—
FLYP + PVL	0.976	62.6	62.7	—
DiVE (AVL + PVL)	0.981	63.2	63.7	82.5

Key outcomes (Suzuki et al., 13 Nov 2025):

• AVL alone or PVL alone yield large gains in OOD and zero-shot accuracy (over +13%), with almost perfect RSA correlation, indicating near-complete geometric retention.
• PVL provides a supplementary 0.3–0.5% boost over AVL, consistently across benchmarks.
• DiVE (with both AVL and PVL) slightly improves OOD and zero-shot accuracy while fully maintaining the pre-trained embedding structure.

6. Advantages, Limitations, and Significance

Advantages:

• PVL explicitly protects local multimodal alignment in encoder embedding shifts.
• Computationally efficient, requiring only batch-wise $\ell_2$ distances.
• Universally applicable for multimodal models leveraging paired data.

Limitations:

• Tuning $\lambda$ is dataset- and model-specific.
• PVL is inherently tied to the existence of matched image–caption pairs; its utility for unpaired modalities is not demonstrated in (Suzuki et al., 13 Nov 2025).
• Does not enforce global rigid transforms alone; global structure preservation additionally depends on AVL.

Significance:

• PVL, as part of DiVE, enables robust fine-tuning without sacrificing OOD and zero-shot performance—a constraint that contrastive-only approaches consistently fail to meet (Suzuki et al., 13 Nov 2025).
• The methodology marks a substantial improvement in retaining generalization capabilities of pre-trained vision-language architectures under adaptation.

7. Contextual Placement and Future Directions

PVL's development fits into a broader movement towards structural regularization in deep representation learning. The approach directly addresses geometric distortions induced by aggressive task-specific adaptation, building on prior contrastive frameworks while making explicit the need for geometric integrity.

This suggests potential extensions of PVL to other domains where paired structural regularity is critical, such as graph neural nets or multimodal retrieval. A plausible implication is that similar pairwise regularization schemes may benefit language-only or image-only transfer learning settings where ground-truth correspondences are available.

Ongoing work may investigate the application of PVL and AVL in domains with approximate or weakly paired data, or in scenarios requiring the preservation of task-agnostic embedding characteristics across fine-tuning cycles.