AVL: Vector Loss for Coherent Predictions

• AVL is a vector-based loss function defined as the averaged L₂-norm between predicted and true vectors, capturing overall magnitude and orientation differences.
• It integrates physical and geometric principles from fluid dynamics and vision-language models to preserve multi-scale flow patterns and semantic structure.
• Hybrid formulations combining AVL with MSE balance pixel-level accuracy with global coherence, markedly reducing KL divergence in empirical studies.

Average Vector Loss (AVL) is a vector-based loss function that quantifies the discrepancy between sets of predicted and target vectors by directly penalizing differences in their overall magnitudes and orientations. Originally developed in fluid-dynamics inpainting and later adapted for contrastive vision-LLMs, AVL provides a principled approach for preserving physically and semantically meaningful structures in the output of machine learning systems.

1. Mathematical Definitions

In its canonical form for vector field prediction, AVL is defined as the averaged L₂-norm between corresponding predicted and true vectors. For $N$ points (e.g., pixels) with true velocity vectors $\mathbf{v}_i \in \mathbb{R}^2$ and network predictions $\hat{\mathbf{v}}_i \in \mathbb{R}^2$: $$L_{\mathrm{AVL}} = \frac{1}{N} \sum_{i=1}^{N} \bigl\| \hat{\mathbf{v}}_i - \mathbf{v}_i \bigr\|_2$$ This formulation, also referred to as "average vector L₂" or "vector-magnitude difference", accounts for the holistic difference between predicted and target vectors, in contrast to coordinate-wise losses.

In the context of model fine-tuning in vision-language embeddings, AVL operates on difference vectors between the outputs of pre-trained and fine-tuned encoders. For a mini-batch of $B'$ reference image-text pairs $(\bm{x}_i^{\rm ref}, \bm{t}_i^{\rm ref})$: $$u(\bm{x}_i^{\rm ref}) = f_{\theta_{\rm ft}}(\bm{x}_i^{\rm ref}) - f_{\theta_{\rm pre}}(\bm{x}_i^{\rm ref})$$

$$v(\bm{t}_i^{\rm ref}) = g_{\phi_{\rm ft}}(\bm{t}_i^{\rm ref}) - g_{\phi_{\rm pre}}(\bm{t}_i^{\rm ref})$$

These difference vectors are constrained to cluster around their exponential moving average $\bm{m}$, resulting in the loss: $$\mathcal{L}_{\rm avl} = \frac{1}{B'} \sum_{j=1}^{B'} \left\|u(\bm{x}_j^{\rm ref}) - \bm{m}\right\|_2^2 + \left\|v(\bm{t}_j^{\rm ref}) - \bm{m}\right\|_2^2$$ with $\bm{m}$ updated per batch as an EMA: $$\bm{m} = \alpha\,\bm{m}_{\rm prev} + (1-\alpha)\,\frac{1}{B'}\sum_{j=1}^{B'} \frac{u(\bm{x}_j^{\rm ref}) + v(\bm{t}_j^{\rm ref})}{2}$$ where $\alpha \in [0,1)$ is a momentum hyperparameter.

2. Physical and Geometric Motivation

In turbulent flow reconstruction (Baker et al., 6 Sep 2025), AVL arises from the need to respect the vectorized nature of data, such as velocity fields from particle image velocimetry (PIV). Standard mean square error (MSE) losses operate component-wise: $$L_{\mathrm{MSE}} = \frac{1}{N}\sum_{i} \left( (\hat{v}_{i,x} - v_{i,x})^2 + (\hat{v}_{i,y} - v_{i,y})^2 \right)$$ which ignores the spatial and energetic coherence of underlying flow structures. AVL, by penalizing the entire vector difference, directly enforces fidelity in the energy distribution across multiple scales and supports recovery of coherent vortical features.

In vision-LLM fine-tuning (Suzuki et al., 13 Nov 2025), the geometric structure of embedding spaces encodes semantic similarity. Uniform shifts in embeddings (enforced by AVL) preserve relative distances between data points, thus maintaining global structure and robustness in out-of-distribution and zero-shot generalization. Without such constraints, vanilla fine-tuning distorts pairwise relationships, degrading generalization.

3. Hybrid Loss Formulations

For applications prioritizing both global vector coherence and pixel-wise accuracy, AVL is often blended with standard losses. The hybrid loss takes the form: $$L_{\mathrm{hyb}} = \alpha\,L_{\mathrm{AVL}} + (1-\alpha)\,L_{\mathrm{MSE}}$$ where $0 \leq \alpha \leq 1$ controls the trade-off. Empirical studies in turbulent flow inpainting show that $\alpha=0.2$ yields an effective balance, marginally increasing L₂ error (by $\approx 0.3\%$) while decreasing the Kullback–Leibler (KL) divergence in speed distributions by a factor of $2.7$ compared to pure MSE. This hybridization is critical when both subtle spatial alignment and large-scale coherence are desired.

4. Practical Implementation and Parameterization

For turbulent flow inpainting, AVL is computed over the set of missing/gap pixels in a masked region, and is agnostic to model architecture—only requiring per-pixel vector outputs. Key design choices:

• Gap size and topology: In (Baker et al., 6 Sep 2025), a $13 \times 13$ central block ($\sim$10% of the field) in $50 \times 49$ vector grids is masked.
• Training regimen: 600 epochs, using U-Net, Adam optimizer, weight decay $=1 \times 10^{-3}$, learning-rate halved every 100 epochs.
• Evaluation: Both normalized L₂ (pixel-level) and KL divergence (distributional) metrics used to characterize performance.

For embedding regularization in vision-LLMs (Suzuki et al., 13 Nov 2025):

• Batch-level reference pairs: $B'$ pairs per batch; $\alpha=0.99$ for EMA of reference average.
• Loss scaling: $\lambda=1000$ (empirically selected) multiplies the AVL term relative to core contrastive and pairwise vector losses.
• Exponential moving average: Ensures stable adaptation of $\bm{m}$ across training iterations.
• Implementation: No architectural constraints; applicable wherever embedding “movements” are accessible.

An explicit calculation for a batch is provided in (Suzuki et al., 13 Nov 2025), illustrating the vector average computation and loss evaluation.

5. Quantitative Results and Empirical Impact

The following summarizes key findings reported for AVL and hybrid losses in turbulent flow field inpainting ((Baker et al., 6 Sep 2025), Table 2):

Loss Type	L₂ Error	KL Divergence
Cosine only	0.645	0.268
MI only	0.485	0.014
Vector (AVL)	0.467	0.013
Hybrid ($\alpha=0.2$)	0.463	0.017
MSE	0.460	0.046

Notable observations:

• Cosine-only loss collapses predictions to near-zero magnitude, yielding a KL divergence nearly $200\times$ worse than the best configuration.
• AVL and MI dramatically reduce KL divergence, recovering multi-scale flow patterns and energetic coherence, at a cost of a minor increase (1–2%) in pixel-wise error.
• The hybrid loss nearly matches MSE in L₂ error ($<0.3\%$ difference) but achieves a $2.7\times$ reduction in KL divergence, indicating improved preservation of turbulent structures.

Empirically, AVL-based losses enable faithful recovery of high-speed jets and vortex structures within inpainted gaps—features systematically underestimated by MSE-only objectives.

6. Conceptual Extensions and Significance

The core principle behind AVL is the explicit acknowledgment of inter-component and inter-sample relationships—either preserving holistic physical quantities (e.g., fluid flow energy) or semantic geometry in feature spaces. By averaging vector differences rather than treating coordinates or samples in isolation, AVL can encode domain-driven constraints without the necessity for hand-defined physics loss terms or excessive post-hoc regularization.

This suggests broader applicability of AVL and its variants wherever preservation of continuum structure, coherent motion, or geometric relations is critical, including climate modeling, robotics, graph-based learning, and representation consolidation during transfer or domain-adaptive fine-tuning.

A plausible implication is that hybridization of AVL with domain-standard losses, calibrated according to empirical trade-offs between pixel-level accuracy and structure preservation, will remain a promising strategy in domains rich in multi-scale, vectorial, or high-dimensional semantic data.