Text-Visual Embedding Alignment (TVEA)

Updated 18 July 2026

TVEA is a family of methods that learn aligned embedding functions for textual and visual inputs to place semantically related pairs close in a joint space.
Approaches include global contrastive alignment, partial matching for videos, and token-level supervision in multimodal models, leading to improved retrieval and segmentation metrics.
The framework spans various tasks—retrieval, segmentation, generation, and editing—using strategies like local/global fusion and asymmetric alignment to tackle modality discrepancies.

Text–Visual Embedding Alignment (TVEA) denotes a family of methods that learn embedding functions for textual and visual inputs so that semantically related pairs are close in a joint space and unrelated pairs are far apart. In one explicit formulation, given a textual description $t$ and a visual input $v$ (an image or video), TVEA learns embedding functions $f_t(t)$ and $f_v(v)$ so that semantically related text–visual pairs lie close in a joint space, while unrelated pairs are far apart (Li et al., 28 Jul 2025). Across the literature, the term covers global image–caption retrieval, text–video retrieval, token-level alignment in multimodal LLMs (MLLMs), text-guided editing of visual embeddings, and task-specific modules that use textual priors to regularize visual prediction (Yin et al., 2024).

1. Scope and definitions

In classical image–text retrieval, TVEA is typically cast as learning to match a caption with an image by minimizing a global contrastive loss over entire representations. In text-to-video retrieval, however, full-representation alignment often fails because videos contain many frames, only some of which are described by $t$ , and $t$ is typically much shorter than the rich, temporally extended content of $v$ ; this yields partial alignment, in which only a subset of frames or frame–text substructures should be aligned (Li et al., 28 Jul 2025). A common consequence is that directly aligning text representations with video representations can result in incorrect supervision, ignoring the inequivalence of information (Li et al., 28 Jul 2025).

The term is also used in narrower, task-specific senses. In source-free cross-domain few-shot segmentation, the TVEA module leverages textual priors from pre-aligned multi-modal features, such as CLIP, to guide cross-modal adaptation; it uses a frozen CLIP model to produce dense class activation maps, thresholds those CAMs into pseudo-masks $M_q^p$ , and enforces consistency between those pseudo-masks and the rough query masks $\hat M_q$ produced by the Task-Specific Attention Adapter (Liu et al., 7 Aug 2025). In work on typographic prompt injection attacks, TVEA is defined as the distance between the embedding of the clean text prompt and the embedding of the same prompt rendered as an image, with the primary metric given by the Euclidean distance between normalized embeddings (Balakrishnan et al., 14 Apr 2026).

This distribution of uses suggests that TVEA is better understood as a class of alignment problems than as a single canonical objective. Some papers treat it as global shared-space retrieval; others treat it as token-level supervision, partial matching, post hoc editing, or a diagnostic proxy for model behavior.

2. Foundational formulations and early architectures

Early TVEA systems used relatively direct common-space constructions. “Order embeddings and character-level convolutions for multimodal alignment” projects VGG-19 image features and character-level textual features into a common nonnegative embedding space $\mathbb{R}^d_+$ , with similarity defined by the order-violation penalty

$v$ 0

and trains with a bidirectional contrastive ranking loss on Microsoft COCO (Wehrmann et al., 2017). The textual encoder replaces word embeddings and recurrent neural networks with character-level convolutions, and the paper reports that the largest variant, Arch D, reaches 47.2% Recall@1 for image-to-text retrieval and 37.5% Recall@1 for text-to-image retrieval on COCO test (Wehrmann et al., 2017).

“Multitask Text-to-Visual Embedding with Titles and Clickthrough Data” keeps the visual side shallow by extracting pre-trained VGG-19 or ResNet features and learns text embeddings with stacked LSTM layers over FastText word vectors. Its distinctive contribution is a positive-aware ranking loss,

$v$ 1

together with mini-batch hard negative sampling and multitask training on image titles and click-through data (Aggarwal et al., 2019). On Adobe Stock, the paper reports Average R@1 of 0.133 for PATR, compared with 0.120 for triplet loss and 0.094 for $v$ 2, while joint training on both streams outperforms click-only and titles-only training (Aggarwal et al., 2019).

“T2VLAD: Global-Local Sequence Alignment for Text-Video Retrieval” moves beyond a single global vector by introducing shared semantic centers and complementary local and global similarity. Its local alignment aggregates textual and visual sequences with learnable centers, computes per-center similarities, and combines them with a global alignment score; the resulting complexity is reduced from naive pairwise matching to $v$ 3 per text–video pair (Wang et al., 2021). On MSR-VTT (1k-A split), it pushes text→video R@1 from approximately 26% to 29.5% and video→text R@1 from 27% to 31.8% (Wang et al., 2021).

“ContextCLIP” adds a contextual loss to a CLIP-style framework built from ResNet-50, BERT-base, and 256-dimensional projection heads. The contrastive loss aligns image–text pairs at the point level, while the contextual loss aligns the two modalities at the feature level by treating each embedding dimension as a feature-point (Grover et al., 2022). The reported zero-shot results illustrate both the promise and the unevenness of alignment: ContextCLIP improves CIFAR-10, CIFAR-100, BirdsNap, and MNIST relative to CLIP, but RESISC45 decreases from 3.41 / 12.19 to 1.48 / 11.05 for top-1 / top-5 accuracy (Grover et al., 2022).

“LAViTeR: Learning Aligned Visual and Textual Representations Assisted by Image and Caption Generation” explicitly names its main module Visual Textual Alignment (VTA), assisted by GAN-based image synthesis and Image Captioning, and proposes a new evaluation metric measuring the similarity between the learnt visual and textual embedding. Its experiments on CUB and MS-COCO are reported to demonstrate superior visual and textual representation alignment in the joint feature embedding space (Hashemi et al., 2021).

Formulation	Representative mechanism	Representative papers
Global shared-space alignment	Order-violation ranking, PATR, shared semantic centers	(Wehrmann et al., 2017, Aggarwal et al., 2019, Wang et al., 2021)
Contextual or local alignment	Feature-level contextual loss, local/global fusion	(Grover et al., 2022, Wang et al., 2021)
Auxiliary-task alignment	Image synthesis and image captioning assist alignment	(Hashemi et al., 2021)

3. Fine-grained, decomposed, and asymmetric alignment

A major line of work argues that a single monolithic text embedding is often too coarse. “Context-Adaptive Multi-Prompt Embedding” replaces the single text embedding in CLIP-style models with $v$ 4 structured prompts of the form “[text]. The [APT $v$ 5] of this image means:”, where each adaptive prompt token is randomly initialized and trained end-to-end (Kim et al., 3 Aug 2025). The prompts are encoded jointly in a single forward pass through a pretrained LLM with prompt-wise attention masking, and the resulting vectors are projected to dimension $v$ 6 and concatenated channel-wise into a single $v$ 7-dimensional text embedding $v$ 8 (Kim et al., 3 Aug 2025). Training uses the contrastive loss $v$ 9, a diversity regularization loss

$f_t(t)$ 0

and a negation-aware loss $f_t(t)$ 1, combined as

$f_t(t)$ 2

with $f_t(t)$ 3 by default (Kim et al., 3 Aug 2025). With ViT-B/16, $f_t(t)$ 4, and Gemma-2B, the paper reports Flickr30K image→text R@1 of 84.7% versus 81.9% for CLIP-B, Flickr30K text→image R@1 of 68.7% versus 62.1%, MSCOCO image→text R@1 of 58.5% versus 52.4%, and MSCOCO text→image R@1 of 41.4% versus 33.1%; ablations further show that using $f_t(t)$ 5 prompts yields +11.3% absolute gain on Flickr30K over a single prompt, and adding $f_t(t)$ 6 and $f_t(t)$ 7 compounds another +2.3% gain (Kim et al., 3 Aug 2025).

“T2VParser” addresses partial alignment directly in text-to-video retrieval. It introduces Adaptive Decomposition Tokens, a shared set of $f_t(t)$ 8 learnable tokens that extract corresponding semantic slices from both modalities, followed by a dual communication mechanism that filters and fuses these slices before the final contrastive alignment (Li et al., 28 Jul 2025). Its full objective is

$f_t(t)$ 9

Empirically, T2VParser improves STAN from 54.1% to 55.8% R@1 on MSR-VTT-1k, CLIP4Clip from 44.5% to 46.4%, and Mug-STAN from 57.3% to 58.4%; on long-text variants the gains grow to +3–5%, and under synthetic noise CLIP4Clip’s R@1 rises from 21.3% to 32.5% (Li et al., 28 Jul 2025).

“Asymmetric Visual Semantic Embedding” adopts a different strategy for information asymmetry. Its image branch samples the same image twice under different spatial biases by radial bias sampling, while the text branch produces a single view embedding via BERT (Liu et al., 10 Mar 2025). The Asymmetric Embedding Optimal Matching module partitions both modalities into meta-semantic embeddings and computes similarity by max-sum pooling,

$f_v(v)$ 0

instead of full cross-attention (Liu et al., 10 Mar 2025). On Flickr30K, AVSE raises text retrieval R@1 from 74.0% to 76.0% and image retrieval R@1 from 62.5% to 62.7%; on MS-COCO it raises text retrieval R@1 from 44.5% to 45.1% and image retrieval R@1 from 74.0% to 74.3% (Liu et al., 10 Mar 2025).

These methods share a common premise: alignment quality can improve when the model decomposes either the text, the visual content, or both, and performs matching at the level of semantic slices rather than whole vectors.

4. Long texts, token-level supervision, and cross-space alignment

A separate line of research addresses the mismatch between short-caption pre-training and longer textual inputs. “Meet At The Embedding” replaces the text encoder of the VLM with a pretrained LLM-based encoder and introduces a learnable projection module

$f_v(v)$ 1

implemented as either a small MLP with three linear layers or a single linear layer (Jang et al., 2024). Training proceeds in two stages without requiring explicit image–long-text pair annotations: text–text alignment on 16 M image captions, then 532 K MSMARCO query–document pairs, followed by image–text fine-tuning on 585 K image–caption pairs (Jang et al., 2024). On DOCCI test, zero-shot caption retrieval reaches 73.45 / 93.78 / 98.94 / 99.67 for R@1 / R@5 / R@25 / R@50; on Infoseek document retrieval, zero-shot mAP@5 / 10 / 25 / 50 reaches 14.51 / 19.29 / 24.95 / 27.44 (Jang et al., 2024). The paper also reports zero-shot multilingual transfer, for example Chinese captions on COCO-CN (Jang et al., 2024).

“SEA: Supervised Embedding Alignment for Token-Level Visual-Textual Integration in MLLMs” moves the alignment problem to visual tokens and LLM vocabulary embeddings. A frozen vision-LLM such as CLIP or SigLIP supplies semantic labels for each visual patch, the adapter output $f_v(v)$ 2 is contrasted with the corresponding LLM embedding $f_v(v)$ 3, and the model is optimized with the usual autoregressive loss plus a symmetric contrastive alignment loss (Yin et al., 2024). The method is designed to improve MLLMs without adding extra data or inference computation (Yin et al., 2024). On a Vicuna-7B backbone, the reported gains over LLaVA-1.5 include VQA-v2 78.8→79.1, GQA 62.0→63.2, ScienceQA 67.9→69.4, VizWiz 45.7→48.8, and MM-VET 30.7→31.9; the language retention result on MMLU is 47.1→46.4 with SEA, compared with 47.1→38.9 without pre-training (Yin et al., 2024).

“Inverse-LLaVA” challenges the assumption that vision–language alignment requires a separate pre-training stage. Instead of learning $f_v(v)$ 4, it learns a text-to-vision projection

$f_v(v)$ 5

and injects the projected text into selected transformer layers through additive fusion components in the attention mechanism (Zhan et al., 17 Aug 2025). The model uses 0 alignment samples, reports 45% lower total GPU cost, and presents a clear trade-off: MM-VET 31.2 (+0.1), VizWiz 50.95 (+0.95), ScienceQA-IMG 67.84 (+1.04), and MME cognitive reasoning +27.2%, alongside decreases in celebrity recognition (-49.5%) and TextVQA / OCR (-21.3%) (Zhan et al., 17 Aug 2025).

“Kernel-based Unsupervised Embedding Alignment for Enhanced Visual Representation in Vision-LLMs” offers another variant of cross-space alignment: it aligns CLIP’s visual representation with that of DINOv2 while maintaining compatibility with frozen text embeddings (Gong et al., 3 Jun 2025). After image-only alignment fine-tuning, the visual encoder retains compatibility with the frozen text encoder and exhibits significant improvements in zero-shot object recognition, fine-grained spatial reasoning, and localization; downstream MLLMs also demonstrate enhanced performance (Gong et al., 3 Jun 2025).

5. TVEA beyond retrieval: segmentation, generation, and representation editing

TVEA has been repurposed for tasks that are not standard image–text retrieval. In source-free cross-domain few-shot segmentation, the TVEA module sits alongside the Visual–Visual Embedding Alignment module and uses a frozen CLIP model to produce dense class activation maps from image–text prompts, thresholds those CAMs into pseudo-masks via Otsu’s method, and applies the self-supervised cross-entropy

$f_v(v)$ 6

to align TSAA rough masks with CLIP pseudo-masks (Liu et al., 7 Aug 2025). Under both 1-shot and 5-shot settings, the paper reports average segmentation accuracy improvements of 2.18\% and 4.11\%, respectively, across four cross-domain datasets (Liu et al., 7 Aug 2025).

In text-to-audio generation, “DiffAVA” fine-tunes lightweight visual-text alignment modules with frozen modality-specific encoders to update visual-aligned text embeddings as the condition (Mo et al., 2023). The architecture comprises a multi-head self-attention transformer for temporal aggregation of video features and a dual multi-modal residual network that fuses temporal visual representations with text embeddings; contrastive learning then matches visual-aligned text embeddings with audio features (Mo et al., 2023). On AudioCaps, DiffAVA reports Inception Score 7.37, KL 1.69, FAD 4.23, and FD 32.21, compared with AudioLDM’s 6.90, 1.97, 2.43, and 29.48 (Mo et al., 2023).

In diffusion-based text-to-image generation, “STEDiff” treats semantic alignment as a text-embedding problem. It strengthens sub-sentence semantics in the CLIP text-embedding space, uses the [EOT] token as a semantic anchor, amplifies singular values by

$f_v(v)$ 7

with $f_v(v)$ 8 and $f_v(v)$ 9, and applies the semantic-enhancement loss

$t$ 0

during the early reverse-diffusion steps (Zhang et al., 9 Jun 2026). On T2I-CompBench, STEDiff reports BLIP-VQA scores of 0.701, 0.697, and 0.610 on Color, Texture, and Shape, with human-preference scores of 0.623, 0.722, and 1.129 (Zhang et al., 9 Jun 2026).

“TEVI” addresses the modality gap by editing visual embeddings rather than retraining the vision encoder. A sparse autoencoder disentangles frozen CLIP image embeddings into a latent representation $t$ 1, and a text-conditioned masking module produces $t$ 2 so that the conditioned embedding is reconstructed as

$t$ 3

The masking module is trained with a modified InfoNCE objective over conditioned embeddings (Mahajan et al., 5 Jun 2026). TEVI improves retrieval on both short- and long-caption benchmarks: MS COCO image→text R@1 32.98→35.66 and text→image R@1 21.38→23.12; Flickr30k 59.66→64.20 and 42.46→44.75; DOCCI 20.38→24.20 and 7.16→8.55; IIW 50.98→55.72 and 16.88→19.37 (Mahajan et al., 5 Jun 2026).

TVEA also appears in zero-shot skeleton-based action recognition through “Dual Visual-Text Alignment,” which combines Direct Alignment, Augmented Alignment, and Semantic Description Enhancement. On NTU-60, the reported accuracies are 79.28 for the 55/5 split and 44.14 for the 48/12 split; on NTU-120 they are 74.89 for 110/10 and 51.81 for 96/24 (Kuang et al., 2024).

6. Evaluation, robustness, and recurring trade-offs

Evaluation protocols for TVEA vary with the task. Retrieval work typically reports Recall@K or mAP@K; segmentation work reports average segmentation accuracy; and some alignment studies evaluate alignment indirectly through downstream error modes. The typographic prompt injection literature is especially explicit about this latter strategy (Balakrishnan et al., 14 Apr 2026).

“Reading Between the Pixels” measures the Euclidean distance between the embedding of a clean text prompt and the embedding of the same prompt rendered as an image, using JinaCLIP and Qwen3-VL-Embedding as embedding backbones (Balakrishnan et al., 14 Apr 2026). Across GPT-4o, Claude Sonnet 4.5, Mistral-Large-3, and Qwen3-VL-4B-Instruct, the paper reports strong negative correlations between text-image embedding distance and attack success rate: $t$ 4, $t$ 5, $t$ 6, and $t$ 7 for JinaCLIP, and $t$ 8, $t$ 9, $t$ 0, and $t$ 1 for Qwen3-VL-Embedding, all with $t$ 2 (Balakrishnan et al., 14 Apr 2026). At 6 px, mean distance is 1.265 (std = 0.032); at 28 px, mean distance is 1.090 (std = 0.045), a 14% reduction (Balakrishnan et al., 14 Apr 2026). Heavy blur increases distance to 1.244 (+12%) and reduces attack success sharply, for example Claude 18.0%→0.7% and Mistral 74.5%→26.7%; triple degradation produces $t$ 3 (+10.5%) with corresponding attack reductions, while rotation has asymmetric effects across models (Balakrishnan et al., 14 Apr 2026).

These findings complicate a simple “more alignment is always better” narrative. In the typographic-attack setting, robust text–visual alignment is both a prerequisite for correct reading and a double-edged sword, because lower embedding distance is associated with higher attack success (Balakrishnan et al., 14 Apr 2026). In text-to-video retrieval, full-representation alignment can be incorrect supervision because only part of the video is described (Li et al., 28 Jul 2025). In multimodal instruction following, Inverse-LLaVA shows that eliminating explicit alignment pre-training can improve reasoning-intensive and cognitive tasks while reducing performance on perception tasks requiring memorized visual-text associations (Zhan et al., 17 Aug 2025).

The forward directions named in the literature are correspondingly diverse. MATE identifies image–long-text retrieval, document retrieval, and zero-shot multilingual transfer as active settings for embedding-based connection between images and long texts (Jang et al., 2024). T2VParser presents partial alignment as a modality-agnostic mechanism that could extend to image–text retrieval, video question answering, captioning, and efficient indexing (Li et al., 28 Jul 2025). Inverse-LLaVA raises open questions about multi-layer fusion, hybrid schemes that combine text→vision inversion with vision→text projection, and extension to other continuous modalities such as audio, sensor data, and molecular graphs (Zhan et al., 17 Aug 2025).

Taken together, the literature describes TVEA not as a single settled recipe but as a spectrum of alignment strategies: global versus local, full versus partial, static versus token-level, pre-training versus post hoc editing, and retrieval-oriented versus task-conditioned. That breadth is one reason the term remains productive across contemporary vision-language research.