Text2VLM: Multimodal Safety & Alignment
- Text2VLM is a multimodal evaluation framework that adapts text-only benchmarks into paired text and typographic images to reveal alignment vulnerabilities.
- It employs salient concept extraction to shift harmful content into images, challenging VLMs with OCR-like reconstruction and cross-modal integration.
- Empirical results show that typographic prompt injection reduces refusal behavior, indicating gaps in safety alignment for current VLMs.
Searching arXiv for the specified paper and closely related Text2VLM-style work to ground the article in current literature. Text2VLM is a dataset-adaptation and evaluation pipeline for visual LLMs (VLMs) that converts text-only safety benchmarks into multimodal prompts combining ordinary text with typographic images, with the explicit goal of testing alignment under typographic prompt injection (Downer et al., 28 Jul 2025). In its original formulation, the method addresses a blind spot in VLM safety assessment: a model that appears well aligned under text-only prompting may become vulnerable when harmful intent is split across textual and visual channels and must be reconstructed through OCR-like reading and cross-modal integration. In adjacent literature, the same general intuition also appears in broader systems that render text as images, compress documents into visual tokens, or translate structured visual content into textual intermediates for downstream reasoning (Xie et al., 7 May 2026).
1. Definition and threat model
Text2VLM is situated in multimodal alignment evaluation rather than in conventional jailbreak optimization. Its central claim is that current alignment evaluation for VLMs is incomplete because it relies heavily on text-only prompting even though deployed VLMs operate over both language and vision (Downer et al., 28 Jul 2025). The consequence is a specific failure mode: harmful intent can be hidden in an image, while the surrounding text merely instructs the model to retrieve and use that content.
The threat model is typographic visual prompt injection. An attacker starts from a harmful text-only request and relocates the dangerous or intent-defining phrases into a separate image. The remaining textual scaffold contains placeholders such as “<insert item 1 from the attached image>,” which forces the VLM to perform OCR-like reconstruction, align numbered placeholders with numbered image items, and then decide whether to comply or refuse. In this framing, vulnerability arises from the conjunction of two tasks: multimodal comprehension and safety alignment. A refusal is meaningful only if the model has first recognized the harmful semantics distributed across modalities.
The method is explicitly motivated against prior multimodal jailbreak lines such as HADES, Visual Roleplay, FigStep, and MM-SafetyBench. Those methods are described as requiring expensive optimization, being limited to specific prompt structures, depending on rephrasing that may distort intent, or relying on static scenario sets. Text2VLM instead aims to be scalable, dataset-agnostic, and reusable across domains by adapting arbitrary text-only benchmarks into multimodal instances (Downer et al., 28 Jul 2025).
2. Pipeline architecture
The Text2VLM pipeline begins with a text-only dataset and converts each prompt into a text-plus-image pair through a multi-stage procedure (Downer et al., 28 Jul 2025). If a source prompt exceeds 200 characters, it is summarized using Dolphin-2.9-Llama3-8B. The paper states that this step is necessary because open-source VLMs often have limited OCR ability, so rendering too much text into a typographic image may make the multimodal input unintelligible. Prompts under 200 characters are left unchanged.
The next stage is salient concept extraction and tagging. GPT-4o-mini identifies the “salient concepts” that make the prompt harmful, or whose removal or substitution would make it benign. These extracted words or phrases are removed from the textual prompt and replaced with numbered placeholders. The result is not a full text-to-image conversion. Instead, the semantically dangerous content is selectively transferred into the visual channel while the request structure remains in text. This selectivity is central to the method because it preserves the original task semantics while shifting the alignment-critical fragments into an image.
The extracted concepts are then rendered into a typographic image using Matplotlib as a numbered list. The design is intentionally typographic rather than naturalistic. The paper argues that many harmful concepts, especially in cybersecurity, are abstract and not naturally representable through ordinary images; terms such as “SQL injection,” “sandbox escape,” or “data obfuscation” are better preserved as text rendered in image form. This also makes the attack more controlled, because the transferred content is the original phrase rather than a paraphrase or visual metaphor.
The final multimodal prompt pairs the modified textual scaffold with the generated image. The text instructs the VLM to retrieve the numbered items from the image; the image supplies them. The target model must therefore read the image, align it with the placeholders, reconstruct the original request, and then answer or refuse (Downer et al., 28 Jul 2025).
3. Evaluation design, metrics, and benchmark sources
Text2VLM separates two questions: whether the model understood the multimodal prompt at all, and whether it remained aligned after understanding it (Downer et al., 28 Jul 2025). Relevance is judged by GPT-4o-mini using a prompt that checks semantic relevance, completeness, and correctness with respect to the original query. Refusal is determined by a rule-based classifier using refusal markers such as “I’m unable,” “It is not ethical,” sentences beginning with “As a” or “As an” that mention AI or LLM, and keywords such as “illegal” or “unethical.”
The paper defines understanding as follows:
This definition treats refusal as evidence of understanding, because a correct refusal implies that the model reconstructed the harmful request. Unsafe response rate is defined conceptually as the number of times the model did not give a refusal but also gave a relevant answer.
The adapted datasets span four malicious domains plus one benign control set.
| Dataset | Role in Text2VLM | Domain |
|---|---|---|
| MITRE | Malicious benchmark | Cyberattacks |
| Interpreter | Malicious benchmark | Code-interpreter exploitation |
| ToxiGen | Malicious benchmark | Hate speech and toxic language |
| MedSafetyBench | Malicious benchmark | Unsafe medical advice |
| MT-Bench | Benign control | Non-malicious task understanding |
The pipeline is also validated by human evaluation on a random subset of 80 data points, with 20 from each of the four malicious datasets. Summarization quality was rated Great or Good for 91.25% of reviewed points, all extracted salient concepts were valid in 93.75% of cases, and no salient concepts were missed in 75% of cases. On the output side, 87% of relevance judgments were rated Good or Great, and 93% of refusal classifications were correct (Downer et al., 28 Jul 2025).
4. Empirical findings on alignment degradation
The experiments evaluate four open-source VLMs: LLaVA-1.6 34B, LLaVA-1.6 7B, VILA-1.5 40B, and VILA-1.5 8B (Downer et al., 28 Jul 2025). The critical comparison is always between the original text-only input and the corresponding Text2VLM multimodal version, so the measured effect is a modality shift rather than a change in underlying task intent.
The first major result is a decline in task understanding once prompts are split across text and typographic image. On the benign MT-Bench control, text-only understanding is near perfect, but understanding drops when words are removed from the text and placed into an image. The same degradation appears on the malicious datasets. The discussion attributes this to weak OCR and multimodal integration in current open-source VLMs, and VILA 8B is described as especially poor.
The more consequential safety result is that, among prompts the model did understand, multimodal typographic injection reduced refusal rates relative to text-only prompting. This indicates that the visual channel weakens alignment defenses. The paper emphasizes MedSafetyBench as the clearest case: text-only medical prompts were comparatively well defended, but refusal rates dropped considerably after Text2VLM transformation. In MITRE and ToxiGen, the effect was less dramatic because baseline text-only safety was already weak, leaving less room for additional degradation. The overall conclusion is a consistent reduction in refusal behavior and a corresponding rise in unsafe relevant responses when harmful content is redistributed into visual form (Downer et al., 28 Jul 2025).
A frequently cited claim in the paper’s abstract is “a significant performance gap compared to closed-source frontier models.” Within the paper itself, however, that comparison is not directly tested. No closed-source frontier model is evaluated; the claim is inferential and presented as broader context rather than as an experimental result. This distinction matters because it constrains what the reported evidence actually establishes (Downer et al., 28 Jul 2025).
5. Position within the broader Text2VLM literature
A plausible broader reading of the literature is that “Text2VLM” now refers both to the specific alignment-evaluation pipeline introduced in (Downer et al., 28 Jul 2025) and to a wider family of methods that route textual content through visual or multimodal interfaces. In this wider sense, the literature covers at least three recurrent operations: rendering text as images, compressing text into compact multimodal representations, and using text-conditioned signals to control what visual evidence a model processes.
LensVLM renders long text as compressed images, lets a VLM scan them, and then selectively expands only the relevant images through tool use. Its main contribution is a selection-then-expansion framework that maintains accuracy comparable to the full-text upper bound at 4.3x effective compression and remains competitive up to 10.1x effective compression across seven text QA benchmarks (Xie et al., 7 May 2026). The relation to Text2VLM is direct: text becomes a visual input, but the system adds dynamic decompression rather than relying on a single fixed rendering.
DocVLM addresses a different but adjacent problem: it converts OCR text plus 2D layout into a compact learned token representation that can be inserted into a frozen VLM. With 64 learned queries, it improves low-token document understanding substantially, including DocVQA gains from 56.0% to 86.6% with InternVL2 and from 84.4% to 91.2% with Qwen2-VL in the limited-token regime (Nacson et al., 2024). Here the textual content is not merely appended as prompt text; it is encoded as an auxiliary modality.
T2Vid, introduced in the paper titled “Sparrow,” translates long-context text instruction data into ordered sequences of text-rendered images so that image-LLMs can be adapted to video understanding with much less real video supervision. A 30K hybrid mixture of 20K real video samples and 10K synthetic text-derived pseudo-video samples surpasses a 200K real-video-only setup on Video-MME overall (Yin et al., 2024). This extends the same modality-bridging logic from safety evaluation to video-LLM training.
TextFlow takes the complementary direction of converting visual structure into explicit text. It uses a Vision Textualizer to produce Graphviz, Mermaid, or PlantUML from flowchart images and a Textual Reasoner to answer downstream questions from those serializations. On FlowVQA, it reports 82.74 versus 76.61 for the best end-to-end baseline, and on FlowLearn 80.57 versus 77.00 (Ye et al., 2024). This suggests that, in structured domains, an explicit text bottleneck can improve controllability, explainability, and tool use.
Related work also shows that text can guide visual processing directly rather than only being rendered into images. FlashVLM uses text-guided visual token selection, computing explicit cross-modal similarity in the language-model space and retaining only a compact subset of visual tokens. Under identical token budgets, it slightly surpasses the unpruned baseline while pruning up to 77.8% of visual tokens on LLaVA 1.5 and maintains 92.8% accuracy under 94.4% compression (Cai et al., 23 Dec 2025). Text-Animator, in turn, treats visible text as a first-class object in video generation by injecting glyph, position, and camera controls, achieving Sentence Accuracy and NED on its benchmark (Liu et al., 2024).
6. Limitations, misconceptions, and future directions
The immediate limitations of Text2VLM are operational rather than conceptual (Downer et al., 28 Jul 2025). Summarization for prompts longer than 200 characters introduces possible fidelity loss relative to the original benchmark. Salient concept extraction is imperfect: in 25% of reviewed cases, one or more important concepts were missed. Typographic images can be distorted if extracted phrases are too long or too few, because VLM image preprocessing may stretch or crop the rendered text. The refusal classifier is heuristic string matching and may miss nuanced refusals or misclassify safe explanations.
A second limitation is scope. Typographic rendering is a meaningful proxy for OCR-mediated multimodal risk, but it does not cover the full space of image-based attacks, such as natural-image prompt injection, hidden text, steganography, or cross-modal distractors (Downer et al., 28 Jul 2025). Text2VLM therefore exposes one major class of multimodal vulnerability rather than exhausting the problem.
A common misconception is to treat Text2VLM as merely another jailbreak attack. The paper’s own framing is different: the method is primarily an evaluation framework for converting existing text benchmarks into multimodal tests while preserving underlying task semantics (Downer et al., 28 Jul 2025). Its purpose is to measure whether alignment transfers across modalities, not simply to maximize attack success on a fixed target.
The broader literature suggests several likely future directions. One is stronger OCR-aware moderation and multimodal refusal policy design, since both Text2VLM and document-oriented systems such as DocVLM indicate that text embedded in images is a distinct operational bottleneck (Nacson et al., 2024). Another is dynamic context management, as in LensVLM’s selective expansion, which suggests that compressed visual text can be used for coarse localization while finer textual recovery is invoked only when needed (Xie et al., 7 May 2026). A plausible implication is that robust future VLM alignment will require joint treatment of OCR, cross-modal reconstruction, and safety classification, rather than separate text-safety and image-safety components.
In that sense, Text2VLM marks a shift in evaluation philosophy. Text-only safety scores cannot be assumed to transfer to VLMs, because harmful intent can be reintroduced through typographic images in ways that preserve understanding while reducing refusal behavior. The method’s lasting significance lies less in a single attack template than in establishing multimodal prompt reconstruction itself as an alignment benchmark (Downer et al., 28 Jul 2025).
