T2I-VeRW: Diverse Benchmarks for T2I Models

• T2I-VeRW is a multi-use term covering diverse benchmarks such as vehicle retrieval with fine-grained annotations, adversarial robustness, and commonsense visual reasoning challenges.
• The vehicle retrieval dataset [2605.06012] features 14,668 images and 1,796 vehicle IDs with detailed part-level segmentation achieved via a two-stage pipeline combining automated and manual text generation.
• Research explores angle-robust embeddings for maintaining adversarial patch effectiveness across viewpoints and evaluates commonsense consistency in generated images.

In current arXiv usage, T2I-VeRW is not a single standardized expansion. The designation appears as a new large-scale dataset called T2I-VeRW for text-to-image vehicle re-identification (Wang et al., 7 May 2026), as “Text-to-Image View Robustness” in work on physically view-invariant text-to-image adversarial patches (Ji et al., 11 Jun 2025), and as “Text-to-Image Visual Reasoning and World knowledge” in a rebranding of the Commonsense-T2I Challenge (Fu et al., 2024). The term therefore denotes distinct research objects across cross-modal retrieval, physical adversarial robustness, and commonsense evaluation, and its meaning must be resolved from local paper context rather than from the acronym alone.

1. Nomenclature and scope

Usage in the literature	Expansion	Reference
Vehicle retrieval benchmark	T2I-VeRW	(Wang et al., 7 May 2026)
Physical adversarial robustness	Text-to-Image View Robustness	(Ji et al., 11 Jun 2025)
Commonsense evaluation benchmark	Text-to-Image Visual Reasoning and World knowledge	(Fu et al., 2024)

A separate consolidated description also presents VersaT2I as a “T2I-VeRW” framework, even though the paper title is “VersaT2I: Improving Text-to-Image Models with Versatile Reward” (Guo et al., 2024). This suggests that acronym-level comparisons are potentially misleading unless anchored to the full paper title and task definition.

2. T2I-VeRW as a benchmark for text-to-image vehicle retrieval

In (Wang et al., 7 May 2026), T2I-VeRW is introduced as a new large-scale dataset called T2I-VeRW, constructed for text-to-image vehicle re-identification. The motivating scenario is explicit: in many real-world surveillance and forensic settings, the only available cue is a witness’s verbal description, whereas conventional vehicle Re-ID systems require an image of the target. The benchmark is positioned against existing datasets such as T2I-VeRI, which are described as severely limited in scale (<2.5K images, 776 IDs) and annotation granularity.

The dataset statistics are specific. T2I-VeRW contains 14,668 images covering 1,796 vehicle identities with fine-grained part-level annotations. The annotation schema includes pixel-wise segmentation masks (discretized to 24×24) for six vehicle components: Windows, Wheels, Doors, Mirrors, Lights, and Roof. The text side consists of rich, ∼50-word average captions covering body color, grille style, headlight shape, wheel design, roof accessories, window tint, bumper details, etc.

The paper’s broader claim is that T2I-VeRW provides a challenging testbed for fine-grained cross-modal vehicle matching. This is tied to three concrete attributes emphasized in the dataset description: Scale, Granularity, and Linguistic richness. In that formulation, T2I-VeRW is not merely a larger benchmark; it is intended to force models to resolve the relationship between textual descriptions and rigid, symmetric, view-dependent vehicle geometry.

3. Data acquisition, annotation, and split protocol

The image corpus is derived from the VERI-Wild surveillance dataset, with vehicles captured across hundreds of non-overlapping urban cameras (Wang et al., 7 May 2026). The covered viewpoints are front, rear, left-side, and right-side views under varied lighting (day/night), weather (sunny, overcast), and exposure settings. The environmental conditions explicitly include occlusion, motion blur, and sensor noise, so the benchmark reflects operational surveillance imagery rather than studio-style captures.

Text descriptions are produced in a two-stage pipeline. Qwen3-VL-32B multimodal LLM is used for automatic generation, with prompts requesting details on grille style, headlight shape, wheel spokes, roof features, window layout, bumper design. These captions are then subjected to manual verification, where human annotators review and refine each caption for fluency and correctness. The resulting text statistics are reported as Avg. caption length = 50.35 words (vs. 27.6 in T2I-VeRI), Max text tokens (CLIP tokenizer) = 77, and a sentence length distribution: 30–70 words (peak at 45–55).

Part-level segmentation is produced with SAM3 (Segment Anything Model 3) using multiple textual prompts per category, such as “car windows, windshield, side window, rear window”. Each part mask is stored as a 24×24 binary grid. Quality control is concrete: Random sampling of 5% of mask overlays for visual inspection; masks with mis-segmentations are re-prompted and corrected.

The split protocol uses a non-overlapping 7:3 identity split to ensure no shared vehicles between splits. The resulting partitions are Training Set: 10,265 image-text pairs, 1,251 IDs and Validation Set: 4,403 image-text pairs, 545 IDs. This split design makes the retrieval problem identity-disjoint by construction, which is standard for Re-ID evaluation and particularly important for cross-modal generalization.

4. Evaluation protocol and model performance on the vehicle-retrieval benchmark

The evaluation protocol follows standard cross-modal retrieval practice and uses Rank-k accuracy and mean Average Precision (mAP) (Wang et al., 7 May 2026). For a query set of size $Q$, Rank-k is defined as

$$\mathrm{Rank\text{-}k}=\frac{1}{Q}\sum_{q=1}^{Q}\mathbf{1}\bigl(\mathrm{pos}_q \le k\bigr),$$

where $\mathrm{pos}_q$ is the rank position of the first true match for query $q$. The benchmark also reports

$$\mathrm{mAP}=\frac{1}{Q}\sum_{q=1}^{Q}AP_q,\qquad AP_q=\frac{1}{m_q}\sum_{k=1}^{N}P_q(k)\,\mathrm{rel}_q(k),$$

with $m_q$ ground-truth matches and $N$ the gallery size per query.

The principal model introduced with the benchmark is PFCVR, described as a Part-level Fine-grained Cross-modal Vehicle Retrieval model. PFCVR constructs locally paired images and texts at the part level and introduces learnable part-query tokens that aggregate both part-specific and full-sentence context before aligning with visual part features. On top of this explicit local alignment, a bi-directional mask recovery module allows each modality to reconstruct its masked content under the guidance of the other, thereby implicitly bridging local correspondences into global feature alignment.

The reported benchmark results on T2I-VeRW are:

• TIPCB (2022): Rank-1 29.1%, Rank-5 53.3%, Rank-10 64.5%, mAP 17.2%
• LGUR (2022): Rank-1 44.2%, Rank-5 73.0%, Rank-10 83.1%, mAP 18.5%
• SSAN (2021): Rank-1 39.1%, Rank-5 67.3%, Rank-10 78.6%, mAP 21.6%
• TransReID (2021): Rank-1 22.0%, Rank-5 41.7%, Rank-10 51.0%, mAP 12.4%
• ALBEF (2021): Rank-1 22.8%, Rank-5 49.2%, Rank-10 52.3%, mAP 15.6%
• IRRA (2023): Rank-1 47.3%, Rank-5 75.6%, Rank-10 85.3%, mAP 22.8%
• UP-Person (2025): Rank-1 50.7%, Rank-5 80.6%, Rank-10 88.7%, mAP 22.6%
• FMFA (2025): Rank-1 47.8%, Rank-5 78.9%, Rank-10 87.6%, mAP 22.1%
• GA-DMS (2025): Rank-1 52.2%, Rank-5 81.4%, Rank-10 89.9%, mAP 23.5%
• VFE-TPS (2025): Rank-1 53.7%, Rank-5 82.0%, Rank-10 89.7%, mAP 24.4%
• MARS (2025): Rank-1 52.8%, Rank-5 78.9%, Rank-10 86.3%, mAP 25.2%
• PFCVR (2026): Rank-1 55.2%, Rank-5 82.8%, Rank-10 90.3%, mAP 26.2%

Two summary comparisons are highlighted. First, PFCVR outperforms the second-best (VFE-TPS) by +1.5% on Rank-1 and +1.8% on mAP. Second, on T2I-VeRI, PFCVR achieves 29.2% Rank-1 accuracy, improving over the best competing method by +3.7% percentage points. The module analysis reported on T2I-VeRI attributes +1.5% Rank-1 to Part-level Local Fine-grained Alignment (PLFA) and another +1.5% Rank-1 to Bidirectional Mask Recovery Implicit Alignment (BMRIA), with further validation that explicit part grounding and cross-modal reconstruction are complementary.

5. T2I-VeRW as text-to-image view robustness for physical adversarial patches

In (Ji et al., 11 Jun 2025), T2I-VeRW (“Text-to-Image View Robustness”) denotes the ability of adversarial patches generated by text-to-image diffusion models to maintain their attack effectiveness when observed from arbitrary viewpoints in the physical world. The definition is explicit: a patch should continue to trigger a high false-positive detection confidence across the angular spectrum $\theta\in[-90^\circ,90^\circ]$, or in physical tests $\theta\in[-70^\circ,70^\circ]$ at 10° steps.

The fundamental challenge is also stated explicitly. First, the 2D image appearance changes nonlinearly under perspective projection. Second, most existing T2I methods optimize for a single or narrow range of viewpoints. Third, naive prompt engineering such as “detectable from all angles” does not translate into robustness, because the model’s text encoder cannot internalize such abstract, goal-oriented instructions. The core failure mode is that a single text embedding used to steer the generator typically encodes only a single canonical appearance; off-axis viewing introduces geometric foreshortening, shading changes and loss of distinctive features that break the adversarial effect.

The proposed remedy is Angle-Robust Concept Learning (AngleRoCL), which learns a dedicated “angle-robust” concept embedding $\mathbf{F}_c$. This embedding can be inserted into prompts through a learnable concept token, so that a generated patch is guided toward inherent resistance to viewpoint variations. In the implementation described, the method uses a pre-trained T2I diffusion model (e.g. Stable Diffusion v1.5) and optimizes only the concept embedding. Training samples nine angles:

$$\mathrm{Rank\text{-}k}=\frac{1}{Q}\sum_{q=1}^{Q}\mathbf{1}\bigl(\mathrm{pos}_q \le k\bigr),$$0. To reduce rendering cost, the method generates one frontal view and then applies a projective warp to approximate oblique angles. Optimization uses AdamW (learning rate $$\mathrm{Rank\text{-}k}=\frac{1}{Q}\sum_{q=1}^{Q}\mathbf{1}\bigl(\mathrm{pos}_q \le k\bigr),$$1) for 50 000 steps, with an angle-aware detection loss and scaling factor $$\mathrm{Rank\text{-}k}=\frac{1}{Q}\sum_{q=1}^{Q}\mathbf{1}\bigl(\mathrm{pos}_q \le k\bigr),$$2 set to 10.

Deployment is plug-and-play. After training, the “<angle-robust>” token is frozen and can be inserted into prompts such as “a <angle-robust> blue square stop sign” or “a <angle-robust> stop sign with ‘hello’ on it”. The experimental protocol spans Faster R-CNN, YOLOv3, YOLOv5, RT-DETR and the latest YOLOv10. Digital testing uses $$\mathrm{Rank\text{-}k}=\frac{1}{Q}\sum_{q=1}^{Q}\mathbf{1}\bigl(\mathrm{pos}_q \le k\bigr),$$3 in 1° steps, and physical testing uses $$\mathrm{Rank\text{-}k}=\frac{1}{Q}\sum_{q=1}^{Q}\mathbf{1}\bigl(\mathrm{pos}_q \le k\bigr),$$4 in 10° increments, with patches printed on paper and affixed to real stop signs.

The main metric is Angle-Aware Attack Success Rate (AASR):

$$\mathrm{Rank\text{-}k}=\frac{1}{Q}\sum_{q=1}^{Q}\mathbf{1}\bigl(\mathrm{pos}_q \le k\bigr),$$5

The quantitative results are reported as follows. In digital tests, average AASR across six nuImage scenes is 0.78% for AdvPatch, 23.79% for NDDA, 36.02% for NDDA+AngleRoCL with +51.4% relative, 26.26% for MAGIC, and 32.51% for MAGIC+AngleRoCL with +23.8% relative. In physical tests, average AASR is 0.00% for AdvPatch, 28.37% for NDDA, 51.75% for NDDA+AngleRoCL with +82.4% relative, 22.72% for MAGIC, and 65.86% for MAGIC+AngleRoCL with +189.9% relative.

The paper interprets these results as evidence that textual concept learning (via a single learned embedding) can encode complex physical-world invariances—here, viewpoint robustness—beyond what ad-hoc prompt engineering achieves. It further reports that the learned token systematically up-weights non-robust yet discriminative features such as color textures and high-frequency patterns, supported by cosine-similarity analysis against CLIP tokens for “red”, “blue”, “checkerboard”, etc. A plausible implication is that T2I-VeRW, in this sense, functions as a bridge between prompt semantics and physical attack invariance.

6. T2I-VeRW as visual reasoning and world knowledge

In (Fu et al., 2024), T2I-VeRW (Text-to-Image Visual Reasoning and World knowledge) is described as essentially the Commonsense-T2I Challenge rebranded. Its purpose is to evaluate whether a text-to-image model can “fill in the gaps” left by a brief prompt using everyday, real-world commonsense. The benchmark therefore departs from conventional fidelity or object-composition tests and instead asks whether a model’s outputs conform to physics, biology, or social practice.

The dataset contains 150 hand-curated test cases organized as adversarial prompt pairs. For each case $$\mathrm{Rank\text{-}k}=\frac{1}{Q}\sum_{q=1}^{Q}\mathbf{1}\bigl(\mathrm{pos}_q \le k\bigr),$$6, there are two prompts $$\mathrm{Rank\text{-}k}=\frac{1}{Q}\sum_{q=1}^{Q}\mathbf{1}\bigl(\mathrm{pos}_q \le k\bigr),$$7 and $$\mathrm{Rank\text{-}k}=\frac{1}{Q}\sum_{q=1}^{Q}\mathbf{1}\bigl(\mathrm{pos}_q \le k\bigr),$$8 with small linguistic differences, such as “A lightbulb without electricity” versus “A lightbulb with electricity”, paired with mutually exclusive expected outputs such as “The lightbulb is unlit” and “The lightbulb is lit.” Each case also carries a likelihood score on a 0–10 scale, and cases scoring below 7 were discarded during curation. The annotation workflow is defined in four steps: commonsense categories are defined up front, GPT-4-turbo is used to propose candidate scenarios, experts rewrite and refine each pair, and a second pass removes ambiguities.

The benchmark taxonomy consists of Physical Laws (32.7%), Human Practices (30.0%), Biological Laws (11.3%), Daily Items (14.0%), and Animal Behaviors (12.0%). Evaluation is pairwise and strict: a sample is counted as correct only if each prompt yields its own expected output and not the other’s. Average accuracy is computed over $$\mathrm{Rank\text{-}k}=\frac{1}{Q}\sum_{q=1}^{Q}\mathbf{1}\bigl(\mathrm{pos}_q \le k\bigr),$$9 samples.

The reported results show a substantial commonsense gap. The human (expert) upper bound is 100% (by construction). Model accuracies are 18.8% for Stable Diffusion v2.1, 24.9% for Stable Diffusion XL, approximately 20% for Openjourney v4, approximately 20% for Playground v2.5, approximately 20% for Latent Consistency Models, 34.0% for DALL·E 3 w/o GPT revision, and 48.9% for DALL·E 3 (default, with GPT prompt enrichment). The prompt-enrichment ablation reports that disabling GPT-based prompt revision on DALL·E 3 drops accuracy by ≈15 points, which confirms that richer text context helps but does not close the gap.

The benchmark’s analysis attributes these failures to embedding ambiguity, insufficient commonsense knowledge, and prompt-engineering limits. A specific observation is that there is a strong correlation between CLIP text-embedding similarity of $\mathrm{pos}_q$0 vs. $\mathrm{pos}_q$1 and low pairwise accuracy. Future directions proposed in the paper include integrating structured commonsense graph embeddings, contrastive fine-tuning on paired adversarial examples, end-to-end multimodal LLM-diffusion hybrids, and data augmentation with synthetic commonsense scenes. In this usage, T2I-VeRW serves as an evaluation framework for whether generation models can transform minimal textual contrasts into the qualitatively different images demanded by commonsense.