Novel Visual References Dataset (NVRD)

• The paper demonstrates the use of controlled perturbation sequences to probe fast mapping and generalization of new visual concepts using nonce labels.
• The dataset comprises 19,176 images across 90 objects, split into known, composed, and fully novel entities to test conflicts with prior training.
• Systematic quality control and human–model comparisons reveal that models often overgeneralize novel visual references compared to human judgments.

The Novel Visual References Dataset (NVRD) is a corpus and benchmark designed to probe how vision–LLMs (VLMs) and humans acquire and generalize truly new visual concepts after minimal exposure, especially when those concepts contradict prior pretraining knowledge (Tür et al., 3 Jun 2026). It contains 19,176 images spanning 90 visual concepts across different levels of visual novelty, each paired with a nonce word and accompanied by controlled perturbation sequences that systematically increase visual distance from an original object. In the associated study, Ada Defne Tür, Gaurav Kamath, Joyce Chai, Siva Reddy, and Benno Krojer evaluate 3 open- and 2 closed-source models alongside 2,400 human judgments, using NVRD to compare fast mapping, generalization, and sensitivity to visual change in humans and machines (Tür et al., 3 Jun 2026).

1. Conceptual scope and motivation

NVRD was introduced to study fast mapping of new visual references to language in VLMs and humans, and to evaluate whether models learn and generalize novel concepts with human-like biases, particularly when new labels conflict with prior knowledge (Tür et al., 3 Jun 2026). The dataset targets a setting that the paper describes as largely underexplored: how learners map novel visual references to language after exposure, especially when those references contradict prior knowledge from pre-training.

The central notion of a “novel visual reference” is a new visual concept that is unfamiliar to the learner. NVRD operationalizes this with nonce labels such as “dax” and “blomwich,” forcing new mappings between image and language. These mappings are assigned to three types of referents: fully novel entities, composed entities, and known entities. Fully novel entities are stimuli designed to not correspond to any real-world object. Composed entities are merged or texture-transferred hybrids of known objects. Known entities are familiar objects renamed with nonce words, inducing conflict with pretraining knowledge.

For known entities, the contradiction is explicit: models must apply a novel label to a familiar object, such as a “chair” labeled “blomwich.” The paper states that this creates tension with pretraining mappings and surfaces mutual exclusivity-like behavior, namely a preference for vanilla names over introduced nonce labels (Tür et al., 3 Jun 2026). This design distinguishes NVRD from work on visual augmentations of familiar concepts, because its stimuli are entirely novel and open-ended, constructed from scratch to mirror how humans encounter genuinely new concepts.

2. Dataset composition and stimulus taxonomy

NVRD contains 19,176 images total and 90 base objects, equally divided across three novelty categories (Tür et al., 3 Jun 2026). Each base object has up to 20 compounding levels per perturbation axis across 11 axes, although some axes saturate earlier.

Component	Count	Description
Total images	19,176	Images across all concepts and perturbations
Base objects	90	Visual concepts in three novelty categories
Known entities	30	Standard objects likely present in pretraining data
Composed entities	30	Shape–shape and shape–texture compositions
Fully novel entities	30	Entirely new objects built via design specifications

The three novelty categories are defined by distinct criteria. Known entities are standard objects likely present in pretraining data; conflict is induced by assigning a nonce word to a familiar referent. Composed entities are novel combinations of known components, including shape–shape compositions, in which two objects are merged into one cohesive entity, and shape–texture compositions, in which a known shape is rendered with an atypical texture such as crochet. Fully novel entities are entirely new objects built via design specifications, including silhouettes, materials, structural rules, surface details, and palettes; the paper states that they do not correspond to any real-world object (Tür et al., 3 Jun 2026).

The stimulus design pipeline uses Gemini-3 Pro Image to generate base images with white background and realistic style for known and composed items. For fully novel entities, prompt compositions specify silhouettes, materials, structural rules, surface details, and palettes. Examples given in the paper include a “lopsided hourglass with one chamber collapsed,” “slimy translucent elastomer with cloudy streaks,” “exactly six curly protrusions alternating direction,” “puckered dimples scattered unevenly,” and “sickly pastel green with oily black shadows” (Tür et al., 3 Jun 2026).

Each base image is paired with a nonce word $r$, constructed by prompting GPT-4o for candidates and filtering to exactly three tokens. Metadata tracked per image include object category, perturbation axis, level index, nonce word $r$, and, for higher-level edits, generator prompts used. Images are collected per concept and per perturbation axis/level, with accompanying JSON/CSV metadata. The dataset is presented as a benchmark, and no train/validation/test split is mandated (Tür et al., 3 Jun 2026).

3. Perturbation framework and quality control

A defining feature of NVRD is its perturbation scheme: controlled sequences that probe generalization and sensitivity to specific visual dimensions such as shape, parts, texture, style, background, and color (Tür et al., 3 Jun 2026). For a base image $x_0$ and perturbation $p$, the general sequence is

$$x_0 \rightarrow x_1 \rightarrow x_2 \rightarrow \dots \rightarrow x_L$$

with each $x_\ell$ applying $p$ to $x_{\ell-1}$, ideally yielding a monotonically increasing visual distance from $x_0$.

The 11 perturbation axes divide into low-level edits and higher-level edits. The low-level edits are programmatic and do not change shape or structure: Gaussian Noise, Scale, Pixelation, JPEG Compression, and Color Shift. The higher-level edits are Texture Shift, Background Replacement, Artistic Style, Shape Deformation, Part Addition, and Part Removal. Texture Shift uses linear interpolation across levels,

$$x_\ell = (1 - t_\ell) x_{\text{orig}} + t_\ell x_{\text{textured}}, \quad t_\ell = \ell / L, \quad L = 20$$

while Background Replacement composites the object onto target scenes with increasing scene opacity over levels. Artistic Style follows a fixed 20-step trajectory from photorealistic rendering to nearly blank image, with average saturation approximately 16.3 levels. Shape Deformation applies strong warping or mutation of silhouette and geometry. Part Addition appends a clearly visible extraneous part per level and ensures at least $r$0 new parts at level $r$1. Part Removal removes a distinct part per level, guided by an ordered “removable parts” list with later entries more semantically significant, and saturates at approximately 16 levels on average (Tür et al., 3 Jun 2026).

The paper emphasizes ordering and graded difficulty: compounding ensures later levels are visually farther from $r$2, a “global sequence judge” enforces monotonic progression, and saturation is detected for scale, style, and part-removal axes (Tür et al., 3 Jun 2026). Quality control combines two-stage automated VLM judging with post-hoc cleaning and manual validation. Gemini-2.5 Flash serves as a per-level judge that validates whether the requested axis change is clear, without contamination from other axes, proposes revised prompts, retries up to 10 times, and flags saturation. A global sequence judge evaluates monotonic divergence across the full sequence, flags regressions or plateaus, performs mid-sequence checks every 5 levels, and permits re-generations. Post-hoc sequence cleaning assigns 0–100 “object intactness” scores per level, removes duplicates or stagnant levels, and fills the largest score gaps by generating intermediate levels. In manual validation, the authors checked 100 high-level perturbation pairs and found that 18% were noisy or undesirable, either because the wrong axis changed or because extra unintended changes appeared (Tür et al., 3 Jun 2026).

4. Evaluation protocols and measurement

NVRD supports VLM in-context learning experiments, likelihood-based probing, and direct human–model comparisons via a dual-image Likert judgment protocol (Tür et al., 3 Jun 2026). The first evaluation setup is in-context learning for name generation. A multi-image pool $r$3 contains $r$4 captioned with the nonce word $r$5 using one of five templates, four distractor image–caption pairs selected as the most visually similar images to $r$6 from a pool of 20,000 PixMoCap images via CLIP ViT-B/32, and $r$7 presented last with the fill-in-the-blank target “This image is best described by the reference: ____.” Greedy decoding is used, with up to 3 re-generations if the response length is less than 2 characters; images are shuffled, and $r$8 is always last (Tür et al., 3 Jun 2026).

For open-source models, the paper also computes average per-token log probability of the target nonce:

$r$9

where $x_0$0 is obtained via log-softmax over output logits. Probabilities are also computed for the vanilla label or labels, such as “tree frog,” to evaluate genuine acquisition versus defaulting to known mappings (Tür et al., 3 Jun 2026).

The primary human–model comparison task uses dual-image Likert ratings. Participants or models are shown $x_0$1 with the caption “Let’s call the object in this image ‘[nonce word]’.” They then see $x_0$2 and are asked, “Could both of these images be called ‘[nonce word]’?” The response is a single integer in $x_0$3, where 1 = Strongly Disagree and 7 = Strongly Agree (Tür et al., 3 Jun 2026).

The main tasks and metrics are recognition or acquisition, generalization across perturbations, and human–model agreement. The paper reports Spearman correlation $x_0$4 between human mean Likert ratings and model ratings, overall and per-perturbation type or leave-one-out; it also reports cross-task Spearman correlations between generation, Likert, and log-probability paradigms. The formulas included in the paper are:

$x_0$5

for Pearson $x_0$6, and

$x_0$7

for Spearman $x_0$8, interpreted as Pearson on ranks. The reported Spearman correlations are generally significant with $x_0$9, except Idefics-3 Generation ↔ Log Prob., which is non-significant (Tür et al., 3 Jun 2026).

5. Human study and model behavior

The human study comprises 800 unique dual-image trials, focusing on higher-level perturbations most relevant to cognition: shape, parts, style, texture, background, and color, with eight levels sampled (Tür et al., 3 Jun 2026). Participants were 30 anonymous native English speakers from the US, Canada, UK, and Ireland via Prolific. The study collected 2,400 total ratings; each participant rated 80 image pairs, and each pair received 3 independent judgments. Inter-rater reliability was $p$0 with $p$1 raters per item. Average duration was 6 min 47 s per participant, compensation averaged £17.79/hour, 5 attention checks used unrelated image pairs, and no participant failed the checks (Tür et al., 3 Jun 2026).

Five VLMs were evaluated: Qwen-2 VL 7B, Idefics-3 8B, and Molmo-2 8B as open-source models; GPT-4o Mini and Gemini-2.5 Flash as closed-source models. All models required multi-image capability. Greedy decoding was used across tasks, with short-output re-tries, and CLIP ViT-B/32 was used externally to select visually similar distractors for the in-context pool (Tür et al., 3 Jun 2026).

Quantitatively, acquisition differs strongly by novelty category. Known entities yield the lowest nonce usage and a strong preference for vanilla labels; GPT-4o Mini shows pronounced vanilla bias. Composed entities yield intermediate nonce usage, with shape–texture greater than shape–shape. Fully novel entities yield the highest nonce usage, because the absence of competing known labels reduces the threshold for adopting nonce mappings. Closed-source models generally adopt nonce references more often, and Gemini-2.5 Flash is highest across models (Tür et al., 3 Jun 2026).

Log-probability curves generally decline with perturbation level. Qwen-2 VL 7B shows the steepest decline on Part Removal, with z-scored drop approximately 0.8. Known entities show flat, low nonce log-probabilities. Shape–shape compositions decline more than shape–texture, consistent with higher structural sensitivity (Tür et al., 3 Jun 2026).

Likert generalization curves show that shape-related axes—Part Removal, Part Addition, and Shape Deformation—yield lower ratings even at early levels; Part Removal drops to approximately 1–2 at strong levels. Texture has a smaller effect, with some sensitivity in GPT-4o Mini and Molmo-2. Low-level edits such as Scale and Color Shift minimally affect ratings, although Molmo-2 assigns lower ratings, below 5, for Color Shift (Tür et al., 3 Jun 2026).

Human–model agreement on Likert ratings is high overall:

Model	Overall Spearman $p$2
Idefics-3 8B	.742
Molmo-2 8B	.771
Qwen-2 VL 7B	.812
Gemini-2.5 Flash	.905
GPT-4o Mini	.915

Leave-one-out by perturbation type ranges from .688 to .770 for Idefics-3, .752 to .836 for Qwen-2, .694 to .843 for Molmo-2, .858 to .915 for Gemini-2.5 Flash, and .899 to .925 for GPT-4o Mini. At the single-type level, the strongest alignments include Molmo-2 on Color Shift at .988, GPT-4o Mini on Part Removal and Style Degradation at .929, and Gemini on Style at .881; weak or negative values include Idefics-3 on Color Shift at −.521, Qwen-2 on Color Shift at −.481, and Gemini on Background at −.253 (Tür et al., 3 Jun 2026).

The paper’s principal behavioral result is not merely correlation but overgeneralization. Humans’ ratings drop below 3, corresponding to Somewhat Disagree, by perturbation level 10 for shape-based edits such as part removal and shape deformation. Models often remain between 4–6 for the same levels, extending labels that humans reject (Tür et al., 3 Jun 2026). The prompt-agreement ablation illustrates this tendency: on Qwen-2 VL 7B, many adversarial trials received “3” instead of “1–2,” and 30% received “6” (Agree). Most “6” judgments occurred when the second image was fully novel, suggesting a tendency to accept labels for novel-looking stimuli even when inappropriate (Tür et al., 3 Jun 2026). A common misconception would be that high human–model correlation implies human-like category boundaries; NVRD shows the opposite pattern, namely correlated sensitivity to perturbation type together with systematically broader model generalization.

6. Practical use, limitations, and relation to adjacent datasets

NVRD is released as a corpus with an interactive explorer, and full generation prompts and procedures are documented in the appendices (Tür et al., 3 Jun 2026). The released contents include base objects and perturbation sequences across 11 axes with up to 20 levels, per-image metadata fields, and documentation for generation settings, VLM judge prompts, post-hoc cleaning procedures, and human study materials. The recommended evaluation pipeline in the paper consists of five steps: sampling object categories and perturbation axes or levels; teaching a novel concept through the in-context setup; probability probing for open-source models; collecting dual-image Likert judgments; and analyzing generalization curves, Spearman correlations, and overgeneralization patterns (Tür et al., 3 Jun 2026).

The paper notes several limitations. Gemini-3 Pro Image and Gemini-2.5 Flash embed biases in structure, materials, and style; residual artifacts likely remain, especially for fully novel and compositional items. Some perturbation types saturate before 20 levels, and monotonicity is enforced but imperfect. Manual validation found 18% of high-level perturbation pairs noisy or undesirable (Tür et al., 3 Jun 2026). Ethical reporting for the human study specifies 30 adult participants, anonymity, native English speakers from the US, Canada, UK, and Ireland, informed consent, no PII collected, aggregate reporting, and attention checks (Tür et al., 3 Jun 2026).

The paper recommends NVRD for cognitively grounded evaluations of fast mapping and generalization, and proposes extensions with additional novelty types such as functional changes, richer contexts, or temporal exposure sequences. It also identifies prompt design effects, model architecture and training impacts, and conflict resolution strategies when labels contradict prior knowledge as future directions (Tür et al., 3 Jun 2026). This suggests that NVRD is not only a benchmark for aggregate accuracy or correlation, but also a diagnostic substrate for studying how pretraining knowledge constrains new label acquisition.

NVRD should not be conflated with the dataset introduced for Multimodal Reference Visual Grounding. The MRVG paper states explicitly that it does not introduce or mention any dataset named “Novel Visual References Dataset (NVRD),” and that the dataset introduced for the MRVG task is named MultimodalGround, with no stated relationship to NVRD (Lu et al., 2 Apr 2025). That distinction matters because the two resources target different problems: NVRD probes novel concept learning and generalization under contradiction with prior knowledge, whereas MultimodalGround evaluates visual grounding with a database of reference images (Lu et al., 2 Apr 2025).