Would you still call this Dax? Novel Visual References in VLMs and Humans

Abstract: Vision-LLMs (VLMs), like human learners, are frequently exposed to new visual concepts, but how they map novel visual references to language after exposure remains largely underexplored, particularly when those references contradict prior knowledge from pre-training. To study this, we present the Novel Visual References Dataset (NVRD): 19,176 images spanning 90 visual concepts across different levels of visual novelty, each with up to 20 increasingly perturbed versions of the original object to probe generalization. Unlike prior work on visual augmentations of familiar concepts, NVRD comprises entirely novel, open-ended stimuli constructed from scratch, mirroring how humans encounter genuinely new concepts. We evaluate 3 open- and 2 closed-source models alongside 2,400 human judgments for direct human-model comparison, and find that (i) models struggle to acquire novel concepts in-context when they contradict prior knowledge, and (ii) while models and humans show correlated sensitivity to visual perturbations, models significantly overgeneralize, extending learned labels to stimuli that humans reject. We contribute NVRD as a corpus and benchmark for research on visual concept learning in both humans and machines.

• The paper introduces the Novel Visual References Dataset (NVRD) and controlled experiments to evaluate how VLMs and humans map newly coined labels to visual entities.
• It demonstrates that VLMs tend to overgeneralize novel labels, often misaligning with human sensitivity to both low-level and high-level visual perturbations.
• The study highlights gaps in model inductive biases and suggests improving training data and architectural designs to achieve more human-like referential precision.

Novel Visual Reference Generalization in Vision-LLMs Versus Humans

Introduction and Motivation

Vision-LLMs (VLMs) increasingly underlie applications in perception, interaction, and knowledge grounding, yet their responses to truly novel visual referents—especially under conflicting prior knowledge—are inadequately characterized. The paper "Would you still call this Dax? Novel Visual References in VLMs and Humans" (2606.05409) directly addresses this critical gap by introducing a large-scale dataset, the Novel Visual References Dataset (NVRD), and a set of controlled experiments designed to probe VLM and human sensitivity to new visual concepts, compositionality, and systematic perturbations.

The NVRD comprises over 19,000 images encompassing 90 visually distinct concepts and up to 20 graded perturbations per object, enabling fine-grained analysis of generalization. The core objective is to examine the mechanisms and extent by which both VLMs and humans generalize newly assigned, out-of-distribution labels (nonce words) to altered exemplars and to assess whether models' inductive biases and mapping strategies align with or diverge from human patterns.

Figure 1: Task setup showing visual comparison trials for both models and human annotators, rating agreement with a novel referent label attached to objects and their perturbed variants.

The Novel Visual References Dataset (NVRD)

The dataset operationalizes visual novelty at multiple axes: it includes familiar categories (e.g., chairs), composed objects (combining shapes or textures from common objects), and fully synthetic entities with no correspondence in any known dataset. For each object, systematic perturbations are applied, categorized as either low-level (e.g., Gaussian noise, color shifts, JPEG compression) or high-level semantic (texture transfer, background replacement, artistic style degradation, shape deformation, part addition/removal).

Figure 2: Pipeline for NVRD construction, including generation of object categories and systematic perturbations along semantic and non-semantic axes.

This design enables comprehensive tests for concepts such as shape-bias, mutual exclusivity, and compositional generalization, drawing directly on the cognitive science literature on human fast mapping and referent assignment.

Experimental Methodology

Three experimental paradigms were deployed across five VLMs (Qwen-2 VL 7B, Idefics-3 8B, Molmo-2 8B, GPT-4o Mini, Gemini-2.5 Flash) and 2,400 human judgments:

1. Name Generation In-Context: Models exposed to an image with a nonce label alongside distractors, then tasked with labeling a perturbed image from the same pool.
2. Token Probability Assignment: Calculation of log probabilities (normalized) for nonce vs. vanilla labels in context, restricted to open-source models.
3. Dual-Image Likert Scale: Agreement ratings (1–7) on whether a perturbed variant still merits the original nonce label, enabling direct human-model comparison.

The approach allows for both production (generative) and recognition (probability/logit or agreement) evaluation, capturing not only whether a model will use a new label, but its confidence and generalization across non-semantic and semantic perturbation gradients.

Figure 3: Example analysis of nonce versus vanilla label adoption, showing higher model proclivity for nonce labeling in more novel or partially novel (composed) entities.

Key Empirical Findings

Model Acquisition and Generalization of Novel References

• Novel reference adoption is inversely correlated with prior knowledge: Known objects nearly always elicit vanilla labels, especially in closed-source GPT-4o Mini; composed objects fall between; only fully novel entities reliably prompt consistent mapping to nonce words given a single in-context exposure.
• Generalization across perturbation gradients is robust but overextended: Once a model commits a label to an object, it typically continues applying that label as visual perturbations increase, even at extremities where human raters strongly reject the mapping.

  Figure 4: Model performance breakdown by both name generation and log probability, standardized for cross-model comparability.

Structural Sensitivity and Shape Bias

• Strong sensitivity to shape-based structural perturbations is observed in both models and humans: Shape deformation, part removal, and part addition are the perturbations most likely to cause both to withdraw label agreement. However, models are less discriminative; human Likert ratings can fall two full points lower than models (on a 1–7 scale) at severe perturbation, especially for highly novel shapes.
• Low-level changes (color, JPEG compression, background modifications): These have minimal effect on model ratings, in line with previous literature on shape-versus-texture bias, but confirming that models track some aspects of this inductive bias even with zero-shot novel entities.

  Figure 5: Comparison of human and model agreement ratings, highlighting sensitivity to structural changes (shape, part addition/removal, style) versus resilience to surface-level/noise perturbations.

Model-Human Alignment and Overgeneralization

• Human–model statistically significant correlation: Spearman coefficients between model and mean human ratings reach as high as 0.92 (GPT-4o Mini), but always with a pronounced tendency for models to overgeneralize—continuing to assign novel labels to objects humans judge as no longer instances of the original.
• Object category divergence: Humans are more sensitive to category novelty, lowering labels for synthetic objects under high perturbation; VLMs largely fail to differentiate, indicating an absence of mutual exclusivity and lexical constraint biases beyond the effect of label novelty alone.

Failure Cases and Sycophancy

• Prompt-randomization ablations demonstrate model sycophancy: When presented with mismatched image pairs, VLMs frequently agree that two unrelated objects should share the same reference—response patterns inconsistent with principled referential mapping and indicative of excessive agreement rather than symbolic alignment.

  Figure 6: Model agreement scores in adversarial "failure case" trials, illustrating inflated agreement especially with fully novel referents.

Implications and Future Directions

The NVRD corpus offers a controlled and high-variance testbed for advancing VLM research on generalization, label assignment, and alignment with human symbolic reasoning. The results highlight:

• Insufficiency of current in-context learning for referential precision: VLMs indiscriminately extend labels to highly perturbed or even mismatched objects.
• Partial acquisition of human-like biases: Shape bias is present but not adequately calibrated; mutual exclusivity and context re-use constraints are absent or extremely weak.
• Implications for human-agent communication: Overgeneralization risks breakdowns in referent tracking in practical, human-in-the-loop deployments, especially as environments introduce genuinely novel or dynamic entities.
• Need for improved inductive bias alignment: Both training data augmentation (with more challenging novel-concept mapping tasks) and explicit architectural or objective modifications (e.g., biases enforcing mutual exclusivity or stricter referential precision) may be required.

Theoretically, these findings reinforce the necessity of cross-disciplinary approaches, embedding cognitive science insights into VLM design, especially when extending beyond perceptual matching into pragmatic communication and shared reference evolution.

Conclusion

By systematically dissecting how VLMs and humans map and generalize novel visual references across a graded series of semantic and surface perturbations, this work demonstrates clear overgeneralization and bias miscalibration in state-of-the-art VLMs relative to humans. The NVRD benchmark establishes a new standard for cognitively-motivated model evaluation and will be critical for future research aiming to bridge the gap between statistical association and principled, human-aligned referential reasoning in artificial agents.