Show Me Examples: Inferring Visual Concepts from Image Sets

Abstract: Vision-LLMs (VLMs) can follow complex textual instructions, yet they struggle to reason from purely visual context. In particular, current models fail to infer shared concepts from sets of example images and apply them to new inputs. We introduce Visual Concept Inference from Sets (VICIS), a task that evaluates this capability. Given a small context set of images sharing a concept and a query image, the model must generate new images that preserve the context-defined concept while remaining consistent with the query. We show that state-of-the-art VLMs perform poorly on this task, often ignoring the visual context or defaulting to biased generations. To address this gap, we propose a training framework and architecture that learn to infer visual concepts from image sets and extract concept-specific embeddings from queries. Experiments on synthetic data and large-scale ImageNet/WordNet data show that our model generates more accurate and diverse outputs and generalizes to unseen concepts and modalities such as sketches.

• The paper presents VICIS, a novel task where a model infers latent visual concepts from a set of example images and applies them to a query for controlled generation.
• The proposed architecture, combining a Set Learner, Instantiation Module, and Diffusion Generator, achieves 46.3% per-concept and 54.5% per-instantiation accuracy.
• VICIS demonstrates robust generalization and scalability, enabling applications like few-shot classification, visual analogies, and language-free human-AI interaction.

Visual Concept Inference from Image Sets: VICIS

Introduction

The paper "Show Me Examples: Inferring Visual Concepts from Image Sets" (2607.02402) defines and addresses the Visual Concept Inference from Sets (VICIS) problem, a direct evaluation of nonverbal visual concept learning and abstraction in generative vision models. Distinctly, VICIS requires a model to infer a latent visual concept from a small set of example images (the context) and to apply this concept to a query image, such that generated outputs embody precisely the specified concept instantiation while disregarding irrelevant visual information. This scenario closely mimics natural, language-free human concept learning but has so far remained unsolved by state-of-the-art vision-LLMs (VLMs).

Figure 1: The VICIS task: infer a visual concept from a context set of images and extract its instantiation from a query image for controlled generation; existing VLMs default to copying the query, while the proposed model adapts generation based on inferred context.

Task Formulation and Core Motivation

Current VLMs excel at following explicit textual instructions but largely fail at purely visual in-context learning, especially under ambiguous, label-free conditions. The VICIS challenge is motivated by critical failures observed in leading VLMs: when tasked with extracting a visual concept from image sets and instantiating it in generated images, they ignore the visual context and reproduce trivial or biased content (Figure 2). This exposes a fundamental limitation in visual reasoning and context utilization, despite apparent generalization in natural language domains.

Figure 2: VLM failure cases on VICIS: models persistently ignore context, producing outputs dominated by query image bias rather than reflecting the specified concept.

To systematically evaluate this capability, VICIS is defined as follows:

• The model receives a context set of $N$ images, all sharing a latent visual concept (e.g., color, object class, shape, taxonomic branch) but otherwise diverse.
• A query image provides a specific instantiation of the concept.
• The model must generate new images that preserve only the specified concept's instantiation from the query, allowing irrelevant factors to vary freely.

Success on VICIS is not about retrieving or replicating exemplars, but about abstracting the shared structure, disentangling relevant from irrelevant features, and transferring this abstraction to new content. Task performance is quantified on both accuracy (did the model generate the correct concept instantiation?) and diversity (does it avoid trivial solutions such as simply copying the context or query?).

Model and Training Framework

The authors propose a tailored architecture for VICIS, specifically addressing the limitations of existing VLMs. The pipeline comprises three major components: a Set Learner, an Instantiation Module, and a Diffusion Generator.

Set Learner: This module accepts the context set, encodes each image into feature tokens (via a pretrained ViT), and jointly infers concept-specific direction vectors spanning a subspace in the high-dimensional embedding space. These directions define the latent manifold corresponding to the shared concept.

Instantiation Module: The query image is embedded and projected onto the learned concept subspace, yielding a disentangled, concept-conditioned embedding corresponding to the required instantiation. Only the relevant dimensions are preserved, while other information is removed.

Diffusion Model: Conditioned on the projected query embedding, a diffusion-based generator is trained to produce output images that manifest the specified concept, using rectified flow matching as the generative objective. This encourages smooth sample interpolation and supports robust, conditional synthesis.

Figure 3: Model architecture: the Set Learner discovers the concept manifold from context; the query is projected onto this space and used to condition a diffusion generator.

The system is trained end-to-end. Training data are constructed using auxiliary groupings (such as the WordNet hierarchy underlying ImageNet), which allows for scalable, weakly-supervised assembly of context-query-target tuples without individual concept labeling.

Evaluation: Controlled and Real-World Settings

Experiments rigorously evaluate the VICIS framework on both synthetic datasets (where concept factors are precisely controlled) and large-scale, hierarchical real-world datasets based on ImageNet/WordNet. In both cases, the main criterion is the model's ability to (a) abstract the correct latent concept from few examples, and (b) robustly instantiate that concept in novel images according to the specified query.

Figure 4: Toy dataset evaluation: the model preserves only the desired concept (e.g., shape or color) as defined by context, allowing other factors to vary, demonstrating strong abstraction and disentanglement.

Hierarchical Concept Specification: The use of the WordNet concept hierarchy enables flexible definition of concepts at various granularity (e.g., "animal" vs. "feline" vs. "tiger"). The model must resolve at which abstraction level to instantiate the query, performing compositional reasoning over a multi-level taxonomy.

Figure 5: Hierarchical concept space visualization leveraging the WordNet hierarchy for flexible, multi-level concept inference.

Quantitative Metrics:

• Accuracy: The proposed model achieves $46.3\%$ mean per-concept accuracy and $54.5\%$ per-instantiation accuracy, outperforming strong Visual Prompting and VLM baselines by an absolute margin of $>10\%$.
• Diversity: The model maintains a diversity score of $0.81$ (relative entropy ratio), indicating it generates varied images within the abstracted concept space—avoiding mode collapse or trivial replications.
• Robustness: Accuracy increases with larger context set size, plateaus after four images, and degrades gracefully with corrupted context sets, highlighting the model's sensitivity and resilience.
• Generalization: The model extrapolates to previously unseen modalities (e.g., sketches) and concept classes, only moderately reducing performance, demonstrating genuine abstraction rather than rote memorization.

  Figure 6: Out-of-distribution generalization: model trained only on images can extrapolate to sketch queries and context, aligning queries with the closest point in the learned concept manifold.

State-of-the-Art Comparisons: Commercial VLMs (Gemini, Qwen, FLUX, etc.) do not reliably solve the VICIS task, with closed models failing to exploit visual context and defaulting to replication or bias, and open models (ILLUME+, BAGEL) significantly lagging in both accuracy and diversity despite extensive prompt engineering.

Analysis and Applications

Concept Representation Analysis: The learned embedding space is highly structured. Using Fisher Discriminant Ratio, the authors demonstrate that their method achieves separability comparable to LDA but with far fewer samples, in contrast to statistical methods (PCA, LDA) that do not encode explicit concept structure.

Scalability & Versatility: VICIS generalizes not only to concept inference but also enables few-shot classification, learning of visual analogies, and transfer to arbitrary downstream visual tasks, all without explicit class labels or retraining.

Figure 7: Application of VICIS for arbitrary visual transformation learning—contextual examples specify an operation that is then applied to the query image.

Theoretical and Practical Implications

The VICIS benchmark surfaces a previously unaddressed gap in visual abstraction and reasoning for current VLMs. By formalizing and scaling visual context-driven concept inference, the methodology introduces a more natural and powerful interface for visual communication and guidance, especially in scenarios where language or explicit labeling is infeasible or undesirable.

Practically, this opens avenues for robust, inference-time adaptation of generative models, personalized and flexible control, and language-free human-AI interaction. The work sets a new technical baseline for context-dependent image generation, bridging the gap toward human-like, nonverbal generalization.

Conclusion

VICIS precisely targets visual in-context reasoning, demanding the inference and instantiation of visual concepts from unlabelled example sets. The authors show that current VLMs, despite their multi-modal scope, are largely incapable of such reasoning. In contrast, the domain-specific architecture introduced here delivers strong accuracy, high diversity, graceful scaling with context quality, and generalization to unseen tasks and modalities. This represents a substantial qualitative and quantitative step forward in machine visual reasoning, offering foundational insights and tools for future research into generalizable, label-free concept learning in AI.

Figure 8: Qualitative samples: model-generated outputs showcase both accurate concept instantiation and diversity among generated images under hierarchical contexts.