VICIS: Visual Concept Inference from Sets

• VICIS is a framework that infers and leverages shared visual concepts from image sets to condition tasks like image generation and classification.
• It uses permutation-invariant set learners and diverse representations—hierarchical tokens, text-anchored activations, or graphs—to capture latent features.
• Empirical evaluations demonstrate enhanced accuracy, diversity, and interpretability compared to traditional single-image approaches.

Visual Concept Inference from Sets (VICIS) denotes the problem of inferring a visual concept from a set of example images and using that inferred concept as a structured conditioning signal for downstream tasks such as generation, classification, and set-level description. In its most explicit formulation, a model receives a small context set of images sharing a latent concept and a query image, and must generate new images that preserve the context-defined concept while remaining consistent with the query (Stracke et al., 2 Jul 2026). Closely related work treats the inferred concept as a hierarchy of sub-concepts encoded as learned textual tokens, as a bag of text-anchored concept activations for recognition, or as a graph of relational triplets for set summarization (Vinker et al., 2023, Zhang et al., 2023, Riccio et al., 25 Mar 2025). Taken together, these formulations define VICIS as a family of set-conditioned visual inference problems centered on concept extraction, compositional representation, and permutation-invariant reasoning over image collections.

1. Task definition and conceptual scope

VICIS is introduced as a task of visual in-context learning in which the specification is purely visual rather than textual. An image $x \in X$ is associated with a set of visual concepts $C_x \subseteq C$, where $C$ is the space of possible visual concepts such as animal kind, material, color, or shape. The input consists of a context set $X_{set} = \{x_i\}_{i=1..k}$ that shares one concept $c \in C$ and a query image $x_{query}$; the output is a distribution over images $y$ that preserve the instantiation of $c$ present in $x_{query}$ while allowing other factors to vary (Stracke et al., 2 Jul 2026).

The formal decomposition in the task paper separates set-level inference from query-specific extraction. A permutation-invariant set function produces a concept representation $z_c = f(X_{set})$, and a query-conditioned function produces $C_x \subseteq C$0, after which generation proceeds via $C_x \subseteq C$1. The reported architecture realizes $C_x \subseteq C$2 not as a single pooled vector but as a small set of learned concept directions, emphasizing that the shared concept may be better represented as a low-dimensional subspace than as a point embedding (Stracke et al., 2 Jul 2026).

Earlier work instantiated closely related ideas without introducing the VICIS term explicitly. In "Concept Decomposition for Visual Exploration and Inspiration" (Vinker et al., 2023), the input is a set of images $C_x \subseteq C$3 depicting a single visual concept, and the goal is to infer a hierarchical decomposition into sub-concepts encoded as learned token embeddings. In "Cross-Modal Concept Learning and Inference for Vision-LLMs" (Zhang et al., 2023), an image is represented as a set or vector of concept activations derived from a dictionary of text concepts, and classification is performed from that set-level representation. In "ImageSet2Text: Describing Sets of Images through Text" (Riccio et al., 25 Mar 2025), the target is a set-level concept graph $C_x \subseteq C$4 and a textual description $C_x \subseteq C$5 summarizing dominant content and relations across the set. This suggests that VICIS is not limited to conditional image generation; it more generally concerns the inference of reusable concept structure from image sets.

A recurring premise across these formulations is that whole-image matching is insufficient when classes, scenes, or exemplars are mixtures of semantic parts, attributes, and relations. The set input is therefore treated as evidence for a latent common factor, while irrelevant variation is to be discarded. In the generative VICIS task, failure modes include ignoring the context set, copying the query, or defaulting to biased generations (Stracke et al., 2 Jul 2026). In recognition-oriented and descriptive settings, analogous failures appear as reliance on spurious background concepts, over-coarse representations, or poor set-level validation (Zhang et al., 2023, Riccio et al., 25 Mar 2025).

2. Representational forms of inferred concepts

A central distinction among VICIS-style methods lies in how the inferred concept is represented. One line of work represents a concept hierarchically as learned token embeddings injected into a pretrained text-to-image diffusion model. In that formulation, the hierarchy is a binary tree $C_x \subseteq C$6, where each node $C_x \subseteq C$7 is a learned vector embedding and associated placeholder word $C_x \subseteq C$8 encoding a sub-concept. A root-level image set $C_x \subseteq C$9 denotes the original concept, whereas internal and leaf nodes encode increasingly specific aspects. Parent-child edges encode a binary reconstruction constraint: siblings together reconstruct their parent’s concept (Vinker et al., 2023).

A second line represents concepts as coordinates in a fixed semantic dictionary. CCLI constructs a dictionary of $C$0 text concepts $C$1 and computes text embeddings $C$2 with $C$3 “The photo is”. Description-specific visual concept prototypes are obtained by scoring training image features against each text concept and averaging the top-$C$4 most similar images. The resulting concept-score vector is

$C$5

which is interpreted as a bag-of-concepts: a permutation-invariant indicator of which concepts are present or compatible with the image (Zhang et al., 2023).

A third representation is graph-structured and relational. ImageSet2Text instantiates the concept representation as a concept graph

$C$6

where each triplet contains a subject $C$7, predicate $C$8, and object $C$9. At $X_{set} = \{x_i\}_{i=1..k}$0, the graph contains a root node $X_{set} = \{x_i\}_{i=1..k}$1 “image” linked to pending predicates “content”, “background”, and “style”. New triplets are hypothesized from subsets of images, generalized via WordNet, and validated on the full set before being accepted into the graph (Riccio et al., 25 Mar 2025). This representation emphasizes relations rather than latent coordinates.

The explicit VICIS architecture of 2026 represents the inferred concept as a small set of directions $X_{set} = \{x_i\}_{i=1..k}$2 in a learned embedding space of dimension 256. A transformer-based Set Learner operates on combined token embeddings from the whole set and predicts these directions. The query image is encoded into a CLS embedding $X_{set} = \{x_i\}_{i=1..k}$3, projected onto the subspace spanned by $X_{set} = \{x_i\}_{i=1..k}$4 using

$X_{set} = \{x_i\}_{i=1..k}$5

The projected embedding is then used as the concept-specific query representation for generation (Stracke et al., 2 Jul 2026).

These representations differ in granularity and intended use, but they share a common structural property: the concept is inferred from a set and stored in a form that can be reused, recombined, or validated downstream. A plausible implication is that VICIS is best understood as a representational design space rather than a single architecture.

3. Learning and inference mechanisms

The token-hierarchy approach learns concept structure by optimizing textual token embeddings inside Stable Diffusion v1.5. Each new node embedding is added to the text token dictionary and linked to a placeholder word. Child embeddings are initialized with the embedding of the common word “object” and trained using a Textual Inversion-style procedure in the text-encoder embedding space. When splitting a parent node $X_{set} = \{x_i\}_{i=1..k}$6 into children $X_{set} = \{x_i\}_{i=1..k}$7 and $X_{set} = \{x_i\}_{i=1..k}$8, optimization uses the latent diffusion loss on an image set $X_{set} = \{x_i\}_{i=1..k}$9 representing the parent concept with the prompt “A photograph of $c \in C$0 $c \in C$1”, enforcing binary reconstruction:

$c \in C$2

The method further skews timestep sampling according to

$c \in C$3

and selects among $c \in C$4 optimization seeds using a CLIP-based coherency criterion that favors high self-consistency and low cross-sibling similarity (Vinker et al., 2023).

CCLI, by contrast, freezes CLIP encoders and learns only lightweight heads. It initializes $c \in C$5 with description-specific concept prototypes $c \in C$6, $c \in C$7 with class prototypes $c \in C$8, and $c \in C$9 at zero for a text residual adapter. The final logits are

$x_{query}$0

and training minimizes cross-entropy while updating only $x_{query}$1, $x_{query}$2, $x_{query}$3, and $x_{query}$4 with AdamW. No auxiliary losses are used; affinity sharpening via $x_{query}$5 and $x_{query}$6 acts as calibration (Zhang et al., 2023).

ImageSet2Text performs no additional training. Its pipeline is inference-time and iterative. At each step, it selects a random subset $x_{query}$7 of size $x_{query}$8, chooses the closest leaf predicate to the root, generates a question using an LLM, applies VQA to every image in the subset, summarizes the answers into a candidate triplet $x_{query}$9, and proposes continuation predicates. The object is then generalized upward in WordNet through a chain $y$0 with at most $y$1 steps. Each candidate is validated on the full set using Open-CLIP ViT-bigG-14 embeddings and $y$2NN classification with $y$3, accepting the most specific non-rejected hypothesis that satisfies

$y$4

The graph is expanded only with accepted triplets, making the bottleneck explicit and interpretable (Riccio et al., 25 Mar 2025).

The 2026 VICIS model trains end-to-end with flow matching. A pretrained DINOv2 ViT-L encoder produces token embeddings for each set image, a Set Learner transformer predicts $y$5 concept directions, the query is projected into the inferred subspace, and a SiT-L generator conditioned through the timestep embedding pathway learns to generate a target image. The loss is

$y$6

There are no explicit contrastive or classification losses; the training episode itself enforces concept inference because the only way to predict the target consistently is to infer the shared concept from the set and isolate its instantiation in the query (Stracke et al., 2 Jul 2026).

Across these mechanisms, a common principle is structural sufficiency: the model is given a set, not a label, and must discover the latent factor that explains the commonality. The main methodological differences concern whether that factor is recovered through token optimization, frozen-encoder concept mining, graph-validated reasoning, or subspace inference.

4. Generation, recombination, and downstream use

In the hierarchical token model, each node in the learned tree affords “endless visual sampling.” A prompt containing a node token conditions Stable Diffusion, and the standard denoising process generates images coherent with the corresponding sub-concept. Because only token embeddings are learned and no UNet weights are fine-tuned, aspects can be composed both within and across trees by natural-language prompts such as “A photograph of $y$7 $y$8”. The paper emphasizes that composition is performed textually rather than by arithmetic vector operations, and reports examples such as combining a “saucer with a drawing” with the “creature” from a mug or replacing a Buddha with a cat sculpture while preserving stone base aspects (Vinker et al., 2023).

The explicit VICIS generation task is more constrained. The output should preserve only the concept shared by the context set as instantiated in the query, while other factors vary. This makes diversity part of the target behavior rather than a by-product. The same query can yield different valid outputs depending on the context-defined concept, and the reported qualitative behavior shows the model switching appropriately between different contextual definitions such as vehicle versus animal. The framework also extends to “visual transformations by example,” where the context set provides pairs $y$9 and the model applies the illustrated transformation to a new query (Stracke et al., 2 Jul 2026).

CCLI uses concept inference for classification rather than synthesis. The description-specific concept scores form a reusable representation for few-shot learning and domain generalization, while class-specific prototypes and a text adapter provide complementary evidence. The method’s VICIS relevance lies in converting an image into a set of concept activations aligned with a language-defined dictionary and then learning a set-to-label aggregator. The paper further notes that the reported implementation uses whole-image features, but a patch-level extension could define concept sets over regions and aggregate them with Deep Sets pooling, attention pooling, or a transformer (Zhang et al., 2023). This suggests a direct route from concept-set inference to part-based visual reasoning.

ImageSet2Text uses the inferred concept structure as an interpretable bottleneck for natural-language set description. After graph construction terminates, pending predicates are discarded and an LLM converts the finalized graph into a coherent description. For group captioning, the longer description can be “captionized” into a concise sentence by an additional LLM call. The graph can also support set difference captioning, where richer structural detail improves the ability to distinguish fine-grained differences between paired image sets (Riccio et al., 25 Mar 2025).

These uses establish three recurrent functions for VICIS representations: conditioning generation, supporting recognition, and mediating explanation. A plausible implication is that the utility of VICIS depends less on the modality of the output than on whether the inferred concept representation remains reusable, compositional, and set-grounded.

5. Empirical evaluation and observed performance

The 2026 VICIS paper evaluates performance primarily on hierarchical ImageNet/WordNet episodes and synthetic controlled data. On the main ImageNet/WordNet animal subtree, the copy-query baseline has diversity $c$0. ILLUME+ 3B attains per concept accuracy $c$1, per instantiation accuracy $c$2, diversity $c$3; BAGEL 7B-MoT attains $c$4, $c$5, and $c$6; Visual Prompting attains $c$7, $c$8, and $c$9; the proposed method attains $x_{query}$0, $x_{query}$1, and $x_{query}$2 (Stracke et al., 2 Jul 2026). On sketch queries with real context, it reports $x_{query}$3 per concept, $x_{query}$4 per instantiation, diversity $x_{query}$5; with sketch context and real query, $x_{query}$6, $x_{query}$7, and $x_{query}$8; with sketch context and query, $x_{query}$9, $z_c = f(X_{set})$0, and $z_c = f(X_{set})$1; and with unseen ImageNet-21k classes in the context, $z_c = f(X_{set})$2, $z_c = f(X_{set})$3, and $z_c = f(X_{set})$4. Closed-source VLMs on a simplified human-judged test achieve $z_c = f(X_{set})$5 for Nano Banana and $z_c = f(X_{set})$6 for Gemini 2.5 Flash Image, compared with $z_c = f(X_{set})$7 for the VICIS model. Under varying context size, per concept accuracy moves from $z_c = f(X_{set})$8 to $z_c = f(X_{set})$9, per instantiation accuracy from $C_x \subseteq C$00 to $C_x \subseteq C$01, and diversity from $C_x \subseteq C$02 to $C_x \subseteq C$03 across $C_x \subseteq C$04 images. Under noisy context, clean performance is $C_x \subseteq C$05, $C_x \subseteq C$06, $C_x \subseteq C$07; with one noisy image, $C_x \subseteq C$08, $C_x \subseteq C$09, $C_x \subseteq C$10; with two noisy images, $C_x \subseteq C$11, $C_x \subseteq C$12, $C_x \subseteq C$13. On the synthetic multiple-shared-concepts test, one shared concept yields mean accuracy $C_x \subseteq C$14 and mean entropy $C_x \subseteq C$15, while two shared concepts yield mean accuracy $C_x \subseteq C$16 and mean entropy $C_x \subseteq C$17 (Stracke et al., 2 Jul 2026).

The hierarchical token approach evaluates decomposition quality using CLIP-based consistency. In a human study with $C_x \subseteq C$18 participants over $C_x \subseteq C$19 pairs of sets from $C_x \subseteq C$20 objects, CLIP consistency matches human judgments in $C_x \subseteq C$21 of cases. Over $C_x \subseteq C$22 concepts, reconstruction quality $C_x \subseteq C$23 averages $C_x \subseteq C$24, and sibling distinctness $C_x \subseteq C$25 averages $C_x \subseteq C$26. In a user study on $C_x \subseteq C$27 objects with $C_x \subseteq C$28 aspects each and $C_x \subseteq C$29 participants, recognition of the originating object from aspect sets reaches $C_x \subseteq C$30. Ablations report that for two versus three children per split, average self-consistency of the best two nodes is $C_x \subseteq C$31 and $C_x \subseteq C$32, whereas the third drops to $C_x \subseteq C$33. Depth statistics over $C_x \subseteq C$34 trees show level 1 self-consistency average $C_x \subseteq C$35, level 2 self-consistency average $C_x \subseteq C$36, level 1 sibling consistency average $C_x \subseteq C$37, and level 2 sibling consistency average $C_x \subseteq C$38 (Vinker et al., 2023).

CCLI evaluates on $C_x \subseteq C$39 few-shot datasets and on domain generalization from ImageNet to ImageNet-V2, -Sketch, -A, and -R. On ImageNet few-shot with ResNet-50, it reports $C_x \subseteq C$40 at $C_x \subseteq C$41 shots. The average gain over Tip-Adapter-F across $C_x \subseteq C$42 datasets is approximately $C_x \subseteq C$43, with largest improvements up to $C_x \subseteq C$44 versus CoOp on UCF101 and up to $C_x \subseteq C$45 versus Tip-Adapter-F on UCF101 at 16-shot. For domain generalization with ResNet-50, source ImageNet is $C_x \subseteq C$46 and OOD average is $C_x \subseteq C$47 versus $C_x \subseteq C$48 for TPT, a $C_x \subseteq C$49; with ViT-B/16, OOD average is $C_x \subseteq C$50 versus $C_x \subseteq C$51 for TPT, a $C_x \subseteq C$52. Ablations show ImageNet 16-shot rising from $C_x \subseteq C$53 to $C_x \subseteq C$54 with concept inference alone and to $C_x \subseteq C$55 with concept inference plus text adapter; removing $C_x \subseteq C$56 drops performance from $C_x \subseteq C$57 to $C_x \subseteq C$58, and removing $C_x \subseteq C$59 drops it to $C_x \subseteq C$60 (Zhang et al., 2023).

ImageSet2Text evaluates on GroupConceptualCaptions, GroupWikiArt, and PairedImageSets. On GroupConceptualCaptions, it reports CIDEr-D $C_x \subseteq C$61, SPICE $C_x \subseteq C$62, METEOR $C_x \subseteq C$63, ROUGE-L $C_x \subseteq C$64, BERTScore $C_x \subseteq C$65, LLM-Judge $C_x \subseteq C$66, and CLIPScore $C_x \subseteq C$67. On GroupWikiArt, it reports CIDEr-D $C_x \subseteq C$68, SPICE $C_x \subseteq C$69, METEOR $C_x \subseteq C$70, ROUGE-L $C_x \subseteq C$71, BERTScore $C_x \subseteq C$72, and LLM-Judge $C_x \subseteq C$73. In completeness evaluation on PairedImageSets, augmenting the VisDiff proposer-ranker with ImageSet2Text graphs yields Easy $C_x \subseteq C$74, Medium $C_x \subseteq C$75, Hard $C_x \subseteq C$76 for Acc@1/Acc@5, compared with $C_x \subseteq C$77, $C_x \subseteq C$78, and $C_x \subseteq C$79 for VisDiff alone. A user study with $C_x \subseteq C$80 valid participants reports higher scores than controls for Clarity $C_x \subseteq C$81 versus $C_x \subseteq C$82, Accuracy $C_x \subseteq C$83 versus $C_x \subseteq C$84, Detail $C_x \subseteq C$85 versus $C_x \subseteq C$86, and Flow $C_x \subseteq C$87 versus $C_x \subseteq C$88 (Riccio et al., 25 Mar 2025).

Collectively, these results indicate that set-based concept inference can be evaluated along at least four axes: fidelity to the intended concept, discrimination among sibling instantiations, diversity or non-triviality of outputs, and interpretability of the inferred representation.

6. Relation to adjacent paradigms, limitations, and open directions

VICIS overlaps with several neighboring research areas but is not reducible to any one of them. It is adjacent to personalization methods such as Textual Inversion and DreamBooth, yet those methods generally learn a single token or fine-tune model weights to capture an entire concept, offering little internal structure; the hierarchical token method instead organizes multiple learned tokens in a binary tree and selects them for coherency and distinctness (Vinker et al., 2023). It is adjacent to prompt learning for CLIP, but CCLI argues that whole-image matching with a single class-specific text description is brittle because images from the same class contain different semantic objects and parts; concept dictionaries and set-level concept activations provide a more compositional alternative (Zhang et al., 2023). It is adjacent to concept bottleneck models, and ImageSet2Text explicitly draws inspiration from CBMs while replacing a closed concept inventory with an iteratively built graph of validated triplets (Riccio et al., 25 Mar 2025). The 2026 task paper positions VICIS as a form of image-only in-context learning that avoids text instructions, labels at test time, and concept-specific fine-tuning (Stracke et al., 2 Jul 2026).

Several limitations recur across formulations. The hierarchical token method reports background leakage when input images have similar viewpoints or backgrounds, incomprehensible splits that are coherent by CLIP but not aesthetically meaningful, dominant sub-concepts that prevent meaningful separation, large sibling overlap, and degradation with deeper trees or more than two children per node (Vinker et al., 2023). CCLI notes concept ambiguity among overlapping words, susceptibility to spurious background concepts, and weak class-specific prototypes in very low-shot regimes for some fine-grained datasets (Zhang et al., 2023). ImageSet2Text identifies CVL embedding limitations, WordNet contradictions, sampling bias from small subsets, dependence on a proprietary LLM, and degradation on highly heterogeneous sets with few shared elements (Riccio et al., 25 Mar 2025). The explicit VICIS model remains dependent on a strong pretrained encoder, on WordNet-based weak supervision for episode construction, and on a hierarchy-aware classifier for diversity evaluation; ambiguous or noisy context sets reduce accuracy, even if performance degrades gracefully (Stracke et al., 2 Jul 2026).

These limitations reveal a persistent tension between openness and control. Methods that allow unconstrained discovery may extract unexpected or uninterpretable aspects; methods that rely on external semantic scaffolds may inherit lexical, taxonomic, or embedding-space biases. A plausible implication is that future VICIS systems will need stronger uncertainty handling, richer part- and relation-level structure, and better mechanisms for distinguishing meaningful shared concepts from incidental correlations.

The open directions named across the papers converge on several themes: automatic tree construction and stronger disentanglement regularizers for decomposition (Vinker et al., 2023); region-level concept assignment, Deep Sets or attention aggregation, and probabilistic concept selection for recognition (Zhang et al., 2023); probabilistic validation, multimodal knowledge graphs, subgroup discovery in heterogeneous sets, and open-source LLM alternatives for set description (Riccio et al., 25 Mar 2025); and richer compositional concepts, reduced reliance on hierarchical scaffolds, explicit uncertainty modeling, and tighter integration with editable generation controls for the generative VICIS task (Stracke et al., 2 Jul 2026). Together, these directions suggest that VICIS is developing toward a general theory of concept induction from image sets, in which the key problem is not merely recognizing similarity, but identifying which shared factor should be abstracted, how it should be represented, and how that representation should constrain downstream reasoning.