Virtual Exemplars: Concepts and Applications

• Virtual exemplars are non-literal proxies—such as stored feature vectors, synthetic renderings, or soft prompt tokens—that preserve the functional role of examples while altering their representation.
• They are applied in diverse domains including online action detection, retrieval-augmented language modeling, photometric stereo, active learning, zero-shot recognition, and embodied memory systems.
• Key computational mechanisms include attention-based consultation, example-based reconstruction, adversarial synthesis, and abstraction-driven memory distillation to improve efficiency and robustness.

Virtual exemplars are non-literal stand-ins for ordinary examples: stored feature vectors, synthetic rendered observations, learned prototype points, soft prompt tokens, class-level visual prototypes, or distilled memory programs that preserve the operational role of examples while changing their representation, storage, or mode of use. Across the literature, they are introduced to reduce annotation or inference cost, improve robustness, compress context, or expose structure that raw examples do not make explicit. The concept therefore spans online video understanding, retrieval-augmented language modeling, photometric stereo, active learning, zero-shot recognition, embodied memory systems, and related exemplar-selection methods (Yang et al., 2022, Jain et al., 6 Oct 2025, Hui et al., 2015, Changpinyo et al., 2018, Sarch et al., 2024).

1. Definitions and conceptual scope

In online action detection, an exemplar is a stored feature vector rather than a raw frame. Colar distinguishes between historical exemplars, which are previous-frame features within the same video segment, and category exemplars, which are representative frame features per action class obtained by K-means clustering. The exemplars are therefore “virtual” in the sense that the model consults feature prototypes rather than raw pixel data or full videos (Yang et al., 2022).

In retrieval-augmented language modeling, MHA-RAG treats conventional retrieved demonstrations or documents as inputs to a transformation that produces a short set of soft prompt vectors. These vectors are continuous, query-conditioned, and order-invariant, so the retrieved examples remain functionally present while no longer appearing as literal text in the prompt. In that formulation, the virtual exemplar is a learned soft token that substitutes for many textual exemplars (Jain et al., 6 Oct 2025).

In photometric stereo, a virtual exemplar is a synthetic image-appearance vector rendered from a BRDF dictionary and known lighting for a hypothetical surface normal and material combination. The crucial substitution is from physical calibration exemplars to offline-rendered appearance exemplars, which are then matched against observed intensities (Hui et al., 2015).

In active learning for satellite image change detection, virtual exemplars are learned synthetic points in feature space rather than selected unlabeled samples. They are optimized to be representative, diverse, and ambiguous with respect to the current classifier, then instantiated as real display items by nearest-neighbor mapping back to actual patch pairs (Sahbi et al., 2022, Sahbi et al., 2022).

In zero-shot learning, virtual exemplars are synthetic visual prototypes for unseen classes. They are predicted from semantic descriptions and then used either directly for nearest-neighbor classification or as denoised replacements for raw semantic embeddings in downstream zero-shot pipelines (Changpinyo et al., 2018).

Embodied-agent memory systems broaden the notion further. ICAL stores optimized trajectories plus language abstractions as reusable memory elements, effectively turning prior experience into self-generated exemplars with executable structure and causal annotation. Exemplar-based contrastive self-supervised learning makes a related move in representation space, where prototypes, neighborhood aggregates, feature memories, and graph nodes act as compact surrogates for many raw examples (Sarch et al., 2024, Chang, 2022).

2. Representational forms

The cited literature uses several recurrent representational forms for virtual exemplars.

Regime	Virtual exemplar representation	Operational role
Online action detection	Stored frame features or class cluster centers	Memory read for temporal and category guidance
RAG / ICL	Soft prompt tokens $Z \in \mathbb{R}^{H \times d}$	Compact prefix replacing exemplar text
Photometric stereo	Rendered appearance vectors $\mathbf{b}_k(\mathbf{n})$	Matching and reconstruction under candidate normals
Active learning	Synthetic feature-space centers $v_k$	Query targets mapped to real samples
Zero-shot learning	Predicted class prototypes $\hat v_u$	Nearest-neighbor or semantic-denoising surrogate
Embodied memory	$e=(\xi_{\text{optimized}}, L)$	Retrieved program-like memory for action generation

Colar formalizes both dynamic and static exemplar memories in feature space. For a current frame feature $\mathbf{f}_0$, historical exemplars $\{\mathbf{f}_t\}_{t=-T}^{0}$ are projected into key/value spaces, compared by cosine similarity, softmax-normalized, and aggregated as

$$\mathbf{f} = \sum_{t=-T}^{0} \hat{\mu}_t \cdot \mathbf{f}_t^{v}.$$

Its category-exemplar bank stores $$\mathcal{E}_c = [\mathbf{e}_{c,1},\dots,\mathbf{e}_{c,M}]$$ for each class $c$, where the $\mathbf{b}_k(\mathbf{n})$0 are K-means cluster centers over class-specific training features (Yang et al., 2022).

MHA-RAG defines retrieved exemplars $\mathbf{b}_k(\mathbf{n})$1, encodes them, and maps them by multi-head attention into a soft prompt

$\mathbf{b}_k(\mathbf{n})$2

with one virtual token per head. Because each head aggregates over a set of retrieved exemplars by attention, the resulting prompt is permutation-invariant with respect to exemplar order (Jain et al., 6 Oct 2025).

In photometric stereo, the virtual exemplar matrix for a candidate normal $\mathbf{b}_k(\mathbf{n})$3 is

$\mathbf{b}_k(\mathbf{n})$4

whose columns are predicted intensity vectors under all $\mathbf{b}_k(\mathbf{n})$5 illuminations for each dictionary BRDF. Any non-negative combination $\mathbf{b}_k(\mathbf{n})$6 is then a realizable appearance under that normal (Hui et al., 2015).

The active-learning formulations parameterize virtual exemplars as a set $\mathbf{b}_k(\mathbf{n})$7 together with a membership matrix $\mathbf{b}_k(\mathbf{n})$8, $\mathbf{b}_k(\mathbf{n})$9. The exemplars are not labels or raw samples but optimized centers in the same feature space as the data, regularized by representativity, diversity, and ambiguity terms (Sahbi et al., 2022, Sahbi et al., 2022).

In zero-shot learning, the central object is the predicted unseen-class exemplar

$v_k$0

where $v_k$1 is the semantic representation of unseen class $v_k$2, and $v_k$3 is learned on seen classes to map semantic descriptions to visual class means after PCA (Changpinyo et al., 2018).

ICAL represents each memory item as a pair $v_k$4, where $v_k$5 is an improved trajectory and $v_k$6 is a structured set of language abstractions containing task and causal abstractions, state changes, task decomposition and subgoals, and state abstraction. This makes the exemplar partly symbolic, partly procedural, and explicitly reusable (Sarch et al., 2024).

3. Core computational mechanisms

A first mechanism is attention-like consultation. Colar measures cosine similarity between the current frame and historical or category exemplars, normalizes the similarities with softmax, and reads out a weighted sum used for classification. MHA-RAG applies the same broad pattern at the language-model interface: query-conditioned multi-head attention compresses a retrieved exemplar set into a short soft prefix. In both cases, the exemplar bank functions as an addressable memory rather than a static concatenation of raw instances (Yang et al., 2022, Jain et al., 6 Oct 2025).

A second mechanism is example-based reconstruction. In photometric stereo, the system searches over candidate normals and, for each one, solves a non-negative least-squares problem

$v_k$7

The residual selects the most plausible normal, while the BRDF weights are estimated afterward under non-negativity and additional priors such as sparsity and spatial coherence. Virtual exemplars are therefore not merely retrieved; they are generated by the forward image-formation model and tested by reconstruction fidelity (Hui et al., 2015).

A third mechanism is adversarial or uncertainty-seeking synthesis. In satellite change detection, virtual exemplars are learned by optimizing objectives that jointly encourage closeness to the unlabeled data distribution, balanced usage across exemplars, and maximal classifier ambiguity. The learned centers are then mapped to the nearest real, not-yet-labeled patch pairs for oracle annotation. This makes the virtual exemplar an intermediate optimization variable governing which real examples should be labeled next (Sahbi et al., 2022, Sahbi et al., 2022).

A fourth mechanism is abstraction and memory distillation. ICAL transforms noisy trajectories into optimized trajectories plus structured reasoning, stores the result in an external memory, and later retrieves top-$v_k$8 examples by a weighted combination of instruction, textual-state, and visual similarity. Exemplar-based CSSL describes a related compression process in which feature memories, nearest-neighbor supports, class prototypes, Neural Gas centroids, and Exemplar Relation Graphs act as virtual surrogates for many examples in few-shot class incremental learning (Sarch et al., 2024, Chang, 2022).

4. Major application regimes

In online action detection, virtual exemplars support two distinct priors. Historical frame features encode within-video temporal evolution, while class-level cluster centers provide a global, cross-video prior over typical appearances and motions within each action category. The resulting architecture uses two parallel branches—dynamic and static—with a consistency loss between their predictions, so temporal consultation and category consultation are jointly enforced (Yang et al., 2022).

In retrieval-augmented generation and in-context learning, virtual exemplars address three specific liabilities of textual demonstrations: inference cost from long contexts, degraded performance in specialized or out-of-distribution domains, and exemplar-order variance. MHA-RAG operationalizes the alternative by keeping the foundation model frozen and learning only the soft-prompt encoder and LoRA adapters in the sentence encoder, so retrieved examples are compressed before they reach the LLM (Jain et al., 6 Oct 2025).

In photometric stereo and spatially varying BRDF estimation, virtual exemplars replace physical calibration objects. A known BRDF dictionary and calibrated lighting define a precomputable table of synthetic observations over candidate normals, enabling example-based matching without captured exemplar spheres and avoiding iterative joint optimization over normals and reflectance parameters (Hui et al., 2015).

In active learning for remote sensing, virtual exemplars are display prototypes rather than decision-time memory reads. They are optimized in feature space, but the oracle is shown only real patch pairs nearest to those prototypes. The framework is therefore interactive yet indirect: the virtual exemplar shapes the query, while the human never labels the synthetic point itself (Sahbi et al., 2022, Sahbi et al., 2022).

In zero-shot learning, virtual exemplars provide a bridge between semantic side information and visually grounded discrimination. The predicted prototype can be used directly in nearest-neighbor classification or fed back into existing zero-shot methods as a denoised semantic representation. This shifts the emphasis from embedding images into a noisy semantic space to predicting where a class should lie in the visual space (Changpinyo et al., 2018).

In VR simulation, Robo-Saber uses style exemplars, contextual gameplay exemplars, and reference segments as compact virtual surrogates for human players. Each exemplar includes both 3-point motion and local game state, allowing a style encoder to infer a latent representation of skill and movement patterns that conditions future motion generation. In embodied-agent memory, ICAL’s programs of thought serve a comparable role: they are reusable multimodal memories that encode how a task should be carried out and why (Kim et al., 20 Feb 2026, Sarch et al., 2024).

5. Empirical properties, efficiency, and prompt-level selection

Several papers justify virtual exemplars primarily through efficiency or robustness. Colar reports 66.9% mAP on THUMOS14 with pre-extracted TSN-Kinetics features, compared with 65.2% for OadTR, while reducing GPU memory from 4375M to 2235M and processing a one-minute video in 50.7 s rather than 51.2 s. With raw frames and ResNet-I3D, it handles a one-minute video in 9.8 s with 58.8% mAP. The same study reports 88.1% mcAP on TVSeries and 30.6% mAP on HDD, with the best configuration requiring both dynamic and static exemplar consultation plus consistency loss (Yang et al., 2022).

MHA-RAG reports an average improvement of approximately $v_k$9 effective-accuracy points over standard RAG and approximately 10× lower inference FLOPs, with zero variance under random permutations of exemplars in Table 2. The reported gains include a ClinTox result for Qwen3-4B improving from 44.24 with RAG to 96.32 with MHA-RAG, and trainable-parameter counts of approximately 13M for Qwen3-4B MHA-RAG with $\hat v_u$0, versus approximately 132M for LoRA rank 64 on the same base model (Jain et al., 6 Oct 2025).

The active-learning studies emphasize label frugality under severe imbalance. One Jefferson dataset configuration contains 2,200 patch pairs with 2,161 negatives and 39 positives, and the full representativity-plus-diversity-plus-ambiguity objective yields the best average EER with an AUC of 10.47. The fully supervised classifier on the same benchmark achieves EER approximately 0.94%, establishing the ceiling toward which the frugal methods are measured (Sahbi et al., 2022). The adversarial variant likewise reports that its learned surrogate display model outperforms random, maxmin, uncertainty, and the related IGARSS 2021 strategy across iterations (Sahbi et al., 2022).

ICAL reports strong gains from turning raw demonstrations into abstracted memory exemplars. In TEACh, ICAL reaches 35.1% success rate and 49.3% goal-condition success, compared with 17.2% and 26.6% for raw visual demos, 26.5% and 29.5% for raw kinesthetic demos, and 34.5% and 36.7% for HELPER hand-written examples. In VisualWebArena, GPT-4V with Set-of-Marks improves from 14.3% average success to 22.7% with ICAL. In Ego4D, ICAL improves few-shot GPT-4V action-edit distance from 0.9414 to 0.9242 and noun-edit distance from 0.7575 to 0.6934 (Sarch et al., 2024).

Prompt-level exemplar optimization occupies an adjacent but important boundary of the topic. EASE treats a learned ordered exemplar sequence as a task-level “virtual prompt,” reports the best validation accuracy on 22 out of 24 Instruction Induction tasks, and shows that joint optimization of instructions and exemplars can raise auto_categorization from 30.0 to 46.7 and second_word_letter from 50 to 100 (Wu et al., 2024). EXPLORA describes a selected pool of reasoning subsets as a virtual curriculum, reduces LLM calls to approximately 11% of those required by LENS, and reports a 12.24% improvement together with gains such as 69.37 to 77.86 on GSM8K and 54.75 to 59.46 on FinQA (Purohit et al., 2024). These results indicate that virtualization can occur not only in exemplar content, but also in the organization, compression, and reuse of exemplar sets.

6. Limitations and open problems

Virtual exemplars inherit the assumptions of the space in which they are constructed. Colar notes that K-means category exemplars may miss rare modes under limited or imbalanced data, that complexity grows with $\hat v_u$1, and that the method is built on a closed-set exemplar bank; the authors explicitly note possible performance drops in open-set or long-tail real-world scenarios (Yang et al., 2022).

MHA-RAG identifies several unresolved issues around retrieval tailored to soft-prompt generation, persistent soft-prompt memories, lifelong updating of the encoder, interpretability of head specializations, and long-document “lost-in-the-middle” effects. The framework is order-invariant by design, but the paper does not provide extensive interpretability analyses of what each virtual token represents (Jain et al., 6 Oct 2025).

The photometric-stereo formulation depends on accurate lighting calibration, minimal cast shadows and interreflections, and a BRDF dictionary whose non-negative span covers the materials in the scene. Performance degrades when the dictionary is not representative, and precomputing virtual exemplars for many normals and BRDFs can be memory- and time-intensive even if the cost is amortized offline (Hui et al., 2015).

The remote-sensing active-learning papers depend heavily on the learned feature space, hyperparameter balance among representativity, diversity, ambiguity, and entropy regularization, and accurate registration of bi-temporal patches. The methods are computationally manageable for modest $\hat v_u$2, but every active-learning round still requires classifier retraining and inner optimization over memberships and exemplar positions (Sahbi et al., 2022, Sahbi et al., 2022).

Zero-shot virtual exemplars remain limited by the quality of semantic representations and by the fact that a single class mean cannot capture strong intra-class multimodality. Exemplar-based CSSL and continual-learning formulations raise related questions about which exemplars or feature memories to retain, how to scale neighborhood or graph structures, and how to combine explicit prototype modeling with incremental updates under tight memory budgets (Changpinyo et al., 2018, Chang, 2022).

Embodied and interactive systems add further constraints. Robo-Saber is demonstrated only on Beat Saber, depends on a large open gameplay dataset, and currently samples exemplars randomly from a player’s history rather than stratifying by difficulty or movement mode (Kim et al., 20 Feb 2026). ICAL can be misled by poor demonstrations or incorrect human feedback, assumes a fixed action API, and inherits the grounding limitations of current VLMs (Sarch et al., 2024). EXPLORA and EASE, finally, require a labeled validation set and depend on the quality of the embedding model used to represent exemplar sequences or subsets; both therefore treat prompt-level virtualization as an offline optimization problem whose success depends on the stability of the task distribution (Purohit et al., 2024, Wu et al., 2024).

A plausible implication is that “virtual exemplars” are best understood not as a single data structure but as a family of exemplar-preserving transformations. The common invariant is functional substitution: the model still reasons with examples, but the examples are compressed, synthesized, abstracted, or reorganized into forms better matched to the computational and statistical constraints of the task.