Exemplar Optimization Methods

• Exemplar optimization is a methodology for selecting a representative subset of data points to enable efficient learning, inference, and visualization.
• It employs diverse techniques such as submodular maximization, bilevel optimization, and variational pseudocoreset construction to balance accuracy, memory constraints, and computational efficiency.
• Empirical results show that these methods enhance in-context learning, memory-constrained setups, and model calibration while addressing scalability and validation challenges.

Exemplar optimization is a broad set of methodologies for selecting or constructing a small, representative, and often highly informative subset of data points—called exemplars—from a larger collection, with the aim of enabling efficient and effective learning, inference, or visualization. The optimization criteria and the algorithmic landscape are highly problem-dependent, spanning submodular maximization, bilevel optimization, combinatorial Bayesian optimization, variational pseudocoreset construction, convex relaxation, and hybrid strategies with semantic or feedback-driven regularization. Exemplar optimization is central in domains such as in-context learning for LLMs, memory-constrained lifelong learning, data visualization, generative modeling, and specialized tasks such as exemplar-based object detection and translation.

1. Formal Problem Definitions and Core Objectives

At its most abstract, exemplar optimization seeks the subset $S \subseteq \mathcal{E}$ (of size $k$), or weighted pseudocoreset, that maximizes a task-specific utility function $F(S)$, possibly under constraints (cardinality, memory, semantic diversity, etc.). Notable instantiations include:

• Subset selection for in-context learning: $$S^* = \arg\max_{S,\,|S|=k} \mathbb{E}_{(x,y)\sim \mathcal{D}_{\text{val}}}[g(f_{\mathrm{LLM}}(I^*,S,x),y)]$$, where $g$ is the evaluation metric and $f_{\mathrm{LLM}}$ the black-box LLM (Wan et al., 2024).
• Memory-bounded exemplar compression: Jointly optimize compressed exemplars to maximize downstream incremental learning accuracy subject to a strict memory budget (Luo et al., 2023).
• Exemplar-based priors in generative models: Minimize the KL divergence between a pseudocoreset-induced prior and the full dataset prior, $$\min_{U,w} D_{\mathrm{KL}}(p_\phi(z|U,w) \Vert p_\phi(z|X))$$ (Ai et al., 2021).
• Multi-objective optimization: Simultaneously maximize accuracy and minimize calibration error, yielding the Pareto frontier of exemplar sets (Luo et al., 1 Oct 2025).

Table: Representative Problems and Optimization Criteria

Application Domain	Objective Function	Constraints
ICL for LLMs	Accuracy, calibration (ECE), generalizability	Fixed-shot, context length
Class Incremental Learning	Final/test accuracy, memory use, mask sparsity	Global memory bound
Bayesian deep modeling	KL divergence to full-data prior, marginal likelihood	Pseudocoreset size/weights
Clustering, facility-location	Submodular facility or representativity function	Cardinality (matroid)
Exemplar SVM calibration	False positive count at full positive coverage	Coverage of all positives

The objectives are often nonconvex and combinatorial (especially for large $|\mathcal{E}|,k$), motivating approximate algorithms, sampling, and surrogate modeling.

2. Algorithmic Approaches and Representative Methods

Several algorithmic paradigms have emerged, tailored to different structure and constraints:

• Combinatorial Bayesian Optimization (BO): Constructs surrogate models (e.g., Gaussian processes with Hamming kernel) over the $\{0,1\}^m$ selection space, using acquisition functions (e.g., NEHVI) to efficiently explore the Pareto frontier in multi-objective settings (Luo et al., 1 Oct 2025).
• Submodular Maximization and Facility-location: Models representativity as a monotone submodular function $F(S)$ and uses greedy algorithms under matroid constraints (per-class or global) with mean-field or semantic priors (Ronando et al., 26 Dec 2025, Honysz et al., 2021). The greedy algorithm yields a $(1-1/e)$-approximation in such cases, and GPU acceleration addresses the computational bottleneck (Honysz et al., 2021).
• Convex Relaxation & Greedy Frank–Wolfe: Relaxes the Boolean selection problem to an $\ell_{2,1}$-ball, solved by projection-free Frank–Wolfe/Conditional Gradient with kernelization (K-FWSR), achieving sparse selection in $O(n^2)$ per iteration and enjoying global linear convergence under mild conditions (Cheng et al., 2018).
• Bilevel Optimization: Inner loop trains/replays incremental learners on compressed exemplars, while an outer loop optimizes the exemplar- or mask-generation parameters for best validation accuracy, subject to memory constraints (Luo et al., 2023).
• Variational Pseudocoreset Optimization: Minimizes KL-divergence between the induced latent prior of a weighted pseudocoreset and that of the complete dataset, using stochastic gradients and Monte Carlo estimation (Ai et al., 2021).
• Neural Bandit and Surrogate-based Acquisition: Surrogate neural networks using order-aware embeddings and bandit-style (UCB) acquisition functions jointly optimize exemplar sets and instruction text under a fixed evaluation query budget (Wu et al., 2024).
• Evolutionary and Enumerative Black-box Search: Mutation/random search over exemplar combinations, especially in in-context learning and prompt optimization, often outperform instruction-only optimization under equivalent LLM evaluation budgets (Wan et al., 2024).
• Prioritized Memory Replay: Incorporates historical feedback and exemplars with dynamic scoring/fading, supporting experience-balanced selection and reducing catastrophic forgetting in iterative prompt refinement (Yan et al., 2024).

3. Theoretical Guarantees and Complexity

Theoretical properties depend on the problem class and surrogate structure:

• Submodular maximization: Guarantees $(1-1/e)$-approximation via greedy under monotonicity and matroid constraints (Ronando et al., 26 Dec 2025, Honysz et al., 2021).
• Frank–Wolfe approaches: Convex relaxations with $\ell_{2,1}$-ball constraints admit global geometric (linear) convergence under suitable step selection and kernel regularity (Cheng et al., 2018).
• Bayesian optimization: Sample efficiency and convergence to the true Pareto frontier depend on surrogate fidelity and acquisition function, with NEHVI encouraging optimal exploration-exploitation (Luo et al., 1 Oct 2025).
• Bandit-based subset selection: Under linear surrogates, PAC sample complexity is $O(d/\Delta^2 \log(1/\delta))$ for best-arm and top-$\ell$ identification, where $d$ is dimension, $\Delta$ the utility gap, and $\delta$ the failure probability (Purohit et al., 2024).
• Bilevel optimization for memory constraints: Alternating gradient scheme converges to local minima; empirically yields consistent improvement under memory constraint (Luo et al., 2023).
• Pseudocoreset variational minimization: KL divergence minimization over pseudodata converges to a local optimum; the resultant prior approximates the full mixture prior with $O(M)$ instead of $O(N)$ complexity (Ai et al., 2021).
• Joint calibration of ensemble SVMs: Pruned DFS with equivalence and difficulty ordering guarantees global optimality for hundreds of exemplars and any-time approximation for thousands (Modolo et al., 2015).

4. Empirical Findings and Benchmarking

Exemplar optimization has demonstrated consistent, significant empirical gains across modalities and tasks:

• Few-shot human activity recognition: LLM-guided exemplars (with facility location objective and semantic priors) yield Macro F1 of 88.79%, outperforming random, herding, $k$-center, with ablations confirming the necessity of both facility and semantic components (+19.7 and +2.1 points) (Ronando et al., 26 Dec 2025).
• Class-incremental learning: Adaptive mask-based exemplar compression boosts ImageNet-1000 10-phase average accuracy from 54.72% (FOSTER) to 59.48%, with similar gains for other datasets and memory budgets (Luo et al., 2023).
• In-context learning for LLMs: Exemplar optimization methods like EASE and EXPLORA improve fixed-prompt performance by 5–15% compared to retrieval-based or diversity-based heuristics, and require 10x fewer LLM API calls than exhaustive search (Wu et al., 2024, Purohit et al., 2024).
• Pareto-optimality between accuracy and calibration: COM-BOM advances both metrics—on Qwen3, accuracy 45.3% vs. 43.6% (baselines) and ECE 25.2% vs. 28.1%—while reducing API calls (Luo et al., 1 Oct 2025).
• Prompt and instruction optimization synergy: Exemplar search (mutation/random) alone can outperform instruction optimization; best results combine both (76.1% vs. 70.8% or 72.9% alone) (Wan et al., 2024).
• Data embedding and visualization: Exemplar-centered parametric t-SNE (dt-SEE/hot-SEE) matches or exceeds classical t-SNE in 1-NN test error and neighborhood preservation, while being robust to batch size and perplexity (Min et al., 2017).
• Exemplar SVM ensembles: Joint calibration lowers window-classification AP and detection mAP by 2–3 points over independent calibration (Modolo et al., 2015).

5. Domain-Specific Extensions and Hybridization

Exemplar optimization is often hybridized with semantic, structure-aware, or adversarial elements:

• LLM-generated semantic priors: Incorporation of feature-importance and confusability matrices regularizes selection, especially in human activity recognition (Ronando et al., 26 Dec 2025).
• Chain-of-thought and feedback memory: In iterative prompt-refinement, exemplar-guided reflection with prioritized feedback accelerates convergence and reduces query cost (Yan et al., 2024).
• Ordering sensitivity: In ICL settings, correct ordering of exemplars in the prompt can yield 5–15% higher test accuracy than order-invariant methods; EASE addresses this via embedding of ordered exemplar sequences (Wu et al., 2024).
• Dynamic vs. static selection: Static optimization (single prompt for all test queries) is computationally and privacy-efficient, but dynamic selection (retrieval-based, per-query) can accommodate non-stationarity at the cost of inference complexity (Purohit et al., 2024).
• Online vs. offline adaptation: Exemplar optimization frameworks such as OEFT operate in an online, per-instance regime, leveraging adaptive correspondence and GAN inversion for exemplar-based translation without offline domain training (Kang et al., 2020).

6. Practical Considerations, Limitations, and Open Directions

Exemplar optimization's scalability, efficiency, and generalizability depend critically on algorithm design and resource management:

• Computational bottlenecks: Embedding large candidate pools (EASE), optimizing over huge subset spaces (submodular or combinatorial), and LLM API evaluation budgets are recurring constraints (Wu et al., 2024, Purohit et al., 2024, Luo et al., 1 Oct 2025).
• Memory and test-time complexity: Fixed-prompt (static) methods and exemplar compression mitigate test-time cost and privacy leakage (Luo et al., 2023, Purohit et al., 2024).
• Validation dependence: Many algorithms require held-out validation sets for surrogate fitting or frontier estimation; low-data regimes remain challenging (Wu et al., 2024).
• Theory gaps: While convex and submodular formulations admit provable bounds, more expressive surrogate-based and feedback-memory approaches lack end-to-end regret or convergence guarantees in high dimensions (Yan et al., 2024, Wu et al., 2024).
• Hybrid and joint optimization: Empirical evidence favors joint optimization of exemplars and instructions, as well as memory-based replay and experience prioritization. The development of scalable, theoretically principled approaches for such combinations remains ongoing (Wan et al., 2024, Yan et al., 2024).
• Generalization beyond input-label pairs: Extensions to chain-of-thought reasoning, open-domain demonstration retrieval, and compressed/synthetic exemplars are active areas (Purohit et al., 2024, Ai et al., 2021).

A plausible implication is that exemplar optimization, in its varied guises, will remain a central substrate for resource-aware, interpretable, and high-performing ML systems, as the dependence on a small, well-chosen memory, demonstration, or support set is both computationally and statistically beneficial across domains.