UCS: Estimating Unseen Coverage for Improved In-Context Learning

Abstract: In-context learning (ICL) performance depends critically on which demonstrations are placed in the prompt, yet most existing selectors prioritize heuristic notions of relevance or diversity and provide limited insight into the coverage of a demonstration set. We propose Unseen Coverage Selection (UKS), a training-free, subset-level coverage prior motivated by the principle that a good demonstration set should expose the model to latent cluster unrevealed by the currently selected subset. UCS operationalizes this idea by (1) inducing discrete latent clusters from model-consistent embeddings and (2) estimating the number of unrevealed clusters within a candidate subset via a Smoothed Good--Turing estimator from its empirical frequency spectrum. Unlike previous selection methods, UCS is coverage-based and training-free, and can be seamlessly combined with both query-dependent and query-independent selection baselines via a simple regularized objective. Experiments on multiple intent-classification and reasoning benchmarks with frontier LLMs show that augmenting strong baselines with UCS consistently improves ICL accuracy by up to 2-6% under the same selection budget, while also yielding insights into task- and model-level latent cluster distributions. Code is available at https://github.com/Raina-Xin/UCS.

• The paper introduces UCS, a demonstration selection method that estimates latent cluster coverage using a Smoothed Good-Turing estimator.
• It employs model-consistent embeddings with PCA and DBSCAN clustering to capture both dominant and singleton clusters within demonstration sets.
• Empirical results show robust ICL accuracy gains across benchmarks by reducing redundancy and targeting long-tail conceptual variations.

UCS: Estimating Unseen Coverage for Improved In-Context Learning

Introduction and Motivation

The paper "UCS: Estimating Unseen Coverage for Improved In-Context Learning" (2604.12015) addresses the critical challenge of demonstration selection in in-context learning (ICL) for LLMs. Empirically, ICL performance demonstrates extreme sensitivity to which demonstrations are presented in the prompt under a fixed selection budget. Traditional retrievers optimize for instance-level objectives such as relevance or diversity, but rarely provide guarantees or direct control over coverage of the task's latent structure, leading to issues with insufficient generalization when novel or underrepresented concepts occur at test time.

The core hypothesis of the work is that demonstration coverage---the extent to which the demonstration set reveals the latent clusters underlying the task---is fundamental for robust ICL. The authors present UCS (Unseen Coverage Selection), a framework to explicitly regularize demonstration selection by estimating, in a statistically principled manner, how much of the latent cluster space remains unseen in the selected subset. By operationalizing coverage as a subset-level prior using a Smoothed Good--Turing (SGT) estimator over clustered model-consistent embeddings, UCS can be combined with strong query-independent and query-dependent selectors (including DPP, MDL, VoteK) without additional training.

Figure 1: Contrast of prior instance-level methods with UCS, which reasons over latent clusters and optimizes subset coverage.

UCS Framework and Methodological Innovations

Model-Consistent Embeddings and Clustering

UCS first embeds all demonstrations in the candidate set using the same LLM utilized at inference, ensuring inductive alignment between the latent space for clustering and the target model. For each demonstration, mean-pooled representations from a designated layer are extracted. These embeddings are then processed via Principal Component Analysis (PCA) for dimensionality reduction.

A dictionary learning step decomposes the reduced embeddings into codes over a latent basis. Instead of naive argmax assignment to dictionary atoms—-which fails to capture compositionality in the code space—the codes are clustered using DBSCAN under cosine distance, after normalization. Singleton clusters correspond to points labeled as noise, preserving tail semantics. This two-stage clustering design enables discovery of dense, recurring latent structures as well as fine-grained outlier units, as validated by the heavy-tailed cluster size distributions observed empirically.

Figure 2: Schematic of UCS including embedding, clustering, SGT-based coverage estimation, and regularization of demo selection.

Smoothed Good–Turing Unseen Cluster Estimator

For a given demonstration subset, the frequency-of-frequencies spectrum of clusters is computed. The SGT estimator then extrapolates, with stabilizations for finite-sample scenarios, how many new clusters would be uncovered by subsampling additional demonstrations from the same candidate pool. The UCS coverage score, $\Phi_\mathrm{UCS}(S)$, is defined as the sum of covered (seen) cluster types and the SGT-estimated number of unseen clusters in the subset. The estimator is truncated at moderate counts with optional parametric smoothing, and includes offset-based stabilization of higher-order terms for robustness under sparse sampling.

Importantly, $\Phi_\mathrm{UCS}$ is a true subset-level functional and non-decomposable over instances. It thus regulates coverage gain at the collective level, unlike instance-level utilities.

Integration with Base Selection Methods

UCS is instantiated as a light-weight regularizer term added to the base selection objective of state-of-the-art selectors:

• For query-dependent selectors such as DPP and MDL, a greedy or global maximization of $$U_\mathrm{base}(S; x_\mathrm{test}) + \lambda\, \Phi_\mathrm{UCS}(S)$$ is performed.
• For query-independent selectors like VoteK, a corpus-level SGT-weighted rarity prior is applied to cluster vote scores.

Implementation is efficient: most computation is amortized in the offline embedding and clustering phase, with online demonstration selection incurring only marginal additional cost.

Empirical Results

ICL Accuracy Gains

Across three intent classification benchmarks (BANKING77, CLINC150, HWU64), three challenging BBEH reasoning tasks, and several LLM backbones (Qwen2.5-7B-Instruct, Llama-3.2-3B-Instruct, Gemma-2-9B-it), UCS consistently improves ICL accuracy under a fixed demonstration budget. Notably, query-independent selectors such as VoteK benefit most: gains of up to +6.2% (HWU64, Qwen2.5-7B) and +4.1% (CLINC150) are observed, eliminating redundancy and forcing selection of singleton clusters.

For strong query-dependent baselines (MDL, DPP), UCS still yields robust, if smaller, improvements, particularly in the low- and mid-accuracy regime. For reasoning tasks, the UCS regularizer enables selectors to target long-tail conceptual variation, leading to improvements (e.g., +8.4% on Causal Understanding with MDL, Qwen2.5-7B).

Analysis of Latent Cluster Structure

Cluster size distributions induced by UCS clustering reveal strong heavy-tailed behavior: the majority of demonstrations reside in small clusters, with a tiny fraction in dominant modes. Coverage-focused selection directly increases the number of unique clusters represented in the prompt and reduces average per-prompt cluster size, especially for corpus-level selectors.

Figure 3: Cluster-size distributions across datasets/LLMs; most clusters are singletons, with a few large dominant units.

Ablations and Controls

• Clustering Approaches: Dictionary+DBSCAN-based clustering outperforms direct embedding clustering or argmax atom assignment. Ridge coding and density-based clustering yield clusters better aligned with meaningful latent structures.
• SGT Scoring: The SGT-based unseen cluster extrapolation is essential; replacing the coverage prior with naive rarity weighting fails to recover the UCS performance gains and has inconsistent impact.
• Sensitivity: UCS is robust to SGT hyperparameters and selection budget, and maintains advantage across a range of these settings. Multi-model embedding alignment (joint dictionary learning) does not outperform model-consistent single-source clustering, highlighting limits in aligning LLM representations.

Theoretical and Practical Implications

UCS reframes demonstration selection as a coverage-driven problem, introducing subset-level regularization grounded in classical statistics of unseen species estimation. This enables systematic control over prompt-induced cluster coverage and reduces the occurrence of brittle prompt sets that neglect rare or novel latent concepts. By decoupling coverage gain from base selector utility, UCS unifies the instance-level and subset-level selection paradigms.

Practically, UCS is training-free, efficient, and retrofits onto established automated selection methods. Its SGT-based latent cluster statistics also serve as interpretable diagnostics of the representational breadth of LLMs and task datasets. This property opens avenues for future work in distributional shift detection, active learning under annotation constraints, and prompt set design for performance and robustness guarantees.

Theoretically, integrating subset-level coverage priors could inform advances in submodular optimization, representation learning, and meta-ICL strategies, especially as models and tasks scale. The UCS framework could bridge to RL-based or iterative selection policies by serving as a coverage-aware reward or constraint, with implications for reinforcement learning-driven demo acquisition and generalized compositionality in LLMs.

Conclusion

UCS provides a statistically rigorous approach for regularizing demonstration selection in ICL by leveraging unsupervised clustering of model-consistent embeddings and extrapolating latent cluster coverage with an SGT estimator. The method consistently improves or matches existing selectors across tasks, backbones, and evaluation regimes. Beyond accuracy, UCS introduces a means to analyze and control the latent semantic diversity of demonstration sets, enabling more robust AI systems that better generalize to unseen concepts. Future research should explore semantically grounded hybrid clustering, extension to complex input regimes, and integration with sequential policy-driven selection.