Few-Shot Evaluation: Protocols & Insights

• Few-shot evaluation is a framework where models adapt from a small labeled support set to predict a disjoint query set under extreme label scarcity.
• It spans tasks like classification, relation extraction, and continual learning, with evaluation protocols varying from episodic to task-level schemes.
• Protocols incorporate realistic challenges such as domain shift, spurious bias, and query imbalance to better assess model robustness and deployment viability.

Few-shot evaluation denotes a family of evaluation protocols in which a model is assessed under extreme label scarcity, typically by adapting to a small labeled support set and then predicting labels for a disjoint query set. The paradigm is central to few-shot classification, relation classification and extraction, natural language understanding, speech, dense retrieval, and continual learning, but its concrete realization varies substantially across benchmarks. Some frameworks emphasize episodic $N$-way $K$-shot testing, some target task-level performance estimation for a single deployment task, and others introduce realistic stressors such as domain shift, spurious bias, query imbalance, or heavy none-of-the-above prevalence (Han et al., 2018, Bragg et al., 2021, Mukherjee et al., 2021, Veilleux et al., 2022, Sun et al., 2023, Luo et al., 28 Feb 2026).

1. Formal definitions and evaluation objects

In the canonical episodic formulation, an evaluation episode consists of a support set and a query set. A standard definition is an $N$-way $K$-shot task with support set

$S=\{(x_1,y_1),\ldots,(x_{N\cdot K},y_{N\cdot K})\}$

containing $K$ examples of each of $N$ novel classes, and query set

$Q=\{(x'_1,y'_1),\ldots,(x'_M,y'_M)\}.$

Episode accuracy is then

$$\mathrm{acc}(S,Q)=\frac{1}{M}\sum_{i=1}^{M}\mathbf{1}\{\hat y'_i=y'_i\},$$

with aggregate statistics computed over many sampled episodes (Chen et al., 2019). FewRel uses the same structure for relation classification, defining a support set of size $N\times K$ over relations $K$0 and a classifier $K$1 that maps each query instance to its relation label in $K$2 (Han et al., 2018).

A second, distinct object of evaluation is the single task rather than the benchmark average. In task-level evaluation, a task distribution $K$3 induces individual tasks $K$4 with support set $K$5 and query set $K$6. The true task performance is

$K$7

whereas conventional few-shot benchmarks typically report only the aggregated estimate

$K$8

This distinction is fundamental in deployment settings, where only the support set of a single task is available and the challenge is to estimate $K$9 from $N$0 alone (Shimabucoro et al., 2023).

Few-shot evaluation has also been generalized beyond immediate prediction from a support set. In continual learning, few-shot evaluation measures the performance of a checkpoint $N$1 after adaptation on at most $N$2 examples from a target task: $N$3 Varying $N$4 yields a learning curve that captures how rapidly a model can reacquire past tasks or acquire future tasks (Inamdar et al., 2 Jun 2026).

This diversity of formalizations implies that “few-shot evaluation” is not a single protocol but a broader methodological category. A plausible implication is that benchmark-average performance, single-task estimability, and post-hoc adaptability should be treated as related but non-equivalent targets.

2. Benchmark construction and split design

Benchmark design determines which generalization problem is actually being measured. FewRel is a large supervised few-shot relation classification benchmark with 100 relation classes, exactly 700 human-verified sentences per relation, 124,577 unique tokens, and a 64/16/20 split for meta-training, meta-validation, and meta-testing (Han et al., 2018). FewCLUE constructs balanced Chinese few-shot support sets with task-dependent shot counts, such as 32 examples per label when $N$5, 16 per label for TNEWS, and 8 per label for CSLDCP and IFLYTEK, while using the remaining held-out public test set as the query set and releasing multiple random splits (Xu et al., 2021). CLUES instead fixes few-shot pools of size $N$6 with nested splits across five seeds and evaluates on a single held-out test set of approximately 200 examples per task, with no development set (Mukherjee et al., 2021).

FLEX broadens the notion of few-shot evaluation by defining four transfer settings—class transfer, domain transfer, task transfer, and pretraining transfer—plus zero-shot evaluation, and by freezing episode samples so that all methods are tested on identical episodes (Bragg et al., 2021). FEWTRANS makes task variability itself a design variable: it contains ten datasets, samples 600 tasks per dataset, draws the number of classes per task from $N$7 with class imbalance allowed, and restricts shots to $N$8 (Luo et al., 28 Feb 2026). FewDR, for few-shot dense retrieval, explicitly partitions 60 Wikidata-derived relation classes into 30 base and 30 novel classes and uses class-wise uniform sampling so that each novel class contributes exactly $N$9 shots under each few-shot split, thereby preventing novel-class leakage and reducing split variance (Sun et al., 2023). Realistic few-shot relation extraction imposes a different constraint set: disjoint relations across train, development, and test, limited support size, non-uniform class frequencies, many negatives, and relabeling non-target relations as NOTA (Alam et al., 2024).

Design axis	Representative benchmark	Key choice
Fixed meta-splits	FewRel	64/16/20 relations for meta-train/val/test
Multiple random support splits	FewCLUE, CLUES	five random splits; nested $K$0-shot splits
Unified transfer settings	FLEX	class, domain, task, and pretraining transfer
Base/novel disjointness	FewDR	30 base and 30 novel classes
Realistic negative prevalence	realistic FSRE	query labels in $K$1
Variable-way imbalance	FEWTRANS	2–15 classes per task, class-imbalanced

The central methodological point is that sampling policy, class partitioning, and whether validation data are permitted all directly affect what counts as “few-shot ability.”

3. Metrics, reporting conventions, and human baselines

The most common primary metric is classification accuracy. FewRel reports

$K$2

averaged over all queries in all sampled episodes, and presents results as mean $K$3 standard deviation over 10,000 episodes (Han et al., 2018). “A Closer Look at Few-shot Classification” similarly reports mean and either standard deviation or a $K$4 confidence interval over 600 episodes (Chen et al., 2019). FEWTRANS reports mean Top-1 accuracy with $K$5 confidence intervals over 600 independent tasks (Luo et al., 28 Feb 2026).

Other modalities require different metrics. Realistic few-shot relation extraction reports precision, recall, $K$6, and accuracy, with test-time aggregation over random episodes (Alam et al., 2024). CLUES uses a unified instance-level span-set metric $K$7 in addition to task-specific accuracy or macro-$K$8 when appropriate, thereby supporting classification, sequence labeling, and machine reading comprehension within one evaluation framework (Mukherjee et al., 2021). The Polish benchmark reports Accuracy for multi-class tasks and positive-class $K$9 for binary tasks (Hadeliya et al., 2024). FewDR reports Recall@$S=\{(x_1,y_1),\ldots,(x_{N\cdot K},y_{N\cdot K})\}$0 and MRR for retrieval (Sun et al., 2023). FLEURS evaluates ASR with character-level error rate, speech language identification with accuracy, speech translation with SacreBLEU, and cross-modal retrieval with Precision@1, always under fixed $S=\{(x_1,y_1),\ldots,(x_{N\cdot K},y_{N\cdot K})\}$1-shot adaptation per language (Conneau et al., 2022).

Several recent protocols introduce robustness-sensitive metrics rather than relying only on mean performance. FewSTAB defines standard accuracy, class-wise worst-case accuracy $S=\{(x_1,y_1),\ldots,(x_{N\cdot K},y_{N\cdot K})\}$2, and the gap

$S=\{(x_1,y_1),\ldots,(x_{N\cdot K},y_{N\cdot K})\}$3

where $S=\{(x_1,y_1),\ldots,(x_{N\cdot K},y_{N\cdot K})\}$4 is computed on randomly sampled tasks and $S=\{(x_1,y_1),\ldots,(x_{N\cdot K},y_{N\cdot K})\}$5 on FewSTAB’s bias-controlled adversarial tasks (Zheng et al., 2024). Continual-learning evaluation adds AUAC-based per-shot plasticity, SAUCE, to measure the rate of improvement under small-shot adaptation rather than only endpoint performance (Inamdar et al., 2 Jun 2026).

Human baselines remain important where they are available. FewRel reports human performance of $S=\{(x_1,y_1),\ldots,(x_{N\cdot K},y_{N\cdot K})\}$6 on 5-way 1-shot and $S=\{(x_1,y_1),\ldots,(x_{N\cdot K},y_{N\cdot K})\}$7 on 10-way 1-shot, far above the best model in those settings (Han et al., 2018). CLUES reports few-shot human baselines across classification, NER, and QA tasks, including approximately $S=\{(x_1,y_1),\ldots,(x_{N\cdot K},y_{N\cdot K})\}$8 on ReCoRD under few-shot conditions (Mukherjee et al., 2021). FewCLUE reports a human average of $S=\{(x_1,y_1),\ldots,(x_{N\cdot K},y_{N\cdot K})\}$9, compared with substantially lower model averages (Xu et al., 2021).

4. Reliability, model selection, and the problem of scarce validation data

A recurring finding is that reliable evaluation is itself difficult in the few-shot regime. Task-level analysis shows that standard performance estimators can have large bias and variance when only a small support set is available. For 5-way 5-shot episodes, hold-out estimation has MAE of approximately 15–25 percentage points depending on model and dataset, bootstrapping yields approximately 10–20 percentage points, and 5-fold cross-validation yields approximately 5–14 percentage points, making 5-fold cross-validation the best off-the-shelf estimator for direct task-level accuracy estimation (Shimabucoro et al., 2023). The same study finds that bootstrapping with $K$0 or leave-one-out cross-validation is better for ranking candidate models or tuning hyperparameters, even though direct performance estimation remains noisy (Shimabucoro et al., 2023).

FewNLU addresses a related issue with its Multi-Splits framework. It repeatedly splits the small labeled set into train and development folds with fixed ratio $K$1, averages development performance over $K$2 random splits, selects hyperparameters by average development score, and reports mean and standard deviation over the corresponding test evaluations (Zheng et al., 2021). The framework explicitly treats test performance, dev–test correlation, and stability as separate evaluation properties and finds that Multi-Splits outperforms alternative split strategies in all three (Zheng et al., 2021).

FLEX frames sample-size selection as a statistical design problem. Under its simulation-based analysis, $K$3 must exceed approximately 60 episodes to attain approximately $K$4 confidence-interval coverage, and for a moderate budget of 48 GPU-hours the best trade-off is approximately $K$5 and $K$6 test examples per episode (Bragg et al., 2021). FEWTRANS diagnoses two additional sources of unreliability: the “sampling lottery,” caused by evaluating on too few tasks, and the “validation set illusion,” caused by requiring unrealistic held-out validation sets in data-scarce settings (Luo et al., 28 Feb 2026). Its Hyperparameter Ensemble protocol replaces single hyperparameter selection with logit averaging over a fixed grid of configurations,

$K$7

and reports that results change by less than $K$8 Top-1 accuracy when the hyperparameter grid is expanded (Luo et al., 28 Feb 2026).

These results indicate that few-shot evaluation has two coupled statistical problems: estimating model performance under limited support and selecting hyperparameters without a trustworthy development set. This suggests that benchmark design and estimator design are inseparable parts of the evaluation problem.

5. Realistic stress tests: imbalance, bias, negatives, and domain shift

Several benchmarks argue that conventional few-shot protocols are artificially easy or insufficiently realistic. A major example is transductive few-shot learning under query-set imbalance. Standard benchmarks for miniImageNet, tieredImageNet, and CUB typically assume perfectly balanced query sets, effectively fixing the marginal class distribution to $K$9. “Realistic Evaluation of Transductive Few-Shot Learning” replaces this with query marginals sampled from a Dirichlet distribution and shows substantial performance drops for transductive methods when class proportions deviate from uniform, sometimes below inductive baselines (Veilleux et al., 2022). On miniImageNet 5-way with mild imbalance ($N$0), PT-MAP drops from $N$1 to $N$2, Entropy Minimization drops from $N$3 to $N$4, TIM drops from $N$5 to $N$6, and LaplacianShot drops from $N$7 to $N$8 (Veilleux et al., 2022).

FewSTAB introduces a different realism criterion: robustness to spurious bias. It constructs few-shot tasks by choosing support images so that each class has a planted attribute as the only consistent cue, then chooses query images that violate that cue and minimize alternative shortcuts. Under this protocol, class-wise worst-case accuracy on FewSTAB tasks is much lower than on random tasks; on miniImageNet 5-way 5-shot, the average gap $N$9 is approximately 15 points, with random tasks reporting approximately $Q=\{(x'_1,y'_1),\ldots,(x'_M,y'_M)\}.$0 worst-case accuracy and FewSTAB reporting approximately $Q=\{(x'_1,y'_1),\ldots,(x'_M,y'_M)\}.$1 (Zheng et al., 2024). The benchmark therefore measures a failure mode that mean accuracy on randomly sampled episodes can conceal.

Relation extraction benchmarks reveal still other realism gaps. FewRel already showed that even strong few-shot relation classification models struggle relative to humans and that successful prediction may require pattern matching, common-sense world knowledge, logical inference or composition, and co-reference and discourse understanding (Han et al., 2018). The later realistic FSRE benchmark adds unseen test relations, heavy NOTA prevalence, non-uniform class frequencies, and the restriction that models cannot rely on large background data for test relations; under this setup no single method is best across datasets and overall performance is often below 35 $Q=\{(x'_1,y'_1),\ldots,(x'_M,y'_M)\}.$2 (Alam et al., 2024).

Cross-domain studies expose another vulnerability. “A Closer Look at Few-shot Classification” reports that in the miniImageNet $Q=\{(x'_1,y'_1),\ldots,(x'_M,y'_M)\}.$3 CUB setting, a simple baseline with standard fine-tuning achieves $Q=\{(x'_1,y'_1),\ldots,(x'_M,y'_M)\}.$4 in 5-shot with ResNet-18, outperforming the evaluated meta-learning methods (Chen et al., 2019). In histopathology, training on Mini-ImageNet and testing on histopathological datasets leads to a severe accuracy drop of approximately 15–20 percentage points relative to training on a large histopathology corpus, and within-domain standard methods slightly outperform episodic methods in high-shot regimes (Sekhar et al., 2024).

Taken together, these stress tests show that few-shot evaluation can fail through at least four distinct artefacts: balanced-query assumptions, accidental attribute shortcuts, unrealistic label spaces without NOTA, and weak domain-shift protocols.

6. Empirical patterns across modalities and open directions

Across domains, a common empirical pattern is that benchmark performance under few-shot constraints remains substantially below human or full-data performance. In FewRel, Prototypical Networks reach $Q=\{(x'_1,y'_1),\ldots,(x'_M,y'_M)\}.$5 in 5-way 1-shot and $Q=\{(x'_1,y'_1),\ldots,(x'_M,y'_M)\}.$6 in 10-way 1-shot, compared with human performance of $Q=\{(x'_1,y'_1),\ldots,(x'_M,y'_M)\}.$7 and $Q=\{(x'_1,y'_1),\ldots,(x'_M,y'_M)\}.$8 respectively (Han et al., 2018). CLUES finds that prompt-tuning substantially outperforms classic fine-tuning in few-shot classification, yet large human–machine gaps persist for token-level and span-extraction tasks; on ReCoRD, humans are approximately $Q=\{(x'_1,y'_1),\ldots,(x'_M,y'_M)\}.$9 few-shot while pretrained LLMs remain at or below $$\mathrm{acc}(S,Q)=\frac{1}{M}\sum_{i=1}^{M}\mathbf{1}\{\hat y'_i=y'_i\},$$0 (Mukherjee et al., 2021). FewCLUE reports PET and P-Tuning as the best overall few-shot methods, with RoBERTa+PET and RoBERTa+P-Tuning both at $$\mathrm{acc}(S,Q)=\frac{1}{M}\sum_{i=1}^{M}\mathbf{1}\{\hat y'_i=y'_i\},$$1 average accuracy, still far below the human average of $$\mathrm{acc}(S,Q)=\frac{1}{M}\sum_{i=1}^{M}\mathbf{1}\{\hat y'_i=y'_i\},$$2 (Xu et al., 2021). The Polish benchmark similarly finds that in-context learning is best, with GPT-3.5 at $$\mathrm{acc}(S,Q)=\frac{1}{M}\sum_{i=1}^{M}\mathbf{1}\{\hat y'_i=y'_i\},$$3 for 16-shot average performance, while full fine-tuning of HerBERT-large on all training data reaches $$\mathrm{acc}(S,Q)=\frac{1}{M}\sum_{i=1}^{M}\mathbf{1}\{\hat y'_i=y'_i\},$$4 (Hadeliya et al., 2024).

In speech, FLEURS shows steep gains from 1-shot to 10-shot and diminishing returns thereafter. For ASR, mSLAM improves macro-averaged CER from $$\mathrm{acc}(S,Q)=\frac{1}{M}\sum_{i=1}^{M}\mathbf{1}\{\hat y'_i=y'_i\},$$5 at 1-shot to $$\mathrm{acc}(S,Q)=\frac{1}{M}\sum_{i=1}^{M}\mathbf{1}\{\hat y'_i=y'_i\},$$6 at 10-shot and $$\mathrm{acc}(S,Q)=\frac{1}{M}\sum_{i=1}^{M}\mathbf{1}\{\hat y'_i=y'_i\},$$7 at 50-shot; for speech language identification, it improves from $$\mathrm{acc}(S,Q)=\frac{1}{M}\sum_{i=1}^{M}\mathbf{1}\{\hat y'_i=y'_i\},$$8 at 1-shot to $$\mathrm{acc}(S,Q)=\frac{1}{M}\sum_{i=1}^{M}\mathbf{1}\{\hat y'_i=y'_i\},$$9 at 10-shot and $N\times K$0 at 50-shot (Conneau et al., 2022). In dense retrieval, FewDR reports that 5-shot performance on novel classes is approximately 23 percentage points below the full-shot upper bound and that the slope of Recall@10 as $N\times K$1 increases is similar across models, suggesting that current few-shot gains are largely driven by stronger base representations rather than specialized adaptation (Sun et al., 2023). FEWTRANS reaches a similar conclusion for vision transfer: the choice of pre-trained model dominates performance, adapter and parameter-efficient methods cluster within approximately $N\times K$2 of full fine-tuning, and the main jump comes from stronger pre-training rather than from sophisticated transfer algorithms (Luo et al., 28 Feb 2026). At the same time, architecture changes still matter under fixed protocols: ViT-ProtoNet reports up to a $N\times K$3 gain over a Conv-4 ProtoNet baseline in 5-shot image classification and competitive performance against transformer-based alternatives on several datasets (Mutlu et al., 12 Jul 2025).

Open directions identified by these benchmarks are concrete. FewRel points to external commonsense or world knowledge bases, coreference resolution, discourse context, richer embedding spaces, and hybrid metric-based and optimization-based meta-learners (Han et al., 2018). Task-level evaluation work calls for Bayesian estimators, meta-learning of validation procedures, side information, and performance metrics beyond simple accuracy (Shimabucoro et al., 2023). FewSTAB proposes more complex spurious structures and multimodal extensions (Zheng et al., 2024). Continual-learning evaluation argues for reporting $N\times K$4-shot forward and backward curves and using regret-based metrics such as SAUCE to quantify adaptation speed (Inamdar et al., 2 Jun 2026).

Few-shot evaluation has therefore evolved from a narrow episodic accuracy protocol into a broader discipline of benchmark design, statistical estimation, robustness testing, and deployment-oriented model assessment. The resulting picture is technically consistent across modalities: evaluation outcomes depend not only on the learning algorithm but also on how tasks are sampled, how hyperparameters are selected, which failure modes are stress-tested, and whether the protocol measures mean performance, worst-case robustness, or adaptation speed.