---
title: 'Few-Shot Evaluation: Protocols & Insights'
url: https://www.emergentmind.com/topics/few-shot-evaluation
type: topic
---

# Few-Shot Evaluation: Protocols & Insights

Few-shot evaluation denotes a family of evaluation protocols in which a model is assessed under extreme label scarcity, typically by adapting to a small labeled support set and then predicting labels for a disjoint query set. The paradigm is central to few-shot classification, relation classification and extraction, natural language understanding, speech, dense retrieval, and continual learning, but its concrete realization varies substantially across benchmarks. Some frameworks emphasize episodic $N$-way $K$-shot testing, some target task-level performance estimation for a single deployment task, and others introduce realistic stressors such as domain shift, spurious bias, query imbalance, or heavy none-of-the-above prevalence [1810.10147] [2107.07170] [2111.02570] [2204.11181] [2304.05845] [2603.00478].

## 1. Formal definitions and evaluation objects

In the canonical episodic formulation, an evaluation episode consists of a support set and a query set. A standard definition is an $N$-way $K$-shot task with support set
\[
S=\{(x_1,y_1),\ldots,(x_{N\cdot K},y_{N\cdot K})\}
\]
containing $K$ examples of each of $N$ novel classes, and query set
\[
Q=\{(x'_1,y'_1),\ldots,(x'_M,y'_M)\}.
\]
Episode accuracy is then
\[
\mathrm{acc}(S,Q)=\frac{1}{M}\sum_{i=1}^{M}\mathbf{1}\{\hat y'_i=y'_i\},
\]
with aggregate statistics computed over many sampled episodes [1904.04232]. FewRel uses the same structure for relation classification, defining a support set of size $N\times K$ over relations $R=\{r_1,\ldots,r_N\}$ and a classifier $F(R,S,\cdot)$ that maps each query instance to its relation label in $R$ [1810.10147].

A second, distinct object of evaluation is the single task rather than the benchmark average. In task-level evaluation, a task distribution $\mathcal P$ induces individual tasks $\tau$ with support set $S\sim p_\tau^n$ and query set $Q\sim p_\tau^m$. The true task performance is
\[
\mu(\tau)=\frac{1}{m}\sum_{(x,y)\in Q}\mathbf{1}[y=h_{\theta,S}(x)],
\]
whereas conventional few-shot benchmarks typically report only the aggregated estimate
\[
\hat\mu_{\mathrm{agg}}=\frac{1}{T}\sum_{i=1}^{T}\mu(\tau_i).
\]
This distinction is fundamental in deployment settings, where only the support set of a single task is available and the challenge is to estimate $\mu(\tau)$ from $S$ alone [2307.02732].

Few-shot evaluation has also been generalized beyond immediate prediction from a support set. In continual learning, few-shot evaluation measures the performance of a checkpoint $\theta_t$ after adaptation on at most $k$ examples from a target task:
\[
A_k(\theta,\mathcal T)=\mathrm{Accuracy}(U_k(\theta,\mathcal T^{train}),\mathcal T^{test}).
\]
Varying $k\in\{0,1,2,5,10\}$ yields a learning curve that captures how rapidly a model can reacquire past tasks or acquire future tasks [2606.03843].

This diversity of formalizations implies that “few-shot evaluation” is not a single protocol but a broader methodological category. A plausible implication is that benchmark-average performance, single-task estimability, and post-hoc adaptability should be treated as related but non-equivalent targets.

## 2. Benchmark construction and split design

Benchmark design determines which generalization problem is actually being measured. FewRel is a large supervised few-shot relation classification benchmark with 100 relation classes, exactly 700 human-verified sentences per relation, 124,577 unique tokens, and a 64/16/20 split for meta-training, meta-validation, and meta-testing [1810.10147]. FewCLUE constructs balanced Chinese few-shot support sets with task-dependent shot counts, such as 32 examples per label when $N\leq 3$, 16 per label for TNEWS, and 8 per label for CSLDCP and IFLYTEK, while using the remaining held-out public test set as the query set and releasing multiple random splits [2107.07498]. CLUES instead fixes few-shot pools of size $k\in\{10,20,30\}$ with nested splits across five seeds and evaluates on a single held-out test set of approximately 200 examples per task, with no development set [2111.02570].

FLEX broadens the notion of few-shot evaluation by defining four transfer settings—class transfer, domain transfer, task transfer, and pretraining transfer—plus zero-shot evaluation, and by freezing episode samples so that all methods are tested on identical episodes [2107.07170]. FEWTRANS makes task variability itself a design variable: it contains ten datasets, samples 600 tasks per dataset, draws the number of classes per task from $[2,15]$ with class imbalance allowed, and restricts shots to $K\leq 10$ [2603.00478]. FewDR, for few-shot dense retrieval, explicitly partitions 60 Wikidata-derived relation classes into 30 base and 30 novel classes and uses class-wise uniform sampling so that each novel class contributes exactly $k$ shots under each few-shot split, thereby preventing novel-class leakage and reducing split variance [2304.05845]. Realistic few-shot relation extraction imposes a different constraint set: disjoint relations across train, development, and test, limited support size, non-uniform class frequencies, many negatives, and relabeling non-target relations as NOTA [2404.04445].

| Design axis | Representative benchmark | Key choice |
|---|---|---|
| Fixed meta-splits | FewRel | 64/16/20 relations for meta-train/val/test |
| Multiple random support splits | FewCLUE, CLUES | five random splits; nested $k$-shot splits |
| Unified transfer settings | FLEX | class, domain, task, and pretraining transfer |
| Base/novel disjointness | FewDR | 30 base and 30 novel classes |
| Realistic negative prevalence | realistic FSRE | query labels in $C_{\mathrm{target}}\cup\{\mathrm{NOTA}\}$ |
| Variable-way imbalance | FEWTRANS | 2–15 classes per task, class-imbalanced |

The central methodological point is that sampling policy, class partitioning, and whether validation data are permitted all directly affect what counts as “few-shot ability.”

## 3. Metrics, reporting conventions, and human baselines

The most common primary metric is classification accuracy. FewRel reports
\[
\mathrm{Acc}=\frac{1}{|Q|}\sum_{(x_j,y_j)\in Q}\mathbb{I}(\hat y_j=y_j),
\]
averaged over all queries in all sampled episodes, and presents results as mean $\pm$ standard deviation over 10,000 episodes [1810.10147]. “A Closer Look at Few-shot Classification” similarly reports mean and either standard deviation or a $95\%$ confidence interval over 600 episodes [1904.04232]. FEWTRANS reports mean Top-1 accuracy with $95\%$ confidence intervals over 600 independent tasks [2603.00478].

Other modalities require different metrics. Realistic few-shot relation extraction reports precision, recall, $F_1$, and accuracy, with test-time aggregation over random episodes [2404.04445]. CLUES uses a unified instance-level span-set metric $S_1$ in addition to task-specific accuracy or macro-$F_1$ when appropriate, thereby supporting classification, sequence labeling, and machine reading comprehension within one evaluation framework [2111.02570]. The Polish benchmark reports Accuracy for multi-class tasks and positive-class $F_1$ for binary tasks [2404.17832]. FewDR reports Recall@$k$ and MRR for retrieval [2304.05845]. FLEURS evaluates ASR with character-level error rate, speech language identification with accuracy, speech translation with SacreBLEU, and cross-modal retrieval with Precision@1, always under fixed $n$-shot adaptation per language [2205.12446].

Several recent protocols introduce robustness-sensitive metrics rather than relying only on mean performance. FewSTAB defines standard accuracy, class-wise worst-case accuracy $\mathrm{wAcc}$, and the gap
\[
\Delta=\mathrm{wAcc}_R-\mathrm{wAcc}_A,
\]
where $\mathrm{wAcc}_R$ is computed on randomly sampled tasks and $\mathrm{wAcc}_A$ on FewSTAB’s bias-controlled adversarial tasks [2409.02882]. Continual-learning evaluation adds AUAC-based per-shot plasticity, SAUCE, to measure the rate of improvement under small-shot adaptation rather than only endpoint performance [2606.03843].

Human baselines remain important where they are available. FewRel reports human performance of $92.22\pm 5.53\%$ on 5-way 1-shot and $85.88\pm 7.40\%$ on 10-way 1-shot, far above the best model in those settings [1810.10147]. CLUES reports few-shot human baselines across classification, NER, and QA tasks, including approximately $94\%$ on ReCoRD under few-shot conditions [2111.02570]. FewCLUE reports a human average of $82.5\pm 1.1$, compared with substantially lower model averages [2107.07498].

## 4. Reliability, model selection, and the problem of scarce validation data

A recurring finding is that reliable evaluation is itself difficult in the few-shot regime. Task-level analysis shows that standard performance estimators can have large bias and variance when only a small support set is available. For 5-way 5-shot episodes, hold-out estimation has MAE of approximately 15–25 percentage points depending on model and dataset, bootstrapping yields approximately 10–20 percentage points, and 5-fold cross-validation yields approximately 5–14 percentage points, making 5-fold cross-validation the best off-the-shelf estimator for direct task-level accuracy estimation [2307.02732]. The same study finds that bootstrapping with $B\geq 100$ or leave-one-out cross-validation is better for ranking candidate models or tuning hyperparameters, even though direct performance estimation remains noisy [2307.02732].

FewNLU addresses a related issue with its Multi-Splits framework. It repeatedly splits the small labeled set into train and development folds with fixed ratio $r=0.5$, averages development performance over $K$ random splits, selects hyperparameters by average development score, and reports mean and standard deviation over the corresponding test evaluations [2109.12742]. The framework explicitly treats test performance, dev–test correlation, and stability as separate evaluation properties and finds that Multi-Splits outperforms alternative split strategies in all three [2109.12742].

FLEX frames sample-size selection as a statistical design problem. Under its simulation-based analysis, $N$ must exceed approximately 60 episodes to attain approximately $95\%$ confidence-interval coverage, and for a moderate budget of 48 GPU-hours the best trade-off is approximately $N=90$ and $\bar m=470$ test examples per episode [2107.07170]. FEWTRANS diagnoses two additional sources of unreliability: the “sampling lottery,” caused by evaluating on too few tasks, and the “validation set illusion,” caused by requiring unrealistic held-out validation sets in data-scarce settings [2603.00478]. Its Hyperparameter Ensemble protocol replaces single hyperparameter selection with logit averaging over a fixed grid of configurations,
\[
S(x)=\frac{1}{N}\sum_{i=1}^{N} g_{\phi_i}(x),
\]
and reports that results change by less than $0.2\%$ Top-1 accuracy when the hyperparameter grid is expanded [2603.00478].

These results indicate that few-shot evaluation has two coupled statistical problems: estimating model performance under limited support and selecting hyperparameters without a trustworthy development set. This suggests that benchmark design and estimator design are inseparable parts of the evaluation problem.

## 5. Realistic stress tests: imbalance, bias, negatives, and domain shift

Several benchmarks argue that conventional few-shot protocols are artificially easy or insufficiently realistic. A major example is transductive few-shot learning under query-set imbalance. Standard benchmarks for miniImageNet, tieredImageNet, and CUB typically assume perfectly balanced query sets, effectively fixing the marginal class distribution to $1/K$. “Realistic Evaluation of Transductive Few-Shot Learning” replaces this with query marginals sampled from a Dirichlet distribution and shows substantial performance drops for transductive methods when class proportions deviate from uniform, sometimes below inductive baselines [2204.11181]. On miniImageNet 5-way with mild imbalance ($\alpha=2$), PT-MAP drops from $85.3$ to $67.1$, Entropy Minimization drops from $82.1$ to $74.8$, TIM drops from $83.9$ to $79.8$, and LaplacianShot drops from $82.1$ to $81.6$ [2204.11181].

FewSTAB introduces a different realism criterion: robustness to spurious bias. It constructs few-shot tasks by choosing support images so that each class has a planted attribute as the only consistent cue, then chooses query images that violate that cue and minimize alternative shortcuts. Under this protocol, class-wise worst-case accuracy on FewSTAB tasks is much lower than on random tasks; on miniImageNet 5-way 5-shot, the average gap $\Delta$ is approximately 15 points, with random tasks reporting approximately $55\%$ worst-case accuracy and FewSTAB reporting approximately $40\%$ [2409.02882]. The benchmark therefore measures a failure mode that mean accuracy on randomly sampled episodes can conceal.

Relation extraction benchmarks reveal still other realism gaps. FewRel already showed that even strong few-shot relation classification models struggle relative to humans and that successful prediction may require pattern matching, common-sense world knowledge, logical inference or composition, and co-reference and discourse understanding [1810.10147]. The later realistic FSRE benchmark adds unseen test relations, heavy NOTA prevalence, non-uniform class frequencies, and the restriction that models cannot rely on large background data for test relations; under this setup no single method is best across datasets and overall performance is often below 35 $F_1$ [2404.04445].

Cross-domain studies expose another vulnerability. “A Closer Look at Few-shot Classification” reports that in the miniImageNet $\rightarrow$ CUB setting, a simple baseline with standard fine-tuning achieves $65.57\pm 0.70$ in 5-shot with ResNet-18, outperforming the evaluated meta-learning methods [1904.04232]. In histopathology, training on Mini-ImageNet and testing on histopathological datasets leads to a severe accuracy drop of approximately 15–20 percentage points relative to training on a large histopathology corpus, and within-domain standard methods slightly outperform episodic methods in high-shot regimes [2410.09176].

Taken together, these stress tests show that few-shot evaluation can fail through at least four distinct artefacts: balanced-query assumptions, accidental attribute shortcuts, unrealistic label spaces without NOTA, and weak domain-shift protocols.

## 6. Empirical patterns across modalities and open directions

Across domains, a common empirical pattern is that benchmark performance under few-shot constraints remains substantially below human or full-data performance. In FewRel, Prototypical Networks reach $69.20\pm 0.20$ in 5-way 1-shot and $56.44\pm 0.22$ in 10-way 1-shot, compared with human performance of $92.22\pm 5.53$ and $85.88\pm 7.40$ respectively [1810.10147]. CLUES finds that prompt-tuning substantially outperforms classic fine-tuning in few-shot classification, yet large human–machine gaps persist for token-level and span-extraction tasks; on ReCoRD, humans are approximately $94\%$ few-shot while pretrained language models remain at or below $21\%$ [2111.02570]. FewCLUE reports PET and P-Tuning as the best overall few-shot methods, with RoBERTa+PET and RoBERTa+P-Tuning both at $59.9\pm 2.2$ average accuracy, still far below the human average of $82.5\pm 1.1$ [2107.07498]. The Polish benchmark similarly finds that in-context learning is best, with GPT-3.5 at $59.5\pm 1.4$ for 16-shot average performance, while full fine-tuning of HerBERT-large on all training data reaches $79.9\pm 0.6$ [2404.17832].

In speech, FLEURS shows steep gains from 1-shot to 10-shot and diminishing returns thereafter. For ASR, mSLAM improves macro-averaged CER from $68.5\%$ at 1-shot to $38.2\%$ at 10-shot and $23.9\%$ at 50-shot; for speech language identification, it improves from $21.8\%$ at 1-shot to $63.0\%$ at 10-shot and $78.2\%$ at 50-shot [2205.12446]. In dense retrieval, FewDR reports that 5-shot performance on novel classes is approximately 23 percentage points below the full-shot upper bound and that the slope of Recall@10 as $k$ increases is similar across models, suggesting that current few-shot gains are largely driven by stronger base representations rather than specialized adaptation [2304.05845]. FEWTRANS reaches a similar conclusion for vision transfer: the choice of pre-trained model dominates performance, adapter and parameter-efficient methods cluster within approximately $\pm 1\%$ of full fine-tuning, and the main jump comes from stronger pre-training rather than from sophisticated transfer algorithms [2603.00478]. At the same time, architecture changes still matter under fixed protocols: ViT-ProtoNet reports up to a $3.2\%$ gain over a Conv-4 ProtoNet baseline in 5-shot image classification and competitive performance against transformer-based alternatives on several datasets [2507.09299].

Open directions identified by these benchmarks are concrete. FewRel points to external commonsense or world knowledge bases, coreference resolution, discourse context, richer embedding spaces, and hybrid metric-based and optimization-based meta-learners [1810.10147]. Task-level evaluation work calls for Bayesian estimators, meta-learning of validation procedures, side information, and performance metrics beyond simple accuracy [2307.02732]. FewSTAB proposes more complex spurious structures and multimodal extensions [2409.02882]. Continual-learning evaluation argues for reporting $k$-shot forward and backward curves and using regret-based metrics such as SAUCE to quantify adaptation speed [2606.03843].

Few-shot evaluation has therefore evolved from a narrow episodic accuracy protocol into a broader discipline of benchmark design, statistical estimation, robustness testing, and deployment-oriented model assessment. The resulting picture is technically consistent across modalities: evaluation outcomes depend not only on the learning algorithm but also on how tasks are sampled, how hyperparameters are selected, which failure modes are stress-tested, and whether the protocol measures mean performance, worst-case robustness, or adaptation speed.

Source: https://www.emergentmind.com/topics/few-shot-evaluation