Hold-One-Shot-Out (HOSO) in Few-Shot Learning

• HOSO is a mechanism that holds out a one-shot example to prevent overfitting, leakage, and improper reuse of limited supervision.
• It enables precise calibration by excluding the self-induced predictor, ensuring unbiased aggregation in conformal prediction or hyperparameter tuning scenarios.
• HOSO is applied in distinct settings, including leave-one-out calibration in CAOS and validation-free optimization in CLIP adapters for few-shot tasks.

Searching arXiv for the cited HOSO-related papers to ground the article in current arXiv records. Hold-One-Shot-Out (HOSO) is a context-dependent hold-out mechanism for one-shot or few-shot learning in which a single labeled example, or the predictor induced by that example, is excluded from part of the training or calibration pipeline to control overfitting, leakage, or invalid reuse of scarce supervision. In the arXiv literature represented here, the term has two distinct technical realizations. In "CAOS: Conformal Aggregation of One-Shot Predictors" (Waldron, 8 Jan 2026), the underlying idea appears as leave-one-out calibration for one-shot conformal prediction: each calibration example is scored after removing the one-shot predictor induced by that same example. In "Hold-One-Shot-Out (HOSO) for Validation-Free Few-Shot CLIP Adapters" (Vorster et al., 4 Mar 2026), HOSO denotes a validation-free procedure that holds out exactly one labeled image per class from the few-shot support to learn a CLIP-adapter blending ratio. By contrast, "NAS-Bench-1Shot1: Benchmarking and Dissecting One-shot Neural Architecture Search" (Zela et al., 2020) does not define or use HOSO.

1. Terminological scope and conceptual core

The shared conceptual core of HOSO is selective exclusion under severe label scarcity. A one-shot element is held out so that the remaining data can be used for model fitting, aggregation, or calibration, while the held-out element provides an unbiased or less biased signal for a secondary objective. The exact object being held out differs by setting.

In CAOS, what is held out is the one-shot predictor induced by the calibration example itself. The paper does not use the name “HOSO”; it uses “leave-one-out calibration.” The provided formulation identifies HOSO with that leave-one-out calibration idea specialized to one-shot predictors: for each labeled example, the calibration score is computed after removing that example’s own induced predictor from the aggregation pool (Waldron, 8 Jan 2026).

In the CLIP-adapter setting, HOSO is the explicit name of a protocol in which exactly one labeled image per class is removed from the few-shot support and used as a micro-validation cache to optimize the blending ratio $\alpha$, while the adapter is trained on the remaining support examples (Vorster et al., 4 Mar 2026).

These two usages are analogous in spirit but not identical in mechanics. One addresses conformal uncertainty quantification with finite-sample coverage guarantees; the other addresses hyperparameter learning under a strict validation-free few-shot protocol. This suggests that “HOSO” is best treated not as a single algorithm, but as a family resemblance among one-shot hold-out constructions.

2. HOSO as leave-one-out calibration in CAOS

In CAOS, the problem setting is one-shot prediction with foundation models under a small labeled dataset $D = \{(X_i,Y_i)\}_{i=1}^n$. Each labeled example induces its own one-shot predictor through a fixed mechanism $\pi$:

$$\pi_i(\cdot \mid x) := \pi(\cdot \mid x;\,(X_i,Y_i)).$$

Each predictor has an associated nonconformity score $s_{\pi_i}(x,y) \in \mathbb{R}$ (Waldron, 8 Jan 2026).

The paper gives two concrete instantiations. For landmark transfer in vision via patch similarity,

$$s_{\pi_i}(x,y) = 1 - \mathrm{sim}\!\bigl(e_x(y), e_{X_i}(Y_i)\bigr),$$

with point prediction

$$\hat y_i = \argmin_{y \in \mathcal{Y}} s_{\pi_i}(x,y).$$

For one-shot text classification with LLMs via in-context learning,

$$s_{\pi_i}(x,y) = \mathrm{AvgNLL}\!\bigl(y \mid prompt(X_i,Y_i,x)\bigr),$$

again with

$$\hat y_i = \argmin_{y \in \mathcal{Y}} s_{\pi_i}(x,y).$$

The motivation is that split conformal is inefficient in the one-shot regime because it requires disjoint reference and calibration splits and equips each one-shot predictor independently. The baseline for a fixed $i$ uses calibration scores

$D = \{(X_i,Y_i)\}_{i=1}^n$0

threshold

$D = \{(X_i,Y_i)\}_{i=1}^n$1

and prediction set

$D = \{(X_i,Y_i)\}_{i=1}^n$2

with

$D = \{(X_i,Y_i)\}_{i=1}^n$3

CAOS replaces this with adaptive aggregation across multiple one-shot predictors. For any reference set $D = \{(X_i,Y_i)\}_{i=1}^n$4 and target $D = \{(X_i,Y_i)\}_{i=1}^n$5, it defines the score multiset

$D = \{(X_i,Y_i)\}_{i=1}^n$6

and the $D = \{(X_i,Y_i)\}_{i=1}^n$7-min sum operator

$D = \{(X_i,Y_i)\}_{i=1}^n$8

where $D = \{(X_i,Y_i)\}_{i=1}^n$9 are the ordered values in $\pi$0.

The test-time aggregated nonconformity score is

$\pi$1

HOSO enters through the leave-one-out calibration statistic. For each $\pi$2, define

$\pi$3

and compute

$\pi$4

The calibration quantile is

$\pi$5

and the final prediction set is

$\pi$6

Operationally, HOSO here means that when calibrating on $\pi$7, the unique one-shot predictor generated from $\pi$8 is excluded. This prevents a “self-score” advantage, avoids self-score leakage, and allows all $\pi$9 labeled examples to participate in calibration without data splitting (Waldron, 8 Jan 2026).

3. Algorithmic structure and theory of CAOS-HOSO

The CAOS construction proceeds in three stages. First, multiple one-shot predictors are built by instantiating $$\pi_i(\cdot \mid x) := \pi(\cdot \mid x;\,(X_i,Y_i)).$$0 and $$\pi_i(\cdot \mid x) := \pi(\cdot \mid x;\,(X_i,Y_i)).$$1 for each labeled example. Second, HOSO or leave-one-out calibration scores $$\pi_i(\cdot \mid x) := \pi(\cdot \mid x;\,(X_i,Y_i)).$$2 are computed using $$\pi_i(\cdot \mid x) := \pi(\cdot \mid x;\,(X_i,Y_i)).$$3, followed by the quantile $$\pi_i(\cdot \mid x) := \pi(\cdot \mid x;\,(X_i,Y_i)).$$4. Third, for each candidate $$\pi_i(\cdot \mid x) := \pi(\cdot \mid x;\,(X_i,Y_i)).$$5 at test time, the aggregated score $$\pi_i(\cdot \mid x) := \pi(\cdot \mid x;\,(X_i,Y_i)).$$6 is computed over the full labeled pool $$\pi_i(\cdot \mid x) := \pi(\cdot \mid x;\,(X_i,Y_i)).$$7, and $$\pi_i(\cdot \mid x) := \pi(\cdot \mid x;\,(X_i,Y_i)).$$8 is returned (Waldron, 8 Jan 2026).

The aggregation rule is the $$\pi_i(\cdot \mid x) := \pi(\cdot \mid x;\,(X_i,Y_i)).$$9-min-sum operator, averaged by $s_{\pi_i}(x,y) \in \mathbb{R}$0. The paper states that this selects the $s_{\pi_i}(x,y) \in \mathbb{R}$1 most supportive one-shot predictors for a given $s_{\pi_i}(x,y) \in \mathbb{R}$2, avoiding dilution by weak or irrelevant references. In experiments, $s_{\pi_i}(x,y) \in \mathbb{R}$3 is used throughout. The operator is described as permutation-invariant and monotone in the reference set.

The finite-sample coverage theorem is stated under exchangeability and a self-score condition. The required assumption is

$s_{\pi_i}(x,y) \in \mathbb{R}$4

Under exchangeability of the labeled data $s_{\pi_i}(x,y) \in \mathbb{R}$5 and the test example $s_{\pi_i}(x,y) \in \mathbb{R}$6, and under this self-score optimality assumption, the CAOS prediction set satisfies

$s_{\pi_i}(x,y) \in \mathbb{R}$7

The proof route is not classical exchangeability of the score sequence, because adaptive aggregation and leave-one-out calibration violate the usual form. Instead, the paper introduces a full CAOS construction with a hypothetical test label $s_{\pi_i}(x,y) \in \mathbb{R}$8:

$s_{\pi_i}(x,y) \in \mathbb{R}$9

For each calibration index $$s_{\pi_i}(x,y) = 1 - \mathrm{sim}\!\bigl(e_x(y), e_{X_i}(Y_i)\bigr),$$0, the full CAOS score is

$$s_{\pi_i}(x,y) = 1 - \mathrm{sim}\!\bigl(e_x(y), e_{X_i}(Y_i)\bigr),$$1

The corresponding threshold is

$$s_{\pi_i}(x,y) = 1 - \mathrm{sim}\!\bigl(e_x(y), e_{X_i}(Y_i)\bigr),$$2

and the full conformal set is

$$s_{\pi_i}(x,y) = 1 - \mathrm{sim}\!\bigl(e_x(y), e_{X_i}(Y_i)\bigr),$$3

The key lemmas are: symmetry of the full CAOS score, monotonicity of the CAOS score with respect to enlarging the reference set, and a comparison lemma asserting

$$s_{\pi_i}(x,y) = 1 - \mathrm{sim}\!\bigl(e_x(y), e_{X_i}(Y_i)\bigr),$$4

The argument then yields $$s_{\pi_i}(x,y) = 1 - \mathrm{sim}\!\bigl(e_x(y), e_{X_i}(Y_i)\bigr),$$5 together with test-score equivalence under self-score optimality, giving

$$s_{\pi_i}(x,y) = 1 - \mathrm{sim}\!\bigl(e_x(y), e_{X_i}(Y_i)\bigr),$$6

and hence the CAOS coverage guarantee. The significance of this construction is that CAOS attains exact marginal coverage without adding extra slack, despite score-level non-exchangeability (Waldron, 8 Jan 2026).

A plausible implication is that HOSO in this sense is not merely a data-efficiency heuristic; it is part of a formally justified conformal design whose validity depends on monotonicity, permutation symmetry, exchangeability of examples, and self-score optimality.

4. HOSO for validation-free few-shot CLIP adapters

In the CLIP-adapter literature, HOSO denotes a distinct procedure aimed at learning the blending ratio $$s_{\pi_i}(x,y) = 1 - \mathrm{sim}\!\bigl(e_x(y), e_{X_i}(Y_i)\bigr),$$7 without a labeled validation set. The setting begins with frozen CLIP encoders: an image encoder $$s_{\pi_i}(x,y) = 1 - \mathrm{sim}\!\bigl(e_x(y), e_{X_i}(Y_i)\bigr),$$8 and text encoder $$s_{\pi_i}(x,y) = 1 - \mathrm{sim}\!\bigl(e_x(y), e_{X_i}(Y_i)\bigr),$$9, both producing $$\hat y_i = \argmin_{y \in \mathcal{Y}} s_{\pi_i}(x,y).$$0-normalized embeddings. For class $$\hat y_i = \argmin_{y \in \mathcal{Y}} s_{\pi_i}(x,y).$$1, the class prototype is

$$\hat y_i = \argmin_{y \in \mathcal{Y}} s_{\pi_i}(x,y).$$2

For image $$\hat y_i = \argmin_{y \in \mathcal{Y}} s_{\pi_i}(x,y).$$3, $$\hat y_i = \argmin_{y \in \mathcal{Y}} s_{\pi_i}(x,y).$$4 with $$\hat y_i = \argmin_{y \in \mathcal{Y}} s_{\pi_i}(x,y).$$5, and zero-shot probabilities are

$$\hat y_i = \argmin_{y \in \mathcal{Y}} s_{\pi_i}(x,y).$$6

CLIP-Adapter introduces an adapter $$\hat y_i = \argmin_{y \in \mathcal{Y}} s_{\pi_i}(x,y).$$7, a bottleneck MLP, with adapted feature

$$\hat y_i = \argmin_{y \in \mathcal{Y}} s_{\pi_i}(x,y).$$8

The blended representation is

$$\hat y_i = \argmin_{y \in \mathcal{Y}} s_{\pi_i}(x,y).$$9

followed by $$s_{\pi_i}(x,y) = \mathrm{AvgNLL}\!\bigl(y \mid prompt(X_i,Y_i,x)\bigr),$$0 normalization and classification using the same text prototypes and temperature $$s_{\pi_i}(x,y) = \mathrm{AvgNLL}\!\bigl(y \mid prompt(X_i,Y_i,x)\bigr),$$1:

$$s_{\pi_i}(x,y) = \mathrm{AvgNLL}\!\bigl(y \mid prompt(X_i,Y_i,x)\bigr),$$2

Here $$s_{\pi_i}(x,y) = \mathrm{AvgNLL}\!\bigl(y \mid prompt(X_i,Y_i,x)\bigr),$$3 is the blending ratio, $$s_{\pi_i}(x,y) = \mathrm{AvgNLL}\!\bigl(y \mid prompt(X_i,Y_i,x)\bigr),$$4 are adapter parameters, and the protocol is explicitly validation-free (Vorster et al., 4 Mar 2026).

The HOSO step is to hold out exactly one labeled image per class from the few-shot support $$s_{\pi_i}(x,y) = \mathrm{AvgNLL}\!\bigl(y \mid prompt(X_i,Y_i,x)\bigr),$$5 to form a tiny cache $$s_{\pi_i}(x,y) = \mathrm{AvgNLL}\!\bigl(y \mid prompt(X_i,Y_i,x)\bigr),$$6 of size $$s_{\pi_i}(x,y) = \mathrm{AvgNLL}\!\bigl(y \mid prompt(X_i,Y_i,x)\bigr),$$7 images. Those examples are removed from the adapter’s training set, leaving $$s_{\pi_i}(x,y) = \mathrm{AvgNLL}\!\bigl(y \mid prompt(X_i,Y_i,x)\bigr),$$8. The held-out shot is selected randomly per class once at the start and kept fixed. The blending ratio is a single global scalar per dataset, not per-class and not per-layer.

To constrain the optimization, HOSO parameterizes $$s_{\pi_i}(x,y) = \mathrm{AvgNLL}\!\bigl(y \mid prompt(X_i,Y_i,x)\bigr),$$9 by a logit:

$$\hat y_i = \argmin_{y \in \mathcal{Y}} s_{\pi_i}(x,y).$$0

so that $$\hat y_i = \argmin_{y \in \mathcal{Y}} s_{\pi_i}(x,y).$$1. The stated purpose is to stabilize optimization and avoid degenerate solutions that fully discard either CLIP or the adapter.

Optimization is decoupled. On $$\hat y_i = \argmin_{y \in \mathcal{Y}} s_{\pi_i}(x,y).$$2, the adapter is trained with

$$\hat y_i = \argmin_{y \in \mathcal{Y}} s_{\pi_i}(x,y).$$3

updating only adapter parameters $$\hat y_i = \argmin_{y \in \mathcal{Y}} s_{\pi_i}(x,y).$$4 while keeping $$\hat y_i = \argmin_{y \in \mathcal{Y}} s_{\pi_i}(x,y).$$5 fixed during that step. On the cache $$\hat y_i = \argmin_{y \in \mathcal{Y}} s_{\pi_i}(x,y).$$6, HOSO updates only $$\hat y_i = \argmin_{y \in \mathcal{Y}} s_{\pi_i}(x,y).$$7 via

$$\hat y_i = \argmin_{y \in \mathcal{Y}} s_{\pi_i}(x,y).$$8

equivalently

$$\hat y_i = \argmin_{y \in \mathcal{Y}} s_{\pi_i}(x,y).$$9

in single-example notation. The paper uses two separate SGD optimizers, initializes $i$0 to $i$1 via $i$2, and updates the ratio each epoch using the hold-out cache (Vorster et al., 4 Mar 2026).

This HOSO formulation differs sharply from CAOS-HOSO. The hold-out object is no longer a one-shot predictor within a conformal pipeline, but a one-shot-per-class micro-validation set for learning a hyperparameter under strict few-shot constraints.

5. Empirical results and comparative behavior

The empirical evidence for CAOS-HOSO is reported on one-shot facial landmarking and RAFT text classification. In the facial landmarking setup with 478 landmarks and $i$3 patches, the reported results are as follows.

Setting	Coverage / size
$i$4, SCOS Avg.	$i$5 / $i$6
$i$7, SCOS Best	$i$8 / $i$9
$D = \{(X_i,Y_i)\}_{i=1}^n$00, CAOS	$D = \{(X_i,Y_i)\}_{i=1}^n$01 / $D = \{(X_i,Y_i)\}_{i=1}^n$02
$D = \{(X_i,Y_i)\}_{i=1}^n$03, Oracle	$D = \{(X_i,Y_i)\}_{i=1}^n$04 / $D = \{(X_i,Y_i)\}_{i=1}^n$05
$D = \{(X_i,Y_i)\}_{i=1}^n$06, SCOS Avg.	$D = \{(X_i,Y_i)\}_{i=1}^n$07 / $D = \{(X_i,Y_i)\}_{i=1}^n$08
$D = \{(X_i,Y_i)\}_{i=1}^n$09, SCOS Best	$D = \{(X_i,Y_i)\}_{i=1}^n$10 / $D = \{(X_i,Y_i)\}_{i=1}^n$11
$D = \{(X_i,Y_i)\}_{i=1}^n$12, CAOS	$D = \{(X_i,Y_i)\}_{i=1}^n$13 / $D = \{(X_i,Y_i)\}_{i=1}^n$14
$D = \{(X_i,Y_i)\}_{i=1}^n$15, Oracle	$D = \{(X_i,Y_i)\}_{i=1}^n$16 / $D = \{(X_i,Y_i)\}_{i=1}^n$17
$D = \{(X_i,Y_i)\}_{i=1}^n$18, SCOS Avg.	$D = \{(X_i,Y_i)\}_{i=1}^n$19 / $D = \{(X_i,Y_i)\}_{i=1}^n$20
$D = \{(X_i,Y_i)\}_{i=1}^n$21, SCOS Best	$D = \{(X_i,Y_i)\}_{i=1}^n$22 / $D = \{(X_i,Y_i)\}_{i=1}^n$23
$D = \{(X_i,Y_i)\}_{i=1}^n$24, CAOS	$D = \{(X_i,Y_i)\}_{i=1}^n$25 / $D = \{(X_i,Y_i)\}_{i=1}^n$26
$D = \{(X_i,Y_i)\}_{i=1}^n$27, Oracle	$D = \{(X_i,Y_i)\}_{i=1}^n$28 / $D = \{(X_i,Y_i)\}_{i=1}^n$29

The paper summarizes this as CAOS consistently attaining target coverage while reducing set sizes relative to split conformal baselines and approaching oracle efficiency in many landmarks. For RAFT one-shot text classification with Llama2-7B at $D = \{(X_i,Y_i)\}_{i=1}^n$30, CAOS produces smaller prediction sets in $D = \{(X_i,Y_i)\}_{i=1}^n$31 tasks; in coverage terms, $D = \{(X_i,Y_i)\}_{i=1}^n$32 tasks are hit by both CAOS and SCOS, $D = \{(X_i,Y_i)\}_{i=1}^n$33 only by CAOS, and $D = \{(X_i,Y_i)\}_{i=1}^n$34 are missed by both, with the note that small test-set size leads to noisy coverage estimates (Waldron, 8 Jan 2026).

For HOSO-Adapter, the validation-free few-shot protocol uses $D = \{(X_i,Y_i)\}_{i=1}^n$35 shots per class in the main experiments and evaluates on 11 standard few-shot classification datasets: ImageNet, Caltech101, OxfordPets, StanfordCars, Flowers102, Food101, FGVCAircraft, SUN397, DTD, EuroSAT, and UCF101. The paper states that HOSO-Adapter outperforms the CLIP-Adapter baseline by more than 4 percentage points on average across 11 standard few-shot datasets, and that in the 8- and 16-shot settings it outperforms CLIP-Adapter even with the optimal blending ratio selected on the test set (Vorster et al., 4 Mar 2026).

Ablations on RN50, 16-shot, reported as averages, are:

Variant	Average accuracy
Full HOSO-Adapter	76.43%
Without decoupled training	73.02%
Keep cache samples in adapter training set	73.35%
Use 2 shots per class in cache	76.04%
Use 8 shots per class in cache	73.68%

The RN50 16-shot snapshot, averaged over 3 runs, gives HOSO-Adapter at 75.25%, CLIP-Adapter (validation-free reimplementation) at 73.35%, CLIP zero-shot at 57.71%, TIP-Adapter (training-free) at 64.61%, and PathCLIP (reimplementation) at 73.35%. The ViT-B/16 16-shot snapshot gives HOSO-Adapter at 80.33%, CLIP-Adapter at 75.82%, and CLIP-Adapter “best $D = \{(X_i,Y_i)\}_{i=1}^n$36” tuned on the test set over $D = \{(X_i,Y_i)\}_{i=1}^n$37 at 81.07%, which the paper notes is not strictly comparable. Reported per-dataset improvements versus validation-free CLIP-Adapter include DTD $D = \{(X_i,Y_i)\}_{i=1}^n$38, FGVCAircraft $D = \{(X_i,Y_i)\}_{i=1}^n$39, and EuroSAT $D = \{(X_i,Y_i)\}_{i=1}^n$40 points (Vorster et al., 4 Mar 2026).

The paper also reports across-shot averages. RN50 HOSO-Adapter averages are 63.03 for 2-shot, 65.13 for 4-shot, 71.78 for 8-shot, and 75.25 for 16-shot. ViT-B/16 HOSO-Adapter averages are 69.76, 71.61, 77.43, and 80.33, respectively. It further reports that jointly learning $D = \{(X_i,Y_i)\}_{i=1}^n$41 on the support set without hold-out causes $D = \{(X_i,Y_i)\}_{i=1}^n$42 to increase monotonically during training and overfit, whereas HOSO’s $D = \{(X_i,Y_i)\}_{i=1}^n$43 stays conservative and drops when overfitting is detected, reducing train–test gaps across datasets (Vorster et al., 4 Mar 2026).

6. Practical conditions, limitations, and relation to adjacent literature

The practical guidance for CAOS-HOSO is to use all available labeled examples to instantiate one-shot predictors, compute each calibration score $D = \{(X_i,Y_i)\}_{i=1}^n$44 with $D = \{(X_i,Y_i)\}_{i=1}^n$45, and choose a small $D = \{(X_i,Y_i)\}_{i=1}^n$46, with $D = \{(X_i,Y_i)\}_{i=1}^n$47 used throughout the experiments. The score functions should be comparable across references, and the aggregation should remain symmetric to reference permutations, monotone in the reference set, and compatible with the self-score optimality assumption. The paper identifies possible failure modes: violating self-score optimality can break the set-inclusion step in the proof; retraining-dependent scores or aggregation that depends on model parameters fitted on augmented data may violate the required assumptions. Computationally, per test input and candidate $D = \{(X_i,Y_i)\}_{i=1}^n$48, computing $D = \{(X_i,Y_i)\}_{i=1}^n$49 is $D = \{(X_i,Y_i)\}_{i=1}^n$50; finding the $D = \{(X_i,Y_i)\}_{i=1}^n$51 smallest can be done in $D = \{(X_i,Y_i)\}_{i=1}^n$52 with selection or $D = \{(X_i,Y_i)\}_{i=1}^n$53 with a heap; overall complexity is $D = \{(X_i,Y_i)\}_{i=1}^n$54 per test input, with $D = \{(X_i,Y_i)\}_{i=1}^n$55 preprocessing for calibration (Waldron, 8 Jan 2026).

The practical guidance for HOSO-Adapter is correspondingly specific: hold out exactly one random sample per class, train the adapter on $D = \{(X_i,Y_i)\}_{i=1}^n$56, optimize $D = \{(X_i,Y_i)\}_{i=1}^n$57 on the cache $D = \{(X_i,Y_i)\}_{i=1}^n$58, use decoupled optimization with two optimizers, and keep $D = \{(X_i,Y_i)\}_{i=1}^n$59 in $D = \{(X_i,Y_i)\}_{i=1}^n$60 with initialization near $D = \{(X_i,Y_i)\}_{i=1}^n$61. The reported defaults are an adapter bottleneck MLP with reduction factor 4 and ReLU, 200 epochs, batch size 32, SGD with momentum 0.9, weight decay 0.0005, cosine learning-rate schedule, initial learning rate 0.002 for the adapter, and a separate ratio optimizer with learning rate 0.1. The paper warns against jointly optimizing $D = \{(X_i,Y_i)\}_{i=1}^n$62 and the adapter on the same few-shot data, keeping the held-out samples in the adapter training set, or using large cache sizes; it reports that one-shot per class is best on average (Vorster et al., 4 Mar 2026).

The literature boundary is also explicit. "NAS-Bench-1Shot1: Benchmarking and Dissecting One-shot Neural Architecture Search" does not introduce, name, or evaluate a Hold-One-Shot-Out protocol. Its one-shot supernet is trained on the full one-shot search space; architectures are not held out from supernet training; and the only hold-outs described are standard train/validation data splits for updating weights and architectural parameters, plus use of NAS-Bench-101 validation data for BOHB objective computation during hyperparameter tuning (Zela et al., 2020).

Taken together, these sources support a precise but non-unified view of HOSO. In one strand, HOSO is a leave-one-out conformal calibration device for aggregating one-shot predictors with exact finite-sample marginal coverage under exchangeability and self-score optimality. In another, HOSO is a one-shot-per-class micro-validation mechanism for learning a CLIP-adapter blending ratio under a strict validation-free few-shot protocol. A plausible implication is that the label “HOSO” now denotes a broader methodological pattern: using minimal, explicitly excluded one-shot supervision to estimate a secondary quantity—such as a calibration threshold or blending ratio—without sacrificing the structural constraints of the underlying low-data regime.