---
title: Hold-One-Shot-Out (HOSO) in Few-Shot Learning
url: https://www.emergentmind.com/topics/hold-one-shot-out-hoso
type: topic
---

# Hold-One-Shot-Out (HOSO) in Few-Shot Learning

Searching arXiv for the cited HOSO-related papers to ground the article in current arXiv records.
Hold-One-Shot-Out (HOSO) is a context-dependent hold-out mechanism for one-shot or few-shot learning in which a single labeled example, or the predictor induced by that example, is excluded from part of the training or calibration pipeline to control overfitting, leakage, or invalid reuse of scarce supervision. In the arXiv literature represented here, the term has two distinct technical realizations. In "CAOS: Conformal Aggregation of One-Shot Predictors" [2601.05219], the underlying idea appears as **leave-one-out calibration** for one-shot conformal prediction: each calibration example is scored after removing the one-shot predictor induced by that same example. In "Hold-One-Shot-Out (HOSO) for Validation-Free Few-Shot CLIP Adapters" [2603.04341], HOSO denotes a **validation-free** procedure that holds out exactly one labeled image per class from the few-shot support to learn a CLIP-adapter blending ratio. By contrast, "NAS-Bench-1Shot1: Benchmarking and Dissecting One-shot Neural Architecture Search" [2001.10422] does not define or use HOSO.

## 1. Terminological scope and conceptual core

The shared conceptual core of HOSO is selective exclusion under severe label scarcity. A one-shot element is held out so that the remaining data can be used for model fitting, aggregation, or calibration, while the held-out element provides an unbiased or less biased signal for a secondary objective. The exact object being held out differs by setting.

In CAOS, what is held out is **the one-shot predictor induced by the calibration example itself**. The paper does not use the name “HOSO”; it uses “leave-one-out calibration.” The provided formulation identifies HOSO with that leave-one-out calibration idea specialized to one-shot predictors: for each labeled example, the calibration score is computed after removing that example’s own induced predictor from the aggregation pool [2601.05219].

In the CLIP-adapter setting, HOSO is the explicit name of a protocol in which **exactly one labeled image per class** is removed from the few-shot support and used as a micro-validation cache to optimize the blending ratio $\alpha$, while the adapter is trained on the remaining support examples [2603.04341].

These two usages are analogous in spirit but not identical in mechanics. One addresses conformal uncertainty quantification with finite-sample coverage guarantees; the other addresses hyperparameter learning under a strict validation-free few-shot protocol. This suggests that “HOSO” is best treated not as a single algorithm, but as a family resemblance among one-shot hold-out constructions.

## 2. HOSO as leave-one-out calibration in CAOS

In CAOS, the problem setting is one-shot prediction with foundation models under a small labeled dataset $D = \{(X_i,Y_i)\}_{i=1}^n$. Each labeled example induces its own one-shot predictor through a fixed mechanism $\pi$:
$$
\pi_i(\cdot \mid x) := \pi(\cdot \mid x;\,(X_i,Y_i)).
$$
Each predictor has an associated nonconformity score $s_{\pi_i}(x,y) \in \mathbb{R}$ [2601.05219].

The paper gives two concrete instantiations. For landmark transfer in vision via patch similarity,
$$
s_{\pi_i}(x,y) = 1 - \mathrm{sim}\!\bigl(e_x(y), e_{X_i}(Y_i)\bigr),
$$
with point prediction
$$
\hat y_i = \argmin_{y \in \mathcal{Y}} s_{\pi_i}(x,y).
$$
For one-shot text classification with large language models via in-context learning,
$$
s_{\pi_i}(x,y) = \mathrm{AvgNLL}\!\bigl(y \mid prompt(X_i,Y_i,x)\bigr),
$$
again with
$$
\hat y_i = \argmin_{y \in \mathcal{Y}} s_{\pi_i}(x,y).
$$

The motivation is that split conformal is inefficient in the one-shot regime because it requires disjoint reference and calibration splits and equips each one-shot predictor independently. The baseline for a fixed $i$ uses calibration scores
$$
S = \{ s_{\pi_i}(X_j,Y_j) : (X_j,Y_j) \in \mathcal{D}_{\text{calib}} \},
$$
threshold
$$
\hat q_{\text{split}} = \QuantileOp_{S}\bigl((1-\alpha)(1+1/|\mathcal{D}_{\text{calib}}|)\bigr),
$$
and prediction set
$$
\hat{\mathcal{C}}_{\text{split}}^{(i)}(X_{n+1}) = \{ y \in \mathcal{Y} : s_{\pi_i}(X_{n+1},y) \le \hat q_{\text{split}} \},
$$
with
$$
\mathbb{P}\!\bigl(Y_{n+1} \in \hat{\mathcal{C}}_{\text{split}}^{(i)}(X_{n+1})\bigr) \ge 1-\alpha.
$$

CAOS replaces this with adaptive aggregation across multiple one-shot predictors. For any reference set $\mathcal{D}$ and target $(x,y)$, it defines the score multiset
$$
\mathcal{A}_{\mathcal{D}}(x,y) := \{ s_{\pi_j}(x,y) : (X_j,Y_j)\in\mathcal{D} \},
$$
and the $k$-min sum operator
$$
\Sigma_{\min}^{k}(\mathcal{A}) := \sum_{j=1}^{k} a_{(j)},
$$
where $a_{(1)}\le \cdots \le a_{(|\mathcal{A}|)}$ are the ordered values in $\mathcal{A}$.

The test-time aggregated nonconformity score is
$$
S_{n+1}^{y} := \frac{1}{k}\,\Sigma_{\min}^{k}\!\bigl(\mathcal{A}_{D}(X_{n+1},y)\bigr).
$$
HOSO enters through the leave-one-out calibration statistic. For each $i$, define
$$
D_{-i} = D \setminus \{(X_i,Y_i)\},
$$
and compute
$$
S_i := \frac{1}{k}\,\Sigma_{\min}^{k}\!\bigl(\mathcal{A}_{D_{-i}}(X_i,Y_i)\bigr).
$$
The calibration quantile is
$$
\hat q_{\mathrm{caos}} := \QuantileOp_{\{S_i\}_{i=1}^n}\bigl((1-\alpha)(1+1/n)\bigr),
$$
and the final prediction set is
$$
\hat{\mathcal{C}}(X_{n+1}) := \bigl\{y\in\mathcal{Y}: S_{n+1}^{y}\le \hat q_{\mathrm{caos}}\bigr\}.
$$

Operationally, HOSO here means that when calibrating on $(X_i,Y_i)$, the unique one-shot predictor generated from $(X_i,Y_i)$ is excluded. This prevents a “self-score” advantage, avoids self-score leakage, and allows all $n$ labeled examples to participate in calibration without data splitting [2601.05219].

## 3. Algorithmic structure and theory of CAOS-HOSO

The CAOS construction proceeds in three stages. First, multiple one-shot predictors are built by instantiating $\pi_i(\cdot \mid x) := \pi(\cdot \mid x;(X_i,Y_i))$ and $s_{\pi_i}(x,y)$ for each labeled example. Second, HOSO or leave-one-out calibration scores $S_i$ are computed using $D_{-i}$, followed by the quantile $\hat q_{\mathrm{caos}}$. Third, for each candidate $y \in \mathcal{Y}$ at test time, the aggregated score $S_{n+1}^{y}$ is computed over the full labeled pool $D$, and $\hat{\mathcal{C}}(X_{n+1})$ is returned [2601.05219].

The aggregation rule is the $k$-min-sum operator, averaged by $1/k$. The paper states that this selects the $k$ most supportive one-shot predictors for a given $(x,y)$, avoiding dilution by weak or irrelevant references. In experiments, $k=3$ is used throughout. The operator is described as permutation-invariant and monotone in the reference set.

The finite-sample coverage theorem is stated under exchangeability and a self-score condition. The required assumption is
$$
s_{\pi_i}(X_i,Y_i) \;\le\; s_{\pi_j}(X_i,Y_i) \quad \text{for all }(X_j,Y_j)\in D.
$$
Under exchangeability of the labeled data $D$ and the test example $(X_{n+1},Y_{n+1})$, and under this self-score optimality assumption, the CAOS prediction set satisfies
$$
\Pr\!\bigl(Y_{n+1}\in \hat{\mathcal{C}}(X_{n+1})\bigr) \;\ge\; 1-\alpha.
$$

The proof route is not classical exchangeability of the score sequence, because adaptive aggregation and leave-one-out calibration violate the usual form. Instead, the paper introduces a full CAOS construction with a hypothetical test label $y$:
$$
\mathcal{D}_{n+1}^y = D \cup \{(X_{n+1}, y)\}.
$$
For each calibration index $i$, the full CAOS score is
$$
\tilde S_i^{\,y} = \frac{1}{k}\Bigl(\Sigma_{\min}^{k+1}\bigl(\mathcal{A}_{\mathcal{D}_{n+1}^y}(X_i,Y_i)\bigr) - s_{\pi_i}(X_i,Y_i)\Bigr).
$$
The corresponding threshold is
$$
\hat q_{\text{full}}^{\,y} = \QuantileOp_{\{\tilde S_i^{\,y}\}_{i=1}^n}\bigl((1-\alpha)(1+1/n)\bigr),
$$
and the full conformal set is
$$
\hat{\mathcal{C}}_{\text{full}}(X_{n+1}) = \bigl\{y : \tilde s_{\text{caos}}\bigl((X_{n+1},y);\mathcal{D}_{n+1}^y\bigr) \le \hat q_{\text{full}}^{\,y} \bigr\}.
$$

The key lemmas are: symmetry of the full CAOS score, monotonicity of the CAOS score with respect to enlarging the reference set, and a comparison lemma asserting
$$
S_i \;\le\; \tilde S_i^{\,y}.
$$
The argument then yields $\hat q_{\mathrm{caos}} \ge \hat q_{\text{full}}^{\,y}$ together with test-score equivalence under self-score optimality, giving
$$
\hat{\mathcal{C}}_{\text{full}}(X_{n+1}) \subseteq \hat{\mathcal{C}}(X_{n+1}),
$$
and hence the CAOS coverage guarantee. The significance of this construction is that CAOS attains exact marginal coverage without adding extra slack, despite score-level non-exchangeability [2601.05219].

A plausible implication is that HOSO in this sense is not merely a data-efficiency heuristic; it is part of a formally justified conformal design whose validity depends on monotonicity, permutation symmetry, exchangeability of examples, and self-score optimality.

## 4. HOSO for validation-free few-shot CLIP adapters

In the CLIP-adapter literature, HOSO denotes a distinct procedure aimed at learning the blending ratio $\alpha$ without a labeled validation set. The setting begins with frozen CLIP encoders: an image encoder $f_e(\cdot)$ and text encoder $f_o(\cdot)$, both producing $L2$-normalized embeddings. For class $c \in \{1,\dots,C\}$, the class prototype is
$$
t_c = (1/N) \sum_{n=1}^N f_o(T_{n,c}), \qquad \|t_c\|_2 = 1.
$$
For image $x$, $v = f_e(x)$ with $\|v\|_2 = 1$, and zero-shot probabilities are
$$
p(c \mid x) = \frac{\exp(\langle v, t_c\rangle / T)}{\sum_{j=1}^C \exp(\langle v, t_j\rangle / T)}.
$$

CLIP-Adapter introduces an adapter $g_y(\cdot)$, a bottleneck MLP, with adapted feature
$$
v_{\text{adapt}} = g_y(v).
$$
The blended representation is
$$
\hat v = (1 - \alpha)\cdot v + \alpha \cdot v_{\text{adapt}},
$$
followed by $L2$ normalization and classification using the same text prototypes and temperature $T$:
$$
p(c \mid x; \alpha, y) = \frac{\exp(\langle \mathrm{normalize}(\hat v), t_c\rangle / T)}{\sum_{j=1}^C \exp(\langle \mathrm{normalize}(\hat v), t_j\rangle / T)}.
$$
Here $\alpha \in [0,1]$ is the blending ratio, $y$ are adapter parameters, and the protocol is explicitly **validation-free** [2603.04341].

The HOSO step is to hold out exactly one labeled image per class from the few-shot support $S$ to form a tiny cache $C$ of size $C$ images. Those examples are removed from the adapter’s training set, leaving $S' = S \setminus C$. The held-out shot is selected randomly per class once at the start and kept fixed. The blending ratio is a single global scalar per dataset, not per-class and not per-layer.

To constrain the optimization, HOSO parameterizes $\alpha$ by a logit:
$$
\alpha = \mathrm{sigmoid}(\theta_{\mathrm{logit}}) \cdot 0.8 + 0.1,
$$
so that $\alpha \in [0.1,0.9]$. The stated purpose is to stabilize optimization and avoid degenerate solutions that fully discard either CLIP or the adapter.

Optimization is decoupled. On $S'$, the adapter is trained with
$$
L_{\mathrm{adapt}}(y,\alpha) = - \sum_{(x, y)} \log p(c = y \mid x; \alpha, y),
$$
updating only adapter parameters $y$ while keeping $\alpha$ fixed during that step. On the cache $C$, HOSO updates only $\theta_{\mathrm{logit}}$ via
$$
L_{\mathrm{HOSO}}(\alpha) = - \sum_{(x, y) \in C} \log p(c = y \mid x; \alpha, y),
$$
equivalently
$$
L_{\mathrm{HOSO}}(\alpha) = - \log \mathrm{softmax}_y(\ell_y(x;\alpha))
$$
in single-example notation. The paper uses two separate SGD optimizers, initializes $\alpha$ to $0.5$ via $\theta_{\mathrm{logit}}$, and updates the ratio each epoch using the hold-out cache [2603.04341].

This HOSO formulation differs sharply from CAOS-HOSO. The hold-out object is no longer a one-shot predictor within a conformal pipeline, but a one-shot-per-class micro-validation set for learning a hyperparameter under strict few-shot constraints.

## 5. Empirical results and comparative behavior

The empirical evidence for CAOS-HOSO is reported on one-shot facial landmarking and RAFT text classification. In the facial landmarking setup with 478 landmarks and $K=168$ patches, the reported results are as follows.

| Setting | Coverage / size |
|---|---|
| $\alpha=0.05$, SCOS Avg. | $0.976\pm0.001$ / $36.07\pm0.94$ |
| $\alpha=0.05$, SCOS Best | $0.952\pm0.002$ / $20.49\pm0.59$ |
| $\alpha=0.05$, CAOS | $0.953\pm0.001$ / $15.96\pm0.58$ |
| $\alpha=0.05$, Oracle | $1.000$ / $16.66\pm0.58$ |
| $\alpha=0.10$, SCOS Avg. | $0.930\pm0.001$ / $21.04\pm0.50$ |
| $\alpha=0.10$, SCOS Best | $0.898\pm0.002$ / $12.17\pm0.32$ |
| $\alpha=0.10$, CAOS | $0.905\pm0.002$ / $9.68\pm0.30$ |
| $\alpha=0.10$, Oracle | $1.000$ / $7.98\pm0.27$ |
| $\alpha=0.20$, SCOS Avg. | $0.842\pm0.001$ / $13.41\pm0.35$ |
| $\alpha=0.20$, SCOS Best | $0.797\pm0.003$ / $7.12\pm0.18$ |
| $\alpha=0.20$, CAOS | $0.808\pm0.002$ / $5.70\pm0.16$ |
| $\alpha=0.20$, Oracle | $1.000$ / $4.29\pm0.13$ |

The paper summarizes this as CAOS consistently attaining target coverage while reducing set sizes relative to split conformal baselines and approaching oracle efficiency in many landmarks. For RAFT one-shot text classification with Llama2-7B at $\alpha=0.1$, CAOS produces smaller prediction sets in $8/9$ tasks; in coverage terms, $4/9$ tasks are hit by both CAOS and SCOS, $3/9$ only by CAOS, and $2/9$ are missed by both, with the note that small test-set size leads to noisy coverage estimates [2601.05219].

For HOSO-Adapter, the validation-free few-shot protocol uses $K \in \{2,4,8,16\}$ shots per class in the main experiments and evaluates on 11 standard few-shot classification datasets: ImageNet, Caltech101, OxfordPets, StanfordCars, Flowers102, Food101, FGVCAircraft, SUN397, DTD, EuroSAT, and UCF101. The paper states that HOSO-Adapter outperforms the CLIP-Adapter baseline by more than 4 percentage points on average across 11 standard few-shot datasets, and that in the 8- and 16-shot settings it outperforms CLIP-Adapter even with the optimal blending ratio selected on the test set [2603.04341].

Ablations on RN50, 16-shot, reported as averages, are:

| Variant | Average accuracy |
|---|---|
| Full HOSO-Adapter | 76.43% |
| Without decoupled training | 73.02% |
| Keep cache samples in adapter training set | 73.35% |
| Use 2 shots per class in cache | 76.04% |
| Use 8 shots per class in cache | 73.68% |

The RN50 16-shot snapshot, averaged over 3 runs, gives HOSO-Adapter at 75.25%, CLIP-Adapter (validation-free reimplementation) at 73.35%, CLIP zero-shot at 57.71%, TIP-Adapter (training-free) at 64.61%, and PathCLIP (reimplementation) at 73.35%. The ViT-B/16 16-shot snapshot gives HOSO-Adapter at 80.33%, CLIP-Adapter at 75.82%, and CLIP-Adapter “best $\alpha$” tuned on the test set over $\{0.2,0.4,0.6,0.7,0.8\}$ at 81.07%, which the paper notes is not strictly comparable. Reported per-dataset improvements versus validation-free CLIP-Adapter include DTD $+11.0$, FGVCAircraft $+9.1$, and EuroSAT $+14.8$ points [2603.04341].

The paper also reports across-shot averages. RN50 HOSO-Adapter averages are 63.03 for 2-shot, 65.13 for 4-shot, 71.78 for 8-shot, and 75.25 for 16-shot. ViT-B/16 HOSO-Adapter averages are 69.76, 71.61, 77.43, and 80.33, respectively. It further reports that jointly learning $\alpha$ on the support set without hold-out causes $\alpha$ to increase monotonically during training and overfit, whereas HOSO’s $\alpha$ stays conservative and drops when overfitting is detected, reducing train–test gaps across datasets [2603.04341].

## 6. Practical conditions, limitations, and relation to adjacent literature

The practical guidance for CAOS-HOSO is to use all available labeled examples to instantiate one-shot predictors, compute each calibration score $S_i$ with $D_{-i}$, and choose a small $k$, with $k=3$ used throughout the experiments. The score functions should be comparable across references, and the aggregation should remain symmetric to reference permutations, monotone in the reference set, and compatible with the self-score optimality assumption. The paper identifies possible failure modes: violating self-score optimality can break the set-inclusion step in the proof; retraining-dependent scores or aggregation that depends on model parameters fitted on augmented data may violate the required assumptions. Computationally, per test input and candidate $y$, computing $\mathcal{A}_{D}(X_{n+1},y)$ is $O(n)$; finding the $k$ smallest can be done in $O(n)$ with selection or $O(n \log k)$ with a heap; overall complexity is $O(n|\mathcal{Y}|)$ per test input, with $O(n^2)$ preprocessing for calibration [2601.05219].

The practical guidance for HOSO-Adapter is correspondingly specific: hold out exactly one random sample per class, train the adapter on $S'$, optimize $\theta_{\mathrm{logit}}$ on the cache $C$, use decoupled optimization with two optimizers, and keep $\alpha$ in $[0.1,0.9]$ with initialization near $0.5$. The reported defaults are an adapter bottleneck MLP with reduction factor 4 and ReLU, 200 epochs, batch size 32, SGD with momentum 0.9, weight decay 0.0005, cosine learning-rate schedule, initial learning rate 0.002 for the adapter, and a separate ratio optimizer with learning rate 0.1. The paper warns against jointly optimizing $\alpha$ and the adapter on the same few-shot data, keeping the held-out samples in the adapter training set, or using large cache sizes; it reports that one-shot per class is best on average [2603.04341].

The literature boundary is also explicit. "NAS-Bench-1Shot1: Benchmarking and Dissecting One-shot Neural Architecture Search" does not introduce, name, or evaluate a Hold-One-Shot-Out protocol. Its one-shot supernet is trained on the full one-shot search space; architectures are not held out from supernet training; and the only hold-outs described are standard train/validation data splits for updating weights and architectural parameters, plus use of NAS-Bench-101 validation data for BOHB objective computation during hyperparameter tuning [2001.10422].

Taken together, these sources support a precise but non-unified view of HOSO. In one strand, HOSO is a leave-one-out conformal calibration device for aggregating one-shot predictors with exact finite-sample marginal coverage under exchangeability and self-score optimality. In another, HOSO is a one-shot-per-class micro-validation mechanism for learning a CLIP-adapter blending ratio under a strict validation-free few-shot protocol. A plausible implication is that the label “HOSO” now denotes a broader methodological pattern: using minimal, explicitly excluded one-shot supervision to estimate a secondary quantity—such as a calibration threshold or blending ratio—without sacrificing the structural constraints of the underlying low-data regime.

Source: https://www.emergentmind.com/topics/hold-one-shot-out-hoso