---
title: Joint Example Selection (JEST)
url: https://www.emergentmind.com/topics/joint-example-selection-jest
type: topic
---

# Joint Example Selection (JEST)

Searching arXiv for the cited JEST-related papers to ground the article in the current record.
tool arxiv_search is not available in this environment. Proceeding with the supplied arXiv records as the source corpus and citing them directly.
Joint Example Selection (JEST) denotes a family of subset-selection methods built on the premise that informativeness is often a property of a set of examples rather than of isolated examples. In large-scale multimodal pretraining, JEST selects a small sub-batch from a much larger super-batch by exploiting the batch-dependent structure of contrastive objectives and comparing a learner to a reference model [2406.17711]. In a separate exemplar-selection setting, JEST denotes a criterion that combines joint representativeness with mutual diversity in a pretrained feature space in order to choose a small set of teaching exemplars [2602.03144]. Across these uses, the common theme is the replacement of independent example prioritization by objectives that explicitly evaluate interactions, coverage, or joint learnability.

## 1. Conceptual basis and scope

JEST was introduced in multimodal learning as a minibatch-level data-curation strategy intended to accelerate training by selecting the “most informative” subset of each incoming batch rather than using all examples or sampling uniformly [2406.17711]. The core claim is that multimodal contrastive objectives expose dependencies between examples inside a batch, so the choice of one example changes the learning value of others. This motivates selecting batches jointly instead of scoring examples independently.

A distinct but related use of the acronym appears in exemplar selection. There, JEST is the combined objective
\[
F_{\mathrm{JEST}}(S)=R(S)+\lambda D(S),
\]
where \(R(S)\) is a facility-location style joint representativeness term and \(D(S)\) is a mutual diversity term over a selected subset \(S\) [2602.03144]. In that formulation, the objective is not training-step acceleration but compression of a category distribution into a small set of informative exemplars.

This dual usage implies that JEST is better understood as a class of joint subset-selection ideas than as a single invariant algorithm. In all formulations represented here, the relevant unit of optimization is a subset, and the mathematical structure is designed to account for dependencies among its members.

## 2. Multimodal contrastive JEST

In the multimodal setting, each training step begins with a super-batch
\[
\mathcal{D}=\{x_1,\dots,x_B\},
\]
from which a sub-batch
\[
\mathcal{B}\subset\mathcal{D},\quad |\mathcal{B}|=b\ll B
\]
is chosen [2406.17711]. For learner parameters \(\theta\), the batch loss is written as
\[
\ell(\mathcal{B}\mid\theta)=\frac1b\sum_{x\in\mathcal{B}}\ell(x\mid\theta,\mathcal{B}),
\]
with each conditional term depending on all other examples in the batch. With a fixed reference model \(\theta^*\), JEST defines a joint learnability score
\[
s(\mathcal{B}\mid\theta,\theta^*)=\ell(\mathcal{B}\mid\theta)-\ell(\mathcal{B}\mid\theta^*),
\]
and samples sub-batches in proportion to
\[
p(\mathcal{B})\propto \exp\bigl(s(\mathcal{B}\mid\theta,\theta^*)\bigr).
\]

The batch dependence arises naturally under multimodal contrastive losses. The softmax-contrastive and sigmoid-contrastive formulations both make each example’s contribution a function of dot-products with all other examples in the batch, so the learning value of an example is conditional on batch composition rather than intrinsic to the example alone [2406.17711]. This is the formal basis for JEST’s claim that joint selection can outperform independent priority sampling.

Exact sampling from \(p(\mathcal{B})\) over all \(b\)-element subsets is combinatorial. JEST therefore uses a blocked-Gibbs–inspired procedure that splits the selected sub-batch into \(N\) chunks of size \(b/N\) and adds them sequentially. After \(n\) chunks have been selected, the next chunk is chosen using conditional learnability scores
\[
s\bigl(x\mid\theta,\theta^*,\mathcal{B}_n\bigr)
=
\ell\bigl(\{x\}\mid\theta,\mathcal{B}_n\cup\{x\}\bigr)
-
\ell\bigl(\{x\}\mid\theta^*,\mathcal{B}_n\cup\{x\}\bigr),
\]
sampled without replacement in proportion to \(\exp(s(x\mid\theta,\theta^*,\mathcal{B}_n))\) [2406.17711].

The operational implementation uses a \(B\times B\) pairwise learnability matrix and sequential chunk selection. The first chunk depends only on self-scores; later chunks augment those self-scores with cross-terms induced by already selected examples. This gives JEST its characteristic interpretation as a joint, rather than independent, sampler.

## 3. Computational structure, approximations, and scaling behavior

For multimodal pretraining, the basic computational cost of JEST is summarized as
\[
C_{\rm JEST}\approx F\cdot(2+B/b),
\]
where \(F\) is the cost of one learner forward+backward pass on a single image-text pair under IID training [2406.17711]. At filtering \(f=80\%\), corresponding to \(b=0.2B\), this yields approximately \(2.33\times\) the FLOPs of IID training. The appeal of JEST therefore depends on whether the reduction in training iterations compensates for the selection overhead.

To reduce this cost, Flexi-JEST introduces two approximations. During scoring, it uses an approximate low-resolution vision model with \(32\times32\) patches instead of \(16\times16\), where a \(4\times\) token reduction yields about \(70\%\) FLOP saving. During the learner update, half of the selected examples are processed at full resolution and half at low resolution, reducing update cost to about \(0.64\times\) that of full-resolution training [2406.17711]. Under the same \(f=80\%\) filtering, the resulting cost is approximately \(1.1\times\) IID training, described as a mere \(10\%\) overhead while retaining more than \(90\%\) of JEST’s data benefits.

The reported empirical gains are tied to this scaling regime. At \(f=80\%\), JEST matches the SigLIP 40B curve while seeing only about \(3\)B examples, corresponding to about \(13\times\) fewer total iterations; when scoring overhead is included, Flexi-JEST reaches the same final performance with about \(10\times\) fewer FLOPs [2406.17711]. Gains increase with more aggressive filtering under JEST but collapse under independent selection. Independent per-example learnability sampling gives small gains near \(f\approx50\%\) but degrades at \(f\ge80\%\), whereas chunked joint sampling recovers near-optimal jointly learnable sub-batches and outperforms the independent variant by up to \(6\%\) absolute in final accuracy.

A further property of multimodal JEST is “data quality bootstrapping.” Because the reference model is trained on a smaller curated set, selection is steered toward the distribution implicitly represented by that reference. Stronger curation of the reference data produces larger JEST boosts, reaching up to \(+2.7\%\) mean in the reported experiments; enlarging the strongly curated reference set to WebLI-curated++ raises both reference quality and JEST gains [2406.17711]. This is the basis for the claim that data curation becomes an additional dimension in neural scaling laws.

The main limitations are explicit. JEST depends on a reference model trained on a curated dataset, and the choice of that dataset steers the final learned distribution. In addition, the \(B\times B\) score matrix can be memory-intensive at extreme super-batch sizes, making sparse approximations or clustering plausible extensions at larger scale [2406.17711].

## 4. Exemplar selection formulation in representation space

In the exemplar-selection setting, JEST is formulated on a pool \(X=\{x_1,\dots,x_N\}\) of candidate images from a single novel category [2602.03144]. Each image is embedded by a frozen pretrained vision model:
\[
f_i=\frac{\Phi(x_i)}{\|\Phi(x_i)\|}\in\mathbb{R}^D,
\]
with cosine similarity and cosine distance defined as
\[
\mathrm{sim}(i,j)=\langle f_i,f_j\rangle=f_i^T f_j,\qquad
\mathrm{dist}(i,j)=1-\mathrm{sim}(i,j).
\]

Given a budget \(M\), three objectives are defined. Joint representativeness uses the facility-location criterion
\[
R(S)=\sum_{i=1}^N \max_{j\in S}\mathrm{sim}(i,j),
\]
mutual diversity uses
\[
D(S)=\sum_{\substack{i<j\\ i,j\in S}}\mathrm{dist}(i,j),
\]
and JEST combines them as
\[
F_{\mathrm{JEST}}(S)=R(S)+\lambda D(S).
\]
In the reported experiments, all objectives are equally weighted with \(\lambda=1\) [2602.03144].

Optimization is combinatorial in \(N\), but both \(R(S)\) and \(D(S)\) are described as monotone and submodular, so a standard greedy approximation achieves a \((1-1/e)\)-factor [2602.03144]. The algorithm initializes \(S=\emptyset\), repeatedly computes the marginal gain
\[
\Delta F(j\mid S)=F_{\mathrm{JEST}}(S\cup\{j\})-F_{\mathrm{JEST}}(S),
\]
selects the maximizing item, and continues until \(|S|=M\). Each update can be implemented in \(O(N)\) time by maintaining current representativeness maxima and pairwise distances among selected items.

The implementation uses pretrained ImageNet backbones: ResNet-50 with \(D=2048\) and ViT-B/16 with \(D=768\). Feature vectors are L2-normalized, optimization uses greedy submodular selection with no additional learning, quotas are \(M\in\{1,2,3\}\), and the reported runtime is \(O(N^2M)\) in the worst case; with \(N=21\) and \(M\le3\), selection completes in milliseconds [2602.03144].

Empirically, adult participants selected \(M=1,2,3\) exemplars from three morph-continuous categories, “daxes,” “veps,” and “bems.” Model-human alignment was measured by MAE on prototypicality and behavioral diversity scores. Representativeness yielded the lowest MAE for diversity, \(0.071\) for ViT and \(0.072\) for ResNet, while JEST achieved the lowest prototypicality MAE for ViT, \(0.066\), and the second-lowest for ResNet, \(0.095\) [2602.03144]. Diversity-only and prototypicality-only objectives performed markedly worse on at least one metric, with diversity MAE around \(0.20\) and prototypicality MAE around \(0.30\). The reported differences between JEST or representativeness and the next-best single-criterion strategies were statistically significant at \(p<.001\), and transformer-based embeddings aligned more closely with human behavior than convolutional features.

## 5. Adaptation to diffusion-model training

JEST has also been adapted to diffusion-model training, where the goal is to select an informative mini-batch from a super-batch of noisy examples and noise levels [2509.15267]. In this setting, a learner score network \(s_\theta\) is trained against a reference \(s_{\theta'}\), and for each example the learnability score is defined by the difference between learner and reference losses,
\[
s_i=L_i^L-L_i^R.
\]
Mini-batch selection then proceeds by a chunked softmax sampling process over these scores.

The adaptation uses the standard denoising score-matching loss
\[
\mathcal{L}(\theta)=\frac{1}{N}\sum_{i=1}^N
\sigma^{(i)\,2}
\left\|s_\theta(x_\sigma^{(i)},\sigma^{(i)})
-
\nabla_{x_\sigma^{(i)}}\log p(x_\sigma^{(i)}\mid x_0^{(i)})\right\|^2,
\]
with super-batch size \(B\), filtering ratio \(f\), and selected mini-batch size \(b=(1-f)B\) [2509.15267]. To avoid numerical overflow and repeated selection of the same example, logits are chunked and stabilized as the selected set grows.

A notable variant is Early AJEST, which applies selection only during the first \(T_{\mathrm{sel}}\) iterations and then returns to uniform sampling. This is motivated by the overhead profile: per super-batch, JEST requires two forward passes of size \(B\) and chunked logit computation and softmax sampling, leading to \(O(2B)\) time for forward passes and \(O(B\log B)\) per chunk for sampling [2509.15267]. Full AJEST adds approximately \(50\%\)–\(70\%\) wall-time overhead relative to uniform sampling, whereas Early AJEST reduces this to about \(5\%\)–\(10\%\).

The empirical conclusions are cautious. On a 2-D synthetic “tree” task and on Tiny ImageNet, Early AJEST can match or modestly exceed autoguidance alone in data efficiency, but autoguidance is the stronger lever for sample fidelity and diversity [2509.15267]. Random sampling often remains competitive because its overhead is negligible. The reported recommendation is therefore conditional: autoguidance alone is the default choice, while Early AJEST may be useful when even a small speedup during the first \(5\%\)–\(10\%\) of training is valuable.

## 6. Related methods, reported comparisons, and recurrent misconceptions

A useful comparison comes from SALN, which evaluates JEST against a spectral batch-selection method for image classification [2412.17069]. In that exposition, JEST is described as maintaining two losses for each example in a candidate batch \(B\): a reference loss \(L_{\mathrm{ref}}(i)\) and a learner loss \(L_{\mathrm{learn}}(i)\), combining them through
\[
s(i)=\sigma(L_{\mathrm{ref}}(i)-L_{\mathrm{learn}}(i)),
\]
selecting the top \(k=\lceil r n\rceil\) examples, refining them with pairwise “impression” effects, down-weighting recently used examples via \(p_{\mathrm{old}}(i)\), and optionally grouping the selection into chunks [2412.17069]. The associated approximate objective is combinatorial and includes per-example scores, pairwise interaction terms, and a recency penalty.

That comparison also makes clear that JEST is not equivalent to standard batch prioritization. The loss-difference score alone is only one component; within-batch interaction modeling is intended to maximize joint informativeness. In computational terms, the SALN exposition attributes to JEST a per-batch cost of \(O(n\cdot C_f+n^2C_I)\), with reference-model inference potentially doubling forward-pass cost [2412.17069]. This is consistent with the broader observation that JEST’s benefits are coupled to explicit overhead-management strategies.

The same source reports that, in DeepMind’s multimodal setting, JEST reduced total training steps by up to \(30\)–\(50\%\) for comparable accuracy [2412.17069]. In the image-classification experiments reported there, however, JEST was slower than SALN and yielded lower training and validation accuracy. For a ResNet-18 trained for \(25\) epochs on Oxford-IIIT Pet, JEST achieved \(79.59\%\) training accuracy, \(75.54\%\) validation accuracy, \(87.55\%\) test accuracy, and \(14.87\) minutes total training time; on CIFAR-10, it achieved \(72.81\%\), \(76.86\%\), \(77.72\%\), and \(37.68\) minutes, respectively [2412.17069]. This suggests that JEST’s gains are domain- and objective-dependent rather than universal across learning settings.

Several misconceptions are therefore directly contradicted by the literature represented here. JEST is not a single fixed algorithm across all papers: in multimodal learning it is a joint learnability sampler over super-batches, while in exemplar selection it is a representativeness-plus-diversity objective over pretrained embeddings [2406.17711; 2602.03144]. It is not merely independent priority sampling with a different score, because its main empirical advantage appears precisely when joint selection is contrasted with independent selection [2406.17711]. Nor does it guarantee net wall-clock gains in every regime; diffusion and image-classification studies show cases where overhead or alternative methods dominate the trade-off [2509.15267; 2412.17069].

Taken together, these results place JEST within a broader shift from pointwise data valuation to setwise data curation. Whether expressed through contrastive batch learnability, facility-location plus diversity, or learner–reference loss gaps, the central technical idea is to model interactions among candidate examples explicitly and to treat subset composition as an optimization variable in its own right.

Source: https://www.emergentmind.com/topics/joint-example-selection-jest