Budgeted Comparison Curation

• Budgeted Comparison Curation is the strategy of selecting comparisons, examples, or evidence fragments under explicit resource constraints to maximize downstream outcomes.
• It involves various budget types such as training steps, token allocations, or annotation costs and caters to curated objects like image–text pairs, completion pairs, and pairwise judgments.
• Methodological families span online ranking, active acquisition, and market-based selection, leading to practical gains in efficiency, top-k recall, and adaptive optimization.

Budgeted comparison curation denotes, in the cited literature, the problem of deciding which comparisons, examples, evidence fragments, or judgment modalities should be selected, queried, retained, or reweighted under an explicit resource constraint. The resource can be training steps, comparison count, token budget, annotation cost, retained-memory capacity, or evaluation budget; the curated object can be image–text pairs, completion pairs, pairwise judgments, approval bundles, or source-backed evidence capsules. Across these settings, the shared principle is that scarce budget should be spent on the comparisons or items that most improve the downstream objective, rather than on exhaustive coverage or static preselection (Xu et al., 2023, Xu et al., 2 Jul 2026, Han et al., 17 Jun 2026, Jha et al., 2 Oct 2025, Dong et al., 19 Jan 2026).

1. Scope, budget types, and curated objects

The literature instantiates budgeted comparison curation in several distinct but structurally related ways. In vision–language pretraining, the budget is the training budget $b$, instantiated as the number of weight-update steps, and the curated object is a task-relevant subset $\mathcal{D}_C \subseteq \mathcal{D}$ drawn online from a large pool of image–text pairs (Xu et al., 2023). In pairwise LLM judging, the budget is a fixed comparison budget $B$, and the curated object is the set of ordered pairs $(a,b)$ presented to a judge so as to identify the top-$k$ items under latent quality $\theta$ (Xu et al., 2 Jul 2026). In preference-based post-training, the budget is the number $n$ of labeled within-prompt completion pairs, with the design variable being a sampling distribution $D\in\Delta(\mathcal E)$ over admissible edges (Han et al., 17 Jun 2026). In multi-signal subset selection, the budget can be a token budget $B$, a kept fraction, or a fixed count, with examples scored by a market-derived price-per-cost rule $\rho_i=p_i/\ell_i^\gamma$ (Jha et al., 2 Oct 2025). In mixed human-supervision settings, the budget is annotation spend, constrained by $\mathcal{D}_C \subseteq \mathcal{D}$0, and the curator must choose between full labels and pairwise preferences over AI-generated outputs (Dong et al., 19 Jan 2026).

Setting	Curated unit	Budget variable
CiT	image–text pairs	$\mathcal{D}_C \subseteq \mathcal{D}$1, $\mathcal{D}_C \subseteq \mathcal{D}$2, refresh frequency
Active top-$\mathcal{D}_C \subseteq \mathcal{D}$3 judging	ordered comparison pairs	$\mathcal{D}_C \subseteq \mathcal{D}$4
DPO comparison design	within-prompt completion pairs	$\mathcal{D}_C \subseteq \mathcal{D}$5
Market-based subset selection	examples	token budget $\mathcal{D}_C \subseteq \mathcal{D}$6 or kept fraction
PCAL	labels versus preferences	$\mathcal{D}_C \subseteq \mathcal{D}$7

This range of formulations makes clear that “comparison” is not restricted to explicit pairwise preference queries. In some works it refers to dense embedding comparisons between candidate captions and metadata (Xu et al., 2023); in others it refers to pairwise judgments among candidate outputs (Xu et al., 2 Jul 2026, Han et al., 17 Jun 2026); elsewhere it includes comparing supervision modalities under heterogeneous costs (Dong et al., 19 Jan 2026). A plausible implication is that the field is better understood as a family of budgeted selection problems whose common currency is not the data object itself, but the marginal value of allocating limited resources to one acquisition or retention choice rather than another.

2. Formal problem structures

A recurring structure is a finite candidate set or stream, an explicit budget, and a downstream objective defined over the final selected set or trained model. In CiT, the raw source is

$\mathcal{D}_C \subseteq \mathcal{D}$8

and there is “no notion of a fixed dataset or training epochs over $\mathcal{D}_C \subseteq \mathcal{D}$9; instead, we view the data source as an online data stream.” Curation and training alternate through

$B$0

with the outer loop scoring captions against task metadata by

$B$1

Selection is hard-thresholded, with fallback to a minimum curation ratio $B$2 when too few examples exceed threshold $B$3 (Xu et al., 2023).

In top-$B$4 LLM judging, the core object is not the full ranking but the membership of

$B$5

After observing comparison data $B$6, the model estimates posterior inclusion probabilities

$B$7

and the active rule scores candidate pairs by

$B$8

thereby prioritizing comparisons that are both informative and relevant to top-$B$9 boundary uncertainty (Xu et al., 2 Jul 2026).

In DPO-oriented comparison design, the candidate pool is a set of admissible within-prompt edges

$(a,b)$0

and a sampling design is a distribution $(a,b)$1. The objective is not local preference accuracy but downstream KL-regularized RLHF value,

$(a,b)$2

The theory shows that comparison selection affects policy quality through the design-dependent information matrix

$(a,b)$3

and yields the criterion

$(a,b)$4

for budgeted comparison curation under DPO (Han et al., 17 Jun 2026).

In mixed-supervision acquisition, the budget allocation problem is formalized by the monotone missing-data constraint

$(a,b)$5

or, in the covariate-aware version,

$(a,b)$6

Here the curator decides whether each item should receive a full label, a preference, or no annotation, and chooses the policy $(a,b)$7 by minimizing the asymptotic variance of an EIF-based estimator (Dong et al., 19 Jan 2026).

A different but related formalism appears when budgeted curation must satisfy structured quotas. For laminar matroid selection, the objective is

$(a,b)$8

with

$(a,b)$9

This formulation is directly suited to curation tasks with a single global budget and hierarchical upper-bound rules (Doron-Arad et al., 2023).

3. Methodological families

One methodological family treats curation as online retrieval or ranking tightly coupled to model optimization. CiT is exemplary: it reuses the current text encoder to embed captions and metadata, performs cheap caption-to-metadata similarity comparisons, and updates the selection policy as the encoder improves (Xu et al., 2023). The paper explicitly contrasts this with offline filtering, arguing that adaptive coupling is why online curation outperforms both no curation and one-shot offline curation.

A second family formulates budgeted comparison curation as active acquisition. In Bayesian dynamic ranking, the comparison process is a finite-horizon Bayesian decision problem in which the action at each stage is a pair $k$0, or a triplet $k$1 when worker reliability is also unknown. The reward is the stage-wise increase in posterior expected Kendall’s tau, and the practical policy is an approximated knowledge gradient with Dirichlet and Beta moment matching (Chen et al., 2016). In the LLM-judge setting, active acquisition is top-$k$2-aware rather than full-ranking-aware, and the data model explicitly includes judge-specific verbosity and position bias (Xu et al., 2 Jul 2026).

A third family emphasizes principled aggregation of heterogeneous utility signals before budgeted selection. The market-based selector aggregates standardized signals into shares

$k$3

then selects under cost by

$k$4

Its theoretical interpretation is maximum-entropy exponential weighting under a convex LMSR cost, while its operational interpretation is that signals act as “traders” and the resulting prices are then converted into price-per-token or price-per-cost densities (Jha et al., 2 Oct 2025).

A fourth family organizes curation as rule design under explicit social or structural constraints. In approval-based budgeting, a budgeting rule $k$5 chooses a feasible set $k$6 with

$k$7

by combining a satisfaction function $k$8 with either a max rule, greedy rule, or proportional greedy rule. The proportional greedy family chooses an item maximizing

$k$9

which is the clearest budget-normalized comparison rule in that framework (Faliszewski et al., 2018). This line of work is not about ML training or LLM judging, but it provides a formal comparative language for selecting budget-feasible bundles under different normative objectives.

4. Representative applications and empirical patterns

In vision–language pretraining, budgeted comparison curation is used to replace static filtering pipelines with online metadata-conditioned selection. On YFCC15M, an offline curation baseline reaches $\theta$0 ImageNet zero-shot accuracy, no curation reaches $\theta$1, and online CiT reaches $\theta$2. The curation comparisons are cheap because only the text encoder is used; the paper gives the concrete figure that the text encoder “only uses $\theta$3 of the ViT-L image-encoders’ compute.” At larger scale, the efficiency claims are stronger: on LAION400M, CiT with ViT-B/16 + MoCo-v3 reaches $\theta$4 ImageNet zero-shot accuracy in 26 hours versus 981 hours for OpenCLIP, a $\theta$5 speedup; with larger models, CiT ViT-L/16 + AugReg reaches $\theta$6 in 27 hours, while OpenCLIP ViT-L/14 gets $\theta$7 in 6803 hours, described as $\theta$8 faster and $\theta$9 more accurate (Xu et al., 2023).

In pairwise LLM judging, the empirical lesson is that extra budget cannot repair systematic confounding if the latent model is wrong. On biased-but-competent judges, naive aggregation plateaus at the wrong top-$n$0, while the bias-aware model lifts recall from roughly $n$1–$n$2 to $n$3–$n$4. The top-$n$5-aware acquisition rule then improves sample efficiency further: on the Llama benchmark with the bias-aware model, recall reaches $n$6 at $n$7, $n$8 at $n$9, and $D\in\Delta(\mathcal E)$0 at $D\in\Delta(\mathcal E)$1, versus $D\in\Delta(\mathcal E)$2, $D\in\Delta(\mathcal E)$3, and $D\in\Delta(\mathcal E)$4 for round-robin (Xu et al., 2 Jul 2026).

In preference-based post-training, the main empirical pattern is that pair curation improves downstream policy quality under the same labeling budget. Synthetic tabular and linear-contextual experiments show that oracle and plug-in designs outperform uniform and $D\in\Delta(\mathcal E)$5-weighted baselines, and the same tendency persists in GPT-2-large IMDb DPO and Pythia-2.8B Anthropic-HH experiments (Han et al., 17 Jun 2026). The notable conceptual shift is that one first generates a larger completion pool and then labels only informative pairs, rather than generating only a few completions and labeling all induced pairs.

In comparing-based evaluation, UniCBE treats budget allocation itself as the object of optimization. On AlpacaEval, UniCBE saves over $D\in\Delta(\mathcal E)$6 of evaluation budgets while achieving a Pearson correlation with ground truth exceeding $D\in\Delta(\mathcal E)$7. In scenarios where new models are continuously introduced, it can save over $D\in\Delta(\mathcal E)$8 of evaluation costs (Yuan et al., 17 Feb 2025). The paper attributes these gains to simultaneous optimization of sampling-bias suppression, balanced uncertainty descent, and updating uncertainty.

In alignment data-mixture search, MOSAIC shows that budgeted curation can operate over dataset mixtures and slice-level failure profiles rather than explicit pairwise judgments. Under a fixed 1M-token budget and five rounds of independent fine-tuning from the same base model, MOSAIC improves internal XGuard from $D\in\Delta(\mathcal E)$9 to $B$0 while keeping OrBench at $B$1 and IFEval at $B$2, and its final Pareto solution generalizes better than a random static LoRA baseline on independent attack, over-refusal, and capability tests (Dou et al., 19 Mar 2026). Although this is not pair selection in the narrow sense, it is an instance of budgeted curation driven by comparative slice diagnostics.

In long-horizon memory, EMBER shifts curation earlier in the pipeline: before the query is known, the system must decide which evidence should survive under a retained source-evidence token budget. On LongMemEval-RR at the 8192-token retained-evidence comparison point, EMBER-14B reaches $B$3 F1, compared with $B$4 for the strongest non-EMBER budgeted baseline, and improves both Retain-Recall and Read-Recall (Li et al., 4 Jun 2026). This suggests that, in some settings, the relevant “comparison” is not a queried pair but a future opportunity to recover and contrast source-backed evidence that was either preserved or lost during ingestion.

5. Evaluation criteria and budget-aware metrics

A notable feature of the literature is that curation policies are judged by downstream task utility under budget, not merely by local acquisition heuristics. In DPO pair selection, the core theoretical quantity is the RLHF optimality gap $B$5, with upper and lower bounds both scaling with

$B$6

(Han et al., 17 Jun 2026). In active top-$B$7 judging, the main outcome is top-$B$8 recall under a fixed comparison budget, accompanied by overlap checks to prevent invalid matched-length regressions when length distributions do not support comparison (Xu et al., 2 Jul 2026).

Several works use explicitly budget-normalized deployment metrics. The inference-efficiency study on concise VLM pretraining defines FLOPs per response, FLOPs per correct answer, dollar Cost-of-Pass, and tokens per correct answer. Its principal metric is cost per correct answer, expressed as

$B$9

and its matched-length regression reports an average marginal effect of $\rho_i=p_i/\ell_i^\gamma$0 percentage points for curated over uncurated models, with $\rho_i=p_i/\ell_i^\gamma$1 CI $\rho_i=p_i/\ell_i^\gamma$2 (DatologyAI et al., 24 Jun 2026). This is not a pair-selection paper, but it clarifies how budgeted curation can be evaluated when output length, not only model size, is part of the resource envelope.

For evidence-retention systems, the budget-aware metrics are explicitly decomposed: $\rho_i=p_i/\ell_i^\gamma$3 These distinguish failure to preserve evidence from failure to retrieve preserved evidence, which is particularly useful when the curation action occurs before the final query is known (Li et al., 4 Jun 2026).

For comparative evaluation protocols, summary statistics such as Pearson and Spearman correlation to exhaustive evaluation remain important, but UniCBE also defines uniformity diagnostics $\rho_i=p_i/\ell_i^\gamma$4, $\rho_i=p_i/\ell_i^\gamma$5, and $\rho_i=p_i/\ell_i^\gamma$6 via cosine similarity to ideal uniform allocation patterns (Yuan et al., 17 Feb 2025). This reflects a broader theme: budgeted comparison curation is often evaluated not only by end-task accuracy, but also by whether the allocation policy maintains the desired distribution of budget across samples, pairs, models, or topics.

6. Misconceptions, limitations, and open directions

A recurrent misconception is that more comparisons automatically imply better curation. Several papers reject this directly. In bias-prone LLM judging, naive aggregation “plateaus at a wrong top-$\rho_i=p_i/\ell_i^\gamma$7 on biased judges regardless of budget,” so additional pairwise labels simply estimate the wrong target more precisely unless the model corrects for bias covariates such as verbosity or position (Xu et al., 2 Jul 2026). In mixed label/preference acquisition, the opposite misconception also fails: comparison-only supervision is generally insufficient. Corollary 4.3 in the PCAL paper shows that as $\rho_i=p_i/\ell_i^\gamma$8, the loss diverges when $\rho_i=p_i/\ell_i^\gamma$9, so some direct labels are indispensable in general (Dong et al., 19 Jan 2026).

Another limitation is modality dependence. CiT assumes informative text exists and that task metadata are meaningful; if captions are uninformative, absent, or misaligned with the downstream taxonomy, the selector has little signal. The authors explicitly note low keep rates and poor transfer for some tasks such as SST2 or PCAM, and also emphasize that repeated rescoring over gigantic pools is not free even if text-only curation is much cheaper than image forwarding (Xu et al., 2023). This suggests that budget-aware curation can fail either because the proxy is weak or because the proxy itself becomes costly at scale.

Structured optimization methods can also be restrictive. The laminar-matroid FPTAS assumes additive profit, one global knapsack budget, and laminar constraints only. The paper explicitly states that pairwise redundancy penalties, submodular coverage objectives, overlapping non-laminar group constraints, and multiple independent budgets are not handled directly (Doron-Arad et al., 2023). A plausible implication is that many realistic curation systems inherit the tractability of these models only after simplifying diversity, fairness, or interaction effects into modular surrogates.

Finally, several works emphasize that their strongest theory is local or setting-specific. The DPO comparison-design theory is derived specifically for DPO and a KL-regularized RLHF objective; the implementation for large neural LLMs relies on feature-space approximations to $\mathcal{D}_C \subseteq \mathcal{D}$00 and $\mathcal{D}_C \subseteq \mathcal{D}$01, and the focus is offline design rather than adaptive label acquisition (Han et al., 17 Jun 2026). The LLM-judge debiasing framework likewise warns that covariates such as length may be legitimate signal on some benchmarks, so blind correction can hurt when the presumed bias variable is actually correlated with the deployment objective (Xu et al., 2 Jul 2026).

Taken together, these limitations delineate the field’s main open problems. The literature already provides strong templates for adaptive subset selection, top-$\mathcal{D}_C \subseteq \mathcal{D}$02-aware pair acquisition, cost-normalized example pricing, hierarchical quota satisfaction, and slice-aware mixture search. What remains unresolved is how to combine these strengths when budgets are heterogeneous, objectives are non-additive, judges are biased, and the relevant comparisons are revealed only after part of the budget has already been spent.