Cost-Effective Model Evaluation with Meta-Learning

Abstract: The rapid growth of machine learning has produced an ever-expanding ecosystem of models, making it increasingly challenging to verify the reliability of newly released models on unseen, unlabeled data. Conventional evaluation pipelines depend on expensive annotation, repeated fine-tuning, or narrow assumptions that fail to transfer across model families. We present MetaEvaluator, a cost-effective, model-agnostic framework for rapid, label-free assessment of unseen models spanning diverse architectures and modalities. MetaEvaluator leverages meta-learning over a pool of reference models to obtain a transferable initialization, enabling accurate evaluation of new models while amortizing cost across the pool and removing the need for per-model retraining. To the best of our knowledge, this is the first model-agnostic framework capable of evaluating new models on entirely unlabeled datasets. Extensive experiments show that MetaEvaluator produces stable and accurate performance estimates at substantially reduced cost compared to conventional approaches, making scalable benchmarking of emerging models on unlabeled data practical.

• The paper introduces MetaEvaluator, a model-agnostic meta-learning framework that adapts a lightweight per-model context vector to estimate unseen model accuracy without labels or model retraining.
• MetaEvaluator achieves approximately 3–4 percentage points of MAE across cross-domain Text2SQL and image-classification transfers, reducing error by roughly threefold versus leading baselines while evaluating a model in 1–2 minutes.
• The study constructs a multimodal MetaDataset for about $1,000 and shows that larger reference-model pools improve accuracy, although generalization beyond the pool and empirical uncertainty calibration remain open challenges.

Problem and motivation

Organizations increasingly face a selection problem in which both the candidate model and the target workload are new, and the target data carry no labels. The paper formalizes this as a "double challenge": estimating the dataset-level performance $M^\star$ of an unseen model $f_{\mathrm{new}}$ on an unlabeled target set $\mathcal{D}_T$, without ground truth ($\xi1$), without modifying the model ($\xi4$), and under strict efficiency constraints ($\xi5$). Existing label-free evaluation methods—AutoEval [deng2021labels], DoC [guillory2021predicting], ATC [garg2022leveraging], AGD [jiang2022assessing], PseudoAutoEval [boyeau2025autoeval], SelfTrainEns [chen2021detecting]—are trained for a specific backbone or require per-model retraining, pseudo-labeling, or auxiliary ensembles. LLM-judge systems and NL2SQL-BUGS [Liu2025nl2sqlbugs] involve substantial human involvement or query-level analysis with low throughput. The authors state that no prior work addresses the double challenge across modalities, and claim MetaEvaluator is "the first model-agnostic framework capable of evaluating new models on entirely unlabeled datasets"—a strong claim that rests on their specific framing of the problem.

The central hypothesis, motivated by observations of structured performance trends across pretrained systems [schurholt2022model, zhang2023model], is that an evaluator can be meta-learned from a pool of reference models so that it transfers to unseen architectures on unlabeled workloads.

Formulation

The paper lifts supervised label-free evaluation to a meta-learning setting. For each reference model $m \in \mathcal{M}$, a task-specific dataset $$\mathcal{D}^{(m)} = \{(\mathrm{SD}_i^{(m)}, a_i^{(m)})\}$$ pairs shift descriptors—summaries of train–test mismatch—with true accuracy. Unlike MAML [finn2017maml], where each dataset constitutes a task, here each model is treated as a task. At test time, given an unseen model and unlabeled $\mathcal{D}_T$, the framework computes shift descriptors between the model's training data and $\mathcal{D}_T$, adapts a small number of parameters, and outputs $f_{\mathrm{new}}$0. Beyond point estimates, the objective includes calibrated prediction intervals satisfying coverage at level $f_{\mathrm{new}}$1, though the experimental sections do not report empirical interval coverage—a gap between stated objectives and evaluated results.

Five challenges are enumerated explicitly: absence of ground truth, cross-model generalization beyond the reference pool, distribution shift, black-box access only, and computational efficiency.

Methodology

MetaDataset

MetaDataset is organized around model–shift pairs rather than fixed benchmark configurations. Two principles govern construction: controllable diversity of shifts and low annotation cost. For Text2SQL (3,373,204 examples), the pipeline builds multi-table databases from TabLib and KaggleDBQA using GPT-5 for schema refinement, generates queries spanning simple projections to nested analytics via SQLForge, PARSQL, and semantics-preserving rewriting across SQLite, PostgreSQL, and Snowflake dialects, and pairs queries with multiple natural-language realizations including injected distractors. For Image Classification (2,487,936 images), label-preserving edits are proposed by a vision–language controller and executed by a latent diffusion model, instantiating five shift families (illumination, material/surface, camera perturbations, background relocation, contextual changes) at three severity levels, with CLIP-based filtering before and after to prevent label drift.

Notably, the entire corpus was generated under a fixed budget of $f_{\mathrm{new}}$2 USD, with itemized costs yielding $f_{\mathrm{new}}$3 and $f_{\mathrm{new}}$4. This cost accounting is a concrete contribution, though it depends on current API pricing assumptions that may not hold over time.

MetaEvaluator learning

MetaEvaluator is a three-layer MLP (hidden dimensions 256/128/64) regressing accuracy from concatenated shift descriptors comprising a Gaussian Fréchet term, a Mahalanobis term, and a sliced Wasserstein term—capturing global embedding statistics, low-density examples, and directional geometric shifts respectively. Training follows a two-loop scheme: an inner loop updates only a per-model context vector $f_{\mathrm{new}}$5 (dimension 512) on a training meta-set while global parameters $f_{\mathrm{new}}$6 are frozen; an outer loop updates $f_{\mathrm{new}}$7 on a validation meta-set with all contexts frozen. At deployment, only the newly initialized context vector of the unseen model is adapted for a few steps; $f_{\mathrm{new}}$8 remains fixed. This design removes any need for per-model retraining and amortizes cost across the reference pool.

Experimental results

Estimation accuracy

Across source–target transfers unseen during meta-learning (e.g., Spider→BIRD, WikiSQL→Spider 2.0 for Text2SQL; MNIST→SVHN, COCO→ImageNet for vision), MetaEvaluator achieves MAE of approximately 3–4 percentage points, versus 9.61 for NL2SQL-BUGS on Text2SQL and 11.30 for SelfTrainEns on Image Classification—the strongest prior baselines. DoC and ATC degrade sharply under shift (MAE above 15). The roughly threefold error reduction over the best baselines is the headline quantitative result, and calibration plots show MetaEvaluator tracking ground truth closely where ATC and DoC systematically overestimate. An implication worth noting: because MAE is measured against true accuracies obtained offline, deployment-time decisions can be made with confidence intervals narrow enough (~±0.8 at 95%) to distinguish models whose true accuracies differ by several points.

Benchmarking capability

MetaEvaluator requires about 1–2 minutes per unseen model, placing it on a strictly better accuracy–latency Pareto frontier than retraining-based baselines. Total latency grows much more slowly than baselines as the number of unseen models increases, since adaptation involves only lightweight forward/backward passes on the context vector. MAE decreases monotonically as the reference pool expands, and Hessian spectra remain stable across pool sizes, suggesting no growing optimization difficulty—an empirical argument that scaling the pool improves generalization without destabilizing training. Practitioners can therefore trade pool size against a target MAE.

Ablations

Against alternative meta-learning algorithms, MetaEvaluator attains MAE 3.26 ± 0.96 with only 3 adaptation steps and 0.12M extra parameters, compared to MAML (11.63, 12 steps), FO-MAML (8.88), Reptile (9.12), Meta-SGD (8.21), ANIL (8.47), and ProtoNet (12.45). The comparison indicates that explicit adaptation mechanisms outperform metric-based approaches for this problem. On meta-set size, the MLP continues improving up to $f_{\mathrm{new}}$9K sample sets while simpler regressors saturate earlier; beyond 30K, cost rises sharply with marginal gains, giving practitioners a practical stopping point.

Limitations and open questions

Several caveats bear directly on the results. First, generalization to unseen models is bounded by the diversity of the reference pool; the paper shows improving MAE with pool size but does not characterize failure when a novel architecture lies far outside the pool's support. Second, the calibrated uncertainty intervals specified in the objectives are never empirically validated in the experiments, leaving open whether the claimed coverage guarantee holds in practice. Third, MetaDataset generation relies on GPT-5 and diffusion models with CLIP-based filtering; residual label drift or template bias in synthetic shifts could inflate apparent transfer performance, and the t-SNE coverage analysis is qualitative rather than quantitative. Fourth, evaluation covers two modalities only; extension to other domains (graph, speech, medical) is asserted as feasible but untested. Finally, the cost figures depend on per-operation API prices at the time of generation, and the paper does not study joint evaluation of multiple candidate models, which it identifies as future work.

Conclusion

This paper formulates label-free evaluation of unseen models on unlabeled data as a meta-learning problem over reference models, contributes a large multimodal corpus of model–shift pairs built under a documented $1{,}000 budget, and demonstrates that a compact MLP evaluator with per-model context adaptation reduces estimation MAE to roughly 3–4 percentage points—at about one-third the error of the strongest prior methods and one to two minutes per model. The results support scalable pre-deployment screening of rapidly arriving models, contingent on reference-pool coverage and synthetic-shift fidelity, both of which remain open questions.