ModelMeta: Label-Free Performance Estimation

• ModelMeta is a meta-learning framework that estimates dataset-level performance for unseen models by leveraging statistical shift descriptors from model outputs and latent embeddings.
• It employs a three-layer MLP and adapts model-specific context via meta-training across a diverse reference pool, ensuring architecture- and modality-agnostic evaluation.
• Empirical results demonstrate significantly lower MAE compared to baselines in Text-to-SQL and image classification, offering fast evaluation with controlled cost.

ModelMeta is a meta-learning formulation for label-free evaluation of unseen machine learning models on unlabeled datasets, instantiated by the framework MetaEvaluator. It addresses the setting in which a new, fixed model $f_{\text{new}}(\cdot \mid \psi_{\text{new}})$ must be assessed on an unlabeled target workload $D_T=\{x_i^T\}_{i=1}^n$ without human annotation, per-model retraining, or modification of $\psi_{\text{new}}$. The central objective is to estimate the true dataset-level metric

$M^*=\frac{1}{n}\sum_{i=1}^n m(\hat y_i, y_i^{T*}),$

where $$\hat y_i=f_{\text{new}}(x_i^T \mid \psi_{\text{new}})$$ and $m(\cdot,\cdot)$ is task-specific, such as exact-match for Text-to-SQL or accuracy for classification. ModelMeta does so by learning a transferable evaluator that maps unlabeled, model-output-derived shift descriptors to performance estimates, using labeled supervision only during meta-training across a pool of reference models and controlled distribution shifts (Pham et al., 22 May 2026).

1. Problem setting and design objectives

ModelMeta is formulated around a pool of reference models $M$ and a collection of unlabeled target datasets. At deployment time, the task is to estimate the dataset-level performance of an unseen model on an unlabeled target dataset, using only black-box access to model outputs and latent representations. The framework assumes that shift descriptors computed from the model’s source data and the unlabeled target data capture distributional and confidence shifts that correlate with performance. Under this assumption, the evaluator predicts performance without access to target labels (Pham et al., 22 May 2026).

The framework is explicitly described as model-agnostic and modality-agnostic. Model-agnostic means that no architectural assumptions are imposed on the target model; only outputs and latent embeddings are required. Modality-agnostic means that the same shift-descriptor machinery is applied across learned embedding spaces and confidence distributions. The paper instantiates this in two modalities: Text-to-SQL, where embeddings may come from SQL token-level or sequence representations and confidence from decoding log-probabilities, and image classification, where embeddings may come from penultimate layers or CLIP-like features and confidence from softmax distributions.

The evaluation target is dataset-level rather than example-level. Estimation quality is measured, once labels become available, using mean absolute error:

$$\mathrm{MAE}=\frac{1}{N}\sum_{i=1}^{N}\left|\widehat{M}_i-M_i^*\right|.$$

The reported tasks are Text-to-SQL exact match (EM), execution accuracy (EX), and image classification accuracy (Acc). This framing shifts evaluation from annotation-heavy benchmarking to performance inference from unlabeled shift statistics.

2. Meta-learning formulation

ModelMeta treats each reference model $m \in M$ as a meta-task with many controlled shifts. For a reference model, the task dataset is

$D^{(m)}=\{(SD_i^{(m)}, a_i^{(m)})\}_{i=1}^{N_m},$

where each $D_T=\{x_i^T\}_{i=1}^n$0 is a shift descriptor and $D_T=\{x_i^T\}_{i=1}^n$1 is the true dataset-level accuracy of model $D_T=\{x_i^T\}_{i=1}^n$2 on a shifted workload. MetaDataset supplies these labeled model–shift pairs and is used to train the evaluator (Pham et al., 22 May 2026).

The evaluator is

$D_T=\{x_i^T\}_{i=1}^n$3

a three-layer MLP with hidden sizes $D_T=\{x_i^T\}_{i=1}^n$4, ReLU, layer norm, and dropout $D_T=\{x_i^T\}_{i=1}^n$5, conditioned on a model-specific context vector $D_T=\{x_i^T\}_{i=1}^n$6. The meta-learning structure is MAML-like but updates the context vector in the inner loop and the global evaluator parameters in the outer loop. The explicit forms given are

$D_T=\{x_i^T\}_{i=1}^n$7

and

$D_T=\{x_i^T\}_{i=1}^n$8

Each meta-training episode samples $D_T=\{x_i^T\}_{i=1}^n$9 and $\psi_{\text{new}}$0 from a task’s pool of shifts, computes shift descriptors, updates $\psi_{\text{new}}$1, and then updates $\psi_{\text{new}}$2. The losses are MSE or RMSE. This design amortizes training across a reference pool, so that new-model evaluation requires only lightweight context adaptation rather than retraining the target model.

3. Shift descriptors and unlabeled signals

The core unlabeled signal in ModelMeta is the composite shift descriptor $\psi_{\text{new}}$3, computed from source and target embeddings and model outputs. For any dataset $\psi_{\text{new}}$4, latent embeddings are collected as $\psi_{\text{new}}$5 through a fixed feature extractor from the model representation. The empirical mean and covariance are denoted by $\psi_{\text{new}}$6 and $\psi_{\text{new}}$7.

The descriptor concatenates three statistical distances. The Fréchet term is

$\psi_{\text{new}}$8

which captures global shifts in embedding statistics. The Mahalanobis term is

$\psi_{\text{new}}$9

which emphasizes rare or low-density changes. The sliced Wasserstein term is

$M^*=\frac{1}{n}\sum_{i=1}^n m(\hat y_i, y_i^{T*}),$0

where the $M^*=\frac{1}{n}\sum_{i=1}^n m(\hat y_i, y_i^{T*}),$1 are random unit directions. The final descriptor is the concatenation

$M^*=\frac{1}{n}\sum_{i=1}^n m(\hat y_i, y_i^{T*}),$2

These terms are intended to summarize train–test mismatch through representation geometry and confidence shifts. The paper states that, empirically, larger shift-descriptor magnitudes indicate greater train–test mismatch and hence lower accuracy, and that the meta-learner maps these patterns to expected performance in a modality- and architecture-invariant way (Pham et al., 22 May 2026).

4. Evaluation procedure for a new model

For an unseen model $M^*=\frac{1}{n}\sum_{i=1}^n m(\hat y_i, y_i^{T*}),$3, ModelMeta initializes a new context vector $M^*=\frac{1}{n}\sum_{i=1}^n m(\hat y_i, y_i^{T*}),$4, for example as zero or an average context, and adapts it using labeled meta-tasks from MetaDataset while keeping the learned evaluator parameters $M^*=\frac{1}{n}\sum_{i=1}^n m(\hat y_i, y_i^{T*}),$5 frozen. The update is

$M^*=\frac{1}{n}\sum_{i=1}^n m(\hat y_i, y_i^{T*}),$6

After $M^*=\frac{1}{n}\sum_{i=1}^n m(\hat y_i, y_i^{T*}),$7 lightweight steps, a target shift descriptor $M^*=\frac{1}{n}\sum_{i=1}^n m(\hat y_i, y_i^{T*}),$8 is computed between the new model’s training data $M^*=\frac{1}{n}\sum_{i=1}^n m(\hat y_i, y_i^{T*}),$9 and the unlabeled target dataset $$\hat y_i=f_{\text{new}}(x_i^T \mid \psi_{\text{new}})$$0, and the final estimate is

$$\hat y_i=f_{\text{new}}(x_i^T \mid \psi_{\text{new}})$$1

Aggregation is dataset-level: embeddings and confidence outputs are collected over all points in $$\hat y_i=f_{\text{new}}(x_i^T \mid \psi_{\text{new}})$$2, and normalization is implicit in the statistical distances. The target model itself is never retrained. This is the operational distinction from pseudo-labeling or fine-tuning baselines, which must be repeated per model.

The framework also includes an optional uncertainty calibration step. A global residual variance

$$\hat y_i=f_{\text{new}}(x_i^T \mid \psi_{\text{new}})$$3

is estimated on held-out validation meta-tasks. Under a normal approximation, a $$\hat y_i=f_{\text{new}}(x_i^T \mid \psi_{\text{new}})$$4 prediction interval is

$$\hat y_i=f_{\text{new}}(x_i^T \mid \psi_{\text{new}})$$5

The paper characterizes this as a simple post-hoc calibration supported by stable prediction-to-ground-truth alignment in experiments (Pham et al., 22 May 2026).

5. Empirical performance, cost, and scalability

The experimental coverage spans Text-to-SQL datasets including WikiSQL, Spider, SParC, CoSQL, BIRD, ScienceBenchmark, EHRSQL, SynSQL-2.5M, and a Text-to-SQL MetaDataset of $$\hat y_i=f_{\text{new}}(x_i^T \mid \psi_{\text{new}})$$6 examples, as well as image datasets including MNIST, USPS, SVHN, COCO, PASCAL VOC, ImageNet ILSVRC12, and an image MetaDataset of $$\hat y_i=f_{\text{new}}(x_i^T \mid \psi_{\text{new}})$$7 examples. The meta-set size is reported as approximately $$\hat y_i=f_{\text{new}}(x_i^T \mid \psi_{\text{new}})$$8K sample sets, each of size approximately $$\hat y_i=f_{\text{new}}(x_i^T \mid \psi_{\text{new}})$$9K, chosen to balance coverage and cost (Pham et al., 22 May 2026).

Headline MAE results are reported as mean $m(\cdot,\cdot)$0 confidence interval. In Text-to-SQL, across Meta-Llama-3-70B, Qwen2.5-32B, XiYanSQL-14B, Ministral-3-14B, and gemma-2-2b, MetaEvaluator achieves an average MAE of $m(\cdot,\cdot)$1, whereas the best baseline, NL2SQL-BUGS, has an average of $m(\cdot,\cdot)$2. In image classification, across ResNeXt-50-32x4d, RegNetY-8GF, ConvNeXt-Tiny, ViT-Tiny, and DeiT-Small, MetaEvaluator achieves an average MAE of $m(\cdot,\cdot)$3, while the best baseline, SelfTrainEns, has an average of approximately $m(\cdot,\cdot)$4.

Domain	MetaEvaluator average MAE	Best baseline average MAE
Text-to-SQL	$m(\cdot,\cdot)$5	$m(\cdot,\cdot)$6
Image Classification	$m(\cdot,\cdot)$7	$m(\cdot,\cdot)$8

Ablations attribute part of the improvement to the specific meta-learning design. In the meta-learning comparison, MAML, FO-MAML, Reptile, Meta-SGD, ANIL, and ProtoNet produce MAEs of $m(\cdot,\cdot)$9, $M$0, $M$1, $M$2, $M$3, and $M$4, respectively, whereas MetaEvaluator achieves $M$5 with only $M$6 adaptation steps and approximately $M$7M extra parameters for context.

The cost model is explicit. MetaDataset is built under a budget of $M$8 USD. Text-to-SQL, at approximately $M$9M examples, contributes approximately $$\mathrm{MAE}=\frac{1}{N}\sum_{i=1}^{N}\left|\widehat{M}_i-M_i^*\right|.$$0 USD; images, at approximately $$\mathrm{MAE}=\frac{1}{N}\sum_{i=1}^{N}\left|\widehat{M}_i-M_i^*\right|.$$1M examples, contribute approximately $$\mathrm{MAE}=\frac{1}{N}\sum_{i=1}^{N}\left|\widehat{M}_i-M_i^*\right|.$$2 USD; the total is approximately $$\mathrm{MAE}=\frac{1}{N}\sum_{i=1}^{N}\left|\widehat{M}_i-M_i^*\right|.$$3 USD. Training is amortized once across the reference pool, while inference for a new model requires $$\mathrm{MAE}=\frac{1}{N}\sum_{i=1}^{N}\left|\widehat{M}_i-M_i^*\right|.$$4 context updates plus one forward pass from $$\mathrm{MAE}=\frac{1}{N}\sum_{i=1}^{N}\left|\widehat{M}_i-M_i^*\right|.$$5 to $$\mathrm{MAE}=\frac{1}{N}\sum_{i=1}^{N}\left|\widehat{M}_i-M_i^*\right|.$$6. Empirical evaluation latency is reported as approximately $$\mathrm{MAE}=\frac{1}{N}\sum_{i=1}^{N}\left|\widehat{M}_i-M_i^*\right|.$$7–$$\mathrm{MAE}=\frac{1}{N}\sum_{i=1}^{N}\left|\widehat{M}_i-M_i^*\right|.$$8 minutes per model, with substantially flatter growth than retraining-based baselines (Pham et al., 22 May 2026).

6. Robustness, limitations, and operational use

ModelMeta is reported to generalize across architectures and modalities because it operates on representation-level distances rather than model-specific heuristics. The experiments show stable calibration across datasets and controlled bias on harder shifts, including enterprise-scale Text-to-SQL settings such as BIRD and Spider 2.0. Pool-size ablations show that MAE decreases as the reference model pool expands, and Hessian spectra remain stable, indicating no optimization instability (Pham et al., 22 May 2026).

The framework’s limitations are also explicit. Reference pool composition matters: broader coverage improves generalization, while narrow pools may limit extrapolation to unseen behaviors. Heavy domain shifts whose embedding statistics are poorly captured by Gaussian or sliced Wasserstein distances may reduce accuracy. Text-to-SQL remains challenging under extreme schema evolution or compositional queries. The method also assumes access to latent embeddings or outputs for shift-descriptor computation; if such access is blocked, shift-descriptor quality degrades. More fundamentally, the approach assumes that representation shifts correlate with performance, which adversarial or highly nonstationary targets may violate.

Operationally, ModelMeta is intended for rapid screening of many candidate models against unlabeled workloads, particularly in fast deployment cycles or privacy-constrained settings. It requires building or adopting a MetaDataset with labeled accuracies for a reference pool, implementing shift-descriptor computation on both source and target data, and integrating a lightweight feature extractor. The paper lists active selection of reference shifts, semi-supervised fine-tuning with small labeled subsets, uncertainty-aware decision-making, and extension to audio, graphs, or multimodal tasks as natural continuations of the framework. A plausible implication is that ModelMeta recasts evaluation itself as a transferable capability, rather than a repeated per-model benchmarking procedure.