Pre-hoc Model Selection Overview

• Pre-hoc model selection is a suite of techniques that use preliminary signals to decide among candidate models before full estimation.
• It spans methods in fine-tuning prediction, Bayesian priors, AutoML trimming, and information-criterion screening to balance efficiency and predictive performance.
• The approach aims to control predictive risk and computational costs by estimating performance from limited probes and guided resource allocation.

Pre-hoc model selection denotes procedures that decide among candidate models, model classes, model configurations, or search regions before full estimation, exhaustive search, or full fine-tuning is incurred. In current usage, the term covers several closely related practices: estimating final fine-tuning outcomes from static signals and shallow probes; choosing priors so that larger Bayesian models remain predictively well behaved without pre-emptive pruning; trimming AutoML search spaces from dataset descriptors; deciding coefficient types in large spatial regressions before full operational deployment; and ruling out submodels a priori through information-criterion bounds (Luo et al., 16 Jun 2026, Riha et al., 22 Jun 2026, Vanhoefer et al., 13 Dec 2025, Belkhiter et al., 2 Oct 2025). The unifying objective is to control predictive risk, complexity, and compute under explicit information or budget constraints rather than after the full model space has already been explored.

1. Scope and meanings

The term is not restricted to a single methodological tradition. In statistical modeling, it often means choosing or constraining a model family before seeing the full inferential consequences of all candidate models. In machine learning systems, it can mean predicting downstream performance before committing to expensive training. In computational search, it can mean discarding portions of the model lattice before calibration. These uses differ in mechanics, but they share a common asymmetry: a small amount of ex ante information is used to avoid a much larger ex post cost.

Setting	Pre-hoc decision	Primary information
Fine-tuning prediction	whether to continue training, which model/configuration to prioritize, how much budget to invest	static signals $X_s$ and dynamic signals $X_d^{(c)}$
Bayesian nested modeling	whether smaller-model pruning is needed at all	prior predictive implications and prior predictive checks
AutoML for tabular data	which family or specific model to include in a trimmed search	statistical meta-features and textual dataset descriptions
Spatial additive mixed modeling	whether each predictor is constant, SVC, NVC, or S&NVC	pre-computed inner products and REML-based marginal BIC
Subset search with information criteria	which submodels can be discarded without calibration	lower bounds derived from calibrated supermodels

A recurrent distinction is between pre-hoc and post-hoc procedures. Pre-hoc methods commit to priors, stopping rules, screening rules, or class-level decisions before full fitting; post-hoc methods first fit or search extensively and only then compare or interpret the resulting models. In breast DCE-MRI, for example, a pre-hoc pipeline localizes candidate lesions and then classifies them, whereas a post-hoc pipeline first diagnoses the whole volume and then localizes evidence (Maicas et al., 2018).

2. Formalizations of the decision problem

One influential formulation treats pre-hoc selection as stochastic estimation under information constraints. A fine-tuning task instance is written as $T=(M,D,A)$, where $M$ is the pretrained model, $D$ is the downstream dataset, and $A$ is the stochastic optimization algorithm. Executing $T$ yields a scalar outcome $R\in\mathbb{R}$ drawn from $P(R\mid T)$. With information available at compute budget $c$ denoted by $X_d^{(c)}$0, the task-conditional risk is

$X_d^{(c)}$1

and the population risk is

$X_d^{(c)}$2

At the Bayes-optimal reference, risk decomposes as

$X_d^{(c)}$3

where $X_d^{(c)}$4 is an intrinsic limit floor and $X_d^{(c)}$5 is reducible optimization variance. Dynamic probing can reduce risk only if $X_d^{(c)}$6, and in a locally regular regime the reducible term obeys the necessary envelope $X_d^{(c)}$7 as $X_d^{(c)}$8. Under linear cost $X_d^{(c)}$9, the tradeoff optimum satisfies

$T=(M,D,A)$0

which yields a budget-aware stopping principle rather than a fixed probing depth (Luo et al., 16 Jun 2026).

A different formalization appears in nested empirical risk methods. For sequentially nested classes $T=(M,D,A)$1, the successive empirical excess risk is

$T=(M,D,A)$2

The NER estimator uses the test

$T=(M,D,A)$3

and sets $T=(M,D,A)$4. Under Glivenko–Cantelli assumptions, NER chooses the true model order with high probability, while S-NER sorts candidate expansions so that the minimum empirical risk decreases more rapidly along the selected chain (Hajiani et al., 2024).

A third perspective places pre-hoc selection inside multi-objective optimization. There, model fit and model complexity are treated as separate objectives, for example $T=(M,D,A)$5 and $T=(M,D,A)$6, and pre-hoc preferences are encoded by scalarization:

$T=(M,D,A)$7

On this reading, AIC, BIC, MDL, ridge, and lasso are pre-optimization statements about how much complexity one is willing to exchange for fit (Williams et al., 2018). At the class level, Model Class Selection defines

$T=(M,D,A)$8

and then constructs a confidence set of classes that likely contain at least one optimal model via held-out risk differences and hypothesis testing (Cecil et al., 14 Nov 2025).

3. Priors, predictive consistency, and objective Bayes formulations

A major line of work argues that some pre-hoc model selection problems are actually prior-design problems. For nested Bayesian models $T=(M,D,A)$9 with induced signal distribution $M$0, second-order predictive consistency requires

$M$1

while predictive parsimony requires

$M$2

In this framework, pre-hoc pruning of larger nested models to “prevent overfitting” is often unnecessary or even harmful in finite samples when priors are chosen so that prior predictive implications remain stable as complexity grows. For linear models and centered global–local priors, predictive consistency holds iff

$M$3

and the limit exists; with simplex priors such as R2D2 or Dirichlet–Laplace, this reduces to requiring $M$4 independently of $M$5 (Riha et al., 22 Jun 2026).

Objective Bayesian model choice develops a complementary pre-hoc program by deriving priors from invariance, predictive matching, information consistency, and multiplicity control. In the normal linear variable-selection problem, the common parameters use the right-Haar prior $M$6, while model-specific coefficients use a scale mixture of $M$7-priors,

$M$8

with recommended hyperparameters $M$9, $D$0, and $D$1. The corresponding model prior is

$D$2

which is uniform over model size and uniform within size. This construction is designed to satisfy group invariance, exact predictive matching, information consistency, intrinsic prior consistency, and model selection consistency (Bayarri et al., 2012).

Another objective prior at the model-space level assigns

$D$3

so that the induced prior on model size is Binomial$D$4 with $D$5. With the recommended default $D$6, the prior is minimally informative and parsimony-friendly, and posterior model probabilities are computed as $D$7 (Villa et al., 2015).

Robustness to distributional misspecification motivates yet another pre-hoc formulation. Quasi-Bayesian variable selection replaces a fully specified likelihood by a quasi-likelihood determined only by mean and variance functions,

$D$8

and then defines a model quasi-posterior through the quasi-marginal likelihood

$D$9

This yields quasi-Bayes factors, inclusion probabilities, and asymptotic rejection of underspecified models without requiring a fully correct likelihood specification (Hadj-Amar et al., 19 Jan 2026).

4. Search-space reduction, stopping rules, and operational workflows

Operational pre-hoc selection methods differ mainly in what they screen and how they certify the screen. In pre-hoc fine-tuning prediction, the practical workflow is explicit: define the evaluation metric $A$0 and static compatibility signals $A$1; run lightweight probes at depths $A$2; fit

$A$3

compute $A$4; and select candidates with lower predicted risk $A$5 under the chosen budget or tradeoff policy. With a shared budget over multiple candidates, probe steps are allocated so as to equalize marginal gains, using $A$6 (Luo et al., 16 Jun 2026).

In AutoML, pre-hoc model selection trims the search space from dataset descriptors. The metadata used include number of samples, number of features, number of numerical features, number of categorical features, class imbalance, number of missing values, target entropy, number of outliers, average variance, average skewness, and average kurtosis. A meta-model predicts the expected performance of algorithm family $A$7 from descriptors $A$8 through $A$9, yielding the top-1 rule $T$0 or a top-$T$1 family set for trimmed search. The reported workflow then allocates budget only across the selected families in AWS AutoGluon’s TabRepo environment rather than over the full portfolio (Belkhiter et al., 2 Oct 2025).

For information-criterion search, branch-and-bound provides a different kind of pre-hoc elimination. If a calibrated supermodel $T$2 has already been fit, every uncalibrated submodel $T$3 satisfies

$T$4

For AIC this becomes

$T$5

and analogous lower bounds hold for BIC, AICc, and Mallows’s $T$6. If the bound already exceeds the best observed criterion value, the submodel can be pruned without calibration, even though the criterion is non-monotonic over the model lattice (Vanhoefer et al., 13 Dec 2025).

Large spatial additive mixed models use yet another pre-hoc strategy. The target is not a subset of predictors but the coefficient type for each predictor: constant, spatially varying coefficient, non-spatially varying coefficient, or both. The method pre-computes small inner products such as $T$7, $T$8, $T$9, and $R\in\mathbb{R}$0, so that iterative REML fitting and marginal BIC comparisons become independent of sample size $R\in\mathbb{R}$1. The default “simple selection method” and the randomized “MC selection method” are implemented in the R package spmoran (Murakami et al., 2020).

5. Empirical regimes and application domains

Empirical work increasingly treats pre-hoc selection as an object of direct measurement rather than as a purely heuristic preprocessing step. In fine-tuning prediction, experiments on Qwen (0.5B, 2.5B, 7B-Instruct), LLaMA-3-8B-Instruct, and Mistral-7B across SST-2, GSM8K, MMLU, SAMSum, and TruthfulQA define a predictability phase diagram with three regimes: Static-Sufficient, Dynamic-Critical, and Noise-Dominant. For each task and probing depth $R\in\mathbb{R}$2, the study used $R\in\mathbb{R}$3 independent runs with different seeds and found approximately linear log–log uncertainty decay within the fitting range. Regime agreement versus the full-fit reference was 69% at 25 steps, 78% at 50, 85% at 75, 90% at 100, and 91% at 150/200, which supports the practical recommendation that moderate probing depths around 100 steps can recover stable regime-level behavior (Luo et al., 16 Jun 2026).

In out-of-distribution computer vision, pre-hoc choice of pretrained backbone and pretraining data can outweigh downstream algorithmic refinements. A large-scale study evaluated 100 models across four DomainBed datasets and Camelyon17 using more than 120,000 GPU hours. Table 2 reports that, on PACS, ResNet-50 ERM scored 88.1, “Best Alg.” scored 89.1, and the best model choice, ViT-L IN21k, scored 96.7; on OfficeHome the corresponding numbers were 62.7, 64.7, and 87.4; on DomainNet, 58.4, 59.5, and 77.8; and on VLCS, 76.4, 77.5, and 82.2. The same study found that larger models and larger pre-training datasets improved both OOD accuracy and calibration, with ConvNeXt and ViT generally dominating older CNNs at similar parameter counts (Naganuma et al., 2023).

Tabular AutoML gives a different empirical picture. On 175 OpenML datasets from AWS AutoGluon’s TabRepo benchmark, traditional pre-hoc predictors based on metadata reached family-selection accuracy 0.5143 with Random Forest and model-selection accuracy 0.3714 with KNN or Random Forest, while RoBERTa text encodings reached family accuracy 0.6111. LLM agents outperformed random baselines but lagged behind the stronger metadata-driven methods; for example, Llama-3.1-8b in zero-shot no-RAG achieved family accuracy 0.3829 and model accuracy 0.2000 (Belkhiter et al., 2 Oct 2025).

Spatial regression illustrates how pre-hoc structural decisions affect both inference and prediction. In the Tokyo crime application, the selected spatial additive mixed models achieved adjusted $R\in\mathbb{R}$4 values 0.914 for bicycle theft and 0.928 for shoplifting, with computation times 67.3 s and 52.0 s respectively. For prediction of 2019 Q1 crime density, the selected model yielded RMSE 3.53 versus KDE 5.77 for bicycle theft and 1.62 versus KDE 6.93 for shoplifting, indicating that coefficient-type selection can materially affect downstream predictive performance as well as interpretability (Murakami et al., 2020).

6. Limitations, misconceptions, and adjacent debates

A central misconception is that pre-hoc model selection is always synonymous with choosing a simpler model in advance to “prevent overfitting.” One body of work argues the opposite: when predictively consistent priors stabilize the prior predictive implications of larger nested models, pre-hoc pruning can be unnecessary or even harmful in finite-sample predictive settings. On this view, the right pre-hoc intervention is not mandatory simplification but prior calibration in predictive space, for example by controlling $R\in\mathbb{R}$5 or implied Bayesian $R\in\mathbb{R}$6 as complexity increases (Riha et al., 22 Jun 2026).

Another misconception is that a model which wins a pre-specified selection criterion must reflect genuine signal. In ecology, a model-selection permutation test is proposed precisely because the “best” model under AIC, AICc, or cross-validated mean ignorance can still arise by chance when many correlated candidates are ranked. The method permutes outcomes, recomputes the full candidate ranking for each permuted dataset, and evaluates whether the observed best statistic is unusually favorable relative to this null distribution. In the ibex case study, using $R\in\mathbb{R}$7 permutations, none produced a best statistic as favorable as observed for either AIC or cross-validated mean ignorance; in the reindeer study, the full candidate set gave $R\in\mathbb{R}$8 for AICc and $R\in\mathbb{R}$9 for mean ignorance, whereas a restricted candidate set gave non-significant $P(R\mid T)$0 and $P(R\mid T)$1 (Wheatcroft, 2020).

Pre-hoc and post-hoc approaches also trade off supervision, interpretability, and diagnostic objective. In breast DCE-MRI, the pre-hoc detector–classifier pipeline requires strong supervision and achieved AUC 0.85 breast-wise and 0.81 patient-wise, whereas the weakly supervised post-hoc approach achieved AUC 0.90 breast-wise and 0.91 patient-wise. However, the paper also reports that the post-hoc method’s localization performance remained challenging because it was trained only with image-level labels. This suggests that pre-hoc structure can remain preferable when lesion-level localization, rather than only patient-level diagnosis, is the primary objective (Maicas et al., 2018).

Failure modes are framework-specific. Risk decomposition analyses assume a Bayes-optimal reference and a locally regular stochastic optimization regime; the fitted $P(R\mid T)$2 is task- and optimizer-dependent, and observable uncertainty proxies $P(R\mid T)$3 are only surrogates for the theoretical Bayes risk (Luo et al., 16 Jun 2026). Quasi-Bayesian variable selection requires correct mean and variance functions, and finite-sample dispersion estimation can be unstable when $P(R\mid T)$4 (Hadj-Amar et al., 19 Jan 2026). Branch-and-bound guarantees depend on reliable calibration of supermodels; if nonconvex fitting returns suboptimal likelihoods, safe pruning can be compromised (Vanhoefer et al., 13 Dec 2025). More broadly, distribution shift, misspecified priors, unstable cross-validation, or poorly chosen screening statistics can turn a pre-hoc decision rule into an opaque source of bias rather than a cost-saving surrogate for full search.

Pre-hoc model selection therefore does not designate a single algorithmic family. It is better understood as a class of ex ante decision procedures that substitute information design, predictive calibration, safe pruning, or shallow evaluation for exhaustive model fitting. Across current formulations, its central technical question is not simply which model is best, but what can be known early enough, and with what guarantees, to justify not evaluating everything.