AutoSelection: Adaptive Replacement of Heuristics

• AutoSelection is a framework that replaces static selection rules with instance- or search-conditioned decisions, optimizing algorithm or hyperparameter choices via learned predictions.
• It employs diverse formulations such as learned thresholds in 3D tracking, portfolio selection in optimization, and recipe search in data fine-tuning, leading to measurable performance gains.
• The approach leverages constrained optimization and surrogate evaluations to allocate evaluation budgets effectively, enhancing task-specific performance across multiple domains.

AutoSelection designates a family of methods that replace manually chosen rules, thresholds, subsets, algorithms, or hyperparameters with an automatic procedure conditioned on the current instance, dataset, or search history. In the literature, the selected object ranges from 3D detections in tracking-by-detection pipelines and MAPF solvers to supervised fine-tuning recipes, active spaces, test suites, CNN subensembles, and stability-selection thresholds (Weng et al., 2020, Sigurdson et al., 2019, Wu et al., 13 May 2026, Golub et al., 2020, Marques et al., 2022, Güldoğuş et al., 2023, Huang et al., 28 May 2025). A recurring pattern is that selection is treated not as a fixed preprocessing heuristic, but as a learned prediction problem, a constrained optimization problem, or a budgeted black-box search problem.

1. Problem formulations and selection objects

AutoSelection methods differ primarily in what is being selected and how the objective is defined. In multi-agent pathfinding, the selector is a mapping

$\pi:\mathcal{I}\rightarrow \mathcal{P},$

where $\mathcal{I}$ is an instance set and $\mathcal{P}$ is a portfolio of algorithms; total quality is

$Q(\pi)=\sum_{i\in\mathcal{I}} Q(\pi(i),i),$

with completion rate as the primary quality function, ties broken by distance traveled and then goal achievement time (Sigurdson et al., 2019). In continuous black-box optimization, the formulation is likewise a mapping

$m:\mathcal{I}\rightarrow \mathcal{A},$

but solver quality is measured by expected runtime and relative ERT under COCO/BBOB (Kerschke et al., 2017). In combined algorithm selection and hyperparameter optimization, the algorithm identity itself is treated as a top-level hyperparameter in a single hierarchical search space over WEKA classifiers, meta-methods, ensemble methods, and feature-selection pipelines (Thornton et al., 2012).

A distinct formulation appears in supervised fine-tuning data curation. There, the search target is not a ranked list of instances but an executable recipe

$$r = \big((o_\ell,\theta_\ell)\big)_{\ell=1}^{L(r)}, \qquad L(r)\le L_{\max},$$

acting on a fixed raw pool through filtering, mixing, deduplication, and sampling operators. The solver is budgeted by the number of full evaluations, where a full evaluation means recipe execution, fine-tuning, and benchmark evaluation (Wu et al., 13 May 2026). Test-suite selection uses yet another object: from a generated test pool $T$, select a subset $T' \subseteq T$ with $|T'| = n$ that maximizes a monotone coverage measure on a canonical solution (Marques et al., 2022).

These formulations show that AutoSelection is not tied to a single modality. This suggests a unifying description: the method replaces a fixed global choice with an instance-conditioned or search-conditioned decision rule over a constrained candidate set.

2. Learned selection in perception and vision

In 3D multi-object tracking, AutoSelect addresses the weakness of fixed confidence thresholding in tracking-by-detection pipelines. The paper argues that detector confidence is only an imperfect proxy for whether a detection is a true positive or a false positive, and that the “right” threshold changes with detector, object category, sequence, and even frame. The frame-level variant uses multi-modal features from tracked objects and current detections, max pooling to handle variable numbers of objects, and a Threshold Regression Network implemented as an MLP $128 \Rightarrow 32 \Rightarrow 8 \Rightarrow 1$. The network predicts a scalar threshold $\mathcal{I}$0, and detections whose raw score exceeds that threshold are kept before association. The target threshold is derived from matched detections and ground-truth boxes, using the minimum true-positive score minus a buffer term $\mathcal{I}$1, with an upper bound $\mathcal{I}$2; training uses an $\mathcal{I}$3 loss between $\mathcal{I}$4 and the ground-truth threshold. The instance-level variant replaces a single frame threshold with per-detection probabilities $\mathcal{I}$5 of being true positives, trained with binary cross-entropy and applied at test time with $\mathcal{I}$6 (Weng et al., 2020).

The reported effect is primarily false-positive suppression without recall loss. On KITTI, AutoSelect filters out $\mathcal{I}$7 false positives while maintaining recall. At IoU $\mathcal{I}$8, frame-level AutoSelect improves over GNN3DMOT from sAMOTA $\mathcal{I}$9 to $\mathcal{P}$0 and from AMOTA $\mathcal{P}$1 to $\mathcal{P}$2, with FP reduced from $\mathcal{P}$3 to $\mathcal{P}$4. The instance-level version reaches sAMOTA $\mathcal{P}$5 and AMOTA $\mathcal{P}$6, with FP reduced to $\mathcal{P}$7. On nuScenes, frame-level AutoSelect improves MOTA from $\mathcal{P}$8 to $\mathcal{P}$9 over AB3DMOT, and the instance-level version raises it to $Q(\pi)=\sum_{i\in\mathcal{I}} Q(\pi(i),i),$0 (Weng et al., 2020).

A related vision example is automatic selection of local feature detectors. There the pipeline extracts global features for light reduction, blur, and JPEG compression, uses a linear-kernel SVM for transformation type detection, activates a type-specific SVM for transformation amount detection, and then applies a rule

$Q(\pi)=\sum_{i\in\mathcal{I}} Q(\pi(i),i),$1

to choose the detector expected to achieve the highest repeatability. Transformation-type accuracy is reported as above $Q(\pi)=\sum_{i\in\mathcal{I}} Q(\pi(i),i),$2, and the full pipeline from image loading to detector selection takes about $Q(\pi)=\sum_{i\in\mathcal{I}} Q(\pi(i),i),$3 ms on the reported hardware (Ferrarini et al., 2016).

In both cases, the selected object is not the final task output but an upstream control variable—threshold or detector—that materially changes downstream performance.

3. Portfolio, solver, and model selection

The portfolio-selection strand of AutoSelection assumes that no single algorithm dominates across all instances. In MAPF, the portfolio contains WHCA*, FAR, and BMAA*, and each instance is converted into a $Q(\pi)=\sum_{i\in\mathcal{I}} Q(\pi(i),i),$4 image in which blocked cells are white, free cells black, starts green, and goals red; the start-to-goal assignment is anonymized. A pretrained AlexNet with its final layer changed to three classes is trained on 70% of the data and tested on 30%, with results averaged over 10 random splits. The learned selector attains $Q(\pi)=\sum_{i\in\mathcal{I}} Q(\pi(i),i),$5 completion rate, outperforming the best single algorithm, FAR at $Q(\pi)=\sum_{i\in\mathcal{I}} Q(\pi(i),i),$6, and reaching $Q(\pi)=\sum_{i\in\mathcal{I}} Q(\pi(i),i),$7 of oracle performance; correct best-algorithm identification is about $Q(\pi)=\sum_{i\in\mathcal{I}} Q(\pi(i),i),$8 (Sigurdson et al., 2019).

For continuous black-box optimization, selection is based on Exploratory Landscape Analysis features computed from an improved Latin hypercube design with $Q(\pi)=\sum_{i\in\mathcal{I}} Q(\pi(i),i),$9 evaluations on $m:\mathcal{I}\rightarrow \mathcal{A},$0. Using 102 landscape features, a 12-solver portfolio, and leave-one-function-out cross-validation, the best selector is a classification-based $m:\mathcal{I}\rightarrow \mathcal{A},$1-GA-selected model, called Model 2, with mean relERT $m:\mathcal{I}\rightarrow \mathcal{A},$2 including feature costs, compared with the single best solver HCMA at $m:\mathcal{I}\rightarrow \mathcal{A},$3. The earlier Model 1, a classification-based SVM with sffs feature selection, attains $m:\mathcal{I}\rightarrow \mathcal{A},$4 including feature costs and $m:\mathcal{I}\rightarrow \mathcal{A},$5 excluding them (Kerschke et al., 2017).

Auto-WEKA generalizes this logic by coupling algorithm choice and hyperparameter tuning. Its hierarchical search space contains 39 classifiers in total—27 base classifiers, 10 meta-methods, and 2 ensemble methods—plus 3 feature-selection search methods and 8 feature evaluators, yielding a space on the order of $m:\mathcal{I}\rightarrow \mathcal{A},$6 configurations if crudely discretized. Bayesian optimization back ends SMAC and TPE are used to search this space. Across 21 datasets, Auto-WEKA is better than Random Grid in 20/21 datasets for cross-validation performance, and SMAC outperforms TPE in 19/21 cases for that metric (Thornton et al., 2012).

A further extension appears in automatic neural-network model selection. AMS uses a modified micro-genetic algorithm, list-based genotype encoding, specialized crossover and mutation operators, and a scalarized fitness

$m:\mathcal{I}\rightarrow \mathcal{A},$7

combining normalized performance and model size. On MNIST, the best reported AMS model with $m:\mathcal{I}\rightarrow \mathcal{A},$8 reaches $m:\mathcal{I}\rightarrow \mathcal{A},$9 test accuracy with size $$r = \big((o_\ell,\theta_\ell)\big)_{\ell=1}^{L(r)}, \qquad L(r)\le L_{\max},$$0, while $$r = \big((o_\ell,\theta_\ell)\big)_{\ell=1}^{L(r)}, \qquad L(r)\le L_{\max},$$1 yields a smaller model of size $$r = \big((o_\ell,\theta_\ell)\big)_{\ell=1}^{L(r)}, \qquad L(r)\le L_{\max},$$2 with $$r = \big((o_\ell,\theta_\ell)\big)_{\ell=1}^{L(r)}, \qquad L(r)\le L_{\max},$$3 accuracy (Laredo et al., 2019).

These systems exemplify the classical algorithm-selection premise in modern form: the selector does not invent new solvers, but exploits complementarity among existing ones.

4. Data, sample, and recipe selection

In supervised fine-tuning, AutoSelection reframes subset construction as fixed-pool data recipe search rather than top-$$r = \big((o_\ell,\theta_\ell)\big)_{\ell=1}^{L(r)}, \qquad L(r)\le L_{\max},$$4 instance ranking. The raw pool is fixed, no new samples are generated, rewritten, or augmented, and the search target is an ordered program over grounded operators such as MONA top-$$r = \big((o_\ell,\theta_\ell)\big)_{\ell=1}^{L(r)}, \qquad L(r)\le L_{\max},$$5, IFD top-$$r = \big((o_\ell,\theta_\ell)\big)_{\ell=1}^{L(r)}, \qquad L(r)\le L_{\max},$$6, Varentropy top-$$r = \big((o_\ell,\theta_\ell)\big)_{\ell=1}^{L(r)}, \qquad L(r)\le L_{\max},$$7, N-gram top-$$r = \big((o_\ell,\theta_\ell)\big)_{\ell=1}^{L(r)}, \qquad L(r)\le L_{\max},$$8, AO top-$$r = \big((o_\ell,\theta_\ell)\big)_{\ell=1}^{L(r)}, \qquad L(r)\le L_{\max},$$9, SemDedup, Random top-$T$0, and Mix. The system is described as a two-layer solver. Layer A canonicalizes the raw pool, caches reusable task-, data-, and model-side signals, executes candidate recipes cheaply on the cached pool, and summarizes realized subsets as state vectors. Layer B uses warmup probes, local recipe edits, GP-assisted ranking, and stagnation-triggered reseeding to decide which recipe receives the costly full SFT evaluation. With a budget $T$1, stagnation patience $T$2, and proposer candidate set size 5, the method achieves the best in-distribution reasoning average on all three base models, with gains over the strongest non-AutoSelection baseline of $T$3, $T$4, and $T$5, and gains over full-data training of $T$6, $T$7, and $T$8 (Wu et al., 13 May 2026).

Few-shot learning provides a complementary example in which the selected object is the support set. ACSESS studies 20 sample-selection strategies across meta-learning, few-shot fine-tuning, and in-context learning, then combines strategies automatically by forward selection, backward selection, and a Datamodels-style LASSO procedure over 150 random strategy combinations. Scores are normalized to $T$9 and combined either uniformly or with learned weights. The method reports improvements of up to about $T' \subseteq T$0 percentage points for gradient-based few-shot learning and up to about $T' \subseteq T$1 percentage points for in-context learning, and improvement over the ICL-specific LENS baseline is reported as statistically significant with $T' \subseteq T$2 by Wilcoxon signed-rank test (Pecher et al., 2024).

Active Selection of Classification Features addresses a different acquisition problem. Here cheap selection variables $T' \subseteq T$3 are available, expensive classification features $T' \subseteq T$4 are not, and the aim is to select those instances whose acquisition of $T' \subseteq T$5 will improve the final classifier $T' \subseteq T$6 most. U-ASCF uses bootstrapped regressors $T' \subseteq T$7 and selects instances with large imputation variance; S-ASCF predicts $T' \subseteq T$8, evaluates it through a probabilistic classifier, and assigns a probability-based utility. On three UCI datasets, both methods are most useful early in the acquisition process, and in the neuroimaging case study S-ASCF consistently outperforms random selection while U-ASCF also improves over random, though less consistently (Kok et al., 2021).

Taken together, these methods move beyond the premise that selection is equivalent to ranking isolated items. This suggests that operator order, compositional structure, and proxy-based acquisition can be first-class optimization variables.

5. Constraint-based, convex, and scientific AutoSelection systems

A large group of AutoSelection systems are formulated as exact or convex optimization, or as domain-specific surrogate prediction. TestSelector is built for student programming projects and separates coverage measurement from selection optimization. The instructor supplies a canonical solution, a JSON configuration specifying the coverage measure and target suite size, and a test pool $T' \subseteq T$9; the framework then selects $|T'| = n$0 with $|T'| = n$1 maximizing the chosen monotone coverage measure. Selection uses a specialized Seesaw algorithm with an implicit hitting set approach, while coverage is treated as a black-box monotone function. Built-in measures include block coverage, array coverage, loop coverage, decision coverage, and condition coverage. On a benchmark of 1,637 student projects, 447,538 lines of student code, and 6,402 available input tests overall, all coverage measures outperform random selection in finding bugs; the average failure rates reported are roughly around $|T'| = n$2 for the best coverage-based selected suites under progression search, versus $|T'| = n$3 for random selection in the table shown (Marques et al., 2022).

Ensemble pruning yields a convex-optimization analogue. The SOCP-based autoselection method for CNN ensembles assigns weights to 300 heterogeneous CNNs and optimizes a loss combining accuracy and entropy-based diversity, with sparsity induced through an $|T'| = n$4 relaxation of an $|T'| = n$5 penalty. The resulting second-order cone program keeps networks with nonzero or sufficiently large weights and prunes those below threshold $|T'| = n$6. Reported full versus pruned ensemble accuracies are $|T'| = n$7 versus $|T'| = n$8 on CIFAR-10 with 300 versus 56 models, $|T'| = n$9 versus $128 \Rightarrow 32 \Rightarrow 8 \Rightarrow 1$0 on CIFAR-100 with 300 versus 98 models, and $128 \Rightarrow 32 \Rightarrow 8 \Rightarrow 1$1 versus $128 \Rightarrow 32 \Rightarrow 8 \Rightarrow 1$2 on MNIST with 300 versus 170 models (Güldoğuş et al., 2023).

Scientific computing supplies further domain-specific examples. Automatic active-space selection for strongly correlated systems uses a neural network trained to predict single-orbital entropies $128 \Rightarrow 32 \Rightarrow 8 \Rightarrow 1$3, which are then used to rank orbitals by importance. The training set contains 5,796 artificial transition-metal complexes, each with 36 orbitals, for a total of $128 \Rightarrow 32 \Rightarrow 8 \Rightarrow 1$4 orbital samples. The selected reference architecture is a 5-hidden-layer, 896-neuron-per-layer model using the feature space $128 \Rightarrow 32 \Rightarrow 8 \Rightarrow 1$5, for a total of 29 features, and the paper reports strong transferability to larger systems such as dichromate, Fe(II)-porphyrin, oxo-Mn(salen), and $128 \Rightarrow 32 \Rightarrow 8 \Rightarrow 1$6 (Golub et al., 2020).

Automatic threshold calibration for stability selection illustrates a statistically explicit version of the same theme. ATS orders empirical selection probabilities into a scree plot and identifies an elbow by maximizing a profile log-likelihood; EATS first estimates a null exclusion threshold

$128 \Rightarrow 32 \Rightarrow 8 \Rightarrow 1$7

from a shuffled dataset and then applies ATS only above that cutoff. The method is benchmarked against static thresholds $128 \Rightarrow 32 \Rightarrow 8 \Rightarrow 1$8, and EATS is often better than ATS when $128 \Rightarrow 32 \Rightarrow 8 \Rightarrow 1$9, while ATS is especially competitive when $\mathcal{I}$00 (Huang et al., 28 May 2025).

6. Recurring principles, limitations, and misconceptions

The literature is consistent on several constraints. First, AutoSelection is almost always portfolio- or library-bounded. The MAPF selector can choose only among WHCA*, FAR, and BMAA* (Sigurdson et al., 2019); the continuous black-box selector is restricted to a 12-solver portfolio (Kerschke et al., 2017); Auto-WEKA searches only within the WEKA hierarchy (Thornton et al., 2012); TestSelector can optimize only with respect to the supplied tests and monotone coverage measure (Marques et al., 2022); and fixed-pool data recipe search explicitly forbids generating, rewriting, or augmenting training samples (Wu et al., 13 May 2026). A common misconception is therefore that automatic selection “finds” new methods. In these papers, it does not: it reallocates search effort among existing candidates.

Second, the gains are typically objective-dependent and regime-dependent. ACSESS reports strong modality, dataset, and approach dependence, together with a strong dependence on the number of shots (Pecher et al., 2024). ASCF is most useful early in the acquisition process and is not uniformly better than random across all budgets and datasets (Kok et al., 2021). ATS can be too permissive in large-$\mathcal{I}$01, low-SNR settings, motivating EATS (Huang et al., 28 May 2025). Auto-WEKA improves cross-validation performance more than test performance, indicating overfitting risk in very large hierarchical search spaces (Thornton et al., 2012).

Third, most systems are designed around budgeted evaluation. Fixed-pool data recipe search makes this explicit by allowing at most $\mathcal{I}$02 full evaluations and screening other candidates through cached execution and surrogate ranking (Wu et al., 13 May 2026). The continuous-optimization and AutoDock lines likewise emphasize that feature costs or expensive exact evaluations must be amortized or replaced by surrogates (Kerschke et al., 2017, Rakhshani et al., 2018). This suggests that AutoSelection is often less a replacement for evaluation than a method for deciding where evaluation budget should be spent.

A final recurrent principle is the conversion of a manually tuned control into a prediction target. In 3D MOT, thresholding becomes “a learned prediction problem instead of a manually tuned hyperparameter” (Weng et al., 2020). In stability selection, the stable threshold $\mathcal{I}$03 is estimated from the empirical selection-probability structure rather than fixed a priori (Huang et al., 28 May 2025). In supervised fine-tuning, the question shifts from “which examples should we keep?” to “which data curation recipe over a fixed pool should we run next?” (Wu et al., 13 May 2026). The broader implication is that AutoSelection is best understood not as a single algorithm, but as a research program for replacing static heuristics with adaptive, data-dependent selection rules.