Layered TPOT: Accelerated AutoML Pipeline Search

• Layered TPOT is a modification of TPOT that employs a stratified, multi-layer framework to efficiently explore machine learning pipelines on large datasets.
• It leverages independent layer-wise genetic operators and evaluations on progressively larger data subsets to maintain computational efficiency and sample-size integrity.
• Empirical results show that Layered TPOT achieves up to 2–4× faster discovery of high-quality pipelines while maintaining comparable accuracy to standard TPOT.

Layered TPOT (LTPOT) is a modification of the Tree-Based Pipeline Optimization Tool (TPOT), a well-known AutoML system employing genetic programming to evolve optimal machine learning pipelines. LTPOT introduces a stratified, multi-layer search procedure designed to accelerate pipeline discovery on large datasets without sacrificing final model quality. By evaluating candidate pipelines initially on small data subsets and advancing only the most promising ones to compete on larger subsets, LTPOT achieves significant efficiency gains while maintaining the core evolutionary framework and Pareto-based multi-objective selection of TPOT (Gijsbers et al., 2018).

1. Layered Evolutionary Framework

LTPOT extends the standard TPOT evolutionary paradigm by partitioning the population into $M$ hierarchical layers. Each layer $l$ maintains its own population $L_l$ of candidate pipelines, which are evaluated on stratified subsamples $D_l \subset D$ of the training data. The evolutionary process within each layer—encompassing variation (crossover, mutation), evaluation, and Pareto-based selection—is performed independently. Periodically, the top-performing pipelines are promoted to higher layers, where they face more challenging evaluations on larger data subsets. The global search is thus structured as a series of interleaved, layer-specific evolutionary runs with staged advancement based on interim fitness (see Table 1).

Table 1: Layered Population Structure and Advancement

Layer Index ($l$)	Data Subset Size ($s_l$)	Population ($L_l$)
1	$N / 2^{M-1}$	Initial + periodic random addition
2	$N / 2^{M-2}$	Receives top $P/2$ from layer 1
...	...	...
$l$0	$l$1	Final selection (best pipeline)

Algorithmically, LTPOT is formalized in "Layered Evolutionary Algorithm (LayeredEA)," which specifies initialization, per-generation operations, intra-layer variation/selection, inter-layer promotion, and periodic population renewal (Gijsbers et al., 2018).

2. Data Subset Hierarchy and Promotion Policy

Layered data subsets are defined such that each subsequent layer uses a data subset twice the size of the previous layer: $l$2 where $l$3 is the stratified sample size for layer $l$4, and $l$5 is the total number of instances. Generalization to geometric schedules $l$6 is supported but not evaluated in the presented experiments.

Promotion of pipelines between layers occurs every $l$7 generations: after performance evaluation and selection within each layer, the top $l$8 individuals from layer $l$9 are advanced to layer $L_l$0. The bottom half of each layer’s population is eliminated, and layer 1 is replenished with $L_l$1 new random pipelines to ensure exploration coverage. Layers are "shut off" toward the end of a run if there is insufficient time remaining for their survivors to reach the top layer (Gijsbers et al., 2018).

3. Multi-objective Fitness and Pareto Selection

Fitness assessment in LTPOT is multi-objective, using the vector: $L_l$2 where $L_l$3 denotes the mean 3-fold CV accuracy of pipeline $L_l$4 on data subset $L_l$5, and $L_l$6 is the number of primitives (pipeline components). Pareto dominance—as operationalized in NSGA-II—is employed: pipelines are ranked that simultaneously minimize $L_l$7 (i.e., maximize accuracy) and pipeline length, with crowding-distance to break ties (Gijsbers et al., 2018). This ensures that pipelines progressing to higher layers are both performant and parsimonious.

4. Layer-wise Genetic Operators and Independent Competition

Variation within a layer leverages both crossover and mutation, with constraints:

• Crossover: Only between parent pipeline trees sharing at least one primitive; the corresponding subtree is exchanged.
• Mutation: Performed by altering a hyperparameter, or by inserting, deleting, or swapping a primitive node in the pipeline’s genetic representation.

Crucially, evolutionary competition, variation, and selection are confined to individuals evaluated on the same subset size within a given layer. Direct comparisons or genetic exchanges between pipelines from different layers are disallowed, maintaining sample-size integrity for performance estimation (Gijsbers et al., 2018).

5. Computational Complexity and Empirical Speedup

No formal convergence proof is presented, but LTPOT's design offers a clear computational advantage. Evaluating a pipeline on $L_l$8 points takes time $L_l$9. A standard TPOT generation (with population $D_l \subset D$0) incurs cost $D_l \subset D$1. LTPOT, by distributing evaluations across $D_l \subset D$2 layers using exponentially increasing $D_l \subset D$3, incurs per-generation cost: $D_l \subset D$4 Due to early layer evaluations being much cheaper, and because higher layers are scheduled less frequently, the realized speedup is substantial. Empirically, pipelines of equivalent quality are discovered in $D_l \subset D$5–$D_l \subset D$6 less wall-clock time compared to vanilla TPOT, especially on large datasets (Gijsbers et al., 2018).

6. Empirical Evaluation and Findings

LTPOT is benchmarked in two variants (LTPOT-2: $D_l \subset D$7; LTPOT-16: $D_l \subset D$8) against standard TPOT on 18 classification tasks with at least $D_l \subset D$9 instances, using an 8-hour wall-clock budget and 9 random seeds per dataset. Key results:

• On smaller subsets, pipeline ranking shows significant positive Spearman correlation with ranking on the full dataset, supporting the multi-layer funnel approach.
• Across most of the search window, LTPOT-2 consistently achieves better Friedman rank than TPOT, indicating faster acquisition of high-quality pipelines. LTPOT-16 is initially competitive but tends to stagnate later due to excessive early filtering.
• Final AUROC scores for the best pipelines do not differ significantly across methods (p > 0.05), confirming parity in eventual solution quality.
• When LTPOT delivers a solution as good as TPOT, it frequently does so hundreds of minutes sooner. When TPOT is superior, the time gap is comparatively minor (Gijsbers et al., 2018).

A plausible implication is that multi-layer population architectures can reduce optimization time without sacrificing the diversity or ultimate effectiveness of evolved pipelines, provided that performance on data subsets adequately predicts performance on the full set.

7. Significance and Applications

LTPOT addresses the principal bottleneck of evolutionary AutoML—high computational cost on large datasets—by exploiting the correspondence between small-sample and full-sample pipeline rankings. The layered approach is particularly effective in scenarios with severe wall-clock constraints, large search spaces, or limited computational resources. The empirical findings indicate its suitability for large-scale, automated model search tasks across domains where minimizing time-to-quality is critical (Gijsbers et al., 2018).