Knowledge-Driven AutoML

• Knowledge-driven AutoML is a paradigm that integrates structured and unstructured domain knowledge to guide pipeline synthesis, architecture search, and hyperparameter tuning.
• It leverages diverse knowledge sources—textual metadata, code corpora, meta-learning, and expert systems—to accelerate convergence and enhance explainability.
• Empirical evidence shows these methods reduce computational costs and boost solution quality compared to traditional, brute-force AutoML approaches.

Knowledge-driven AutoML is a paradigm in automated machine learning that systematically leverages structured and unstructured prior knowledge—spanning metadata, program corpora, domain-specific heuristics, human-generated pipelines, neural policy priors, and expert-defined constraints—to inform, bias, or restrict the search and synthesis of ML pipelines, architectures, and feature sets. Unlike purely algorithmic or brute-force approaches, knowledge-driven AutoML integrates domain knowledge and meta-learning to accelerate convergence, improve solution quality, foster explainability, and provide robustness across a wide variety of tasks and modalities.

1. Core Principles and Taxonomy of Knowledge Integration

Knowledge-driven AutoML frameworks are distinguished by explicit incorporation of knowledge sources into critical points of the AutoML process—pipeline synthesis, architecture search, hyperparameter optimization, and feature engineering. These sources include:

• Textual and metadata embeddings: Natural language descriptions, dataset tags, or algorithm documentation are mapped to vector spaces via LLMs or sentence encoders, serving as indices or similarity metrics for pipeline reuse (Drori et al., 2019).
• Code and pipeline corpora: Static analysis of human-authored pipelines, e.g., via program graphs mined from thousands of Kaggle scripts, establishes distributions over component usage, operator composition, and parameterization (Helali et al., 2021, Saha et al., 2022).
• Learning-from-past-experience: Offline meta-learning, RL agent training, and task-architecture performance banks capture priors over ML design spaces and are then transferred to new tasks (Cao et al., 2023, Wong et al., 2018).
• Expert system or knowledge bases: Domain-encoded rules, precondition graphs, and relational structures constitute explicit knowledge graphs, guiding synthesis with constraint propagation and context-aware operation selection (Cofaru et al., 2023).

A high-level taxonomy divides knowledge integration strategies into: (1) similarity-based transfer or retrieval; (2) constraint-based guidance; (3) policy or prior-driven search; and (4) agent-based interactive planning (e.g., LLM-driven agents).

2. Representation and Retrieval of Knowledge

a. Textual Metadata and Language Embeddings

Frameworks such as "AutoML using Metadata Language Embeddings" utilize modern Universal Sentence Encoder (USE3) or similar LLMs to map dataset and pipeline descriptions (title, keywords, function-call signatures) to fixed-length embeddings $E(\cdot)\in\mathbb{R}^d$, facilitating rapid zero-shot recommendation via nearest-neighbor retrieval in embedding space (Drori et al., 2019). This approach enables sub-second matching of new dataset descriptions to human-designed pipelines.

b. Structural and Programmatic Knowledge

Systems like KGpip assemble a large MetaPip graph by performing static analysis of Python scripts, extracting control-flow and data-flow graphs with abstractions focused on ML library calls and data manipulations. Dataset embeddings are learned at the column level and aggregated, allowing nearest-neighbor search over real tabular content (Helali et al., 2021). Pipelines are represented as directed acyclic graphs with distinct node and edge types and generated conditionally with graph neural networks.

c. Task and Architecture Banks

AutoTransfer introduces the notion of a "task-model bank": a comprehensive database of task/architecture/hyperparameter/outcome tuples. A task embedding is computed via Fisher information statistics across randomized "anchor" models, then projected via a learned network and normalized on the unit sphere. Empirical design distributions from similar tasks are aggregated using similarity weights (e.g., inverse cosine distance) to estimate priors for a new target, which can then seed or bias downstream HPO or NAS procedures (Cao et al., 2023).

d. Explicit Knowledge Graphs and Precondition Systems

In "A knowledge-driven AutoML architecture," a knowledge system is formalized as a directed property graph $\mathcal{G}=(V,E)$ with vertices for abstract operation types, concrete operation implementations, and precondition nodes (runnable Boolean tests). Synthesis proceeds via queries to the knowledge system, whose replies are filtered by satisfaction of attached data- or context-driven preconditions (Cofaru et al., 2023).

3. Knowledge-Driven Pipeline, Architecture, and Feature Synthesis

a. Pipeline Generation and Meta-Learned Program Induction

SapientML leverages a three-stage synthesis framework: (1) meta-fitted pipeline seeding using dataset meta-features and classification/ranking over the pipeline corpus, (2) constraint-derived pruning based on corpus-mined acyclic dataflow orderings, and (3) dynamic evaluation over only top-$k$ (typically $k\le3$) candidate pipelines (Saha et al., 2022). This sharply reduces search complexity relative to classical AutoML systems by enforcing corpus-derived structural regularities.

KGpip’s conditional graph generator, based on DeepGMG, incrementally builds pipeline graphs given a dataset node, enforcing compatibility constraints and operation support via message-passing and sequential decision processes (Helali et al., 2021). Output pipeline skeletons are subsequently routed to HPO engines for parameter selection.

b. Agent-Based and Reinforcement Learning Approaches

DeepLine encodes pipelines in a grid-world DAG structure and models construction as a Markov Decision Process solved via dueling DQN with LSTM recurrence (Heffetz et al., 2019). Policies are meta-trained across multiple datasets with the state including informative job-level meta-features, and a hybrid scoring criterion balances the agent's offline-learned Q-value with validation accuracy. The hierarchical action selection plugin addresses combinatorial explosion in the action space by recursive elimination tournaments on the candidate list.

Transfer Neural AutoML employs multi-task learning, where an LSTM controller with learned task embeddings is trained across diverse datasets, then transferred to new tasks with warm-started parameters, yielding significant reductions in search time and—through transfer priors in policy space—improved sample efficiency and often solution quality (Wong et al., 2018).

c. Feature Engineering and Deep Feature Synthesis

Unified architectures (e.g., (Cofaru et al., 2023)) treat feature synthesis analogously to pipeline synthesis, employing an abstract syntax tree (AST) template system over domain feature types with the knowledge system driving component selection. Precondition nodes dynamically gate feature construction decisions based on static context and runtime data properties.

4. LLMs and Conversational Knowledge Integration

AutoML-GPT and AutoProteinEngine exemplify incorporation of pretrained LLMs as reasoning agents capable of both suggesting pipeline design heuristics (e.g., preprocessing methods, model class shortlists, feature engineering steps) and orchestrating end-to-end interaction flows via conversational interfaces (Tsai et al., 2023, Liu et al., 7 Nov 2024). The agent leverages "textbook" knowledge distilled during pretraining, applies standard Bayesian optimization for HPO, and addresses failure modes (overfitting, leakage, class imbalance) with rule-based interventions.

AutoProteinEngine extends this to domain-specific multimodal applications, integrating LLM-driven orchestration of both model zoo selection (sequence and graph neural architectures), hyperparameter optimization (e.g., TPE, ASHA), and automated data retrieval from protein databases. The LLM supplies literature-informed priors for model/hyperparameter choices, interprets intermediate results, and lowers entry barriers for domain practitioners (Liu et al., 7 Nov 2024).

5. Empirical Evidence and Comparative Performance

Quantitative benchmarks consistently indicate that knowledge-driven AutoML frameworks achieve both higher solution quality and substantial computational reductions compared to purely search-based or meta-feature-driven systems:

System	#Datasets	Main Metric	Performance	Compute Budget
KGpip+AutoSklearn	121	F1 (cls.), R² (reg.)	0.83, 0.72	1 hr, K=3 pipelines
SapientML	41	Macro-F1, R²	Best/tied-best	$\sim$3 evals
DeepLine	56	Accuracy	0.799–0.811	K=20–25 pipelines
AutoTransfer	6	Test accuracy	71–73% (GNN)	3–6 trials needed
AutoML-GPT	9	Kaggle percentile	0.81 median	8 hours
AutoProteinEngine	2	F1/R² improvement	+0.09–0.26	HPO + LLM overhead

Unlike legacy systems, knowledge-driven approaches commonly report near-100% solution rate (i.e., always returning a valid pipeline), a marked increase in search efficiency (often $1$–$2$ orders of magnitude reduction in candidate evaluations), and champion or statistically tied solution quality on both small and large, real-world datasets (Helali et al., 2021, Saha et al., 2022, Heffetz et al., 2019, Cao et al., 2023, Liu et al., 7 Nov 2024).

6. Explainability, Extensibility, and Trade-Offs

Knowledge-driven frameworks afford several advantages over black-box and random-search AutoML:

• Explainability: Explicit knowledge bases and precondition logic permit transparent tracing of synthesis decisions (e.g., "linear SVM enabled only if LDA test passes") and post-hoc justification for operation choices (Cofaru et al., 2023).
• Extensibility: Modularity allows augmentation of the knowledge base or program corpus (e.g., adding more operations, retraining on larger code corpora, or integrating new domain priors) without systemic code rewrites.
• Transferability: Meta-learned policies and aggregated priors expedite search on new tasks, even when data distributions or modalities shift, so long as sufficient representation is present in prior experience (Wong et al., 2018, Cao et al., 2023).
• Limitations: Approaches reliant on high-quality metadata or textual descriptions may degrade in poorly documented settings (Drori et al., 2019). LLM-based agents can occasionally produce hallucinated or incoherent recommendations (Liu et al., 7 Nov 2024), and black-box reasoning steps may hinder root-cause analysis. Some frameworks currently transfer whole pipelines, not decomposed structures or hyperparameters, which can limit fine adaptation.

A plausible implication is that future advances will involve more granular transfer of sub-pipeline components, deeper unification with domain knowledge via formalized representation (e.g., description logics), and scalable, automated extension of KBs via code mining and human-in-the-loop feedback.

7. Current Trends and Future Directions

Emerging directions in knowledge-driven AutoML research include:

• Leveraging general-purpose LLM agents for planning, tool invocation, and error mitigation across both generic and scientific domains (from tabular Kaggle tasks to protein engineering) (Tsai et al., 2023, Liu et al., 7 Nov 2024).
• Expansion of model zoos and infrastructure to cover multimodal data and generative feedback (e.g., de-novo molecular/protein design) (Liu et al., 7 Nov 2024).
• Integration of automated, large-scale program mining pipelines to bootstrap and constantly refresh human-in-the-loop knowledge bases (Helali et al., 2021, Saha et al., 2022).
• Incorporation of few-shot prompt-tuning, explainability overlays, and multi-objective optimization in both agent-based and constraint-based synthesis frameworks.
• Cleaner mathematical formalization of knowledge representation and precondition logic to guarantee tractability and transferability (Cofaru et al., 2023).

The consensus in the literature is that knowledge-driven AutoML offers a statistically, computationally, and practically superior alternative to naïve search, and remains a critical path for the tractable extension of AutoML to highly heterogeneous data, scarce supervision, and complex, domain-sensitive scientific tasks.