KGmetaSP: Unified Meta-Learning Framework

• The paper introduces KGmetaSP, a meta-learning framework that unifies dataset and pipeline metadata in a knowledge graph for effective performance estimation.
• It employs RDF2Vec with MKGA tokenization on MetaExe-KG to capture dataset attributes and pipeline interactions, facilitating both PPE and DPSE tasks.
• Experimental results show KGmetaSP’s superior performance in classification and similarity retrieval compared to traditional, configuration-specific meta-learning approaches.

Searching arXiv for the cited paper and closely related works mentioned in the source material. KGmetaSP is a meta-learning framework that represents datasets and machine-learning pipelines together in a single knowledge graph and learns knowledge-graph embeddings over that graph to support two tasks: Pipeline Performance Estimation (PPE), which predicts how well a pipeline configuration will perform on a target dataset, and Dataset Performance-based Similarity Estimation (DPSE), which identifies datasets with similar performance patterns (Klironomos et al., 20 Mar 2026). Its central premise is that classical meta-learning based only on dataset meta-features overlooks past experimental results and pipeline metadata, and therefore captures little of the dataset–pipeline interaction structure that underlies performance similarity patterns. KGmetaSP addresses this by constructing a unified knowledge graph, MetaExe-KG, learning embeddings for all nodes, and using those representations for a single pipeline-agnostic meta-model for PPE and for distance-based retrieval in DPSE (Klironomos et al., 20 Mar 2026).

1. Definition and conceptual basis

KGmetaSP is defined by the joint representation of datasets and machine-learning pipelines within one heterogeneous knowledge graph. In the formulation given for the framework, datasets are not represented only by vectors of meta-features such as number of instances, entropy, or landmarkers, and pipelines are not treated as opaque identifiers. Instead, datasets, executable pipeline configurations, methods, hyper-parameters, and historical links between datasets and pipelines are all encoded in a relational structure (Klironomos et al., 20 Mar 2026).

The framework targets two meta-learning problems. PPE predicts the performance value for a pair $(d_i, p_j)$, where $d_i$ is a dataset and $p_j$ is a pipeline configuration. DPSE estimates similarity between datasets in terms of how pipelines perform on them, and retrieves performance-similar datasets. The paper positions KGmetaSP against classical approaches that, for PPE, train one regression model per pipeline configuration from meta-features to performance, and, for DPSE, define similarity as a distance between meta-feature vectors (Klironomos et al., 20 Mar 2026).

The conceptual contribution is the unification of intrinsic dataset properties, pipeline structure, and historical pipeline usage patterns in a single knowledge graph. The source material states that this relational representation supports shared neighborhoods, compositional generalization, and multi-view representations. This suggests that the framework is designed not merely as a feature engineering device, but as a mechanism for exposing latent dataset–pipeline interaction structure through graph context (Klironomos et al., 20 Mar 2026).

2. MetaExe-KG and MetaExe-Bench

The knowledge graph used by KGmetaSP is MetaExe-KG. It is built by integrating two existing semantic resources: MLSea-KG, based on MLSO, and Exe, or Executable KGs. MLSea-KG contributes rich dataset metadata and meta-features, whereas Exe represents machine-learning pipelines as executable graphs containing dataset variables, tasks, methods, hyper-parameters, and data-flow edges (Klironomos et al., 20 Mar 2026).

The authors use the OpenML API to fetch a set of 170 OpenML datasets $\mathcal{D}$ and, for each dataset $d_i$, the top 10 pipeline configurations per pipeline that were trained and evaluated on $d_i$, restricted to scikit-learn pipelines. According to the source material, this filters out low-quality runs, reduces knowledge-graph size, and focuses on informative historical patterns. These pipelines are reconstructed as Exe graphs and aligned with MLSea-KG, yielding MetaExe-KG (Klironomos et al., 20 Mar 2026).

MetaExe-KG includes several entity types. From Exe, it contains DataEntity nodes for dataset variables, Task nodes for workflow steps, Method nodes for concrete implementations such as StandardScaler or RandomForestClassifier, Hyper-parameter nodes for specific hyper-parameters and values, and Pipeline configuration nodes. From MLSea-KG, it contains Dataset nodes, textual dataset metadata such as description, title, keywords, and tags, and numerical meta-feature attributes. The graph also includes relations corresponding to data flow and structural decomposition, as well as cross-links between datasets and the pipeline configurations historically evaluated on them (Klironomos et al., 20 Mar 2026).

The scale reported for MetaExe-KG is 170 datasets, 2,616 pipeline configurations, approximately 1.5M entities, approximately 4.5M triples, 149–155 attributes, and 371k–385k literals. The benchmark derived from this graph is MetaExe-Bench, which is created by executing 2,616 scikit-learn pipelines sparsely on 170 OpenML datasets, yielding 144,177 evaluations (Klironomos et al., 20 Mar 2026).

MetaExe-Bench is constructed in several steps. The procedure identifies the 170 OpenML datasets present in MetaExe-KG, extracts the top 10 scikit-learn pipeline configurations per pipeline from historical OpenML experiments, filters pipeline configurations to ensure end-to-end executability under scikit-learn 1.5.1, executes the 2,616 pipeline configurations sparsely across the 170 datasets, obtains performance metrics including accuracy and precision for classification tasks, measures fit time, and labels experiments invalid if runtime exceeds 10 times the original OpenML runtime. If a pipeline cannot be fit to a dataset, the worst possible metric value is assigned as $v_{i,j}$ (Klironomos et al., 20 Mar 2026).

3. Embedding methodology and representation design

The primary embedding method used in KGmetaSP is RDF2Vec combined with MKGA for handling numeric literals. The paper states that triple-score models such as TransE, DistMult, and ComplEx, as well as GNNs, can struggle with extreme scale and sparsity or can be computationally heavy, whereas walk-based methods work well for real-world sparse knowledge graphs. Because MetaExe-KG is large and sparse, RDF2Vec is selected as the main embedding model (Klironomos et al., 20 Mar 2026).

The RDF2Vec procedure is specified as follows. Random walks are generated on the graph with 10 walks per entity and walk length 20, using both entities and relations, as well as MKGA tokens. These walks are treated as sentences and used to train a Word2Vec model with dimension $100$, window $5$, negative sampling $5$, epochs $d_i$0, and min_count = 0. The objective is the Word2Vec Skip-gram or CBOW objective. For Skip-gram, the formulation given is

$d_i$1

and, with negative sampling, the per-pair objective is

$d_i$2

MKGA transforms numeric literals into binned tokens using strategies such as propConst_LOF_del_numbers, so that numeric values are discretized and become tokens in the random walks. In this way, dataset meta-features and other literals influence the local neighborhoods sampled by RDF2Vec and therefore affect the learned embedding geometry (Klironomos et al., 20 Mar 2026).

The framework defines several aggregate embeddings. Let $d_i$3 denote the RDF2Vec embedding of node $d_i$4. The dataset variable-based embedding is

$d_i$5

where $d_i$6 is the number of DataEntity nodes of $d_i$7. The pipeline embedding for a pipeline configuration $d_i$8 is

$d_i$9

where $p_j$0 is the $p_j$1-th Method node in $p_j$2 and $p_j$3 is the number of Method nodes in $p_j$4. The dataset pipeline-context embedding is

$p_j$5

where $p_j$6 is the number of pipelines associated with $p_j$7. The combined dataset embedding is a simple average of the two dataset views:

$p_j$8

The paper also reports appendix experiments with TransE, DistMult, and ComplEx trained with PyKEEN for link prediction, using dimension $p_j$9, 1500 epochs, batch size $\mathcal{D}$0, learning rate $\mathcal{D}$1, 3 negatives per positive triple, and margin $\mathcal{D}$2. However, the reported main results are based on RDF2Vec because it consistently outperforms these link-prediction-based embeddings in PPE experiments (Klironomos et al., 20 Mar 2026).

4. Meta-learning tasks and model construction

For PPE, the problem is to learn a meta-model $\mathcal{D}$3 such that

$\mathcal{D}$4

where $\mathcal{D}$5 are characteristics of dataset $\mathcal{D}$6 and $\mathcal{D}$7 are characteristics of pipeline configuration $\mathcal{D}$8. In KGmetaSP, the dataset representation for PPE uses classical meta-features only, not dataset embeddings. These meta-features are organized into several sets: MF All, MF Statistical, MF Information Theory, and MF Landmarkers. The pipeline representation is the pipeline embedding $\mathcal{D}$9, with $d_i$0 for RDF2Vec. The input to the meta-model is the concatenation

$d_i$1

The meta-models are pipeline-agnostic single models that handle all $d_i$2 pairs. The implementation uses Random Forests from scikit-learn for both regression and classification: RandomForestRegressor for meta-regression and RandomForestClassifier for meta-classification. Hyper-parameters are tuned by 10-fold cross-validation on the training set over grids of n_estimators, max_depth, and min_samples_split. The ensemble prediction is described as

$d_i$3

where $d_i$4 are decision trees (Klironomos et al., 20 Mar 2026).

The target metrics considered in MetaExe-Bench for classification tasks are accuracy and precision, each used in two target formulations: raw continuous values for meta-regression and quantile-binned three-class targets for meta-classification. The evaluation uses two generalization settings. In the unseen-datasets setting, the train/test split is at dataset level with a 70/30 split, only pipeline configurations with at least 50 training samples are kept, and the evaluation covers 123,920 dataset–pipeline pairs across 1,028 pipeline configurations. The baseline consists of 1,028 configuration-specific Random Forest models using only dataset meta-features. KGmetaSP instead uses one pipeline-agnostic model with meta-features and pipeline embeddings. In the unseen-pipelines setting, the split is at pipeline configuration level with a 70/30 split. The baselines are average performance and closest embedding; KGmetaSP again uses the same pipeline-agnostic model (Klironomos et al., 20 Mar 2026).

For DPSE, each dataset $d_i$5 is associated with a performance profile vector

$d_i$6

defined over the set of pipelines evaluated on it. Ground-truth similarity between two datasets is the cosine similarity of their performance profiles restricted to pipelines evaluated on both:

$d_i$7

KGmetaSP computes embedding-based similarity by cosine similarity over $d_i$8, $d_i$9, or $d_i$0:

$d_i$1

The evaluation ranks all other datasets for each query dataset and compares the ranking to the ground-truth performance-based similarity using Hit@K and NDCG@K, with hit thresholds $d_i$2 of 0.8 or 0.9 (Klironomos et al., 20 Mar 2026).

5. Integration of meta-features and experiment data

A defining characteristic of KGmetaSP is that meta-features enter the framework in two distinct ways. First, they appear as classical meta-feature vectors $d_i$3 and are used directly as raw inputs to the PPE meta-models. In this usage, there is no complex fusion; the meta-feature vector is simply concatenated with the pipeline embedding:

$d_i$4

Second, meta-features appear within the knowledge graph itself as numeric literals attached to dataset nodes in MLSea-KG. Through MKGA discretization, these literals are transformed into tokens and included in random walks, allowing them to influence the RDF2Vec embeddings implicitly (Klironomos et al., 20 Mar 2026).

The source material describes an ablation on the integration of MLSea-KG. In unseen-dataset PPE with MF All and RDF2Vec, adding MLSea-KG changes target accuracy regression from MSE 0.0103 and $d_i$5 0.6114 to MSE 0.0101 and $d_i$6 0.6181, and changes classification accuracy and F1 from 0.7302 and 0.7320 to 0.7413 and 0.7427. For target precision in the same setting, the ablation shows that MSE and $d_i$7 improve slightly with MLSea-KG, but classification accuracy and F1 are slightly better without it. The paper therefore characterizes the effect in unseen datasets as beneficial for meta-regression and slightly beneficial for accuracy prediction, while some precision-classification metrics favor the raw ExeKG variant (Klironomos et al., 20 Mar 2026).

In the unseen-pipelines setting, the ablation is more consistent. With MLSea-KG, target accuracy yields MSE 0.0070, $d_i$8 0.7361, accuracy 0.8250, and F1 0.8244, compared with MSE 0.0074, $d_i$9 0.7246, accuracy 0.8225, and F1 0.8220 without MLSea-KG. For target precision, the corresponding comparison is MSE 0.0131, $v_{i,j}$0 0.6264, accuracy 0.8251, and F1 0.8247 with MLSea-KG versus MSE 0.0135, $v_{i,j}$1 0.6163, accuracy 0.8216, and F1 0.8211 without it. The paper concludes that, for unseen pipelines, MLSea-KG consistently improves both regression and classification (Klironomos et al., 20 Mar 2026).

For DPSE, the best KGmetaSP configuration with MLSea-KG reports Hit@1 0.7905, Hit@2 0.8649, Hit@5 0.9257, NDCG@1 0.8811, NDCG@2 0.8642, and NDCG@5 0.8541. Without MLSea-KG, the corresponding values are Hit@1 0.7432, Hit@2 0.8649, Hit@5 0.9527, NDCG@1 0.8551, NDCG@2 0.8606, and NDCG@5 0.8560. The paper interprets this as a significant improvement in top-1 retrieval and NDCG from MLSea-KG, with slightly better deeper-rank results without it. A plausible implication is that semantic enrichment is particularly useful when the objective is to identify the most similar datasets rather than merely good candidates within a larger result list (Klironomos et al., 20 Mar 2026).

6. Empirical findings, implications, and limitations

In PPE for unseen datasets, KGmetaSP is compared with the average performance of 1,028 configuration-specific models. For target accuracy with MF All, the configuration-specific models achieve MSE 0.0081, $v_{i,j}$2, accuracy 0.7363, and F1 0.7358, while KGmetaSP with MF All and RDF2Vec achieves MSE 0.0101, $v_{i,j}$3, accuracy 0.7413, and F1 0.7427. The reported pattern is that configuration-specific models perform better on regression, but KGmetaSP performs better on classification with a single meta-model rather than 1,028 models. Similar classification improvements are reported for MF Information Theory, MF Landmarkers, and MF Statistical. For target precision with MF All, configuration-specific models achieve MSE 0.0133, $v_{i,j}$4, and accuracy 0.7347, whereas KGmetaSP reports MSE 0.0164, $v_{i,j}$5, and accuracy 0.7537, again indicating stronger classification performance from the pipeline-agnostic formulation (Klironomos et al., 20 Mar 2026).

In PPE for unseen pipelines, KGmetaSP substantially outperforms the reported baselines. For target accuracy, average performance yields MSE 0.0267, $v_{i,j}$6, accuracy 0.3303, and F1 0.1640; closest embedding yields MSE 0.0127, $v_{i,j}$7, accuracy 0.7748, and F1 0.7747; and KGmetaSP with MF All and RDF2Vec yields MSE 0.0070, $v_{i,j}$8, accuracy 0.8250, and F1 0.8244. For target precision, average performance yields MSE 0.0352 and accuracy 0.3323, closest embedding yields MSE 0.0225, $v_{i,j}$9, and accuracy 0.7773, and KGmetaSP yields MSE 0.0131, $100$0, and accuracy 0.8251. The paper interprets these results as evidence that the framework generalizes to unseen pipelines in a way configuration-specific models cannot (Klironomos et al., 20 Mar 2026).

Appendix results compare RDF2Vec with link-prediction embeddings. In the unseen-pipelines PPE setting, TransE, DistMult, and ComplEx all improve over the naive baselines but perform worse than RDF2Vec. For example, in accuracy regression the best link-prediction model, DistMult, reports MSE 0.0080 and $100$1, while RDF2Vec reports MSE 0.0070 and $100$2. The authors use this to support the claim that walk-based embeddings are particularly suitable for MetaExe-KG (Klironomos et al., 20 Mar 2026).

For DPSE, the reported best Hit@5 under threshold $100$3 is achieved by KGmetaSP with $100$4 at 0.9459, followed by KGmetaSP with $100$5 at 0.9324 and SiFi-Pip. at 0.9324. Under $100$6 and $100$7, KGmetaSP with $100$8 reaches 0.9662, followed by SiFi-Pip. at 0.9527 and both KGmetaSP with $100$9 and MF Norm at 0.9324. For NDCG@1, KGmetaSP with $5$0 achieves 0.8811, which is the best result reported in that column. The paper describes KGmetaSP as competitive with strong textual baselines such as SiFi-Dat. based on Sentence-BERT embeddings over dataset descriptions, and it emphasizes the importance of the pipeline-context embeddings $5$1 and $5$2 for recovering performance-based neighbors (Klironomos et al., 20 Mar 2026).

The practical uses described for KGmetaSP include estimating candidate pipeline performance on new datasets by adding the dataset to the graph, generating embeddings, applying the trained pipeline-agnostic meta-model to $5$3, and ranking candidate pipelines by predicted performance. The framework is also described as suitable for retrieving similar datasets through cosine similarity on $5$4 or $5$5, which can provide hints for model selection, hyper-parameter priors, and transfer of pipeline designs. The paper further notes that KGmetaSP can be used to warm-start AutoML or restrict the search space, and that pipeline embeddings capture structural properties that may be useful for NAS or pipeline structure optimization (Klironomos et al., 20 Mar 2026).

The reported limitations are specific. MetaExe-KG currently focuses on scikit-learn pipelines recorded in OpenML, not other libraries such as TensorFlow or PyTorch. RDF2Vec is trained on a fixed knowledge graph, so updated graphs require retraining or retrofitting. The quality and coverage of OpenML experiment logs affect the usefulness of the learned pipeline-context embeddings. Random Forests, while standard and robust, may not optimally exploit high-dimensional embeddings. Embedding training dominates runtime, at approximately 24 hours and approximately 80GB RAM, although the paper characterizes this as a one-time cost per graph snapshot (Klironomos et al., 20 Mar 2026).

The future directions identified in the source material include extending the graph to other machine-learning frameworks and platforms, adopting incremental RDF2Vec or other incremental knowledge-graph embedding methods for evolving graphs, using the PPE model as a surrogate for end-to-end hyper-parameter optimization, and exploring more advanced meta-models and fusion architectures such as neural architectures, attention, or metric learning. The authors release KGmetaSP code, MetaExe-KG, and MetaExe-Bench at https://github.com/dtai-kg/KGmetaSP, and present them as a reference point for future meta-learning research (Klironomos et al., 20 Mar 2026).