Kernel of Partition Paths: A Unified Representation for Tree Ensembles

Abstract: A recent line of work has reframed individual decision trees as linear models on engineered features associated with their splits, opening routes for oracle inequalities and feature-importance reinterpretation, but leaving open the question of what unified geometric object a forest induces when one indexes its feature map by nodes rather than by splits. The present paper studies that object. KPP indexes the feature map by the nodes of the forest, weighted by a path metric that turns each coordinate into a component of a squared-Euclidean path-isometric embedding. KPP unifies four pillars under a single non-diagonal Gram that carries a metric: prediction, exact additive attribution, deterministic Lipschitz robust radius in the KPP metric, and uniform Rademacher risk bounds for regression and classification under fixed, honest, or cross-fit conditioning. All probabilistic guarantees are conditional on the representation and are stated under three explicit conditioning regimes; the robust-radius guarantee is deterministic in the KPP metric rather than in a norm on the raw input. Conjectured fast-rate refinements for both regression and classification are stated as open problems and are not claimed as theorems.

• The paper introduces the Kernel of Partition Paths (KPP) that creates a node-indexed, path-metric geometric embedding for tree ensembles.
• It unifies prediction, exact additive attribution, deterministic robustness certificates, and Rademacher-based generalization bounds.
• Empirical benchmarks demonstrate competitive performance and provide interpretability with structural diagnostics for tree-based models.

Kernel of Partition Paths (KPP): Unified Geometric Representation for Tree Ensembles

Overview and Motivation

The paper introduces the Kernel of Partition Paths (KPP) framework as a geometric, node-indexed representation for tree ensembles. The crux is to index the ensemble's feature map by the nodes in the forest (not by splits or leaves), weighting each coordinate with a tree-specific path metric. This induces a squared-Euclidean, path-isometric embedding whose geometry both encodes partition structure and supports a non-diagonal Gram matrix. KPP unifies four pillars: prediction, exact additive attribution, deterministic robust radius certificates in the KPP metric, and Rademacher-based generalization bounds for regression and classification. All probabilistic guarantees are conditional on the representation and stated under three explicit regimes (fixed, honest, cross-fit).

Geometric Construction: Node-indexed Path Embedding

KPP is predicated on representing each input $x$ by a feature vector $(x)$ whose coordinates correspond to forest nodes, weighted via a path metric. This metric is constructed as follows: node weights capture relevance (e.g., impurity decrease); each edge weight is defined as the mean of its adjacent nodes' weights. The root-to-leaf path for each input $x$ is mapped to the respective coordinates via $$\phi_{t, v}(x) = \sqrt{a_t(par(v)) + a_t(v)/2}\,1_{x \in S_v}$$, where $S_v$ denotes the node's region.

A rigorous isometry theorem (Corollary~KPP Isometry, forest level) establishes that the Euclidean distance between embedded vectors reconstructs the normalized sum of weighted path distances across all trees, i.e., $(x)-(x')||^2 = d_T(x, x')/S_T$ for total forest mass $S_T$. Unlike split-based views (RuleFit, MDI+), where Gram matrices are diagonal and lack path geometry, or leaf/proximity-based kernels (RF-GAP, KMERF), which encode similarity without explicit metric, KPP yields a non-diagonal Gram encoding full path-metric structure.

Unified Guarantees: Prediction, Attribution, Robustness, Generalization

KPP delivers four explicit theoretical pillars. Prediction is framed as linear models in the KPP space ($x \mapsto w^T(x) + b$), with population and empirical risk for standard regression (squared loss) and classification (logistic loss).

Exact additive attribution is realized at both node and variable levels: scores decompose as $g(x) = b + \sum_u w_u \phi_u(x)$, and variable contributions are summed over their respective splits. The empirical baseline variant supports dataset-level centering.

Deterministic robustness is instantiated as Lipschitz certificates in the KPP metric: $|g(x) - g(x')| \leq ||w||_2 \sqrt{\delta(x,x')}$. Classification robust radius is certified via margin: for margin $(x)$0, any $(x)$1 within $(x)$2 preserves label, leading to robust-accuracy curves on benchmarks.

Uniform generalization bounds are obtained via trace-based Rademacher complexity: $(x)$3, where $(x)$4. Ridge regression admits bias-variance decompositions, and ridge effective dimension is defined via eigenvalue trace. Uniform loss bounds for regression and logistic risk (calibrated to $(x)$5 error) are proved under fixed/honest/cross-fit regimes.

Proof Boundary and Conditioning Regimes

Critical methodological discipline is enforced in all statistical claims: they are conditional on the learned representation (the forest and node-indexed feature map), not fully end-to-end. Three regimes are formalized: fixed representation, honest representation (partition/fit split), and cross-fit (multiple folds, each fit with representation learned excluding that fold’s labels). This boundary is motivated by label dependence in tree-growing and guarantees for the final convex learner (ridge/logistic).

Benchmark and Empirical Diagnostics

KPP's predictive behavior is empirically benchmarked against six standard baselines (RF, GBM, XGBoost, LightGBM, CatBoost, ridge/logistic on raw features) across five tabular datasets (classification and regression). Results are competitive: KPP achieves rank 1 on spambase_full and wine_quality_red, rank 2 on concrete, with mixed performance on others.

KPP differentiates itself via post-hoc structural guarantees—robust radius certificates, trace/effective-dimension diagnostics, partition-quality oracles—not available in standard boosting pipelines. Empirical dashboards quantitatively monitor model capacity, conditioning, forest stability, partition quality, and margin robustness.

Figure 1: Robust-radius certificate curves on five benchmarks; Top row: $(x)$6 (classification), Bottom: $(x)$7 (regression). Certificates reflect deterministic robustness in the KPP metric, not raw input perturbations.

Partition Oracles and Attribution

The leaf-wise oracle risk in both regression and classification decomposes as root risk minus telescoping sum of local refinement gains (variance or entropy). At the forest level, the per-tree average is the correct invariant, not the raw sum across all nodes; this precludes incorrect interpretations of ensemble gains.

Exact node-level and variable-level attribution follow directly from the geometric embedding, supporting rigorous, interpretable decompositions of forest predictions.

Methodological Limitations and Open Problems

Several limitations and explicit open conjectures frame the scope:

• Fast-rate excess-risk (sub-$(x)$8) in regression requires distributional assumptions (Bernstein-type, strong convexity) not proved here (FR-R-01).
• In classification, geometric-margin tail in the KPP metric does not imply score-margin tail needed for fast rates; direct score-margin assumption is necessary (FR-C-01).
• End-to-end unconditional bounds for algorithms coupling tree construction and ERM require stability or uniform complexity analysis of the representation-building step.
• Detected empirical methodology asymmetry: unconstrained ridge/logistic fits often yield trivial uniform bounds at the realized $(x)$9, motivating stricter conditioning regimes.

Extensions such as multiclass, oblique splits, alternative loss functions, or multi-ensemble aggregation can inherit the path-isometry skeleton, contingent on analogous normalizations and proof-boundary discipline.

Implications and Future Directions

KPP provides a rigorous geometric foundation for tree ensemble representations with explicit, interpretable metrics, unifying prediction, attribution, robustness, and generalization. In practical contexts, it empowers deterministic robustness diagnostics and precise attribution not accessible in conventional boosting frameworks. Theoretically, it catalyzes further investigation into margin-based fast rates, stability of geometric representation, and honest cross-fit guarantees.

Future developments may include: multiclass path-metric kernels, oblique partition paths, localized complexity for fast-rate bounds, and stability-based unconditional generalization analyses. KPP's geometric structure is likely to influence ensemble interpretability, robust certification protocols, and hybrid representations combining partition-based geometry with linear tractability.

Conclusion

The Kernel of Partition Paths framework introduces a node-indexed, path-metric geometric embedding for tree ensembles that supports prediction, exact additive attribution, deterministic robustness certificates, and rigorous generalization bounds within a unified, non-diagonal Gram matrix. All statistical guarantees are conditional on representation, with explicit honesty/cross-fit regimes. Empirical benchmarks confirm competitive performance and highlight interpretability and robustness advantages. Open questions center on fast-rate risk bounds and end-to-end pipeline guarantees, positioning KPP as a structurally rigorous and extensible representation for tree-based ensemble learning.