Nested tree space: a geometric framework for co-phylogeny

Abstract: Nested (or reconciled) phylogenetic trees model co-evolutionary systems in which one evolutionary history is embedded within another. We introduce a geometric framework for such systems by defining $σ$-space, a moduli space of fully nested ultrametric phylogenetic trees with a fixed leaf map. Generalizing the $τ$-space of Gavryushkin and Drummond, $σ$-space is constructed as a cubical complex parametrised by nested ranked tree topologies and inter-event time coordinates of the combined host and parasite speciation events. We characterise admissible orderings via binary \textit{nesting sequences} and organise them into a natural poset. We show that $σ$-space is contractible and satisfies Gromov's cube condition, and is therefore CAT(0). In particular, it admits unique geodesics and well-defined Fréchet means. We further describe its geometric structure, including boundary strata corresponding to cospeciation events, and relate it to products of ultrametric tree spaces via natural forgetful maps.

• The paper introduces σ-space, a novel CAT(0) cubical complex for modeling nested ultrametric phylogenies with reconciled leaf maps.
• It extends existing single-tree spaces by incorporating combinatorial nesting sequences that capture cospeciation and duplication events.
• The CAT(0) property guarantees unique geodesics and supports advanced statistical tests and inference methods in co-phylogeny studies.

Nested Tree Space: A Geometric Framework for Co-Phylogeny

Introduction

This work introduces the $\sigma$-space, a cubical complex that provides an intrinsic geometric framework for co-phylogenetic systems by modeling nested evolutionary histories. The formulation is designed to address the analysis of systems where one phylogeny (e.g., parasites or genes) is naturally embedded within another (e.g., hosts or species), under the assumption that host-switching events are absent or rare. The paper expands on the well-established frameworks for single-tree spaces, notably $\tau$-space [ultra], generalizing geometric and combinatorial constructions to the domain of nested ultrametric phylogenies. $\sigma$-space fundamentally encodes the space of all fully nested, ultrametric, reconciled phylogenetic trees with fixed leaf associations.

Background: Phylogenetic Spaces and Nested Trees

The construction of geometric spaces for individual phylogenetic trees (spaces such as Billera-Holmes-Vogtmann (BHV) space [BHV] and $\tau$-space [ultra]) enabled the development of statistical methodologies on phylogenies, including well-defined metrics, geodesics, and statistical means. For ultrametric trees, $\tau$-space utilizes inter-event time coordinates and is piecewise Euclidean, non-positively curved (CAT(0)), admitting efficient computation of geodesics and explicit calculation of Fréchet means.

The reconciliation of phylogenies—such as gene/species or host/parasite systems—requires the embedding of one evolutionary history in another, capturing events such as cospeciation, duplication, and sorting, while host-switches are absent in the current framework. The main challenge is to construct a space encoding not just individual tree topologies and their event times, but also the combinatorics describing which events in the embedded tree correspond or are coupled to events in the embedding tree, all under fixed leaf mapping constraints.

Definitions: Nested Ultrametric Trees, Nesting Sequences, and Topologies

A nested phylogenetic tree is defined by a triple $(T_H,T_P,\ell)$:

• $T_H$ (host) and $T_P$ (parasite) are rooted, weighted ultrametric trees,
• $\ell$ is a leaf map associating each parasite tip to a host tip,
• Host-parasite compatibility is ensured via ultrametric inequalities on the induced pairwise distances (guaranteeing that parasite divergence events do not precede those possible for their hosts).

Key to the combinatorial structure are nesting sequences: binary strings recording the interleaving of parasite and host speciation events by rank, under the constraints imposed by the leaf mapping and tree topology. The admissibility of a nesting sequence is characterized by cumulative inequalities (reflecting that parasite events can never outpace host events beyond what is allowed by the leaf map degeneracies). The poset structure on nesting sequences describes allowable rank changes that interpolate between maximally-cospeciating and fully-decoupled evolutionary scenarios.

The $\sigma$-Space Construction

$\tau$0-space, denoted $\tau$1 for $\tau$2 hosts, $\tau$3 parasites, and fixed mapping $\tau$4, is a cubical complex where:

• Each maximal orthant corresponds to a fully resolved nested ranked topology and an admissible nesting sequence.
• Coordinates within each orthant are inter-event times (generalizing $\tau$5-coordinates) for the combined sequence of host and parasite events. Each coordinate is labeled by its biological nature (host, parasite, or cospeciation).

Boundaries between orthants arise from:

• Rank changes (switching order of adjacent events),
• Nearest-neighbor interchanges (NNI) in either tree,
• Event time degeneracies (leading to multifurcations),
• Cospeciation boundaries (codimension-1 external boundaries corresponding to simultaneous host and parasite speciation events in the same lineage).

The faces of the complex provide a stratification by evolutionary scenario: interleaved regions correspond to maximal cospeciation, while decoupled regions correspond to independent duplication/speciation events.

(Figure 1)

Figure 1: Illustration of the space $\tau$6, depicting orthant adjacency, cospeciation boundaries (thick black lines), and the cone point (red dot) as a degenerate nested star tree.

Geometric and Topological Properties

The central analytical result is that $\tau$7-space is CAT(0): it is contractible, admits unique geodesics between any two points (i.e., any pair of nested trees), and supports well-defined Fréchet means and additional statistical tools. This follows from the verification of Gromov's cube condition via a detailed analysis of the orthant poset, specifically proving the absence of three-cycles in the link of the origin (i.e., the complex's local combinatorics).

Noteworthy geometric features include:

• Cospeciation boundaries, which are not shared between orthants and correspond to strict cospeciation events,
• The domain of perfect cospeciation, a locus where host and parasite trees are perfectly topologically and metrically matched; in the bijective case, this is homeomorphic to standard ultrametric tree space,
• Explicit mapping (forgetful) from $\tau$8-space to products of single-tree $\tau$9-spaces, with the image characterized by compatibility congruent with the original leaf mapping and the absence of horizontal transfer.

For small instances (e.g., two hosts and three parasites), the complex is visualized as the gluing of polytopes related by the possible combinatorial arrangements of speciation and cospeciation (see Figure 1).

Implications and Applications

The development of $\sigma$0-space offers a rigorous geometric substrate for statistical analysis on collections of reconciled trees, supporting principal geodesic analysis (PGA), statistical hypothesis tests concerning co-speciation, and the formulation of confidence regions or kernel densities for co-evolving tree pairs. Its CAT(0) property ensures the tractability of such analyses, paralleling established computational results for single-tree spaces [OwenProvan2011].

The framework integrates with existing reconciliation, simulation, and inference algorithms by providing a natural metric for tree space, facilitating geometric comparisons, averaging, and uncertainty quantification. The treatment of cospeciation boundaries and the explicit characterization of interleaved and decoupled loci allow for quantification and testing of cophylogenetic hypotheses. The lack of host-switching limits direct application in some biological systems, but the geometric construction is amenable to further extension, such as to networks or more general graphs encoding reticulate events.

From a theoretical perspective, $\sigma$1-space generalizes and refines the earlier approaches by providing an intrinsic metric structure, accommodating the nested combinatorics and evolutionary constraints inherent in cophylogeny, and situating reconciled tree analysis within the broader landscape of non-positively curved polyhedral complexes.

Future Directions

Potential extensions include:

• Generalization to accommodate host-switches, loss, and lateral gene transfer (embedding in phylogenetic networks),
• Efficient algorithms for geodesic computation in arbitrary $\sigma$2-spaces (generalizing the Owen-Provan approach),
• Statistical methodology for hypothesis testing in co-evolutionary scenarios (e.g., detecting cospeciation versus duplication dominance),
• Enumeration of nested ranked topologies for large $\sigma$3 or structural classes,
• Integration with empirical data in microbiome-host phylosymbiosis and symbiont/parasite systems.

Conclusion

$\sigma$4-space equips the study of nested phylogenies with a contractible, CAT(0) cubical complex, uniting combinatorial reconciliation and metric embedding in a unified framework. This geometric foundation enables both rigorous comparison and statistical inference for co-evolving evolutionary histories, opening prospects for deeper integration of geometry and biology in evolutionary studies.

(Figure 2)

Figure 2: Illustration of the space $\sigma$5, summarizing the gluing of decoupled and interleaved orthants, with cospeciation boundaries and the degenerate cone point highlighted.