Exchangeable Data Unit Hierarchy

• Hierarchy of exchangeable data units is a structured model that organizes datasets into nested, tree-like partitions based on label invariance.
• It underpins frameworks such as NOMAD/ESL and NDF, enabling consistent metadata exchange and reproducibility in computational and scientific applications.
• The approach leverages probabilistic representations and real tree models to enforce exchangeability, supporting rigorous statistical analysis and robust data interoperability.

A hierarchy of exchangeable data units formalizes the organization and exchange of structured information, typically in scientific data systems or probabilistic models, by imposing a nested, tree-like schema where both the structure and the underlying units are compatible with relabelings (exchangeability). Such hierarchies serve as the foundation for rigorous statistical analysis, semantic data exchange, and reproducibility in computation-heavy fields, ensuring that metadata, provenance, and heterogeneous domain-specific components are consistently and robustly managed. The concept connects probabilistic representations (as in exchangeable random hierarchies) with incentive-compatible and modular data storage formats, and appears in domains ranging from combinatorial probability to computational materials science and astronomy.

1. Formal Definitions and Probabilistic Underpinnings

A hierarchy (or total partition) $\mathcal{H}$ on a set $S$ is a collection of subsets of $S$ satisfying: (1) $S \in \mathcal{H}$, (2) $\{s\} \in \mathcal{H}$ for every $s \in S$, and (3) for all $A, B \in \mathcal{H}$, $A \cap B \in \{A, B, \emptyset\}$—that is, blocks are either nested or disjoint; partial overlap is forbidden (Forman et al., 2011). Any hierarchy $\mathcal{H}$ on a finite set $[n]$ is equivalent to an unlabeled finite rooted tree with $n$ leaves, where each vertex corresponds to a block and the leaves to singletons.

A hierarchy on $\mathbb{N}$ is a sequence $(H_n, n\geq 1)$, with $H_n$ a hierarchy on $[n]$ satisfying the consistency condition $H_n = \{A \cap [n]: A \in H_{n+1}\}$. Exchangeability is defined in analogy with de Finetti’s theorem: the law of $(H_n)$ is invariant under finite permutations of labels.

The de Finetti-type mixture for hierarchies states that every exchangeable hierarchy on $\mathbb{N}$ is a mixture of extreme, independently generated (e.i.g.) laws—a hierarchy law is e.i.g. if, when restricted to pairwise-disjoint finite subsets, the induced hierarchies are independent (Forman et al., 2011).

Equivalently, every exchangeable random hierarchy of positive integers has a distribution induced by sampling i.i.d. from a random real tree $(\mathcal{T}, p)$: for each sampled $t_j$, the blocks correspond to those indices $j$ such that $t_j$ lies in the fringe subtree $F_x$ rooted at some $x \in \mathcal{T}$. There is an alternative yet equivalent representation using interval hierarchies on $[0,1]$, where sets $\{j : U_j \in B\}$ correspond to block membership via i.i.d. uniform draws and a random hierarchy $\mathscr{H}$ on $[0,1]$ (Forman et al., 2011).

2. Hierarchical Data Models in Scientific Computing

A hierarchy of exchangeable data units undergirds data interoperability frameworks such as the NOMAD/ESL metadata schema (Ghiringhelli et al., 2016) and the N-Dimensional Data Format (NDF) (Jenness et al., 2014). Both employ rooted, typed trees to impose structure, ensure semantic equivalence across formats, and enable robust exchange and comparison of scientific results.

The NOMAD/ESL schema is organized into nested "sections," each a typed metadata container. At the top level, section_run (provenance of the computation) contains references to section_system (atomic configuration), section_method (theoretical model), and section_single_configuration_calculation (results for one system–method pairing), among possible others. Nodes can contain both key–value fields and other sections, forming a practical directed tree, sometimes with cross-links. The approach enforces semantic consistency and guarantees that every data unit—ranging from experimental parameters to intermediate SCF iterations—is annotated by units, references to baselines, and links to domain-specific conventions (Ghiringhelli et al., 2016).

NDF, by contrast, defines a minimal core (mandatory DATA_ARRAY) with extensible optional substructures—e.g., VARIANCE, QUALITY, AXIS, WCS, HISTORY, PROVENANCE (a DAG of ancestors), and a MORE container for arbitrary application extensions—thereby enabling exchange and round-trip conversion across astronomy analysis pipelines (Jenness et al., 2014). This strict yet extensible hierarchy, enforced by libraries rather than by file schema alone, supports substantial heterogeneity without sacrificing interoperability.

3. Mechanisms Ensuring Exchangeability and Reproducibility

Core to the guarantee of exchangeability in these data hierarchies is the imposition of strict shape and type constraints, propagation and registration rules, and semantic coupling to domain information.

For example, in the NDF model, an object is recognized as a valid NDF if its root contains a DATA_ARRAY node (duck-typing rule). All primary arrays must agree in shape, and application-defined lists enforce consistent propagation of components (e.g., WCS, AXIS, HISTORY), preventing accidental metadata loss. Provenance is stored as an explicit DAG to support reproducibility (Jenness et al., 2014).

In the NOMAD/ESL ecosystem, all metadata possess explicit "units" attributes (defaulting to SI), with additional annotations (e.g., zero_reference for energies) to standardize relative measurements. The schema enforces hierarchical encapsulation and bidirectional links among sections, e.g., iterations reference parent methods and systems, handling the exchange and recomputation of results in distributed and cross-code workflows (Ghiringhelli et al., 2016).

Semantic interoperability is realized by maintaining a central metadata dictionary spanning names, types, and conventions for all fields; this registry is consumed by both converters and native libraries, ensuring that exchange is lossless and exact regardless of the origin (Ghiringhelli et al., 2016).

4. Two Representations of Exchangeable Hierarchies: Trees and Interval Hierarchies

A central result (Forman et al., 2011) establishes that any exchangeable hierarchy of positive integers admits two equivalent representations:

• Random real tree representation: Given a random rooted, weighted real tree $(T, d, r, p)$, i.i.d. samples $t_j \sim p$ generate blocks $\{j : t_j \in F_x\}$ for each $x \in T$, collecting all singleton subsets and $\mathbb{N}$. This representation captures the full law of any exchangeable hierarchy and supports decomposition of the measure $p$ into atomic, continuous, and diffuse parts, analogous to Kingman’s theory for exchangeable partitions.
• Interval hierarchy on $[0,1]$: The real tree can be traversed in "depth-first" fashion to yield a random hierarchy $\mathscr{H}$ of intervals on $[0,1]$. Blocks for indices $j$ are then determined by membership of i.i.d. samples $U_j \sim \operatorname{Unif}[0,1]$ in Borel sets from $\mathscr{H}$, with the distribution reconstructing the original hierarchy law.

These constructions connect infinite-dimensional probability, combinatorial structures, and practical data models.

5. Hierarchical Metadata Models: Design Patterns and Implementation

Practices from NDF and NOMAD/ESL offer converging lessons for actionable, extensible hierarchical metadata:

• Minimal core and extension pattern: Both systems define a minimal required structure (e.g., DATA_ARRAY, section_run) and accommodate domain-specificity through structured extension (NDF’s MORE container, new NOMAD sections).
• Component propagation and compatibility: Applications enforce predetermined propagation lists to ensure all recognized metadata move intact through data processing, supporting auditability and compliance.
• Formal registration and shape consistency: A central registry of names/types, shape constraints, and versioning rules guarantees that extensions do not conflict and that existing applications can process new fields safely.
• Semantic annotation of quantities: Use of standardized units, explicit zero-references, and domain tags enables meaningful cross-comparison and algorithmic post-processing, critical for computational reproducibility (Ghiringhelli et al., 2016, Jenness et al., 2014).
• Extensibility and namespaces: The use of an extension tree with name uniqueness prevents collision and supports arbitrary vendor- or code-specific annotation.

This design converges on the principle that a strict but extensible hierarchy, enforced by libraries and central registries, most effectively supports long-lived and interoperable scientific data exchange.

6. Performance, Scalability, and Outlook

While performance benchmarks are rarely published, systems such as NDF document mechanisms for chunked processing, lazy loading, and provenance compression, enabling handling of GB-scale datasets in astronomy. The HDS and associated libraries provide automatic type conversion, byte-swapping, and block-based access for efficiency. Provenance tracking is optimized to prevent exponential growth due to DAG depth by storing nodes in a bespoke packed integer format (Jenness et al., 2014).

In computational materials, direct support for hierarchical metadata in parallel I/O, either via converters (from legacy code output to standardized formats) or via native library APIs (writing directly into the schema), brings immediate compatibility and reproducibility (Ghiringhelli et al., 2016).

Outlook and Open Problems

The uniqueness (up to measure-preserving isomorphism) of tree representations for exchangeable hierarchies remains open. Similarly, the development of canonical "moment-formulas" for exchangeable hierarchies in analogy with Kingman’s simplex for partitions is unresolved (Forman et al., 2011). The ongoing evolution of metadata dictionaries and the challenge of balancing minimality with extensibility continue to shape best practices in hierarchical exchangeable data management.

---

References:

• (Forman et al., 2011): "A representation of exchangeable hierarchies by sampling from real trees"
• (Ghiringhelli et al., 2016): "Towards a Common Format for Computational Material Science Data"
• (Jenness et al., 2014): "Learning from 25 years of the extensible N-Dimensional Data Format"