Koza: Modular Biomedical Data Ingestion

• Koza is a standalone Python package that modularizes biomedical data ingestion into interoperable, KGX- and Biolink-compliant property graphs.
• It leverages declarative YAML configurations and Python transforms to control readers, writers, and schema validations at write time.
• The system automates monthly ingests via GitHub Actions, facilitating reproducible graph assembly from over 30 gold-standard biomedical sources.

Koza is a standalone Python package for streamlining the ingestion of raw biomedical data into a KGX-compliant property graph, and the Koza-Hub is an associated, continuously growing collection of modular ingest repositories for more than thirty gold-standard biomedical data sources (Korn et al., 11 Sep 2025). The system is designed around modularity, declarative YAML configuration, reusability, transparency, and standards-centric output targeting KGX and the Biolink Model, with the stated aim of producing “born interoperable” knowledge graph artifacts. In the formulation presented for Koza, knowledge graph construction is reduced to a set of primitive operations, with schema compliance enforced at write time and monthly execution of source-specific ingests via GitHub Actions (Korn et al., 11 Sep 2025).

1. Definition and design objectives

Koza is presented as a standalone Python package in which each data source is handled by a self-contained “ingest.” This modular decomposition makes development, testing, and documentation straightforward on a per-source basis. Configuration is declarative rather than embedded in hard-coded scripts: readers and writers are parameterized through YAML, while source-specific logic is placed in Python transform modules. The package further emphasizes reusability and transparency by making transformation steps explicit, versioned, and testable, thereby reducing redundant labor across projects (Korn et al., 11 Sep 2025).

The standards-centric aspect of the design is central to its definition. Koza targets KGX, the Knowledge Graph Exchange format, and the Biolink Model schema, so that outputs are not merely graph-structured but conformant with a shared exchange and semantic layer. The paper characterizes this as producing outputs that are “born interoperable” (Korn et al., 11 Sep 2025). In context, this means that interoperability is treated as a property of the ingest process itself, rather than as a downstream reconciliation step.

The Koza-Hub extends this model operationally. It is described as a collection of modular Koza ingest repositories hosted on GitHub, covering 30+ gold-standard biomedical data sources. These ingests are run each month through GitHub Actions, producing updated KGX artifacts that can be trivially merged into a custom knowledge graph (Korn et al., 11 Sep 2025). A plausible implication is that Koza distinguishes between a core transformation runtime and a repository layer that curates reusable source adapters.

2. Pipeline architecture and execution model

Koza’s architecture is organized around three pluggable components: a Reader, a Transform, and a Writer. The paper depicts the end-to-end flow as raw data in formats such as CSV, JSON, XML, or SQL passing through a YAML-driven Reader into a stream of records, then through Python transform hooks that emit Biolink Nodes and Edges, and finally through a YAML-driven Writer that serializes to nodes.tsv and edges.tsv in KGX format (Korn et al., 11 Sep 2025).

The Reader parses raw files into Python dict records. Its configuration is expressed in YAML and may specify items such as delimiter, header row, or JSON path. It can also declare record filters using a mini matching language. This positioning of filtering at the Reader level is significant because it allows coarse selection before transform logic is executed.

The Transform is implemented as a Python module with decorator hooks such as @transform_record or @transform. It is responsible for converting each input record into zero or more Biolink Node and Edge objects. The transform context, kt, exposes kt.write(obj) for emitting objects, kt.state as a persistent dictionary for cross-record state, and kt.log() for centralized logging. This design concentrates source-specific semantics in a narrow API surface while leaving I/O and validation to the surrounding framework (Korn et al., 11 Sep 2025).

The Writer serializes Node and Edge objects to KGX TSV and validates them against the Biolink Model. Its behavior is also controlled by YAML, including file paths, column ordering, and chunk sizes. The formalization given in the paper makes the execution model explicit: if $R=\{r_1,\ldots,r_n\}$ is the raw record stream, then the transform is a function $T:r \to (N \cup E)$, where $N$ and $E$ are sets of Node and Edge objects. The final KGX graph $G=(V,E')$ is assembled by taking the union of node and edge emissions across records (Korn et al., 11 Sep 2025).

This decomposition suggests a deliberate separation between syntactic ingestion, semantic mapping, and standards-constrained serialization. In practical terms, it permits the same writer and validation machinery to be reused across heterogeneous biomedical sources.

3. Declarative configuration and transformation semantics

Every ingest defines at least one koza.yaml or similar configuration file. The example given in the paper configures a CSV Reader with a path, delimiter, header presence, and a filter requiring evidence_score >= 0.8; a KGX Writer with output paths and a Biolink vocabulary URI; and a transform module/function binding for a gene–disease association ingest (Korn et al., 11 Sep 2025). The paper’s example makes clear that YAML carries both data-access parameters and high-level transform parameters, while the Python transform expresses the source-to-schema mapping.

In the corresponding Python transform, a record is mapped to a Node for a gene, a Node for a disease, and an Edge with predicate biolink:gene_associated_with_condition, with confidence stored as a property. The transform uses kt.write to emit each object. The paper states that the YAML reader parameters ensure only records satisfying evidence_score ≥ 0.8 reach the transform, while the writer parameters guarantee KGX-compliant TSV output (Korn et al., 11 Sep 2025).

The transformation model is further specified through primitive operations. Filtering is defined by a predicate $$p(r): \text{record} \to \{\text{true}, \text{false}\}$$, with records failing the predicate dropped before transform. Node creation is formalized as a function producing an object $n=(id, category, name, properties)$, where the id, name, and property values are computed from the input record and category is a Biolink entity type. Edge creation is analogously defined as $e=(subject, predicate, object, properties)$ with subject and object identifiers and a Biolink relation. Arbitrary record fields may be promoted to edge qualifiers or node properties by passing a dictionary to the constructor, and stateful aggregation is supported through kt.state, for example when grouping synonyms across records (Korn et al., 11 Sep 2025).

These semantics matter because they make Koza’s abstraction layer intentionally small. Rather than supplying a large domain-specific language, the package exposes a concise set of primitives whose outputs are already aligned with Biolink data structures. This suggests that extensibility is achieved through Python transforms and YAML parameterization rather than through a separate transformation calculus.

4. Schema compliance and interoperability guarantees

A defining feature of Koza is that schema compliance is enforced during serialization. The Writer fetches the Biolink JSON-LD and builds an in-memory schema of allowed categories, predicates, and required fields. Before writing each line, it validates that Node.category belongs to the allowed categories, that Edge.predicate belongs to the allowed relations, and that Edge.subject and Edge.object are non-empty and well-formed CURIEs (Korn et al., 11 Sep 2025).

The paper also states formal constraints for edges: for every edge $e$, $\mathrm{subject}(e) \in I$, $T:r \to (N \cup E)$0, and $T:r \to (N \cup E)$1, where $T:r \to (N \cup E)$2 is the set of valid CURIE identifiers in Biolink and $T:r \to (N \cup E)$3 is the set of predicates (Korn et al., 11 Sep 2025). This places interoperability on a concrete syntactic and semantic basis, rather than leaving it as an informal design goal.

On validation failure, the offending object is logged with record context through kt.log.error. By default, Koza halts with a SchemaValidationError, although the behavior can be configured to skip invalid records. This mechanism is consequential for reproducibility: invalid outputs are not silently admitted into the graph unless the ingest author explicitly chooses a permissive mode.

The phrase “born interoperable” can be misunderstood as implying that downstream integration problems disappear automatically. The description of Koza does not support that stronger claim. What it does support is that outputs conform to KGX and the Biolink Model at production time (Korn et al., 11 Sep 2025). A plausible implication is that Koza reduces a major class of interoperability failures—format and schema incompatibility—even though cross-source deduplication, qualifier reconciliation, and broader harmonization may still be required elsewhere in the pipeline.

5. Representative workflows and graph assembly

The paper describes a ClinGen pathogenic variant ingest as a concrete workflow. The raw source is a ClinGen JSON release containing sequence variants with associated gene and disease mappings. A JSONReader is configured with the path to the ClinGen JSON and a json_path targeting the “pathogenicity” array. Records are filtered so that only those with classification == "Pathogenic" are processed. The transform creates a Biolink:SequenceVariant node with identifier ClinGenVar:<variant_id>, looks up the associated gene as a Biolink:Gene, and emits edges from the sequence variant to the gene using biolink:has_variant_location and from the sequence variant to the disease using biolink:related_to. The KGXWriter then outputs nodes.tsv and edges.tsv, yielding a monthly refreshed KGX fragment of ClinGen pathogenic variants that is directly mergeable into Monarch KG (Korn et al., 11 Sep 2025).

A second workflow, the Model Organism KG (MOKG), illustrates multi-source assembly. The paper specifies nine modular Koza ingests, including HGNC, HPOA, PANTHER, PomBase, Dictybase, Alliance, and NCBI Gene. These outputs are concatenated with the koza-merge CLI into a unified KGX representation, after which global deduplication and qualifier merging are performed across sources. The result is described as a specialized Neo4j database for model organism research, updated monthly (Korn et al., 11 Sep 2025).

These examples clarify the intended scope of Koza. It is not limited to one-off conversion of a single source into an exchange format; it also supports recurrent production of modular graph fragments and their subsequent composition. The ClinGen example emphasizes source-specific semantic mapping under schema enforcement, while the MOKG example emphasizes compositionality across independently maintained ingests.

6. Operational characteristics, scaling behavior, and constraints

Koza’s performance model is explicitly tied to streaming execution. The paper states that the streaming design keeps memory usage to $T:r \to (N \cup E)$4 per record, with the exception that the Koza state dictionary grows if stateful transforms use it. Typical runtimes are reported as less than five minutes on a single CPU core for small CSV or JSON sources below 100 MB, and 30–45 minutes for medium datasets of several GB. Average throughput is reported as 50–100 K records per second for simple node/edge transforms on modern hardware (Korn et al., 11 Sep 2025).

Scalability is described as source- or module-level parallelism. Ingest jobs are said to be trivially parallelizable by source/module, and Koza-Hub runs 30 CI jobs in parallel via GitHub Actions. A single ingest typically uses less than 2 GB RAM and can run on a standard 4-core virtual machine (Korn et al., 11 Sep 2025). These details position Koza as a framework optimized for many moderately sized, independently executable ingests rather than for monolithic, tightly coupled distributed graph ETL.

The paper also identifies concrete limitations. GitHub Artifact size limits of 2 GB cap the size of published KGX modules, motivating migration of large ingests to alternative object stores such as AWS S3 or Zenodo. Real-time updates are not native: modular ingests must be re-run to reflect source changes, with incremental or webhook-triggered updates identified as future work. Schema evolution in Biolink may break existing ingests, leading to a plan for version-pinned Biolink releases and compatibility shims. Very large datasets, exemplified by gnomAD at 1.6 TB, exceed typical compute capacity; proposed directions include collaboration with providers for native KGX exports or integration of a distributed ingestion framework such as Spark (Korn et al., 11 Sep 2025).

The planned enhancement of the primitive set—adding joins, lookups, and near-duplicate merging primitives—indicates that the current abstraction deliberately prioritizes a minimal core. This suggests a tension, common in ETL systems, between keeping the primitive API simple and accommodating increasingly complex transformation logic without bespoke code.

7. Position within biomedical knowledge graph engineering

Koza is situated in the paper as a response to redundant labor in biomedical knowledge graph construction, with those redundancies attributed to the lack of data standards and the scarcity of “knowledge-graph ready” source data (Korn et al., 11 Sep 2025). Its answer is not a new graph schema, but a workflow formalization: raw biomedical information is ingested through YAML-configured readers, transformed through explicit Python hooks using a small primitive set, and emitted as KGX under Biolink validation.

Within biomedical knowledge graph engineering, its significance lies in the combination of modular source adapters, explicit and versioned transformation logic, schema-constrained output, and routine artifact publication through the Koza-Hub. These characteristics support rapid, reproducible, and interoperable knowledge graph builds and make it straightforward to merge source-specific outputs into custom graph products (Korn et al., 11 Sep 2025).

A plausible implication is that Koza serves as infrastructure for institutionalizing ingest maintenance as a shared, testable, and automatable practice rather than as a collection of isolated conversion scripts. In that sense, its main contribution is methodological as much as technical: it turns knowledge graph ingest into a standardized software engineering problem centered on reusable modules, declarative configuration, and validation against a common semantic model.