---
title: 'Koza: Modular Biomedical Data Ingestion'
url: https://www.emergentmind.com/topics/koza
type: topic
---

# Koza: Modular Biomedical Data Ingestion

Koza is a standalone Python package for streamlining the ingestion of raw biomedical data into a KGX-compliant property graph, and the Koza-Hub is an associated, continuously growing collection of modular ingest repositories for more than thirty gold-standard biomedical data sources [2509.09096]. The system is designed around modularity, declarative YAML configuration, reusability, transparency, and standards-centric output targeting KGX and the Biolink Model, with the stated aim of producing “born interoperable” knowledge graph artifacts. In the formulation presented for Koza, knowledge graph construction is reduced to a set of primitive operations, with schema compliance enforced at write time and monthly execution of source-specific ingests via GitHub Actions [2509.09096].

## 1. Definition and design objectives

Koza is presented as a standalone Python package in which each data source is handled by a self-contained “ingest.” This modular decomposition makes development, testing, and documentation straightforward on a per-source basis. Configuration is declarative rather than embedded in hard-coded scripts: readers and writers are parameterized through YAML, while source-specific logic is placed in Python transform modules. The package further emphasizes reusability and transparency by making transformation steps explicit, versioned, and testable, thereby reducing redundant labor across projects [2509.09096].

The standards-centric aspect of the design is central to its definition. Koza targets KGX, the Knowledge Graph Exchange format, and the Biolink Model schema, so that outputs are not merely graph-structured but conformant with a shared exchange and semantic layer. The paper characterizes this as producing outputs that are “born interoperable” [2509.09096]. In context, this means that interoperability is treated as a property of the ingest process itself, rather than as a downstream reconciliation step.

The Koza-Hub extends this model operationally. It is described as a collection of modular Koza ingest repositories hosted on GitHub, covering 30+ gold-standard biomedical data sources. These ingests are run each month through GitHub Actions, producing updated KGX artifacts that can be trivially merged into a custom knowledge graph [2509.09096]. A plausible implication is that Koza distinguishes between a core transformation runtime and a repository layer that curates reusable source adapters.

## 2. Pipeline architecture and execution model

Koza’s architecture is organized around three pluggable components: a Reader, a Transform, and a Writer. The paper depicts the end-to-end flow as raw data in formats such as CSV, JSON, XML, or SQL passing through a YAML-driven Reader into a stream of records, then through Python transform hooks that emit Biolink Nodes and Edges, and finally through a YAML-driven Writer that serializes to `nodes.tsv` and `edges.tsv` in KGX format [2509.09096].

The Reader parses raw files into Python `dict` records. Its configuration is expressed in YAML and may specify items such as delimiter, header row, or JSON path. It can also declare record filters using a mini matching language. This positioning of filtering at the Reader level is significant because it allows coarse selection before transform logic is executed.

The Transform is implemented as a Python module with decorator hooks such as `@transform_record` or `@transform`. It is responsible for converting each input record into zero or more Biolink Node and Edge objects. The transform context, `kt`, exposes `kt.write(obj)` for emitting objects, `kt.state` as a persistent dictionary for cross-record state, and `kt.log()` for centralized logging. This design concentrates source-specific semantics in a narrow API surface while leaving I/O and validation to the surrounding framework [2509.09096].

The Writer serializes Node and Edge objects to KGX TSV and validates them against the Biolink Model. Its behavior is also controlled by YAML, including file paths, column ordering, and chunk sizes. The formalization given in the paper makes the execution model explicit: if $R=\{r_1,\ldots,r_n\}$ is the raw record stream, then the transform is a function $T:r \to (N \cup E)$, where $N$ and $E$ are sets of Node and Edge objects. The final KGX graph $G=(V,E')$ is assembled by taking the union of node and edge emissions across records [2509.09096].

This decomposition suggests a deliberate separation between syntactic ingestion, semantic mapping, and standards-constrained serialization. In practical terms, it permits the same writer and validation machinery to be reused across heterogeneous biomedical sources.

## 3. Declarative configuration and transformation semantics

Every ingest defines at least one `koza.yaml` or similar configuration file. The example given in the paper configures a CSV Reader with a path, delimiter, header presence, and a filter requiring `evidence_score >= 0.8`; a KGX Writer with output paths and a Biolink vocabulary URI; and a transform module/function binding for a gene–disease association ingest [2509.09096]. The paper’s example makes clear that YAML carries both data-access parameters and high-level transform parameters, while the Python transform expresses the source-to-schema mapping.

In the corresponding Python transform, a record is mapped to a `Node` for a gene, a `Node` for a disease, and an `Edge` with predicate `biolink:gene_associated_with_condition`, with `confidence` stored as a property. The transform uses `kt.write` to emit each object. The paper states that the YAML reader parameters ensure only records satisfying `evidence_score ≥ 0.8` reach the transform, while the writer parameters guarantee KGX-compliant TSV output [2509.09096].

The transformation model is further specified through primitive operations. Filtering is defined by a predicate $p(r): \text{record} \to \{\text{true}, \text{false}\}$, with records failing the predicate dropped before transform. Node creation is formalized as a function producing an object $n=(id, category, name, properties)$, where the `id`, `name`, and property values are computed from the input record and `category` is a Biolink entity type. Edge creation is analogously defined as $e=(subject, predicate, object, properties)$ with subject and object identifiers and a Biolink relation. Arbitrary record fields may be promoted to edge qualifiers or node properties by passing a dictionary to the constructor, and stateful aggregation is supported through `kt.state`, for example when grouping synonyms across records [2509.09096].

These semantics matter because they make Koza’s abstraction layer intentionally small. Rather than supplying a large domain-specific language, the package exposes a concise set of primitives whose outputs are already aligned with Biolink data structures. This suggests that extensibility is achieved through Python transforms and YAML parameterization rather than through a separate transformation calculus.

## 4. Schema compliance and interoperability guarantees

A defining feature of Koza is that schema compliance is enforced during serialization. The Writer fetches the Biolink JSON-LD and builds an in-memory schema of allowed categories, predicates, and required fields. Before writing each line, it validates that `Node.category` belongs to the allowed categories, that `Edge.predicate` belongs to the allowed relations, and that `Edge.subject` and `Edge.object` are non-empty and well-formed CURIEs [2509.09096].

The paper also states formal constraints for edges: for every edge $e$, $\mathrm{subject}(e) \in I$, $\mathrm{object}(e) \in I$, and $\mathrm{predicate}(e) \in R$, where $I$ is the set of valid CURIE identifiers in Biolink and $R$ is the set of predicates [2509.09096]. This places interoperability on a concrete syntactic and semantic basis, rather than leaving it as an informal design goal.

On validation failure, the offending object is logged with record context through `kt.log.error`. By default, Koza halts with a `SchemaValidationError`, although the behavior can be configured to skip invalid records. This mechanism is consequential for reproducibility: invalid outputs are not silently admitted into the graph unless the ingest author explicitly chooses a permissive mode.

The phrase “born interoperable” can be misunderstood as implying that downstream integration problems disappear automatically. The description of Koza does not support that stronger claim. What it does support is that outputs conform to KGX and the Biolink Model at production time [2509.09096]. A plausible implication is that Koza reduces a major class of interoperability failures—format and schema incompatibility—even though cross-source deduplication, qualifier reconciliation, and broader harmonization may still be required elsewhere in the pipeline.

## 5. Representative workflows and graph assembly

The paper describes a ClinGen pathogenic variant ingest as a concrete workflow. The raw source is a ClinGen JSON release containing sequence variants with associated gene and disease mappings. A `JSONReader` is configured with the path to the ClinGen JSON and a `json_path` targeting the “pathogenicity” array. Records are filtered so that only those with `classification == "Pathogenic"` are processed. The transform creates a `Biolink:SequenceVariant` node with identifier `ClinGenVar:<variant_id>`, looks up the associated gene as a `Biolink:Gene`, and emits edges from the sequence variant to the gene using `biolink:has_variant_location` and from the sequence variant to the disease using `biolink:related_to`. The `KGXWriter` then outputs `nodes.tsv` and `edges.tsv`, yielding a monthly refreshed KGX fragment of ClinGen pathogenic variants that is directly mergeable into Monarch KG [2509.09096].

A second workflow, the Model Organism KG (MOKG), illustrates multi-source assembly. The paper specifies nine modular Koza ingests, including HGNC, HPOA, PANTHER, PomBase, Dictybase, Alliance, and NCBI Gene. These outputs are concatenated with the `koza-merge` CLI into a unified KGX representation, after which global deduplication and qualifier merging are performed across sources. The result is described as a specialized Neo4j database for model organism research, updated monthly [2509.09096].

These examples clarify the intended scope of Koza. It is not limited to one-off conversion of a single source into an exchange format; it also supports recurrent production of modular graph fragments and their subsequent composition. The ClinGen example emphasizes source-specific semantic mapping under schema enforcement, while the MOKG example emphasizes compositionality across independently maintained ingests.

## 6. Operational characteristics, scaling behavior, and constraints

Koza’s performance model is explicitly tied to streaming execution. The paper states that the streaming design keeps memory usage to $O(1)$ per record, with the exception that the Koza state dictionary grows if stateful transforms use it. Typical runtimes are reported as less than five minutes on a single CPU core for small CSV or JSON sources below 100 MB, and 30–45 minutes for medium datasets of several GB. Average throughput is reported as 50–100 K records per second for simple node/edge transforms on modern hardware [2509.09096].

Scalability is described as source- or module-level parallelism. Ingest jobs are said to be trivially parallelizable by source/module, and Koza-Hub runs 30 CI jobs in parallel via GitHub Actions. A single ingest typically uses less than 2 GB RAM and can run on a standard 4-core virtual machine [2509.09096]. These details position Koza as a framework optimized for many moderately sized, independently executable ingests rather than for monolithic, tightly coupled distributed graph ETL.

The paper also identifies concrete limitations. GitHub Artifact size limits of 2 GB cap the size of published KGX modules, motivating migration of large ingests to alternative object stores such as AWS S3 or Zenodo. Real-time updates are not native: modular ingests must be re-run to reflect source changes, with incremental or webhook-triggered updates identified as future work. Schema evolution in Biolink may break existing ingests, leading to a plan for version-pinned Biolink releases and compatibility shims. Very large datasets, exemplified by gnomAD at 1.6 TB, exceed typical compute capacity; proposed directions include collaboration with providers for native KGX exports or integration of a distributed ingestion framework such as Spark [2509.09096].

The planned enhancement of the primitive set—adding joins, lookups, and near-duplicate merging primitives—indicates that the current abstraction deliberately prioritizes a minimal core. This suggests a tension, common in ETL systems, between keeping the primitive API simple and accommodating increasingly complex transformation logic without bespoke code.

## 7. Position within biomedical knowledge graph engineering

Koza is situated in the paper as a response to redundant labor in biomedical knowledge graph construction, with those redundancies attributed to the lack of data standards and the scarcity of “knowledge-graph ready” source data [2509.09096]. Its answer is not a new graph schema, but a workflow formalization: raw biomedical information is ingested through YAML-configured readers, transformed through explicit Python hooks using a small primitive set, and emitted as KGX under Biolink validation.

Within biomedical knowledge graph engineering, its significance lies in the combination of modular source adapters, explicit and versioned transformation logic, schema-constrained output, and routine artifact publication through the Koza-Hub. These characteristics support rapid, reproducible, and interoperable knowledge graph builds and make it straightforward to merge source-specific outputs into custom graph products [2509.09096].

A plausible implication is that Koza serves as infrastructure for institutionalizing ingest maintenance as a shared, testable, and automatable practice rather than as a collection of isolated conversion scripts. In that sense, its main contribution is methodological as much as technical: it turns knowledge graph ingest into a standardized software engineering problem centered on reusable modules, declarative configuration, and validation against a common semantic model.

Source: https://www.emergentmind.com/topics/koza