OpenHealth Lake: Federated Data Lakehouse

• OpenHealth Lake is a federated data management platform for health and bioinformatics that combines flexible lakehouse storage with strong governance and access control.
• The platform integrates open-source tools, metadata extraction, version tracking, and supports multiple storage backends like Google Cloud Storage, Amazon S3, and Apache HDFS.
• Its multi-interface design—via a website, REST API, and Python/R libraries—enables scalable, FAIR-compliant data management for collaborative research.

OpenHealth Lake is a prototype data management platform for health and bioinformatics applications built around a data lakehouse architecture, combined with data federation and guided by the FAIR principles. Implemented with open-source tools and exposed through a website, a REST API, and client libraries for Python and R, it is intended for organisations that generate and share large, heterogeneous, decentralized datasets, including structured and unstructured research data, while preserving governance, access control, and flexibility in where data is stored. The motivating use case was the INFORM Africa research hub, a multinational health collaboration focused on managing health-related research data from Nigeria and South Africa (Silva et al., 19 May 2026).

1. Conceptual position

OpenHealth Lake occupies an intermediate position between a raw data lake, a governed lakehouse, and a federated biomedical platform. In the broader platform literature, a data lake is defined as “a system for storing data as objects, where the objects have an associated GUID and (object) metadata, but there is no data model for interpreting the data within the object,” whereas data commons and data ecosystems add shared software services, governance, APIs, harmonization workflows, and interoperability across resources (Grossman, 2018). OpenHealth Lake follows the lakehouse direction by combining flexible heterogeneous storage with metadata-driven organization, and it follows the federated direction by connecting multiple external storage systems rather than forcing all data into one centralized repository (Silva et al., 19 May 2026).

This places it near a family of health-data infrastructures that separate logical access from physical custody. The HEAL Data Platform, for example, is a cloud-based federated system that provides a single point of search, discovery, and analysis across connected repositories while preserving repository autonomy over the actual data and approvals (Larrick et al., 19 Dec 2025). By contrast, OpenHealth Lake is not to be conflated with OpenHealth, the wearable health monitoring platform whose emphasis is device/platform standardization, local processing, extensibility, and community-driven interoperability rather than a centralized storage fabric or data lake architecture (Bhat et al., 2019).

2. Design goals and requirements

The platform was developed in response to several recurring problems in health sciences and bioinformatics. Health and bioinformatics workflows now produce very large and varied datasets, especially with technologies such as Next Generation Sequencing, while collaborative global health initiatives require secure storage and sharing, data governance, access control, and centralized access over distributed holdings. The authors argue that traditional federated databases may not scale well to the large, heterogeneous, and unstructured data common in bioinformatics, and that conventional data lakes, warehouses, or simple NoSQL systems do not fully combine flexible raw-data storage, structured indexing and discovery, analytics-readiness, governance and access control, and portability across storage environments (Silva et al., 19 May 2026).

From these premises, OpenHealth Lake was designed to support both functional and non-functional goals. Its core functions include user account management, authentication, uploading and downloading files, storage of raw and processed data, grouping files into logical repositories called collections, metadata extraction and indexing, catalogue browsing and file search, access-request workflows for restricted data, dataset access control, version tracking of uploaded files, and connection to multiple external storage environments. Its non-functional objectives are flexibility, scalability, security, governance, adaptability, portability, reproducibility, usability, and interoperability. The assumption throughout is a multi-user, inter-institutional research environment in which data may remain in distributed storage, governance constraints may differ across organisations, and some users will require a website rather than a code-first interface (Silva et al., 19 May 2026).

A plausible implication is that OpenHealth Lake is oriented less toward a single institutional warehouse and more toward a configurable coordination layer for collaborative research infrastructures.

3. Architecture and implementation

OpenHealth Lake is organized into three layers: a data ingestion layer, a data storage layer, and a data governance layer. The ingestion and governance layers are implemented through a custom REST API, while the storage layer consists of external file-storage back ends. In the main architecture, the API sits between an application database and the file storage layer: the database stores metadata, credentials, indexes, permissions, and application-management data, while the storage layer holds the actual files. This is the central lakehouse move in the system: raw and processed files remain in flexible storage, while structured metadata and governance functions are maintained separately (Silva et al., 19 May 2026).

The storage layer is intentionally data-agnostic. It stores raw and processed data, accepts data without mandatory preprocessing or ETL pipelines, and can handle both structured and unstructured files. Federation appears in the platform’s ability to connect to multiple external storage systems, specifically Google Cloud Storage, Amazon S3, and Apache HDFS. The implementation therefore uses a hybrid approach that supports cloud-based and self-hosted storage and allows organisations to combine storage credentials and environments under a single logical management plane (Silva et al., 19 May 2026).

The application database is implemented with Couchbase. It stores user credentials, encrypted storage access keys, dataset access requests, user permissions, collection metadata, file metadata, and the indexes that together form the data catalogue. The API is built with FastAPI and is itself structured into three layers: a routes layer for endpoints, a services layer for business logic, and a repository layer for database connection and CRUD operations. The implementation also uses the Builder pattern for database query construction and the Facade pattern to manage interactions between endpoints and lower-level components. Containerized deployment is provided through Docker, and the public interfaces are complemented by Python and R packages that wrap the API for common analytical environments (Silva et al., 19 May 2026).

4. Collections, metadata, governance, and FAIR

The basic organizational and governance unit in OpenHealth Lake is the collection. A collection is a logical repository grouping one or more files, analogous to a Git repository for code or data; every file uploaded to the platform must belong to a collection. The collection index stores metadata about collections, while the file index stores metadata extracted from uploaded files. The paper does not enumerate a full metadata schema, but examples include file name, file collection, file category, bucket location, and version information. Together, these indexes constitute the data catalogue and support collection browsing, file search, locating versions, and downloading files (Silva et al., 19 May 2026).

To support provenance and avoid silent overwriting, the upload process includes automated version control. If an uploaded file matches an existing file according to metadata such as file name, collection, category, and bucket location, the system creates a new versioned path rather than overwriting the old one, with examples such as collection_name/v2/file_name and collection_name/v3/file_name. Users may also assign versions manually when uploading externally processed or independently versioned data. Access control is handled separately from login authentication: OpenHealth Lake uses a passport strategy integrated with the GA4GH Passport Broker, and when a collection is created a corresponding visa is generated; collection owners can then grant or revoke visas to users. Security features include password encryption, encrypted storage of cloud bucket and HDFS access keys in Couchbase, and Fernet symmetric encryption for external storage credentials (Silva et al., 19 May 2026).

The platform’s FAIR alignment is implemented operationally rather than semantically. Findability is supported through the data catalogue, metadata extraction, browsing, search, and version records. Accessibility is provided through the website, REST API, Python package, and R package, with controlled rather than open-by-default access. Interoperability is supported by open REST interfaces, language bindings, support for multiple storage back ends, containerized deployment, and integration with GA4GH Passport Broker. Reusability is supported through open-source implementation, Dockerized deployment, structured metadata, provenance-aware versioning, and storage portability. At the same time, the platform does not report use of specific ontologies or community health-data standards. This contrasts with semantic integration architectures that use FHIR-compliant semantic web based data templates or combine FHIR and RDF to make disparate healthcare data queryable and susceptible to automated inference (Seneviratne et al., 2 Mar 2026); (Cotter et al., 2019). This suggests that OpenHealth Lake’s current strength lies in governance and deployment portability rather than semantic harmonization.

5. Interfaces, workflows, and usability evaluation

OpenHealth Lake was designed to be used through multiple interfaces. The website supports direct human-facing interaction and was particularly used for the access-request workflow. The REST API exposes all major platform operations to authenticated external applications. The Python and R packages encapsulate the same functionality for researchers who prefer scripted access. This multi-interface design reflects the target user population, which includes both computational specialists and health researchers with limited software engineering expertise (Silva et al., 19 May 2026).

The evaluation was a usability study focused on the platform’s main functions as exposed through the Python and R libraries, plus one website-based task. The study had four components: an introductory presentation, informed consent, a task-based user study with 11 tasks plus setup and installation, and a final feedback form. It received ethics approval from the Stellenbosch University REC: SBE ethics committee under project ID 30945. Thirty-one participants took part, 26 completed all tasks, and participants came from bioinformatics, microbiology, biology, computer science, and data science. The tested tasks included authentication, listing collections, listing files, searching files, fetching a dataframe, downloading a file, listing buckets, creating a collection, uploading a file, uploading a dataframe, and requesting data access (Silva et al., 19 May 2026).

The reported results indicate that the prototype was generally usable and useful. The overall dropout rate was 17%. Python was chosen by 14 participants and R by 15. Across all responses, most tasks were rated Easy or Very Easy: 89% Easy/Very Easy, 10% Neutral, and 2% Difficult. Task 11, the website-based access request task, had one of the lowest completion times and required no coding. Task 4, file search and especially advanced querying, emerged as the most difficult in practice when combining time and answer accuracy. Qualitative findings identified documentation clarity, advanced search/query usage, Windows path-format issues for R users, and confusion around upload parameters such as final_file_name as the main sources of friction rather than core functional failure (Silva et al., 19 May 2026).

6. Relation to earlier healthcare data-lake architectures and current limitations

OpenHealth Lake inherits a substantial architectural lineage from earlier healthcare big-data systems. Hadoop-based healthcare data lake proposals emphasized native-format HDFS storage, schema-on-read, removal of data silos, and layered architectures for ingestion, governance, security, and analytics; one such system reported that loading and storing data in the data lake takes “nearly 50% less” time than in a traditional data warehouse and reported lower Euclidean-distance values for patient clusters across four clusters than the warehouse baseline (Rangarajan et al., 2018). Other open-source healthcare platforms combined HDFS, Spark, Hive, HBase, Ceph, Kubernetes, and JupyterHub to support heterogeneous health data and notebook-centric reproducible analytics at scale, including evaluation with a clinical dataset of 69M patients (Rouzbeh et al., 2020). Domain-specific repositories for waveform and vital-sign data have likewise adopted object storage plus metadata indexing, nightly batch extraction, de-identification, and linkage to OMOP and ancillary data types such as EEG, radiology, and video monitoring (Malunjkar et al., 2021).

Against that background, OpenHealth Lake is comparatively stronger in user-facing governance and cross-back-end portability than in semantic standardization or domain-specific data modeling. It does not claim conformance to a particular domain metadata standard or ontology for health-data content, and its current limitations are explicit: version control is internal and relatively simple; storage-back-end support is restricted to Google Cloud Storage, Amazon S3, and Apache HDFS; the upload process can create “ghost” file entries if upload completion fails; permissions operate at the collection level rather than the file level; and documentation and examples require improvement, especially for non-technical users and advanced search functions (Silva et al., 19 May 2026).

In this sense, OpenHealth Lake is best understood as a prototype federated lakehouse platform for health data management rather than a complete semantic interoperability stack or a finished production platform. Its central contribution is the demonstration of a viable, adaptable, open-source architecture for securely storing, cataloguing, governing, and sharing large heterogeneous datasets across collaborating organisations without forcing all data into one centralized system (Silva et al., 19 May 2026).