OpenHealth Lake: Designing and testing a data lakehouse platform for health applications

Abstract: Data management can be a complex challenge in fields such as bioinformatics and health sciences, which continuously generate extensive heterogeneous datasets. In the context of collaborative global health initiatives, secure storage and sharing of data are crucial to support impactful research. However, the absence of a unified data management platform complicates efficient data exchange and governance within these initiatives. In this paper, we introduce the design process of OpenHealth Lake, a data management prototype platform based on a data lakehouse architecture, data federation, and the FAIR principles. The platform is designed using open-source tools, guided by system requirements identified in previously published studies and complemented by insights from the existing literature. The current prototype platform comprises a user-friendly website, an open API, Python and R packages, allowing users to interact with the platform in multiple ways. Through a user study that included participants with varying technical backgrounds, we showed that our proposed data management prototype is both usable and useful. Our prototype design showcases the adaptability, scalability, and reproducibility of a lakehouse system that can be used by any organisation. It is designed as a flexible and complementary approach that allows organisations to customise data management systems to their specific requirements and resources, including cloud-based or self-hosted storage choices.

• The paper presents OpenHealth Lake, a novel platform that integrates heterogeneous health data using a lakehouse architecture and open-source solutions.
• The paper details a robust system design employing FastAPI, containerized deployment, hybrid cloud storage, and Couchbase for scalable data governance.
• The paper’s empirical evaluation shows high usability and adaptability across interdisciplinary users, while highlighting areas for enhanced documentation and advanced functionality.

OpenHealth Lake: Design and Empirical Assessment of a Lakehouse Platform for Health Data Management

Motivation and Background

The exponential increase of heterogeneous data in bioinformatics and health sciences, particularly fueled by advances such as Next Generation Sequencing, has overwhelmed traditional data management paradigms. Collaborative, cross-institutional health initiatives require not only scalable storage but also robust data governance, privacy, and provenance mechanisms—demands inadequately served by conventional data warehouses or single-paradigm data lakes. The data lakehouse architecture has emerged as a plausible alternative, synergizing the flexibility and scaling of data lakes with the structured access and governance features of data warehouses.

OpenHealth Lake is introduced as a prototype implementation of such a data lakehouse, explicitly targeting the needs of collaborative global health research consortia. The design is grounded in open-source solutions and incorporates contemporary best practices, including data federation and adherence to the FAIR (Findable, Accessible, Interoperable, Reusable) principles.

System Architecture and Implementation

OpenHealth Lake's technical architecture is organized into three principal layers: data ingestion, storage, and governance. This model enables federated, heterogeneous data management while accommodating evolving governance needs (Figure 1).

Figure 1: Lakehouse Infrastructure design, illustrating the application DB, REST API, and hybrid file storage layer (cloud/HDFS).

The core of the system is a FastAPI-based REST API, containerized with Docker for reproducible deployment. Application and metadata storage leverage Couchbase, a distributed non-relational database selected over MongoDB for multifactorial reasons: hybrid key-value/document support, built-in memory-first caching, intuitive clustering, and multidimensional scaling. File storage is implemented as a hybrid model spanning Google Cloud Storage, Amazon S3, and Hadoop HDFS, allowing data zone flexibility and supporting both cloud-based and self-hosted deployment models.

Logical data grouping via “collections” provides access control granularity and operational grouping for data governance. File metadata and collection indices facilitate data catalogue construction to enable efficient discovery and management. Credentials and access tokens are encrypted via Fernet symmetric encryption, mitigating risk in case of database compromise.

The API itself adheres to conventional separation-of-concerns: route handlers, service logic, and repository/Data Access Layer. Data access control is managed through a “passport” approach, directly integrating GA4GH Passport Broker services for visa-based permission assignment.

Figure 2: Simplified API data flow diagram. Highlights data validation, storage operations, and passport-based access controls across system endpoints.

Figure 3: API infrastructure diagram detailing service decomposition and mapping to core health data management functionalities.

Interoperability is further improved via dedicated Python and R client libraries, each exposing complete API-driven data ingestion, query, and management functions—enabling integration into analysis pipelines independent of user programming environment.

Empirical Usability Evaluation

An empirical study with 31 participants (from bioinformatics, computer science, biology, microbiology, and data science domains) was performed to assess OpenHealth Lake’s usability and utility. Participants spanned the spectrum from undergraduate to PhD, with the majority holding advanced degrees and >2 years’ experience.

Completion, engagement, and dropout rates suggest broad accessibility, with only a 17% dropout over 12 tasks. Task-by-task analysis reveals higher skip rates and perceived difficulty in upload and advanced search functionalities (especially Tasks 5, 9, and 10), mapping directly to increased argument complexity or less-familiar interaction paradigms.

Figure 4: Total participants vs number per task skipping; complexity/pitfalls increase skips for advanced features.

Usability scores—collected on a Likert scale—show 89% of tasks rated as "Easy" or "Very Easy." Notably, Python library users reported higher usability than R users (47% vs 41% in the most positive categories), a result likely attributable to both background and library interaction paradigms rather than significant functional differences between language bindings.

Figure 5: Usability ratings by task—majority cluster in positive categories; hardship isolated to select advanced operations.

Task completion time analytics reveal outliers primarily during early onboarding and advanced search/upload tasks, with prolonged documentation consultation a key factor. However, near-uniform correctness rates even for challenging tasks suggest that, despite initial friction, participants successfully adopted intended workflows.

Figure 6: Distributions of time spent per task across participant pool; advanced or unfamiliar tasks generate longer tails.

Accuracy metrics for responses confirm the majority of study participants were able to achieve correct outcomes even on advanced tasks, although advanced file search (Task 4) had elevated error rates—corroborating observed usability challenges.

Figure 7: Rate of correct responses per task; critical drop in advanced file search, with consistently high correctness elsewhere.

Usefulness ratings are overwhelmingly positive, reflecting recognition that the provided functionalities map to core operational requirements for researchers managing health data.

Figure 8: Usefulness ratings per functionality; strong positive skew for all core features.

Implications, Limitations, and Future Directions

The successful realization of OpenHealth Lake demonstrates the technical feasibility and user-level acceptance of lakehouse architectures for health data federation, even in interdisciplinary consortia with varied technical backgrounds. The empirical ablation between Python and R user experience surfaces the importance of both API design choices and user background in adoption, highlighting the need for enhanced documentation and onboarding pathways for less-technical users.

Practical implications extend to compliant, federated data integration across jurisdictions, with flexible storage backends and access models readily adaptable to diverse organizational contexts. Architecturally, the system positions itself for extensibility: support for additional low-cost storage backends (MinIO, Ceph) is straightforward, and integration of externalized data versioning tools is anticipated.

However, current limitations include only coarse-grained (collection-level) access controls and lack of proactive cleanup (“ghost” file) routines when uploads fail, potentially introducing catalogue inconsistencies. Granular file-level access management and event-driven integrity verification are suggested as concrete future enhancements. Moreover, advanced functionality designs (e.g., search, upload operations) should be revisited, particularly for users less familiar with programmatic data pipelines.

Theoretically, OpenHealth Lake’s design affirms the maturity of FAIR-aligned lakehouse paradigms in handling highly decentralized, heterogeneous data while preserving governance and analytic flexibility. Its empirical validation lays groundwork for evaluating more advanced governance and automation features.

Conclusion

OpenHealth Lake exemplifies a robust and adaptable open-source lakehouse platform tailored for heterogeneous, federated health data management. The design foregrounds flexibility, deployability, and adherence to best practices (data federation, FAIR principles, external API access). Measured empirical feedback supports the system’s usability and practical utility, with most core operations rated as easy and useful even across a technically diverse participant base.

Nonetheless, the study reveals actionable directions for refinement—improved documentation, expanded storage/access flexibility, and advanced governance tools. The prototype’s demonstrated scalability and interdisciplinary applicability indicate strong potential for further adoption in health and beyond, while also informing future architectural decisions in the evolution of open scientific data management platforms.