HEAL Data Platform Overview

• HEAL Data Platform is a cloud-based, federated data mesh that integrates heterogeneous biomedical datasets from the NIH HEAL Initiative using FAIR principles.
• It leverages open-source technologies like Gen3 and Pennsieve to manage metadata, secure access, and scalable analytic workflows.
• The platform supports high-throughput data ingestion, real-time cohort generation, and collaborative analysis with robust data governance and provenance tracking.

The HEAL Data Platform is a cloud-based, federated data mesh designed to serve as an integrated hub for search, discovery, secondary analysis, and sharing of data generated under the NIH Helping to End Addiction Long-term (HEAL) Initiative. It is implemented using open-source technologies, notably the Gen3 framework, and also operates in complementary configurations based on the Pennsieve platform for specific sub-initiatives. The system is engineered to enforce rigorous data governance, FAIR (Findable, Accessible, Interoperable, and Reusable) compliance, and scalable, secure workflow orchestration for multimodal biomedical data, including but not limited to clinical, imaging, genomics, and device-generated datasets (Larrick et al., 19 Dec 2025, Goldblum et al., 16 Sep 2024, Cohen et al., 2021).

1. Architectural Foundations and Federation Model

The HEAL Data Platform is fundamentally a mesh architecture, built atop the Gen3 framework, which orchestrates metadata and persistent identifier (PID) management across a heterogeneous landscape of NIH and third-party data repositories. The platform’s core comprises a suite of microservices, including:

• Authentication/Authorization Service (Fence): Implements OIDC/OAuth 2.0, supports SSO via NIH eRA Commons, Google, ORCID, and other providers, and issues JWTs with policy-scoped entitlements.
• Index Service (Indexd): Registers and resolves global PIDs for every dataset, file, and data object, enabling cross-repository linkage.
• Metadata Service (Sheepdog): Facilitates ingestion and query of structured study- and variable-level metadata through both REST and GraphQL APIs.
• Policy Engine (PXP): Governs fine-grained role-based access and audit policies.
• Web Interface (Windmill): Centralized portal for search, study registration, and spawning secure analytical workspaces.

This architecture enables the platform to stitch together metadata and access controls while federating data storage: source data remain in their respective repositories, each of which must minimally expose PIDs, metadata APIs, data APIs, and authorization endpoints (Larrick et al., 19 Dec 2025).

In parallel, the Pennsieve-based HEAL Data Platform instance underpins specific research programs (e.g., PRECISION Human Pain Network, RE-JOIN Initiative) and offers modular microservices for storage (backed by AWS S3), custom metadata graph models, API gateways, and workspaces, supporting both cloud and on-premises deployments (Goldblum et al., 16 Sep 2024).

2. Metadata Management and FAIR Data Principles

The central value proposition of the HEAL Data Platform is to maximize secondary use and reusability of HEAL-funded data by enforcing FAIR principles:

• Findable: Each study, dataset, and file receives a globally unique PID (Indexd for Gen3; DOI for Pennsieve) resolvable via open APIs; metadata catalogs index over 1,000 studies.
• Accessible: Open, queryable metadata APIs; data access via authenticated, policy-governed endpoints.
• Interoperable: Metadata schemas are JSON-based, mapped to domain CDEs, and harmonized across repositories; variable-level metadata follows Frictionless Data’s “JSON Table Schema” (Gen3) or custom model graphs (Pennsieve).
• Reusable: All metadata capture extensive provenance, licensing, and usage terms; integration with CEDAR assures ontological compatibility; curation workflows ensure peer-review of datasets (Larrick et al., 19 Dec 2025, Goldblum et al., 16 Sep 2024).

The metadata model encompasses:

• Study-Level Metadata (SLMD): Extends Dublin Core with fields for objectives, design, instruments, and outcomes.
• Variable-Level Metadata: Annotated with CDE tags; schemas are versioned, public, and promoted via GitHub (HEAL-metadata-schemas).

3. Data Ingestion and Processing Workflows

The HEAL Data Platform supports sustained, high-throughput multimodal data ingestion and robust curation:

• Pipeline Structure (Gen3-based): Ingestion controllers monitor incoming S3 buckets, emit message queue jobs, and dispatch ETL workers that parse and translate records atomic to the PostgreSQL warehouse. Manual uploads, EHR connectors, and device-stream captors are all supported.
• Performance Metrics: With six ETL workers, the platform sustains ~15,000 patient records/hour, each record reflecting high-frequency (e.g., 300 Hz ECG), unstructured notes, and structured clinical data. Processing efficiency is modeled as $R = W \cdot \mu$, with empirical worker throughput $\mu \approx 0.7~\text{records/sec}$.
• QA/QC Automation: Translation failures or outlier values route to a clinician-facing correction queue, after which corrected entries are reincorporated, flagged for audit compliance.

In Pennsieve, ingestion is structured as a three-step process—manifest generation, chunked file upload (with parallelism and verification), and manifest validation. Files up to 5 TB and aggregate parallel uploads are supported; throughput is governed by $T = D/t$ (bytes/time) (Goldblum et al., 16 Sep 2024, Cohen et al., 2021).

4. Search, Discovery, and Cohort Generation

The HEAL data portal acts as a single point of discovery, aggregating metadata from 19+ repositories (as of December 2025). Users can perform semantic and structured search by keyword, ontology, CDE, instrument, or study parameter.

• Cohort Generation (Gen3 flow): Researchers define high-level phenotyping queries, which are decomposed into parallelized subqueries against the metadata warehouse. Intermediate results are staged to S3 in Parquet format for efficient merging and statistical summarization.
• Complexity Models: For $N_{\mathrm{pat}}$ patients, $C$ cohort criteria, and $M$ workers:
  • Filtering is $O((N_{\mathrm{pat}} \cdot C)/M)$.
  • Final cohort time $$T_{\mathrm{cohort}} \approx \frac{\gamma}{M} N_{\mathrm{pat}} C$$, with $\gamma$ the per-row filter cost.
• Preliminary Analytics: Univariate/multivariate statistics are computed automatically; typical cohort creation for $10^5$ rows and $10^2$–$10^3$ variables ranges from 30 s to 2.5 min (Cohen et al., 2021).

Pennsieve’s platform further supports metadata graph traversals and multicriteria filtering, enabling patient-level, sample-level, and time-based queries using both REST and programmatic API endpoints (Goldblum et al., 16 Sep 2024).

5. Collaborative and Secure Analysis Environments

The platform provisions secure, cloud-based analytic workspaces:

• Gen3 Workspaces: On-demand launch of JupyterLab, RStudio, or Stata environments, all integrated with NIH STRIDES for cloud compute credits and with data egress limited by Kubernetes network policy to approved buckets.
• Auth Flow: Token-based authentication is persisted from portal to workspace and to all in-workspace data-access APIs.
• Collaboration: Real-time dataset versioning, optimistic concurrency control, fork/merge capabilities, audit trails, and annotation/commenting features all support collaborative data wrangling and analysis.
• Compliance: All writes/reads to data stores are logged to an immutable audit stream; encryption at rest (AES-256) and in transit (TLS ≥ 1.2) is enforced; each data object is tagged with IRB, user, and data-use metadata for RBAC enforcement (Larrick et al., 19 Dec 2025, Cohen et al., 2021).

Pennsieve workspaces extend this model by supporting on-premises Compute Nodes, versioned DOI publishing of datasets (including embargoes), and peer-reviewed curation workflows for public release (Goldblum et al., 16 Sep 2024).

6. Integration with HEAL Program Initiatives and Impact Metrics

The HEAL Data Platform serves >1,000 HEAL-funded studies, with hundreds of registered or enriched datasets spanning genomics, imaging, clinical trials, surveys, and real-world data. Platform metrics as of late 2025 include:

Metric	Count	Source
Searchable studies	1,078	(Larrick et al., 19 Dec 2025)
Registered studies	516	(Larrick et al., 19 Dec 2025)
SLMD-enhanced studies	398	(Larrick et al., 19 Dec 2025)
Variable-metadata studies	74	(Larrick et al., 19 Dec 2025)
Available datasets	118	(Larrick et al., 19 Dec 2025)
Connected repositories	19	(Larrick et al., 19 Dec 2025)
Public datasets (Pennsieve)	350+	(Goldblum et al., 16 Sep 2024)

Through its mesh federation, the platform enables cross-study meta-analyses, accelerates legal/technical onboarding of new repositories, and provides harmonized cloud analysis for both NIH and non-NIH investigators.

The Pennsieve-backed instantiation supports large collaborative consortia (e.g., SPARC, HEAL PRECISION Human Pain, HEAL RE-JOIN), hosting >125 TB data (as of 2024), with 35 TB public, and 80+ research groups leveraging private and shared cloud/on-premises workspaces (Goldblum et al., 16 Sep 2024).

7. Ongoing Developments and Limitations

Expansion priorities include further automation of repository onboarding pipelines and SIA/agreement generation, as well as the development of federated query layers (potentially leveraging GraphQL stitching) for rapid, cross-repository subset retrieval.

Major technical challenges persist in harmonizing metadata across diverse, evolving schemas and in standardizing APIs for programmatic, federated access. A plausible implication is that ongoing tooling for schema alignment and user-driven annotation is essential for maintaining interoperability and scaling secondary analyses.

Future enhancements under active development include advanced provenance (W3C PROV), lineage-tracking, and interactive user feedback on metadata quality. Cross-platform curation standards, robust audit trails, and immediate STRIDES integration further minimize barriers for secondary and collaborative research (Larrick et al., 19 Dec 2025, Goldblum et al., 16 Sep 2024).

---

References:

(Cohen et al., 2021): A Methodology for a Scalable, Collaborative, and Resource-Efficient Platform to Facilitate Healthcare AI Research (Larrick et al., 19 Dec 2025): The HEAL Data Platform (Goldblum et al., 16 Sep 2024): Pennsieve: A Collaborative Platform for Translational Neuroscience and Beyond