Gen3: Open-Source Data Commons Platform

• Gen3 is an open-source data platform that builds and operates data commons by curating, harmonizing, and exposing research data through FAIR APIs and interactive portals.
• It integrates analysis and visualization tools with automated data surface generation to support secure, standards-based access and community governance.
• Its flexible, cloud-agnostic architecture and narrow middle design simplify adding new data sources and analytics applications over heterogeneous datasets.

Gen3 is an open-source data platform for building and operating data commons, defined as cloud-based platforms that manage, analyze, and share data for a research community. In this usage, a data commons is more than a passive repository: it curates and harmonizes submitted data to a defined data model, integrates analysis and visualization tools, and exposes FAIR interfaces for both interactive and programmatic use. Gen3 has been used to develop and operate over 15 data commons worldwide and, in aggregate, these systems make over 28 PB of data and more than 64 million FAIR data objects available to the research community, including information on over 2 million research subjects (Barnes et al., 7 Aug 2025).

1. Definition, scope, and deployment context

Gen3 is designed to lower the barriers to sharing and re-use of research data at scale, to support integrated analytics over heterogeneous data, and to provide governance and security controls tailored to each community. Its intended domain is community-scale research infrastructure rather than a single laboratory repository or a narrow-purpose application. The platform manages structured, semi-structured, and unstructured data, while coupling curation and harmonization to a formal schema (Barnes et al., 7 Aug 2025).

Its deployments include components of the NIH NCI Cancer Research Data Commons (CRDC), the NHLBI BioData Catalyst platform, the NIBIB MIDRC imaging commons, the Kids First Data Resource Center, the Veterans Precision Oncology Data Commons, BloodPAC for liquid biopsy data, CHORDS, and the Pandemic Response Commons, which pivoted to Long COVID research. The Center for Translational Data Science (CTDS, University of Chicago) itself operates ten data commons using Gen3 (Barnes et al., 7 Aug 2025).

The term “Gen3” is not unique across technical literature. It also appears in unrelated contexts such as Kinova robotic platforms, RFSoC generations, and other domain-specific systems (Chen et al., 2022, Redondo et al., 2023). In research data infrastructure, however, “Gen3” denotes the data commons platform described in “Managing, Analyzing and Sharing Research Data with Gen3 Data Commons” (Barnes et al., 7 Aug 2025).

2. Data model and autogenerated platform surfaces

A Gen3 data commons begins with definition of a data model. Administrators specify a graph data model in YAML using a dictionary approach: nodes represent entities such as patient, diagnosis, biospecimen, sequencing experiment, clinical visit, and file; edges represent relationships such as sample derived-from patient; and properties capture attributes with types, value constraints, and required flags. Many deployments begin from a basic dictionary and adapt it to local requirements (Barnes et al., 7 Aug 2025).

This graph-first specification is operational rather than merely documentary. Once the data model is defined, Gen3 automatically generates three major surfaces: a data portal for searching and exploring structured data, a submission portal for ingesting structured data and registering data objects, and FAIR APIs that expose the modeled data and objects programmatically through REST and GraphQL (Barnes et al., 7 Aug 2025).

The search and exploration portal includes an Exploration page with faceted search and cohort building and a Discovery page for dataset-level metadata. The submission portal supports ingest of structured records and registration of data objects. The FAIR APIs expose both metadata and data objects for downstream automation and integration (Barnes et al., 7 Aug 2025).

A significant consequence of this design is that the platform evolves with the schema. Updating the data model triggers regeneration of the APIs and portals, which keeps the interface layer aligned with evolving curation and data governance requirements. This suggests a relatively tight coupling between community ontology work and application-layer behavior, with less manual reimplementation than in platforms where schemas and interfaces evolve independently (Barnes et al., 7 Aug 2025).

3. Architecture and core services

Gen3 follows a “narrow middle” architecture, described as an end-to-end pattern in which a small set of data mesh services provides the core APIs while data producers and consumers sit at the edges. The stated purpose of this design is to make it straightforward to add new data sources or analytics applications over time without altering the foundation (Barnes et al., 7 Aug 2025).

The core services are standards-based. For identity and access, Gen3 uses OpenID Connect and OAuth 2.0 for authentication and authorization, together with a policy engine for fine-grained access control to registered and controlled-access or sensitive data. For persistent identifiers and indexing, it uses GA4GH DRS-compliant persistent digital IDs; indexing services mint GUIDs and record essential object attributes such as file size, name, md5 hash, and storage location. These identifiers are opaque and decoupled from physical location, which eases data migration (Barnes et al., 7 Aug 2025).

Metadata management is split by modality. A Metadata Service stores semi-structured JSON metadata, including dataset descriptors and public sample metadata, while structured submissions are validated against the YAML-defined graph data model and written to a Postgres-backed graph database. Invalid submissions return detailed errors pointing to mismatches such as wrong type. For exploration, the Tube microservice performs ETL from Postgres to Elasticsearch, enabling rapid faceted search in the Exploration page (Barnes et al., 7 Aug 2025).

The platform also incorporates workflow and compute integration. Analysis Workspaces integrate with the commons and other data resources, and workflows can be orchestrated via a Workflow Execution API, with Nextflow given as an example. Workspaces provide persistent, user-specific storage and environments such as JupyterLab or RStudio (Barnes et al., 7 Aug 2025).

Deployment is containerized and cloud-agnostic. Services run on Kubernetes, with autoscaling via Karpenter; Helm charts streamline deployment, and Terraform may be used for cloud infrastructure management. The platform is described as cloud-agnostic across AWS, GCP, Azure, and OpenStack (Barnes et al., 7 Aug 2025).

4. FAIR interfaces, querying, and data lifecycle

Gen3 is explicitly organized around FAIR support. Findability is provided through persistent GA4GH DRS IDs, search portals powered by Elasticsearch and the Metadata Service, and clear data dictionaries and graph schema that are browsable via a Dictionary Viewer. Accessibility is implemented through standards-based authentication and authorization, controlled access via the policy engine, and APIs that expose structured and object metadata. Interoperability is pursued through standards-compliant identifiers and APIs, support for bulk-FHIR and self-describing export formats such as PFB/Avro, and compatibility of the graph data model with paradigms including OMOP, relational, and FHIR. Reusability is supported by curated and harmonized data aligned to explicit schemas, exportable and versionable structured datasets with persistent IDs, provenance captured via graph relationships, and community-tail