---
title: 'EGI FedCloud: Federated Research Cloud Platform'
url: https://www.emergentmind.com/topics/egi-fedcloud-platform
type: topic
---

# EGI FedCloud: Federated Research Cloud Platform

The EGI FedCloud Platform is a pan-European federated infrastructure aggregating heterogeneous cloud resources—including OpenStack and OpenNebula IaaS sites—under a consolidated access and authentication framework tailored for research data and compute workloads. Developed through collaborative efforts such as the European Grid Infrastructure, INDIGO-DataCloud, and ESCAPE projects, EGI FedCloud enables elastic, SLA-backed resource provisioning, cross-site orchestration, and federated identity management supporting diverse science communities. Its architecture harmonizes IaaS, PaaS, and microservices-based science gateways, exposing common interfaces for workload portability and FAIR-compliant data management across distributed, multi-institutional sites [2301.09978][1911.06560][1711.01981][1712.02574][1806.04986].

## 1. Federation Architecture and Core Principles

EGI FedCloud operates as an aggregation layer spanning both public and private IaaS providers, each running OpenStack, OpenNebula, or other compatible cloud stacks. Site-local cloud fabrics (compute nodes, block/object storage, network fabrics) expose standardized APIs—principally OpenStack’s Keystone, Nova, Glance, Neutron, and provider-specific OCCI (Open Cloud Computing Interface)—to a central platform management layer [2301.09978][1711.01981].

The federation is coordinated via:
- **Authentication & Authorization Infrastructure (AAI):** Enforces user identity and group/VO membership via OpenID Connect (e.g., InDiGo-IAM, EGI Check-in), X.509 certificates, or VOMS proxies.
- **Resource Orchestration:** Brokering workloads across sites using dynamic discovery (Keystone catalog, OCCI endpoints), quotas, and site metadata (flavors, images, network mappings).
- **Service Exposure:** Common REST/CLI endpoints enable the instantiation and lifecycle management of virtual machines (VMs), block devices, and containers.
- **Data Federation:** Onedata and VOSpace interfaces unify storage access, ensuring consistent POSIX, HTTP, and IVOA-compliant data abstraction [1806.04986][1712.02574].

A typical component and data flow (per [2301.09978]):

```latex
\begin{tikzpicture}[node distance=1.2cm,auto,>=latex']
\node[draw,rectangle] (User) {ESAP User};
\node[draw,rectangle,right=of User] (ESAPFE) {ESAP Front-end/API};
\node[draw,rectangle,below=of ESAPFE] (Auth) {InDiGo-IAM (OIDC)};
\node[draw,rectangle,right=of ESAPFE] (Prov) {ESAP Provisioning Service};
\node[draw,rectangle,right=of Prov] (FedClient) {fedcloudclient};
\node[draw,rectangle,below=of FedClient] (EGI_IAM) {EGI Check-in (OIDC)};
\node[draw,rectangle,right=of FedClient] (OSAPI) {OpenStack APIs};
\node[draw,rectangle,right=of OSAPI] (EGI_Cloud) {EGI FedCloud Site};
% Detailed edge list omitted for brevity
\end{tikzpicture}
```

## 2. Authentication, Authorization, and Identity Federation

The AAI subsystem permits seamless cross-site access while supporting legacy and modern protocols:
- **OpenID Connect (OIDC):** Used in ESCAPE/ESAP and by INDIGO IAM for portal-to-cloud trust, federated via eduGAIN, Shibboleth, and campus IdPs. JWTs are the principal credential format exchanged via OIDC code flow, with group and role claims managed by IAM/Check-in brokers [2301.09978][1711.01981].
- **VOMS/X.509 PKIX:** For grid-oriented workloads, each user holds an IGTF-certified X.509 certificate, optionally augmented with VOMS attributes (VO, group, role). The OpenStack Keystone-VOMS extension enables VO-based authorization and token issuance, mapping identities into cloud tenants [1806.04986][1712.02574].
- **Token Translation (WaTTS, motley-cue):** For services requiring other credential types (e.g., SSH keys, Kerberos), OIDC or VOMS tokens are programmatically mapped to appropriate local credentials [2301.09978][1711.01981].

Policy enforcement relies on group/VO membership (usually group-based for resource ACLs in, e.g., VOSpace), with persistence handled via LDAP and relational DBs. Fine-grained access is possible at the data node (file, container) or infrastructure level [1712.02574]. The incorporation of token-based and certificate-based credentialing supports both modern portal-centric science gateways and more traditional batch/grid workflows.

## 3. Resource Provisioning, Orchestration, and Automation

VM, container, and storage lifecycle operations are mediated by site-local and federated clients:
- **fedcloudclient:** Python-based wrapper around standard OpenStack CLI, orchestrating sequence of API invocations—image/flavor discovery, server instantiation, floating IP association—backed by OIDC or VOMS tokens [2301.09978].
- **Orchestrator/Infrastructure Manager (IM):** TOSCA-template driven federated deployment of complex, multi-site workflows, optionally leveraging Cloud Provider Rankers for SLA-aware brokering and Mesos/Marathon for containerized tasks [1711.01981].
- **OCCI and rOCCI-cli:** Front endpoints for VM/block storage via OCCI-compliant orchestration handlers [1911.06560].

Provisioning performance is summarized as follows:
| Operation           | Typical Latency (CESGA)         |
|---------------------|---------------------------------|
| Keystone token+catalog | 0.5–1 s                     |
| Image/flavor listing   | 0.2–0.5 s per API call       |
| VM creation (boot/network/config) | 60–90 s         |

A generalized provisioning latency equation:
\[
T_{\mathrm{prov}} \approx T_{\mathrm{auth}} + N_{\mathrm{list}} T_{\mathrm{list}} + T_{\mathrm{create}} + T_{\mathrm{config}}
\]
where $T_{\text{config}}$ is cloud-init/post-installation time [2301.09978].

## 4. Data Management and Interoperability Mechanisms

EGI FedCloud enables distributed, FAIR-compliant storage and data movement:
- **VOSpace 2.1:** Exposes IVOA-standard node-based storage model (nodes, containers, data nodes, link nodes) over RESTful APIs. Metadata abstraction, fine-grained ACLs, and object/byte-level transfer are supported via HTTP(S), with DataNodes mapped to POSIX or Swift/CDMI backends [1806.04986][1712.02574].
- **Onedata:** Provides global, POSIX-like federated filesystem abstraction over disparate sites; supports caching, QoS profiles, and multi-GB/s throughput for scientific workflows [1711.01981].
- **CernVM-FS (CVMFS):** For software distribution within VMs, CVMFS mounts large, versioned runtime environments (e.g., SAS/HEAsoft) as read-only POSIX filesystems, minimizing image size and accelerating contextualization [1911.06560].
- **Cross-site transfers:** Credential delegation (IVOA CDP protocol) enables remote service calls under a single federated identity, facilitating access to foreign VOSpace resources and inter-cloud data migration.

Typical storage throughput reaches $R_{\mathrm{IO}} \approx 20$–$50$ MB/s per VM (scratch disks), with pilot-scale cross-atlantic FITS data transfers reported at $\sim$80 MB/s [1911.06560][1712.02574].

## 5. Limitations, Challenges, and Proven Solutions

Key challenges and adopted workarounds include:
- **Network Naming and Discovery:** Site-specific public network names (“public”, “ext-net”, etc.) complicate automation. Solutions involve site metadata collectors running periodic network enumeration (e.g. `openstack network list`) and maintaining a catalog of mappings [2301.09978].
- **VO Support and Site Capability Discovery:** Lack of an API to enumerate which sites accept a given VO is mitigated by parsing and caching site lists from fedcloudclient and tagging VO support [2301.09978].
- **Flavor/Image Metadata Ambiguity:** Absence of flavor/image naming conventions is addressed by programmatic extraction and caching of CPU, RAM, and disk properties using OpenStack show calls, publishing this metadata for workflow configuration [2301.09978].
- **Authentication Friction:** Inconsistent AAI systems (InDiGo IAM vs. EGI Check-in) and per-site SSH credentialing are unified through federated OIDC trust brokers and services such as motley-cue for user mapping [2301.09978].
- **Certificate and Credential Management:** Heavy reliance on X.509 and VOMS creates a usability barrier; proposed remediation involves adoption of OAuth2/OIDC-based identity federation and increased use of token translation [1806.04986][1712.02574].

Operational best practices include deploying microservice-based integration layers (as in PortalTS and EXTraS), round-robin site selection for load distribution, strict monitoring, and embracing auto-scaling triggers based on job queue depth and site utilization [1911.06560].

## 6. Science Gateways, Use Cases, and Performance Observations

Science gateways such as ESAP (ESCAPE), EXTraS, and CANFAR-OATs-INAF leverage the EGI FedCloud Platform to deliver domain-specific analysis as-a-service on distributed data:
- **ESAP (ESCAPE):** API gateway and portal for brokering EOSC-linked datasets, leveraging Django/Python microservices and fedcloudclient for VM/container instantiation [2301.09978].
- **EXTraS:** Microservice-driven portal (PortalTS) for astrophysical transient analysis, using OCCI CLI for multi-site job distribution, CVMFS for on-demand software layers, and per-job contextualized VMs via cloud-init [1911.06560].
- **CANFAR/OATs-INAF:** Federated deployment for astronomical data analysis, combining IVOA-compliant services, OpenStack VMs with VOSpace clients, and transcontinental data access for co-located storage and compute [1712.02574][1806.04986].

Performance metrics (where reported) demonstrate high throughput in terms of aggregate CPU time (e.g., $15\,795$ h), efficient SLA-backed vCPU and storage quotas, and efficient job scheduling paradigms (e.g., round-robin, fair-share, preemptible instances) [1911.06560][1711.01981].

## 7. Roadmap, Recommendations, and Future Directions

Adopted and proposed enhancements:
- **Standardize site metadata and naming conventions (e.g., “public-net” network naming, flavors.json catalogs).**
- **Federate InDiGo-IAM and EGI Check-in at broker level; map group/role claims via JWT for seamless AAI.**
- **Develop VO → site support discovery endpoints in fedcloudclient for programmatic capability matching.**
- **Deploy site metadata collector microservices for continuous inventory of site resources, capabilities, and VO support.**
- **Instrument workflow for detailed provisioning/operation metrics, supporting future SLA guarantees; establish end-to-end monitoring with dashboards (e.g., Grafana/Prometheus).**
- **Integrate container and notebook platforms (JupyterHub, Kubernetes, Docker) for interactive workloads.**
- **Pursue transition to token-based credentialing (OAuth2/OIDC) for user-friendly, scalable identity and access control [2301.09978][1712.02574][1711.01981].**

These directions are fundamental for advancing EGI FedCloud as a robust, federated PaaS for high-throughput, data-intensive scientific research environments, facilitating transparent, policy-driven and programmable compute/data workflows across an expanding set of European and international sites.

Source: https://www.emergentmind.com/topics/egi-fedcloud-platform