Rucio: Distributed Data Management Framework

• Rucio is an open-source scientific data management framework that enables organized, policy-driven handling of massive data across heterogeneous infrastructures.
• It abstracts data through unique identifiers (DIDs) and storage endpoints (RSEs), facilitating efficient data replication, transfer, and protection.
• Rucio integrates with diverse workflow and authorization systems, proving essential for experiments like ATLAS, LVK, and Rubin Observatory.

Searching arXiv for recent and foundational papers on Rucio to ground the article. Rucio is an open-source scientific data management framework that provides scientific collaborations with the functionality to organize, manage, and access their data at scale across heterogeneous data centers at widely distributed locations. It was originally developed to meet the requirements of the high-energy physics experiment ATLAS, but has been continuously extended to support the LHC experiments and other diverse scientific communities. Across the literature, Rucio appears not merely as a file catalog or transfer launcher, but as a policy-driven control plane for data organization, replication, protection, monitoring, and workflow-coupled data logistics in infrastructures ranging from WLCG to Virtual Research Environments, gravitational-wave computing, radio astronomy prototypes, dark-matter experiments, and Rubin Observatory data movement (Barisits et al., 2019, Gazzarrini et al., 2023, Bagnasco, 2023, Hernandez et al., 17 Jul 2025).

1. Origins, scope, and design intent

Rucio was developed to address a modern scientific data-management problem in which instruments and computing workflows generate enormous volumes of data while storage and compute resources are spread across many sites, institutions, and administrative domains. The original ATLAS requirements emphasized very large data volumes and file counts, globally distributed collaboration and storage, heterogeneous storage and network technologies, the need for autonomy and automation in data movement, support for detector data, simulation data, and user data, robust operation in the presence of failures, and integration with workflow systems such as PanDA and ProdSys. The system builds on lessons from DQ2 and expands on functionality, scalability, and robustness. A central design principle is that users express what they want, such as a dataset copied to a region or protected from deletion, while Rucio decides how to achieve it (Barisits et al., 2019).

In this formulation, Rucio is a Distributed Data Management system in the strong sense: it unifies data organization, data access, data distribution, data protection, dataflow automation, consistency and recovery, and monitoring and analytics. That breadth explains why later papers repeatedly describe it as a backbone or federation layer rather than as a narrow transfer service. The Virtual Research Environment paper presents Rucio as the software layer that makes distributed storage usable as a coherent, policy-driven service for scientific users; the LVK paper places it in the offline data distribution layer; and the Rubin paper uses it as the central system for inter-site file replication and transfer orchestration (Gazzarrini et al., 2023, Bagnasco, 2023, Hernandez et al., 17 Jul 2025).

A recurring misconception is that Rucio directly serves every data-access pattern in every scientific stack. The papers are more specific. In LVK, Rucio is part of the offline/high-latency distribution layer, while low-latency transfers are handled by igwn-lldd leveraging Apache Kafka. In Rubin, only the files that are meant to be replicated between sites are registered in a dedicated Rucio instance. In XENON1T, Rucio is reserved for the high-value, high-volume raw-data layer, while processed data and minitrees follow different paths (Bagnasco, 2023, Hernandez et al., 17 Jul 2025, Ahlin et al., 2019).

2. Namespace, storage abstraction, and policy model

Rucio’s data model is centered on Data Identifiers (DIDs). The namespace has three levels: file, dataset, and container. A file corresponds to an actual file on storage; a dataset is a logical grouping of files; a container is a grouping of datasets and/or containers. Containers consist of containers or datasets, datasets consist of files only, files can belong to multiple datasets, and collections are datasets or containers. Each DID is a tuple of scope and name, written as scope:name, and the combination must be globally unique. A critical design principle is that DIDs are identified forever: once a DID has been used, it can never be reused, even if the corresponding data is deleted. If data changes, the name must change too (Barisits et al., 2019).

The catalog stores metadata and availability information. Built-in metadata includes checksums, with MD5 and Adler32 explicitly supported. A file DID can be in one of three availability states: available, lost, or deleted. Collections carry additional attributes: open means content may be added and once closed it cannot be reopened; monotonic means content may not be removed once set; complete means all files have available replicas. Rucio also supports archives such as ZIP files, with registered constituents and location resolution via the archive file (Barisits et al., 2019).

The storage abstraction is the Rucio Storage Element, or RSE. An RSE is the abstraction of a globally addressable storage endpoint and includes hostname, port, protocol, local path, and arbitrary key-value attributes. Files are materialized as replicas on RSEs. Rucio supports deterministic and non-deterministic RSEs: in the deterministic case the path can be derived from DID information alone, while in the non-deterministic case additional metadata or path information is needed. It is transport-protocol agnostic and supports heterogeneous protocol priorities for read, write, deletion, and third-party copy; the cited protocols include gsiftp, SRM, ROOT, WebDAV, and S3, and the paper lists systems such as EOS, dCache, XRootD, StoRM, DPM, Dynafed, and CASTOR (Barisits et al., 2019).

The policy engine is built around subscriptions and replication rules. A subscription is a standing policy that automatically creates replication rules for future DIDs matching a metadata filter. A replication rule defines the minimum number of replicas that must exist for a DID on a set of RSEs, applies continuously to all constituent files of the DID, follows content changes in datasets, is owned by an account, protects replicas from deletion, and requests transfers if data is not yet present. A rule requires at least the DID, an RSE expression, the number of copies, and the rule lifetime. Placement decisions are recorded as replica locks, which stabilize data placement by preventing later re-evaluation or reshuffling (Barisits et al., 2019).

These abstractions reappear almost unchanged in later domain-specific deployments. Rubin uses RSEs for facility storage endpoints, datasets as units of grouping and closure, and subscriptions and replication rules to trigger movement to processing and access facilities. SKAO exposes regional storage systems as RSEs with a name, a protocol endpoint, and a path prefix. XENON1T treats the LNGS buffer disk itself as an RSE and uses transfer rules and a file catalog mediated by Ruciax. The continuity of the abstractions across these papers is notable: the same DID/RSE/rule model spans HEP, astronomy, and gravitational-wave data logistics (Hernandez et al., 17 Jul 2025, Parra-Royón et al., 2023, Ahlin et al., 2019).

3. Architecture, asynchronous control, and operational behavior

Rucio is described as a four-layer distributed architecture consisting of clients, server, core, and daemons, with supporting storage/transfer tools and a persistence layer. Clients include command-line tools, a Python client API, and a JavaScript-based web UI. The main external interface is REST. The server is passive: it receives REST requests, forwards them to the core, and returns results. The core represents the global system state. Large operations such as huge replication-rule creation are accepted by the server and then processed asynchronously in daemons (Barisits et al., 2019).

The persistence layer uses SQLAlchemy as ORM and Alembic for schema migration, with SQLite, MySQL, PostgreSQL, and Oracle listed as supported RDBMSs. The database stores both catalog data and daemon state. The paper emphasizes optimization against row-lock contention and deadlocks, and it identifies work sharding across daemons as a particularly important design: daemons of the same type select disjoint subsets of work via hashing over request attributes, enabling lock-free parallelism and failover via a heartbeat mechanism (Barisits et al., 2019).

The transfer pipeline exemplifies the separation between policy and execution. Transfers are only initiated as a consequence of rules. Creation of a replication rule creates transfer requests; the transfer-submitter daemon ranks sources, chooses protocols, and submits batches to the transfer tool; the transfer-poller daemon polls the transfer tool for success or failure; the transfer-receiver daemon listens to messages; and the transfer-finisher updates replication rules from transfer outcomes. If a transfer fails, the rule can become STUCK, after which the rule-repairer tries to resubmit to another destination or retries after delay. Deletion is similarly asynchronous and policy-driven, with greedy and non-greedy modes. Consistency management distinguishes lost files from dark files, and recovery can recreate replicas from other copies or, if the last copy is gone, remove the file from the dataset and notify external services and owners (Barisits et al., 2019).

Rucio also incorporates asynchronous messaging and monitoring. The cited messaging backends include STOMP-compatible queues such as ActiveMQ, with schema-free JSON payloads and event types such as transfer-done and deletion-queued. Monitoring is partitioned into internal monitoring, dataflow monitoring, and reporting, with metrics flowing through statsd and Graphite into Grafana, and transfer/deletion traces flowing through ActiveMQ, Kafka, Hadoop-based processing, Elasticsearch, InfluxDB, and Grafana dashboards. This operational instrumentation is central to how later systems integrate with Rucio rather than incidental to it (Barisits et al., 2019).

Production figures from ATLAS illustrate the scale for which the architecture was designed. By the end of 2018, ATLAS managed roughly 450 PB total volume, 25 million containers, 13 million datasets, 960 million unique files, 860 RSEs, and 1.2 billion replicas. Transfer rates reached at least 30 PB/month and peaked at 55 PB/month, with 50–70 million files transferred per month and about 10 million failed transfer attempts/month, mostly due to storage or network misconfiguration but automatically recovered. Average server interaction rate was around 250 Hz, with spikes of 400–500 Hz and average response time under 50 ms (Barisits et al., 2019).

4. Authentication and authorization evolution

A major transition in the Rucio literature concerns authorization, especially in WLCG’s move from the legacy X.509-based model to token-based authorization. The motivating pressures were both technical and usability-driven: certificate management created a steep learning curve for researchers, the X.509-based authorization model was outdated by modern privacy expectations, integration with commercial and third-party services had become harder because many modern services already support federated identity via SAML or OIDC, and users were accustomed to web-style authorization flows such as OAuth2/OIDC delegation. Within this transition, Rucio is presented as a concrete, operational example of how a critical WLCG service can adapt to JWTs, OIDC, and the WLCG Common JWT Profiles schema (Bockelman et al., 2020).

The WLCG model uses JSON Web Tokens provisioned via OpenID Connect and OAuth2 workflows. The architecture centers on INDIGO IAM, which proxies multiple authentication channels, centralizes authorization, translates tokens to and from X.509, and serves as a uniform token issuer for downstream WLCG services. The token schema references RFC 6749, RFC 6750, OpenID Connect, RFC 7519, OpenID Discovery, and RFC 8414, and includes claims such as sub, exp, iss, acr, aud, iat, nbf, jti, eduperson_assurance, wlcg.ver, wlcg.groups, auth_time, and scope. Two authorization forms are supported in parallel: group-based authorization, semantically equivalent to VOMS groups, and capability-based authorization, where a capability is permission to perform an action, optionally at a specific path (Bockelman et al., 2020).

For Rucio specifically, the paper states that during 2019 its authentication and authorization mechanisms were extended to support JWTs using OIDC, following OAuth 2.0 specifications. The implementation is based on the OIDC-certified pyoidc library and the WLCG specification. For Rucio 1.22.0.dev3, a user can log in via the authorization code flow implemented in both the Rucio WebUI and the Rucio command line interface, and the CLI also supports username/password login and session validation using a browser. The Rucio REST API is configured as an OAuth2 protected resource and accepts tokens from trusted OIDC providers, including the WLCG token issuer (Bockelman et al., 2020).

The same paper extends the discussion downstream. Tests were successfully performed to demonstrate token-based authorization from Rucio to the File Transfer Service. When a Rucio user requests a file transfer, Rucio can use either a JWT issued for the user or a JWT issued to the Rucio Admin user itself, acting on the user’s behalf. To allow fine-grained downstream access control in the path Rucio → FTS3, token exchange and token refresh grant flows were implemented. The first functional third-party copy tests were performed in the chain Rucio → FTS3 → dCache, and a Rucio version supporting these features was deployed on the DOMA instance. The paper therefore treats Rucio not only as compatible with the token-based model, but as a working demonstration of the transition strategy (Bockelman et al., 2020).

5. Domain-specific deployments and scientific reuse

The extension of Rucio beyond ATLAS is visible in several mature domain-specific deployments. In the CERN Virtual Research Environment, Rucio is the core data-management framework underpinning the Data Lake. The VRE Rucio installation consists of Rucio servers, daemons, and webUI deployed on the CERN cloud using Helm charts, a central relational database hosted at CERN, and multiple Rucio Storage Elements at partner institutions with storage quotas ranging from 1 TB to 300 TB. These RSEs support heterogeneous storage backends such as EOS, StoRM, dCache, DPM, and XRootD, while data access is available through gridFTP, HTTP(S), XRootD, and S3. The paper distinguishes data upload, download, and streaming implemented using GFAL from third-party asynchronous transfers and deletion implemented through CERN’s File Transfer Service, with Rucio daemons providing the policy and control plane (Gazzarrini et al., 2023).

The newer VRE paper emphasizes the user-facing layer built around this infrastructure. It states that the ESCAPE Rucio instance provides a data management solution framework for the collaboration’s users and partners, connects the storage elements of different data centers, and manages the data transfers between them on users’ needs. It also states that Rucio uses FTS3 and GFAL2 to enable a transparent and seamless interaction between different storage technologies and file transfer protocols. A CERN EOS instance hosted at eospilot.cern.ch serves as a Rucio Storage Element, and a Rucio JupyterLab extension was developed to ease access and discoverability of data from a Data Lake by bridging the gap between the Rucio CLI and the Jupyter-based analysis platform. If a Rucio storage element is FUSE mounted on the user’s session, the extension allows injection of files present on this volume into a Jupyter Notebook as variables and can trigger replication of data to the attached volume if the data is not present (García et al., 4 Mar 2025).

In gravitational-wave computing, the LVK paper places Rucio in the offline distribution layer of the common IGWN infrastructure. Data transfers to the Origin servers are principally managed using the Rucio data management framework. These Origin servers publish proprietary interferometer data, Open Science Center public data, and pre-processed products. The paper describes one Origin for LIGO proprietary data at Caltech, one for Virgo data at UCLouvain, one for GWOSC Open Data at Caltech, and two for pre-processed LVK data, one at Caltech and one at SurfSARA in Amsterdam, mirroring each other. At the same time, the deployment is hybrid: some long-standing transfers, such as Virgo raw data transfer from EGO to custodial storage at CNAF and CC-IN2P3, are still managed with proprietary tools (Bagnasco, 2023).

The SKAO/SRCNet prototype uses Rucio as the core data-management layer of a storage prototype intended to provide a shared distributed storage fabric for a future network of regional centres. The Spanish prototype followed a four-phase deployment process based on StoRM as the storage resource manager and WebDAV as the transfer/access protocol, with OIDC-based authentication and authorization through ESCAPE IAM. After the endpoint was exposed externally, it was added administratively to Rucio using the RSE name SPSRC_STORM, the WebDAV endpoint https://spsrc14.iaa.csic.es:18027, and the prefix /disk/storm/sa/. Validation covered direct access with davix, FTS interaction using fts-rest-* tools, and copying to and from the RSE using the Rucio CLI. The paper also notes that object-based storage integration remains open (Parra-Royón et al., 2023).

In XENON1T, Rucio is the central distributed data management system for raw-data distribution across Open Science Grid and European Grid Infrastructure storage systems. The Rucio server and file catalogue are hosted at the University of Chicago, Rucio clients are distributed via CVMFS, and the collaboration uses the XENON-specific wrapper Ruciax because the deployed Rucio version was not compatible with Python 3. The LNGS buffer disk is treated as a Rucio Storage Element, raw data are uploaded into the catalogue by Ruciax, distribution to worldwide distributed RSEs is initiated from there, and the meta-database is updated when transfers complete. The paper states that raw data on the LNGS RSE are kept for about four days, which is enough time to complete distribution to the other RSEs (Ahlin et al., 2019).

Rubin Observatory uses a dedicated instance of Rucio as the central system for inter-site replication between Chile, the Archive Center within the US Data Facility at SLAC, European facilities, and annual Data Access Center distribution. Rucio is the orchestrator of replication between Rubin facilities, while FTS performs the actual transport. Rubin developed three key tools to bridge Rucio’s model with the Rubin Data Butler: rucio_register, which selects files from a Butler repository, attaches Rubin-specific metadata, and registers them into Rucio datasets; HermesK, a modified Rucio Hermes daemon that relays transfer-completion notifications through Kafka; and ingestd, a Rubin daemon at each destination facility that ingests newly replicated files into the local Butler repository. The paper is explicit that Rubin registers only files that actually need replication across facilities, because intermediate files can outnumber final products by about 10 times and storing them in Rucio would not be cost-effective (Hernandez et al., 17 Jul 2025).

6. Workflow coupling, network awareness, and analytic scrutiny

Rucio’s role increasingly extends into workflow orchestration and network-aware control. The 2022 network-services paper treats Rucio as the central data movement planner in a Rucio → FTS → XRootD stack for CMS/LHC workflows. In that baseline chain, Rucio decides what must be moved, FTS acts as the transfer queue and execution layer, and XRootD clusters provide the storage endpoints. The key architectural extension is a Rucio–SENSE coordination path mediated by a Data Movement Manager. Rucio sends the DMM data amount, source site, destination site, and relative priority; the DMM asks SENSE for a priority network path with source and destination sites, bandwidth, time duration, and IPv6 subnets; and the resulting IPv6 information is returned to Rucio so that FTS can begin the transfer. The paper emphasizes that no major changes are anticipated to Rucio–FTS interaction itself; the network-service intelligence is introduced before the normal FTS submission step (Lehman et al., 2022).

The later DMM prototype for SENSE-Rucio interoperation makes this coupling more concrete. Rucio remains the authoritative data management layer that creates and tracks transfer rules, while DMM periodically polls Rucio through the Rucio Python client to discover new rules and metadata such as source site, destination site, rule priority, and total transfer size. DMM maintains a daemon-based state machine backed by a SQL database, with states INITIALIZED → ALLOCATED → PROVISIONED → FINISHED. For point-to-point rules only, it allocates SENSE-controlled endpoints, computes priority-aware bandwidth allocations, provisions a SENSE circuit, tunes FTS concurrency, monitors host-level throughput and FTS job metrics, and later releases the endpoints. Its central optimization is formulated as

$\text{Objective:} \quad \text{Maximizing} \ c^T x$

subject to

$Ax \leq b,\ x \geq l \geq 0$

where $x$ is the bandwidth allocation vector, $b$ is the port-capacity vector, and $c$ is the priority vector. The objective is to maximize the priority-weighted sum of bandwidths so that higher-priority rules receive proportionally more bandwidth (Arora et al., 1 Aug 2025).

Rucio also appears as the data subsystem in higher-level workflow orchestration. The iDDS paper defines the division of labor as PanDA for workload management and job execution, Rucio for data management, and iDDS as the orchestration layer that coordinates what should run next, when, and under what conditions. The paper highlights data-aware orchestration in which iDDS stores workflow state, dependencies, input/output datasets, and scheduling metadata, while Rucio provides the authoritative system for dataset state and movement. In the ATLAS Data Carousel example, iDDS adds file-level granularity so that processing starts as files become available from tape rather than waiting for an entire dataset to be staged (Guan et al., 3 Oct 2025).

Analytic studies have also treated Rucio as an object of performance diagnosis rather than as an assumed black box. The 2025 PanDA–Rucio analysis paper describes Rucio as the distributed data-management counterpart to PanDA, managing replicas, enforcing replication rules, and orchestrating file transfers so that input data are available where jobs run and outputs are properly registered. By mid-2024, it reports that Rucio managed over 1 exabyte of ATLAS data across more than 180 storage elements worldwide. The study identifies redundant or unnecessary transfers, prolonged staging delays, bandwidth underutilization, error-associated transfer behavior, and mismatches between local placement and actual performance. To make these effects measurable, it develops a file-level metadata-matching algorithm connecting PanDA jobs and Rucio transfers and reports exact and relaxed matching results over an 8-day period, including 30,380 transfers and 7,907 jobs under exact matching, rising to 60,593 transfers and 16,501 jobs under the most relaxed method (Hsu et al., 1 Oct 2025).

Taken together, these papers suggest a broadening role for Rucio. The core abstractions—DIDs, RSEs, replication rules, subscriptions, asynchronous daemons, and transport-tool integration—remain stable. What changes is the control context around them: token-based authorization replaces X.509-centric access patterns; notebook environments and analysis facilities expose Rucio interactively; domain-specific platforms bridge Rucio with their own metadata and workflow systems; and orchestration layers increasingly treat data state, transfer state, and even network-service state as first-class inputs. A plausible implication is that Rucio’s enduring significance lies less in any single transfer protocol or experiment-specific deployment than in its role as a reusable scientific data-management control plane across heterogeneous, federated infrastructures (Barisits et al., 2019, Bockelman et al., 2020, Arora et al., 1 Aug 2025).