Intelligent Distributed Dispatch & Scheduling

• Intelligent Distributed Dispatch & Scheduling (iDDS) is a workflow orchestration platform integrating data-aware execution, conditional logic, and programmable workflows across distributed systems.
• It employs a modular, agent-based architecture with RESTful services and event-driven mechanisms to coordinate workflow state, data state, and dynamic execution without a central scheduler.
• iDDS has proven effective in reducing data staging overhead and optimizing large-scale scientific workflows in projects like ATLAS, Rubin Observatory, and machine learning experiments.

Searching arXiv for the most relevant iDDS papers and closely related distributed dispatch/scheduling work. arXiv search query: "iDDS Intelligent Distributed Dispatch and Scheduling workflow orchestration ATLAS" Intelligent Distributed Dispatch and Scheduling (iDDS) is a workflow orchestration system for large-scale distributed scientific computing that coordinates execution decisions using both workflow structure and data state across heterogeneous infrastructures. In earlier ATLAS papers, the same acronym denoted the intelligent Data Delivery Service, reflecting an origin in fine-grained data transformation and delivery; later work broadened iDDS into an experiment-agnostic orchestration layer above workload and data management systems such as PanDA and Rucio, with support for programmable workflows, conditional execution, and adaptive control (Guan et al., 2020, Guan et al., 2021, Guan et al., 3 Oct 2025).

1. Origins, nomenclature, and motivating problem

The historical core of iDDS lies in ATLAS distributed computing at the Large Hadron Collider. The ATLAS Event Service (ES) introduced fine-grained event processing to exploit opportunistic resources such as HPC hole-filling, spot-market clouds, volunteer computing, and shared grid resources, but ES still required replicas of full input files even when workers processed only subsets of events. The ATLAS Event Streaming Service (ESS) addressed part of that limitation by pre-transforming input files at remote sites and asynchronously delivering only the required events over the WAN. iDDS was then proposed as the next step: not merely event streaming for ES, but a more general service that can transform data on demand, coordinate downstream processing, and support workflows beyond the ESS prototype (Guan et al., 2020).

This lineage explains the shift in the meaning of the acronym. In the 2020 and 2021 ATLAS design papers, iDDS is presented as the intelligent Data Delivery Service, with emphasis on reducing replicas, decreasing WAN traffic, and decoupling the “cold format” of stored data from the format best suited for execution. “Cold format” denotes how data are stored economically at rest, while the processing format is the representation preferred by the compute workflow. By 2025, iDDS is presented as Intelligent Distributed Dispatch and Scheduling, a broader orchestration platform that extends traditional workload and data management with data-aware execution, conditional logic, and programmable workflows (Guan et al., 2020, Guan et al., 3 Oct 2025).

The scaling pressure behind this evolution is explicit. The ATLAS motivation was tied to rapidly growing data volume, constrained storage and WAN bandwidth, and the unsustainability of retaining expensive full replicas. The earlier design papers frame iDDS as one response to HL-LHC growth, including a factor-of-20 increase in data volume by 2038, and as a cleaner architectural home for cross-cutting functions such as dynamic data placement, on-demand derivation production, and tape-aware incremental processing (Guan et al., 2020).

2. Architecture and software substrate

The early architectural description presents iDDS as a standalone, experiment-agnostic service composed of a general RESTful head service and daemon-mode agents. In that design, the RESTful service registers and queries requests and exposes catalog access to collections or contents. The active agents include a Transporter, which finds input replicas from external distributed data management systems; a Transformer, which transforms expensive input replicas into the desired processing format; and a Conductor, which notifies downstream consumers in a fine-grained way as new transformed output becomes available. The software stack is layered into an ORM layer, Core, API, RESTful services, daemons, and clients, with Python clients and a plugin architecture based on a base plugin class for integration with systems such as PanDA, pilot, crab, and others (Guan et al., 2020).

The later orchestration architecture is more explicit about persistent workflow state and asynchronous control flow. It is organized around five major components: workflow engine, base infrastructure, RESTful service, agents, and monitors. The database and event bus provide the infrastructure backbone; the REST layer is implemented as a Flask application behind WSGI daemons and Apache HTTP server; and persistent state is managed through SQLAlchemy and Alembic with support for Oracle, PostgreSQL, MySQL, and SQLite. The event bus is modular, with LocalEventBus, DBEventBus, MsgEventBus, and optional integration with external systems such as ActiveMQ. The API groups are authentication, ping, request, cache, catalog, monitor, message, and log (Guan et al., 3 Oct 2025).

The agent model also became more differentiated. The principal agents are Clerk, Transformer, Carrier, and Coordinator. The Carrier further contains Submitter, Poller, Finisher, Conductor, Receiver, and Trigger sub-agents. The 2025 paper states that operations in iDDS are fully distributed, with no central scheduler; the scheduling logic is instead distributed across specialized agents operating asynchronously. At the same time, iDDS retains shared services such as a relational database, event bus, REST interface, and monitoring layer. This suggests an architecture without a single scheduler process, rather than one without central infrastructure (Guan et al., 3 Oct 2025).

3. Workflow model and execution semantics

The internal workflow model revolves around four abstractions: Work, Workflow, Condition, and Parameter. Each combines a Template, understood as reusable static logic, with Metadata, understood as dynamic runtime context. A Work unit is the atomic executable entity and corresponds to a group of jobs with similar attributes in the underlying workload system. A Workflow is a collection of Work units linked by dependencies. A Condition is the runtime control structure used to decide whether downstream Work should be created or triggered. A Parameter is a key/value mechanism for dataset identifiers, runtime settings, thresholds, model configurations, and related execution inputs; parameters may be hierarchical and dynamically generated during execution (Guan et al., 3 Oct 2025).

The 2021 paper gives a closely related but slightly earlier object model. A client submits a JSON-based Request through the RESTful head service; the Clerk converts that request into a Workflow object; the Marshaller manages directed acyclic graphs and splits Workflow objects into Work objects; the Transformer creates Processing objects by managing input/output associations and interacting with the DDM system; the Carrier submits Processing objects to the workflow management system and polls status; and the Conductor monitors outputs and triggers consumers. The same paper then broadens this from DAG management to directed graph (DG) workflow management, explicitly supporting not only DAGs but also graphs with cycles (Guan et al., 2021).

These abstractions support two complementary workflow styles. The first is template-based, in which the graph structure and dependencies among Work, Conditions, and Parameters are defined before submission. The second is Function-as-a-Task, a Python-native model inspired by Function-as-a-Service. In that model, local Python functions are decorated, serialized into Work objects, uploaded through an HTTP cache, submitted as tasks or jobs to remote workers, reconstructed by a wrapper in a remote execution environment, and returned asynchronously through STOMP or RESTful HTTP. This style is meant for runtime decisions, loops, external API calls, and adaptive ML workflows rather than only predeclared production DAGs (Guan et al., 3 Oct 2025).

Execution progression is hybrid. iDDS uses an event-driven mechanism for low-latency reactions to task completions, data availability, errors, and status changes, but also polls the database as a fallback to guarantee eventual progress if messages are lost. The papers describe a state machine for Workflow and Work lifecycle tracking, but they do not provide a formal state-transition diagram or a mathematical state model (Guan et al., 3 Oct 2025).

4. Data-aware dispatch and scheduling logic

The defining feature of iDDS is that execution decisions are conditioned on data readiness rather than only on predeclared workflow order. In the original data-delivery formulation, iDDS orchestrates workflow management and distributed data management so that transient data are transformed, delivered, processed, and removed at fine granularity. Instead of staging or replicating whole datasets before compute starts, iDDS can trigger transformation of expensive source data into a cheaper or more appropriate format, release only the transformed subset needed for current processing, and notify downstream consumers as soon as a new collection is available. This enables partial processing, shorter cache residency, lower storage occupancy, reduced replica usage, lower WAN traffic, and earlier start of downstream work (Guan et al., 2020).

The later orchestration papers preserve that logic but generalize it. The Transformer decides whether Work is ready to execute by checking prerequisites such as input data availability and selecting an execution system according to availability, efficiency, and policy constraints. The Clerk evaluates Conditions and generates downstream Work from Parameter values. The Carrier submits Work to systems such as PanDA or HTCondor and monitors execution. The Trigger and related agents incrementally release downstream jobs as dependencies are satisfied. The system therefore couples data state, workflow state, and execution state in one control plane (Guan et al., 3 Oct 2025).

The “intelligence” in iDDS is described architecturally rather than through an explicit optimization model. The papers do not provide a formal ranking equation, utility function, queueing model, or pseudocode for scheduling. Instead, the claimed intelligence lies in data-aware readiness checks, runtime condition evaluation, dependency-triggered incremental release, dynamic generation of new work from results and parameters, selection of execution backends according to availability, efficiency, and policy, and feedback-driven iteration in ML and active-learning workflows (Guan et al., 3 Oct 2025).

A common misconception is that iDDS is merely a data mover. The 2025 formulation is broader: iDDS is an orchestration layer between workflow logic, data state, and distributed execution. Conversely, another misconception is that iDDS replaces PanDA or Rucio. The papers are explicit that PanDA remains the workload execution system and Rucio remains the data management system; iDDS sits above them, translating Work into tasks or jobs, consuming data-state information, and coordinating progression across stages (Guan et al., 3 Oct 2025).

5. Scientific applications and production use

The breadth of iDDS is most visible in its applications, which span ATLAS, Rubin Observatory, and the Electron–Ion Collider ecosystem.

Use case	Workflow pattern	Reported status or scale
ATLAS Data Carousel	File-level data-driven incremental release	Production since mid-2020
Rubin Observatory	Large DAG with job-level dependencies	Production since mid-2021; over 100,000 jobs
Hyperparameter optimization	Iterative feedback loop	In production for ATLAS ML workflows
Active learning	Closed-loop adaptive workflow	Fully automated re-analysis reported
AID2E detector design	Function-as-a-Task optimization loop	Grid, cloud, and HPC execution

For ATLAS Data Carousel, iDDS introduced file-level orchestration into workflows that had previously been constrained to dataset granularity. Processing can start as files become available from tape rather than after full dataset staging, and processed data can be released from cache promptly. The reported outcomes are reduced data staging overhead, elimination of redundant transfers and caching, lower disk footprint, and sustained large-scale reprocessing throughput. The 2025 paper reports production use since mid-2020 and shows processed requests from 2021 to 2025, with one request per dataset (Guan et al., 3 Oct 2025).

For Rubin Observatory, iDDS manages large workflow graphs with job-level dependencies, including workflows with over 100,000 jobs. Rubin middleware dynamically generates a graph per payload submission, and iDDS performs dependency resolution and incremental release. The paper reports a dedicated PanDA–iDDS instance at SLAC and Rubin production use since mid-2021 for ComCam and DRP campaigns (Guan et al., 3 Oct 2025).

For machine learning, iDDS supports hyperparameter optimization as an iterative cycle of candidate sampling, distributed training and evaluation, metric collection, and search-space refinement. The 2025 paper cites Bayesian optimization and segmented HPO and notes production use in ATLAS ML projects such as FastCaloGAN. Active learning is treated as a closed-loop workflow in which simulation, reconstruction, significance evaluation, and proposal of new parameter points are chained through Conditions and dynamically generated downstream Work. One named example is the $H \rightarrow ZZ_d \rightarrow 4l$ analysis, which used PanDA and REANA in an automated re-analysis loop (Guan et al., 3 Oct 2025).

The AID2E detector-design framework gives the most detailed view of Function-as-a-Task in practice. There, iDDS serves as the workflow-orchestration layer above PanDA in a three-layer stack with Ax/BoTorch for candidate generation and surrogate updates, iDDS for packaging and orchestration, and PanDA for distributed execution on grid, cloud, Kubernetes, and HPC resources. The paper reports a representative DTLZ2 benchmark with 5 objectives and 100 parameters, and a realistic dRICH study with 7 parameters targeting pion–kaon separation, kaon–proton separation, and detector acceptance. It reports similar hypervolume convergence between local and distributed execution, increased concurrency under distributed execution, and hypervolume that increases monotonically with the number of trials in the dRICH study (Anderson et al., 31 Mar 2026).

6. Limitations, interpretations, and ongoing directions

The iDDS papers are strongest as architecture-and-operation papers rather than as formal scheduling theory. They do not provide explicit mathematical scheduling objectives, queueing relations, cost functions, or comparative scaling laws for the internal control plane. Quantitative evidence is present mainly as production deployment status, selected operational plots, and use-case outcomes rather than as systematic benchmarks of agent throughput, event latency, or failure-recovery cost. The 2025 paper also notes that event delivery can be lossy, which is why database polling remains necessary as a fallback (Guan et al., 3 Oct 2025).

Several limitations therefore recur. First, iDDS depends heavily on external systems such as PanDA, Rucio, HTCondor, and site infrastructure. Second, although operations are fully distributed in the sense of agent-based orchestration, the system still relies on a central persistent metadata store and shared service layers. Third, the “intelligence” remains largely rule-, state-, and condition-driven in the published descriptions rather than expressed as a formal optimizer. Fourth, some workflow classes remain more mature than others: ATLAS Data Carousel and Rubin DAG orchestration are production services, whereas other modes, such as active learning in earlier papers, began as prototypes (Guan et al., 2021, Guan et al., 3 Oct 2025).

Future directions are explicit. The original design paper states that the next step is to extend iDDS to support more workflows and improve its intelligence (Guan et al., 2020). The 2025 orchestration paper adds broader support for interactive workflows, serverless workflows, and cloud-native ecosystems, as well as general user-experience improvements (Guan et al., 3 Oct 2025). The AID2E paper adds larger ML models, more detector components, LLMs for workflow automation and decision support, SLURM-based runners, and broader generalization beyond current detector-design use cases (Anderson et al., 31 Mar 2026).

Taken together, these papers position iDDS as a control-plane system that unifies workflow state, data state, and distributed execution. Its historical importance lies in moving ATLAS computing away from replica-centric, bulk-synchronous data handling toward fine-grained, demand-driven orchestration; its broader significance lies in showing that the same abstractions—Work, Conditions, Parameters, data-aware triggering, and programmable distributed execution—can span tape-aware reprocessing, massive scientific DAGs, hyperparameter optimization, active learning, and AI-assisted detector design (Guan et al., 2020, Guan et al., 3 Oct 2025).