Unified Execution Pipelines Framework

• Unified Execution Pipelines are computational frameworks that model end-to-end workflows with directed acyclic graphs (DAGs) to reduce fragmentation and improve scalability.
• They systematically map logical DAG structures to resource-aware execution plans using optimization methods such as heuristics or mixed-integer programming for efficient load balancing.
• These pipelines employ decentralized, data-driven activation that eliminates central bottlenecks, enabling real-time execution on heterogeneous systems.

A unified execution pipeline is a computational framework in which a single, coherent data and control model governs the end-to-end execution of complex workflows, integrating diverse components, resource scheduling, data management, and system-level orchestration. The principal goal is to eliminate fragmentation and manual intervention across the pipeline life cycle—enabling modularity, scalability, and reproducibility across highly heterogeneous computation environments and scientific domains. Unified execution pipelines are critical in domains ranging from large-scale astronomical data reduction to machine learning, data discovery, hybrid HPC-quantum computing, and stream data fusion. Approaches such as DALiuGE exemplify this methodology by abstracting both datasets and algorithms into a single graph execution substrate, decoupling logical workflow expression from resource-aware execution and highly scalable runtime orchestration (Wu et al., 2017).

1. Graph-Based Logical Abstractions

At the core of unified execution pipelines is the formalization of the workflow as a directed acyclic graph (DAG), denoted $G = (V, E)$, where $V$ is partitioned into data nodes $D$ (immutable data artifacts) and application nodes $A$ (algorithmic tasks), and $E \subseteq V \times V$ captures all data dependencies and transformation steps. In DALiuGE, for example, a node $v \in V$ is either a data-drop $d_i \in D$ (e.g., partition of astronomical data) or an application-drop $a_j \in A$ (e.g., a deconvolution step); edges indicate that one drop is required as an input to another (Wu et al., 2017). This abstraction enables logical pipeline modeling independently from physical deployment, supports validation of acyclicity, and provides the substrate for automatic dependency discovery and type-driven orchestration.

Logical DAGs serve as the interface between domain scientists and pipeline engineers, supporting declaration of complex, multi-stage workflows without requiring manual specification of execution order, batching, or parallelism. Systems like the Rubin Observatory Butler layer similarly employ a registry-driven task-relation model in which all workflow elements declare only their data input/output types and processing dimensions, allowing middleware layers to synthesize an executable graph via dependency analysis (Lust et al., 2023).

2. Mapping Logical Graphs to Resource-Aware Execution

Unified pipelines systematically map the logical DAG to concrete execution plans over distributed, heterogeneous resources. This process is governed by resource capacity constraints, placement heuristics, and optimization objectives such as makespan minimization, load balancing, and memory/I/O/cpu utilization.

Formal mappings use mixed-integer programming or heuristics. In DALiuGE, let $x_{v,r} \in \{0,1\}$ be an indicator if node $v$ is mapped to resource $r$. The scheduler solves: $$\min T = \max_v (finish\_time(v)), \quad finish\_time(v) = start\_time(v) + cost(v)$$ subject to per-resource constraints: $$\sum_v x_{v,r} \cdot usage_k(v) \leq cap_k(r), \quad \sum_r x_{v,r} = 1 \quad \forall v$$ Heuristics include topological sorting, best-fit placement by projected finish time, and local exchanges to further reduce total completion time (Wu et al., 2017).

This separation enables stakeholders to independently optimize either the pipeline logic or the resource allocation without interference, and supports flexible adaptation to evolving cluster, cloud, or HPC environments. Systems may coalesce small tasks into bundles to further minimize scheduling overhead and improve throughput.

3. Decentralized, Data-Driven Activation and Execution

A distinctive property of several modern unified execution pipelines is their data-activated, decentralized execution model. Rather than a central scheduler issuing all control commands, the lifecycle of each application or data node is governed by completion and activation messages that propagate along the DAG according to data dependencies.

In DALiuGE, each data-drop tracks its pending predecessors; completion of the last predecessor triggers (“activates”) subsequent application-drops, and upon completion of these, further data-drops are emitted. All requisite control information is piggybacked on the data flow itself (metadata, task status, event). Pseudocode for drop activation:

on DataDropComplete(di): for aj ∈ succ(di): aj.pending_inputs– if aj.pending_inputs == 0: send Activate(aj) to the node hosting aj on Activate(aj): launch application aj with inputs emit DataDropComplete(outputs of aj)

This fully distributed control eliminates central bottlenecks and enables the pipeline to scale from a handful of concurrent tasks on a laptop to tens of millions on top-tier supercomputers (Wu et al., 2017).

4. Separation of Concerns in Runtime Architecture

Unified execution pipeline frameworks strictly separate the logical workflow, resource mapping, and distributed runtime orchestration:

• Graph Manager (GM): Interprets the logical pipeline, partitions it, and invokes mapping algorithms.
• Drop Manager (DM): Deployed on each compute host, it manages the runtime state of assigned subgraphs (data and application-drops).
• Communication Layer: Implements efficient low-latency messaging for control signals (e.g., ZeroMQ for activation and completion events).

Stakeholders are cleanly separated:

• Algorithm developers specify only logical graphs.
• System operators determine resource pool description and scheduler configurations.
• Resource providers expose compute/storage/network through a pluggable resource description API.

This architectural split promotes modularity, operational flexibility, and maintainability of large, multi-institution data processing workflows.

5. Scalability Analysis and Empirical Performance

Scalability is analyzed via an overhead model—per-drop overhead is usually linear in the number of drops: $T_{ov}(n) \approx n \cdot t_{overhead}$. With pipelined activation, the system's throughput on $P$ cores is: $\Theta(P) = \frac{1}{t_{task}/P + t_{overhead}}$ DALiuGE exhibits empirical scaling laws $\Theta(P) \approx \alpha \cdot P^{0.95}$ up to $P \sim 4$k with $\alpha \approx 200$ tasks/sec per 1k cores, and sustains high utilization and low per-task overhead even on large production workloads (Wu et al., 2017).

Comparable pipeline frameworks exploiting a unified execution substrate (e.g., the Butler system in the Rubin Observatory) demonstrate linear throughput scaling to $\geq 10^5$ tasks and efficient orchestration on both workstation and large-scale HPC/cloud environments (Lust et al., 2023).

6. Case Studies Highlighting Unified Pipeline Impact

CHILES (COSMOS H I Large Extragalactic Survey)

• Pipeline: $\sim$60 application-drops, $\sim$120 data-drops; maximum width: 30 parallel calibration drops.
• Execution: Mapped to 512 cores, task bundling reduced scheduler load.
• Result: End-to-end runtime 3.2 hours versus 16 hours for a legacy sequential script (5$\times$ speed up), peak throughput 1200 tasks/sec.

MUSER (Mingantu Ultrawide Spectral Radioheliograph)

• Pipeline: 40 highly parallel drops, data parallel over frequency channels.
• Execution: Scaled to 2k cores, >95% utilization, sub-second latency per processed slice (real-time).

These results demonstrate that unified pipelines can deliver order-of-magnitude speedups, real-time responsiveness, and efficient scaling, confirming their suitability for next-generation scientific data analysis (Wu et al., 2017).

---

Unified execution pipelines, exemplified by the architecture and empirical validation of DALiuGE, embody a principled approach in which a single logical DAG model governs pipeline specification, physical mapping, decentralized activation, and distributed execution, thereby enabling high scalability, flexibility, and sustainable development in data-intensive science and engineering. The paradigm continues to influence large-scale pipeline frameworks, promoting modularity, provenance, and operational efficiency (Wu et al., 2017).