---
title: 'Globus Flows: Automated Research Workflows'
url: https://www.emergentmind.com/topics/globus-flows
type: topic
---

# Globus Flows: Automated Research Workflows

Globus Flows is a cloud-hosted research automation service central to the reproducible, efficient, and secure orchestration of multi-step workflows involving disparate scientific instruments, compute, storage, and human-in-the-loop tasks. At its core, Globus Flows enables declarative specification and execution of heterogeneous research processes—spanning spatial and temporal scales—by linking instruments, HPC clusters, storage endpoints, and external services in robust, parameterizable pipelines [2208.09513], [2111.11330].

## 1. Flow Specification and Execution Model

A "flow" in the Globus platform is a state-machine–style, multi-step process defined in JSON or a Python-oriented domain-specific language, extending the Amazon States Language (ASL). Each flow comprises:

- A `"StartAt"` field marking the initial state.
- A `"States"` mapping of state names to definitions.

Supported state types include:
- **Action:** Invokes asynchronous external action providers (e.g., data transfers, compute functions).
- **Choice:** Conditional branching.
- **Wait:** Time-based delays.
- **Pass:** No-op, context reshaping.
- **Fail:** Flow termination with failure.

Each Action state specifies:
- An `ActionUrl` (registered REST endpoint of the action provider),
- `Parameters` (input, mapped via JSONPath from flow context),
- `ResultPath` (output location in context),
- `WaitTime` (timeout),
- `Next` or `End` for control flow.

Flows declare an input schema (JSON Schema) both to validate initial context and to drive GUI or SDK front ends [2208.09513].

## 2. Architecture and System Integration

Globus Flows and companion services (Queues, Triggers, Timers) are fully cloud-native, leveraging AWS managed services for reliability, scaling, and resilience. Key components include:

- **Front-end API:** Implemented as AWS ECS Docker containers behind load balancers.
- **Persistent State:** AWS RDS for flow/run metadata and DynamoDB for OAuth tokens.
- **Execution Engine:** AWS Step Functions (ASF), driving each flow instance as a state machine, with ASF Task states enqueueing on Amazon SQS. AWS Lambda functions dequeue and invoke action providers; exponential backoff polling assures liveness.
- **Queues Service:** REST layer over SQS, with Globus Auth–protected roles to guarantee in-order, at-least-once message delivery.
- **Triggers:** Back-end workers that poll queues, evaluate user-defined predicates, transform event payloads, and post flow invocations.
- **Timers:** Priority-queued schedulers for periodic or time-triggered actions that persist through outages [2208.09513].

Integration with resources is federation-based:
- At instrument (edge), data is produced into directories or storage endpoints.
- Compute clusters (e.g., ALCF's ThetaGPU) host endpoints for Globus Transfer (data staging) and funcX (remote function execution).
- Globus Auth enables passwordless, single-consent access, with delegated OAuth2 tokens for flows to call Transfer/funcX on users’ behalf [2111.11330].

## 3. Action Provider API and Authorization Delegation

Every action provider is a registered RESTful service supporting:
- `GET <action_url>/`: Introspect provider, report scope/schema.
- `POST <action_url>/run`: Execute action, returning action_id and status.
- `GET <action_url>/<action_id>/status`: Poll state and output.
- `POST <action_url>/<action_id>/cancel`: Request cancellation.
- `POST <action_url>/<action_id>/release`: Free provider-side state.

OAuth2 scopes are issued and registered via Globus Auth per provider and per flow, with scope dependencies such that user consent to a flow’s run scope transits to all underlying actions. On invocation, the Flows front-end acquires needed tokens and caches them securely; Lambda backends retrieve and use these tokens as needed. `"RunAs"` directives allow flow states to execute as group-managed or alternate identities [2208.09513].

## 4. Event-Driven and Parameterized Workflow Automation

Globus Flows orchestrates automation using event decoupling:
- **Event Generators** (file watchers, instrument controllers) emit JSON payloads to named queues.
- **Triggers** poll queues, evaluate predicates, map events to flow inputs, and invoke parameterized flows.
- **Timers** periodically enqueue flow/action invocations based on schedule/count/end-time, ensuring orchestration over long timescales and resilience to service interruption.

Flows often parameterize actions per experiment (e.g., `{scan_id}` for beamline data), enabling massive concurrent sub-flow launches—e.g., one per sample, scan, or view [2111.11330].

## 5. Use Cases and End-to-End Orchestration

Globus Flows underpins diverse, distributed research automations:

- **Synchrotron Imaging:** At APS, a driver detects new ptychographic scan directories, launching parameterized flows that (1) transfer raw data (bulk directory) from the beamline to ThetaGPU via Globus Transfer, (2) invoke GPU-accelerated reconstruction via funcX, and (3) stream results back to the beamline, with retries and dynamic job scheduling [2111.11330]. Concurrent flows (up to 64 GPUs) maximize pipeline overlap of transfers and compute for hundreds of views.
- **Real-Time X-ray Crystallography:** Multi-step per-image and per-batch flows automate image transfer, processing, visualization, catalog submission, and merging [2208.09513].
- **ML Training and Inference:** Flows tie together image staging, HPC-based MIDAS analysis, neural net training, and model distribution to edge devices.
- **Data Publication and Human-in-the-loop:** Flows coordinate transfers, interactive metadata entry, curator approval, DOI minting, and Search indexing in the Materials Data Facility.
- **Analysis-as-a-Service:** e.g., AlphaFold job orchestration, storage allocation, compute, publication, and user notification [2208.09513].

A characteristic flow for ptychographic reconstruction includes chained states with error and retry logic:

```python
ptycho_flow = Flow(
  id="ptycho-recon",
  states={
    "CreateDirs":   Action("mkdir", params={...}, next="TransferIn", on_failure="Fail"),
    "TransferIn":   Action("transfer", params={...}, next="Compute", retry=3, on_retry="TransferIn", on_failure="Fail"),
    "Compute":      Action("funcx_run", params={...}, next="TransferOut", retry=2, on_retry="Compute", on_failure="Fail"),
    "TransferOut":  Action("transfer", params={...}, next="Done", retry=3, on_retry="TransferOut", on_failure="Fail"),
    "Done":         Succeed(),
    "Fail":         Fail()
  },
  start_state="CreateDirs"
)
```
States are linked via the `next` field; each action has retry and failure-handling paths [2111.11330].

## 6. Performance, Scaling, and Mathematical Foundations

End-to-end throughput depends on transfer, compute, and orchestration overheads. In experimental deployments:

- Single-node (8 A100 GPUs): For 1.8K frames, Catalyst dataset (~20 GB), ~1.0 s/iteration on one GPU, with best scaling at 1–2 GPUs due to communication overhead. Larger datasets (Coin 8K/16K, Siemens 32K) achieve >90% scaling to 8 GPUs.
- Multi-node: 8 nodes (64 GPUs) process 168 views in ~85 s end-to-end (vs. ~350 s serial/1 node, ~2500 s per GPU workstation).
- Flow run overhead (per step): For simple cases, mean overhead ~2.88 s; action provider (Transfer, funcX) median latencies are 3–4 s for trivial cases; authentication costs ~200–400 ms [2208.09513], [2111.11330].

The only explicit mathematical relation is for the ptychography iterative solver:
$$
\psi^{(i+1)} = F(d, p, \psi^i)
$$
where $d$ is raw diffraction data, $p$ is the probe, and $F$ implements the update via Fourier transforms and projections [2111.11330].

The composite makespan per view is:
$$
M(G) = T_\text{transfer in} + N_\text{iter} \cdot t_\text{iter}(G) + T_\text{transfer out} + T_\text{overheads}
$$
for $G$ GPUs and $N_\text{iter}$ iterations. For $V$ views processed with $P$ GPUs in parallel, the wall-clock time is approximately
$$
\lceil V/P \rceil \cdot M(G)
$$
modulo pipeline overlaps [2111.11330].

## 7. Design Insights, Limitations, and Future Directions

Adapting ASL for declarative state-machine logic yields familiarity but can result in verbose flow specifications; higher-level SDKs and GUIs are in development [2208.09513]. The action provider API is general, covering transfers, compute, DOI generation, search, notification, and manual tasks, though building robust providers requires custom implementation, token management, and scaling constructs.

Cloud-hosted, managed-service implementation eliminates user-side installations and provides reliable execution for long-lived flows. Throughput, not sub-second latency, is prioritized; overheads (1–4 s/step) limit applicability to real-time control loops but are acceptable for most distributed scientific automation use cases.

The OAuth2 authorization delegation model offers robust cross-resource security but can engender consent-management challenges for long-running, multi-domain workflows. Group-based access mitigates some user-experience limitations.

Early adopters report sufficient throughput and reliability for campaigns ranging from sub-second instrument triggers to multi-day HPC processing [2208.09513]. Enhanced provenance capture, debugging, and analytics tools are recognized as desirable for future system evolution.

---

References:  
- [2208.09513] "Globus Automation Services: Research process automation across the space-time continuum"  
- [2111.11330] "High-Performance Ptychographic Reconstruction with Federated Facilities"

Source: https://www.emergentmind.com/topics/globus-flows