AstraFlow: Dataflow RL & Radio Pipeline

• AstraFlow is a dataflow-oriented reinforcement learning system that decouples rollout, data management, and training for collaborative, multi-policy LLM fine-tuning.
• It enables elastic scaling and heterogeneous, cross-region operation by using a modular, auto-scaling design that improves training speed and efficiency.
• It also functions as a high-performance, GPU-accelerated pipeline for radio single-pulse searches, showcasing its versatile, modular architecture.

AstraFlow denotes a dataflow-oriented RL system for collaborative, agentic training of LLMs and also refers to a high-performance, GPU-accelerated radio single-pulse search pipeline. In both contexts, AstraFlow is defined by modular decomposition, explicit orchestration of data movement and compute, and, in the case of RL, system-level abstractions enabling elastic, heterogeneous, and cross-region operation (Zheng et al., 15 May 2026, Lin et al., 4 Nov 2025). The following article focuses on AstraFlow in RL for agentic LLMs, with a summary comparison to the radio transient search pipeline where relevant.

1. Motivations and Limitations of Prior Architectures

The expanding scope of RL applications to LLM fine-tuning—including reasoning, coding, tool use, and multi-agent workflows—necessitates RL systems that can coordinate complex, multi-policy workloads across elastic, heterogeneous, and geographically distributed compute resources. Prior trainer-centered RL architectures entangle rollout scheduling, data selection, staleness handling, and optimization within a single monolithic control loop. This rigidity impedes the integration of new features (multi-policy, cross-region execution, composable data algorithms), requiring system-level modifications for each new use case (Zheng et al., 15 May 2026).

AstraFlow addresses three challenges:

• Trainer-Centered Rigidity: Existing single-policy, single-trainer frameworks tightly couple rollout, data movement, and policy optimization, making system extension costly.
• Elastic & Heterogeneous Compute: Production RL workloads must utilize autoscaled/preemptible GPU pools, diverse accelerators, and wide-area networked nodes, all presenting dynamic constraints on throughput and bandwidth.
• Multi-Policy & Agentic RL: Modern agentic applications (e.g., solver+verifier, tool-using agents) need seamless collaborative training of multiple policies.

AstraFlow's key design insight is to replace this monolithic control with a dataflow-layer coordination, decomposing the system into three minimal, autonomous, and substitutable components: rollout services, dataflow manager, and trainers.

2. Dataflow-Oriented Architecture

AstraFlow’s architecture is defined by the decoupling of rollout services (RaaS nodes), dataflow management, and trainers, each interacting exclusively via programmable interfaces for data and state exchange. This decomposition engenders composability, elasticity, and multi-policy support without feature-specific engineering.

Core components:

• Prompt Sources: External data sources generate input prompts to be ingested for training.
• Dataflow Layer: Central abstraction that buffers prompts, agent trajectories, metadata, and training batches as first-class data objects. Exposes programmable “dataflow policy” hooks for curriculum scheduling, sampling, filtering, staleness correction, replay, etc.; all “control” arises from data availability and routing, not trainer-side coordination.
• Rollout-as-a-Service (RaaS) Nodes: Stateless, autonomously operating services that pull rollout tasks from the Dataflow Layer, fetch corresponding policy weights, generate trajectories plus rewards, and push results back to dataflow. RaaS pools can expand or contract elastically, substituting underlying agent-serving backends as needed.
• Trainers: Each trainer pulls batches relevant to its associated policy from the Dataflow Layer, applies optimization (e.g., PPO, M₂PO), and publishes updated weights via a central Weight Store (with versioning and delta sparsity). Trainers do not orchestrate rollouts or data movement, and multi-policy is natively handled by routing data objects to the appropriate trainer.
• Weight Store: Acts as a registry/transfer interface for publishing and pulling model weights, supporting versioning and sparse delta synchronization.

Scheduling and orchestration are performed by the Dataflow Layer based on data availability, not hard-coded control logic.

3. Formal Objectives and System Abstractions

RL Objectives

• Single-Policy:

For policy $\pi_\theta$ parameterized by $\theta$, AstraFlow maximizes expected return:

$$J(\theta) = \mathbb{E}_{\tau\sim\pi_\theta} \left[ \sum_{t=0}^{T} \gamma^t r(s_t,a_t) \right]$$

Proximal Policy Optimization (PPO) or M₂PO² is used; per-batch loss:

$$L^{PPO}(\theta) = -\mathbb{E}_t\left[ \min\left( \rho_t(\theta) A_t, \text{clip}(\rho_t(\theta),1-\epsilon,1+\epsilon)\cdot A_t \right) \right] + \beta \cdot D_{KL}[ \pi_{\theta_{old}}(\cdot|s_t) \| \pi_\theta(\cdot|s_t) ]$$

with $$\rho_t(\theta)=\pi_\theta(a_t|s_t)/\pi_{\theta_{old}}(a_t|s_t)$$, $A_t$ advantage, $\epsilon$ clip ratio, and $\beta$ KL penalty.

• Multi-Policy Collaborative RL:

For $N$ roles ($i=1..N$), each with policy $\theta$0 rewards $\theta$1,

$\theta$2

Trajectories are tagged by producing policy and corresponding reward; trainers only process their associated trajectories.

Elastic Scaling (Three-Zone Controller)

Given current GPU pool size $\theta$3, trainer wait fraction $\theta$4, rollout production/consumption rates $\theta$5, and shrink margin $\theta$6, the auto-scaler targets pool size:

$\theta$7

Dynamics ensure resource elasticity tracks bottlenecks in rollout or training.

4. Core Algorithms and System Operation

AstraFlow's operational primitives are implemented as independent processes:

$\theta$8 $\theta$9

• Dataflow core: Implements prompt ingestion, rollout scheduling (with curriculum or policy-driven selection), trajectory ingestion and filtering, and batch construction and routing to trainers.
• Auto-scaler (Agentic Maintainer): Periodically reads “balance report” statistics from the Dataflow Layer and issues scaling commands to RaaS pools. This maintains equilibrium between rollout supply and batch consumption under fluctuating resource and workload conditions.

5. Empirical Evaluation and System Capabilities

AstraFlow is evaluated across three axes: multi-policy collaborative RL, system flexibility (elasticity/heterogeneity), and data-algorithm composability (Zheng et al., 15 May 2026).

Multi-Policy RL:

• Math solver+verifier on AIME24/AIME25/MATH500/Minerva (Qwen3-8B, M₂PO, 1200 iters):
  • Single solver: 51.1%
  • Dr. MAS baseline (von Neumann–verifier): 54.4% (Δ+3.3) @ 212.6 s/iter
  • AstraFlow solver+verifier: 56.5% (Δ+5.4) @ 77.7 s/iter (2.7× speedup)
• Code generation:
  • Single solver: 30.29%, solver+selector: 32.14% (+1.85), solver+test-case generator: 34.55% (+4.26)
• No system code changes are needed for supporting new workflows; only the workflow logic (prompt selection/switching) is modified.

System Flexibility:

• Auto-scaling (on Qwen3-14B, math):
  • 6 GPUs (fixed): 35.8 h, 358.0 GPU-h, 26.9% wait
  • 11 GPUs (fixed): 23.9 h, 359.2 GPU-h, 2.1% wait
  • 6–11 GPUs (auto): 24.4 h, 312 GPU-h (–13%), 3.0% wait
• Cross-region heterogeneity:
  • 3×4-GPU RaaS pools (one local, two remote – constrained bandwidth/RTT) with sparse delta weight transfer and full syncs yield minimal trainer stall, final accuracy 67.6% (homogeneous: 68.0%).

Dataflow Algorithm Composability:

• Three dataflow policies tested:
  • Vanilla: baseline accuracy, high rollout cost
  • Dynamic sampling + replay: similar accuracy, rollout cost ∼700k (3.5×)
  • Selective rollout/Replay (with GRESO): baseline accuracy at ∼200k rollouts

6. Elasticity, Heterogeneity, and Data Algorithm Integrability

AstraFlow's minimal data and weight interface boundaries make elasticity, cross-region operation, and expert dataflow logic accidental byproducts of the design:

• Elasticity: Resources (RaaS nodes) are scaled at runtime with no effect on trainers or orchestration; this supports dynamic adaptation to workload or hardware availability.
• Heterogeneous and Cross-Region Pooling: RaaS nodes can be deployed across diverse accelerator pools or distinct regions, exploiting sparse, pull-based weight synchronization (exploiting ≥98% bf16 weight equality) to hide transfer latency and bandwidth limits.
• Composable Dataflow Algorithms: Advanced selection, curriculum, or trajectory filtering algorithms are injected as policy hooks in the Dataflow Layer. This eliminates the need for system-level code changes when experimenting with new data management strategies.

7. Comparison: Astroflow for Real-Time Radio Transient Search

The term "AstraFlow/Astroflow" is also used for a GPU-accelerated pipeline for single-pulse searches in radio astronomy (Lin et al., 4 Nov 2025). While architecturally unrelated to agentic RL, both systems emphasize high modularity and compute efficiency:

• Radio AstraFlow integrates RFI excision, incoherent dedispersion, intelligent tiling, and deep YOLO-based detection for FRBs and related phenomena, achieving 10×–20× faster-than-real-time end-to-end throughput on consumer GPUs.
• Both variants achieve scalability and extensibility via modular component design (C++/CUDA core with a Python interface in radio; dataflow/trainer/RaaS separation in RL).

This suggests that the term “AstraFlow/Astroflow” has become an archetype for high-performance, data-centric pipelines in both machine learning and domain science applications.

---

References:

• "AstraFlow: Dataflow-Oriented Reinforcement Learning for Agentic LLMs" (Zheng et al., 15 May 2026)
• "ASTROFLOW: A Real-Time End-to-End Pipeline for Radio Single-Pulse Searches" (Lin et al., 4 Nov 2025)