FlowBot: Inducing LLM Workflows with Bilevel Optimization and Textual Gradients

Abstract: LLM workflows, which coordinate structured calls to individual LLMs (each augmented with varying instructions and tools) to achieve a particular goal, offer a promising path towards extending the capabilities of LLMs and building powerful systems that can tackle diverse tasks. However, existing approaches for building such workflows generally rely on human-crafted pipelines and prompts, which presents a substantial bottleneck in real world deployment. How can automatically induce and optimize such workflows in a data-driven way? This paper describes a simple data-driven approach for automatically inducing LLM workflows. We formulate workflow induction as a bilevel optimization problem: an outer loop which optimizes a high-level sketch of the workflow (in particular how the LLM calls should be structured), and an inner loop which optimizes each individual LLM call one-by one. Both loops are optimized with textual gradients'' where for the inner loop we optimize each component in a modular way throughbackpropagating'' textual gradients layer-by-layer. We find that LLM workflows discovered through our \textsc{FlowBot} (work\textbf{flow} induction through \textbf{b}ilevel \textbf{o}ptimization and \textbf{t}extual gradients) approach performs competitively against strong baselines that make use of human-crafted or automatically-generated workflows.

• The paper introduces FlowBot, a fully data-driven framework that jointly optimizes LLM workflow structures and component prompts using bilevel optimization with textual gradients.
• It demonstrates that modular textual backpropagation enables efficient, interpretable prompt updates that consistently outperform prior prompt optimization algorithms on multiple benchmarks.
• Empirical evaluations reveal FlowBot’s cost-efficiency and scalability, achieving competitive performance with fewer API calls and reducing the need for human intervention.

FlowBot: Bilevel Optimization and Textual Gradients for Automated LLM Workflow Induction

Problem Formulation and Approach

The paper introduces FlowBot, a framework for inducing and optimizing LLM-centric workflows in a fully data-driven manner, eliminating the need for human-crafted pipelines or bespoke prompts. The induction of such workflows is posed as a bilevel optimization problem in text space: the outer optimization loop searches over workflow structures (high-level sketches dictating the composition and ordering of LLM calls and tool usage), while the inner loop optimizes the behavior of individual workflow components (detailed prompts for each LLM invocation). Both optimization levels leverage natural language, "textual gradients"—LLM-generated feedback grounded in execution traces and evaluation results. The inner loop performs layerwise modular prompt optimization using textual backpropagation, analogous to gradient descent in neural nets, propagating feedback through each workflow stage based on downstream error signal attribution. The outer loop, distinctively, edits the overarching workflow abstraction itself—adding, removing, or restructuring steps in response to suggested textual gradients.

This approach generalizes optimization over static LLM programs, as used in prior prompt optimization works, by jointly optimizing both program structure and component prompts. The workflow representation is explicit in text, and each LLM call is parameterized by a prompt $\theta_i$ generated and iteratively improved based on batch feedback. Notably, neither the structural search nor the modular prompt updates require program synthesis: the whole workflow lifecycle is driven by LLM feedback, thereby unifying both architecture search and weight optimization in the language modality.

Experimental Evaluation and Results

FlowBot undergoes comprehensive evaluation on ten public benchmarks encompassing multi-hop QA (HotpotQA, HoVer), instruction following under constraints (IFBench), privacy-conscious delegation (PUPA), mathematical reasoning (GSM8K, MATH), program synthesis (HumanEval, MBPP), and discrete reasoning (DROP). The experimental protocol tightly couples model selection and validation, with model updates selected based on validation set performance, mirroring best practices in neural architecture search.

The results demonstrate several strong claims:

• Workflow Discovery vs. Optimization-only Baselines: FlowBot surpasses the strongest prior prompt optimization algorithm (GEPA) on retrieval/verification-heavy tasks and matches or exceeds performance on others, even though GEPA is initialized with human-designed workflows and manually tuned prompts. FlowBot's induced workflows, discovered entirely from scratch, consistently outperform when both structure and prompts are allowed to adapt [(2604.26258), Table 1].
• Comparison with Automated Workflow Synthesis Methods: Compared to AFlow and ADAS—which induce agentic workflows in program or code space—FlowBot achieves higher accuracy on four of six benchmarks when using the same LLM executor and comparable optimization budgets, often at lower cost. The improvements are maintained after ablating for model scale and API call count [(2604.26258), Table 2].
• Cost and Data Efficiency: Empirical results on API cost/usage indicate that FlowBot induces high-quality workflows using an order-of-magnitude fewer API invocations than AFlow, emphasizing its practicality for expensive executor LLMs in real-world settings.
• Component Ablations: Removal of bilevel optimization (i.e., optimizing the entire workflow as a monolithic prompt) or collapsing the inner loop (single-step prompt updates across all components rather than modular backpropagation) leads to substantial degradation in accuracy, confirming the necessity of both the bilevel structure and strictly modular, layer-wise updates.
• Cross-Model Transferability: FlowBot generalizes well to open-weight LLMs (e.g. Qwen3.5-397B), outperforming GPT-OSS baselines. Improvements are amplified when strong meta-LLMs (as optimizers) are paired with smaller executor LLMs, which has practical consequences for deployment on resource-constrained systems.
• Unified Workflow Induction: An extension experiments with learning a single conditional workflow to route and solve multiple tasks jointly. While there is a performance gap compared to task-specialized workflows, the unified model achieves competitive scores for most tasks, splitting data-driven meta-steps (reading comprehension, math, code generation) and demonstrating the method's flexibility in more generalizable workflow induction.

Analysis of Induced Workflows and Textual Backpropagation

Comparison of induced workflows to human-engineered baselines reveals that FlowBot discovers modular decompositions consistent with expert priors, but often with finer granularity in verification and retrieval logic. For instance, in multi-hop QA, FlowBot inserts explicit disambiguation and attribute-targeted retrieval steps, not present in human designs, to address observed structural failures. Qualitative analyses of layerwise textual gradients (textual "chain rule") show meaningful, localizable attributions of downstream error that guide effective prompt updates. For instance, when a hallucinated fact in downstream reasoning negates a correct retrieved answer, the backward-pass gradients instruct the system to (1) cross-reference multiple authoritative sources, (2) add uncertainty estimates, and (3) defer commitment to a single answer—directly addressing latent sources of systematic error. These instance-level gradients, when aggregated, lead to structural modifications and targeted prompt editing that are verifiably reflected in improved empirical performance.

Theoretical and Practical Implications

FlowBot concretely demonstrates that bilevel optimization, a major paradigm in neural architecture search, can be ported to LLM workflow induction—in text space, with no fixed API or hand-crafted program search. Modular textual backpropagation yields interpretable attributions for workflow improvement, and the framework absorbs insights from both neural program synthesis (via continuous structure search) and meta-learning (via optimizer LLMs).

Practically, FlowBot reduces the design and deployment costs for multi-stage LLM applications, learning nontrivial, domain-specific workflow decompositions in a data-driven manner with minimal human intervention. The explicit decoupling of meta-LLM (gradient/prompt updater) and executor-LLM (inference engine) reduces the energy and cost footprints in deployment by front-loading expensive search into the optimization phase. The success of the unified workflow induction experiment suggests possible application in generalized LLM-based solutions, potentially accelerating tool learnability and compositional system integration—key for future agentic AI systems.

Future Directions

Several avenues naturally extend from this work:

• Scaling Workflow Search: Although FlowBot's search space is currently restricted to feedforward, acyclic chains of LLM calls, extending the method to induce workflows with explicit loops, conditionals, and agentic subroutines would further expand its expressiveness.
• Batchwise Meta-Learning and Generalization: The rapid convergence and data-efficiency of gradient propagation suggest a path toward few-shot or even meta-learning formulations, where general workflow templates are adapted with minimal additional data.
• Continual Structural Adaptation: Integration with continual learning to support persistent and dynamic structure evolution as new tasks and domains are encountered in production remains an open research problem.
• Improved Feedback Signal: Alignment with richer, model- or human-in-the-loop reward signals, and leveraging explicit execution traces for causal error attribution, would facilitate finer-grained and more robust optimization.

Conclusion

FlowBot offers a principled, scalable approach to automated LLM workflow induction, unifying modular textual backpropagation with outer-loop structure optimization. Empirical evidence across a suite of reasoning, program synthesis, and constraint-following tasks positions it as a strong competitor to both hand-designed and code-synthesized workflow baselines, with the added advantages of interpretability, cost-efficiency, and extensibility. The paradigm outlined by this work is well-positioned to drive research in scalable, generalized LLM-based agents and could serve as an architectural foundation for more flexible and autonomous compound AI systems (2604.26258).