Compiling Agentic Workflows into LLM Weights: Near-Frontier Quality at Two Orders of Magnitude Less Cost

Abstract: Agent orchestration frameworks have proliferated, collectively exceeding 290,000 GitHub stars across LangGraph, CrewAI, Google ADK, OpenAI Agents SDK, Semantic Kernel, Strands, and LlamaIndex. All follow the same pattern: an external orchestrator above the LLM, injecting instructions and routing decisions every turn. Recent work has shown this architecture is dominated for procedural tasks by simply providing the procedure in a frontier model's system prompt [Dennis et al., 2026a], at the cost of consuming the context window, requiring a frontier model for every conversation, and exposing proprietary procedures to third-party providers. Compiling the procedure into the weights of a small fine-tuned model -- creating a subterranean agent -- should resolve all of these concerns, and prior work (SimpleTOD, FireAct, SynTOD, WorkflowLLM, Agent Lumos) has shown the technique works. Yet developer adoption has overwhelmingly favored orchestration. We identify three perceived barriers and address each empirically across travel booking (14 nodes), Zoom support (14 nodes, product-specific knowledge), and insurance claims (55 nodes, 6 decision hubs).

• The paper compiles agentic workflows into LLM weights, achieving near-frontier performance with 128–462× lower inference costs.
• It employs full-parameter fine-tuning on synthetic dialogues from directed graphs to internalize complex procedural logic.
• Empirical results show compiled agents outperform orchestrated baselines across diverse domains like travel booking, technical support, and claims processing.

Compiling Agentic Workflows into LLM Weights: Near-Frontier Quality at Two Orders of Magnitude Less Cost

Architectural Framework and Workflow Compilation

The paper systematically evaluates the paradigm of compiling agentic workflows directly into LLM weights, eliminating the need for external agent orchestration frameworks. Traditionally, agent orchestration frameworks (e.g., LangGraph, CrewAI, Semantic Kernel) inject instructions and routing decisions at each turn, operating on top of a large-scale LLM via APIs. However, this architecture incurs context window consumption, exposes proprietary procedures, and scales poorly in cost with procedure complexity.

By contrast, the compilation approach represents procedural knowledge as directed graphs (nodes, decision hubs, terminal states) and generates synthetic training dialogues that traverse all valid workflow paths. Full-parameter fine-tuning on these data results in a "subterranean agent": an LLM that directly internalizes the workflow structure in its weights, requiring only minimal initialization prompts at inference.

Figure 1: Travel booking flowchart (14 nodes, 3 decision hubs, 3 terminal states), illustrating procedural routing complexity.

Empirical Evaluation Across Domain Complexity

Three procedural domains are evaluated:

• Travel Booking (14 nodes): Tests simple agent procedures with moderate routing and looping.
• Zoom Technical Support (14 nodes): Incorporates product-specific troubleshooting and knowledge, testing domain transfer.
• Insurance Claims Processing (55 nodes): Substantially more complex, with nested loops and hierarchical decision logic.

  Figure 2: Zoom technical support flowchart, mirroring domain-aware troubleshooting pathways.

  

  Figure 3: Insurance claims workflow, featuring 55 nodes and 6 decision hubs for deeply hierarchical procedures.

Quality, Efficiency, and Failure Analysis

Quality Metrics

Results demonstrate that compiled models (Qwen 2.5 3B and Qwen3-8B) consistently outperform same-capacity orchestrated baselines and approach frontier-level performance:

• A 3B compiled model beats its orchestrated counterpart on 4/5 metrics (significant at $p < 0.001$), particularly on task success, consistency, and graceful handling.
• The 8B compiled agent achieves 87--98% of the in-context baseline’s quality, with graceful handling/naturalness reaching 92--97%.
• On the insurance claims task, the compiled agent leads the LangGraph orchestrator in graceful handling, naturalness, and consistency, with failure rates nearly halved relative to orchestration.

Cost Structure

Compiled models are 128--462$\times$ cheaper per conversation than in-context baselines, due to efficient self-hosted inference and elimination of repeated procedural prompts. Token consumption overhead from context window inflation grows with workflow complexity, further amplifying cost savings for elaborate procedures. Self-hosted 8B inference on commodity GPUs yields per-token costs that are roughly $65\times$ lower than frontier API rates.

Flexibility

Contrary to developer perceptions, the workflow compilation cycle completes within 30–50 minutes on production GPU clusters (or ~3–4 hours on a single A100), paralleling standard CI/CD deployment cycles. This enables frequent updates to procedural logic without prohibitive retraining.

Comparative Analysis and Failure Modes

Orchestration frameworks introduce fragmentation of conversational reasoning, routing failures at decision hubs, and unnatural response granularity. Compiled agents learn holistic procedure execution, eliminate routing failures by construction, and produce more audit-friendly, one-question-per-turn interviews. This structural advantage compensates for substantial model capacity differences: compiled 8B models achieve competitive performance against orchestrators using $70\times$ larger models.

Implications and Future Directions

Theoretical Implications

The results challenge established beliefs about LLM-based agent development:

• For procedural domains, persistent structure is more effectively realized through weight internalization than through prompt-injected orchestration.
• The dominant failure modes of modular agent frameworks—local reasoning, routing errors, and template-constrained responses—are absent in compiled agents.
• Fine-tuning procedural knowledge cannot be efficiently achieved via low-rank adaptation (LoRA, QLoRA), requiring full parameter optimization.

Practical Implications

• Compiled agents render in-context prompting and orchestration obsolete for procedural tasks, enabling practical deployment at two orders of magnitude less cost.
• Deployment cycles are rapid, practical for continuous integration, and scale gracefully with procedure complexity.
• This architecture vastly reduces inference latency by obviating repeated API calls for routing.

Prospects for Generalized Agentic Intelligence

Future developments may extend workflow compilation to:

• Large-scale enterprise agents with dynamically evolving workflows, exploiting flexible recompile cycles.
• Enhanced context-aware function dispatch and goal decomposition, leveraging full sequence modeling for stateful task management.
• Integration with open-source agentic LLMs and methodical benchmarking against frontier models for non-procedural domains.

Conclusion

Compiling procedural agent workflows into LLM weights achieves near-frontier conversational quality (87--98%) with a 128--462$\times$ reduction in inference cost. The compilation pipeline offers rapid flexibility and eliminates orchestrator-induced failure modes. As procedural knowledge persists beyond a single conversation, compilation emerges as the canonical solution for persistent agent structure, with significant implications for the evolution of agentic LLM systems (2605.22502).