Tool-Making and Self-Evolving LLM Agents in Low-Latency Systems

Abstract: Production LLM agents often waste latency and reliability by regenerating code for the same procedural steps on every request. We replace this inference-time coding loop with an agentic tool-making pipeline that compiles repeated SOP steps into validated, versioned tools before deployment. The tool-maker grounds synthesis in the live environment as it collects execution traces, observes backend schemas and values, generates candidate tools, and repairs them against labeled cases. At runtime, the production agent calls these tools directly and falls back to code generation only when needed. We deploy the approach in a Fulfillment Center alarm-triage system, where an agent diagnoses alarms against a 44-node SOP over heterogeneous metric backends. In production, tool calls reduce p50 latency by 42%. On 1,500 historical alarms, they reduce end-to-end error rate by up to 53% by suppressing run-to-run variance in repeated steps. Because tools return compact structured verdicts, they also enable a simpler direct-call architecture, reducing p50 latency by a further 62% in a controlled ablation. Versioned tools also improve auditability and expose specification gaps and upstream data drift. Our results show that self-evolving agents can make industrial LLM systems faster, more reliable, and easier to operate.

• The paper introduces a self-evolving pipeline that compiles SOP steps into validated, versioned tools to eliminate redundant code generation.
• It employs real-time execution traces and repair loops to achieve over 90% pass@1 accuracy while reducing token costs and latency.
• The approach enhances industrial LLM deployment by lowering error rates and enabling deterministic, auditable execution in latency-critical environments.

Tool-Making and Self-Evolving LLM Agents in Low-Latency Systems

Motivation: Bottlenecks in LLM Coding Agents for Latency-Critical Industrial Systems

LLM agents deployed in operational environments, such as fulfillment centers, typically operate by programmatically interpreting natural language SOPs, generating code at inference time per procedural node, and iteratively executing these actions. This CodeAct paradigm, however, incurs severe latency, cost, and reliability penalties: procedural steps are redundantly reinterpreted and re-coded for stable backends, creating unnecessary inference overhead and run-to-run variance. The paper proposes a self-evolving agentic tool-making pipeline, which compiles frequently executed SOP steps into validated, versioned tools prior to deployment and shifts repeated code generation out of the latency-critical path.

(Figure 1)

Figure 1: Architecture overview: offline, each SOP node is compiled into a tool using traces and a test-repair loop; online, the agent invokes tools directly, eliminating redundant code generation per node.

Pipeline Methodology: SOP Compilation via Grounded Tracing and Repair Loop

The proposed pipeline employs two key grounding mechanisms: real-time execution traces and repair loops. SOP text alone is insufficient due to frequent omission of schema-specific details and edge-case handling, requiring grounding in live environment data and execution. A data-collector sub-agent runs SOP steps against MCP (Model Context Protocol), capturing query code, schema metadata, and verdicts. Subsequently, a tool-maker LLM generates functionally uniform tools per node, tested against labeled cases provided by domain experts. Failures are diagnosed by a reflector LLM and repaired by the tool-maker in a bounded loop, deploying the highest pass@1 tool.

This results in tools whose signature maps $$(\text{warehouse}, \text{timestamp}, \text{context}) \rightarrow (\text{verdict}, \;\text{details})$$, ensuring deterministic, auditable execution at inference. Runtime, the agent either delegates to sub-agents invoking tools or calls them directly, preserving fallback capability for unavailable or failing tools.

Empirical Evaluation: Ablation and Agentic Improvements

Tool-Making Ablation and Component Contributions

The pipeline is validated across 44 SOP nodes, with per-node pass@1 as correctness metric. Experimental comparisons using GLM-4.7, GLM-5, Qwen3 235B, and GLM-4.7 Flash demonstrate that only the configuration incorporating both the data-collection trace and test-repair loop (SOP+D+R) achieves >90% pass@1.

Figure 2: Per-node pass@1 and generation cost for configurations; SOP+D+R attains 94.5% pass@1 on GLM-4.7, with component ablations significantly trailing.

Specification Bottleneck and Data Efficiency

Residual errors concentrate in underspecified SOP nodes, with edge cases and ambiguous averaging windows. Targeted augmentation raises pass@1 from 94.5% to 96.6%; clarifying SOP specification further lifts it to 99.9%. Notably, grounded tool generation attains high accuracy with minimal labeled data; label-free versions trail the labeled ceiling by less than 3pp.

Figure 3: pass@1 as a function of training set size; data grounding enables efficient tool synthesis from few labeled cases.

Repair Strategy Analysis

Repair strategies (reflection vs. non-reflection vs. hypothesis branching) are dissected. Reflection peaks pass@1 at 94.3% within 10 rounds, but underperforms at lower rounds absent trace grounding. Multi-hypothesis branching achieves similar ceilings more quickly at increased cost.

Figure 4: pass@1 across repair rounds for each repair strategy; reflection achieves ceiling at more rounds, multi-hypothesis branching converges faster but at increased cost.

Token Budget and Cost Reduction

Ablation with Claude Opus 4.6 confirms that the trace component reduces token usage by 29% to reach pass@1 comparable to the repair-only ceiling.

Figure 5: Per-node pass@1 and token cost for ablation configurations; SOP+D+R reduces generation cost without accuracy loss.

Production Deployment: Latency and End-to-End Accuracy Improvements

The pipeline is deployed in a fulfillment center alarm-triage agent with two configurations: code-generating sub-agent (baseline) and tool-based inference (direct or delegated calls). Tool calls reduce p50 latency by 42% and output tokens by 58%; architectural redesign with direct tool calls further reduces latency by 62%. End-to-end accuracy rises—error rates drop from 2.8% to 1.8% on Qwen3 32B, and 1.7% to 0.8% on GLM-4.5-Air.

This deterministic execution eliminates failures observed with sub-agent reasoning, including deviation from SOP and incorrect reporting. Mixed-trajectory fine-tuning maintains fallback capability: disabling tools results in statistically indistinguishable accuracy to tool-free fine-tuning.

Practical and Theoretical Implications

Transferability and Generalization

Versioned tools transfer between agents and student models with a one-time generation cost (800K tokens for 44-node library). Parameterized tools, where site-specific quantities (e.g., thresholds) are runtime parameters, enable deployment across diverse environments.

Auditing, Monitoring, and Data Drift Detection

Logging tool outputs unveils upstream environment inconsistencies, data drift, and specification gaps—issues masked in dynamic code generation. Tools surface exceptions (e.g., schema mismatches, metric format shifts) deterministically; fixing involves regenerating tools, not retraining agents, simplifying rollback and hotfix workflows.

Limitations and Future Directions

While effective in SOP-guided alarm triage, generalization to less structured workflows (e.g., free-form runbooks) is untested. Fully automating the improvement loop—removing human oversight and labeled cases—is an open challenge; label-free variants approach the labeled ceiling, but specification disambiguation remains inherently human-driven.

Conclusion

Compiling SOP steps into grounded, validated, versioned tools optimizes production LLM agent workflows by amortizing repeated procedural work. This paradigm enhances latency, reliability, and auditability without sacrificing agentic flexibility. The agent's limiting factor transitions from schema rediscovery to SOP specification quality, offering a scalable and robust framework for industrial LLM deployment in latency-critical environments.

For a comprehensive list of related work on LLM tool-making, agentic refinement, SOP adherence, industrial deployments, and iterative code repair, consult the cited references within the paper (2607.08010).