Distilled Agent Executor

• Distilled Agent Executor is a lightweight execution component that replicates expert decision logic for real-time tool invocation at reduced computational cost.
• It is developed via supervised distillation on curated ReAct trajectories, using a mixed-rationale loss to ensure robust and efficient decision-making.
• Its integration within TURA systems streamlines dynamic sub-task execution by eliminating chain-of-thought reasoning, significantly lowering latency and resource use.

A Distilled Agent Executor is a lightweight execution component—typically a neural or symbolic policy—that enables efficient, accurate, and real-time agentic behavior by replicating the decision logic and tool-use abilities of more complex agents, while drastically reducing computational and inference costs. In modern AI systems such as TURA (Tool-Augmented Unified Retrieval Agent for AI Search), the executor is produced via supervised distillation on high-quality, expert trajectories and is designed to meet industrial-scale performance and latency demands by eschewing runtime chain-of-thought reasoning in favor of direct, efficient action generation (Zhao et al., 6 Aug 2025).

1. Role and Architecture within Agentic Search Frameworks

Within the TURA system, the Distilled Agent Executor forms the final stage of a three-part architecture. After the Intent-Aware Retrieval module decomposes the user query and identifies relevant tool endpoints (termed MCP Servers), and the DAG-based Task Planner organizes task dependencies and parallel execution via a directed acyclic graph, the executor receives each sub-task (constituted as a (refined sub-query, associated tool) pair) and is solely responsible for dynamic tool invocation. This architecture enables TURA to bridge traditional retrieval-augmented generation—which is limited to static corpora—and agentic, tool-augmented execution that can call live APIs and databases, thereby supporting structured queries, dynamic content retrieval, and real-time requirements.

The executor itself is built as a distilled small model (e.g., from the Qwen3 series), fine-tuned on curated expert ReAct-style execution trajectories, and designed to operate at significantly lower latency and resource cost than previous approaches reliant on large, monolithic models (Zhao et al., 6 Aug 2025).

2. Distillation Methodology and Loss Formulation

The training of the Distilled Agent Executor is underpinned by a two-stage dataset curation and supervised fine-tuning process:

• Expert Trajectory Collection: Generation of a log of ReAct-style tuples—comprising observations, explicit chain-of-thought reasoning, and tool-calling actions—created by a high-performance teacher agent (e.g., Deepseek-V3).
• Curation: Correctness filtering (using a judge model to enforce schema adherence and logical consistency) and efficiency filtering (disqualifying redundant or suboptimal steps) are applied to create a distilled, high-quality dataset denoted as 𝒟_distill.

Training employs a mixed-rationale supervised fine-tuning objective: $$\mathcal{L}_{\text{SFT}}(\theta) = -\sum_{(st', \pi') \in \mathcal{D}_{\text{distill}}} \sum_{t=1}^{|\pi'|} \log P_\theta(y_t | y_{<t}, st')$$ where $y_t$ is the token for the predicted thought–action pair and $\theta$ the parameters of the distilled model. This loss ensures the distilled executor acquires robust decision-making capability during training, while inference can be performed without generating full chains-of-thought (see Section 3).

3. Inference and Latency Optimization

A key differentiator is the "train-with-thought, infer-without-thought" execution paradigm. Although the model is exposed to stepwise reasoning during training (i.e., the full thought-action sequence), it is explicitly designed to predict only the concise action (the tool call) at inference time. This dramatically reduces the average number of forward passes and output tokens per sub-task, cutting end-to-end latency to a fraction of that required by chain-of-thought–emitting agents.

Empirically, distilled models such as Qwen3-4B Distilled offer latency compatible with the demands of industrial-scale, synchronous web search and recommendation applications, supporting deployments serving tens of millions of users (Zhao et al., 6 Aug 2025).

4. Integration with Retrieval and Planning Modules

The executor is tightly coupled to both the retrieval and planning systems:

• It consumes sub-tasks and tool-server selections generated by the Intent-Aware Retrieval module, ensuring that tool invocations are contextually relevant.
• As the DAG-based Task Planner traverses the execution graph, the executor receives each (sub-query, tool) node for real-time parallel execution.
• The efficiency of the executor is critical for the overall latency and throughput of the entire TURA system, especially under the parallel execution regime enabled by the DAG planner.

5. Empirical Performance and Industrial Deployment

The executor’s effectiveness is validated in full-stack, production-scale deployment:

• Real-world scenarios (e.g., flight or ticket availability queries) demonstrate the system's ability to return authoritative, real-time results via external API calls—a class of queries that are fundamentally unanswerable via static RAG approaches.
• Experimental measurements report that distilled agent executors match the tool-calling accuracy of teacher agents but at a fraction of the computational cost.
• This design is crucial for maintaining faithfulness (producing answers reliably grounded in dynamic backends) and fulfilling the stringent latency SLAs of modern conversational search engines.

6. Future Directions and Expansion

Potential future improvements involve:

• Enriching the distillation process with an even broader set of expert trajectories, including more diverse domains and multi-modal data.
• Tightening the dynamic interaction between the executor and DAG-based planner, allowing for adaptive behavior and real-time replanning based on tool feedback or observed latency.
• Expanding tool coverage and possibly integrating generalized reasoning to support not only structured tool invocation but also intelligent handling of unstructured or unforeseen data sources.

7. Summary Table: Distilled Agent Executor in TURA

Component	Function	Distillation Approach
Intent-Aware Retrieval	Query decomposition & tool discovery	Not applicable (retrieval system)
DAG-based Task Planner	Task dependency modeling & scheduling	Not applicable (planner system)
Distilled Agent Executor	Real-time tool invocation for each subtask	Mixed-rationale SFT on curated expert ReAct trajectories

This organization highlights the Distilled Agent Executor as the operational core enabling scalable, accurate, and low-latency agentic orchestration within TURA (Zhao et al., 6 Aug 2025), and establishes it as a template for efficient agent deployment in contemporary AI search architectures.