SPORK: Self-Speculative Forking to Accelerate Agentic LLM Inference

Abstract: LLM agents are becoming a common interface for research, coding, and question answering, yet their Thought-Action-Observation loop is often serial: the model reasons, emits a tool call, then idles the GPU until the result returns. This wait consumes 16-37% of wall time in our workloads and 35-61% in prior reports. Speculative tool execution can hide this wait, but existing systems need auxiliary predictors, historical traces, or static workflow graphs, leaving a gap for training-free, day-one deployment. We observe that the model can be its own predictor: a probe forked at the start of generation predicts Qwen3-32B's upcoming tool name with 74.6-99.6% accuracy across five benchmarks. We present SPORK (Self-sPeculative fORKing), a training-free controller that dispatches the speculated tool call early, overlapping its execution with the remaining chain-of-thought decode. A cost model captures when speculation breaks even, and each component improves one of its terms: a prefix-cache fork cuts probe cost, a confidence gate filters mispredictions, and partial-token accept turns rejected probes into speculative-decoding drafts. On acceptance, the tool result is ready when reasoning ends; on rejection, SPORK falls back to serial execution with no correctness penalty. On real-tool benchmarks, SPORK cuts Qwen3-32B's GAIA P95 by 18% (131.9 to 108.1 s); the mechanism holds across model sizes from 4B to 32B and across dense and mixture-of-experts models, with task accuracy within 1 pp of baseline or better wherever measured. SPORK deploys as a thin controller over standard completion APIs (no retraining, no auxiliary models, no offline traces) and is orthogonal to token-level speculative decoding. SPORK is open source at https://github.com/baihuajun24/spork.

• The paper introduces SPORK, a method that reduces tool-wait latency in LLM agents through self-speculative forking.
• It employs a prefix-cache fork, dynamic confidence gating, and partial-token acceptance to achieve up to an 18% reduction in P95 latency.
• The approach integrates with existing systems without retraining, preserving model accuracy while minimizing idle tool execution time.

Self-Speculative Forking for Latency Reduction in Agentic LLM Inference

Motivation and Bottleneck Characterization

The paper addresses the predominant latency bottleneck inherent in current LLM-agent architectures adopting the Thought–Action–Observation (ReAct) workflow. In this paradigm, the model serially reasons, emits a tool call, then idles while awaiting tool execution before resuming with the observed result. Empirical profiling on several agent benchmarks reveals that tool execution typically constitutes 16–37% of end-to-end wall time (Figure 1), with tail latencies dominated by outliers in tool response duration. This tool-wait is recalcitrant to optimization since its roots lie in external API latency, network, and I/O constraints—factors not ameliorated by hardware scaling or token-level speculative decoding.

Figure 1: Tool execution accounts for 16–37% of end-to-end wall time across agent tasks, highlighting its role as the critical latency bottleneck.

Empirical Insights Enabling Self-Speculation

The authors identify that instruction-tuned LLMs, such as Qwen3-32B, frequently "know" the upcoming tool call well before concluding their chain-of-thought (CoT) reasoning. Analyses show that a speculative probe, forked from the main thread immediately after emitting the first token and constrained to decode the tool-call prefix, yields a tool name prediction matching the main thread with accuracy ranging from 74.6% to 99.6% across five benchmarks (Figure 2). Argument-exact match rates are lower initially but rise monotonically as CoT context increases, reaching 97.5% at the chain’s end (Figure 3).

Figure 2: Fork-at-start probes predict the exact tool call name across a range of tasks, demonstrating early intent in agentic LLMs.

Figure 3: Fork accuracy in argument-exact matches increases with the observed CoT context, supporting dynamic gating.

The accuracy gap between tool name and argument prediction motivates the adoption of a dynamic, logprob-based confidence gate. The forked probe's token-level logprobabilities exhibit separability between correct and incorrect predictions, enabling an adaptive commit threshold. The design capitalizes on the empirical finding that most rejected probes nonetheless share a substantial verified prefix (median 18 tokens) with the final tool call (Figure 4).

Figure 4: Even when probes are rejected, a recoverable token prefix is present, enabling draft-based partial acceptance.

SPORK Method and System Design

The "Spork" system operationalizes these insights through three composable mechanisms:

1. D1 (Prefix-Cache Forking): Forks a probe immediately following the main's first token, leveraging KV-cache sharing to minimize probe dispatch overhead and open the speculative window (Figure 5). This ensures probe cost is negligible compared to tool execution time and avoids prefix contention.
2. D2 (Adaptive Confidence Gate): The probe streams its logprobs; upon surpassing a predefined confidence threshold on the tool name, speculative tool execution is triggered. The threshold (typically $\theta=0.90$) maximizes F1 and filters out mispredictions with high precision (Figure 6).
3. D3 (Partial-Token Accept/Draft Verification): On strict rejection (argument mismatch), the main decode only emits the suffix after verifying and accepting the maximal matching prefix from the probe (Figure 7). This mechanism recycles speculative computation and reduces wasted decode tokens.

   Figure 5: System overview—Spork overlaps tool execution with chain-of-thought reasoning via probe forking and confidence gating.

   

   Figure 6: Confidence gate threshold selection cleanly separates correct from incorrect probes using token logprobs.

   

   Figure 7: Partial-token accept mechanism verifies and admits the probe's matching prefix, reducing redundant decode operations.

Quantitative Evaluation

Spork is integrated as a controller over standard vLLM APIs and requires no retraining, auxiliary predictors, or offline traces. Experiments are performed on dense and MoE LLMs (Qwen3-32B, Qwen3.5-35B-A3B, Qwen3-4B) on benchmarks including GAIA, HotpotQA, BrowseComp, and tau2-bench.

Key results:

• On Qwen3-32B (GAIA), Spork achieves an 18% $P_{95}$ latency reduction (131.9 s → 108.1 s), with mean speedup scaling monotonically as tool latency increases (up to 1.18× at a 5 s tool floor).
• Task accuracy (EM, F1) is preserved within 1pp of baseline, occasionally improving, due to strict speculative acceptance gating (Figure 8). No correctness loss is observed because partial-tip drafts only admit tokens matching the main model's greedy output.
• Ablations verify that latency reductions are orthogonal and additive to token-level speculative decoding (Figure 9). D1 is optimal for mean/P50 latency; the synergy of D2+D3 dominates the tail.
• The system outperforms or matches external-drafter approaches, as shown by higher acceptance rates and lower incidence of wasted speculative tool executions (Figure 10).
• Spork’s benefit is neutral or negative only when overlap windows vanish (short tools or non-CoT models) or when forced probes diverge from the agent’s native tool-call schema.

  Figure 11: Spork reduces $P_{95}$ tail latency across all tested models and benchmarks.

  

  Figure 8: Task accuracy is preserved or improved across all configurations and scales due to strict speculative verification.

  

  Figure 9: Full Spork (D1+D2+D3) configuration achieves optimal tail latency by integrating all three mechanisms.

  

  Figure 10: Self-speculation yields greater overlap and higher acceptance rates than smaller cross-model drafters.

  

  Figure 12: Spork operates at or above break-even across the tool-latency envelope, with realized speedup matching cost model predictions.

Practical and Theoretical Implications

Spork demonstrates that tool-wait bottlenecks in agentic LLM inference can be mitigated, in many settings, using self-speculation—without added model complexity, retraining, or trace collection. The critical precondition is the early availability of actionable intent within the active model’s decode trajectory, which is empirically robust for instruction-tuned agents using JSON-based tool APIs. The theoretical framework with explicit cost modeling allows practitioners to anticipate the speedup regime based on measurable parameters ($\alpha$, tool time fraction, overlap, probe overhead), offering clear guidance for deployment feasibility.

The architectural implication is that tool-level self-speculation via cache sharing and dynamic gating is orthogonal and composable with lower-level speculative decoding techniques, including draft models, n-gram drafting, or dynamic early exit. The method's training-free nature and generic deployment suggest applicability to new agent workloads without retraining cost or workflow engineering.

Future Directions

Spork currently limits speculation to read-only, idempotent tools, sidestepping side effect management. Supporting write/non-idempotent tools would necessitate transactional semantics and agent-state checkpointing for speculative rollback—directions compatible with, e.g., Atomix. Deep engine integration and dynamic probe scheduling can further shrink probe overhead under heavy batching. Finally, systematic exploration of early intent emergence in alternative tool-call formats may delineate both failure modes and transferability boundaries for next-generation agentic systems.

Conclusion

Spork establishes that agentic LLMs often encode their next tool call well in advance of completion, and that this early intent can be harnessed for tool-level speculative execution. The methodology delivers measurable latency benefits on real-world benchmarks and is robust to model and architectural variations, provided the CoT window and schema regularity hold. Spork’s design principles and cost-analysis framework lay the foundation for further advances in agentic LLM inference acceleration under practical deployment constraints.