LLM Zeroth-Order Fine-Tuning is an Inference Workload

Abstract: Zeroth-order (ZO) fine-tuning is attractive for LLMs because it replaces backpropagation with forward objective evaluations. Existing implementations nevertheless execute ZO algorithms inside conventional training loops, even though their dominant work is repeated scoring under nearby parameter states. This creates a workload-runtime mismatch: the algorithm asks for structured inference-style scoring, while the system exposes a sequence of fragmented training-loop steps. We show that LLM ZO fine-tuning is an inference-dominated workload and execute its repeated scoring phase through a serving runtime. On OPT-13B SST-2, the resulting vLLM execution path completes the 20k-step LoZO run in 0.51 estimated training hours versus 4.15 hours for the official LoZO baseline under the matched LoRA-only setting, an 8.13x speedup, while reaching 0.922 final evaluation accuracy and 0.931 final full-validation accuracy. In core-step scaling experiments across OPT-1.3B to OPT-13B, the same runtime reorganization gives 2.34x--7.72x speedups. A MeZO-style high-rank factorized experiment shows that the same runtime paradigm can track a MeZO-like loss trajectory while running up to 2.55x faster. More broadly, representing ZO updates as dynamic adapter states suggests a practical path toward inference-time training, where lightweight adaptation can be scheduled as an inference-like workload rather than as a separate training job.

• The paper demonstrates that reframing LLM zeroth-order fine-tuning as an inference workload yields up to 8.13× speedup and preserves convergence accuracy.
• It leverages direct worker-side scoring and in situ GPU low-rank updates to minimize memory writes and latency.
• Empirical results on OPT models show enhanced throughput, faster SST-2 training, and strong statistical fidelity with 99–100% sign agreement.

LLM Zeroth-Order Fine-Tuning as an Inference Workload

Introduction and Motivation

Zeroth-order (ZO) optimization methods for LLM fine-tuning forgo backpropagation in favor of forward-only objective evaluations, thus reducing memory and computation requirements. Yet, the majority of ZO implementations retain the conventional training-loop abstraction, which incurs substantial overhead and obfuscates the actual structure of the dominant computation: repeated forward scoring under slightly perturbed model states. This paper identifies a fundamental workload-runtime mismatch in current ZO training pipelines and makes the core claim that LLM ZO fine-tuning is more effectively executed as an inference-style workload.

Methodological Innovation: Inference-Native ZO Execution

The authors retain the LoZO optimizer's exact semantics—including the low-rank perturbation structure, random direction sampling, and update rules—while reimplementing the repeated scoring phase directly through the vLLM serving runtime. The key technical innovations enabling this shift include:

• Direct worker-side scoring: Objective evaluations (option-token log-probabilities) bypass high-level trainers, invoking vLLM worker paths to decrease latency and orchestration overhead.
• Direct update path: Low-rank parameter updates are performed in situ on GPU worker-side memory using packed QKV updates, eliminating host-device transfer bottlenecks.
• Memory-write reduction: Utilization of factor accumulation within the $\nu$-step interval when direction factors (e.g., LoRA $V$) are held fixed, such that only minimal LoRA-shaped adapters must be modified per step.

This design leverages the fact that both LoRA-style adapter updates and temporary ZO perturbations can be represented as dynamic LoRA adapter slots, seamlessly composable with the base model without explicit materialization or full-weight updates.

Empirical Results

Extensive evaluation was conducted on the OPT-13B model fine-tuned on SST-2 under LoZO; core throughput and convergence metrics were analyzed across OPT-1.3B, 2.7B, 6.7B, and 13B. Key numerical results include:

• Training Time: The vLLM-inference path ($\nu=50$) completes 20k OPT-13B LoZO (LoRA-only) steps in $0.51$ hours ($30.7$ minutes), while the official LoZO baseline requires $4.15$ hours, yielding an $8.13\times$ end-to-end speedup with equivalent or better final evaluation accuracy ($0.922$ vs $0.902$).
• Step Throughput: Core optimization-step speedups across models and batch sizes reach up to $7.7\times$, with $V$0--$V$1 range overall (Figure 1, Figure 2).
• Paradigm Transfer: MeZO-style high-rank factorized ZO evaluation (OPT-1.3B) achieves almost identical loss reduction and increased validation accuracy within $V$2--$V$3 runtime of the MeZO baseline.

Figure 1: OPT-13B SST-2 evaluation versus training time: vLLM $V$4 completes in $V$5 minutes vs $V$6 for LoZO LoRA-only baseline, $V$7 speedup.

Figure 2: OPT-13B SST-2 evaluation loss per optimization step, demonstrating matched convergence behavior across implementations.

A correctness analysis confirms statistical fidelity, with 99–100% sign agreement in high-signal regions between the vLLM and LoZO code paths.

Figure 3: Phase 1 perturbation-score correctness summary, separating high- and low-signal discrepancy distribution.

Figure 4: 300-step side-by-side convergence and per-step ZO coefficient agreement: vLLM and LoZO paths are tightly aligned.

Theoretical and System Implications

The core contribution reframes ZO fine-tuning not as a stripped-down version of backprop but as a structured pattern of inference-like calls over parameterizations that can be mapped onto LoRA adapter slots. This reconceptualization facilitates several practical advances:

• Adapter-centric adaptation: Both the persistent update and current perturbation are LoRA-shaped adapters. This abstraction enables all ZO state to reside in inference-native representations, and allows for hot swapping, batching, and scheduling analogous to batched inference.
• Serving-native adaptation: Since the immutable base model can be heavily quantized and persisted, only adapter states must remain in higher precision. This supports resource isolation, stateless serving models, and fine-grained rollout or rollback of adaptation via adapter slot management.
• Elastic and opportunistic scheduling: ZO probes (forward loss differences) may be batched, canceled, or opportunistically injected in idle GPU cycles. This enables lightweight, request-driven or feedback-driven continual adaptation in both cloud and edge settings, without explicit training job boundaries.

Furthermore, the workload view imposes a new systems criterion on ZO method design: serving compatibility becomes as essential as estimator variance or sample efficiency. Estimators that enable adapter-state-based perturbations, batching, compatibility with quantized models, preemption, and state isolation are favored for deployment.

Limitations

The strongest empirical benchmarks are limited to OPT-13B on SST-2 classification under LoZO, with limited coverage of broader task classes (e.g., generation) or comprehensive model families. The presented vLLM code path remains a minimal prototype rather than an integrated, production-ready runtime feature.

Conclusion

This paper establishes that the typical pattern of zeroth-order LLM fine-tuning is, from a systems perspective, an inference-dominated workload rather than a conventional training workload. Mapping ZO fine-tuning onto the inference runtime yields significant throughput and efficiency gains (up to $V$8 wall-clock speedup for OPT-13B LoZO), while maintaining convergence and accuracy parity. Moreover, this paradigm facilitates lightweight, adapter-scoped, and flexible adaptation that can be scheduled like inference, supporting continual or context-driven parameter updates. The results advocate for the co-design of ZO algorithms and LLM inference runtimes, with adapter-centric representations taking center stage in future LLM adaptation infrastructure.