Continual Zeroth-Order Adaptation (ZOA)

• Continual ZOA is a forward-pass adaptation paradigm that updates models using function-value differences instead of backpropagated gradients.
• It is applied in settings such as PEFT-based continual learning, vision-language systems, and test-time adaptation to optimize memory and stability.
• Selective and hybrid update strategies in continual ZOA balance plasticity and stability, delivering memory savings and significant training speedups.

Continual Zeroth-Order Adaptation (ZOA) is an emerging forward-pass adaptation paradigm in which models are updated over a sequence of tasks, domains, or unlabeled test batches using function-value differences rather than backpropagated gradients. Across recent work, the term covers several closely related settings: continual learning with PEFT modules, continual vision-language learning, memory-constrained fine-tuning of LLMs, and continual test-time adaptation of quantized or full-precision models. The shared premise is that zeroth-order (ZO) optimization can reduce activation-storage costs and, in some settings, alter the stability–plasticity trade-off in ways that are not captured by viewing ZO merely as an approximate substitute for first-order (FO) descent (Deng et al., 4 Aug 2025, Yu et al., 23 Oct 2025, Liu et al., 14 Jun 2025, Wang et al., 5 Mar 2026, Yu et al., 3 Jun 2026).

1. Definition and problem setting

In the continual-learning literature, ZOA is used when adaptation must proceed under tight memory, compute, or deployment constraints and when updates occur repeatedly over time rather than in a single offline fine-tuning phase. In class-incremental and task-sequential settings, the typical architecture is a frozen pretrained backbone with a small trainable PEFT component such as prompts, adapters, or LoRA, and the core question is how to preserve prior knowledge while remaining plastic enough for new tasks (Yu et al., 23 Oct 2025). In continual vision-language learning, the same issue appears in CLIP-based systems, where the optimization problem is complicated by modality-specific instability and by the heterogeneous behavior of shallow and deep layers (Liu et al., 14 Jun 2025).

A closely related deployment setting is continual test-time adaptation. FOZO formulates test-time adaptation as an online stream of unlabeled batches $\{\mathcal{B}_t\}_{t=1}^T$, with unknown domain boundaries and no model reset when the domain changes (Wang et al., 5 Mar 2026). The quantized-model ZOA framework studies the same streaming setting for low-bit neural networks, where backpropagation-based TTA is impractical because quantized inference is discrete or non-differentiable in practice and because backward passes incur activation-storage overhead (Deng et al., 4 Aug 2025). This deployment-driven interpretation is central to the field: continual ZOA is not only a continual-learning algorithmic theme, but also a systems response to resource-limited adaptation.

The application range is already broad. Recent papers study CLIP ViT-B/16 with task-specific adapters in continual vision-language learning, frozen ViT backbones with learnable prompts for test-time adaptation, quantized ViT-B, ViT-L, ViM-S, and ResNet-50 under streaming corruptions, and LLMs such as LLaMA2-7B, Qwen3-8B, and OPT-13B under forward-only fine-tuning workloads (Liu et al., 14 Jun 2025, Wang et al., 5 Mar 2026, Deng et al., 4 Aug 2025, Yu et al., 3 Jun 2026, Li et al., 27 May 2026). This suggests that “continual ZOA” is best understood as a family of forward-only or hybrid adaptation schemes rather than a single algorithm.

2. Optimization formulations and adaptation objects

The canonical ZO update in this literature is the two-point finite-difference estimator. In the MeZO-style formulation, at iteration $t$ with parameters $\theta_t$, minibatch $\mathcal{B}_t$, and random perturbation $z_t$,

$$\widehat{g}_t = \frac{ \mathcal{L}(\theta_t + \epsilon z_t;\mathcal{B}_t) - \mathcal{L}(\theta_t - \epsilon z_t;\mathcal{B}_t) }{2\epsilon} z_t,$$

followed by

$\theta_{t+1} = \theta_t - \eta_t \widehat{g}_t.$

Here $\epsilon$ is the perturbation scale and $\eta_t$ is the learning rate (Yu et al., 3 Jun 2026). FOZO uses the same SPSA-style principle for prompt optimization, averages over $n$ perturbations, and updates only the prompt while leaving the ViT backbone frozen (Wang et al., 5 Mar 2026).

The quantized-model ZOA framework adopts a one-sided SPSA estimator,

$t$0

and sets $t$1, yielding exactly two forward passes: one evaluation at the current parameters and one at the perturbed parameters (Deng et al., 4 Aug 2025). That paper also perturbs two parameter groups simultaneously, a newly learnable domain parameter set $t$2 and aggregation coefficients $t$3, so the same two forward passes suffice for both (Deng et al., 4 Aug 2025).

What is adapted varies sharply across papers. FOZO updates only learnable prompts inserted before the first ViT layer (Wang et al., 5 Mar 2026). ZO-FC applies ZO to a single adapter-based PEFT module while keeping the classifier trained with FO SGD (Yu et al., 23 Oct 2025). The continual vision-LLM work interleaves ZO and FO across branches or layers of CLIP-based architectures (Liu et al., 14 Jun 2025). Dominant-Layer ZO restricts perturbations and updates to one decoding layer in an LLM, while the base model otherwise remains frozen (Yu et al., 3 Jun 2026). This diversity of adaptation objects is one of the field’s defining properties: continual ZOA is less a commitment to a fixed parameterization than a commitment to forward-only or selectively hybrid update rules.

The unsupervised or task loss used for adaptation is likewise setting-specific. FOZO minimizes a combination of deep-shallow feature-statistics alignment and prediction entropy,

$t$4

while the quantized ZOA framework uses the FOA-style loss combining entropy or prediction consistency with feature-statistics alignment to source-domain statistics (Wang et al., 5 Mar 2026, Deng et al., 4 Aug 2025). In LLM fine-tuning and PEFT-based continual learning, the objective is the downstream task loss evaluated by repeated forward scoring under perturbed parameters (Yu et al., 23 Oct 2025, Li et al., 27 May 2026).

3. Selective and hybrid update strategies

A major development in continual ZOA is the move away from naive full-model ZO. In continual vision-language learning, “dual w/ ZO” degrades performance substantially and produces oscillatory loss trajectories, with reported drops of roughly 6%–11% in Last accuracy and 6%–13% in Avg accuracy relative to the FO baseline, depending on dataset and adapter type (Liu et al., 14 Jun 2025). The proposed remedy is selective asymmetry: apply ZO to either the vision branch or the language branch and FO to the complementary branch. The same work further introduces a layer-wise paradigm—Hop-odd, Hop-even, Prefix, and Suffix—in which ZO and FO are interleaved across layers, with Hop-odd and Hop-even reported as the strongest settings (Liu et al., 14 Jun 2025). The paper attributes this to modality-specific instability and to heterogeneous learning dynamics across depth.

A closely related hybrid design appears in ZO-FC. That method argues that naive ZO substitution improves stability and memory efficiency but often undermines plasticity, especially when the model contains a learnable classifier (Yu et al., 23 Oct 2025). The proposed fix is component-specific: use SPSA/ZO-SGD for a single adapter-based PEFT module and keep the classifier updated with standard FO SGD. In the paper’s interpretation, the adapter is the right place for flatness-inducing smoothing, whereas the classifier requires precise directional gradients to maintain decision-boundary plasticity (Yu et al., 23 Oct 2025).

In LLM fine-tuning, update selectivity becomes even more extreme. “Dominant-Layer ZO: A Single Layer Dominates Zeroth-Order Fine-Tuning of LLMs” reports that ZO fine-tuning is sharply dominated by a single decoding layer and that fine-tuning this dominant layer alone can match or even exceed full-model ZO fine-tuning (Yu et al., 3 Jun 2026). The dominant layer is reported as task-agnostic but model-specific and can be identified before training through an inference-only analysis of activation outliers; specifically, it aligns with the first activation-outlier layer in the pretrained model. The paper gives LLaMA2-7B: layer 1 and Qwen3-8B: layer 6 as examples and reports average improvements over full-model MeZO of 0.57% on LLaMA2-7B and 1.15% on Qwen3-8B, together with up to 4.52$t$5 training speedup (Yu et al., 3 Jun 2026).

These results materially reshape the design space of continual ZOA. Rather than treating forward-only adaptation as a blanket optimizer replacement, recent work treats it as a targeted perturbation mechanism that can be assigned to selected branches, layers, adapters, prompts, or even a single dominant decoding block (Liu et al., 14 Jun 2025, Yu et al., 23 Oct 2025, Yu et al., 3 Jun 2026). A plausible implication is that continual ZOA is moving toward structurally sparse adaptation loci rather than full-parameter perturbation.

4. Continual memory, retention, and domain knowledge reuse

One branch of the literature makes continuality explicit by storing and reusing adaptation knowledge across domains. The quantized-model ZOA framework defines a domain knowledge vector after adapting to domain $t$6 as

$t$7

stores these vectors in a memory bank $t$8, and decomposes the current parameters as

$t$9

The aggregation coefficients $\theta_t$0 are normalized by softmax, and the method uses a statistics-based domain-shift detector together with a redundancy-based memory eviction rule based on cosine similarity between stored domain vectors (Deng et al., 4 Aug 2025). In the reported ablation, Save all stores 4015 vectors with 86.2 MB and yields 62.2 accuracy, whereas DKM (ours) keeps 32 vectors, uses 0.7 MB, and yields 61.3 accuracy (Deng et al., 4 Aug 2025). The same paper reports that long-term gains come primarily from this continual knowledge mechanism: V1 (zero-order adaptation only) gives 59.7 / 60.2 at first/last round, whereas ZOA full gives 59.7 / 62.8 (Deng et al., 4 Aug 2025).

A different but related persistence mechanism appears in systems work on LLM ZO fine-tuning. “LLM Zeroth-Order Fine-Tuning is an Inference Workload” represents the base model, accumulated updates, and current perturbation as a composed LoRA adapter state, with the algebraic form

$\theta_t$1

or equivalently

$\theta_t$2

The paper’s “slot” view reserves the first $\theta_t$3 slots for accumulated adaptation history and the last slot for the current probe or perturbation (Li et al., 27 May 2026). This turns repeated adaptation into mutable adapter state rather than repeated rewriting of base weights. The authors explicitly connect this to inference-time training, where updates are lightweight, composable, and schedulable like inference jobs (Li et al., 27 May 2026).

Taken together, these papers describe two complementary notions of memory in continual ZOA: retained optimization state inside the optimizer or adapter, and explicit retained domain knowledge as reusable deltas or composed low-rank blocks (Deng et al., 4 Aug 2025, Li et al., 27 May 2026). This suggests that continuality in ZOA is not only about sequential data exposure, but also about how historical adaptation is stored, merged, and reactivated.

5. Theoretical accounts of stability, forgetting, and adaptivity

A central theoretical claim in this area is that ZO can reduce forgetting because it biases optimization toward flatter minima. “More Than Memory Savings: Zeroth-Order Optimization Mitigates Forgetting in Continual Learning” states the chain

$\theta_t$4

where $\theta_t$5 is the Hessian of the old-task loss at task $\theta_t$6 and $\theta_t$7 measures forgetting (Yu et al., 23 Oct 2025). The same paper derives the smoothed objective

$\theta_t$8

and shows

$\theta_t$9

In that account, ZO descent implicitly penalizes curvature via the Hessian trace, improving stability but often reducing plasticity because finite-difference gradients are noisy and convergence is slower (Yu et al., 23 Oct 2025).

A more geometric explanation is given by “Why Zeroth-Order Adaptation May Forget Less: A Randomized Shaping Theory” (Shu et al., 11 May 2026). Under norm-matched ZO with shape operator $\mathcal{B}_t$0, the paper proves the identity

$\mathcal{B}_t$1

where $\mathcal{B}_t$2 and $\mathcal{B}_t$3 (Shu et al., 11 May 2026). The interpretation is that the isotropic retention floor is preserved while only the anisotropic component is contracted. Projecting this onto the incoming gradient $\mathcal{B}_t$4 yields the FO–ZO quadratic forgetting gap

$\mathcal{B}_t$5

so ZO improves mean forgetting precisely when the FO direction has above-average retention curvature (Shu et al., 11 May 2026). This is a conditional rather than universal advantage.

Adaptivity theory has developed along a partly separate line. “Why Does Adaptive Zeroth-Order Optimization Work?” argues that the empirical standard deviation of queried function values is, with high probability, proportional to the norm of the stochastic gradient, and therefore supports adaptive normalization by $\mathcal{B}_t$6 in updates of the form

$\mathcal{B}_t$7

or

$\mathcal{B}_t$8

in the stochastic setting (Ye et al., 2 Feb 2026). Under generalized $\mathcal{B}_t$9-smoothness with respect to an $z_t$0-norm, the paper derives explicit convergence and query complexity bounds with weak dimension dependence through $z_t$1 and $z_t$2 (Ye et al., 2 Feb 2026).

Other adaptive ZO papers disagree on what form of adaptivity is worthwhile. R-AdaZO refines the second moment by using the squared first moment,

$z_t$3

instead of squared raw ZO gradients, and argues that the first moment acts as a variance reducer; the paper reports faster convergence than existing baselines such as ZO-AdaMM (Shu et al., 3 Feb 2025). By contrast, MEAZO argues that in high-dimensional LLM fine-tuning, ZO gradients lack coordinate-wise heterogeneity and therefore per-coordinate methods such as ZO-Adam offer no convergence advantage over well-tuned ZO-SGD while incurring significant memory overhead (Dbouk et al., 5 May 2026). MEAZO replaces vector-valued adaptive state with a single scalar for global step-size adaptation and is reported to match ZO-Adam’s performance with the memory footprint of ZO-SGD (Dbouk et al., 5 May 2026). The coexistence of these results underscores that “adaptive ZO” is not a single settled design principle.

6. Empirical regimes, systems execution, and open questions

Empirically, continual ZOA has been evaluated in at least four distinct regimes. In quantized-model adaptation, ZOA reports that on quantized W6A6 ViT-B on ImageNet-C, FOA achieves 51.3% average at the 10th round and ZOA achieves 56.3%, a 5.0% improvement (Deng et al., 4 Aug 2025). The same paper reports additional gains on ViT-B W8A8, ViT-B W4A4, ViM-S W8A8, ViT-L W8A8, ViT-B W3A3, and ResNet-50 W2A4, while using only 2 forward passes per test sample and maintaining very low memory overhead; for ViT-B W8A8, reported memory is 207 MB, and the domain memory at $z_t$4 is about 0.7 MB (Deng et al., 4 Aug 2025).

In continual test-time adaptation with frozen ViT backbones and prompt tuning, FOZO reports 59.52% Top-1 accuracy on ImageNet-C (5K, level 5), outperforming the stated SOTA forward-only FOA result of 58.13%, and also reports 64.1 on ImageNet-R and 50.5 on ImageNet-Sketch (Wang et al., 5 Mar 2026). With larger forward-pass budgets, FOZO reports 62.60 at FP=26 and 62.67 at FP=28 on ImageNet-C, above the cited gradient-based and forward-only baselines in that table (Wang et al., 5 Mar 2026). The same paper emphasizes compatibility with quantized INT8 models, reporting 58.00 average for FOZO versus 57.07 for FOA and 56.91 for ZOA on the quoted PTQ4ViT setting (Wang et al., 5 Mar 2026).

In continual vision-language learning, the hybrid FO–ZO method reports state-of-the-art performance on four benchmarks while reducing memory consumption by 89.1% compared to baselines (Liu et al., 14 Jun 2025). In on-device continual learning with PEFT modules, ZO-FC reports about a 6× reduction in peak training memory, with memory around 0.7 GB versus multiple GB for FO PEFT baselines, while often recovering much of the performance lost by direct ZO substitution and exhibiting particularly low forgetting (Yu et al., 23 Oct 2025). These results support the view that the principal empirical value of continual ZOA lies in balancing memory, stability, and sufficient plasticity rather than in uniformly maximizing single-task adaptation speed.

The systems interpretation of LLM ZO has become increasingly important. “LLM Zeroth-Order Fine-Tuning is an Inference Workload” reports that on OPT-13B SST-2, a vLLM, LoRA-only path completes the 20k-step run in 0.51 h versus 4.15 h for the official matched LoRA-only baseline, an 8.13× speedup, while reaching 0.922 final evaluation accuracy and 0.931 final full-validation accuracy (Li et al., 27 May 2026). In scaling experiments across OPT-1.3B to OPT-13B, reported speedups range from 2.3×–2.9× up to 3.8×–7.7×, and for OPT-13B the paper states that scoring accounts for about 94%–99% of the measured vLLM step as batch size increases (Li et al., 27 May 2026). This systems result complements the algorithmic selectivity of dominant-layer ZO, which reports up to 4.52× end-to-end training speedup but also notes that forward-pass cost remains the dominant component (Yu et al., 3 Jun 2026).

Several limitations recur across the literature. ZO-FC emphasizes a stability–plasticity–efficiency trilemma: ZO improves stability and memory efficiency but may underfit new tasks under limited budgets (Yu et al., 23 Oct 2025). The continual vision-language work shows that naive full-ZO is incompatible with modality-specific stability requirements (Liu et al., 14 Jun 2025). Dominant-Layer ZO explicitly notes a remaining gap to FO fine-tuning, limited model coverage, and the absence of combinations with advanced ZO optimizers such as ZO-AdaMM or FZOO (Yu et al., 3 Jun 2026). The runtime work on inference-style LLM adaptation notes that evidence is concentrated on one main end-to-end case, that long-context or generation-heavy workloads may expose different bottlenecks, and that not every online ZO estimate should necessarily become a lasting update (Li et al., 27 May 2026). Collectively, these caveats indicate that continual ZOA is best viewed as a rapidly developing design space whose strongest current results come from selective, structured, and deployment-aware use of zeroth-order updates rather than from indiscriminate replacement of FO optimization.