---
title: Continual Zeroth-Order Adaptation (ZOA)
url: https://www.emergentmind.com/topics/continual-zeroth-order-adaptation-zoa
type: topic
---

# Continual Zeroth-Order Adaptation (ZOA)

Continual Zeroth-Order Adaptation (ZOA) is an emerging forward-pass adaptation paradigm in which models are updated over a sequence of tasks, domains, or unlabeled test batches using function-value differences rather than backpropagated gradients. Across recent work, the term covers several closely related settings: continual learning with PEFT modules, continual vision-language learning, memory-constrained fine-tuning of LLMs, and continual test-time adaptation of quantized or full-precision models. The shared premise is that zeroth-order (ZO) optimization can reduce activation-storage costs and, in some settings, alter the stability–plasticity trade-off in ways that are not captured by viewing ZO merely as an approximate substitute for first-order (FO) descent [2508.02180] [2510.21019] [2506.12409] [2603.04733] [2606.05516].

## 1. Definition and problem setting

In the continual-learning literature, ZOA is used when adaptation must proceed under tight memory, compute, or deployment constraints and when updates occur repeatedly over time rather than in a single offline fine-tuning phase. In class-incremental and task-sequential settings, the typical architecture is a frozen pretrained backbone with a small trainable PEFT component such as prompts, adapters, or LoRA, and the core question is how to preserve prior knowledge while remaining plastic enough for new tasks [2510.21019]. In continual vision-language learning, the same issue appears in CLIP-based systems, where the optimization problem is complicated by modality-specific instability and by the heterogeneous behavior of shallow and deep layers [2506.12409].

A closely related deployment setting is continual test-time adaptation. FOZO formulates test-time adaptation as an online stream of unlabeled batches $\{\mathcal{B}_t\}_{t=1}^T$, with unknown domain boundaries and no model reset when the domain changes [2603.04733]. The quantized-model ZOA framework studies the same streaming setting for low-bit neural networks, where backpropagation-based TTA is impractical because quantized inference is discrete or non-differentiable in practice and because backward passes incur activation-storage overhead [2508.02180]. This deployment-driven interpretation is central to the field: continual ZOA is not only a continual-learning algorithmic theme, but also a systems response to resource-limited adaptation.

The application range is already broad. Recent papers study CLIP ViT-B/16 with task-specific adapters in continual vision-language learning, frozen ViT backbones with learnable prompts for test-time adaptation, quantized ViT-B, ViT-L, ViM-S, and ResNet-50 under streaming corruptions, and LLMs such as LLaMA2-7B, Qwen3-8B, and OPT-13B under forward-only fine-tuning workloads [2506.12409] [2603.04733] [2508.02180] [2606.05516] [2605.28760]. This suggests that “continual ZOA” is best understood as a family of forward-only or hybrid adaptation schemes rather than a single algorithm.

## 2. Optimization formulations and adaptation objects

The canonical ZO update in this literature is the two-point finite-difference estimator. In the MeZO-style formulation, at iteration $t$ with parameters $\theta_t$, minibatch $\mathcal{B}_t$, and random perturbation $z_t$,
$$
\widehat{g}_t = \frac{ \mathcal{L}(\theta_t + \epsilon z_t;\mathcal{B}_t) - \mathcal{L}(\theta_t - \epsilon z_t;\mathcal{B}_t) }{2\epsilon} z_t,
$$
followed by
$$
\theta_{t+1} = \theta_t - \eta_t \widehat{g}_t.
$$
Here $\epsilon$ is the perturbation scale and $\eta_t$ is the learning rate [2606.05516]. FOZO uses the same SPSA-style principle for prompt optimization, averages over $n$ perturbations, and updates only the prompt while leaving the ViT backbone frozen [2603.04733].

The quantized-model ZOA framework adopts a one-sided SPSA estimator,
$$
\hat{g}(\theta) = \hat{\nabla}\mathcal{L}(;\theta) = \frac{1}{q}\sum_{i=1}^q \frac{\mathcal{L}(;\theta + c\epsilon_i) - \mathcal{L}(;\theta)}{c}\,\epsilon_i^{-1},
$$
and sets $q=1$, yielding exactly two forward passes: one evaluation at the current parameters and one at the perturbed parameters [2508.02180]. That paper also perturbs two parameter groups simultaneously, a newly learnable domain parameter set $\tilde{\theta}$ and aggregation coefficients $\alpha$, so the same two forward passes suffice for both [2508.02180].

What is adapted varies sharply across papers. FOZO updates only learnable prompts inserted before the first ViT layer [2603.04733]. ZO-FC applies ZO to a single adapter-based PEFT module while keeping the classifier trained with FO SGD [2510.21019]. The continual vision-language model work interleaves ZO and FO across branches or layers of CLIP-based architectures [2506.12409]. Dominant-Layer ZO restricts perturbations and updates to one decoding layer in an LLM, while the base model otherwise remains frozen [2606.05516]. This diversity of adaptation objects is one of the field’s defining properties: continual ZOA is less a commitment to a fixed parameterization than a commitment to forward-only or selectively hybrid update rules.

The unsupervised or task loss used for adaptation is likewise setting-specific. FOZO minimizes a combination of deep-shallow feature-statistics alignment and prediction entropy,
$$
\mathcal{L} = \lambda \mathcal{L}_{stats} + \mathcal{L}_{ent},
$$
while the quantized ZOA framework uses the FOA-style loss combining entropy or prediction consistency with feature-statistics alignment to source-domain statistics [2603.04733] [2508.02180]. In LLM fine-tuning and PEFT-based continual learning, the objective is the downstream task loss evaluated by repeated forward scoring under perturbed parameters [2510.21019] [2605.28760].

## 3. Selective and hybrid update strategies

A major development in continual ZOA is the move away from naive full-model ZO. In continual vision-language learning, “dual w/ ZO” degrades performance substantially and produces oscillatory loss trajectories, with reported drops of roughly **6%–11% in Last accuracy** and **6%–13% in Avg accuracy** relative to the FO baseline, depending on dataset and adapter type [2506.12409]. The proposed remedy is selective asymmetry: apply ZO to either the vision branch or the language branch and FO to the complementary branch. The same work further introduces a layer-wise paradigm—Hop-odd, Hop-even, Prefix, and Suffix—in which ZO and FO are interleaved across layers, with Hop-odd and Hop-even reported as the strongest settings [2506.12409]. The paper attributes this to modality-specific instability and to heterogeneous learning dynamics across depth.

A closely related hybrid design appears in ZO-FC. That method argues that naive ZO substitution improves stability and memory efficiency but often undermines plasticity, especially when the model contains a learnable classifier [2510.21019]. The proposed fix is component-specific: use SPSA/ZO-SGD for a single adapter-based PEFT module and keep the classifier updated with standard FO SGD. In the paper’s interpretation, the adapter is the right place for flatness-inducing smoothing, whereas the classifier requires precise directional gradients to maintain decision-boundary plasticity [2510.21019].

In LLM fine-tuning, update selectivity becomes even more extreme. “Dominant-Layer ZO: A Single Layer Dominates Zeroth-Order Fine-Tuning of LLMs” reports that ZO fine-tuning is sharply dominated by a single decoding layer and that fine-tuning this dominant layer alone can match or even exceed full-model ZO fine-tuning [2606.05516]. The dominant layer is reported as task-agnostic but model-specific and can be identified before training through an inference-only analysis of activation outliers; specifically, it aligns with the first activation-outlier layer in the pretrained model. The paper gives **LLaMA2-7B: layer 1** and **Qwen3-8B: layer 6** as examples and reports average improvements over full-model MeZO of **0.57%** on LLaMA2-7B and **1.15%** on Qwen3-8B, together with up to **4.52$\times$ training speedup** [2606.05516].

These results materially reshape the design space of continual ZOA. Rather than treating forward-only adaptation as a blanket optimizer replacement, recent work treats it as a targeted perturbation mechanism that can be assigned to selected branches, layers, adapters, prompts, or even a single dominant decoding block [2506.12409] [2510.21019] [2606.05516]. A plausible implication is that continual ZOA is moving toward structurally sparse adaptation loci rather than full-parameter perturbation.

## 4. Continual memory, retention, and domain knowledge reuse

One branch of the literature makes continuality explicit by storing and reusing adaptation knowledge across domains. The quantized-model ZOA framework defines a domain knowledge vector after adapting to domain $j$ as
$$
\Delta_j = \theta_j - \theta^0,
$$
stores these vectors in a memory bank $\mathcal{T} = \{\Delta_j\}_{j=0}^{N}$, and decomposes the current parameters as
$$
\theta = \theta^0 + \sum_{j=0}^{N}\alpha_j \Delta_j + \tilde{\theta}, \qquad \Delta_j\in\mathcal{T}.
$$
The aggregation coefficients $\alpha_j$ are normalized by softmax, and the method uses a statistics-based domain-shift detector together with a redundancy-based memory eviction rule based on cosine similarity between stored domain vectors [2508.02180]. In the reported ablation, **Save all** stores **4015 vectors** with **86.2 MB** and yields **62.2** accuracy, whereas **DKM (ours)** keeps **32 vectors**, uses **0.7 MB**, and yields **61.3** accuracy [2508.02180]. The same paper reports that long-term gains come primarily from this continual knowledge mechanism: **V1 (zero-order adaptation only)** gives **59.7 / 60.2** at first/last round, whereas **ZOA full** gives **59.7 / 62.8** [2508.02180].

A different but related persistence mechanism appears in systems work on LLM ZO fine-tuning. “LLM Zeroth-Order Fine-Tuning is an Inference Workload” represents the base model, accumulated updates, and current perturbation as a composed LoRA adapter state, with the algebraic form
$$
W_{\mathrm{probe}^{\pm}} = W_0 + \sum_{i=1}^{N-1} A_i B_i^T \pm \epsilon A_p B_p^T,
$$
or equivalently
$$
W_{\mathrm{probe}^{\pm}} = W_0 + A^{\pm} B^T.
$$
The paper’s “slot” view reserves the first $N-1$ slots for accumulated adaptation history and the last slot for the current probe or perturbation [2605.28760]. This turns repeated adaptation into mutable adapter state rather than repeated rewriting of base weights. The authors explicitly connect this to inference-time training, where updates are lightweight, composable, and schedulable like inference jobs [2605.28760].

Taken together, these papers describe two complementary notions of memory in continual ZOA: retained optimization state inside the optimizer or adapter, and explicit retained domain knowledge as reusable deltas or composed low-rank blocks [2508.02180] [2605.28760]. This suggests that continuality in ZOA is not only about sequential data exposure, but also about how historical adaptation is stored, merged, and reactivated.

## 5. Theoretical accounts of stability, forgetting, and adaptivity

A central theoretical claim in this area is that ZO can reduce forgetting because it biases optimization toward flatter minima. “More Than Memory Savings: Zeroth-Order Optimization Mitigates Forgetting in Continual Learning” states the chain
$$
\text{ZO optimization} \Longrightarrow \lambda_{\max}(H_t)\downarrow \Longrightarrow F_t\downarrow,
$$
where $H_t$ is the Hessian of the old-task loss at task $t$ and $F_t$ measures forgetting [2510.21019]. The same paper derives the smoothed objective
$$
L_\varepsilon(\theta)=\mathbb{E}_{\Delta}[L(\theta+\varepsilon\Delta)]
$$
and shows
$$
L_\varepsilon(\theta)=L(\theta)+\frac{\varepsilon^2}{2}\operatorname{Tr}(\nabla^2 L(\theta))+O(\varepsilon^4).
$$
In that account, ZO descent implicitly penalizes curvature via the Hessian trace, improving stability but often reducing plasticity because finite-difference gradients are noisy and convergence is slower [2510.21019].

A more geometric explanation is given by “Why Zeroth-Order Adaptation May Forget Less: A Randomized Shaping Theory” [2605.10658]. Under norm-matched ZO with shape operator $\widetilde{A}$, the paper proves the identity
$$
\mathbb E\!\left[\widetilde{A}^\top H \widetilde{A}\right] = (1-\tau)H + \tau \bar H I,
$$
where $\bar H = \operatorname{tr}(H)/d$ and $\tau = d/(q+d+1)$ [2605.10658]. The interpretation is that the isotropic retention floor is preserved while only the anisotropic component is contracted. Projecting this onto the incoming gradient $g$ yields the FO–ZO quadratic forgetting gap
$$
\mathcal Q^{\text{FO}}-\mathbb E[\mathcal Q_q^{\text{ZO}}] = \frac{\eta^2}{2}\tau \|g\|^2(\lambda_g-\bar\lambda),
$$
so ZO improves mean forgetting precisely when the FO direction has above-average retention curvature [2605.10658]. This is a conditional rather than universal advantage.

Adaptivity theory has developed along a partly separate line. “Why Does Adaptive Zeroth-Order Optimization Work?” argues that the empirical standard deviation of queried function values is, with high probability, proportional to the norm of the stochastic gradient, and therefore supports adaptive normalization by $\sigma_t$ in updates of the form
$$
x_{t+1}=x_t-\eta\,\frac{g(x_t)}{\sigma_t}
$$
or
$$
x_{t+1}=x_t-\eta\,\frac{g(x_t)}{\sigma_t+\beta}
$$
in the stochastic setting [2602.01627]. Under generalized $(L_0,L_1)$-smoothness with respect to an $H$-norm, the paper derives explicit convergence and query complexity bounds with weak dimension dependence through $\operatorname{tr}(H)$ and $\|H\|$ [2602.01627].

Other adaptive ZO papers disagree on what form of adaptivity is worthwhile. R-AdaZO refines the second moment by using the squared first moment,
$$
v_t = \beta_2 v_{t-1} + (1 - \beta_2)m_t^2,
$$
instead of squared raw ZO gradients, and argues that the first moment acts as a variance reducer; the paper reports faster convergence than existing baselines such as ZO-AdaMM [2502.01014]. By contrast, MEAZO argues that in high-dimensional LLM fine-tuning, ZO gradients lack coordinate-wise heterogeneity and therefore per-coordinate methods such as ZO-Adam offer no convergence advantage over well-tuned ZO-SGD while incurring significant memory overhead [2605.03869]. MEAZO replaces vector-valued adaptive state with a single scalar for global step-size adaptation and is reported to match ZO-Adam’s performance with the memory footprint of ZO-SGD [2605.03869]. The coexistence of these results underscores that “adaptive ZO” is not a single settled design principle.

## 6. Empirical regimes, systems execution, and open questions

Empirically, continual ZOA has been evaluated in at least four distinct regimes. In quantized-model adaptation, ZOA reports that on **quantized W6A6 ViT-B** on ImageNet-C, FOA achieves **51.3%** average at the 10th round and ZOA achieves **56.3%**, a **5.0%** improvement [2508.02180]. The same paper reports additional gains on **ViT-B W8A8**, **ViT-B W4A4**, **ViM-S W8A8**, **ViT-L W8A8**, **ViT-B W3A3**, and **ResNet-50 W2A4**, while using only **2 forward passes per test sample** and maintaining very low memory overhead; for **ViT-B W8A8**, reported memory is **207 MB**, and the domain memory at **$N=32$** is about **0.7 MB** [2508.02180].

In continual test-time adaptation with frozen ViT backbones and prompt tuning, FOZO reports **59.52% Top-1 accuracy on ImageNet-C (5K, level 5)**, outperforming the stated SOTA forward-only FOA result of **58.13%**, and also reports **64.1** on ImageNet-R and **50.5** on ImageNet-Sketch [2603.04733]. With larger forward-pass budgets, FOZO reports **62.60** at **FP=26** and **62.67** at **FP=28** on ImageNet-C, above the cited gradient-based and forward-only baselines in that table [2603.04733]. The same paper emphasizes compatibility with quantized **INT8** models, reporting **58.00** average for FOZO versus **57.07** for FOA and **56.91** for ZOA on the quoted PTQ4ViT setting [2603.04733].

In continual vision-language learning, the hybrid FO–ZO method reports state-of-the-art performance on four benchmarks while reducing memory consumption by **89.1%** compared to baselines [2506.12409]. In on-device continual learning with PEFT modules, ZO-FC reports about a **6× reduction in peak training memory**, with memory around **0.7 GB** versus multiple GB for FO PEFT baselines, while often recovering much of the performance lost by direct ZO substitution and exhibiting particularly low forgetting [2510.21019]. These results support the view that the principal empirical value of continual ZOA lies in balancing memory, stability, and sufficient plasticity rather than in uniformly maximizing single-task adaptation speed.

The systems interpretation of LLM ZO has become increasingly important. “LLM Zeroth-Order Fine-Tuning is an Inference Workload” reports that on **OPT-13B SST-2**, a **vLLM, LoRA-only** path completes the **20k-step** run in **0.51 h** versus **4.15 h** for the official matched **LoRA-only** baseline, an **8.13×** speedup, while reaching **0.922** final evaluation accuracy and **0.931** final full-validation accuracy [2605.28760]. In scaling experiments across **OPT-1.3B** to **OPT-13B**, reported speedups range from **2.3×–2.9×** up to **3.8×–7.7×**, and for OPT-13B the paper states that scoring accounts for about **94%–99%** of the measured vLLM step as batch size increases [2605.28760]. This systems result complements the algorithmic selectivity of dominant-layer ZO, which reports up to **4.52×** end-to-end training speedup but also notes that forward-pass cost remains the dominant component [2606.05516].

Several limitations recur across the literature. ZO-FC emphasizes a stability–plasticity–efficiency trilemma: ZO improves stability and memory efficiency but may underfit new tasks under limited budgets [2510.21019]. The continual vision-language work shows that naive full-ZO is incompatible with modality-specific stability requirements [2506.12409]. Dominant-Layer ZO explicitly notes a remaining gap to FO fine-tuning, limited model coverage, and the absence of combinations with advanced ZO optimizers such as **ZO-AdaMM** or **FZOO** [2606.05516]. The runtime work on inference-style LLM adaptation notes that evidence is concentrated on one main end-to-end case, that long-context or generation-heavy workloads may expose different bottlenecks, and that not every online ZO estimate should necessarily become a lasting update [2605.28760]. Collectively, these caveats indicate that continual ZOA is best viewed as a rapidly developing design space whose strongest current results come from selective, structured, and deployment-aware use of zeroth-order updates rather than from indiscriminate replacement of FO optimization.

Source: https://www.emergentmind.com/topics/continual-zeroth-order-adaptation-zoa