---
title: Jacobian-Based Update Methods
url: https://www.emergentmind.com/topics/jacobian-based-update
type: topic
---

# Jacobian-Based Update Methods

to=arxiv_search.search  大发时时彩计划 code  blocks omitted
{"query":"Jacobian-based update arXiv deep equilibrium Jacobian regularization Jacobian descent", "max_results": 10}
Jacobian-based update denotes a family of update mechanisms in which a Jacobian, a Jacobian approximation, or a Jacobian-derived surrogate is the central object used to choose, regularize, or reinterpret an iteration. In the literature, the phrase covers at least four technically distinct patterns: direct descent directions built from a vector-valued Jacobian; quasi-Newton or secant updates that replace an unavailable Jacobian by a recursively updated surrogate; stability-oriented penalties that constrain a Jacobian near a fixed point; and geometric or numerical schemes in which Jacobian determinants, flux Jacobians, or Jacobian actions define the admissible update itself [2406.16232] [2203.04138] [2106.14342]. A recurring theme is that the Jacobian is rarely used only as a matrix of partial derivatives; it is used as a local model of interaction, conditioning, conflict, transport, or deformation.

## 1. General formulation and mathematical role

For a vector-valued objective \(\mathbf f:\mathbb R^n\to\mathbb R^m\), the Jacobian is
\[
J\mathbf f(\mathbf x)=
\begin{bmatrix}
\nabla f_1(\mathbf x)^\top\\
\vdots\\
\nabla f_m(\mathbf x)^\top
\end{bmatrix},
\]
so each row is the gradient of one objective. The first-order approximation
\[
\mathbf f(\mathbf x+\mathbf y)=\mathbf f(\mathbf x)+J\mathbf f(\mathbf x)\mathbf y+o(\|\mathbf y\|)
\]
makes the Jacobian the natural local object for multi-objective update design, just as the gradient is natural in the scalar case [2406.16232]. In nonlinear least squares and inverse problems, the same role appears through residual linearization, where the Jacobian maps parameter perturbations to residual perturbations, and Levenberg–Marquardt-type steps are built from \(\mathbf J^T\mathbf J\) or an approximation thereof [2203.04138].

The same principle persists in modular programs. If \(F=F_q\circ\cdots\circ F_1\), then the total Jacobian is the chain product
\[
J_F(x)=J_{F_q}(x_{q-1})\,J_{F_{q-1}}(x_{q-2})\cdots J_{F_1}(x_0),
\]
but large-scale codes often expose only tangent and adjoint routines rather than explicit elemental Jacobians. In that setting, Jacobian-based updating becomes a problem of choosing how to propagate Jacobian-matrix and matrix-Jacobian products efficiently across a chain of subprograms [2406.11862]. This suggests that “Jacobian-based update” is not a single algorithmic template but a broader design principle: an update is Jacobian-based whenever local derivative structure, exact or approximate, is the primary control variable.

## 2. Direct update rules from exact Jacobian structure

The most literal instance is Jacobian Descent, introduced as the direct analogue of gradient descent for vector-valued objectives. Its generic update is
\[
\mathbf x_{t+1}=\mathbf x_t-\eta\,\mathcal A(J\mathbf f(\mathbf x_t)),
\]
where an aggregator \(\mathcal A:\mathbb R^{m\times n}\to\mathbb R^n\) converts the row-wise objective gradients into a single direction [2406.16232]. The proposed UPGRAD aggregator projects each individual row-gradient into the dual cone of directions that do not conflict with any objective and then averages the projected gradients:
\[
\mathrm{UPGRAD}(J)=\frac{1}{m}\sum_{i=1}^m \operatorname{proj}_J(J^\top e_i).
\]
The method is designed to be non-conflicting, linear under scaling, and weighted, with a convergence theorem in the smooth convex setting stating that the objective values converge to the Pareto front under a prescribed stepsize [2406.16232].

A more specialized recent formulation appears in multi-objective prompt optimization for frozen LLMs. There, the trainable variables are two perturbation groups \(h\) and \(w\), and the relevant local object is a \(2\times2\) block Jacobian
\[
J=
\begin{bmatrix}
\nabla_h \text{ob}_1 & \nabla_w \text{ob}_1\\
\nabla_h \text{ob}_2 & \nabla_w \text{ob}_2
\end{bmatrix}.
\]
The paper compresses this into a six-component J6 score vector consisting of four squared gradient norms and two cross-path alignment terms, then uses either hard routing by \(\arg\max\) or soft routing through temperature-scaled softmax weights and sharpened coefficients in updates of the form
\[
\Delta h = -\eta_h \left( \alpha_0 \nabla_h \text{ob}_1 + \alpha_3 \nabla_h \text{ob}_2 \right), \qquad
\Delta w = -\eta_w \left( \alpha_2 \nabla_w \text{ob}_1 + \alpha_4 \nabla_w \text{ob}_2 \right).
\]
The paper also notes an indexing inconsistency across sections, so the six components are stable but their order is not [2508.12086].

In sparse adversarial optimization, Jacobian-based updates take a discrete rather than continuous form. JSMA, WJSMA, and TJSMA recompute output Jacobians at each perturbed input, build saliency scores from partial derivatives, select one coordinate or a pair of coordinates, and then perturb only those features. WJSMA replaces the raw non-target sum by a probability-weighted sum, and TJSMA further multiplies derivative contributions by the remaining room before a feature saturates [2007.06032]. In this line of work, the Jacobian does not define a global descent direction; it ranks sparse coordinate updates.

## 3. Jacobian approximation, secant updating, and Jacobian-free variants

Many applications require Jacobian-based updates precisely because exact Jacobians are unavailable or too costly. In a Jacobian-free deterministic inverse-problem solver, the nonlinear least-squares objective
\[
S(\mathbf x,\bm\beta)=\tfrac12\sum_{i=1}^m r_i(x_i,\bm\beta)^2
\]
is optimized by Levenberg–Marquardt, but the residual Jacobian \(\mathbf J\) is never formed explicitly. Instead, the method maintains a Broyden matrix \(\mathbf B_k\cong\mathbf J\) updated by
\[
\mathbf B_k=\mathbf B_{k-1}+\frac{(\mathbf t_k-\mathbf B_{k-1}\mathbf s_k)\mathbf s_k^\top}{\|\mathbf s_k\|^2},
\]
with \(\mathbf t_k=\mathbf r_k-\mathbf r_{k-1}\) and \(\mathbf s_k=\bm\beta_k-\bm\beta_{k-1}\). The resulting step is accepted by Armijo backtracking, and the implemented algorithm does not periodically recompute the exact Jacobian; periodic refresh is mentioned only as an external hybrid possibility [2203.04138]. In this usage, a Jacobian-based update means that the update law is still organized around Jacobian information, but only through secant-consistent low-rank surrogates.

A structurally richer variant appears in nonlinear optimal control. In adjoint-based SQP for multiple shooting and lifted collocation, the stagewise Jacobian blocks
\[
A_i^k \approx \frac{\partial F_i}{\partial w_i}(w_i^k)
\]
are updated by a sparsity-preserving two-sided rank-one formula
\[
A^{k+1}_{i} = A^{k}_{i} + \alpha_i^k \left(y_i^k - A^{k}_{i}s_{i}^k\right)\left(\gamma_i^{k^\top} - \sigma_{i}^{k^\top} A^{k}_{i} \right),
\]
using primal secant information \((s_i^k,y_i^k)\) and adjoint information \((\sigma_i^k,\gamma_i^k)\) [1903.08720]. The update can enforce either the forward condition \(A_i^{k+1}s_i^k=y_i^k\) or the adjoint condition \(\sigma_i^{k^\top}A_i^{k+1}=\gamma_i^{k^\top}\), with skip conditions preventing instability when denominators are small. The paper proves convergence of the approximate Jacobians on the null space of the active constraints and states that the asymptotic local contraction rate matches exact-Jacobian Gauss–Newton SQP [1903.08720].

Variational inequalities provide a third approximation regime. There the inexact Jacobian model is
\[
\|\nabla F(v)-J(v)\|\le \delta,
\]
and the update is built from the regularized local operator
\[
\Omega_v^\eta(x)=F(v)+J(v)[x-v]+\eta\delta(x-v)+5L_1\|x-v\|(x-v).
\]
The resulting VIJI method attains the gap bound
\[
O\!\left(\frac{L_1D^3}{T^{3/2}}+\frac{\delta D^2}{T}\right),
\]
matching the lower bound established for methods using only a \(\delta\)-inexact first-order oracle [2405.15990]. To reduce cost further, the paper introduces limited-memory Broyden and damped Broyden updates for \(J(v)\), with the damped variant giving a memory-independent global inexactness bound \(\delta\le 2L_0\) under the stated initialization [2405.15990].

The computational infrastructure for such methods need not be explicit matrix assembly. Matrix-Free Jacobian Chaining reformulates the classical chain-product bracketing problem so that one chooses among explicit dense multiplication, tangent propagation, and adjoint propagation for each subchain. The dynamic programming recurrence solves the dense matrix-free bracketing problem in \(\mathcal O(q^3)\), and a limited-memory version disallows adjoint subchains whose tape requirement exceeds a budget \(\overline M\) [2406.11862]. This is a Jacobian-based update mechanism at the level of derivative scheduling.

## 4. Stability-oriented Jacobian updates in implicit and equilibrium models

In deep equilibrium models, the Jacobian of the hidden-state update map is the central object governing both forward fixed-point convergence and backward implicit differentiation. A DEQ replaces explicit depth
\[
z^{[i+1]}=f_\theta(z^{[i]};x), \qquad L\to\infty
\]
by an equilibrium condition
\[
z^\star=f_\theta(z^\star;x).
\]
The relevant Jacobian is
\[
J_f=\frac{\partial f_\theta(z,x)}{\partial z},
\]
evaluated at \(z^\star\), and the backward gradient depends on
\[
\left(I-J_{f_\theta}(z^\star)\right)^{-1}.
\]
If \(J_f\) is poorly conditioned near equilibrium, then forward solves become expensive and brittle, and the backward fixed-point iteration for implicit gradients becomes noisy or divergent [2106.14342].

The proposed Jacobian-based update does not alter the equilibrium equation itself. Instead, training is modified by adding a Hutchinson-estimated Frobenius penalty,
\[
\mathcal L_{\text{total}}(z^\star)=\mathcal L_{\text{orig}}(z^\star)+\gamma \frac{\|\epsilon^\top J_{f_\theta}(z^\star)\|_2^2}{d},
\]
or, in the stochastic implementation,
\[
\mathcal L_\text{total}(z^\star)=\mathcal L_\text{orig}(z^\star)+\tau\cdot\gamma
\frac{\sum_{m=1}^{M}\|\epsilon^\top J_{f_\theta}(z^\star)\|_2^2}{Md}, \qquad \tau\sim \mathrm{Bernoulli}(p).
\]
This is a Jacobian-based update in the sense that the learned dynamics are biased toward smaller local amplification without imposing a hard monotonicity or Lipschitz architecture [2106.14342].

Empirically, the paper reports fewer function evaluations and more stable backward solves. On WikiText-103, DEQ-Transformers drop from about 30 inference NFEs to 12–14, reducing training time from \(3.1\times\) Transformer-XL to about \(1.5\times\); on CIFAR-10, multiscale DEQ inference NFEs drop from 17 to 6; on ImageNet, NFEs roughly halve, e.g. \(27\to 14\) and \(30\to 15\) [2106.14342]. The regularizer adds roughly a 25–30% net increase in training memory, can cause a small drop in final accuracy or perplexity if applied too strongly, and is therefore used stochastically with probabilities such as \(p=0.35\) on WikiText-103, \(0.05\) on CIFAR-10, and \(0.1\) on ImageNet [2106.14342]. A central result is that early stopping at lower NFEs is not a substitute: it can yield much worse generalization or divergence, whereas Jacobian regularization changes training so that low-NFE solves become genuinely more accurate and stable [2106.14342].

## 5. Stochastic, sketched, and table-based Jacobian updates

In finite-sum optimization, Jacobian-based update has acquired a different but highly systematic meaning. JacSketch maintains an explicit estimate
\[
J^k\in\mathbb R^{d\times n}
\]
of the component-gradient Jacobian
\[
\nabla F(x)=[\nabla f_1(x),\ldots,\nabla f_n(x)].
\]
At iteration \(k\), a random sketch matrix \(S_k\) is sampled, the measurement \(\nabla F(x^k)S_k\) is computed, and the Jacobian estimate is updated by the sketch-and-project rule
\[
J^{k+1}=\arg\min_J \|J-J^k\|_{W^{-1}}^2
\quad \text{subject to} \quad
J S_k=\nabla F(x^k)S_k,
\]
with closed form
\[
J^{k+1}=J^{k}+(\nabla F(x^k)-J^{k})\Pi_{S_k}, \qquad
\Pi_S=S(S^\top W S)^\dagger S^\top W.
\]
The updated \(J^k\) then acts as a control variate in an unbiased gradient estimator [1805.02632].

This reinterpretation unifies several variance-reduction methods. For singleton sketches, the Jacobian update reduces exactly to the SAGA table replacement rule; for minibatch sketches it gives minibatch SAGA; for reduced-memory sketches it averages block columns [1805.02632]. The general convergence theorem uses the Lyapunov function
\[
\Psi^k=\|x^k-x^*\|_2^2+\frac{\alpha}{2\mathcal L_2}\|J^k-\nabla F(x^*)\|_{W^{-1}}^2
\]
and yields linear convergence under one-point strong convexity, expected smoothness, and unbiased sketching. In this setting, a Jacobian-based update means that stochastic optimization is organized around learning the full component-gradient matrix over time rather than sampling gradients and discarding the rest of the information [1805.02632].

A plausible implication is that Jacobian-based update can be understood as matrix learning at the derivative level. In quasi-Newton methods the learned object is usually a Hessian or inverse Hessian; in JacSketch it is the Jacobian of component gradients, and the stochastic direction is assembled from that learned matrix rather than from a single freshly sampled row [1805.02632].

## 6. Discretization, geometry, and transformation reconstruction

In numerical PDEs, Jacobian-based update refers to time-stepping formulas in which temporal advancement is expressed through flux Jacobians and Hessians. In the Picard integration formulation for hyperbolic conservation laws,
\[
U_t + F(U)_x + G(U)_y = 0,
\]
the third-order one-step update is built from time-averaged fluxes
\[
F^{appx} = F^n + \frac{\Delta t}{2} F_t^n + \frac{\Delta t^2}{6} F_{tt}^n,
\]
where
\[
F_t = -\frac{\partial F}{\partial U}(F_x+G_y).
\]
Thus the method is explicitly Jacobian-based because third-order temporal accuracy requires both flux Jacobians \(\partial F/\partial U\), \(\partial G/\partial U\) and flux Hessians \(\partial^2F/\partial U^2\), \(\partial^2G/\partial U^2\) [2006.00096]. The same paper then shows how these derivative tensors can be avoided in implementation by directional finite-difference approximations of Jacobian-vector and Hessian-vector-vector products. The mathematics remains Jacobian-based even when the implementation is Jacobian-free.

A geometrically different meaning appears in diffeomorphism reconstruction. One paper defines the average of diffeomorphisms \(\Phi_i\) by averaging Jacobian determinants and curl fields,
\[
J(\Phi)=\sum_i w_i J(\Phi_i), \qquad \operatorname{curl}(\Phi)=\sum_i w_i \operatorname{curl}(\Phi_i),
\]
and then reconstructing a map by minimizing
\[
E(\Phi,f_0,g_0)=\frac12\int_\Omega \big[(J(\Phi)-f_0)^2 + (\operatorname{curl}(\Phi)-g_0)^2\big]\,dx
\]
subject to a div-curl or Poisson-type constraint on the displacement field [1611.03946]. The motivation is explicit: Euclidean averaging of transformations can collapse cells or mis-handle local rotation, whereas Jacobian determinant and curl are intended to preserve local size and rotation.

A closely related analytical note sharpens the same point. Prescribing a positive Jacobian determinant alone does not uniquely determine a diffeomorphism. The paper conjectures that Jacobian determinant plus curl-vector, together with boundary conditions, should determine the transformation uniquely, and gives a small-perturbation uniqueness argument near the identity by reducing the problem to
\[
\operatorname{div}(u)=\mathcal F(u), \qquad \operatorname{curl}(u)=0
\]
and then to an elliptic bootstrap estimate [1712.03443]. In this literature, Jacobian-based update is therefore inseparable from the question of what information the Jacobian determinant does not encode.

## 7. Recurring trade-offs, misconceptions, and technical cautions

A common misconception is that Jacobian-based update necessarily requires materializing the full Jacobian. Several of the methods above show the opposite. JacSketch stores a structured Jacobian estimate but touches only the sketched part each iteration; Matrix-Free Jacobian Chaining works only with tangent and adjoint products; J6 uses four per-objective gradient blocks rather than a dense output Jacobian; and the DEQ regularizer uses Hutchinson trace estimation instead of explicit spectral computations [1805.02632] [2406.11862] [2508.12086] [2106.14342]. In many applications, the practically relevant quantity is not \(J\) itself but \(Jv\), \(w^\top J\), a block row, a block column, or a scalar surrogate such as \(\|J\|_F\) or \(\det J\).

A second recurring trade-off is between fidelity and cost. Quasi-Newton surrogates reduce the expense of exact Jacobian evaluation, but their quality can drift; the inverse-problem paper explicitly notes that pure Broyden updates can be less accurate than a freshly computed Jacobian and that no periodic refresh is implemented in the core method [2203.04138]. In DEQs, Jacobian regularization improves conditioning and reduces NFEs, but if applied too strongly it constrains the function class and can slightly hurt final task performance [2106.14342]. In variational inequalities, the lower bound makes the same point in oracle form: Jacobian inexactness contributes an unavoidable \(\delta D^2/T\) term, so second-order gains depend directly on how accurate the Jacobian model is [2405.15990].

A third caution is geometric. For transformations involving the Moore–Penrose inverse \(Y=X^+\), full-rank Jacobian formulas such as
\[
(dY)=|X^\top X|^{-n}(dX)
\]
or
\[
(dY)=|XX^\top|^{-m}(dX)
\]
apply only in the full-rank case with Lebesgue measure. In the non-full-rank case, the correct Jacobian must be interpreted on the rank-\(q\) manifold with a Hausdorff or factorized measure, and naive determinant formulas derived in ambient coordinates can be only formal [1911.01596]. This suggests that Jacobian-based update rules are not purely algebraic objects; their meaning can depend on the geometry and measure of the underlying space.

Finally, repeated Jacobian evaluation remains a central bottleneck in stiff simulation. In chemical kinetics, implicit algorithms generally require frequent Jacobian matrix evaluations, which is why analytical Jacobian generators such as pyJac were developed for detailed combustion mechanisms [1605.03262]. A plausible implication is that Jacobian-based update methods form an ecosystem rather than a single research thread: update rules, regularizers, secant surrogates, matrix-free derivative propagation, and analytical generators all address the same bottleneck from different sides.

Source: https://www.emergentmind.com/topics/jacobian-based-update