Jacobian-Based Update Methods

• Jacobian-based update is a design principle that uses exact or approximate derivative information to guide iterative updates in multi-objective and equilibrium models.
• It encompasses various methods, including direct descent, quasi-Newton surrogates, and stability regularization, to optimize computational performance.
• These techniques balance fidelity and cost, often employing matrix-free or sketched approaches to ensure robust convergence in high-dimensional problems.

to=arxiv_search.search 大发时时彩计划 code blocks omitted {"query":"Jacobian-based update arXiv deep equilibrium Jacobian regularization Jacobian descent", "max_results": 10} Jacobian-based update denotes a family of update mechanisms in which a Jacobian, a Jacobian approximation, or a Jacobian-derived surrogate is the central object used to choose, regularize, or reinterpret an iteration. In the literature, the phrase covers at least four technically distinct patterns: direct descent directions built from a vector-valued Jacobian; quasi-Newton or secant updates that replace an unavailable Jacobian by a recursively updated surrogate; stability-oriented penalties that constrain a Jacobian near a fixed point; and geometric or numerical schemes in which Jacobian determinants, flux Jacobians, or Jacobian actions define the admissible update itself (Quinton et al., 2024, Piro et al., 2022, Bai et al., 2021). A recurring theme is that the Jacobian is rarely used only as a matrix of partial derivatives; it is used as a local model of interaction, conditioning, conflict, transport, or deformation.

1. General formulation and mathematical role

For a vector-valued objective $\mathbf f:\mathbb R^n\to\mathbb R^m$, the Jacobian is

$$J\mathbf f(\mathbf x)= \begin{bmatrix} \nabla f_1(\mathbf x)^\top\ \vdots\ \nabla f_m(\mathbf x)^\top \end{bmatrix},$$

so each row is the gradient of one objective. The first-order approximation

$$\mathbf f(\mathbf x+\mathbf y)=\mathbf f(\mathbf x)+J\mathbf f(\mathbf x)\mathbf y+o(\|\mathbf y\|)$$

makes the Jacobian the natural local object for multi-objective update design, just as the gradient is natural in the scalar case (Quinton et al., 2024). In nonlinear least squares and inverse problems, the same role appears through residual linearization, where the Jacobian maps parameter perturbations to residual perturbations, and Levenberg–Marquardt-type steps are built from $\mathbf J^T\mathbf J$ or an approximation thereof (Piro et al., 2022).

The same principle persists in modular programs. If $F=F_q\circ\cdots\circ F_1$, then the total Jacobian is the chain product

$$J_F(x)=J_{F_q}(x_{q-1})\,J_{F_{q-1}}(x_{q-2})\cdots J_{F_1}(x_0),$$

but large-scale codes often expose only tangent and adjoint routines rather than explicit elemental Jacobians. In that setting, Jacobian-based updating becomes a problem of choosing how to propagate Jacobian-matrix and matrix-Jacobian products efficiently across a chain of subprograms (Naumann, 2024). This suggests that “Jacobian-based update” is not a single algorithmic template but a broader design principle: an update is Jacobian-based whenever local derivative structure, exact or approximate, is the primary control variable.

2. Direct update rules from exact Jacobian structure

The most literal instance is Jacobian Descent, introduced as the direct analogue of gradient descent for vector-valued objectives. Its generic update is

$$\mathbf x_{t+1}=\mathbf x_t-\eta\,\mathcal A(J\mathbf f(\mathbf x_t)),$$

where an aggregator $\mathcal A:\mathbb R^{m\times n}\to\mathbb R^n$ converts the row-wise objective gradients into a single direction (Quinton et al., 2024). The proposed UPGRAD aggregator projects each individual row-gradient into the dual cone of directions that do not conflict with any objective and then averages the projected gradients: $$\mathrm{UPGRAD}(J)=\frac{1}{m}\sum_{i=1}^m \operatorname{proj}_J(J^\top e_i).$$ The method is designed to be non-conflicting, linear under scaling, and weighted, with a convergence theorem in the smooth convex setting stating that the objective values converge to the Pareto front under a prescribed stepsize (Quinton et al., 2024).

A more specialized recent formulation appears in multi-objective prompt optimization for frozen LLMs. There, the trainable variables are two perturbation groups $h$ and $$J\mathbf f(\mathbf x)= \begin{bmatrix} \nabla f_1(\mathbf x)^\top\ \vdots\ \nabla f_m(\mathbf x)^\top \end{bmatrix},$$0, and the relevant local object is a $$J\mathbf f(\mathbf x)= \begin{bmatrix} \nabla f_1(\mathbf x)^\top\ \vdots\ \nabla f_m(\mathbf x)^\top \end{bmatrix},$$1 block Jacobian

$$J\mathbf f(\mathbf x)= \begin{bmatrix} \nabla f_1(\mathbf x)^\top\ \vdots\ \nabla f_m(\mathbf x)^\top \end{bmatrix},$$2

The paper compresses this into a six-component J6 score vector consisting of four squared gradient norms and two cross-path alignment terms, then uses either hard routing by $$J\mathbf f(\mathbf x)= \begin{bmatrix} \nabla f_1(\mathbf x)^\top\ \vdots\ \nabla f_m(\mathbf x)^\top \end{bmatrix},$$3 or soft routing through temperature-scaled softmax weights and sharpened coefficients in updates of the form

$$J\mathbf f(\mathbf x)= \begin{bmatrix} \nabla f_1(\mathbf x)^\top\ \vdots\ \nabla f_m(\mathbf x)^\top \end{bmatrix},$$4

The paper also notes an indexing inconsistency across sections, so the six components are stable but their order is not (Wu, 16 Aug 2025).

In sparse adversarial optimization, Jacobian-based updates take a discrete rather than continuous form. JSMA, WJSMA, and TJSMA recompute output Jacobians at each perturbed input, build saliency scores from partial derivatives, select one coordinate or a pair of coordinates, and then perturb only those features. WJSMA replaces the raw non-target sum by a probability-weighted sum, and TJSMA further multiplies derivative contributions by the remaining room before a feature saturates (Combey et al., 2020). In this line of work, the Jacobian does not define a global descent direction; it ranks sparse coordinate updates.

3. Jacobian approximation, secant updating, and Jacobian-free variants

Many applications require Jacobian-based updates precisely because exact Jacobians are unavailable or too costly. In a Jacobian-free deterministic inverse-problem solver, the nonlinear least-squares objective

$$J\mathbf f(\mathbf x)= \begin{bmatrix} \nabla f_1(\mathbf x)^\top\ \vdots\ \nabla f_m(\mathbf x)^\top \end{bmatrix},$$5

is optimized by Levenberg–Marquardt, but the residual Jacobian $$J\mathbf f(\mathbf x)= \begin{bmatrix} \nabla f_1(\mathbf x)^\top\ \vdots\ \nabla f_m(\mathbf x)^\top \end{bmatrix},$$6 is never formed explicitly. Instead, the method maintains a Broyden matrix $$J\mathbf f(\mathbf x)= \begin{bmatrix} \nabla f_1(\mathbf x)^\top\ \vdots\ \nabla f_m(\mathbf x)^\top \end{bmatrix},$$7 updated by

$$J\mathbf f(\mathbf x)= \begin{bmatrix} \nabla f_1(\mathbf x)^\top\ \vdots\ \nabla f_m(\mathbf x)^\top \end{bmatrix},$$8

with $$J\mathbf f(\mathbf x)= \begin{bmatrix} \nabla f_1(\mathbf x)^\top\ \vdots\ \nabla f_m(\mathbf x)^\top \end{bmatrix},$$9 and $$\mathbf f(\mathbf x+\mathbf y)=\mathbf f(\mathbf x)+J\mathbf f(\mathbf x)\mathbf y+o(\|\mathbf y\|)$$0. The resulting step is accepted by Armijo backtracking, and the implemented algorithm does not periodically recompute the exact Jacobian; periodic refresh is mentioned only as an external hybrid possibility (Piro et al., 2022). In this usage, a Jacobian-based update means that the update law is still organized around Jacobian information, but only through secant-consistent low-rank surrogates.

A structurally richer variant appears in nonlinear optimal control. In adjoint-based SQP for multiple shooting and lifted collocation, the stagewise Jacobian blocks

$$\mathbf f(\mathbf x+\mathbf y)=\mathbf f(\mathbf x)+J\mathbf f(\mathbf x)\mathbf y+o(\|\mathbf y\|)$$1

are updated by a sparsity-preserving two-sided rank-one formula

$$\mathbf f(\mathbf x+\mathbf y)=\mathbf f(\mathbf x)+J\mathbf f(\mathbf x)\mathbf y+o(\|\mathbf y\|)$$2

using primal secant information $$\mathbf f(\mathbf x+\mathbf y)=\mathbf f(\mathbf x)+J\mathbf f(\mathbf x)\mathbf y+o(\|\mathbf y\|)$$3 and adjoint information $$\mathbf f(\mathbf x+\mathbf y)=\mathbf f(\mathbf x)+J\mathbf f(\mathbf x)\mathbf y+o(\|\mathbf y\|)$$4 (Hespanhol et al., 2019). The update can enforce either the forward condition $$\mathbf f(\mathbf x+\mathbf y)=\mathbf f(\mathbf x)+J\mathbf f(\mathbf x)\mathbf y+o(\|\mathbf y\|)$$5 or the adjoint condition $$\mathbf f(\mathbf x+\mathbf y)=\mathbf f(\mathbf x)+J\mathbf f(\mathbf x)\mathbf y+o(\|\mathbf y\|)$$6, with skip conditions preventing instability when denominators are small. The paper proves convergence of the approximate Jacobians on the null space of the active constraints and states that the asymptotic local contraction rate matches exact-Jacobian Gauss–Newton SQP (Hespanhol et al., 2019).

Variational inequalities provide a third approximation regime. There the inexact Jacobian model is

$$\mathbf f(\mathbf x+\mathbf y)=\mathbf f(\mathbf x)+J\mathbf f(\mathbf x)\mathbf y+o(\|\mathbf y\|)$$7

and the update is built from the regularized local operator

$$\mathbf f(\mathbf x+\mathbf y)=\mathbf f(\mathbf x)+J\mathbf f(\mathbf x)\mathbf y+o(\|\mathbf y\|)$$8

The resulting VIJI method attains the gap bound

$$\mathbf f(\mathbf x+\mathbf y)=\mathbf f(\mathbf x)+J\mathbf f(\mathbf x)\mathbf y+o(\|\mathbf y\|)$$9

matching the lower bound established for methods using only a $\mathbf J^T\mathbf J$0-inexact first-order oracle (Agafonov et al., 2024). To reduce cost further, the paper introduces limited-memory Broyden and damped Broyden updates for $\mathbf J^T\mathbf J$1, with the damped variant giving a memory-independent global inexactness bound $\mathbf J^T\mathbf J$2 under the stated initialization (Agafonov et al., 2024).

The computational infrastructure for such methods need not be explicit matrix assembly. Matrix-Free Jacobian Chaining reformulates the classical chain-product bracketing problem so that one chooses among explicit dense multiplication, tangent propagation, and adjoint propagation for each subchain. The dynamic programming recurrence solves the dense matrix-free bracketing problem in $\mathbf J^T\mathbf J$3, and a limited-memory version disallows adjoint subchains whose tape requirement exceeds a budget $\mathbf J^T\mathbf J$4 (Naumann, 2024). This is a Jacobian-based update mechanism at the level of derivative scheduling.

4. Stability-oriented Jacobian updates in implicit and equilibrium models

In deep equilibrium models, the Jacobian of the hidden-state update map is the central object governing both forward fixed-point convergence and backward implicit differentiation. A DEQ replaces explicit depth

$\mathbf J^T\mathbf J$5

by an equilibrium condition

$\mathbf J^T\mathbf J$6

The relevant Jacobian is

$\mathbf J^T\mathbf J$7

evaluated at $\mathbf J^T\mathbf J$8, and the backward gradient depends on

$\mathbf J^T\mathbf J$9

If $F=F_q\circ\cdots\circ F_1$0 is poorly conditioned near equilibrium, then forward solves become expensive and brittle, and the backward fixed-point iteration for implicit gradients becomes noisy or divergent (Bai et al., 2021).

The proposed Jacobian-based update does not alter the equilibrium equation itself. Instead, training is modified by adding a Hutchinson-estimated Frobenius penalty,

$F=F_q\circ\cdots\circ F_1$1

or, in the stochastic implementation,

$F=F_q\circ\cdots\circ F_1$2

This is a Jacobian-based update in the sense that the learned dynamics are biased toward smaller local amplification without imposing a hard monotonicity or Lipschitz architecture (Bai et al., 2021).

Empirically, the paper reports fewer function evaluations and more stable backward solves. On WikiText-103, DEQ-Transformers drop from about 30 inference NFEs to 12–14, reducing training time from $F=F_q\circ\cdots\circ F_1$3 Transformer-XL to about $F=F_q\circ\cdots\circ F_1$4; on CIFAR-10, multiscale DEQ inference NFEs drop from 17 to 6; on ImageNet, NFEs roughly halve, e.g. $F=F_q\circ\cdots\circ F_1$5 and $F=F_q\circ\cdots\circ F_1$6 (Bai et al., 2021). The regularizer adds roughly a 25–30% net increase in training memory, can cause a small drop in final accuracy or perplexity if applied too strongly, and is therefore used stochastically with probabilities such as $F=F_q\circ\cdots\circ F_1$7 on WikiText-103, $F=F_q\circ\cdots\circ F_1$8 on CIFAR-10, and $F=F_q\circ\cdots\circ F_1$9 on ImageNet (Bai et al., 2021). A central result is that early stopping at lower NFEs is not a substitute: it can yield much worse generalization or divergence, whereas Jacobian regularization changes training so that low-NFE solves become genuinely more accurate and stable (Bai et al., 2021).

5. Stochastic, sketched, and table-based Jacobian updates

In finite-sum optimization, Jacobian-based update has acquired a different but highly systematic meaning. JacSketch maintains an explicit estimate

$$J_F(x)=J_{F_q}(x_{q-1})\,J_{F_{q-1}}(x_{q-2})\cdots J_{F_1}(x_0),$$0

of the component-gradient Jacobian

$$J_F(x)=J_{F_q}(x_{q-1})\,J_{F_{q-1}}(x_{q-2})\cdots J_{F_1}(x_0),$$1

At iteration $$J_F(x)=J_{F_q}(x_{q-1})\,J_{F_{q-1}}(x_{q-2})\cdots J_{F_1}(x_0),$$2, a random sketch matrix $$J_F(x)=J_{F_q}(x_{q-1})\,J_{F_{q-1}}(x_{q-2})\cdots J_{F_1}(x_0),$$3 is sampled, the measurement $$J_F(x)=J_{F_q}(x_{q-1})\,J_{F_{q-1}}(x_{q-2})\cdots J_{F_1}(x_0),$$4 is computed, and the Jacobian estimate is updated by the sketch-and-project rule

$$J_F(x)=J_{F_q}(x_{q-1})\,J_{F_{q-1}}(x_{q-2})\cdots J_{F_1}(x_0),$$5

with closed form

$$J_F(x)=J_{F_q}(x_{q-1})\,J_{F_{q-1}}(x_{q-2})\cdots J_{F_1}(x_0),$$6

The updated $$J_F(x)=J_{F_q}(x_{q-1})\,J_{F_{q-1}}(x_{q-2})\cdots J_{F_1}(x_0),$$7 then acts as a control variate in an unbiased gradient estimator (Gower et al., 2018).

This reinterpretation unifies several variance-reduction methods. For singleton sketches, the Jacobian update reduces exactly to the SAGA table replacement rule; for minibatch sketches it gives minibatch SAGA; for reduced-memory sketches it averages block columns (Gower et al., 2018). The general convergence theorem uses the Lyapunov function

$$J_F(x)=J_{F_q}(x_{q-1})\,J_{F_{q-1}}(x_{q-2})\cdots J_{F_1}(x_0),$$8

and yields linear convergence under one-point strong convexity, expected smoothness, and unbiased sketching. In this setting, a Jacobian-based update means that stochastic optimization is organized around learning the full component-gradient matrix over time rather than sampling gradients and discarding the rest of the information (Gower et al., 2018).

A plausible implication is that Jacobian-based update can be understood as matrix learning at the derivative level. In quasi-Newton methods the learned object is usually a Hessian or inverse Hessian; in JacSketch it is the Jacobian of component gradients, and the stochastic direction is assembled from that learned matrix rather than from a single freshly sampled row (Gower et al., 2018).

6. Discretization, geometry, and transformation reconstruction

In numerical PDEs, Jacobian-based update refers to time-stepping formulas in which temporal advancement is expressed through flux Jacobians and Hessians. In the Picard integration formulation for hyperbolic conservation laws,

$$J_F(x)=J_{F_q}(x_{q-1})\,J_{F_{q-1}}(x_{q-2})\cdots J_{F_1}(x_0),$$9

the third-order one-step update is built from time-averaged fluxes

$$\mathbf x_{t+1}=\mathbf x_t-\eta\,\mathcal A(J\mathbf f(\mathbf x_t)),$$0

where

$$\mathbf x_{t+1}=\mathbf x_t-\eta\,\mathcal A(J\mathbf f(\mathbf x_t)),$$1

Thus the method is explicitly Jacobian-based because third-order temporal accuracy requires both flux Jacobians $$\mathbf x_{t+1}=\mathbf x_t-\eta\,\mathcal A(J\mathbf f(\mathbf x_t)),$$2, $$\mathbf x_{t+1}=\mathbf x_t-\eta\,\mathcal A(J\mathbf f(\mathbf x_t)),$$3 and flux Hessians $$\mathbf x_{t+1}=\mathbf x_t-\eta\,\mathcal A(J\mathbf f(\mathbf x_t)),$$4, $$\mathbf x_{t+1}=\mathbf x_t-\eta\,\mathcal A(J\mathbf f(\mathbf x_t)),$$5 (Lee et al., 2020). The same paper then shows how these derivative tensors can be avoided in implementation by directional finite-difference approximations of Jacobian-vector and Hessian-vector-vector products. The mathematics remains Jacobian-based even when the implementation is Jacobian-free.

A geometrically different meaning appears in diffeomorphism reconstruction. One paper defines the average of diffeomorphisms $$\mathbf x_{t+1}=\mathbf x_t-\eta\,\mathcal A(J\mathbf f(\mathbf x_t)),$$6 by averaging Jacobian determinants and curl fields,

$$\mathbf x_{t+1}=\mathbf x_t-\eta\,\mathcal A(J\mathbf f(\mathbf x_t)),$$7

and then reconstructing a map by minimizing

$$\mathbf x_{t+1}=\mathbf x_t-\eta\,\mathcal A(J\mathbf f(\mathbf x_t)),$$8

subject to a div-curl or Poisson-type constraint on the displacement field (Chen et al., 2016). The motivation is explicit: Euclidean averaging of transformations can collapse cells or mis-handle local rotation, whereas Jacobian determinant and curl are intended to preserve local size and rotation.

A closely related analytical note sharpens the same point. Prescribing a positive Jacobian determinant alone does not uniquely determine a diffeomorphism. The paper conjectures that Jacobian determinant plus curl-vector, together with boundary conditions, should determine the transformation uniquely, and gives a small-perturbation uniqueness argument near the identity by reducing the problem to

$$\mathbf x_{t+1}=\mathbf x_t-\eta\,\mathcal A(J\mathbf f(\mathbf x_t)),$$9

and then to an elliptic bootstrap estimate (Zhou et al., 2017). In this literature, Jacobian-based update is therefore inseparable from the question of what information the Jacobian determinant does not encode.

7. Recurring trade-offs, misconceptions, and technical cautions

A common misconception is that Jacobian-based update necessarily requires materializing the full Jacobian. Several of the methods above show the opposite. JacSketch stores a structured Jacobian estimate but touches only the sketched part each iteration; Matrix-Free Jacobian Chaining works only with tangent and adjoint products; J6 uses four per-objective gradient blocks rather than a dense output Jacobian; and the DEQ regularizer uses Hutchinson trace estimation instead of explicit spectral computations (Gower et al., 2018, Naumann, 2024, Wu, 16 Aug 2025, Bai et al., 2021). In many applications, the practically relevant quantity is not $\mathcal A:\mathbb R^{m\times n}\to\mathbb R^n$0 itself but $\mathcal A:\mathbb R^{m\times n}\to\mathbb R^n$1, $\mathcal A:\mathbb R^{m\times n}\to\mathbb R^n$2, a block row, a block column, or a scalar surrogate such as $\mathcal A:\mathbb R^{m\times n}\to\mathbb R^n$3 or $\mathcal A:\mathbb R^{m\times n}\to\mathbb R^n$4.

A second recurring trade-off is between fidelity and cost. Quasi-Newton surrogates reduce the expense of exact Jacobian evaluation, but their quality can drift; the inverse-problem paper explicitly notes that pure Broyden updates can be less accurate than a freshly computed Jacobian and that no periodic refresh is implemented in the core method (Piro et al., 2022). In DEQs, Jacobian regularization improves conditioning and reduces NFEs, but if applied too strongly it constrains the function class and can slightly hurt final task performance (Bai et al., 2021). In variational inequalities, the lower bound makes the same point in oracle form: Jacobian inexactness contributes an unavoidable $\mathcal A:\mathbb R^{m\times n}\to\mathbb R^n$5 term, so second-order gains depend directly on how accurate the Jacobian model is (Agafonov et al., 2024).

A third caution is geometric. For transformations involving the Moore–Penrose inverse $\mathcal A:\mathbb R^{m\times n}\to\mathbb R^n$6, full-rank Jacobian formulas such as

$\mathcal A:\mathbb R^{m\times n}\to\mathbb R^n$7

or

$\mathcal A:\mathbb R^{m\times n}\to\mathbb R^n$8

apply only in the full-rank case with Lebesgue measure. In the non-full-rank case, the correct Jacobian must be interpreted on the rank-$\mathcal A:\mathbb R^{m\times n}\to\mathbb R^n$9 manifold with a Hausdorff or factorized measure, and naive determinant formulas derived in ambient coordinates can be only formal (Díaz-García et al., 2019). This suggests that Jacobian-based update rules are not purely algebraic objects; their meaning can depend on the geometry and measure of the underlying space.

Finally, repeated Jacobian evaluation remains a central bottleneck in stiff simulation. In chemical kinetics, implicit algorithms generally require frequent Jacobian matrix evaluations, which is why analytical Jacobian generators such as pyJac were developed for detailed combustion mechanisms (Niemeyer et al., 2016). A plausible implication is that Jacobian-based update methods form an ecosystem rather than a single research thread: update rules, regularizers, secant surrogates, matrix-free derivative propagation, and analytical generators all address the same bottleneck from different sides.