Adversarially-Aligned Jacobian Regularization

• AAJR is a family of techniques that shapes the input–output Jacobian along adversarially relevant directions to strategically control local sensitivity and decision boundaries.
• It utilizes methods such as cosine similarity, norm duality, and optimal transport to align gradients with threat-model perturbations, thereby reducing adversarial transferability.
• Empirical studies show that AAJR variants improve robustness across datasets with measurable gains in accuracy under adversarial conditions, despite increased computational overhead.

Adversarially-Aligned Jacobian Regularization (AAJR) denotes a family of Jacobian-based regularization schemes that explicitly shape the input–output Jacobian of a model in directions that are most relevant to adversarial perturbations, rather than only shrinking sensitivity isotropically. In broad terms, this includes threat-model-aligned Jacobian norm penalties, cosine-based alignment or misalignment of Jacobians across models, adversarial alignment of Jacobian-derived saliency with natural images, optimal-transport-guided Jacobian projections along adversarial feature trajectories, and trajectory-aligned directional Jacobian control in minimax training (Fidel et al., 2020, Adam et al., 2019, Chan et al., 2019, Le et al., 2023, Mumcu et al., 4 Mar 2026). Across these formulations, the central premise is that adversarial robustness is governed by Jacobian geometry: magnitudes determine local sensitivity, while directions, angles, and shared singular structure determine whether adversarial perturbations are transferable, universal, or optimization-stable.

1. Genealogy of the concept

AAJR is best understood as the convergence of several Jacobian-centered research threads. Early Jacobian regularization work treated robustness as a smoothness problem and penalized the Frobenius norm of the input–output Jacobian to enlarge local margins and reduce sensitivity to perturbations (Jakubovitz et al., 2018, Hoffman et al., 2019). In parallel, work on adversarial transferability showed that transfer between models is largely governed by the geometry of their input–output Jacobians, especially the angle between gradients and the mismatch in gradient norms (Adam et al., 2019). This introduced a directly geometric view in which Jacobian direction, not only Jacobian size, becomes a robustness variable.

Subsequent work made that geometric viewpoint progressively more adversarially specific. "Jacobian Adversarially Regularized Networks for Robustness" adversarially regularized the input Jacobian of the loss so that Jacobian-derived images resemble natural training images, thereby enforcing salient and image-like gradients without generating adversarial training examples (Chan et al., 2019). "Adversarial robustness via stochastic regularization of neural activation sensitivity" combined a stochastic perturbation-based regularizer in an $\ell_2$ neighborhood with Jacobian regularization, yielding what it explicitly describes as a concrete instance of the design philosophy later called AAJR (Fidel et al., 2020). "Jacobian Regularization for Mitigating Universal Adversarial Perturbations" connected universal vulnerability to norms of stacked Jacobians and to cross-sample Jacobian similarity, thereby extending Jacobian geometry from per-example attacks to shared adversarial directions (Co et al., 2021).

Later formulations made the alignment target more explicit. "Bridging Optimal Transport and Jacobian Regularization by Optimal Trajectory for Enhanced Adversarial Defense" regularized the Jacobian along sample-specific directions derived from Sliced Wasserstein alignment between clean and adversarial representations, replacing random Jacobian projections with adversarially informative ones (Le et al., 2023). "Jacobian Norm with Selective Input Gradient Regularization for Improved and Interpretable Adversarial Defense" combined Jacobian norm control with selective suppression of non-salient input gradients, using perturbation-based saliency to preserve interpretable predictions under attack (Liu et al., 2022). The term AAJR itself is used explicitly in "Robustness of Agentic AI Systems via Adversarially-Aligned Jacobian Regularization," where it denotes trajectory-aligned directional Jacobian control along adversarial ascent directions in minimax training (Mumcu et al., 4 Mar 2026). In parallel, a formal robust generalization theory showed that $\ell_2$- or $\ell_1$-Jacobian-regularized losses serve as approximate upper bounds on adversarially robust losses under $\ell_2$ or $\ell_\infty$ attacks, respectively (Wu et al., 2024).

2. Mathematical formulations

The common object is the input–output Jacobian. Different papers instantiate it differently. For a classifier $f$, one formulation uses the ground-truth score and defines the input–output Jacobian as

$J_f(x) \equiv \nabla_x f_{gt}(x) \in \mathbb{R}^d,$

while broader Jacobian regularization work uses the full matrix

$J_f(x)=\nabla_x f(x)\in\mathbb{R}^{C\times I}$

or $\nabla_x f(x)\in\mathbb{R}^{k\times d}$ for logits with $k$ classes (Adam et al., 2019, Hoffman et al., 2019, Wu et al., 2024). In JARN and J-SIGR, the regularized quantity is the gradient of a scalar loss or prediction with respect to the input image, treated as a Jacobian-derived saliency map (Chan et al., 2019, Liu et al., 2022).

A central AAJR-style formulation arises from pairwise Jacobian alignment. For two models with Jacobians $\ell_2$0 and $\ell_2$1, the geometric control variable is the cosine similarity

$\ell_2$2

This supports parallel, perpendicular, antiparallel, or arbitrary-angle targets. The explicit AAJR form proposed in this line is

$\ell_2$3

with the more general angle-targeted form

$\ell_2$4

Here $\ell_2$5 corresponds to parallel alignment, $\ell_2$6 to orthogonality, and $\ell_2$7 to anti-parallel alignment (Adam et al., 2019).

A second formulation aligns the Jacobian to the adversarial threat model through norm duality. For an $\ell_2$8 threat, the Jacobian-regularized surrogate loss is

$\ell_2$9

whereas for an $\ell_1$0 threat it is

$\ell_1$1

These losses are derived as approximate upper bounds on the first-order robust loss under $\ell_1$2 and $\ell_1$3 attacks, respectively, and make the alignment between adversarial norm geometry and Jacobian norm explicit (Wu et al., 2024).

A third formulation is trajectory-aligned. In robust minimax training for agentic systems, the inner adversary updates perturbations by projected gradient ascent,

$\ell_1$4

and defines normalized ascent directions

$\ell_1$5

AAJR regularizes only the directional Jacobian amplification actually used by the adversary,

$\ell_1$6

This is the most explicit statement of AAJR as trajectory-aligned directional regularization (Mumcu et al., 4 Mar 2026).

Other formulations substitute the adversarial direction by a learned alignment target. OTJR computes an optimal latent trajectory $\ell_1$7 from Sliced Wasserstein alignment between clean and adversarial features and regularizes

$\ell_1$8

thereby replacing random Jacobian projections with directions induced by adversarial feature transport (Le et al., 2023). JARN instead learns an adaptor $\ell_1$9 and discriminator so that $\ell_2$0 is indistinguishable from natural images, making Jacobian alignment distributional rather than norm-based (Chan et al., 2019).

3. Geometric mechanisms

AAJR rests on several geometric mechanisms that recur across the literature. The first is local sensitivity. A first-order expansion shows that for a small perturbation, the change in logits is governed by $\ell_2$1, so Jacobian norm controls a local Lipschitz constant and therefore the amount by which the network can move under small perturbations (Hoffman et al., 2019). In the pairwise transferability setting, gradient norm also determines how easy it is to attack a single model: the decrease in the ground-truth score under a one-step perturbation is proportional to the squared gradient norm, and the $\ell_2$2 norm lower-bounds the effect of sign-based attacks as well (Adam et al., 2019).

The second mechanism is directional overlap. When two models have nearly parallel Jacobians, perturbations crafted to reduce one model’s ground-truth score tend to reduce the other’s as well; when Jacobians are nearly orthogonal, first-order transfer is strongly reduced; when they are anti-parallel, no one-step perturbation in a convex combination of gradients can decrease both ground-truth scores simultaneously (Adam et al., 2019). The same principle extends from model pairs to data pairs. For universal adversarial perturbations, the relevant object is the stacked Jacobian $\ell_2$3, and strong universal directions arise when Jacobians of different inputs share singular vectors and have proportional singular values (Co et al., 2021). The corresponding similarity metric is

$\ell_2$4

which measures the strength of shared adversarial perturbations across inputs (Co et al., 2021).

The third mechanism is margin and boundary distance. Jacobian regularization decreases the norm of $\ell_2$5, so for fixed logit margin it increases a lower bound on the perturbation required to cross a decision boundary (Hoffman et al., 2019, Fidel et al., 2020). This is one reason Jacobian penalties are commonly interpreted as local margin enlargement. However, exact spectral norm regularization work showed that enlarged decision regions do not fully capture adversarial robustness: methods can produce similarly sized regions yet differ substantially under PGD, TPGD, or Square attack, and weight decay can produce large regions while giving poor adversarial safeguard on KMNIST (Johansson et al., 2022). This suggests that robustness depends not only on local region size but also on the structure of the loss landscape and on which Jacobian directions remain large.

A fourth mechanism is trajectory smoothness. In the agentic minimax setting, the Hessian of the inner objective decomposes into a term $\ell_2$6 plus a residual second-order policy term. Bounding $\ell_2$7 along the update direction yields an effective smoothness bound

$\ell_2$8

which in turn gives step-size conditions for stable projected gradient ascent in the inner loop (Mumcu et al., 4 Mar 2026). Here AAJR is not merely a robustness penalty; it is a curvature-control mechanism aligned to the attack trajectory.

4. Principal variants

One prominent variant is pairwise Jacobian-angle regularization. Simultaneously trained model pairs can be encouraged to have parallel, perpendicular, or antiparallel gradients by modifying a cosine-similarity loss. This framework was introduced to study and control adversarial example transferability. Parallel regularization increases transfer, whereas perpendicular regularization reduces it most effectively in the reported MNIST experiments (Adam et al., 2019). In this setting, AAJR corresponds specifically to the parallel mode; the same geometric machinery also supports adversarial misalignment.

A second variant is threat-model-aligned Jacobian norm regularization. Here the regularizer is not defined relative to another model or to a learned direction, but relative to the adversarial norm ball itself. The combination of NsLoss and JacobRegLoss is the clearest example: NsLoss samples perturbations uniformly on the surface of an $\ell_2$9 sphere of radius NsEps and penalizes expected neuron activation changes, while JacobRegLoss shrinks the input–output Jacobian and pushes decision boundaries away from the data (Fidel et al., 2020). The later robust generalization theory formalizes the same principle by pairing $\ell_\infty$0 threats with Jacobian Frobenius regularization and $\ell_\infty$1 threats with entrywise $\ell_\infty$2 Jacobian regularization (Wu et al., 2024).

A third variant is adversarially regularized saliency alignment. JARN computes the gradient of the classification loss with respect to the input, maps it through a $\ell_\infty$3 convolution plus tanh, and trains a discriminator to distinguish Jacobian-derived images from real images. The classifier and adaptor are updated to fool the discriminator, so Jacobians become more salient and image-like (Chan et al., 2019). J-SIGR extends this idea by constructing perturbation-based saliency maps from Jacobians of noisy inputs and then applying selective input gradient regularization only where saliency is below a threshold $\ell_\infty$4, thereby smoothing mostly background or non-important regions while preserving discriminative salient gradients (Liu et al., 2022).

A fourth variant is optimal-transport-aligned Jacobian regularization. OTJR uses PGD-generated adversarial samples, computes clean and adversarial representations, aligns the two empirical distributions with Sliced Wasserstein distance, and derives per-sample movement directions $\ell_\infty$5 in feature space. Jacobian regularization is then applied along $\ell_\infty$6 rather than along random output-space projections (Le et al., 2023). This makes the regularizer adversarially aligned in a literal sense: the Jacobian is penalized in the directions required to move adversarial features back toward the clean representation distribution.

A fifth variant is trajectory-aligned AAJR for agentic AI. Rather than enforcing global Lipschitz or Jacobian bounds, it constrains sensitivity only along the adversarial ascent trajectory used by the inner maximization. The resulting policy class $\ell_\infty$7 strictly contains the globally constrained class under mild conditions, implying a weakly smaller approximation gap and reduced nominal performance degradation at the same budget level (Mumcu et al., 4 Mar 2026). This formulation generalizes AAJR beyond image classification to robust multi-agent minimax optimization.

5. Optimization and implementation

AAJR implementations inherit the computational burden of Jacobian-based training: explicit Jacobians are expensive, and most practical schemes rely on Jacobian–vector products, random projections, or additional backward passes. For Jacobian Frobenius penalties, a Hutchinson-style estimator computes

$\ell_\infty$8

so each random projection requires only the gradient of a scalar $\ell_\infty$9 with respect to the input. With $f$0, the overhead is a constant factor, about $f$1 standard SGD for LeNet’ on MNIST and approximately $f$2 for ResNet-18 on CIFAR-10 (Hoffman et al., 2019).

Exact spectral Jacobian regularization replaces Frobenius penalties with the local Jacobian spectral norm and computes it by power iteration on the Jacobian using forward- and backward-mode operators within the current activation region. This is substantially tighter than layerwise spectral upper bounds and is far cheaper than forming full Jacobians and performing an SVD, while still targeting the exact local spectral norm of the input–output mapping (Johansson et al., 2022). A plausible implication is that such machinery can be reused in AAJR variants that regularize dominant singular directions rather than isotropic Jacobian energy.

Some AAJR-style methods avoid hidden-layer Jacobian backpropagation but still incur substantial forward overhead. NsLoss requires one clean forward pass and $f$3 perturbed forward passes per batch, increasing training cost by roughly a factor of $f$4; the reported configurations use $f$5 on MNIST and $f$6 on CIFAR-10 (Fidel et al., 2020). OTJR adds PGD adversarial example generation, Sliced Wasserstein computations, and a Jacobian penalty along $f$7, making each epoch slower than PGD-AT, although the reported training still converges in 100 epochs on CIFAR-100 (Le et al., 2023). JARN requires an additional backward pass to compute $f$8 and trains a small adaptor and discriminator jointly with the classifier (Chan et al., 2019).

Agentic AAJR uses inner-loop PGD and, at each inner step, computes Jacobian–vector products $f$9 without forming full Jacobians. This is often cheaper than global Jacobian regularization in high dimensions because it needs only a single vector projection per step, but unrolling the inner loop and differentiating through these directional Jacobian terms is still memory-intensive. Forward-mode AD, implicit differentiation, and memory-efficient unrolling are identified as important implementation directions (Mumcu et al., 4 Mar 2026).

6. Empirical behavior, limitations, and open directions

Empirical results across the literature support the claim that adversarially aligned Jacobian control changes robustness in predictable ways. In the pairwise transferability study on MNIST with LeNet-like models, no regularization produced mean gradient cosine similarity $J_f(x) \equiv \nabla_x f_{gt}(x) \in \mathbb{R}^d,$0, perpendicular training drove it to $J_f(x) \equiv \nabla_x f_{gt}(x) \in \mathbb{R}^d,$1, and antiparallel training drove it to $J_f(x) \equiv \nabla_x f_{gt}(x) \in \mathbb{R}^d,$2. Under IGS-1.0, perpendicular regularization reduced transfer to $J_f(x) \equiv \nabla_x f_{gt}(x) \in \mathbb{R}^d,$3, about a $J_f(x) \equiv \nabla_x f_{gt}(x) \in \mathbb{R}^d,$4 reduction versus the no-regularization transfer rates of about $J_f(x) \equiv \nabla_x f_{gt}(x) \in \mathbb{R}^d,$5–$J_f(x) \equiv \nabla_x f_{gt}(x) \in \mathbb{R}^d,$6; under CW-40 it reduced transfer to $J_f(x) \equiv \nabla_x f_{gt}(x) \in \mathbb{R}^d,$7, about a $J_f(x) \equiv \nabla_x f_{gt}(x) \in \mathbb{R}^d,$8–$J_f(x) \equiv \nabla_x f_{gt}(x) \in \mathbb{R}^d,$9 reduction versus $J_f(x)=\nabla_x f(x)\in\mathbb{R}^{C\times I}$0 in the baseline (Adam et al., 2019). These experiments also showed that antiparallel regularization did not reduce transfer in practice with strong iterative attacks, even though it is theoretically the strongest form of misalignment for one-step attacks (Adam et al., 2019).

Threat-model-aligned Jacobian schemes have also produced strong robustness gains. On CIFAR-10 at $J_f(x)=\nabla_x f(x)\in\mathbb{R}^{C\times I}$1, the combined NsLoss + JacobRegLoss model achieved $J_f(x)=\nabla_x f(x)\in\mathbb{R}^{C\times I}$2 adversarial accuracy under untargeted PGD-$J_f(x)=\nabla_x f(x)\in\mathbb{R}^{C\times I}$3, compared with $J_f(x)=\nabla_x f(x)\in\mathbb{R}^{C\times I}$4 for the standard model and $J_f(x)=\nabla_x f(x)\in\mathbb{R}^{C\times I}$5 for JacobReg-only; on HopSkipJump at the same $J_f(x)=\nabla_x f(x)\in\mathbb{R}^{C\times I}$6, JacobReg-only reached $J_f(x)=\nabla_x f(x)\in\mathbb{R}^{C\times I}$7, confirming that Jacobian regularization was the main contributor to robustness against boundary attacks (Fidel et al., 2020). For universal perturbations, Jacobian regularization reduced MNIST UER from $J_f(x)=\nabla_x f(x)\in\mathbb{R}^{C\times I}$8 to $J_f(x)=\nabla_x f(x)\in\mathbb{R}^{C\times I}$9 and Fashion-MNIST UER from $\nabla_x f(x)\in\mathbb{R}^{k\times d}$0 to $\nabla_x f(x)\in\mathbb{R}^{k\times d}$1 at almost unchanged clean test error, while also sharply lowering median Jacobian similarity across input pairs (Co et al., 2021). OTJR, which can be read as an AAJR variant based on optimal transport, reported $\nabla_x f(x)\in\mathbb{R}^{k\times d}$2 AutoAttack accuracy on CIFAR-10 and $\nabla_x f(x)\in\mathbb{R}^{k\times d}$3 on CIFAR-100, outperforming the compared baselines in that study (Le et al., 2023). J-SIGR reported $\nabla_x f(x)\in\mathbb{R}^{k\times d}$4 clean accuracy and $\nabla_x f(x)\in\mathbb{R}^{k\times d}$5 PGD accuracy on CIFAR-10, while also reducing the success rates of ZOO and Substitute attacks relative to the adversarial-training baseline (Liu et al., 2022). In the MNIST robust generalization study, Jacobian-regularized risk minimization reduced empirical Jacobian norms by orders of magnitude and improved both standard and robust test accuracy under PGD (Wu et al., 2024).

The limitations are equally consistent. Many early demonstrations were restricted to MNIST or LeNet-like architectures, and the 2019 transferability study explicitly leaves behavior on harder datasets and deeper architectures open (Adam et al., 2019). The stochastic sensitivity framework is tuned and evaluated only for $\nabla_x f(x)\in\mathbb{R}^{k\times d}$6 perturbations, and its clean-accuracy trade-off becomes nontrivial on CIFAR-10 (Fidel et al., 2020). Universal-perturbation results are on MNIST and Fashion-MNIST rather than ImageNet-scale settings (Co et al., 2021). Exact spectral Jacobian regularization shows that decision-region enlargement alone is an incomplete explanation of adversarial robustness, which cautions against reducing AAJR to a simple local-margin story (Johansson et al., 2022). The agentic AAJR formulation is currently primarily theoretical, with no detailed large-scale empirical validation on LLM-based multi-agent systems, and emphasizes memory-intensive unrolling as a major obstacle (Mumcu et al., 4 Mar 2026).

Several open directions follow directly from the existing formulations. One is to extend pairwise or cross-sample Jacobian-angle control to large ensembles and many-way Jacobian geometry, where all pairwise angles might be encouraged toward $\nabla_x f(x)\in\mathbb{R}^{k\times d}$7 to reduce multi-model transfer (Adam et al., 2019). Another is adversarial-direction-aware Jacobian regularization that explicitly targets neighborhoods of adversarial examples, rather than average cosine or norm control on clean data alone (Adam et al., 2019, Fidel et al., 2020). A third is to combine local directional AAJR with isotropic or global Jacobian penalties, using the former to focus on adversarial directions and the latter to prevent uncontrolled sensitivity in neglected subspaces; several papers suggest this hybridization implicitly, and the robust generalization theory provides a norm-dual rationale for doing so (Wu et al., 2024). A plausible implication is that future AAJR systems will be defined less by a single penalty and more by how they choose the adversarial alignment target: another model’s Jacobian, a threat-model norm ball, a clean-to-adversarial transport direction, a human-interpretable saliency structure, or an inner-loop optimization trajectory.