Jacobian Steering: Local Linear Control

• Jacobian Steering is a method that uses the Jacobian matrix to capture local linear approximations, enabling precise control of system state changes.
• It underpins applications from robotic kinematics and neural network activation steering to matrix-free numerical simulation by formulating problems as locally linear updates.
• Practical implementations include LQR-based feedback in neural settings and dynamic programming in simulations, achieving improved performance and reduced error.

Jacobian steering refers to a family of methods in control, machine learning, and numerical simulation that exploit the local linearity of complex systems—whether physical robots, neural networks, or modular computation graphs—by using their Jacobian matrices to drive the system state toward desired targets or behaviors. Across domains, the unifying motif is direct manipulation of hidden or control variables based on knowledge (exact or approximate) of the system's Jacobian, yielding interpretable, sample-efficient, and theoretically grounded interventions.

1. Core Principles of Jacobian Steering

Jacobian steering leverages first-order approximations of system dynamics, exploiting the local linearity encoded in the Jacobian matrix, $J = \frac{\partial y}{\partial x}$, to map small input changes to output changes. This approach underlies both classical robotic control and a new generation of neural network alignment methods:

• In robotic kinematics, the Jacobian maps joint velocities $\dot{q}$ to end-effector velocities $\dot{x}$ via $\dot{x} = J(q)\dot{q}$; steering involves computing pseudo-inverse Jacobian updates to reach a desired configuration, typically $\Delta q = \lambda J^{\dagger}(x^*-x)$, with $J^{\dagger}$ the Moore–Penrose pseudoinverse (Przystupa et al., 2021).
• In neural models, especially LLMs, local linearity appears in the form of near-linear transformations at each layer around a reference activation, enabling state-space modeling and optimal feedback control (Skifstad et al., 21 Apr 2026).
• In modular simulation programs, efficient computation and steering through a chain of Jacobians via tangent and adjoint modes allows scalable automatic differentiation and local manipulations across module boundaries (Naumann, 2024).

The shared insight is that a sequence of locally linear updates can steer the system along approximate geodesics in its high-dimensional state or activation space.

2. Methodological Frameworks

a. Activation Steering in Neural Networks

Activation steering manipulates hidden activations at inference to induce or suppress behaviors in fixed models. Preceding methods (e.g., additive or contrastive activation addition) functioned in open-loop, neglecting interactions between layers. Jacobian steering ("Activation-LQR" or A-LQR) models each transformer block as a locally linear map $\phi_k(z) \approx A_k z + b_k$ and constructs an explicit state-space model:

$z_{k+1} = \phi_k(z_k) + u_k$

Linearization around a nominal trajectory yields

$$\delta z_{k+1} \approx A_k\,\delta z_k + B_k\,\delta u_k$$

with $$A_k = \frac{\partial \phi_k}{\partial z}|_{z=\bar{z}_k}$$, $\dot{q}$0 (Skifstad et al., 21 Apr 2026).

A quadratic cost in the deviation from semantic setpoints is minimized using the discrete-time LQR framework, yielding closed-form feedback laws $\dot{q}$1. The gains $\dot{q}$2 are solved once via backward Riccati recursion and reused during inference, enabling predictive, closed-loop interventions with minimal computational overhead.

b. Pullback Fisher Geometry for Optimal Steering

FishBack (Wang et al., 17 May 2026) demonstrates that in neural transformers, the Euclidean metric for activation space is a poor approximation—over 97% deviation in spectral norm—compared to the pullback Fisher metric:

$\dot{q}$3

where $\dot{q}$4 is the Fisher information of the model's output layer and $\dot{q}$5 is the Jacobian from intermediate activations to logits. The optimal minimal-KL intervention along a direction $\dot{q}$6 (with $\dot{q}$7 defining an output-concept) subject to a constraint $\dot{q}$8 is

$\dot{q}$9

This closed-form solution ensures the most semantically efficient steering for a desired attribute change, minimizing off-target distributional drift.

c. Matrix-Free Jacobian Steering in Numerical Simulation

For composite differentiable programs $\dot{x}$0, Jacobian steering arises in the efficient chaining and propagation of seed vectors (tangents) and adjoints through sequences of modules. The Matrix-Free Jacobian Chaining approach (Naumann, 2024) formalizes the optimal selection of forward/reverse propagation at each submodule so as to minimize floating-point operation cost and respect global tape-memory limits, dynamically steering computation in the space of practical AD schedules.

3. Implementation Strategies and Computational Considerations

a. LQR-based Activation Steering

Implementation is staged:

1. Offline:
   • Feature directions $\dot{x}$1 are computed from contrastive datasets.
   • Setpoints $\dot{x}$2 and reference activations $\dot{x}$3 are derived.
   • Jacobians $\dot{x}$4 are computed by automatic differentiation.
   • The Riccati recursion yields feedback gains $\dot{x}$5 for all layers.
2. Online (inference):
   • For each layer, the feature error $\dot{x}$6 is measured.
   • The intervention $\dot{x}$7 is computed and injected.
   • No further backpropagation or optimization is needed, resulting in an additional 10–30% latency per forward pass (Skifstad et al., 21 Apr 2026).

b. Minimal-Distortion Geometry (FishBack)

• Layerwise computation of the pullback Fisher metric—by propagating the output Fisher information back through the chain of Jacobians—yields each layer's anisotropic geometry $\dot{x}$8.
• Steering updates involve solving $\dot{x}$9 for the desired attribute.
• Empirical results show that this spectral-optimal approach outperforms all Euclidean-metric baselines (including ActAdd and CAA), with up to $\dot{x} = J(q)\dot{q}$0 lower off-target KL at a matched concept level (Wang et al., 17 May 2026).

c. Neural Jacobian Steering in Robotics

• Learning-based Jacobian estimators ("Neural Jacobian", "Bi-directional Neural Jacobian", "Neural Kinematics") provide data-driven approximations of $\dot{x} = J(q)\dot{q}$1.
• Steering uses the inverse-Jacobian law $\dot{x} = J(q)\dot{q}$2, optionally with Tikhonov regularization to manage singularities.
• Practical protocols include dense joint-space exploration during training, monitoring of the condition number $\dot{x} = J(q)\dot{q}$3, and blending direct and learned Jacobian models for robustness (Przystupa et al., 2021).

d. Matrix-Free Chaining for Large Programs

• Apply dynamic programming across modules to tactically select between tangent or adjoint propagation per stage, respecting a global tape-memory budget $\dot{x} = J(q)\dot{q}$4.
• Greedy, block-wise heuristics are recommended for large $\dot{x} = J(q)\dot{q}$5, with practical cost reductions confirmed in simulation studies (Naumann, 2024).

4. Theoretical Guarantees and Error Analysis

For LQR-based Jacobian steering in neural networks, theoretical upper bounds on the tracking error are provided. The semiglobal bound (Theorem 5.1 in (Skifstad et al., 21 Apr 2026)) states that

$\dot{x} = J(q)\dot{q}$6

where $\dot{x} = J(q)\dot{q}$7. Projected onto the feature direction, the feature tracking error is similarly controlled.

For FishBack, the spectral cost ratio

$\dot{x} = J(q)\dot{q}$8

quantifies how much higher the KL-cost is for a given metric (e.g., Euclidean) relative to the Fisher-optimal. This cost is governed by the spectrum of $\dot{x} = J(q)\dot{q}$9 and the alignment of $\Delta q = \lambda J^{\dagger}(x^*-x)$0, providing a quantitative metric of method suboptimality (Wang et al., 17 May 2026).

For neural Jacobian learning in robotics, convergence and condition number tests (e.g., $\Delta q = \lambda J^{\dagger}(x^*-x)$1) are used to assess reliability and positive-definiteness, correlating with empirical convergence rates (Przystupa et al., 2021).

5. Empirical Performance Across Domains

a. Neural Activation Steering

• Toxicity mitigation: A-LQR reduces model toxicity rates from 4–5% to 0.1–0.2% on RealToxicityPrompts, with 3–5x improvement over open-loop methods and no loss of n-gram diversity or accuracy.
• Truthfulness: LQR-based steering achieves 10–20 percentage point lift in combined Truth × Informativeness scores over base models.
• Refusal/jailbreaking: The A-LQR⁺ variant attains $\Delta q = \lambda J^{\dagger}(x^*-x)$2 attack success rate, outperforming Angular Steering (75–85%).
• Concept modulation: Varying $\Delta q = \lambda J^{\dagger}(x^*-x)$3 in the setpoint definition modulates arbitrary concept prevalence between $\Delta q = \lambda J^{\dagger}(x^*-x)$40% and $\Delta q = \lambda J^{\dagger}(x^*-x)$5, as judged by an LLM.

b. Pullback Fisher Steering

• FishBack consistently yields off-target KL reductions of $\Delta q = \lambda J^{\dagger}(x^*-x)$6–$\Delta q = \lambda J^{\dagger}(x^*-x)$7 over Euclidean gradient ascent and $\Delta q = \lambda J^{\dagger}(x^*-x)$8 over CAA, with empirical win rates of 72–81% (p < $\Delta q = \lambda J^{\dagger}(x^*-x)$9), reflecting the metric's strong anisotropy and low effective rank (2–17% of activation space) (Wang et al., 17 May 2026).

c. Robotic Control

• Neural Jacobian-based steering in simulation achieves $J^{\dagger}$085% success rate in 7-DOF reaching, vs. $J^{\dagger}$196% for the true Jacobian.
• On a Kinova Gen-3, bi-directional neural Jacobians achieve 91% success in 7-DOF reach, outperforming LL-KNN and Broyden's method, with consistent transferability from learned models (Przystupa et al., 2021).

d. Matrix-Free Steering in Simulation

• Representative numerical experiments show order-of-magnitude fma cost reductions (up to $J^{\dagger}$2122 for $J^{\dagger}$3 modules) compared to all-forward or all-reverse AD strategies. Pareto-optimal trade-offs between total cost and memory overhead are realized by dynamic programming (Naumann, 2024).

6. Limitations and Practical Recommendations

• For neural methods, all heavy computation (Jacobian, Riccati) is performed once offline; the residual online overhead is moderate (Skifstad et al., 21 Apr 2026).
• Effective steering depends on accurate estimation of Jacobians and monitoring of matrix condition numbers; singularities or ill-conditioned Jacobians require regularization or fallback strategies (Przystupa et al., 2021).
• The pronounced anisotropy and low effective rank of neural Fisher metrics suggest that naive Euclidean steering will often be highly suboptimal (Wang et al., 17 May 2026).
• In matrix-free simulation, problem complexity is NP-complete in its most general form, but tractable in practice via dynamic programming and greedy heuristics (Naumann, 2024).
• Sensitivity to noise in perceptual observations (in robotic vision) and unmodeled higher-order nonlinearities (in neural activations) may degrade empirical performance. Online adaptation and hybrid model combinations are recommended (Przystupa et al., 2021).
• Memory limitations in large-scale simulations can be efficiently handled by dynamic allocation between tangent and adjoint modes, using local problem size as a heuristic (Naumann, 2024).

7. Connections, Extensions, and Future Directions

Jacobian steering creates a unifying bridge between optimal control, differentiable programming, and neural model alignment. The core idea—online feedback corrections based on local linearity—enables sample-efficient, interpretable, and theoretically controlled interventions in both physical and virtual systems.

Extensions include:

• Online adaptation of Jacobian estimators for drift compensation (Przystupa et al., 2021).
• Spectral diagnostics for evaluating the suitability of proxy metrics versus Fisher-pullback in neural steering (Wang et al., 17 May 2026).
• Compositional scheduling of AD strategies in simulation, exploiting block-structure for scalability (Naumann, 2024).
• A plausible implication is that as model architectures, tasks, and domains become even higher-dimensional and more multimodal, the role of explicit Jacobian-based steering—especially when paired with spectral or geometry-aware metrics—will become increasingly central both for interpretability and for efficient control.

Ongoing research continues to refine both the mathematical understanding of local linearity in high-dimensional models and the practical tools for exploiting these structures across applications.