--- title: 'Jacobian Steering: Local Linear Control' url: https://www.emergentmind.com/topics/jacobian-steering type: topic --- # Jacobian Steering: Local Linear Control Jacobian steering refers to a family of methods in control, machine learning, and numerical simulation that exploit the local linearity of complex systems—whether physical robots, neural networks, or modular computation graphs—by using their Jacobian matrices to drive the system state toward desired targets or behaviors. Across domains, the unifying motif is direct manipulation of hidden or control variables based on knowledge (exact or approximate) of the system's Jacobian, yielding interpretable, sample-efficient, and theoretically grounded interventions. ## 1. Core Principles of Jacobian Steering Jacobian steering leverages first-order approximations of system dynamics, exploiting the local linearity encoded in the Jacobian matrix, $J = \frac{\partial y}{\partial x}$, to map small input changes to output changes. This approach underlies both classical robotic control and a new generation of neural network alignment methods: - In robotic kinematics, the Jacobian maps joint velocities $\dot{q}$ to end-effector velocities $\dot{x}$ via $\dot{x} = J(q)\dot{q}$; steering involves computing pseudo-inverse Jacobian updates to reach a desired configuration, typically $\Delta q = \lambda J^{\dagger}(x^*-x)$, with $J^{\dagger}$ the Moore–Penrose pseudoinverse [2106.06083]. - In neural models, especially large language models (LLMs), local linearity appears in the form of near-linear transformations at each layer around a reference activation, enabling state-space modeling and optimal feedback control [2604.19018]. - In modular simulation programs, efficient computation and steering through a chain of Jacobians via tangent and adjoint modes allows scalable automatic differentiation and local manipulations across module boundaries [2406.11862]. The shared insight is that a sequence of locally linear updates can steer the system along approximate geodesics in its high-dimensional state or activation space. ## 2. Methodological Frameworks ### a. Activation Steering in Neural Networks Activation steering manipulates hidden activations at inference to induce or suppress behaviors in fixed models. Preceding methods (e.g., additive or contrastive activation addition) functioned in open-loop, neglecting interactions between layers. Jacobian steering ("Activation-LQR" or A-LQR) models each transformer block as a locally linear map $\phi_k(z) \approx A_k z + b_k$ and constructs an explicit state-space model: $$ z_{k+1} = \phi_k(z_k) + u_k $$ Linearization around a nominal trajectory yields $$ \delta z_{k+1} \approx A_k\,\delta z_k + B_k\,\delta u_k $$ with $A_k = \frac{\partial \phi_k}{\partial z}|_{z=\bar{z}_k}$, $B_k = I$ [2604.19018]. A quadratic cost in the deviation from semantic setpoints is minimized using the discrete-time LQR framework, yielding closed-form feedback laws $u_k = -K_k(z_k - \bar{z}_k')$. The gains $K_k$ are solved once via backward Riccati recursion and reused during inference, enabling predictive, closed-loop interventions with minimal computational overhead. ### b. Pullback Fisher Geometry for Optimal Steering FishBack [2605.17231] demonstrates that in neural transformers, the Euclidean metric for activation space is a poor approximation—over 97% deviation in spectral norm—compared to the pullback Fisher metric: $$ g(x) = J(x)^\top F(\theta) J(x) $$ where $F(\theta)$ is the Fisher information of the model's output layer and $J(x)$ is the Jacobian from intermediate activations to logits. The optimal minimal-KL intervention along a direction $q = J^\top \beta$ (with $\beta$ defining an output-concept) subject to a constraint $q^\top \delta x = \rho$ is $$ \delta x^* = \frac{\rho}{q^\top g^{-1} q} g^{-1} q $$ This closed-form solution ensures the most semantically efficient steering for a desired attribute change, minimizing off-target distributional drift. ### c. Matrix-Free Jacobian Steering in Numerical Simulation For composite differentiable programs $F = F_q \circ \cdots \circ F_1$, Jacobian steering arises in the efficient chaining and propagation of seed vectors (tangents) and adjoints through sequences of modules. The Matrix-Free Jacobian Chaining approach [2406.11862] formalizes the optimal selection of forward/reverse propagation at each submodule so as to minimize floating-point operation cost and respect global tape-memory limits, dynamically steering computation in the space of practical AD schedules. ## 3. Implementation Strategies and Computational Considerations ### a. LQR-based Activation Steering Implementation is staged: 1. **Offline**: - Feature directions $v_k$ are computed from contrastive datasets. - Setpoints $\beta_k^*$ and reference activations $\bar{z}_k'$ are derived. - Jacobians $A_k$ are computed by automatic differentiation. - The Riccati recursion yields feedback gains $K_k$ for all layers. 2. **Online (inference)**: - For each layer, the feature error $\alpha_k = \beta_k^* - v_k^\top z_k$ is measured. - The intervention $u_k = K_k \alpha_k v_k$ is computed and injected. - No further backpropagation or optimization is needed, resulting in an additional 10–30% latency per forward pass [2604.19018]. ### b. Minimal-Distortion Geometry (FishBack) - **Layerwise computation** of the pullback Fisher metric—by propagating the output Fisher information back through the chain of Jacobians—yields each layer's anisotropic geometry $g^{(\ell)}$. - Steering updates involve solving $g^{-1}q$ for the desired attribute. - Empirical results show that this spectral-optimal approach outperforms all Euclidean-metric baselines (including ActAdd and CAA), with up to $2.5\times$ lower off-target KL at a matched concept level [2605.17231]. ### c. Neural Jacobian Steering in Robotics - Learning-based Jacobian estimators ("Neural Jacobian", "Bi-directional Neural Jacobian", "Neural Kinematics") provide data-driven approximations of $J(q)$. - Steering uses the inverse-Jacobian law $\Delta q = \lambda J^{\dagger} (x^* - x)$, optionally with Tikhonov regularization to manage singularities. - Practical protocols include dense joint-space exploration during training, monitoring of the condition number $\kappa(J)$, and blending direct and learned Jacobian models for robustness [2106.06083]. ### d. Matrix-Free Chaining for Large Programs - Apply dynamic programming across modules to tactically select between tangent or adjoint propagation per stage, respecting a global tape-memory budget $M_{\max}$. - Greedy, block-wise heuristics are recommended for large $q$, with practical cost reductions confirmed in simulation studies [2406.11862]. ## 4. Theoretical Guarantees and Error Analysis For LQR-based Jacobian steering in neural networks, theoretical upper bounds on the tracking error are provided. The semiglobal bound (Theorem 5.1 in [2604.19018]) states that $$ \|\delta z_k\| \leq \|\Phi_{k,1}\|\|\delta z_1\| + \sum_{i90\%$ attack success rate, outperforming Angular Steering (75–85%). - **Concept modulation:** Varying $\lambda$ in the setpoint definition modulates arbitrary concept prevalence between $\sim$0% and $>80\%$, as judged by an LLM. ### b. Pullback Fisher Steering - FishBack consistently yields off-target KL reductions of $1.3\times$–$2.5\times$ over Euclidean gradient ascent and $1.5\times$ over CAA, with empirical win rates of 72–81% (p < $10^{-4}$), reflecting the metric's strong anisotropy and low effective rank (2–17% of activation space) [2605.17231]. ### c. Robotic Control - Neural Jacobian-based steering in simulation achieves $\geq$85% success rate in 7-DOF reaching, vs. $\approx$96% for the true Jacobian. - On a Kinova Gen-3, bi-directional neural Jacobians achieve 91% success in 7-DOF reach, outperforming LL-KNN and Broyden's method, with consistent transferability from learned models [2106.06083]. ### d. Matrix-Free Steering in Simulation - Representative numerical experiments show order-of-magnitude fma cost reductions (up to $\times$122 for $q=500$ modules) compared to all-forward or all-reverse AD strategies. Pareto-optimal trade-offs between total cost and memory overhead are realized by dynamic programming [2406.11862]. ## 6. Limitations and Practical Recommendations - For neural methods, all heavy computation (Jacobian, Riccati) is performed once offline; the residual online overhead is moderate [2604.19018]. - Effective steering depends on accurate estimation of Jacobians and monitoring of matrix condition numbers; singularities or ill-conditioned Jacobians require regularization or fallback strategies [2106.06083]. - The pronounced anisotropy and low effective rank of neural Fisher metrics suggest that naive Euclidean steering will often be highly suboptimal [2605.17231]. - In matrix-free simulation, problem complexity is NP-complete in its most general form, but tractable in practice via dynamic programming and greedy heuristics [2406.11862]. - Sensitivity to noise in perceptual observations (in robotic vision) and unmodeled higher-order nonlinearities (in neural activations) may degrade empirical performance. Online adaptation and hybrid model combinations are recommended [2106.06083]. - Memory limitations in large-scale simulations can be efficiently handled by dynamic allocation between tangent and adjoint modes, using local problem size as a heuristic [2406.11862]. ## 7. Connections, Extensions, and Future Directions Jacobian steering creates a unifying bridge between optimal control, differentiable programming, and neural model alignment. The core idea—online feedback corrections based on local linearity—enables sample-efficient, interpretable, and theoretically controlled interventions in both physical and virtual systems. Extensions include: - **Online adaptation** of Jacobian estimators for drift compensation [2106.06083]. - **Spectral diagnostics** for evaluating the suitability of proxy metrics versus Fisher-pullback in neural steering [2605.17231]. - **Compositional scheduling** of AD strategies in simulation, exploiting block-structure for scalability [2406.11862]. - *A plausible implication is that as model architectures, tasks, and domains become even higher-dimensional and more multimodal, the role of explicit Jacobian-based steering—especially when paired with spectral or geometry-aware metrics—will become increasingly central both for interpretability and for efficient control.* Ongoing research continues to refine both the mathematical understanding of local linearity in high-dimensional models and the practical tools for exploiting these structures across applications. Source: https://www.emergentmind.com/topics/jacobian-steering