Exactly Linearizable Deep Models

• Exactly linearizable deep models are neural systems whose nonlinear functions become precisely linear under specific transformations, ensuring exact representations rather than approximations.
• Instance-wise linearization in piecewise linear networks guarantees that, within a fixed activation region, the model behaves as an exact affine map, enhancing feature attribution and local interpretability.
• Architectures leveraging induced vector spaces and transformer analysis reveal that deep models can exhibit low-order or globally linear dynamics, which facilitate optimal control and scalable inference.

Exactly linearizable deep models are neural models whose nonlinear behavior admits a linear description under a specified representation, coordinate system, activation regime, or depth-wise surrogate. In the recent literature, the phrase covers several non-equivalent constructions: dynamical models of the form $x^+=\Phi^{-1}(A\Phi(x)+B\Psi(u;x)+c)$ whose transformed dynamics are exactly linear; instance-wise reformulations of piecewise linear networks as $F(x)=W\cdot x+b$; architectures $f(x)=g_y^{-1}(A g_x(x))$ that are linear in induced vector spaces; and transformer analyses showing that, within a fixed context or across residual-stream layers, highly accurate low-order or layer-wise linear descriptions emerge without implying global Euclidean linearity (Kato et al., 24 Sep 2025, Moriyasu et al., 2023, Li et al., 2023, Berman et al., 9 Oct 2025, Nettasinghe et al., 13 Mar 2026, Razzhigaev et al., 2024).

1. Conceptual scope and meanings of exact linearizability

The literature does not use a single universal definition of exact linearizability. In control-oriented deep dynamics, exact linearizability means that a nonlinear state-update law is represented as the inverse image of a truly linear core under learned bijective state and input transformations, so that no approximation is involved in the transformed coordinates (Kato et al., 24 Sep 2025, Moriyasu et al., 2023). In instance-wise interpretation of piecewise linear networks, exactness means that once the activation pattern induced by an input is fixed, the entire forward map for that instance is exactly a matrix multiplication plus bias, $F(x)=W\cdot x+b$, on a local region sharing that pattern (Li et al., 2023). In the Linearizer framework, exactness is defined relative to non-standard vector spaces induced by invertible neural networks, so that $f(x)=g_y^{-1}(A g_x(x))$ is linear with respect to transported addition and scaling operations even though it is nonlinear in standard Euclidean coordinates (Berman et al., 9 Oct 2025).

Transformer results introduce a different, explicitly weaker notion. “As LLMs Scale, Low-order Linear Depth Dynamics Emerge” studies prompt-specific, concept-specific, low-order linear state-space surrogates across depth, and its exactness claim is local in context and perturbation magnitude rather than global in input space (Nettasinghe et al., 13 Mar 2026). “Your Transformer is Secretly Linear” reports near-perfect linear relations between consecutive residual-stream embeddings, but it explicitly distinguishes this empirical near-linearity from exact equivalence to a single global linear operator (Razzhigaev et al., 2024). “The Key to Going Linear” analyzes post hoc linearization of causal self-attention in a frozen-backbone regime and frames the result as an approximation to softmax geometry, not an exact identity (Kuzina et al., 8 Jul 2026).

These distinctions matter because the same phrase can refer to coordinate linearization of dynamics, local activation-region linearity, induced-space linearity, or approximate linear surrogates. A plausible implication is that “exactly linearizable deep models” is best understood as a family of structurally constrained representations rather than a single theorem schema.

2. Coordinate-transformed deep dynamics

A central formulation in the control literature is the discrete-time model

$$x(t+1)=\Phi^{-1}\!\big(A\Phi(x(t))+B\Psi(u(t);x(t))+c\big),$$

where $\Phi$ is a state transformation, $\Psi$ is an input transformation, and $A,B,c$ are linear dynamics parameters in transformed coordinates. Defining $\xi=\Phi(x)$ and $F(x)=W\cdot x+b$0 yields transformed dynamics

$F(x)=W\cdot x+b$1

so the nonlinear model is exactly the inverse transformation of a linear system (Kato et al., 24 Sep 2025). The same paper requires $F(x)=W\cdot x+b$2 to be a diffeomorphism in $F(x)=W\cdot x+b$3 and, for each fixed $F(x)=W\cdot x+b$4, $F(x)=W\cdot x+b$5 to be a diffeomorphism in $F(x)=W\cdot x+b$6.

A related continuous-time formulation appears in “Learning Exactly Linearizable Deep Dynamics Models,” which represents a nonlinear plant by

$F(x)=W\cdot x+b$7

together with

$F(x)=W\cdot x+b$8

The paper defines an Exactly Linearizable (EL) model by requiring $F(x)=W\cdot x+b$9 and $f(x)=g_y^{-1}(A g_x(x))$0 to be bijective in the respective arguments and $f(x)=g_y^{-1}(A g_x(x))$1 to be convex in $f(x)=g_y^{-1}(A g_x(x))$2 for each fixed $f(x)=g_y^{-1}(A g_x(x))$3 (Moriyasu et al., 2023). This construction generalizes the previous Structured Hammerstein–Wiener model by allowing output feedback in the input transform $f(x)=g_y^{-1}(A g_x(x))$4, thereby increasing representational capacity.

Both papers implement the nonlinear transforms with specialized neural architectures rather than unconstrained multilayer perceptrons. In the sign-structured discrete-time model, the input map $f(x)=g_y^{-1}(A g_x(x))$5 and state map $f(x)=g_y^{-1}(A g_x(x))$6 are compositions of monotone layers built from $f(x)=g_y^{-1}(A g_x(x))$7, $f(x)=g_y^{-1}(A g_x(x))$8, affine offsets, and strictly positive elementwise scalings obtained through $f(x)=g_y^{-1}(A g_x(x))$9, which guarantees componentwise strict monotonicity, invertibility, and the origin conditions $F(x)=W\cdot x+b$0 and $F(x)=W\cdot x+b$1 (Kato et al., 24 Sep 2025). In the continuous-time EL model, $F(x)=W\cdot x+b$2 is a bijective neural network, $F(x)=W\cdot x+b$3 is a diagonal-BNN, and $F(x)=W\cdot x+b$4 is a partially input-convex neural network, while $F(x)=W\cdot x+b$5 are parameterized by small neural networks in the disturbance $F(x)=W\cdot x+b$6 (Moriyasu et al., 2023).

The common structural claim is strong: the linear core is not a local Taylor approximation but the exact dynamics in transformed coordinates. What differs across papers is the geometry imposed on the transformations—monotonicity and sign structure in one case, convex constraint handling and disturbance dependence in the other.

3. Sign constraints, monotonicity, and control synthesis

“Modeling and Control of Deep Sign-Definite Dynamics” extends exact linearizability by imposing sign constraints on Jacobian entries, unifying monotonicity, positivity, and sign-definiteness. For a nonlinear system $F(x)=W\cdot x+b$7, the sign of each Jacobian entry is constrained to belong to prescribed sets $F(x)=W\cdot x+b$8, $F(x)=W\cdot x+b$9, and $f(x)=g_y^{-1}(A g_x(x))$0. Special cases recover fixed-sign Jacobians, Kamke monotonicity, and positivity (Kato et al., 24 Sep 2025). The key reduction theorem states that if $f(x)=g_y^{-1}(A g_x(x))$1 is componentwise strictly increasing and, for each fixed $f(x)=g_y^{-1}(A g_x(x))$2, $f(x)=g_y^{-1}(A g_x(x))$3 is componentwise strictly increasing in $f(x)=g_y^{-1}(A g_x(x))$4, then state, input, and origin sign constraints for the nonlinear exactly linearizable model reduce to the corresponding sign conditions on $f(x)=g_y^{-1}(A g_x(x))$5, $f(x)=g_y^{-1}(A g_x(x))$6, and $f(x)=g_y^{-1}(A g_x(x))$7. A second theorem states that positivity of the exactly linearizable model is equivalent to nonnegative sign conditions on $f(x)=g_y^{-1}(A g_x(x))$8, $f(x)=g_y^{-1}(A g_x(x))$9, and $$x(t+1)=\Phi^{-1}\!\big(A\Phi(x(t))+B\Psi(u(t);x(t))+c\big),$$0 under the same monotonicity assumptions.

This reduction enables architectural enforcement of physical structure. The paper parameterizes sign-definite entries by

$$x(t+1)=\Phi^{-1}\!\big(A\Phi(x(t))+B\Psi(u(t);x(t))+c\big),$$1

with analogous constructions for $$x(t+1)=\Phi^{-1}\!\big(A\Phi(x(t))+B\Psi(u(t);x(t))+c\big),$$2 and $$x(t+1)=\Phi^{-1}\!\big(A\Phi(x(t))+B\Psi(u(t);x(t))+c\big),$$3, so the desired sign pattern is satisfied structurally during training (Kato et al., 24 Sep 2025). Training minimizes the one-step prediction loss

$$x(t+1)=\Phi^{-1}\!\big(A\Phi(x(t))+B\Psi(u(t);x(t))+c\big),$$4

Because constraints are enforced by parameterization rather than penalties, gradient-based optimization preserves monotonicity and sign structure exactly.

The control consequence is that model predictive control can be written in transformed coordinates as a convex quadratic program. After the change of variables $$x(t+1)=\Phi^{-1}\!\big(A\Phi(x(t))+B\Psi(u(t);x(t))+c\big),$$5, $$x(t+1)=\Phi^{-1}\!\big(A\Phi(x(t))+B\Psi(u(t);x(t))+c\big),$$6, the dynamics become linear, state and nonnegativity constraints remain order-preserving because $$x(t+1)=\Phi^{-1}\!\big(A\Phi(x(t))+B\Psi(u(t);x(t))+c\big),$$7 and $$x(t+1)=\Phi^{-1}\!\big(A\Phi(x(t))+B\Psi(u(t);x(t))+c\big),$$8 are componentwise strictly increasing, and the resulting transformed MPC problem has a quadratic objective and affine constraints (Kato et al., 24 Sep 2025). The paper states that this yields a unique optimizer and a Lipschitz continuous control law under standard constraint qualifications. In experiments on a two-tank system and a hybrid powertrain, the proposed approach improves prediction accuracy and produces smoother control inputs than existing methods.

The continuous-time EL framework takes a different control route. It uses the learned linear internal dynamics for standard LQR design and combines them with control barrier functions and Integral Control Barrier Functions. For input-affine internal dynamics $$x(t+1)=\Phi^{-1}\!\big(A\Phi(x(t))+B\Psi(u(t);x(t))+c\big),$$9, the safety filter is implemented as a convex quadratic program in a new control variable $\Phi$0, enforcing inequalities derived from $\Phi$1 and $\Phi$2 while penalizing deviation from the nominal feedback $\Phi$3 (Moriyasu et al., 2023). On an automotive engine air-path and combustion-control task, the paper reports $\Phi$4 averages of $\Phi$5 for S–HW, $\Phi$6 for EL(A), and $\Phi$7 for EL(B), together with real-time constrained control at sampling time $\Phi$8. Here exact linearizability is not merely descriptive; it is the mechanism by which linear control and convex safety filtering become compatible with learned nonlinear models.

4. Instance-wise exact linearization of piecewise linear networks

A different notion of exact linearizability arises when the network is piecewise linear. “Instance-wise Linearization of Neural Network for Model Interpretation” observes that for a given input $\Phi$9, a network with ReLU, max pooling, and other piecewise linear components has a fixed activation pattern: every switch induced by the nonlinearities is resolved, so the forward computation becomes exactly linear in $\Psi$0 (Li et al., 2023). Writing the network as

$\Psi$1

with layer form $\Psi$2, the method replaces each layer by an input-specific affine map

$\Psi$3

and aggregates them into

$\Psi$4

The paper gives explicit constructions for fully connected layers, convolution layers interpreted as sparse fully connected layers, max-pooling layers represented by indicator matrices selecting the instance-specific maxima, skip connections reduced to $\Psi$5, and batch normalization expressed as an affine map at inference time (Li et al., 2023). For networks with piecewise linear activations, the paper states a local-region lemma: for $\Psi$6, there exists a maximum $\Psi$7 such that every $\Psi$8 with $\Psi$9 has the same activation pattern as $A,B,c$0, and the same matrix $A,B,c$1 explains both predictions. Within that region,

$A,B,c$2

The instance-wise linearization is therefore exact for the prediction and equal to the Jacobian on the corresponding linear region.

The immediate application is feature attribution. For a $A,B,c$3-output model,

$A,B,c$4

so the term $A,B,c$5 is interpreted as the exact contribution of feature $A,B,c$6 to output $A,B,c$7 on that instance (Li et al., 2023). The paper emphasizes that this attribution is not a heuristic backpropagation rule but the actual linear decomposition of the model inside the local activation region. It extends the same logic to unsupervised parametric t-SNE, where a trained neural embedding map is linearized instance-wise to explain what input features place a sample in a given low-dimensional location.

The paper also identifies limits. For modern CNNs with batch normalization, the bias term $A,B,c$8 often dominates predictions, so $A,B,c$9 alone may not recover the full output well. It further notes that non-piecewise-linear activations such as GELU, SELU, and ELU cannot use the Jacobian matrix to calculate $\xi=\Phi(x)$0 in the same manner; those cases require layer-by-layer treatment and no longer enjoy the same exact piecewise-linear interpretation (Li et al., 2023).

5. Induced vector spaces and the Linearizer framework

“Who Said Neural Networks Aren’t Linear?” reformulates exact linearizability at the level of algebraic structure rather than local activation regions or control coordinates. Given an invertible neural network $\xi=\Phi(x)$1, it defines new operations

$\xi=\Phi(x)$2

and proves that $\xi=\Phi(x)$3 is a vector space over $\xi=\Phi(x)$4 (Berman et al., 9 Oct 2025). With invertible $\xi=\Phi(x)$5 and $\xi=\Phi(x)$6, the architecture

$\xi=\Phi(x)$7

is then linear as a map from $\xi=\Phi(x)$8 to $\xi=\Phi(x)$9, where the input and output operations are induced by $F(x)=W\cdot x+b$00 and $F(x)=W\cdot x+b$01. The paper terms this construction a Linearizer.

Once linearity is defined in the induced spaces, standard linear algebra transports directly. The paper defines an induced inner product

$F(x)=W\cdot x+b$02

proves that the resulting space is a Hilbert space, and gives closed-form expressions for the adjoint,

$F(x)=W\cdot x+b$03

the SVD, and the Moore–Penrose pseudoinverse,

$F(x)=W\cdot x+b$04

all inherited from the corresponding objects of $F(x)=W\cdot x+b$05 (Berman et al., 9 Oct 2025). The composition rule is equally direct: if two Linearizers share the middle invertible map, then their composition is again a Linearizer with core matrix $F(x)=W\cdot x+b$06. This means arbitrarily deep compositions remain exactly linearizable in a fixed induced basis.

The paper uses this structure in three ways. First, for flow matching and diffusion-style models, repeated sampling steps in induced coordinates correspond to repeated multiplication by step matrices, so hundreds of sampling steps collapse into a single operator $F(x)=W\cdot x+b$07 acting in latent space. The paper reports that training diffusion models with this architecture makes the hundreds of sampling steps collapse into a single step and shows one-step samples visually matching multi-step samples on MNIST and CelebA (Berman et al., 9 Oct 2025). Second, idempotency becomes an architectural property: $F(x)=W\cdot x+b$08 for all $F(x)=W\cdot x+b$09 if and only if $F(x)=W\cdot x+b$10. Constraining $F(x)=W\cdot x+b$11 to be a projector therefore produces a globally projective generative model. Third, modular style transfer is implemented by fixing $F(x)=W\cdot x+b$12 and $F(x)=W\cdot x+b$13 while associating different styles with different linear operators $F(x)=W\cdot x+b$14, which are then interpolated algebraically.

This framework is exact in a global sense, but only relative to the induced vector spaces. The paper explicitly emphasizes that linearity is defined relative to the chosen pair of vector spaces, not as an intrinsic Euclidean property of the raw input-output map (Berman et al., 9 Oct 2025).

6. Transformer linearization and emergent low-order depth dynamics

Transformer work has introduced a distinct research program: identifying when LLMs are approximately linearizable in depth or attention while retaining pretrained behavior. “Your Transformer is Secretly Linear” analyzes residual-stream embeddings between consecutive layers in decoder-only transformers, including GPT, LLaMA, OPT, BLOOM, and Mistral, and defines a generalized Procrustes linearity score

$F(x)=W\cdot x+b$15

Across a range of open-source decoder models, the residual-stream layer transitions have scores often around $F(x)=W\cdot x+b$16, whereas linearity decreases substantially when the residual component is removed and only block outputs are considered (Razzhigaev et al., 2024). The same paper attributes much of the high residual-stream linearity to the consistently low norm of transformer block outputs relative to the residual stream. It further reports that removing or linearly approximating some of the most linear blocks causes only minor increases in perplexity and modest degradation on downstream tasks, and that cosine-similarity-based regularization in pretraining improves TinyStories and SuperGLUE while decreasing without-residual linearity.

A more systems-theoretic result appears in “As LLMs Scale, Low-order Linear Depth Dynamics Emerge.” For a fixed prompt $F(x)=W\cdot x+b$17, it treats transformer depth as discrete time and the last-token hidden state as the state variable, linearizes the frozen-context map $F(x)=W\cdot x+b$18 around the prompt-specific operating trajectory, and then projects the Jacobian dynamics onto a reduced basis whose first direction is the concept direction and whose remaining directions are built by a Krylov-style reachability procedure (Nettasinghe et al., 13 Mar 2026). The resulting Linear Layer Variant surrogate is a prompt-conditioned linear time-varying system

$F(x)=W\cdot x+b$19

with a final linear readout for a scalar concept score. On GPT-2-large, a 32-dimensional surrogate reproduces the layerwise sensitivity profile across ten binary NLP tasks spanning sentiment, question answering and entailment, toxicity and bias, and social media concepts. The paper reports Spearman correlations of $F(x)=W\cdot x+b$20 or $F(x)=W\cdot x+b$21 and Pearson correlations of approximately $F(x)=W\cdot x+b$22 on average across all ten datasets for $F(x)=W\cdot x+b$23. It also reports a monotonic scaling pattern across the GPT-2 family under fixed surrogate order: GPT-2 small achieves mean Spearman $F(x)=W\cdot x+b$24 and mean Pearson $F(x)=W\cdot x+b$25, GPT-2-medium $F(x)=W\cdot x+b$26 and $F(x)=W\cdot x+b$27, and GPT-2-large $F(x)=W\cdot x+b$28 and $F(x)=W\cdot x+b$29 (Nettasinghe et al., 13 Mar 2026). In this sense, larger models become more compressible and more predictable by a fixed low-order linear system within context.

A third line of work addresses linear-time inference. “The Key to Going Linear” analyzes causal softmax attention and shows that its first-order behavior can be approximated by equal weights plus a key-dependent orthogonal projection term, motivating delta-style recurrent state updates of the form used by Gated Delta Networks (Kuzina et al., 8 Jul 2026). In a strict frozen-backbone regime, the paper reports that GDN outperforms GLA, kGLA, and Hedgehog in approximating full attention and, when combined with sink tokens, short convolutions, and fixed-budget cache routing, scales across LLaMA and Qwen models up to $F(x)=W\cdot x+b$30B parameters. On Qwen3, the linearized GDN models achieve averages of $F(x)=W\cdot x+b$31 versus $F(x)=W\cdot x+b$32 at $F(x)=W\cdot x+b$33B, $F(x)=W\cdot x+b$34 versus $F(x)=W\cdot x+b$35 at $F(x)=W\cdot x+b$36B, and $F(x)=W\cdot x+b$37 versus $F(x)=W\cdot x+b$38 at $F(x)=W\cdot x+b$39B when compared with the corresponding full-attention models (Kuzina et al., 8 Jul 2026). The paper’s claim is not global exactness but analysis-driven approximation of the geometrically critical part of softmax.

Taken together, these transformer results support a narrower but influential thesis: deep transformer stacks are often not globally linear, yet they can exhibit near-linear residual evolution, accurate low-order linear depth dynamics within a context, or linear-time recurrent surrogates for attention that preserve much of the original model’s quality.

7. Boundaries, misconceptions, and open directions

A recurrent misconception is to treat all uses of linearizability as equivalent. The surveyed work does not support that view. The control papers describe exact linear dynamics in transformed coordinates and use that exactness to obtain convex quadratic programs, unique optimizers, and Lipschitz continuous control laws under stated assumptions (Kato et al., 24 Sep 2025, Moriyasu et al., 2023). The instance-wise interpretation paper proves exact affine behavior only for a fixed input and within the local activation region determined by piecewise linear activations (Li et al., 2023). The Linearizer paper proves global exact linearity only with respect to induced vector-space operations transported by invertible neural networks (Berman et al., 9 Oct 2025). Transformer studies, by contrast, emphasize locality in prompt, magnitude, task, or architectural regime, and explicitly analyze approximation errors, scaling behavior, and structural corrections rather than exact Euclidean equivalence (Nettasinghe et al., 13 Mar 2026, Razzhigaev et al., 2024, Kuzina et al., 8 Jul 2026).

The technical assumptions differ correspondingly. Sign-structured exact linearizable models rely on monotone diffeomorphisms, prescribed sign patterns, and positivity conditions on the linear core (Kato et al., 24 Sep 2025). EL dynamics models rely on bijective state and input maps, convex constrained-output maps, and disturbance-dependent linear internal dynamics (Moriyasu et al., 2023). Instance-wise exactness relies on piecewise linear activations and deterministic inference, and becomes less direct for GELU, SELU, and ELU (Li et al., 2023). Linearizers rely on tractable invertibility of $F(x)=W\cdot x+b$40 and $F(x)=W\cdot x+b$41, which the paper notes can limit engineering flexibility and scaling (Berman et al., 9 Oct 2025). Transformer linearization results are family-specific—GPT-2 for low-order depth surrogates, decoder-only models for Procrustes linearity, and LLaMA/Qwen for analysis-driven attention linearization—and they remain distribution-dependent and architecture-dependent (Nettasinghe et al., 13 Mar 2026, Razzhigaev et al., 2024, Kuzina et al., 8 Jul 2026).

The open questions follow the same fault lines. The control literature asks how far exact linearizable architectures can be extended to broader classes of physical systems and how they interact with safety certification and uncertainty handling (Moriyasu et al., 2023, Kato et al., 24 Sep 2025). The interpretation literature raises the scalability problem of constructing $F(x)=W\cdot x+b$42 explicitly for high-dimensional inputs and more complex operations (Li et al., 2023). The Linearizer framework raises questions of expressivity, null-space structure, and whether large-scale models can be trained competitively under invertibility constraints (Berman et al., 9 Oct 2025). Transformer work asks whether the observed scaling regularity can be formalized, which architectural features promote emergent linear depth dynamics, and how much of softmax can be replaced by structured linear recurrences without sacrificing long-context reasoning (Nettasinghe et al., 13 Mar 2026, Kuzina et al., 8 Jul 2026).

What unifies these otherwise distinct programs is not a single theorem, but a design principle: deep models often become more analyzable, controllable, or compressible when nonlinear computation is factored into invertible transformations, fixed activation patterns, or structurally constrained residual dynamics around a linear core. The exact scope of that principle remains architecture-specific, but the contemporary literature shows that it is already operational across model interpretation, dynamical systems, optimal control, generative modeling, and transformer analysis.