Hardware Linearization Advances

• Hardware linearization is a set of methods that transform nonlinear or irregular computations into forms compatible with hardware primitives like GEMM, shifts, and circuit behaviors.
• It enables efficient mapping of complex algorithms onto platforms such as GPUs, analog circuits, quantum hardware, and DSP systems while preserving core operational fidelity.
• Key trade-offs involve balancing approximation errors, hardware constraints, and throughput to achieve stable and real-time computational performance.

Hardware linearization denotes a family of methods that reformulate a target computation so that it can be executed efficiently, stably, or directly by a given hardware substrate. In the recent literature, the term spans several distinct but related practices: replacing quadratic softmax attention by subquadratic mechanisms that map well to GPU tensor cores; approximating logarithmic, anti-logarithmic, and logarithmic-addition functions by piecewise-linear shift-and-add datapaths; replacing nonlinear circuit elements by linearized surrogates around an operating point to compute gradients in analog hardware; converting irregular coupling graphs into quasi-linear topologies for quantum compilation; and embedding nonlinear dynamics into large linear operators that current quantum hardware can realize (Nguyen et al., 11 Jul 2025, Xiong et al., 2020, Chaffey, 17 Sep 2025, Gao et al., 2024, Ali, 6 Feb 2026).

1. Scope and meanings of the term

In the cited literature, hardware linearization is not a single technique. It is an umbrella term for redesigning an algorithm, a numerical operator, a circuit, or a data layout so that the resulting computation aligns with the primitive operations and constraints of the target hardware. The alignment may concern asymptotic complexity, memory growth, tensor-core compatibility, bounded state, shift-and-add realizability, circuit reciprocity, or linearly addressed memory access (Nguyen et al., 11 Jul 2025, Xiong et al., 2020, 0803.2386).

Domain	What is linearized	Hardware objective
Long-context LLMs	Attention and memory mechanisms	Subquadratic compute, constant-memory inference, tensor-core efficiency
Digital arithmetic	Nonlinear functions such as log, antilog, and log-domain addition	Shift-and-add datapaths, bounded area and latency
Analog equilibrium networks	Nonlinear diode behavior around an operating point	Direct hardware computation of Jacobians and gradients
Quantum compilation	Irregular coupling graphs or nonlinear dynamics	Linear-depth schedules or linear operators compatible with quantum hardware
Array and DSP systems	Multidimensional access patterns or nonlinear plant inverses	Real-time execution, cache locality, linearly addressed memory

This diversity matters because a common misconception is to treat hardware linearization as synonymous with first-order Taylor linearization. In some works it is exactly a local Jacobian construction around an equilibrium point; in others it is a graph transformation, a piecewise-linear approximation, a recurrent-state reformulation, or a compiler-level conversion from multidimensional array semantics to explicit linear address arithmetic (Chaffey, 17 Sep 2025, Gao et al., 2024, Xiong et al., 2020, 0803.2386).

A second recurring feature is behavioral preservation. The target is usually not a new model class for its own sake, but a hardware-compatible surrogate that preserves a reference behavior: a pretrained Transformer teacher, a desired linear system response, a nonlinear arithmetic function, or the original quantum algorithmic specification (Nguyen et al., 11 Jul 2025, Hotz et al., 2014, Xiong et al., 2020).

2. Subquadratic attention and accelerator-oriented linearization in LLMs

For Transformer-based LLMs, hardware linearization means redesigning the attention and memory mechanisms so that algorithmic complexity becomes linear or subquadratic in sequence length and the computation maps efficiently onto modern accelerators, with good use of tensor cores, high throughput, and bounded memory use, without sacrificing the behavioral quality of a strong pretrained Transformer (Nguyen et al., 11 Jul 2025). In this setting, the primary obstacle is the quadratic complexity of causal softmax attention and the growing KV cache.

"Lizard: An Efficient Linearization Framework for LLMs" replaces quadratic softmax attention by a hybrid of gated linear attention and sliding window attention with meta memory (Nguyen et al., 11 Jul 2025). The linear-attention component uses a kernel of the form

$k(q,k)=\phi(q)\phi(k)^\top,$

with the Hedgehog feature map

$$\phi(\mathbf{x})=\left[\exp(\mathbf{x}W)\oplus \exp(-\mathbf{x}W)\right].$$

Gated Linear Attention introduces a learnable decay gate, and in recurrent form the global state is updated as

$$\mathbf{S}_i=\Gamma_i\mathbf{S}_{i-1}+\phi_k(\mathbf{k}_i)\mathbf{v}_i^\top,\qquad \hat{\mathbf{y}}_i=\mathbf{S}_i\phi_q(\mathbf{q}_i).$$

Because only $\mathbf{S}_i$ must be retained, the global memory cost is constant in sequence length. Lizard combines this recurrent global compressor with sliding-window softmax attention augmented by meta tokens, so that long-range dependence and local high-fidelity interaction are handled by different mechanisms within the same attention block (Nguyen et al., 11 Jul 2025).

The hardware-oriented contribution is not only asymptotic. Lizard rewrites cumulative gating in log space by absorbing the cumulative factor into transformed queries and keys,

$$\widetilde{\mathbf{Q}}=\phi_q(\mathbf{Q})\odot \exp(\log \mathbf{C}),\qquad \widetilde{\mathbf{K}}=\phi_k(\mathbf{K})\odot \exp(-\log \mathbf{C}),$$

so that the gated attention reduces to standard batched matrix multiplication in BF16. The stated consequence is that no per-chunk high-precision fallback is needed, tensor cores can be used directly, and gradients remain stable (Nguyen et al., 11 Jul 2025).

The reported empirical gains are both algorithmic and hardware-level. On an NVIDIA A100 80GB with BF16, batch size $B=16$, sequence length $S=2048$, heads $H=32$, and head dimension $D_{\text{head}}=128$, the original GLA kernel requires 4.30 ms per forward, whereas the Lizard kernel requires 3.25 ms per forward, a 32% reduction in inference time. In generation experiments, the teacher with FlashAttention-2 runs out of memory at long sequence lengths, while Lizard maintains constant memory and stable throughput up to 32K and beyond. On 5-shot MMLU, Lizard improves prior linearization methods by 8–18 points, and its scalar gate formulation reaches 61.2, higher than the heavier gating variants reported in the paper (Nguyen et al., 11 Jul 2025).

This usage of hardware linearization therefore combines approximation theory, recurrence, and kernel engineering. The point is not merely to replace softmax by a linear kernel; it is to express the resulting computation entirely as regular matrix multiplies and elementwise operations that are numerically stable and accelerator-native (Nguyen et al., 11 Jul 2025).

3. Piecewise-linear arithmetic and fused nonlinear-function blocks

A classical sense of hardware linearization is the replacement of a nonlinear arithmetic function by a piecewise-linear approximation whose coefficients are chosen to minimize hardware cost subject to an error bound. "Hardware Complexity Aware Design Strategy for a Fused Logarithmic and Anti-Logarithmic Converter" uses this strategy for $\log_2(1+x)$ and $$\phi(\mathbf{x})=\left[\exp(\mathbf{x}W)\oplus \exp(-\mathbf{x}W)\right].$$0 on 10-bit inputs and 10-bit outputs in a half-precision logarithmic number system format (Xiong et al., 2020). The approximation takes the form

$$\phi(\mathbf{x})=\left[\exp(\mathbf{x}W)\oplus \exp(-\mathbf{x}W)\right].$$1

and the design objective is not only approximation fidelity but hardware realizability under a shift-and-add architecture.

The paper defines Maximum Hardware Error as

$$\phi(\mathbf{x})=\left[\exp(\mathbf{x}W)\oplus \exp(-\mathbf{x}W)\right].$$2

uses Maximum Quantization Distance as a proxy during design, and enforces a symmetric error distribution around zero through the objective

$$\phi(\mathbf{x})=\left[\exp(\mathbf{x}W)\oplus \exp(-\mathbf{x}W)\right].$$3

Slopes are quantized in Canonic Signed Digit form, intercepts in binary fixed-point form, and two segmentation schemes are studied: recursive segmentation and MQD-flattened segmentation. The implementation is a shift-and-add datapath with an adder tree block, multiplexer block, and segment index encoder; its complexity is predicted by formulas for adder-tree latency, adder-tree area, and multiplexer-shift complexity (Xiong et al., 2020).

The architectural novelty is fusion. Because the derivatives of the log and antilog functions over $$\phi(\mathbf{x})=\left[\exp(\mathbf{x}W)\oplus \exp(-\mathbf{x}W)\right].$$4 are nearly symmetric when one is flipped around $$\phi(\mathbf{x})=\left[\exp(\mathbf{x}W)\oplus \exp(-\mathbf{x}W)\right].$$5, many of the CSD slope patterns are shared. The paper therefore shares the adder tree block and the multiplexer block between the log and antilog converters, adding only an encoder selection path and one additional right-shift-plus-inverter path. The reported result is that the antilog function can be implemented by the log converter at the cost of additional 14% area and 6% latency (Xiong et al., 2020).

A closely related modern instance appears in "Bitwidth-Specific Logarithmic Arithmetic for Future Hardware-Accelerated Training" (Hamad et al., 20 Oct 2025). There, the hard nonlinear part of logarithmic number system arithmetic is log-domain addition:

$$\phi(\mathbf{x})=\left[\exp(\mathbf{x}W)\oplus \exp(-\mathbf{x}W)\right].$$6

with $$\phi(\mathbf{x})=\left[\exp(\mathbf{x}W)\oplus \exp(-\mathbf{x}W)\right].$$7. The paper linearizes these functions by a piecewise-linear approximation over 16 segments on $$\phi(\mathbf{x})=\left[\exp(\mathbf{x}W)\oplus \exp(-\mathbf{x}W)\right].$$8,

$$\phi(\mathbf{x})=\left[\exp(\mathbf{x}W)\oplus \exp(-\mathbf{x}W)\right].$$9

so that each slope becomes a bit-shift and each segment requires only comparison, shifting, and addition. The approximation is optimized separately for each bitwidth by simulated annealing with a quantization-aware loss computed after quantization and dequantization of LNS sums (Hamad et al., 20 Oct 2025).

The reported significance is twofold. Numerically, the paper shows that 12-bit and 14-bit QAA-LNS training can stay close to FP32 on VGG and ResNet models, whereas non-quantization-aware approximations or bitwidth-mismatched approximations can cause severe degradation. In hardware terms, the synthesized LNS multiply-accumulate units achieve up to 32.5% reduction in area and 53.5% reduction in energy consumption compared to linear fixed-point equivalents (Hamad et al., 20 Oct 2025). Across both papers, the common principle is the same: nonlinear arithmetic is replaced by a piecewise-linear surrogate whose coefficients are chosen with explicit awareness of the underlying datapath.

4. In-situ circuit linearization and analog equilibrium networks

In analog equilibrium models, hardware linearization has a stricter local meaning. "Circuit realization and hardware linearization of monotone operator equilibrium networks" shows that the port behavior of a resistor-diode network corresponds to the solution of a ReLU monotone operator equilibrium network, and that the gradient of such a circuit can be computed directly in hardware by linearizing the physical circuit around its operating point (Chaffey, 17 Sep 2025). The equilibrium equations are written as

$$\mathbf{S}_i=\Gamma_i\mathbf{S}_{i-1}+\phi_k(\mathbf{k}_i)\mathbf{v}_i^\top,\qquad \hat{\mathbf{y}}_i=\mathbf{S}_i\phi_q(\mathbf{q}_i).$$0

Hardware linearization in this setting means physically replacing each nonlinear diode by its linearized counterpart determined by the operating point. For ideal diodes, a forward-biased diode becomes a short with a fixed voltage offset, while a reverse-biased diode becomes an open or a constant current offset. The resulting RTG circuit realizes the differential system

$$\mathbf{S}_i=\Gamma_i\mathbf{S}_{i-1}+\phi_k(\mathbf{k}_i)\mathbf{v}_i^\top,\qquad \hat{\mathbf{y}}_i=\mathbf{S}_i\phi_q(\mathbf{q}_i).$$1

so that $$\mathbf{S}_i=\Gamma_i\mathbf{S}_{i-1}+\phi_k(\mathbf{k}_i)\mathbf{v}_i^\top,\qquad \hat{\mathbf{y}}_i=\mathbf{S}_i\phi_q(\mathbf{q}_i).$$2, and the corresponding output yields $$\mathbf{S}_i=\Gamma_i\mathbf{S}_{i-1}+\phi_k(\mathbf{k}_i)\mathbf{v}_i^\top,\qquad \hat{\mathbf{y}}_i=\mathbf{S}_i\phi_q(\mathbf{q}_i).$$3 directly in hardware (Chaffey, 17 Sep 2025). In effect, the backward pass of an equilibrium model becomes a second circuit evaluation.

This is an exact analog implementation of implicit differentiation rather than a heuristic gradient estimate. The paper further shows that the derivative of the hybrid matrix of reciprocal RTG networks can itself be measured experimentally through unit excitations at individual resistive or conductive elements. That permits training with gradients computed on the actual physical device, including device non-idealities, instead of gradients from an idealized software model (Chaffey, 17 Sep 2025).

The framework extends beyond ideal ReLU. Ideal diodes induce ReLU through the resolvent of the monotone operator. A Shockley diode induces a smooth "diode ReLU," and Zener or current-regulator diodes induce saturation activations. The paper also studies cascades of RTGD networks, which implement more general feedforward or asymmetric architectures. In device-level simulations, training curves from hardware linearization and equilibrium backpropagation are nearly identical, and when resistance noise is introduced, hardware linearization on the true circuit yields smoother training than equilibrium backpropagation on the nominal model (Chaffey, 17 Sep 2025).

This usage of the term is the clearest example of literal local linearization. The nonlinearity is not approximated away globally; it is replaced by its exact linearized device law at the equilibrium that the forward computation has already selected (Chaffey, 17 Sep 2025).

5. Quantum hardware: graph linearization and linear operator embedding

Quantum hardware is intrinsically linear, so hardware linearization appears in quantum computing both as architecture reshaping and as operator embedding. "Linear Depth QFT over IBM Heavy-hex Architecture" addresses the former by transforming the heavy-hex coupling graph into a major line with dangling qubits (Gao et al., 2024). The resulting coupling structure is a path graph plus degree-1 leaves, which supports a recursive mapping of Quantum Fourier Transform circuits. For a special case with one dangling qubit in every five qubits, the paper proves a linear-depth upper bound of

$$\mathbf{S}_i=\Gamma_i\mathbf{S}_{i-1}+\phi_k(\mathbf{k}_i)\mathbf{v}_i^\top,\qquad \hat{\mathbf{y}}_i=\mathbf{S}_i\phi_q(\mathbf{q}_i).$$4

and more generally derives

$$\mathbf{S}_i=\Gamma_i\mathbf{S}_{i-1}+\phi_k(\mathbf{k}_i)\mathbf{v}_i^\top,\qquad \hat{\mathbf{y}}_i=\mathbf{S}_i\phi_q(\mathbf{q}_i).$$5

This is a graph-theoretic linearization: the physical hardware remains heavy-hex, but compilation uses a quasi-linear subgraph aligned with the algorithmic structure of QFT (Gao et al., 2024).

A different but related problem arises when the target problem itself is nonlinear. "Alleviating Post-Linearization Challenges for Solving Nonlinear Systems on a Quantum Computer" starts from nonlinear ODEs and applies Carleman linearization to obtain a high-dimensional infinite linear system, truncated at order $$\mathbf{S}_i=\Gamma_i\mathbf{S}_{i-1}+\phi_k(\mathbf{k}_i)\mathbf{v}_i^\top,\qquad \hat{\mathbf{y}}_i=\mathbf{S}_i\phi_q(\mathbf{q}_i).$$6 (Ali, 6 Feb 2026). The lifted variables are

$$\mathbf{S}_i=\Gamma_i\mathbf{S}_{i-1}+\phi_k(\mathbf{k}_i)\mathbf{v}_i^\top,\qquad \hat{\mathbf{y}}_i=\mathbf{S}_i\phi_q(\mathbf{q}_i).$$7

and the truncated state dimension is

$$\mathbf{S}_i=\Gamma_i\mathbf{S}_{i-1}+\phi_k(\mathbf{k}_i)\mathbf{v}_i^\top,\qquad \hat{\mathbf{y}}_i=\mathbf{S}_i\phi_q(\mathbf{q}_i).$$8

The post-linearization challenge is then to load and implement the resulting sparse linear operator efficiently on a quantum device (Ali, 6 Feb 2026).

The paper’s response is a basis change. Instead of decomposing the Carleman operator in a linear combination of Pauli operators, it introduces the Sigma basis

$$\mathbf{S}_i=\Gamma_i\mathbf{S}_{i-1}+\phi_k(\mathbf{k}_i)\mathbf{v}_i^\top,\qquad \hat{\mathbf{y}}_i=\mathbf{S}_i\phi_q(\mathbf{q}_i).$$9

whose tensor products directly match individual non-zero matrix entries. The stated result is an exponential reduction in the number of decomposition terms compared to the traditional Pauli decomposition. Because the Sigma elements are non-unitary, the paper then constructs a unitary completion for each tensor-product component $\mathbf{S}_i$0, so that it can be implemented as a block unitary in a standard LCU-style quantum circuit (Ali, 6 Feb 2026).

The two quantum cases illustrate two distinct meanings of the same phrase. In QFT mapping, the hardware is logically linearized by selecting a path-like coupling structure. In Carleman-based nonlinear simulation, the problem is linearized first and then the resulting linear operator is further transformed into a basis and circuit family that quantum hardware can actually realize (Gao et al., 2024, Ali, 6 Feb 2026).

6. Signal-processing, optimization, and memory-hierarchy formulations

In digital signal processing, hardware linearization often means constructing a compensator around a nonlinear plant so that the overall cascade behaves linearly. "Linearization of Time-Varying Nonlinear Systems Using A Modified Linear Iterative Method" models a nonlinear device by a time-varying discrete-time Volterra series

$\mathbf{S}_i$1

and then rewrites it as an input-dependent linear time-varying convolution. This permits a modified Richardson iteration,

$\mathbf{S}_i$2

which serves as either a post-linearizer or a predistorter for time-varying nonlinear hardware such as amplifiers (Hotz et al., 2014). The paper provides a sufficient convergence condition,

$\mathbf{S}_i$3

with $\mathbf{S}_i$4 expressed in terms of the first-order kernel deviation from identity and weighted sums of higher-order kernel magnitudes. The significance is that no explicit inverse of the first-order kernel is required, and the algorithm is real-time capable for time-varying kernels (Hotz et al., 2014).

At the level of processor and memory hierarchies, hardware linearization appears as address-space reduction. "Conformal Computing: Algebraically connecting the hardware/software boundary using a uniform approach to high-performance computation for software and hardware applications" treats multidimensional arrays as first-class objects and uses A Mathematics of Arrays with the $\mathbf{S}_i$5-calculus to derive explicit linear address arithmetic from high-level array expressions (0803.2386). The central mapping is the row-major linearization

$\mathbf{S}_i$6

with

$\mathbf{S}_i$7

which sends a multidimensional index to a linear offset in the ravel of an array. Denotational Normal Form expresses what an array program computes; Operational Normal Form expresses how it is executed as explicit loops, starts, stops, and strides. In this literature, tiling, reshape-transpose, and hyper-cube permutations are algebraic linearizations of data movement across registers, caches, main memory, processors, and networks (0803.2386).

A related optimization-oriented reduction appears in "Hardware-Based Linear Program Decoding with the Alternating Direction Method of Multipliers" (Wasson et al., 2016). There the global LP decoding problem is decomposed by ADMM into local variable-node and check-node updates, plus structured projections onto an interval and the parity polytope. The result is a message-passing architecture close to belief propagation in schedule but grounded in convex optimization, and a fixed-point Verilog implementation shows Frame Error Rate performance well within 0.5 dB of double-precision implementations with 10-bit messages (Wasson et al., 2016). Although the paper is not framed around the phrase itself, it exemplifies the same design move: transform a monolithic optimization problem into predominantly linear, local, hardware-regular updates.

Across these examples, hardware linearization operates at three different levels: plant inversion, memory access, and optimization decomposition. What unifies them is the replacement of irregular global computation by structured local updates whose dataflow matches the hardware substrate (Hotz et al., 2014, 0803.2386, Wasson et al., 2016).

7. Design principles, trade-offs, and recurring misunderstandings

Across the cited works, three design patterns recur. First, the transformed computation is expressed in terms of hardware-native primitives: GEMM and elementwise BF16 operations in LLMs, shifts and adds in digital arithmetic, opens and shorts in circuit linearization, multi-controlled Clifford-style constructions in quantum circuits, or explicit stride arithmetic in array compilers (Nguyen et al., 11 Jul 2025, Xiong et al., 2020, Chaffey, 17 Sep 2025, Ali, 6 Feb 2026, 0803.2386). Second, the transformation is usually guided by a measurable resource bottleneck such as quadratic memory growth, adder-tree latency, ancilla-controlled decomposition size, or cache-miss behavior. Third, the transformed system is evaluated against behavioral fidelity to a reference: teacher-model outputs, exact nonlinear functions, equilibrium gradients, or desired linear plant response (Nguyen et al., 11 Jul 2025, Xiong et al., 2020, Hotz et al., 2014).

The main trade-off is between structure and fidelity. In Lizard, the trade-off is controlled by the balance among gated global memory, sliding window size, and meta tokens, and by the attention-mimicry stage that reduces loss relative to the teacher (Nguyen et al., 11 Jul 2025). In arithmetic linearization, the trade-off appears as a Latency–Area–Precision space or as the number and shape of segments in bitwidth-specific logarithmic addition (Xiong et al., 2020, Hamad et al., 20 Oct 2025). In Carleman-based quantum methods, it appears as truncation order $\mathbf{S}_i$8 versus operator dimension, while in Volterra-based predistortion it appears as iteration count and model accuracy versus convergence and real-time cost (Ali, 6 Feb 2026, Hotz et al., 2014).

Two misunderstandings are especially persistent. One is that linearization automatically implies negligible error. The sources do not support that view: teacher performance is recovered only approximately in LLM retrofitting, Carleman embedding is truncated, Volterra inversion has a sufficient convergence condition rather than a universal guarantee, and low-bitwidth logarithmic arithmetic can become unstable when the approximation is not quantization-aware (Nguyen et al., 11 Jul 2025, Ali, 6 Feb 2026, Hotz et al., 2014, Hamad et al., 20 Oct 2025). The other is that hardware linearization always refers to the physical device itself. In fact, some works physically reconfigure circuits, some re-express algorithms for GPUs, some reshape a coupling graph used by a compiler, and some derive linear address formulas at the software/hardware boundary (Chaffey, 17 Sep 2025, Nguyen et al., 11 Jul 2025, Gao et al., 2024, 0803.2386).

A plausible implication is that hardware linearization is best understood not as a narrow approximation technique but as a co-design methodology. The nonlinear, quadratic, or irregular object of interest is transformed only to the extent required for faithful execution on the target substrate, and the success criterion is therefore joint: preserved behavior together with improved hardware realizability.