Per-Element Forward-Mode Differentiation

• Per-element forward-mode differentiation is a computational paradigm that augments each basic operation to propagate both its value and derivative.
• It unifies matrix–vector products, dual number algebra, and first-order Taylor expansion to compute derivatives with high numerical precision.
• This method underpins efficient and robust gradient evaluations in scientific computing, optimization, and machine learning, avoiding common pitfalls like expression swell.

Per-element forward-mode differentiation is a computational paradigm in automatic differentiation (AD) wherein each elementary operation within a function is systematically extended (“lifted”) to compute and propagate both its value and its derivative (or directional derivative) as a pair. This approach structures the entire computational process so that, for each basic function in a composite evaluation trace, both the primal (numerical) value and its derivative information are computed and passed forward, one elementary transformation at a time. Seemingly distinct perspectives—matrix–vector products, dual number algebra, first-order Taylor expansion—are all shown to encode the same per-element computational pattern. This framework underpins algorithmic differentiation in scientific computing, optimization, machine learning, and beyond, offering robust, precise, and efficient computation of derivatives at machine precision.

1. Mathematical Formulations and Core Principles

Three conceptualizations are fundamentally equivalent for per-element forward-mode differentiation:

1. Matrix–Vector Product Approach

Given a composite function $f : \mathbb{R}^n \rightarrow \mathbb{R}^m$ built from $\mu$ elementary functions, forward-mode AD embeds the evaluation as $$f = P_Y \circ \Phi_\mu \circ \dots \circ \Phi_1 \circ P_X$$, with $P_X$, $P_Y$ as input/output projections and $\Phi_i$ as elementary transitions. At a point $c \in \mathbb{R}^n$ and tangent vector $\dot{x} \in \mathbb{R}^n$, the chain rule produces the directional derivative:

$$J_f(c)\cdot \dot{x} = P_Y \cdot \Phi_\mu'(v^{(\mu-1)}) \cdots \Phi_1'(v^{(0)})\cdot P_X \dot{x}$$

where each step involves a local pair:

$(v, v') = (\text{value}, \text{derivative})$

and in particular for an elementary $\varphi_i$ with inputs $v_{i_1},\dots,v_{i_{n_i}}$,

$$\left(\varphi_i(v_{i_1},\dots,v_{i_{n_i}}),\;\nabla\varphi_i(v_{i_1},\dots,v_{i_{n_i}}) \cdot (v'_{i_1},\dots,v'_{i_{n_i}})^T\right)$$

2. Dual Numbers Approach

Each real value $x$ is replaced with a dual number $x + x' \epsilon$, $\epsilon^2 = 0$. Arithmetic and function application are overloaded so that for $h:\mathbb{R}^n\to\mathbb{R}$,

$$\hat{h}(x_1 + x_1' \epsilon, \dots, x_n + x_n' \epsilon) = h(x_1,\dots,x_n) + (\nabla h(x_1,\dots,x_n) \cdot (x_1',\dots, x_n')^T)\epsilon$$

The dual part of the result accumulates the directional derivative, and every elementary operation processes a pair (value, derivative) in parallel with the original evaluation.

3. Taylor Series Expansion

Interpreting function evaluation in forward-mode as evaluating the first-order Taylor polynomial,

$$T(h;x)(x+x'\epsilon) = h(x) + (\nabla h(x)\cdot x')\epsilon$$

Since $\epsilon^2=0$, higher-order terms vanish—again yielding the per-element propagation of derivatives.

All three perspectives formalize a computation in which every step in the calculation graph carries both a value and its derivative, which is updated by local rules derived from the chain rule.

2. Computational Workflow and Per-Element Mechanism

The per-element mechanism involves:

1. Decomposition: Breaking down a target function into a sequence of elementary operations, each with a well-defined local derivative.
2. Lifting: Augmenting each operation (addition, multiplication, sin, exp, etc.) to compute both its value and the derivative, typically by overloading or code transformation.
3. Evaluation: At each step, the current elementary operation receives as input the current values and derivatives, computes its output and the local directional derivative, and propagates both forward to the next operation.
4. Assembly: At the conclusion of the computation, the final output derivative reflects the accumulated per-element contributions, equivalent to a Jacobian–vector product at the evaluation point.

Forward-mode facilitates this by threading both values and derivatives ("Griewank pairs") through the computational trace. Each subexpression is differentiated once at the given input, so the total cost scales linearly in the number of elementary operations—unlike symbolic differentiation, which can suffer from exponential “expression swell.”

3. Applications and Algorithmic Implications

The per-element forward-mode paradigm has wide-ranging implications:

• Efficiency: The method avoids repeated computation of common subexpressions and overhead from symbolic expansion. Its complexity is $O$(number of elementary operations), as opposed to symbolic differentiation’s potential $O(n^2)$ or worse.
• Robustness: Since derivatives are calculated by explicit algebra at machine precision—without recourse to finite differences—numerical errors from step size or truncation are eliminated.
• Breadth: The framework covers not just scalar or vector-valued functions, but also naturally extends to multivariate, higher-order (e.g., Hessian) derivatives (via, e.g., truncated polynomial algebras) and to mappings between differentiable manifolds.
• Machine Learning and Optimization: Forward-mode AD is used for gradient-based optimization, backpropagation (as the forward-mode basis for reverse-mode), sensitivity analysis, and error estimation, where derivatives must be computed efficiently at arbitrary points in the parameter space.

4. Implementation Strategies

The per-element approach can be realized in several ways:

• Operator Overloading: Each numeric type is replaced with a dual-number or pair type; arithmetic operations and functions are overloaded to propagate derivatives.
• Source Code Transformation: The computation is automatically rewritten to produce code that calculates both value and derivative.
• Manual Augmentation: In scientific computing, such as ODE or PDE solvers, the underlying differential equations are explicitly augmented to propagate tangent information alongside the primary state.
• AD Tooling: State-of-the-art AD systems (e.g., ADiMat, JAX, Clad, CoDiPack, and EasyAD) implement per-element forward-mode by chaining together elementary rules per operation.

A summary of the computational model is shown in the table below:

Approach	Mechanism	Key Formula
Matrix–vector	Jacobian–vector product	$J_f(c)\cdot\dot{x}$
Dual number	Overload arithmetic/functions	$$f(x + x'\epsilon) = f(x) + (J_f(x)\cdot x')\epsilon$$
Taylor expansion	First-order polynomial eval	$$h(x + x'\epsilon) = h(x) + (\nabla h(x)\cdot x')\epsilon$$

5. Comparison with Alternative Methods

Forward-mode per-element differentiation is distinct from:

• Reverse-Mode (Backpropagation): Reverse mode computes the vector–Jacobian product, favoring situations with many input variables and few output variables (such as neural networks with scalar loss). It requires a backward pass and tape storage. Forward mode is advantageous for functions with few inputs and many outputs, or when memory is limited.
• Symbolic Differentiation: Symbolic methods may produce large, redundant expressions and suffer from “expression swell.” Per-element forward-mode computes numeric derivatives directly in one sweep of the computation.
• Finite Differences: Finite differences approximate derivatives via function perturbations and suffer from truncation and rounding errors. Per-element forward-mode is exact at machine precision.

6. Extensions and Theoretical Generalizations

The per-element schema admits generalization:

• Higher-Order Derivatives: By replacing dual numbers with truncated polynomial algebras, higher-order derivatives—including Hessians—can be computed via a similar per-element propagation strategy.
• Functionals and Higher-Order Functions: The approach can be lifted to process functionals (functions of functions), as in languages with higher-order differentiable programming support.
• Geometric and Manifold Contexts: Through the push-forward operator, per-element transformations extend to mappings between manifolds.
• Semantics and Correctness: Structure-preserving macros and semantic logical relations in denotational models guarantee that the per-element forward-mode transformation is mathematically correct, even in the presence of recursion, partiality, and higher types.

7. Practical Considerations in Modern Systems

In contemporary scientific computing and machine learning frameworks, per-element forward-mode differentiation is at the core of:

• Automatic Differentiation Libraries: Widely used in optimization engines, physics-based simulation, and neural network training, these leverage native per-element mechanisms for reliable and efficient gradient computation.
• Code Generation: Both source-to-source AD tools and operator-overloading systems instantiate per-element rules at compile or run time.
• Scalability and Parallelism: By structuring the computation as per-element operations, parallelized execution—on CPUs, GPUs, or distributed systems—is facilitated, improving throughput for large-scale models.

The fundamental benefit is that, by computing value and derivative in lockstep and treating each elementary operation independently, per-element forward-mode differentiation achieves a blend of flexibility, simplicity, and computational integrity unmatched by alternative approaches.

---

In summary, per-element forward-mode differentiation formalizes a universal computational rule: augment each operation to propagate its value and derivative pairwise through the computation, assembling exact derivatives (or Jacobian–vector products) in one pass, with robust numerical properties, modular extensibility, and generality for scientific and machine learning applications (Hoffmann, 2014).