Jacobian-Based Nonlinearity Evaluation (JNE)

• JNE is a technique that quantifies, interprets, and leverages nonlinearity in neural or dynamical system mappings using local Jacobian matrices.
• It computes dispersion metrics by aggregating individual Jacobians over multiple inputs to reveal region-specific and hierarchical nonlinearity patterns.
• This method enhances model interpretability by directly comparing local linearizations to predicted outputs, aiding in diagnostics and neural encoding research.

Jacobian-Based Nonlinearity Evaluation (JNE) refers to a class of techniques for quantifying, interpreting, and leveraging the nonlinearity in mappings modeled by neural networks or dynamical systems via statistical analysis of their local linearizations—specifically, their Jacobian matrices—over a population of inputs or states. JNE has recently gained prominence as an interpretability metric for nonlinear neural encoding models, particularly in neuroscience, where the goal is to characterize the stimulus-dependent and region-specific nonlinearities in brain response patterns to complex visual stimuli (Gao et al., 15 Oct 2025). Unlike traditional metrics that compare the overall predictive performance of linear and nonlinear models, JNE directly measures the dispersion of local Jacobians between model representations and predicted outputs, providing a more sensitive and regionally-resolved assessment of nonlinearity.

1. Theoretical Foundation and Core Definitions

The central concept of JNE is to use the Jacobian matrix, $$JM_i = \left(\frac{\partial f(x_i)}{\partial x_i}\right)^T$$, as a local linear approximation of the mapping $f$ at sample $x_i$. For a set of samples $\{x_i\}_{i=1}^N$, the mean Jacobian is

$JM_{\text{mean}} = \frac{1}{N}\sum_{i=1}^N JM_i$

The dispersion from linearity for each sample is quantified as

$s\Delta JM_i = \| JM_i - JM_{\text{mean}} \|_1$

For a given output dimension (e.g., a particular brain voxel indexed by $m$), the JNE metric is defined as the standard deviation

$$\mathrm{JNE}_m = \sigma\left( [s\Delta JM_i]_m : i=1,\dots,N \right)$$

With this construction, $\mathrm{JNE}_m = 0$ exactly if the model is globally linear ($JM_i$ is constant for all samples). Nonzero values indicate sample-specific nonlinearity. This approach generalizes prior work in neural encoding by providing a quantitative, sample-resolved measure of deviation from linear response.

2. Methodological Implementation and Variants

JNE is implemented by computing Jacobians for each test stimulus input and aggregating their dispersion statistics. Computationally, this involves:

• For each input $x_i$, calculating the Jacobian $JM_i$ via forward or reverse-mode differentiation.
• Aggregating $JM_i$ over all test samples to compute $JM_{\text{mean}}$.
• Computing $s\Delta JM_i$ for each sample.
• Calculating $\mathrm{JNE}_m$ as the population standard deviation for each output dimension.

In high-dimensional cases, the $L_1$ norm is preferred for robustness when measuring dispersion. The methodology has been extended to sample-specific analysis ("JNE-SS") by examining the squared deviation

$\mathrm{JNE\mathchar`-SS} = (s\Delta JM - \Delta \mu)^2$

across clusters of stimuli (using t-SNE and k-means for stimulus grouping), enabling the identification of stimulus-selective nonlinear response patterns in subregions of output space (e.g., functional brain areas).

3. Empirical Validation and Hierarchical Nonlinearity Patterns

The paper demonstrates the validity and utility of JNE via both simulation and neuroimaging data (Gao et al., 15 Oct 2025):

• Activation Function Analysis: For typical neuron activation functions (ReLU, Leaky ReLU, GELU, Swish), the JNE curve computed over input windows closely matches the second derivative, thereby reconstructing the true nonlinearity of the function.
• Network-Level Nonlinearity: In feedforward networks, introducing nonlinear activations in deeper layers increases JNE values in accordance with layer position and function type.
• fMRI Data (NSD + CLIP-ViT): Application to real test sets reveals low JNE values in primary visual cortex (V1), and progressively higher JNE scores in higher-order visual and associative areas (EBA, PPA, prefrontal cortex). This establishes that cortical hierarchy is mirrored in increasing model nonlinearity, providing empirical confirmation for established theoretical models of sensory processing.

Sample-specific analysis further reveals that select stimulus categories elicit elevated JNE-SS signatures in specialized regions, supporting the notion of functional specialization in higher-order cortex.

4. Interpretation, Implications, and Comparative Advantages

JNE substantially refines the interpretation of neural encoding models:

• Beyond R² Comparison: Traditional comparisons of prediction performance between linear and nonlinear models can mask underlying sample-dependent nonlinearities, especially when input representations are themselves highly nonlinear. JNE avoids reliance on overall performance, instead focusing directly on the heterogeneity of local response.
• Hierarchical and Regional Insight: By mapping JNE across output dimensions (e.g., brain voxels), it is possible to recover the functional hierarchy in neural architecture—i.e., more linear responses in early sensory areas, and increasingly nonlinear processing in associative and semantic regions.
• Stimulus Selectivity: Through the JNE-SS extension, researchers can isolate input categories eliciting maximal nonlinearity in specific outputs, revealing new functional signatures or potential diagnostic markers for neural specialization.

This framework therefore enables a transition from black-box encoders to interpretable mappings, establishing direct links between model nonlinearity and neural coding properties.

5. Relationship to Other Metrics and Future Directions

JNE represents the first formal interpretability metric that quantifies nonlinear responses directly via local linear mappings (Gao et al., 15 Oct 2025). The method operates in a complementary fashion to existing measures of model fidelity, such as prediction accuracy or traditional variance explained:

• Direct nonlinearity quantification: By tracking the statistical dispersion of local Jacobians, JNE isolates the degree to which neural encoding models diverge from global linearity, regardless of predictive strength.
• Potential for clinical and basic science: This approach enables exploration of pathological or developmentally altered nonlinear response signatures, as well as detailed mapping of hierarchical processing.
• Expansion to other domains: While developed for neural encoding in fMRI datasets, the JNE methodology is broadly transferable to any scientific problem that demands interpretable measurement of local nonlinearities in a multivariate mapping.

A plausible implication is that JNE will inform the design of next-generation neural encoding models and neuroscientific experiments by providing diagnostic feedback rooted in the intrinsic local geometry of predictive mappings.

6. Computational Tools and Resource Availability

The framework and software for JNE are openly released for reproducibility and further research:

• Source code for the complete pipeline—including Jacobian computation, norm evaluation, and simulation tools—is available at https://github.com/Gaitxh/JNE.
• Scripts support both simulation (activation functions, feedforward networks) and analysis of neuroimaging data.

This encourages further theoretical and empirical development, as well as adaptation to new architectures and experimental modalities.

7. Significance Within the Field

JNE establishes a rigorous approach to interpreting complex nonlinear mappings, particularly within the context of neural response modeling and functional neuroimaging. By shifting focus from global accuracy to statistical analysis of local linearizations, the approach supports a refined understanding of hierarchy, specialization, and sample-specific brain computation. The introduction of JNE and its sample-specific extension JNE-SS represents a substantial methodological advance in model-based neuroscience and has implications for broader applications in nonlinear system identification, model-based interpretability, and computational diagnostics.