Sample-Specificity (JNE-SS) in Neural Encoding

• Sample-Specificity (JNE-SS) is a metric that quantifies stimulus-specific deviations in local linear mappings from latent features to neural responses.
• It employs computational procedures that extract deep neural features and compute per-stimulus squared deviations of Jacobians using the L₁ norm.
• Empirical evaluation on fMRI data shows that JNE-SS distinguishes linear patterns in primary visual cortices from nonlinear responses in higher-order brain regions.

Sample-Specificity (JNE-SS) quantifies the degree to which nonlinear neural encoding models exhibit stimulus-dependent variation in local linear mappings—represented by the model’s Jacobians—from latent representations (e.g., artificial neural network activations) to predicted neural responses (e.g., fMRI BOLD signals). Unlike global metrics that summarize nonlinearity over an entire dataset, JNE-SS decomposes this variation at the level of individual stimulus–voxel pairs, providing a fine-grained map of where and how nonlinear computations arise in neural encoding models. This enables the identification of stimulus-selective nonlinear patterns across brain regions and supports interpretation of functional specialization in neural circuitry.

1. Jacobian-Based Nonlinearity Evaluation and Sample-Specificity Extension

The original JNE metric computes, for each voxel, the dispersion of the Jacobians of the neural encoding model over the sample space. Mathematically, for a nonlinear model mapping input features to output BOLD responses, the Jacobian for sample $i$ at voxel $m$ is denoted $JM_{i,m}$, and the contracted deviation (e.g., L₁ norm) is $sΔJM_{i,m}$. JNE summarizes this through voxelwise mean absolute deviation ($Δμ_m$) and standard deviation:

$Δμ_{m} = \frac{1}{N} \sum_{i=1}^N sΔJM_{i,m}$

$$JNE_{m} = \sqrt{\frac{1}{N} \sum_{i=1}^N (sΔJM_{i,m} - Δμ_{m})^2}$$

JNE-SS generalizes this calculation, defining a sample–specific squared deviation for each stimulus $i$ and voxel $m$:

$JNE\text{-}SS_{i,m} = (sΔJM_{i,m} - Δμ_m)^2$

Here, $JNE\text{-}SS_{i,m}$ quantifies, for a particular stimulus–voxel pair, the extent to which the local linear mapping deviates from the global average mapping, offering per-sample resolution of nonlinearity.

2. Computational Procedures

To compute JNE-SS in practice:

• For each stimulus in the test set, extract deep neural features (e.g., CLIP-ViT embeddings).
• Fit both linear and nonlinear neural encoding models from features to the BOLD response for each voxel.
• For each stimulus–voxel pair, calculate $JM_{i,m}$ (the Jacobian of the nonlinear mapping).
• Contract each Jacobian to a scalar value per voxel using the L₁ norm to obtain $sΔJM_{i,m}$.
• Compute $Δμ_m$ across all samples for each voxel.
• Evaluate $JNE\text{-}SS_{i,m}$ as above.

This procedure yields a matrix of size $N \times M$ (number of stimuli × number of voxels), which can be stratified, clustered, or visualized to map the landscape of stimulus-selective nonlinearity.

3. Empirical Validation and Brain Region Implications

Simulation experiments using minimal artificial neural networks confirmed that JNE tracks the local nonlinearity induced by various activation functions (ReLU, GELU, Leaky ReLU, Swish) with JNE-SS extending this to single-stimulus resolution.

Analyses on real fMRI data, specifically the Natural Scenes Dataset (NSD), revealed region-specific and stimulus-selective patterns:

• Primary and intermediate visual cortices showed low JNE-SS values, suggestive of dominantly linear mappings for most stimuli.
• Higher-order visual regions (EBA, PPA, FFA) exhibited high and cluster-specific JNE-SS values, indicating that select stimulus categories (e.g., faces, outdoor scenes) evoke strong nonlinear neural responses.
• Clustering test stimuli via t-SNE and K-means stratification exposed stimulus categories preferentially driving nonlinear responses in select regions, consistent with established functional cortical hierarchies.

4. Interpretative Utility and Applications

JNE-SS provides a novel interpretability metric for neural encoding models with several major applications:

• Disentangles the effect of individual stimulus properties on cortical response nonlinearity, supporting more accurate mappings of stimulus selectivity and functional specialization.
• Enables diagnostic comparisons of modeling choices: when both linear and nonlinear encoding models yield similar prediction R² due to the feature extractor’s prior nonlinearity, JNE-SS directly quantifies the residual nonlinearity at the encoding layer.
• Supports stratified interpretation for identifying which stimulus types elicit nonlinear behavior, informing network model development and functional neuroanatomy.

5. Comparison with Traditional Performance Metrics

Traditional performance comparisons between linear and nonlinear neural encoding models often yield negligible differences, especially when feature extractors are highly nonlinear (e.g., transformer-based models like CLIP-ViT). In these cases, R² fails to capture intrinsic stimulus-dependent nonlinearity in responses.

JNE-SS, by leveraging the dispersion of Jacobians, reveals nuanced spatial and stimulus-specific patterns, uncovering stimulus-driven nonlinearities invisible to global prediction metrics. This makes JNE-SS an essential complement to classic performance scores.

6. Limitations and Future Research Directions

Current open challenges include:

• Establishing absolute JNE-SS thresholds for categorical classification of “linear” vs. “nonlinear” processing.
• Extending JNE-SS quantification to more granular scales, such as sub-voxel regions or microcircuits.
• Broadening evaluation to alternative neural encoding architectures (e.g., convolutional networks, temporal models).
• Integrating JNE-SS analysis with physiological measures (e.g., combining BOLD and electrophysiological recordings) to further validate correspondence between model-based nonlinearity and actual neural circuit dynamics.
• Investigating temporal and state-dependent fluctuations in sample-specific nonlinearity during cognitive tasks.

7. Summary and Significance

Sample-Specificity (JNE-SS) advances the quantification of nonlinearity in neural encoding models by resolving stimulus–response patterns at the granularity of individual inputs and cortical regions. By mathematically formalizing per-sample deviations of local linear mappings and demonstrating their hierarchical and selective structure in real neural data, JNE-SS offers rich interpretative capacity that surpasses conventional aggregate performance metrics. This innovation supports deeper mechanistic insight into brain information processing and lays groundwork for enhanced model development, functional mapping, and neuroscience discovery.