Neural Response Function: Affine Cascade

• Neural response functions are mappings that convert sensory stimuli into neural activity by factorizing spatial (retinal) information from feature tuning.
• The AFRT model employs an affine retinal transform and localized linear pooling to dramatically reduce parameters while preserving biological interpretability.
• Empirical evaluations in macaque V1, V4, and IT demonstrate that this approach achieves higher prediction correlations and less overfitting compared to global models.

In encoding models, a neural response function maps a sensory stimulus to the measured activity of a neuron or recording site. In the retinotopy-centered formulation developed in "Neural encoding with affine feature response transforms" (Le et al., 7 Jan 2025), the function is defined not as an unstructured global weighting of deep features, but as a cascade that separates retinal position and size from feature tuning. The resulting model, the affine feature response transform (AFRT), factorizes each neuron’s encoding into an affine retinal transform with three interpretable parameters and a localized feature response with a small set of feature weights, thereby yielding a compact and biologically interpretable neural response function (Le et al., 7 Jan 2025).

1. Neural response functions in system identification and encoding

Within neuronal system identification, the functional relationship between a stimulus and a neuronal response is also called a mapping function, kernel or transfer function, transducer, or spatiotemporal receptive field (Wu et al., 2018). AFRT adopts this general viewpoint, but specializes it to visual cortical encoding with an explicit retinotopic inductive bias. Its starting point is the common linearizing encoder

$f(s)=w_{\mathrm{global}}^\top \phi(s),$

where $\phi$ is a frozen deep feature extractor and the response is predicted by linearly weighting all spatial features across the visual field (Le et al., 7 Jan 2025).

The central criticism of this baseline is architectural rather than merely statistical. A global weighting over the full feature map is expressive, but it is inefficient and hard to interpret because it ignores the retinotopic organization of visual cortex: real neurons respond only to a localized part of the visual field (Le et al., 7 Jan 2025). AFRT addresses this by defining the neural response function as a composition of two stages: an affine retinal transform that selects and re-parameterizes the local visual field feeding the neuron, and a localized feature response that linearly pools deep features from that local region (Le et al., 7 Jan 2025).

This factorization separates retinal “where” from feature “what.” In AFRT, “where” is encoded by retinal position and size, while “what” is encoded by channel-wise feature tuning inside the selected region. The resulting neural response function remains a linear–deep cascade, but only after the stimulus has been aligned to a neuron-specific local coordinate frame (Le et al., 7 Jan 2025).

2. AFRT formulation and factorization

Let $s \in \mathbb{R}^{H\times W\times C}$ be an image stimulus, let $\phi$ be a frozen convolutional feature extractor, and let $n$ index a neuron or channel. AFRT predicts the response $r_n$ via a constrained spatial transformer $T_{\theta_n}$ followed by localized linear pooling (Le et al., 7 Jan 2025).

The retinal stage uses a $2\times 3$ affine matrix parameterized by translation and uniform scaling:

$$A_n(\theta_n)= \begin{bmatrix} s_n & 0 & t_{x,n}\ 0 & s_n & t_{y,n} \end{bmatrix}.$$

Equivalently,

$x' = s_n x + t_{x,n}, \qquad y' = s_n y + t_{y,n}.$

Here $\phi$0 comprises translation of the receptive-field center in image coordinates and uniform scale of the receptive field. Rotation and shearing are excluded; the affine map is constrained to translation plus uniform scaling to preserve parallelism and keep the parameterization minimal and biologically interpretable (Le et al., 7 Jan 2025).

The spatial transformer resamples the input by warping $\phi$1 with $\phi$2 and cropping to a compact field of view:

$\phi$3

where $\phi$4 denotes differentiable sampling of $\phi$5 at coordinates given by $\phi$6. In practice, $\phi$7 is a square crop whose size is set per feature layer, for example 16, 32, or 64 pixels (Le et al., 7 Jan 2025).

Deep features are then extracted from the warped input:

$\phi$8

AFRT collapses the local spatial dimensions to a single descriptor, for example by average pooling:

$\phi$9

with $s \in \mathbb{R}^{H\times W\times C}$0. The final predicted response is

$s \in \mathbb{R}^{H\times W\times C}$1

where $s \in \mathbb{R}^{H\times W\times C}$2 are localized feature weights. The complete cascade is therefore

$s \in \mathbb{R}^{H\times W\times C}$3

AFRT also admits an explicitly segmented interpretation. Each neuron’s receptive field is decomposed into an affine component with three interpretable parameters $s \in \mathbb{R}^{H\times W\times C}$4 determining the center and size of the local retinal input, and a small set of feature weights $s \in \mathbb{R}^{H\times W\times C}$5 encoding channel-wise tuning within that local region. In expanded form,

$s \in \mathbb{R}^{H\times W\times C}$6

This is a local linearizing model in the coordinate frame induced by the affine retinal transform (Le et al., 7 Jan 2025).

3. Optimization, parameterization, and efficiency

AFRT is trained from paired stimulus–response data $s \in \mathbb{R}^{H\times W\times C}$7 by minimizing mean squared error for each neuron:

$s \in \mathbb{R}^{H\times W\times C}$8

The feature extractor $s \in \mathbb{R}^{H\times W\times C}$9 is frozen; in the reported experiments it is pretrained AlexNet on ImageNet. Both the affine parameters $\phi$0 and the localized weights $\phi$1 are learned jointly with Adam using learning rate $\phi$2 for 100 epochs. Affine transforms are initialized to identity, and $\phi$3 is initialized to uniform average pooling. Because $\phi$4 is differentiable with respect to $\phi$5 via bilinear sampling, gradients propagate through $\phi$6 and $\phi$7 to update both the affine parameters and the feature weights (Le et al., 7 Jan 2025).

A notable feature of AFRT is that regularization is architectural rather than explicit. No sparsity or smoothness priors are added beyond the architectural constraints; regularization arises from the massive parameter reduction and the inductive bias to locality (Le et al., 7 Jan 2025). This reduction can be written schematically as follows. Per neuron, AFRT estimates three affine parameters plus $\phi$8 local feature weights, where $\phi$9 is the local receptive-field size at the chosen layer and $n$0 is the number of feature channels. An unstructured model, by contrast, estimates $n$1 weights per neuron across the full spatial map (Le et al., 7 Jan 2025).

The example reported in the paper makes the scale of this difference concrete. For input size $n$2, $n$3 channels, and a local receptive field of $n$4,

• AFRT uses $n$5 parameters per neuron.
• An unstructured model uses $n$6 parameters per neuron.

This is roughly a three-orders-of-magnitude reduction (Le et al., 7 Jan 2025). The reduction occurs because AFRT replaces global spatial weighting with a small affine alignment and local channel pooling, eliminating the need to learn entangled global spatial weights and constraining the solution to low-order dependencies in a local region (Le et al., 7 Jan 2025).

4. Empirical results in macaque V1, V4, and IT

The reported evaluation uses macaque multi-unit activity recorded from 1024 channels and, after reliability filtering, 667 channels across V1, V4, and IT using 16 Utah arrays: 7 in V1, 4 in V4, and 4 in IT. Stimuli are naturalistic images from the THINGS database, comprising 25,248 images from 1,854 categories. Images were presented briefly for 200 ms in sequences with fixation enforced; original image resolution was 500×500 and images were shifted toward the lower-right fovea by 100 pixels in both axes (Le et al., 7 Jan 2025).

AFRT uses a frozen AlexNet with features from convolutional layers Conv1–Conv5. The affine crop size is set per layer, for example 16, 32, or 64 pixels, so that compact inputs are matched to receptive-field sizes of the chosen layers. The baseline is a standard linearizing encoder, denoted Linear-AlexNet, without affine transforms; its stimuli are uniformly resized to 224×224 and its features are pooled globally. Performance is evaluated by Pearson correlation between predicted and recorded responses on held-out test images (Le et al., 7 Jan 2025).

Several findings are emphasized. AFRT consistently outperforms Linear-AlexNet across V1, V4, and IT, with more models achieving higher correlations, including correlations at or above 0.5, and with higher mean performance despite using drastically fewer parameters. It is also notably less prone to overfitting (Le et al., 7 Jan 2025). The learned retinal transforms align with known receptive-field organization: receptive fields are larger in downstream regions V4 and IT than in V1, and visualizations of affine crops preserve these ventral-stream scaling trends (Le et al., 7 Jan 2025).

Layer usage is likewise anatomically graded. Earlier AlexNet layers better encode V1, whereas deeper layers contribute more to V4 and IT, consistent with hierarchical feature complexity. Roughly half of neurons per region select the most appropriate layer, with some variability. In V1, layer 1 dominates over layer 5, suggesting that overly complex features may overfit simpler early visual responses (Le et al., 7 Jan 2025).

5. Terminological breadth across adjacent literatures

A common misconception is that “neural response function” denotes a single standardized object. The literature instead uses the term, or closely related terms, for several distinct constructions.

Context	Representative formulation	Role
Sensory system identification	mapping function, kernel or transfer function, transducer, STRF	stimulus–response mapping (Wu et al., 2018)
Retinotopic visual encoding	$n$7	localized encoding with affine retinal alignment (Le et al., 7 Jan 2025)
Continuous fMRI encoding	$n$8	continuous implicit function over MNI space (Chen et al., 7 Oct 2025)
Hyperspectral fusion	Neural SRF and PSF layers	learn sensor response functions as differentiable layers (Zheng et al., 2020)
Stochastic integrate-and-fire neurons	susceptibilities $n$9 and $r_n$0	linear response to weak time-dependent stimuli (Klett et al., 10 Mar 2025)

In hyperspectral super-resolution, for example, HyCoNet realizes the spectral response function and point spread function as learnable convolutional layers and explicitly describes them as Neural Response Functions (Zheng et al., 2020). In continuous brain encoding, the Neural Response Function is an anatomically grounded implicit neural field that predicts fMRI activity as a function of both image and MNI coordinate, $r_n$1 (Chen et al., 7 Oct 2025). In stochastic integrate-and-fire theory, response functions are frequency-domain susceptibilities that characterize how membrane voltage, spike train, or subthreshold nonlinearity respond linearly to weak external stimuli (Klett et al., 10 Mar 2025).

This breadth suggests that the term is best understood structurally rather than lexically. Across these usages, a response function is a formal object that links perturbation, stimulus, or coordinate to an observed neural or sensorimotor quantity, but the mathematical realization depends on the scientific problem under study.

6. Practical implications, limitations, and extensions

Within visual encoding, AFRT is designed for localized neural responses from natural images, especially when data are limited. The paper explicitly identifies use cases such as multi-unit activity, single units, or multi-voxels in early or mid visual cortex. Its reported advantages are interpretable receptive-field centers and sizes, significantly fewer parameters, and better generalization than unstructured global linear models (Le et al., 7 Jan 2025).

The same source also delineates the present limits of the formulation. AFRT excludes rotation and shear, even though real receptive fields may exhibit orientation-dependent modulations or more complex geometry due to attention, learning, or anisotropies. The implementation is single-stage and restricted to static images; temporal dynamics and multi-stage spatial alignment are not modeled. Applicability to lower-SNR modalities such as fMRI remains open because of differences in spatial resolution and signal characteristics (Le et al., 7 Jan 2025).

Several extensions are proposed directly. One is to add rotation and possibly non-uniform scaling or shear by using a full 6-parameter affine transform or anisotropic scaling. Another is to cascade multiple constrained warps to model multi-scale or surround effects. A temporal AFRT would extend $r_n$2 to dynamic stimuli such as videos and learn time-varying transforms and feature pooling. End-to-end training, in which the feature extractor $r_n$3 is fine-tuned jointly with AFRT, is suggested as a route to capture dataset-specific features and further improve accuracy (Le et al., 7 Jan 2025).

Taken together, AFRT gives the neural response function a particularly explicit form:

$r_n$4

with

$r_n$5

Its significance lies in making the geometric and feature components of encoding separately identifiable. The retinal transform supplies a compact model of center and size; the local feature weights supply channel-wise tuning. In that sense, AFRT does not merely improve a predictor. It redefines the response function itself as a retinotopy-constrained cascade whose geometry is part of the model rather than a by-product of post hoc interpretation (Le et al., 7 Jan 2025).