Neural Functional Regression Net

• Neural Functional Regression Net is a class of neural architectures that map function-valued data to outputs while preserving intrinsic continuous structure.
• They employ diverse encoding methods such as basis expansion, point evaluation, kernel embedding, and image conversion to capture functional characteristics.
• These models support various regimes—scalar-on-function, function-on-function, and spatial-functional—with applications in epidemiology, energy demand, and beyond.

Searching arXiv for relevant papers on neural functional regression networks and closely related functional neural architectures. Search results reviewed across functional regression, function-on-function models, functional inputs, RKHS functional approximation, and recent BP-free regression variants. “Neural Functional Regression Net” (“NFRN”, Editor's term) denotes a class of neural architectures for regression with functional data, where predictors, responses, or both are functions such as curves, time series, spectra, or images rather than fixed finite-dimensional vectors. In the literature, this umbrella includes scalar-on-function models with function-valued weights, deep models with continuous hidden layers, nonlinear function-on-function regressors, neural networks for scalar input and functional output, spatial and spatio-temporal functional networks, and alternative pipelines based on image conversion, RKHS point evaluations, or kernel embeddings (Thind et al., 2020).

1. Conceptual scope and task taxonomy

The central object in neural functional regression is a map between spaces of functions and/or Euclidean variables. The main regimes represented in the literature are scalar-on-function regression, where functional predictors are mapped to scalar responses; function-on-function regression, where both input and output are functions; function-on-scalar regression, where scalar predictors generate functional responses; and spatial or spatio-temporal regression, where functional covariates are indexed by location and combined with spatial dependence or heterogeneity (Rao et al., 2021).

A useful terminological distinction is that functional regression in functional data analysis is not identical to generic function approximation. In the former, the data objects themselves are functions, often observed discretely or irregularly; in the latter, the goal is to approximate an unknown numerical map $x \mapsto y$ even when inputs are ordinary vectors. Adjacent work on Forward-Forward regression addresses function regression/approximation and BP-free learning, but it is methodologically adjacent rather than identical to the FDA-centered literature (Padmani et al., 8 Oct 2025).

Family	Setting	Representative papers
Functional-input neural nets	Functional and scalar covariates $\rightarrow$ scalar response	(Thind et al., 2020, Rao et al., 2021)
Nonlinear function-on-function nets	Functional predictors $\rightarrow$ functional response	(Rao et al., 2021, Wang et al., 2020)
Neural function-on-scalar models	Scalar predictors $\rightarrow$ functional response	(Wu et al., 2022)
Spatial and spatio-temporal functional nets	Spatially indexed functional covariates $\rightarrow$ scalar response	(Rao et al., 2020, Park et al., 16 Jun 2025)
Alternative functional representations	Image conversion, RKHS point evaluations, kernel embedding	(Galarza et al., 2023, Zhou et al., 2024, Shi et al., 2024)

This taxonomy suggests that a single “net” architecture is not canonical. Rather, the field is defined by a recurrent design principle: preserve or exploit functional structure long enough to avoid naive vectorization, then couple that structure to a neural predictor.

2. Functional representations and input encodings

A foundational representation for scalar-on-function learning is the functional neuron, which replaces the ordinary inner product by an integral against a learned weight function. In the formulation with multiple functional and scalar covariates, the first hidden layer takes the form

$$v^{(1)}_i = g\left( \sum_{k=1}^{K} \int_{\mathcal{T}} \beta_{ik}(t)x_k(t)\,dt + \sum_{j=1}^{J} w^{(1)}_{ij}z_j + b^{(1)}_i \right),$$

with $\beta_{ik}(t)$ parameterized through basis expansions such as B-splines or Fourier functions (Thind et al., 2020). This design keeps the function space structure explicit at the network entrance and yields dynamic functional weights that can be visualized during optimization.

A more unusual representation is image conversion. For functional observations $\{(x_i,f(x_i))\}_{i=1}^n$, the curve is first min-max normalized so that $f:\mathbb{R}\to[0,1]$, then converted to a signed distance matrix $D=[d_{ij}]$ with

$\rightarrow$0

The matrix is interpreted as a $\rightarrow$1 grayscale image and passed to a CNN (Galarza et al., 2023). This representation is intentionally agnostic to handcrafted bases and rephrases one-dimensional functional learning as image-based regression or classification.

A different line of work removes basis integration altogether by exploiting RKHS structure. One formulation represents $\rightarrow$2 through point evaluations on a finite set $\rightarrow$3, so that a standard two-hidden-layer fully connected tanh network operates on $\rightarrow$4. The justification is that point evaluation is a continuous linear functional in an RKHS, and interpolating orthogonal projections can reconstruct an approximation from the sampled values (Zhou et al., 2024).

Kernel-embedded functional networks occupy an intermediate position. In that setting, an input function is first mapped by

$\rightarrow$5

then projected onto leading eigenfunctions of a projection kernel, and only afterward fed into a deep ReLU network (Shi et al., 2024). This architecture is explicitly designed to be adaptive and nearly discretization-invariant, so that arbitrary meshes or noisy sampled curves need not be treated as fixed raw vectors.

These alternatives make clear that “functional encoding” is a primary design axis. Basis expansion, point evaluation, kernel embedding, FPCA projection, and image conversion are not implementation details around a single model; they define substantively different theories of what information from a function should be made visible to the network.

3. Architectural families

The simplest NFRN family is the scalar-response functional neural network in which only the first hidden layer is functional and later layers are ordinary dense layers. This architecture integrates functional covariates with scalar covariates, reduces the number of learned parameters relative to naive discretization, and produces function-valued weights $\rightarrow$6 that summarize where in the domain the response is most sensitive (Thind et al., 2020).

A deeper construction is the Functional Direct Neural Network (FDNN) and the Functional Basis Neural Network (FBNN). In these models, hidden layers are continuous hidden layers composed of continuous neurons. The generic layer is

$\rightarrow$7

with $\rightarrow$8 and $\rightarrow$9 in the function-on-function case (Rao et al., 2021). FDNN learns weight functions directly on a grid, whereas FBNN expands weight and bias functions in bases such as B-splines, wavelets, or Fourier functions. The latter is explicitly more parsimonious and better suited to irregular or sparse functional data.

For scalar input and functional output, a distinct architectural strategy is used. The network outputs basis coefficients or FPCA scores rather than pointwise curve values. In the basis-expansion version, if $\rightarrow$0, then the network predicts $\rightarrow$1, and the functional response is reconstructed from the learned coefficient vector (Wu et al., 2022). The same paper also defines direct response-level losses and roughness penalties so that smoothness is controlled during training rather than imposed post hoc.

Another function-on-function strategy combines multivariate FPCA with a fully connected neural network. In that framework, input time series are projected onto input principal component scores, a dense network learns a nonlinear map from input scores to output scores, and the functional response is reconstructed by the output eigenfunctions: $\rightarrow$2 This formulation explicitly generalizes the function-on-function linear model while retaining irregular-time capability through the functional representation (Wang et al., 2020).

Spatial extensions further diversify the architecture class. GWFNN builds a location-specific functional neural network by weighting training data according to spatial proximity, whereas SARFNN augments a functional network with a spatial autoregressive covariate $\rightarrow$3 to encode response dependence among nearby sites (Rao et al., 2020). DSNet goes further by integrating functional predictors, scalar predictors, spatially varying coefficients, and spatial random effects within a deep architecture whose first hidden layer contains terms of the form

$\rightarrow$4

with location-specific functional weights expanded jointly over temporal and spatial bases (Park et al., 16 Jun 2025).

Across these families, the defining contrast is whether functional structure is used only at the entrance, preserved across all hidden layers, represented through lower-dimensional scores, or absorbed into spatially varying and operator-valued components.

4. Loss functions, optimization, and regularization

Most neural functional regression models are trained by variants of mean squared error, but the target of the loss differs. For function-on-function models with functional responses, FDNN and FBNN minimize the mean squared integrated error

$\rightarrow$5

and compute functional gradients via Fréchet derivatives with respect to weight and bias functions (Rao et al., 2021). This is the direct analogue of backpropagation in function space.

In neural function-on-scalar regression for functional outputs, two supervision strategies are distinguished. One supervises basis coefficients or FPC scores; the other supervises the reconstructed response trajectory directly through

$\rightarrow$6

Smoothness can then be enforced by penalties on second derivatives or on second differences of basis coefficients (Wu et al., 2022). This is important because accurate coefficient prediction alone does not guarantee visually or scientifically plausible curves.

Regularization in the functional literature is not limited to standard deep learning devices, although those appear as well. Early stopping is used to avoid overfitting in FDNN and FBNN, roughness penalties promote smooth and interpretable functional parameters, and dropout is part of the CNN image-based architecture, which includes a 20% dropout layer before the fully connected regression or classification head (Galarza et al., 2023). The image-based workflow also uses ordinary regression loss layers and cross-entropy in the Siamese CNN used for drug dissolution profile similarity (Galarza et al., 2023).

A separate methodological direction explores BP-free regression training. FFR replaces contrastive positive-negative pairs by an ordinal competitive goodness function over neuron groups, combines this with a stratified ladder architecture and hierarchical prediction, and reports that it recovers on average 98.6% of BP’s accuracy across five real-world regression benchmarks while reducing peak training memory to 27% of BP’s at depth 8 and 8% at depth 32, with per-iteration time around 72% of BP’s (Liu et al., 2 Jun 2026). Another Forward-Forward regression method reformulates regression as a classification of trial outputs into “in-tolerance” and “out-tolerance” classes and performs inference by scanning candidate outputs and comparing layer goodness values (Padmani et al., 8 Oct 2025). These methods are not specialized to functional data analysis, but they are relevant to the broader question of how regression-oriented neural architectures can be trained without conventional backpropagation.

5. Approximation theory and statistical guarantees

The strongest formal statistical guarantee in the surveyed literature appears in deep ReLU estimation of the mean function of functional data. With observations

$\rightarrow$7

the estimator is the empirical risk minimizer

$\rightarrow$8

over a sparse ReLU network class (Wang et al., 2020). When $\rightarrow$9, the paper states the optimal nonparametric rate

$\rightarrow$0

independent of the ambient data dimension $\rightarrow$1. When $\rightarrow$2 and $\rightarrow$3 is sufficiently large, the rate can be faster than root-$\rightarrow$4, and the discussion states that $\rightarrow$5 is possible under suitable scaling (Wang et al., 2020). The theoretical mechanism is compositional low-dimensional structure, represented by the function class $\rightarrow$6.

For regression maps defined on RKHSs, universality results are obtained with remarkably simple input representations. A standard fully connected network with two hidden layers and tanh activation can approximate any Hölder-continuous functional on the unit ball of an RKHS to arbitrary accuracy using only point evaluations, and explicit error bounds are given for Sobolev, inverse multiquadric, and Gaussian kernels (Zhou et al., 2024). This directly challenges the view that functional learning necessarily requires integration-type basis expansions.

Kernel-embedded functional deep networks provide another theory path. The architecture based on $\rightarrow$7, eigenfunction projection, and a deep ReLU predictor yields explicit approximation rates on Besov, Sobolev, mixed-smooth Sobolev, and Gaussian RKHS function spaces, together with a two-stage oracle inequality that separates discretization error from network approximation and estimation error (Shi et al., 2024). A central claim is that, once the kernel is sufficiently smooth, a modest number of observation points per function can suffice, which the paper describes as near mesh invariance.

In spatial-functional learning, DSNet introduces a low-rank structure inspired by the spatially varying functional index model. The supplementary theory states

$\rightarrow$8

so the rate depends on the order of interactions $\rightarrow$9 rather than on the full input dimension (Park et al., 16 Jun 2025). This suggests that the functional and spatial bases are not merely computational devices; they are also the mechanism by which the architecture mitigates the curse of dimensionality.

6. Empirical behavior, applications, and interpretability

Empirical work shows that neural functional regression is not confined to synthetic curve benchmarks. The image-converted CNN pipeline has been applied to estimation of exponential growth and decay rates, sine and cosine bandwidths, peak widths and heights, Lyapunov exponents in chaotic systems, transmission rates in SIR epidemic curves, drug dissolution profile similarity, and Parkinson’s disease detection from digitized spirals (Galarza et al., 2023). Across the regression tasks, the paper reports very high correlation between predicted and true parameter values, typically $\rightarrow$0, with only slight degradation at noise level $\rightarrow$1. For exponential growth/decay estimation, the representative correlations are $\rightarrow$2 with no noise, $\rightarrow$3 for $\rightarrow$4, and $\rightarrow$5 for $\rightarrow$6, with slopes near $\rightarrow$7 and intercepts near $\rightarrow$8. The same work states that CNN-based Lyapunov exponent estimation is 600x faster than the Rosenstein algorithm and reports 100% accuracy on validation and test sets for Parkinson’s disease classification (Galarza et al., 2023).

In nonlinear function-on-function regression, FDNN and FBNN perform comparably to traditional methods in simple linear settings but substantially better in nonlinear settings. For the complex quadratic simulation summarized in the paper, FFDNN achieves an RMSE of about $\rightarrow$9, compared with $$v^{(1)}_i = g\left( \sum_{k=1}^{K} \int_{\mathcal{T}} \beta_{ik}(t)x_k(t)\,dt + \sum_{j=1}^{J} w^{(1)}_{ij}z_j + b^{(1)}_i \right),$$0 for FFLM, $$v^{(1)}_i = g\left( \sum_{k=1}^{K} \int_{\mathcal{T}} \beta_{ik}(t)x_k(t)\,dt + \sum_{j=1}^{J} w^{(1)}_{ij}z_j + b^{(1)}_i \right),$$1 for NN, and $$v^{(1)}_i = g\left( \sum_{k=1}^{K} \int_{\mathcal{T}} \beta_{ik}(t)x_k(t)\,dt + \sum_{j=1}^{J} w^{(1)}_{ij}z_j + b^{(1)}_i \right),$$2 for FNN (Rao et al., 2021). On Adelaide electricity demand, the same study reports RMSE values of $$v^{(1)}_i = g\left( \sum_{k=1}^{K} \int_{\mathcal{T}} \beta_{ik}(t)x_k(t)\,dt + \sum_{j=1}^{J} w^{(1)}_{ij}z_j + b^{(1)}_i \right),$$3 for FFLM, $$v^{(1)}_i = g\left( \sum_{k=1}^{K} \int_{\mathcal{T}} \beta_{ik}(t)x_k(t)\,dt + \sum_{j=1}^{J} w^{(1)}_{ij}z_j + b^{(1)}_i \right),$$4 for CAM, $$v^{(1)}_i = g\left( \sum_{k=1}^{K} \int_{\mathcal{T}} \beta_{ik}(t)x_k(t)\,dt + \sum_{j=1}^{J} w^{(1)}_{ij}z_j + b^{(1)}_i \right),$$5 for FFDNN, and $$v^{(1)}_i = g\left( \sum_{k=1}^{K} \int_{\mathcal{T}} \beta_{ik}(t)x_k(t)\,dt + \sum_{j=1}^{J} w^{(1)}_{ij}z_j + b^{(1)}_i \right),$$6 for FBNN (Rao et al., 2021). These results support the narrower claim that deep continuous-hidden-layer models become most useful when interactions and nonlinearities are structurally important.

The multivariate time-series function-on-function model using FPCA plus a fully connected network also reports lower error than multivariate regression, Seq2Seq LSTM, and the functional linear model on electricity demand and short-term traffic prediction (Wang et al., 2020). For electricity demand, the paper states that the proposed model used about $$v^{(1)}_i = g\left( \sum_{k=1}^{K} \int_{\mathcal{T}} \beta_{ik}(t)x_k(t)\,dt + \sum_{j=1}^{J} w^{(1)}_{ij}z_j + b^{(1)}_i \right),$$7 parameters, compared with more than $$v^{(1)}_i = g\left( \sum_{k=1}^{K} \int_{\mathcal{T}} \beta_{ik}(t)x_k(t)\,dt + \sum_{j=1}^{J} w^{(1)}_{ij}z_j + b^{(1)}_i \right),$$8 for the LSTM and $$v^{(1)}_i = g\left( \sum_{k=1}^{K} \int_{\mathcal{T}} \beta_{ik}(t)x_k(t)\,dt + \sum_{j=1}^{J} w^{(1)}_{ij}z_j + b^{(1)}_i \right),$$9 for the linear FFLM, while achieving the lowest RMSE and RMSPE across all seven response dimensions (Wang et al., 2020).

Spatial-functional deep learning has likewise moved from simulation to domain applications. GWFNN and SARFNN were evaluated on precipitation prediction using the Canada weather dataset, and GWFNN achieved the best RMSE with a 30% improvement over FNN in the setting dominated by spatial heterogeneity (Rao et al., 2020). DSNet was developed for county-level corn yield prediction in the U.S. Midwest and is reported to achieve much lower mean squared prediction error than SVFM, functional neural nets without spatial adaptation, and standard NN or XGBoost baselines, while preserving interpretable spatial and temporal effect structure (Park et al., 16 Jun 2025).

Interpretability remains a recurrent, if secondary, design target. Functional-input FNNs expose dynamic functional weights that can be averaged and visualized over the domain, indicating where a predictor contributes positively or negatively to the scalar response (Thind et al., 2020). FDNN and FBNN add roughness penalties to obtain smoother fitted parameter functions, and neural function-on-scalar regression uses roughness penalties to control the smoothness of predicted curves directly (Rao et al., 2021). These mechanisms do not convert the models into classical parametric regressions, but they do create interpretable intermediate objects absent from ordinary dense networks.

7. Limitations, misconceptions, and current directions

A common misconception is that neural functional regression always reduces to basis expansion followed by an ordinary multilayer perceptron. The surveyed literature contradicts this directly: some models use function-valued weights and continuous hidden layers, some use image conversion, some use RKHS point evaluations, and some use kernel embeddings with projection bases learned from operator structure rather than chosen a priori (Zhou et al., 2024). The field is better understood as a family of architectures that differ primarily in how they encode infinite-dimensional input.

Another misconception is that the methods uniformly handle irregular observation designs. They do not. FDNN is described as well suited to dense, regularly gridded data, whereas FBNN is explicitly more parsimonious and more flexible for irregular or sparse functional data (Rao et al., 2021). Likewise, function-on-scalar networks for functional output devote substantial methodological effort to objective functions that remain back-propagable under irregular grids (Wu et al., 2022).

Several limitations recur across papers. Spatial models are sensitive to kernel function and bandwidth choice, and GWFNN may require model fitting for each new spatial location (Rao et al., 2020). Deep functional models can overfit and require tuning of layers, neurons, basis size, activation functions, and grid resolution (Rao et al., 2021). Basis-expansion methods inherit subjectivity from basis selection, which is one reason point-evaluation and kernel-embedding approaches were proposed (Zhou et al., 2024). Interpretability improves relative to black-box dense nets but remains weaker than in linear or additive FDA models (Rao et al., 2020).

Alternative training paradigms are promising but not settled. FFR reports substantial memory savings and strong regression accuracy relative to BP-free competitors, but it is framed as a general regression framework rather than a functional-data architecture (Liu et al., 2 Jun 2026). The Forward-Forward method for function regression yields accurate approximations on several benchmark functions, yet the paper also reports that it is orders of magnitude slower than backpropagation for similar accuracy and that FF training for Deep Physical Neural Networks did not converge successfully (Padmani et al., 8 Oct 2025). A plausible implication is that training methodology remains an open research frontier distinct from the question of how best to represent functional inputs.

Current directions are therefore converging on three themes: richer representations for irregular and high-dimensional functional observations, stronger theory for approximation and generalization in infinite-dimensional settings, and architectures that integrate functional structure with spatial dependence, uncertainty quantification, or local training constraints. This suggests that the enduring problem is not whether neural networks can be used for functional regression, but which architectural and statistical abstractions best preserve the geometry of function-valued data.