---
title: Functional Input Neural Networks (FNN)
url: https://www.emergentmind.com/topics/functional-input-neural-networks-fnn
type: topic
---

# Functional Input Neural Networks (FNN)

Searching arXiv for recent and foundational papers on functional-input neural networks and adjacent formulations.
Functional Input Neural Networks (FNNs) are neural-network models in which the input is not restricted to a finite-dimensional feature vector but may instead be a function \(x(t)\), a collection of functions, a probability measure, or the weights of another neural network. The unifying idea is to replace the ordinary input map by an operator that is native to the input space—for example a functional inner product, a continuous integral transform, a convolution kernel \(u(s-t)\), a learned basis projection, a measure-integration layer, or a permutation-equivariant attention mechanism on weight space—while retaining end-to-end optimization by backpropagation [0709.3641, 2006.09590, 2301.05869, 2106.10414, 2307.03487, 2305.13546].

## 1. Historical development and terminological scope

Early work in this area arose from functional data analysis (FDA), where spectra, temporal series, and related objects are treated as elements of \(L^2\) rather than as raw sampled vectors. Rossi et al. showed how to extend both RBFN and MLP models to functional data observed through noisy input-output pairs \((t_j^i,y_j^i)\), using basis expansion, functional principal component analysis, functional centering and reduction, and differential operators before applying neural models in coefficient space [0709.3641]. In that formulation, the functional aspect is introduced through representation and preprocessing, while the downstream learner remains close to a classical neural network.

Subsequent work made the functional structure part of the network architecture itself. Wang et al. introduced a densely connected feed-forward network for scalar responses with multiple functional and scalar covariates, in which each first-layer neuron carries a dynamic functional weight \(\beta_{ik}(t)\) and produces an activation from a functional inner product \(\int \beta_{ik}(t)x_k(t)\,dt\) [2006.09590]. Thind, Multani and Cao developed the same idea for classification, coupling a functional first layer with a softmax output and cross-entropy training [2010.04305]. Other strands then expanded the term further: shift-invariant functional CNN/MLP architectures for EEG [2301.05869], continuous-hidden-layer function-on-function models [2104.09371, 2107.14151], adaptive basis layers whose basis functions are themselves micro neural networks [2106.10414], neural functionals on weight space via Transformers [2305.13546], and fully connected networks with a measure-integration layer for distribution regression [2307.03487].

A recurring source of confusion is that the abbreviation “FNN” does not denote a single canonical model. In the cited literature it names several related constructions that all modify the network so that functional structure is handled explicitly rather than discarded at preprocessing time. This suggests that FNN is best read as a family of architectures rather than a single fixed design.

## 2. Canonical scalar-output FNNs with functional covariates

The best-known scalar-on-function formulation uses a standard feed-forward network whose first hidden layer accepts functional covariates \(x_k(t)\) and optional scalar covariates \(z_j\). For neuron \(i\) in layer 1, the activation is
\[
v_i^{(1)} \;=\; g\!\Bigl(\sum_{k=1}^K \int_{\mathcal T}\beta_{ik}(t)\,x_k(t)\,\mathrm{d}t \;+\;\sum_{j=1}^J w_{ij}^{(1)}\,z_j\;+\;b_i^{(1)}\Bigr),
\]
with \(g\) an activation such as ReLU or sigmoid, \(w_{ij}^{(1)}\) ordinary scalar weights, and \(b_i^{(1)}\) biases. After this functional layer, the network proceeds through one or more ordinary dense layers and produces a scalar output \(\hat y\) for regression or class logits for classification [2006.09590, 2010.04305].

Because \(\beta_{ik}(t)\) is infinite-dimensional, it is approximated in a finite basis,
\[
\beta_{ik}(t)=\sum_{m=1}^{M_k} c_{ikm}\,\phi_{ikm}(t)=\mathbf c_{ik}^\top\boldsymbol\phi_{ik}(t).
\]
Substitution yields
\[
\int_{\mathcal T}\beta_{ik}(t)x_k(t)\,\mathrm{d}t
=\sum_{m=1}^{M_k}c_{ikm}\int_{\mathcal T}\phi_{ikm}(t)x_k(t)\,\mathrm{d}t
=\sum_{m=1}^{M_k}c_{ikm}\,\tilde\phi_{ikm},
\]
where the quantities \(\tilde\phi_{ikm}\) are precomputed numerically, for example by Simpson’s rule [2006.09590]. The functional input therefore enters the network as a learned linear projection of the curve, but the coefficients are tied so as to define a smooth weight function over the domain.

In classification-oriented versions, the deeper layers are purely multivariate,
\[
v^{(u)}_{\ell}=g\bigl(W^{(u)}v^{(u-1)}_{\ell}+b^{(u)}\bigr),
\]
followed by logits
\[
a_{\ell,h}=\sum_{i=1}^{n_U} w^{(U+1)}_{h,i}\,v^{(U)}_{\ell,i}+b_h^{(U+1)},
\]
softmax probabilities
\[
\hat p_{\ell,h}=\frac{\exp(a_{\ell,h})}{\sum_{\ell'=1}^{H}\exp(a_{\ell,\ell'})},
\]
and the decision rule \(\hat y_\ell=\arg\max_h \hat p_{\ell,h}\) [2010.04305]. Relative to naive discretization, the key distinction is that the first layer parameterizes how the network “looks” across the continuum.

## 3. Major architectural variants

The literature now includes several distinct FNN families, which differ primarily in whether they reduce functions to coefficients immediately, preserve function-valued hidden states, or generalize the notion of “functional input” to measures or neural-network weights.

| Variant | Defining mechanism | Representative paper |
|---|---|---|
| Functional preprocessing + classical RBFN/MLP | Basis expansion, fPCA, centering/reduction, or differential operators produce coefficient vectors \(\psi(x)\) for standard neural models | [0709.3641] |
| Functional-input first layer | First-layer neurons use \(\int \beta(t)x(t)\,dt\), then standard dense layers | [2006.09590], [2010.04305] |
| Functional MLP / functional CNN | Hidden layers output functions; shift-invariance enforced by \(w(s,t)=u(s-t)\) | [2301.05869] |
| FDNN / FBNN or FFDNN / FFBNN | Continuous hidden layers with weight surfaces \(w(s,t)\), either learned directly or via basis coefficients | [2104.09371], [2107.14151] |
| Adaptive basis FNN | Basis Layer with nodes \(\beta_i(t)=\mathrm{nn}_{\Theta_i}(t)\), learned end-to-end | [2106.10414] |
| Distribution-input FNN | Measure-integration layer \(\int_\Omega h(x)\,d\mu(x)\) inserted into a ReLU network | [2307.03487] |
| Neural Functional Transformers | Weight entries of a primary network are treated as tokens for a permutation-equivariant Transformer | [2305.13546] |

In the shift-invariant functional CNN of “Functional Neural Networks: Shift invariant models for functional data with applications to EEG classification,” the hidden representation remains functional:
\[
H_k^\ell(s)=\sigma\!\Bigl(b_k^\ell(s)+\sum_j\int_0^1 u_{j,k}^\ell(s-t)\,H_j^{\ell-1}(t)\,dt\Bigr),
\]
with \(u_{j,k}^\ell\) supported on \([-b,b]\), so translations \(x(t)\mapsto x(t+\tau)\) are handled by construction up to boundary effects [2301.05869].

In FDNN/FBNN and their function-on-function extensions, each neuron is itself a continuous operator. A typical hidden unit has output
\[
H_k^{(l)}(s)=\sigma\!\Bigl(b_k^{(l)}(s)+\sum_j\int_0^1 w_{j,k}^{(l)}(s,t)\,H_j^{(l-1)}(t)\,dt\Bigr),
\]
where \(w_{j,k}^{(l)}(s,t)\) is a weight surface and \(b_k^{(l)}(s)\) a bias function. The basis versions replace the surfaces by finite coefficient arrays in chosen bases such as B-splines, wavelets, or Fourier functions [2104.09371, 2107.14151].

AdaFNN modifies the first-layer projection more radically by learning the basis functions themselves. Each basis node is a micro-MLP,
\[
\beta_i(t)=\mathrm{nn}_{\Theta_i}(t),
\qquad
c_i=\langle \beta_i,X\rangle=\int_0^1 \beta_i(t)\,X(t)\,dt,
\]
so the network jointly learns a parsimonious embedding \(c\in\mathbb R^d\) and the downstream predictor [2106.10414].

The most abstract generalizations no longer take temporal curves as input. In distribution regression, the input is \(\mu\in P(\Omega)\), and the network inserts the expectation layer
\[
H^{(J_1)}(\mu)=\int_\Omega h^{(J_1)}(x)\,d\mu(x),
\]
after which ordinary ReLU layers continue on this integrated representation [2307.03487]. In Neural Functional Transformers, the input is a set of weight tensors \(W=\{W^{(i)}\}\) from a feedforward network, each scalar or channel-vector weight becomes a token, and self-attention is designed to be equivariant exactly to neuron-permutation symmetries [2305.13546].

## 4. Estimation, preprocessing, regularization, and interpretability

Training procedures depend on the architectural family, but several components recur. Functional covariates are often smoothed and normalized before entering the network. In the EEG-oriented functional CNN/MLP framework, the observed multivariate time series are smoothed into continuous functions by local polynomial smoothing, derivatives may be estimated, and each smoothed function is normalized to zero mean and unit \(L^2\)-norm [2301.05869]. In preprocessing-oriented models, basis expansion on B-splines or Fourier functions, Cholesky rescaling of Gram matrices, fPCA, and differential operators turn sampled curves into coordinates compatible with neural training while preserving \(L^2\) inner products and norms [0709.3641].

Optimization in first-layer functional-input FNNs is standard backpropagation through precomputed integrals. For scalar-response regression, one minimizes
\[
R(\theta)=\sum_{\ell=1}^N\bigl(y_\ell-\hat y_\ell(\theta)\bigr)^2
\]
over parameters \(\theta=\{c_{ikm},w_{ij}^{(u)},b_i^{(u)}\}\), and updates follow
\[
a\leftarrow a-\gamma\,\overline{\partial R/\partial a}
\]
with mini-batch Adam or stochastic gradient descent [2006.09590]. Classification models use cross-entropy loss with optional weight decay, together with mini-batch gradient descent, learning-rate decay, early stopping, and dropout [2010.04305].

Continuous-hidden-layer models require functional gradients. In FDNN/FBNN and function-on-function FFDNN/FFBNN, Fréchet derivatives are derived for weight surfaces \(w^{(l)}_{j,k}(s,t)\) and bias functions, and numerical training discretizes \(s\) and \(t\) so that dot products are replaced by numerical integrals on a grid [2104.09371, 2107.14151]. Regularization can include weight decay, dropout, early stopping, direct low-dimensional basis truncation, and roughness penalties such as
\[
\sum_{i,k}\int\bigl[\beta_{ik}''(t)\bigr]^2\,dt
\]
for first-layer weight functions or
\[
\lambda_b \sum_{l,k}\int [b_k^{(l)\prime\prime}(s)]^2 ds
+\lambda_w \sum_{l,j,k}\iint [\Delta w_{j,k}^{(l)}(s,t)]^2 ds\,dt
\]
for continuous hidden layers [2006.09590, 2107.14151].

Interpretability is one of the distinguishing claims of these models. Because the first-layer weights are themselves smooth functions, practitioners can visualize \(\beta_{ik}(t)\) across epochs, or average across neurons via
\[
\hat\beta_k(t)=\frac1{n_1}\sum_i \hat\beta_{ik}(t).
\]
The resulting peaks, valleys, and zero-crossings can be read analogously to a classical functional linear coefficient, but they are learned within a nonlinear network. Reported behavior includes validation error plateauing after approximately 100 epochs and \(\beta_k(t)\) settling into a stable shape [2006.09590].

## 5. Theoretical properties

Several FNN lines are accompanied by explicit approximation or generalization guarantees. The classification-oriented functional-input architecture is argued to retain the expressive power of a fully connected network: by Csáji-type arguments and a Fubini-projection trick, the model is described as having universal approximation capacity while using basis-projected functional inputs [2010.04305].

AdaFNN provides a more direct consistency statement. If the target map factorizes as
\[
T: C([0,1])\xrightarrow{g}\mathbb R^q\xrightarrow{h}\mathbb R,
\]
where \(g\) is linear and continuous and \(h\) is continuous, then under accurate quadrature and sufficient width/depth of the micro-net basis nodes and downstream network, for any \(\epsilon>0\) there exists a parameter choice such that
\[
\sup_{\|f\|_2\le 1} |\widehat T(f)-T(f)|<\epsilon.
\]
The same work also states a generalization-error result for stochastic gradient descent under Lipschitz and compactness assumptions [2106.10414].

For distribution regression, the input-space theory is more explicit. The hypothesis class augments a ReLU network with a measure layer, and constructive approximation theorems show that ridge-type functionals and more general composite-polynomial functionals can be approximated at rate \(O(N^{-\beta})\) by networks of types \((1,2)\) and \((2,3)\), respectively. A two-stage error decomposition—reflecting both the sampling of distributions and the sampling from each distribution—then yields almost-optimal learning rates up to logarithmic factors, with
\[
N\sim m^{\frac{1}{2\beta+1}}
\quad\text{and}\quad
\mathbb E\bigl[\mathcal E(f_{\hat D})-\mathcal E(f_\rho)\bigr]
\lesssim
m^{-\frac{2\beta}{2\beta+1}}\ln m
\]
under the stated sample-size conditions [2307.03487].

Neural Functional Transformers add a different type of theory: their weight-space attention construction is paired with a minimal-equivariance theorem. The attention pattern—rows, columns, and a global term, plus a learned per-layer position encoding—is proved to be equivariant exactly to the neuron-permutation group \(S=S_{n_0}\times\cdots\times S_{n_L}\) and to no larger subgroup of permutations [2305.13546]. This is a structural guarantee rather than an approximation result, but it addresses a central identifiability issue in neural functionals on weight space.

## 6. Empirical performance, software, and limitations

Empirical evaluations span regression, classification, function-on-function prediction, EEG decoding, chemometrics, environmental data, and neural-network weight-space tasks.

| Setting | Reported result | Paper |
|---|---|---|
| Bike rentals, daily count from hourly temperature curves | Cross-validated MSPE \(=0.0669\), \(R^2=0.582\); baselines had \(R^2\) in \([0.048,0.545]\) | [2006.09590] |
| Tecator spectroscopy, fat prediction | Six-layer FNN with approximately 4,000 parameters achieved normalized test-error \( \mathrm{MEP}=0.0088 \) | [2006.09590] |
| Canadian weather, precipitation from daily temperature | Leave-one-out MSPE \(=0.0194\), \(R^2=0.54\) | [2006.09590] |
| EEG classification | Best FNN achieved approximately \(69\%\) accuracy with approximately \(2.3\)k parameters on BCI Competition IV 2A; in sliding-window classification FNN(40) achieved \(56.3\%\) accuracy versus EEGNet’s \(51.8\%\) | [2301.05869] |
| Spectrographic and phoneme classification | Wine \(0.92\pm0.06\), orange juice \(0.81\pm0.08\), fungi \(0.78\pm0.06\), phoneme \(0.91\pm0.04\) | [2010.04305] |
| INR classification with Neural Functional Transformers | MNIST \(98.5\%\) versus \(92.9\%\), FashionMNIST \(79.3\%\) versus \(75.6\%\), CIFAR-10 \(63.4\%\) versus \(46.6\%\), up to \(+17\%\) | [2305.13546] |
| FuncNN package case studies | Gasoline data tuned MSPE approximately \(0.0043\); tecator classification accuracy approximately \(98.2\%\) on hold-out test | [2009.09111] |

Simulation evidence is equally prominent. In scalar-on-function regression, FNNs were reported to recover the true \(\beta(t)\) comparably to a functional linear model when the true link is linear, and far more accurately under exponential, logistic, and log links; across 100 replicates in four nonlinear scenarios they consistently achieved or rivaled the best MSPE against multivariate methods such as MLR, LASSO, random forests, boosting, and PPR as well as standard functional approaches [2006.09590]. In the classification setting, three simulated designs with 150 replicates each showed lower MSPE for FNN than both a standard functional linear model and a conventional neural network on raw discretized curves [2010.04305]. FDNN/FBNN and FFDNN/FFBNN studies likewise reported superior out-of-sample RMSE in increasingly nonlinear data-generating regimes and gains on Tecator, Berkeley growth curves, TIMIT phoneme data, electricity demand, and bike rentals [2104.09371, 2107.14151].

Software support has concentrated in R on top of Keras. The regression paper announced a forthcoming R package built on Keras for general use [2006.09590]. “FuncNN: An R Package to Fit Deep Neural Networks Using Generalized Input Spaces” described an R library presented as the first such package in any programming language, with user-facing functions including `fnn.fit`, `fnn.predict`, `fnn.fnc`, `fnn.cv`, and `fnn.tune`, plus diagnostics such as training-history plots and estimated average coefficient functions \(\hat\beta_k(t)\) [2009.09111].

Limitations are reported consistently across the literature. Additional hyperparameters—basis size, hidden layers, neurons, activation functions, regularization, and learning-rate schedules—require careful tuning; training is slower than simple functional regression; stochastic initialization can introduce variance in \(\hat\beta(t)\); and overfitting remains a concern when models are deep and sample sizes are limited [2006.09590, 2010.04305]. Some models depend heavily on basis selection, whereas others such as AdaFNN are motivated precisely by the inadequacy of fixed, a priori bases for task-relevant variation [2106.10414]. Another common misconception is that FNNs are simply ordinary MLPs fed discretized curves. The published formulations show otherwise: the defining mechanisms are architectural and operator-theoretic, not merely representational, and may involve functional inner products, continuous neurons, convolutional weight functions, adaptive bases, measure integration, or symmetry-preserving attention [2301.05869, 2307.03487, 2305.13546].

Taken together, the literature positions Functional Input Neural Networks as a broad interface between FDA and deep learning. Their central contribution is to preserve the structure of continuous or function-like inputs inside the network itself, thereby combining nonlinear predictive modeling with smoothness, equivariance, or interpretability constraints that are native to the underlying data domain.

Source: https://www.emergentmind.com/topics/functional-input-neural-networks-fnn