Generalized Operational Perceptron (GOP)

• GOP is a neural computational unit that extends the classical perceptron by parameterizing diverse nonlinear operations (e.g., multiplication, exponential, sinusoidal) to enhance expressivity.
• It supports both homogeneous and heterogeneous architectures, enabling progressive, data-driven growth of multilayer networks that adaptively select operator sets.
• Empirical studies show GOP-based networks achieve high accuracy with compact parameter counts and efficient training, making them ideal for resource-constrained applications.

The Generalized Operational Perceptron (GOP) is a neural computational unit that extends the expressivity of the classical perceptron by parameterizing diverse nonlinear operations at the level of each individual synapse, pooling, and post-pooling activation. Designed to mimic a broader range of biological neuronal activities than the standard McCulloch–Pitts neuron, the GOP underpins a family of multilayer neural architectures and progressive learning strategies that seek compact, heterogeneous, and high-performing predictors for a variety of classification and regression tasks (Tran et al., 2018, Tran et al., 2018).

1. Mathematical Definition and Operator Libraries

Unlike the traditional perceptron, which computes a linear weighted sum of inputs followed by a fixed nonlinearity, the GOP neuron at layer $\ell+1$ is specified as follows:

Given inputs $y_k^{\ell}$ ($k=1,\ldots,N_\ell$), synaptic weights $w_{ki}^{\ell+1}$, and bias $b_i^{\ell+1}$, the neuron output is computed in three stages:

1. Nodal (synaptic) operation:

$$z_{ki}^{\ell+1} = \psi_i^{\ell+1}(y_k^{\ell}, w_{ki}^{\ell+1})$$

where $\psi$ is selected from a library $\Psi$ of nonlinear transforms, e.g., - multiplication: $w y$ - exponential: $\exp(w y) - 1$ - quadratic: $w y^2$ - Gaussian: $w \exp(-w y^2)$ - DoG: $w y \exp(-w y^2)$ - sinusoidal: $\sin(w y)$

2. Pooling:

$$x_i^{\ell+1} = \rho_i^{\ell+1}(z_{1i}^{\ell+1}, \ldots, z_{N_\ell,i}^{\ell+1}) + b_i^{\ell+1}$$

with pooling operator $\rho$ drawn from a set $P$ including: - summation: $\sum_k z_k$ - 1- and 2-correlation: $\sum_k z_k z_{k+1}$, $\sum_k z_k z_{k+1} z_{k+2}$ - maximum: $\max_k z_k$

3. Activation:

$y_i^{\ell+1} = f_i^{\ell+1}(x_i^{\ell+1})$

$f$ is chosen from an activation library $F$ such as sigmoid, tanh, ReLU, softplus, inverse absolute, ELU.

Consequently, the forward pass of a GOP is expressed as

$$y = f\left(\rho\left(\left\{\psi(y_k, w_k)\right\}_k\right) + b\right)$$

Using $(\psi, \mathrm{sum}, \mathrm{sigmoid})$, the GOP reduces to the McCulloch–Pitts neuron, but the enlarged operator set renders GOP strictly more expressive (Tran et al., 2018, Tran et al., 2018).

2. Network Topologies and Heterogeneity

A generalized operational perceptron network can be instantiated as either:

• Homogeneous: all neurons within a layer share the same choice of $(\psi, \rho, f)$ (as in standard POP/POPfast).
• Heterogeneous: each neuron may independently select its operator set from the full library, leading to a rich, highly flexible multilayer architecture.

There is no architectural prior fixing the width (neurons per layer) or depth (number of layers); the topology is determined by data-driven progressive methods that add neurons and layers adaptively. Fully heterogeneous multilayer GOPs can thus configure each neuron's computation to optimally capture diverse, task-specific nonlinearities, enabling greater parameter efficiency and potentially superior generalization, especially where mixtures of nonlinear regimes are required (Tran et al., 2018).

3. Progressive Network Construction Algorithms

Several progressive training paradigms have been proposed for GOP-based networks:

3.1 Progressive Operational Perceptron (POP)

POP constructs the network template $T = [I, h_1, h_2, ..., h_N, O]$ one hidden layer at a time:

• For each new layer, layerwise greedy search identifies the best operator set $(\psi, \rho, f)$ for all neurons, followed by backpropagation to optimize synaptic weights.
• Once the new layer reaches a target loss or the fixed architecture is completed, a final joint finetuning stage is performed.
• All GOPs in a given hidden layer share the same operator set in POP.

3.2 POPfast

POPfast removes the need for an output GOP—replacing it with a linear or softmax classifier—which reduces operator-set search by a factor of four per layer and accelerates convergence with equivalent classification performance.

3.3 Fully Heterogeneous Multilayer Learning (HeMLGOP)

HeMLGOP extends the progressive idea to neuron-level granularity:

• At each step, blocks of neurons are candidate-added, with each neuron allowed to pick its operator set independently.
• Each block is assessed for loss reduction using a randomized weight assignment and closed-form output layer, then finetuned with backpropagation.
• Addition of neurons and layers is governed by threshold parameters $\epsilon_n$ and $\epsilon_l$.
• Once growth ceases, joint backpropagation across the full network is optional for global refinement.

3.4 Memory-Augmented Extensions (POPmem-H, POPmem-O)

Memory-based variants incorporate an auxiliary linear projection (via PCA or LDA) from all prior representations to each layer or output, providing additional discriminative input and facilitating deeper architectures (Tran et al., 2018).

Algorithmic complexity is dominated by operator-set randomization and pseudoinverse solving for the output weights during block addition, and then backpropagation for block and whole-network finetuning (Tran et al., 2018).

4. Training Regimes and Regularization

Training a GOP-based network involves several subtasks:

• Operator-set search: random initialization of GOP weights for each candidate triple, batch-normalized forward pass, and closed-form solution for output weights (using Moore–Penrose pseudoinverse).
• Block/batch-level finetuning: mini-batch SGD with batch normalization at hidden outputs; learning-rate schedules vary by dataset scale.
• Regularization: incorporates weight decay (typically $10^{-4}$), optional $\ell_2$-norm clipping (max norm in [1,3]), and dropout at both input (rate 0.2) and hidden outputs (rates in $\{0.1,0.3,0.5\}$).
• Losses: Mean Squared Error or cross-entropy; accuracy is also monitored for stopping conditions.
• Initialization: hidden weights are randomly uniform; output weights are solved with a small ridge parameter ($c \in \{0.1,1,10\}$).

Memory-augmented schemes augment the progressive layerwise input with learned projections (PCA or LDA). Optimization applies dropout (typically 0.5), weight-decay, and plateau-based learning rate decay. Global finetuning is generally performed at a low fixed learning rate over hundreds of epochs (Tran et al., 2018, Tran et al., 2018).

5. Empirical Performance and Benchmarks

Extensive experimental evaluations have established the empirical properties of GOP-based networks:

• Datasets: 11 classification tasks across small- (e.g. PIMA, YEAST, Hollywood3D), medium- (MIT Indoor), and large-scale (Caltech256, CFW60k) regimes.
• Comparison methods: homogeneous GOP (POP, POPfast), perceptron-based progressive systems (PMLP, PLN, BLS), and memory-augmented GOPs.
• Metrics: test accuracy (%), model size (parameter count), CPU training time (seconds), inference FLOPs per sample (Tran et al., 2018, Tran et al., 2018).

Key results include:

• HeMLGOP achieves top or near-top accuracy on all 11 tasks, with the smallest or near-smallest parameter counts. For instance, on PIMA: 81.8% accuracy with ~0.8K parameters vs. >40K in POP/PMLP (Tran et al., 2018).
• Training time: HeMLGOP is approximately 300$\times$ faster than POP, comparable to homogeneous multilayer progressive variants; PLN/BLS train fastest but with inferior accuracy or massive models (Tran et al., 2018).
• On small datasets, memory-augmented variants (POPmem-O) further improve classification accuracy beyond POP/POPfast baselines (e.g., 80.65% vs. 78.03% on Hollywood3D); computational cost increases only by the PCA/LDA projection per layer (Tran et al., 2018).
• On medium/large datasets, the best performance remains with fully heterogeneous or memory-augmented networks (e.g., 79.25% on Caltech256 with POPmem-O-PCA versus 73.93% with POPfast) (Tran et al., 2018).

Summarized results:

Method	Hollywood3D Accuracy (%)	Training Time (s/layer)
POP	78.03	48,484
POPfast	79.42	7,851
POPmem-O-PCA	80.65	10,549

6. Analytical Insights and Implications

• Expressivity: Heterogeneous operator sets per neuron enable the representation of diverse nonlinear patterns within a single model, leading to compactness and parameter efficiency, especially on heterogeneous data (e.g., functions containing both linear and highly nonlinear components) (Tran et al., 2018).
• Progressivity: Progressive neuron-level search, leveraging both randomization and selective backpropagation, provides a balance between architectural search and convergence speed. Selective randomization recovers missed operator-set choices in later neuron additions.
• Practicality: The adoption of batch normalization, dropout, and modern learning rate schedules ensures stable and efficient training.
• Memory-based augmentation: Adding direct linear projections via memory paths enables deeper architectures and improved data representation by circumventing vanishing-discriminability across layers.

This suggests that the GOP framework, particularly in its fully heterogeneous multilayer and memory-augmented instantiations, offers a modular, interpretable, and parameter-efficient paradigm for mixed-regime pattern recognition, distinct from both standard MLPs and layer-homogeneous progressive networks (Tran et al., 2018, Tran et al., 2018).

7. Limitations, Open Questions, and Application Domains

• Limitations:
  • There is no theoretical guarantee that the addition of a new layer or neuron always reduces the training loss, as it depends on improved subspace coverage by the new features; validation-based stopping is recommended.
  • Operator-set randomization introduces variance, though the progressive search may recover optimal configurations missed in early growth phases.
• Future directions:
  • Further expansion of operator-set libraries for even richer expressivity.
  • Deeper theoretical study of the subspace expansion induced by new GOP features.
  • Tighter complexity analyses for the progressive architecture search and finetuning processes.
• Domains of application: Tasks demanding compact MLP-like models with low-compute inference, such as mobile vision, wearable sensor data analysis, biomedical informatics, and embedded classifiers. The methodology is appropriate wherever compactness and nonlinear discriminative power are simultaneously required (Tran et al., 2018).

• "Heterogeneous Multilayer Generalized Operational Perceptron" (Tran et al., 2018)
• "Progressive Operational Perceptron with Memory" (Tran et al., 2018)