Model-less CNNs: Analytical & Random Techniques

• Model-less CNNs are a class of architectures that bypass traditional gradient descent by using fixed, random, or analytically computed filters for feature extraction.
• They leverage learned 1×1 convolutions and classifier-free, fully convolutional designs to enable rapid deployment, implicit regularization, and data efficiency.
• Empirical results show competitive performance on benchmarks like CIFAR-10 and MNIST while reducing training complexity and enhancing model interpretability.

A model-less Convolutional Neural Network (CNN) is a framework in which core elements of conventional CNNs—particularly spatial convolutional filters—are either not learned via standard gradient-based optimization, or all model parameters are analytically or algorithmically determined without iterative training. This paradigm challenges the canonical assumption that supervised training is essential for obtaining expressive and performant CNNs. Recent research has instantiated several ^^^^1^^^^ modalities: (i) the use of entirely fixed, randomly initialized convolutions combined with learned linear recombination layers; (ii) direct analytical computation of all network weights and thresholds from a minimal set of labeled exemplars; and (iii) CNNs trained for regression tasks by eliding fully-connected classifier heads and learning to map directly to continuous feature maps. Each approach refines or abandons the "model-centric" notion of convolutional networks, opening alternative routes for inductive bias, regularization, and data efficiency.

1. Taxonomy and Definitions

The term "model-less CNN" encompasses several non-classical CNN constructions:

• Fixed Random Convolutional Filters Plus Learned Linear Combinations: Filters in each convolutional layer are initialized at random and never updated; the only trainable parameters are in 1×1 pointwise convolutions that mix the outputs of these fixed filters. This maintains the convolutional inductive bias while dispensing with standard filter learning (Gavrikov et al., 2023).
• Analytic Construction of All Network Parameters: All weights, biases, and even channel counts are computed in closed-form from a small set of labeled exemplars, usually without any back-propagation or gradient descent. The operating principle is that task-relevant features can be formalized directly as set-theoretic or statistical relations among prototypes (Geidarov, 26 May 2025).
• Classifier-Free CNNs for Regression: Fully-connected layers are omitted, and the network learns to regress to desired continuous target maps (e.g., for segmentation, detection, or saliency), effectively reframing the learning problem and reducing architectural constraints imposed by classification-specific heads (Yuan et al., 2014).

2. Formalization and Architecture

Fixed-Random Linear Combination CNNs

Let $W_\text{rand} = \{W_i^\text{rand}\}_{i=1}^m$ denote a collection of $m$ fixed, randomly initialized $k\times k$ convolutional kernels. Given input $X\in\mathbb{R}^{c_\text{in}\times H\times W}$:

• The $i$-th random feature map is $R_i = W_i^\text{rand} * X$.
• A learned $1\times 1$ convolution (parameterized by $\{\alpha_i\}$) linearly recombines these maps:
  • $Y = \sum_{i=1}^m \alpha_i \cdot R_i$
  • Or, for $c_\text{out}$ output channels, $Y_p = \sum_{i=1}^m A_{p,i}R_i$ with $A\in\mathbb{R}^{c_\text{out}\times m}$.

This structure is deployed in standard architectures such as ResNet, where each $k\times k$ convolution is replaced by the sequential composition of a frozen spatial convolution and a trainable $1\times 1$ convolution, often without intervening nonlinearity. Only the $1\times 1$ weights, normalization parameters, and final classifier weights are learned (Gavrikov et al., 2023).

Analytically Constructed CNNs

All weights and thresholds across convolutional and fully-connected layers are analytically derived from a fixed, small set of prototypes:

• For MNIST, given 10 prototype images (one per class), $5\times 5$ spatial features are detected by scanning and binarization; filters are constructed per detected feature via

$$W_{ij}^{(0)} = \frac{P_{ij}\times 2}{\max(P_{ij})} - 1$$

• Per-feature and per-layer biases are prescribed according to normalized dot-products and user-set scaling parameters.
• Channel counts in each layer are determined by the number of unique features extracted from the prototypes.
• Fully-connected layers are formed via explicit difference tables and pre-set thresholds, enabling direct mapping from activation maps to class logits without any learned adaptation (Geidarov, 26 May 2025).

Classifier-Free Regression CNNs

For image-to-map regression tasks, model-less CNNs (e.g., Half-CNN) forego all fully-connected heads. Instead, the final output is produced by a $1\times 1$ convolution (linear combination) of the final convolutional layer’s channels, followed by a sigmoid, mapping to a continuous-valued prediction map. The network is inherently fully convolutional and can handle variable input/image sizes. Learning is by minimizing regularized mean squared error between predicted and true maps:

$$L(\theta) = \frac{1}{N}\sum_n \|M\odot(f(X^n;\theta) - Y^n)\|_2^2 + \lambda\|\theta\|_2^2$$

with $M$ as the validity/border mask (Yuan et al., 2014).

3. Theoretical Characteristics and Expressivity

• Expressive Power with Fixed Filters: A central lemma is that a $1\times 1$ convolution over fixed $k\times k$ random filters can synthesize any desired convolutional operator, provided the filter bank is sufficiently overcomplete. In high probability, random filters span the relevant function space, and learned linear mixtures can emulate or approximate any target kernel by the Johnson–Lindenstrauss/random-features principle (Gavrikov et al., 2023).
• Implicit Regularization: Freezing the spatial filters reduces over-parameterization by shrinking the learnable weight set, which empirically enhances generalization and mitigates filter “collapse”—quantitatively measured by filter-variance entropy—and confers mild robustness to adversarial perturbations (Gavrikov et al., 2023).
• Direct Analytic Modeling: By tying all parameters directly to dataset exemplars, analytic model-less CNNs eliminate learnable redundancy, trading adaptability for analytical transparency and rapid deployment (Geidarov, 26 May 2025).
• Classifier-Free Structure: Model-less regression CNNs avoid all constraints and inductive biases associated with fully-connected heads, admitting arbitrary input sizes and outputs, and acting as universal local function approximators over images (Yuan et al., 2014).

4. Empirical Findings and Benchmarks

Approach	Dataset	Accuracy / Metric	Notes
Frozen random filters + 1×1 linear combinations	CIFAR-10	91.9% (ResNet-20-16, E=128)	Surpasses learned baseline (91.6%)
"	CIFAR-100	61.3% (LC-frozen, E=128)	Matches/design-matched baseline
"	ImageNet	72.5% Top-1 (ResNet-18d, E=64)	Near parity with fully learned
Analytic CNN (no training)	MNIST	58.3% (best config, 10 prototypes, 9.2 s build time)	Instant deployment, no training
Regression CNN (Half-CNN)	LFW	>95% face-window retrieval	Direct regressed detection map
"	MIT1003	AUC ≈ 0.75, sAUC ≈ 0.62 (CNN-R)	Saliency, outperforms classic models

• Freezing all spatial convolutions in deep/wide nets with enough expansion (E ≈ 128) nearly closes the gap to fully-trained baselines for both small and large kernel sizes; for small ($k=3$) kernels, learning confers little benefit. Gains from learning filters increase with kernel size due to the i.i.d. variance structure of common initializations (Gavrikov et al., 2023).
• Analytic (training-free) CNNs operate with extremely low latency and require only a handful of labeled inputs, yet their accuracy is below standard learned models unless further fine-tuned. Performance improves with additional prototypes or downstream adaptation (Geidarov, 26 May 2025).
• Half-CNN achieves competitive results in both detection/segmentation (face localization on LFW, >95% recall) and saliency (AUC ≈ 0.75, outperforming AWS, AIM, and other classical algorithms) without any classifier head (Yuan et al., 2014).

5. Practical Methodologies

• Random Linear Combination (LC) Networks: Implemented as a stack of $k\times k$ frozen random convolutions, each followed by a learned $1\times 1$ convolution to allow expressive feature mixing. All non-flat layers (BatchNorm, other normalization) are retained. Trained with standard cross-entropy, SGD + Nesterov momentum, and cosine-annealing learning rate; only pointwise and downstream weights are updated (Gavrikov et al., 2023).
• Analytic Construction: Begins with prototype selection and binarization, feature mask extraction via windowed scanning, filter construction by rescaling and centering per-feature patches, bias computation via normalized dot products, and fully connected construction via explicit “difference tables” among activation maps. Algorithm is implemented fully in C++ arrays for efficiency (Geidarov, 26 May 2025).
• Regression CNNs: Training for 2D regression is performed with mean squared error on masked real-valued maps, with no constraints on input/output spatial size. The final prediction combines channels by a learned linear weighting and applies a sigmoid for bounded outputs. L-BFGS optimization is preferred for rapid convergence in small models (Yuan et al., 2014).

6. Design Considerations, Limitations, and Prospects

• Generalization and Regularization: Model-less CNNs with frozen spatial kernels can outperform baseline learned networks in broad or deep regimes due to strong implicit regularization, but may forfeit adaptation to domain-specific patterns if the random bases or analytically derived filters omit critical structures (Gavrikov et al., 2023).
• Data Efficiency: Analytically constructed CNNs require only a handful of prototypes for deployment, but scale poorly to complex datasets unless supplemented by further training. Peak accuracy on MNIST with ten prototypes is ≈58%; accuracy increases if more exemplars are used or hybridized with brief fine-tuning (Geidarov, 26 May 2025).
• Architectural Implications: These findings suggest CNN architectures may rely more on random or analytically constructed bases, leveraging $1\times 1$ pointwise flexibility, and minimizing filter overparametrization. For regression tasks, classifier-free, fully-convolutional designs offer flexibility in resolution and application scope (Gavrikov et al., 2023, Yuan et al., 2014).
• Limitations: Model-less CNNs provide strong empirical results on select vision benchmarks, but their robustness under domain shift, scaling to novel modalities, and extension to structured generative tasks remain open. For regression tasks, direct map prediction can fail to capture global spatial dependencies unless designed to do so (Yuan et al., 2014).

7. Relationship to Broader Research Themes

Model-less CNNs intersect with several canonical ideas:

• Random Feature Models: Analogous to Rahimi–Recht random kitchen sinks, these networks use overcomplete random bases and learn only the "readout" weights, paralleling methods in kernel machines and reservoir computing.
• Fully-Convolutional Design: Classifier-free regression CNNs instantiate the principle that spatial structure, not dense connections, is foundational to image modeling (Yuan et al., 2014).
• Template and Metric Learning: Analytic CNN construction formalizes feature extraction as a deterministic, prototype-driven process akin to classic template matching and metric-based recognition (Geidarov, 26 May 2025).
• Implicit vs. Explicit Regularization: Freezing large parameter blocks forgoes explicit regularization (such as weight decay) in exchange for hard constraints, which impacts generalization and optimization landscape (Gavrikov et al., 2023).

Model-less CNNs demonstrate that the essential power of deep convolutional networks does not necessarily reside in learned spatial filters, but may be shifted to topological expressivity, pointwise combination, and the statistical richness of the (random or analytic) convolutional bases.