---
title: Primitive-Specific MLP Architecture
url: https://www.emergentmind.com/topics/primitive-specific-multilayer-perceptron-mlp-d4257cbb-f79d-4815-b8b2-2c23d88cb8ed
type: topic
---

# Primitive-Specific MLP Architecture

A primitive-specific multilayer perceptron (MLP) is a feed-forward neural architecture systematically constructed by decomposing complex classification tasks into geometric primitives, training dedicated MLPs for these primitive regions, then algebraically combining these subnetworks to yield a tailored, interpretable classifier. The resulting architecture and its construction principles are grounded in the formalism of MLP-algebra, which provides a suite of closed, compositional operations for synthesizing complex decision boundaries from a finite library of primitive MLPs [1701.04968].

## 1. The Formal Framework: Universe of MLPs and Primitives

The universe $\mathcal{M}$ comprises all feed-forward MLPs with $L \geq 2$ layers, arbitrary finite widths, and (unless otherwise specified) sigmoid activations. Each network $\mathcal{N} \in \mathcal{M}$ is specified by its layer dimensions $(n_1, ..., n_L)$, its collection of weight matrices $\omega^k \in \mathbb{R}^{n_{k+1} \times n_k}$, and threshold vectors $\theta^k \in \mathbb{R}^{n_{k+1}}$. The forward map is realized by iterating $x \mapsto \sigma(\omega^1 x - \theta^1) \rightarrow \cdots \rightarrow \mathcal{N}(x) \in \mathbb{R}^{n_L}$.

A network is designated as "primitive" if it is trainable to act as the characteristic function of a simple geometric set. Canonical examples include:

- **Half-space**: $ \{x : a \cdot x \geq b \} $ (single-layer perceptron).
- **Ball/disk**: $ \{x: \lVert x - c \rVert < r \} $ (two-layer: $n \rightarrow n+1 \rightarrow 1$).
- **Axis-aligned boxes**: $ \{\ell_i \leq x_i \leq u_i\} $ (hidden layer size $2n$).
- **Cartesian products and unions of lower-dimensional primitives**.

For a concrete task, a finite library $\mathcal{G} \subset \mathcal{M}$ of such primitives is selected to serve as the building blocks for higher-complexity MLP construction.

## 2. Algebraic Operations for MLP Composition

The MLP-algebra defines a set of operators that act on compatible networks (i.e., with matching input spaces and structure) to produce new networks also in $\mathcal{M}$. These include:

- **Complement $(\cdot)^c$**: For 1D-output MLP $\mathcal{N}$, the complement $\mathcal{N}^c$ is constructed by negating the final-layer weights and thresholds, thereby inverting the decision boundary via the identity $\sigma(-z) = 1 - \sigma(z)$.
- **Sum (Union) $+$**: Combines two L-layer, scalar-output MLPs so that the resulting network represents the union of the individual decision regions. The final layer applies a sharp sigmoid with weights $\lambda(1,1)$ and threshold $0.5\lambda$, enforcing logical OR.
- **Multi-Sum $\sum$**: Generalizes the union operator to $m$ networks, stacking hidden layers accordingly.
- **Difference $-$**: Set difference realized algebraically as $\mathcal{N}_1 - \mathcal{N}_2 = \mathcal{N}_1 + (\mathcal{N}_2)^c$.
- **$I$-Product (Intersection) $\times$**: Forms the intersection of the positive regions of two compatible MLPs; constructed using block-diagonal combining at each hidden layer and weight $\lambda(1,1)$, threshold $1.5\lambda$ in the final layer.
- **Multi-$I$-Product $\prod$**: Multi-way intersection.
- **Component Extraction $(\cdot)^{(l)}$**: Extracts the $l$-th output neuron from a multi-label MLP, yielding a binary classifier.
- **$\mathcal{O}$-Product $\otimes$**: Concatenates two binary classifiers into a 2-label MLP by direct-sum of hidden layers and stacking the outputs.
- **Identical Extension $T$**: Appends an identity mapping layer (linear or ReLU) to align depth for algebraic summing or multiplication with deeper networks.

These operations are closed in $\mathcal{M}$, enabling hierarchical design without leaving the universe of valid MLPs [1701.04968].

## 3. Key Algebraic Properties

Networks with 1D output under the fundamental operators obey a commutative algebra:

| Property       | Formal Statement                                      | Intuitive Explanation                                        |
|----------------|------------------------------------------------------|--------------------------------------------------------------|
| Commutativity  | $\mathcal{N}_1 + \mathcal{N}_2 = \mathcal{N}_2 + \mathcal{N}_1$; $\mathcal{N}_1 \times \mathcal{N}_2 = \mathcal{N}_2 \times \mathcal{N}_1$ | Order of union/intersection does not affect result           |
| Associativity  | $(\mathcal{N}_1 + \mathcal{N}_2) + \mathcal{N}_3 = \mathcal{N}_1 + (\mathcal{N}_2 + \mathcal{N}_3)$, etc.   | Grouping does not affect the output decision boundary        |
| Distributivity | $\mathcal{N}_1 \times (\mathcal{N}_2 + \mathcal{N}_3) \approx (\mathcal{N}_1 \times \mathcal{N}_2) + (\mathcal{N}_1 \times \mathcal{N}_3)$ (up to scaling/thresholds)         | Logical AND distributes over OR at the decision level        |
| Involution     | $(\mathcal{N}^c)^c = \mathcal{N}$                    | Double complement restores the original decision boundary     |

Closure is immediate from the recursive definitions of weights, thresholds, and dimensions. Associativity and commutativity arise from block-diagonal and concatenation symmetries. Distributivity is approximate, depending on the sharpness parameter $\lambda \gg 1$; exactness can be enforced by further SGD fine-tuning. Involution of complement is algebraic [1701.04968].

## 4. Construction Methodology and Pseudocode

Building a primitive-specific MLP proceeds as follows:

1. For each label $\ell$ in $1 \ldots K$, partition the label-specific data $D_\ell$ into regions approximately corresponding to primitives in $\mathcal{G}$ (e.g., via clustering or geometric heuristics).
2. Train an MLP for each primitive region $P_{\ell, j}$—selecting the appropriate template (half-space, ball, box) and fine-tuning as a characteristic function.
3. Combine the label-specific primitive nets by multi-sum (union): $N_\ell = \sum_j N_{\ell, j}$.
4. Merge all $K$ label-specific nets by the $\mathcal{O}$-product to yield a $K$-output classifier.
5. Optionally, fine-tune the assembled MLP on the full labeled set for several epochs to enhance logical gating.

Pseudocode (appearing verbatim in the source):

```python
function BuildPrimitiveMLP(D, G):
  Input: D = { (xᵢ, yᵢ) } ⊂ ℝⁿ×{1..K}; G = library of primitive template-nets
  Output: an MLP N: ℝⁿ→ℝᴷ

  for each label ℓ in 1..K:
      D_ℓ ← { xᵢ | yᵢ = ℓ }
      Partition D_ℓ into simpler subsets {P_{ℓ,1}, …, P_{ℓ,mℓ}}
          (e.g. by clustering or geometric heuristics)
      for j=1..mℓ:
          choose a primitive template T in G (e.g. half-space, disk, box)
          train N_{ℓ,j} = characteristic MLP of P_{ℓ,j} (fine-tune on positive vs outside)
      N_ℓ_interim = Sum_{j=1..mℓ}( N_{ℓ,j} )
  N_final = N_1_interim ⊗ N_2_interim ⊗ … ⊗ N_K_interim
  Optionally fine-tune N_final on the full labeled set D for a few epochs
  return N_final
```
Regions embedded in projected subspaces can be re-aligned using the $I$-product operation [1701.04968].

## 5. Practical Strategies for Primitive Selection and Capacity

Choosing optimal primitives:

- For clusters near-linearly separable, use half-space MLPs.
- For "blob-like" clusters, employ ball/disk templates ($n \rightarrow n+1 \rightarrow 1$).
- For box-shaped clusters, use axis-aligned box MLPs (hidden layer size $2n$).
- If primitive fitting is nontrivial, approximate with unions of basic shapes.

Architectural and learning considerations:

- Depth increments by one whenever a sum or $I$-product is applied, corresponding to an added combining layer.
- Width increases via block-diagonal concatenation as networks are joined.
- The sharpness parameter $\lambda$ modulates how Boolean-like the logical gates become; higher $\lambda$ yields sharper transitions, with initial value $\lambda \approx 10$ suggested, tuned as needed.
- After construction, re-fine-tuning using SGD on the recombined data often sharpens logical implementation and mitigates softness in composite gates.

## 6. Illustrative Example: Constructing XOR in $\mathbb{R}^2$

For the classic XOR problem on $\mathbb{R}^2$, the positive region is the union of two quadrants:

- $R_1$ : $\{ x_1 > 0, x_2 > 0 \}$
- $R_2$ : $\{ x_1 < 0, x_2 < 0 \}$

Primitive networks:

- $N_1$: half-space $x_1 > 0$ (one-layer)
- $N_2$: half-space $x_2 > 0$

Compositional construction:

- $R_1$ via $N_1 \times N_2$ (intersection)
- $R_2$ via $N_1^c \times N_2^c$
- Union $N_{pos} = (N_1 \times N_2) + (N_1^c \times N_2^c)$

The final binary XOR network is $N_{xor}(x) = N_{pos}(x)$. The explicit architecture includes stacking the hidden representations, forming logical AND via sharp sigmoid ($\lambda$), and summing for the logical OR. SGD fine-tuning for a few epochs enables the MLP to achieve $>99\%$ accuracy on XOR inputs. This construction demonstrates the systematic decomposition and algebraic recombination at the heart of the MLP-algebra approach [1701.04968].

## 7. Significance and Interpretability

By leveraging primitive-specific construction, the resulting MLP encodes the logical structure of the classification task explicitly in its architecture. The compositionality of the framework ensures transparency: internal subnetworks correspond to interpretable geometric regions. For datasets that admit decompositions into simple primitives, this approach offers a systematic, design-theoretic alternative to unguided end-to-end training, with capacity and architectural complexity controlled through algebraic operations and hyperparameters. The method provides provable guarantees about the underlying logical form of the resulting MLP, unique among network construction methodologies [1701.04968].

Source: https://www.emergentmind.com/topics/primitive-specific-multilayer-perceptron-mlp-d4257cbb-f79d-4815-b8b2-2c23d88cb8ed