Multi-layer Logical Perceptron (MLLP)

• MLLP is a continuous neural architecture that relaxes discrete rule-based classifiers (CRS) by replacing binary operations with differentiable functions.
• It employs Random Binarization during training to reconcile the gap between continuous optimization and the extraction of a human-interpretable discrete rule set.
• By substituting binary weights with continuous values in [0,1] and leveraging gradient descent, MLLP ensures both high accuracy and transparency in decision-making.

Searching arXiv for MLLP and directly related papers to ground the article. arxiv_search(query="Multilayer Logical Perceptron", max_results=10) arxiv_search(query="Transparent Classification with Multilayer Logical Perceptrons and Random Binarization", max_results=5) arxiv_search(query="Deepest Neural Networks Rojas", max_results=5) Multi-layer Logical Perceptron (MLLP) is a neural architecture introduced as a continuous version of Concept Rule Sets (CRS), a hierarchical rule-based classifier for transparent classification. In this formulation, the target discrete model is a multi-level alternation of conjunction and disjunction layers, while MLLP replaces binary structure with continuous weights in $[0,1]$ and differentiable logical functions so that gradient descent can be used during training. Its stated purpose is to search the discrete solution of CRS in continuous space and to ensure that the extracted discrete CRS acts almost the same as the corresponding continuous MLLP (Wang et al., 2019).

1. Problem setting and design objective

The motivating problem is the construction of classification models that are both accurate and transparent. The underlying argument is that, in domains such as healthcare, finance, security, politics, and law, users require not only predictive performance but also an intelligible account of why a prediction was made. In the transparency notion used here, the internal components of the model—inputs, parameters, and intermediate calculations—should each have intuitive meaning (Wang et al., 2019).

Within that framing, standard deep neural networks are treated as non-transparent because hidden neuron activations are real-valued and difficult to interpret semantically. Linear models are easy to interpret but have limited expressive power for nonlinear data. Existing rule-based models, including decision trees, rule lists, and rule sets, are transparent because they operate on understandable Boolean rules, but the paper identifies two limitations: optimization is heuristic, and the discrete parameters of rules prevent direct gradient-based learning.

MLLP is introduced to close that gap. The intended workflow is explicit: define a hierarchical rule-based target model, namely CRS; construct a continuous neural counterpart, namely MLLP; optimize that continuous model by gradient descent; and then extract a discrete CRS from the trained network. This makes MLLP less a generic multilayer perceptron and more a differentiable surrogate for a logical classifier. A useful clarification follows from the paper’s terminology: the final transparent predictor is CRS, while MLLP is the trainable continuous relaxation used to obtain it.

2. Formal relationship to Concept Rule Sets

CRS is defined as a multi-level hierarchy in which each level contains a conjunction layer followed by a disjunction layer. If CRS has $L$ levels, then the odd-numbered logical layers are conjunction layers $\mathcal{R}^{(l)}$, $l\in\{1,3,\dots,2L-1\}$, the even-numbered logical layers are disjunction layers $\mathcal{S}^{(l)}$, $l\in\{2,4,\dots,2L\}$, and the input layer is denoted $\mathcal{S}^{(0)}$. Each layer has a binary adjacency matrix $W^{(l)}$ with entries

$W_{i,j}^{(l)} \in \{0,1\},$

where $W_{i,j}^{(l)}=1$ means an edge exists from node $L$0 in layer $L$1 to node $L$2 in layer $L$3 (Wang et al., 2019).

The node semantics are explicitly logical. A conjunction-layer node is a rule,

$L$4

and a disjunction-layer node is a rule set,

$L$5

A rule is a conjunction of Boolean variables,

$L$6

and a rule set is a disjunction of rules, that is, a disjunctive normal form (DNF),

$L$7

MLLP preserves this architecture and node-to-neuron correspondence but relaxes the discrete structure. Binary weights $L$8 are replaced by continuous weights $L$9, and hard Boolean conjunction/disjunction are replaced by differentiable logical functions. The paper defines

$\mathcal{R}^{(l)}$0

$\mathcal{R}^{(l)}$1

with

$\mathcal{R}^{(l)}$2

and

$\mathcal{R}^{(l)}$3

These selector functions have a precise role. For conjunction, $\mathcal{R}^{(l)}$4 yields $\mathcal{R}^{(l)}$5, while $\mathcal{R}^{(l)}$6 yields $\mathcal{R}^{(l)}$7, the identity element for AND. For disjunction, $\mathcal{R}^{(l)}$8 yields $\mathcal{R}^{(l)}$9, while $l\in\{1,3,\dots,2L-1\}$0 yields $l\in\{1,3,\dots,2L-1\}$1, the identity element for OR. Consequently, if $l\in\{1,3,\dots,2L-1\}$2 and $l\in\{1,3,\dots,2L-1\}$3 are binary vectors, then

$l\in\{1,3,\dots,2L-1\}$4

$l\in\{1,3,\dots,2L-1\}$5

In that exact sense, MLLP is not merely inspired by CRS; it is a differentiable relaxation that recovers CRS at binary solutions.

A notable architectural feature is the absence of traditional additive biases or thresholds in the standard perceptron sense. Logical behavior is encoded through multiplicative conjunction/disjunction structure and selector-like weights $l\in\{1,3,\dots,2L-1\}$6. The architecture also does not introduce an explicit learnable NOT operator; negation is instead handled implicitly by discretization and one-hot encoding of the input features.

3. Forward computation and optimization procedure

The forward equations mirror the CRS hierarchy. For conjunction layers,

$l\in\{1,3,\dots,2L-1\}$7

and for disjunction layers,

$l\in\{1,3,\dots,2L-1\}$8

All layers are fully connected, with weight matrices $l\in\{1,3,\dots,2L-1\}$9 whose entries lie in $\mathcal{S}^{(l)}$0 (Wang et al., 2019).

The training pipeline begins with preprocessing. Continuous features are discretized using recursive minimal entropy partitioning, all discrete values are one-hot encoded, and labels are converted to one-hot vectors. This yields binary input vectors

$\mathcal{S}^{(l)}$1

and one-hot labels

$\mathcal{S}^{(l)}$2

The stated loss is

$\mathcal{S}^{(l)}$3

where $\mathcal{S}^{(l)}$4 is mean squared error and $\mathcal{S}^{(l)}$5 is $\mathcal{S}^{(l)}$6 regularization. The paper states that MSE is used to minimize the gap between the continuous output vector and the one-hot target in every dimension, which helps later discrete CRS extraction, while $\mathcal{S}^{(l)}$7 regularization encourages shorter rules.

Backpropagation is possible because the continuous logical units are differentiable. The derivatives with respect to directly connected weights are given explicitly. For conjunction neurons,

$\mathcal{S}^{(l)}$8

and for disjunction neurons,

$\mathcal{S}^{(l)}$9

Two training issues are emphasized. First, using sigmoid or $l\in\{2,4,\dots,2L\}$0 to enforce $l\in\{2,4,\dots,2L\}$1-valued weights aggravates vanishing gradients near 0 and 1, so the paper instead uses clipping,

$l\in\{2,4,\dots,2L\}$2

Second, the multiplicative logical structure itself produces gradient products in $l\in\{2,4,\dots,2L\}$3, which can also vanish. To mitigate that, weights are initialized as

$l\in\{2,4,\dots,2L\}$4

The stated rationale is that if $l\in\{2,4,\dots,2L\}$5 is close to 0, then both $l\in\{2,4,\dots,2L\}$6 and $l\in\{2,4,\dots,2L\}$7 are close to 1, keeping the multiplicative factors near 1 at initialization.

The paper’s default experimental setup is also concrete: $l\in\{2,4,\dots,2L\}$8 logical layers, middle-layer widths between 32 and 256 depending on data, batch size $l\in\{2,4,\dots,2L\}$9, training for $\mathcal{S}^{(0)}$0 epochs, initial learning rate $\mathcal{S}^{(0)}$1, decay by factor $\mathcal{S}^{(0)}$2 every $\mathcal{S}^{(0)}$3 epochs, weight decay $\mathcal{S}^{(0)}$4, and—with Random Binarization—the selected subset of weights is changed every epoch.

4. Random Binarization and discrete rule extraction

A central claim of the framework is that ordinary continuous training followed by simple post-hoc thresholding is insufficient. The paper reports that if MLLP is trained without special treatment and then binarized at the end, the extracted CRS behaves very differently and often performs poorly, especially when deep. Random Binarization (RB) is introduced to reduce that mismatch (Wang et al., 2019).

For each layer $\mathcal{S}^{(0)}$5, RB defines a mask matrix $\mathcal{S}^{(0)}$6 with entries

$\mathcal{S}^{(0)}$7

with

$\mathcal{S}^{(0)}$8

where $\mathcal{S}^{(0)}$9 is the rate of binarization. Each weight is therefore independently selected for binarization with probability $W^{(l)}$0. The temporary training weights are

$W^{(l)}$1

with

$W^{(l)}$2

and $W^{(l)}$3, usually $W^{(l)}$4.

During forward propagation, $W^{(l)}$5 is replaced by $W^{(l)}$6. To freeze selected weights, the paper sets

$W^{(l)}$7

By the chain rule,

$W^{(l)}$8

Hence, if $W^{(l)}$9, the weight is binarized and receives zero gradient; if $W_{i,j}^{(l)} \in \{0,1\},$0, it remains continuous and trainable.

After training, all weights are binarized,

$W_{i,j}^{(l)} \in \{0,1\},$1

and the resulting binary adjacency matrices define the extracted CRS. The paper also notes a second effect of RB: it acts similarly to Dropout, because weights binarized to 0 effectively remove connections, which helps reduce overfitting.

This separation between continuous optimization and discrete deployment is essential to the meaning of MLLP. The continuous network is not itself the final logical model; it is the mechanism used to obtain a discrete hierarchical rule system whose nodes are intended to be human-readable.

5. Interpretability, simplification, and empirical profile

The interpretability claim is structural rather than post-hoc. Each node in CRS corresponds either to a conjunction rule or to a disjunction of lower-level rules, and a user can inspect first-layer rules over binarized input features, higher-level rule sets that combine lower-level concepts, and final class-specific rule sets. The paper measures CRS complexity by number of edges rather than by naive total rule length, because edges better reflect shared structure across rules (Wang et al., 2019).

The extracted CRS can be simplified in two ways. First, dead nodes are removed. A node is dead if no path from top to bottom contains it or if it is never activated by any training instance. Second, redundant rules are eliminated through subset checks. The paper defines

$W_{i,j}^{(l)} \in \{0,1\},$2

For $W_{i,j}^{(l)} \in \{0,1\},$3, redundant weights are

$W_{i,j}^{(l)} \in \{0,1\},$4

The paper illustrates interpretability with a rule set from the Adult dataset for predicting income below \$50K/year, using factors such as lacking capital behavior, not being married, short work hours per week, and low educational background.

Empirically, the evaluation uses 12 UCI datasets—adult, bank-marketing, banknote, blogger, chess, connect-4, letRecog, magic04, mushroom, nursery, tic-tac-toe, and wine—and the primary metric is 5-fold cross-validated Macro F1. The reported average Macro F1 values over the 12 datasets are as follows.

Model	Average Macro F1
CRS	86.98
MLLP($W_{i,j}^{(l)} \in \{0,1\},$5)	84.40
CRS($W_{i,j}^{(l)} \in \{0,1\},$6)	47.31
C4.5	84.48
CART	84.44
RF(e=100)	86.82

These results are used to support two distinct claims. First, CRS achieves the best average performance among all methods, slightly above RF with 100 trees. Second, the contrast between MLLP($W_{i,j}^{(l)} \in \{0,1\},$7) = 84.40 and CRS($W_{i,j}^{(l)} \in \{0,1\},$8) = 47.31 is presented as evidence that RB is crucial: without RB, post-hoc binarization breaks the discrete model badly.

The paper also studies the effect of the binarization rate $W_{i,j}^{(l)} \in \{0,1\},$9 on Connect-4. When $W_{i,j}^{(l)}=1$0, deep CRS performs poorly; with suitable $W_{i,j}^{(l)}=1$1, roughly $W_{i,j}^{(l)}=1$2 to $W_{i,j}^{(l)}=1$3, deep CRS improves substantially and can outperform shallow CRS; if $W_{i,j}^{(l)}=1$4 is too small, consistency with the discrete CRS is insufficient; if $W_{i,j}^{(l)}=1$5 is too close to $W_{i,j}^{(l)}=1$6, too many weights are fixed and training becomes difficult. After simplification, CRS complexity is reported to be close to decision trees, while Random Forest has much larger complexity due to many trees.

6. Conceptual neighbors and scope of the term

The 2019 MLLP formulation sits within a broader line of work that interprets multilayer networks as compositional logical or geometric systems, but those neighboring formulations are not identical to MLLP. One nearby result is Raúl Rojas’s “long chain of perceptrons,” described as a multilayer perceptron with many hidden layers of width one and with the original input vector passed from layer to layer as a shortcut. In that construction, later threshold units receive both $W_{i,j}^{(l)}=1$7 and the previous binary output with a large positive constant $W_{i,j}^{(l)}=1$8, producing a serial logical program over the input with one persistent binary flag. The paper’s universality claim is for classification of two classes in a bounded input domain with arbitrary finite data configurations, and the abstract explicitly warns that the classification procedure is “not necessarily computationally efficient” (Rojas, 2017). This suggests a deep threshold-logic antecedent to MLLP-style thinking, but it differs in using hard perceptrons, explicit input shortcuts, and an existential representational theorem rather than a trainable continuous relaxation.

A second related construction is the interpretable feedforward MLP derived from two-class LDA. There the network is built analytically in three stages: half-space partitionings accomplished by multiple parallel LDAs, subspace isolation where one Gaussian modality is represented by one neuron, and class-wise subspace mergence where each Gaussian modality is connected to its target class. Stage 2 uses weights in $W_{i,j}^{(l)}=1$9, with $L$00 in experiments, and Stage 3 uses weights in $L$01. The resulting semantics are strongly MLLP-like—half-space predicates, conjunctive region isolation, and disjunctive class merging—but the units are ReLU-based and the construction depends on Gaussian or GMM modeling rather than on a CRS relaxation with RB (Lin et al., 2020).

A third conceptual neighbor is “MLP algebra,” which treats trained MLPs as approximate characteristic functions of sets and introduces operators with explicit logical implications: complement, sum net, I-product net, component net, and O-product net. In that framework, complement behaves like NOT, sum net like OR or union, and I-product net like AND. It is therefore relevant to an MLLP interpretation as a compositional predicate calculus, but it does not define MLLP and does not provide a differentiable rule-learning procedure of the CRS type (Peng, 2017).

A common misconception is to treat all of these architectures as interchangeable instances of “logical perceptrons.” The literature summarized here indicates a narrower usage. In the strict sense established by the 2019 paper, MLLP denotes a continuous neural architecture designed specifically to learn hierarchical logical rule models by approximating conjunction and disjunction with differentiable operators and by using Random Binarization to make subsequent discrete extraction viable. The broader family of related work shows that deep narrow threshold chains, LDA-derived region networks, and algebraic compositions of trained MLPs can all be interpreted in logical terms, but they embody different assumptions about activations, training, and the meaning of internal representations.