Predicate Networks: Relational Models

• Predicate networks are machine learning models that represent relational and invariant properties using structured predicates rather than simple feature-level variables.
• Neural predicate networks employ layered memory banks and oscillatory binding via Hebbian learning, while probabilistic predicate networks generalize Bayesian methods to capture cyclic causal dependencies.
• Applications span natural language processing and theory reconstruction, with models achieving high transfer accuracy and robust compositional inference.

Predicate networks are a family of machine learning and probabilistic models in which the fundamental units of representation and inference are predicates—structures that capture relational or invariant properties—rather than solely propositional or feature-level variables. This entry surveys three principal classes of predicate networks: neural predicate networks that discover and operate over latent generative structures via distributed assemblies and oscillatory dynamics; probabilistic predicate networks that generalize Bayesian networks by representing knowledge as a static graph of predicates and handle cyclic causal dependencies; and theory-reconstruction-based predicate networks that formulate predicate invention as an autoencoder over relational data. Additionally, a specific neural class—predicate networks for predicate-argument structure analysis—implements pooling and attention over multi-predicate interactions in sequence models.

1. Neural Predicate Networks: Architecture and Learning Principles

Neural predicate networks, as exemplified by Martin & Doumas (Martin et al., 2018), are structured around the following architectural motifs:

• Layered Banks: The network contains three main banks—Focal Bank (FB) for current input/attentional state, Active-Memory Bank (AMB) as a working buffer, and Long-Term-Memory Bank (LTMB) for persistent storage of learned predicates and role representations.
• Feature Pool: All representations are distributed over a shared pool of $F$ real-valued feature units. Assemblies in the memory banks project to and receive feedback from this pool.
• Predicate and Role-Binding Assemblies: Predicate assemblies encode invariant properties as distributed vectors. Role-binding assemblies ('RBs') bind predicates to arguments through phase-locked or phase-lagged oscillatory activity without conflating their content.
• Connectivity: Projections to and from the feature pool are predominantly excitatory and weighted; competition and sparsity are enforced within banks via lateral inhibition and “yoked” inhibitory interneurons, which also mediate synchronous or desynchronous activation for compositional binding.

Learning Protocol: Learning is an unsupervised two-stage process:

1. Intersection Discovery: Given two input activations $a, b \in \{0,1\}^F$, their intersection $r = a \land b$ captures invariant features.

1. Hebbian Predicate Learning: A new LTMB predicate node $P$ is recruited and its weights are updated according to

$\Delta w_k = \eta \cdot r \odot (1 - w_k)$

or variations thereof. This consolidates the discovered invariant as a reusable predicate assembly.

Bank Mapping: Synaptic mappings between FB and AMB are formed Hebbian-style to allow flexible reactivation of predicates in varying input contexts:

$$\Delta M_{ij} = \eta_{map} \, x_i^{(\text{FB})} \, y_j^{(\text{AMB})}$$

This enables the network to represent and retrieve structured knowledge via dynamic assemblies, rather than static feature conjunctions (Martin et al., 2018).

2. Rhythmic Predicate Binding and Generative Compositionality

Predicate networks achieve compositional generalization through oscillatory binding mechanisms:

• Phase-Lag-1 Binding: Predicate and argument assemblies are activated sequentially in distinct sub-phases within an oscillatory cycle, maintaining their separation while enabling relational binding. For predicate $P$ and argument $A$, firing likelihoods are given by

$$u_P(t) = \sin(\omega t + \phi_0), \quad u_A(t) = \sin(\omega t + \phi_0 + \Delta\phi)$$

with $\Delta\phi \approx \pi/2$.

• Phase-Lag-0 Binding: To encode an entire proposition as a unit (e.g., $(P, A_1, A_2, ...)$), all constituent assemblies fire synchronously, allowing recursive packaging.
• Generative Composition: Compositional inference or action is achieved by co-activating predicate $P$, argument $A$, and their associated role-binding assembly $RB$ in the correct phase relation, formally

$z = w_P + w_{RB}(P \rightarrow A) + w_A$

enabling the network to instantiate new propositions from previously learned components (Martin et al., 2018).

3. Probabilistic Predicate Networks

Probabilistic predicate networks, introduced by Lin (Lin, 2013), extend Bayesian networks by constructing a single static graph whose nodes are unary predicate templates and whose edges represent type inheritance or causal/structural features, with cycles permitted at the schema level.

• Formulation:
  • Events $E$ provide a domain for unary predicates $C = \{c_1, ... c_N\}$.
  • Features $F$ encode causal or structural functions $f: c_i \rightarrow c_j$.
  • Priors $\pi_i$ and conditionals $\theta_f$ parameterize occurrence and feature instantiation probabilities.
• Joint Distribution: Defined through the chain rule over all ground atoms:

$$P(\{c_k(e)\}) = \prod_{e \in E} \prod_{i=1}^N P(c_i(e) \mid \text{Pa}(c_i(e)))$$

• Handling Cycles and Recursion:
  • Type-level cycles (e.g., contagious-disease spread) are valid as long as each ground instantiation terminates.
  • The approach supports recursive processes (e.g., plan recognition, list processing) without dynamic network re-assembly.
• Inference and Learning:
  • Abductive explanations are generated by assembling 'scenarios'—grounded causal trees scored by product of local probabilities.
  • Learning parameters involves frequency-based estimation or EM for hidden structure.

Illustrative Applications:

• Complex diagnoses, plan recognition, and recursive causal explanation are handled within the same static predicate template graph (Lin, 2013).

4. Predicate Networks in Predicate-Argument Structure Analysis

In PAS analysis, particularly for Japanese, predicate networks denote deep bidirectional RNNs with explicit pooling and attention mechanisms capturing cross-predicate interactions (Matsubayashi et al., 2018):

• Model Structure:
  • Each sentence is processed by a stacked bi-RNN; for each predicate-word pair, a vector $h^0_{i,t}$ encodes embeddings and predicate-position indicators.
  • Multi-predicate (MP) input flags further encode all predicate positions, increasing representational power.
• Interaction Layers:
  • Pooling: Max-pooling combines hidden states $h^K_{i,t}$ across predicates.
  • Cross-Predicate Attention: Learns weighted relations over distant predicate-argument pairs, insensitive to token distances.
  • Self-Attention: Captures intra-predicate dependencies, optionally stacked after pooling for enhanced expressivity.
• Loss and Training:
  • Negative log-likelihood over gold argument labels, dropouts, and Adam optimizer.
• Empirical Results:
  • MP+Pool-SelfAtt model achieves a single-model F$_1$ of 83.94 on the NTC 1.5 test set, outperforming pooled or self-attention models alone. Zero (indirect) F$_1$ increases by 1.18, indicating strong gains in long-range dependency resolution.

Pooling and attention-based predicate networks successfully capture “distance-free” dependencies, facilitating argument-sharing and long-range role assignment in language tasks (Matsubayashi et al., 2018).

5. Predicate Networks via Theory Reconstruction and Predicate Invention

Predicate invention can be formalized as a theory reconstruction problem using an autoencoder-like network over relational data (Dumancic et al., 2016):

• Framework:
  • Input is a set of formulas $T$ (from language $\mathcal{L}$) true in a knowledge base $\mathcal{KB}$.
  • Invented predicates $Q$ (not in $P$) are assigned definitions $\mathcal{F}$, and a new hidden theory $T_Q$ is induced.
  • Encoder $\mathcal{E}$ computes activations in hidden space by applying $\mathcal{F}$ to $T$.
  • Decoder $\mathcal{D}$ reconstructs $T$ from $T_Q$.
  • Objective:

  $$\min_{Q, \mathcal{F}} \left\| T \;\triangle\; \mathcal{D} (\mathcal{E}(T)) \right\|_\Delta - q(Q, \mathcal{F}, \mathcal{KB})$$

  where $q(\cdot)$ regularizes for sparsity or brevity.

• Network Parallels:

  • Input: one unit per formula in $\mathcal{L}$.
  • Hidden: one unit per instantiation of each hidden predicate $h \in Q$.
  • Connections are unweighted; structure learning seeks minimal definitions capturing the theory's content.
• Significance:
  • Provides a unified formalism for both predicate invention and theory revision as autoencoding problems.
  • Opens avenues for multi-layer (stacked) predicate discovery and integration with neural and probabilistic models (Dumancic et al., 2016).

6. Compositional Generalization, Inference, and Empirical Metrics

Predicate network models are characterized by strong compositionality and the ability to extrapolate structure across novel domains:

• Neural Predicate Networks: Achieve >90% one-shot transfer accuracy from raw pixels across transformed Atari game environments, without supervised labels, due to abstract predicate representation (Martin et al., 2018).
• Probabilistic Predicate Networks: Efficiently explain indirect observations via abductive scenario assembly, scoring explanations by product of priors and local conditionals (Lin, 2013).
• Evaluation Metrics:
  • Compositionality is assessed by test accuracy of composed propositions $(P, A_1, A_2, ...)$.
  • Predicate vocabulary size and convergence rates empirically support a small set of reusable predicates sufficing for generalization:

  $$\text{accuracy(test)} \approx 1 - \exp(-\alpha \cdot N_\text{predicates})$$

• PAS Predicate Networks: Enhanced $F_1$ on indirect-argument (Zero) prediction reflects increased capacity for multi-predicate generalization through pooling/attention (Matsubayashi et al., 2018).

7. Comparative Summary and Outlook

Predicate networks constitute a central paradigm for bridging symbolic reasoning and neural or probabilistic modeling. Their core attributes are summarized in the table below:

Approach	Representation	Learning/Inference
Neural predicate networks	Assemblies in feature pool; oscillatory binding	Hebbian intersection; phase-based composition (Martin et al., 2018)
Probabilistic predicate networks	Static schema of unary predicates with causal features	Abductive scenario construction; frequency/EM parameter learning (Lin, 2013)
Theory reconstruction networks	Binary units for formulas and invented predicates	Autoencoder minimization of theory reconstruction loss (Dumancic et al., 2016)
PAS predicate networks	Bi-RNNs with cross-predicate pooling and attention	End-to-end supervised learning for multi-predicate structure (Matsubayashi et al., 2018)

Predicate networks overcome the structural limits of propositional models, enable relational abstraction and one-shot generalization, and realize soft unification of logic-based representations with numerical optimization. Ongoing extensions include incorporation of continuous phase-coding, differentiable definition learning, and scaling to complex, uncertain domains.