Stepwise Forward Reason (R())

• Stepwise Forward Reason (R()) is a family of residual-oriented algorithms that enable iterative feature selection and differentiable inference in regression and neuro-symbolic logic.
• It utilizes a greedy, iterative routine in regression, updating residuals using the Moore–Penrose inverse to minimize prediction error effectively.
• In neuro-symbolic reasoning, R() integrates differentiable clause composition and soft logical operations, facilitating end-to-end gradient-based optimization.

Stepwise Forward Reason ($R()$) encompasses a family of residual-oriented algorithms and differentiable reasoning operators used for feature selection in statistical modeling and neuro-symbolic logic inference. In regression, $R()$ formalizes the greedy addition of variables to minimize prediction error, while in differentiable reasoning, $R()$ expresses one-step forward inference under weighted clause composition. The function is central in stepwise regression, Relevance Matching Pursuit (RMP), and Neuro-Symbolic Forward Reasoning (NSFR), and exhibits fundamental connections to best subset selection, the lasso, and differentiable logic programming.

1. Mathematical Definition and Selection Rule

In the classical regression context, $R()$ represents the residual norm after inclusion of a candidate variable. With $X\in\mathbb{R}^{n\times m}$ the feature matrix, $y\in\mathbb{R}^n$ the response, and active set $A$ indexing the selected features, the residual is

$r_A = y - X_A (X_A^+ y)$

where $X_A^+$ is the Moore–Penrose inverse. For candidate $j\notin A$, the residual-based scoring function is

$R(A,j) = \| r_{A\cup\{j\}} \|_2,$

and the variable selected is

$$j^* = \arg\max_{j\notin A}\; \frac{|a_j^\top r_A|}{\| a_j \|_{R_A}},$$

with $R_A = I - X_A X_A^+$ and $\| a_j \|_{R_A}^2 = a_j^\top R_A a_j$ (Ament et al., 2021). This selection fundamentally seeks maximal decrease in squared residual.

For neuro-symbolic reasoning, $R()$ is defined as

$$R : \mathbb{R}^{B\times G} \times \mathbb{N}^{C\times G\times S\times L} \times \mathbb{R}^{M\times C} \to \mathbb{R}^{B\times G}$$

where $B$ is batch size, $G$ is the number of ground atoms, $C$ clauses, $S$ substitutions, $L$ clause body-length, and $M$ the number of soft programs. $R()$ orchestrates clause instantiation, body aggregation, weighted clause composition, and soft logical OR, producing a new batch of truth-valued facts (Shindo et al., 2021).

2. Algorithmic Sequence and Implementation

The regression version of stepwise $R()$ is instantiated via a greedy, iterative routine:

A = [] r = y for t in range(k): scores = [] for j in complement(A): score = (a_j.T @ r)**2 / (a_j.T @ a_j) scores.append(score) jstar = argmax(scores) A.append(jstar) r = y - X_A @ (X_A^+ @ y)

(Ament et al., 2021). This algorithm selects $k$ features sequentially, updating the residual at each step.

For NSFR, $R()$ integrates perceptual neural networks, tensorized object constants, neural predicate valuation, and batched differentiable reasoning:

Z = f_percept(X; Phi) V = zeros([B, G]) for j in 1..G: T_list = [to_tensor(t_i; Z) for i in 1..n] V[:,j] = v_p(T_list; Theta) for t in 0..T-1: for i in 1..C: C_i = clause_function(V, I[i,:,:,:], gamma) H = sum_over_C(Wstar * C) V_new = softor(H, dim=0, gamma) V = softor_stack([V, V_new], dim=1, gamma) y_hat = f_predict(V)

(Shindo et al., 2021). Every primitive—perception, predicate, clause, composition—is differentiable, supporting gradient-based optimization.

3. Connections Among Regression, Neuro-symbolic, and Bayesian Learning

Relevance Matching Pursuit (RMP) introduces a coordinate-ascent algorithm for Sparse Bayesian Learning. As the noise variance $\sigma\rightarrow0$, RMP’s atom addition criterion

$$\arg\max_{i\notin A}|\,\tilde{e}_i^\top r_{A,\sigma}|$$

where $\tilde{e}_i = a_i/\|a_i\|_{R_{A,\sigma}}$, converges to the stepwise-forward regression rule (Ament et al., 2021). This ties Bayesian sparsification procedures to residual-based variable selection.

Similarly, NSFR operationalizes $R()$ as a forward-chaining inference routine within neural logic programming. It constructs new probabilistic facts from antecedent valuations using weighted clause-program composition and iterative soft logical OR, forming a differentiable reasoning pipeline (Shindo et al., 2021).

4. Exact-recovery Guarantees and Statistical Properties

Stepwise $R()$ does not guarantee globally optimal subset selection via forward moves alone. However, backward elimination coupled with a computable bound produces optimal recovery:

Theorem ((Ament et al., 2021), Theorem 4.2): Let $X_A$ have full column rank and $S\subseteq\{1,...,m\}$ be the true support. In $y = X x + \epsilon$ with $\text{supp}(x)=S$, if $$> \|\epsilon\|_2 < \frac{\sigma_{\min}(X)}{\sqrt{2[2-\sigma_{\min}(X)^2]}} \min_{i\in S} |x_i| >$$ then backward elimination from $A=\{1,...,m\}$ exactly recovers the support $S$.

This result depends exclusively on the smallest singular value $\sigma_{\min}$ of the design matrix, enabling polynomial-time computation of the bound. A plausible implication is that backward-augmented stepwise regression can be certifiably optimal under strong signal conditions.

5. Computational Complexity and Efficient Implementation

For regression, each forward step requires $O(n \cdot (m - |A|))$ computations to score and update; the full $k$-step selection runs in $O(k n m)$. Cholesky or QR factorization of $X_A$ accelerates scoring and residual updates to $O(|A| n)$ per step, completing $k$ steps in $O(k^2 n)$. Updating “energy-norm” quantities in RMP adds $O(m|A|^2)$ overhead, often negligible (Ament et al., 2021).

Empirical timings confirm efficiency: on $n=100, p=10$, FS takes $0.006$ s; for $n=500, p=100$, up to $k=50$, $0.82$ s; for $n=50, p=1000$, $0.14$ s—contrasted with minute-level runtimes for best subset selection via mixed integer optimization (Hastie et al., 2017).

6. Comparative Performance and Practical Applications

Forward stepwise selection using $R()$ matches best subset selection in medium/high SNR scenarios and outperforms the lasso in these regimes, but tends to overfit under low SNR where the lasso's shrinkage yields lower generalization error. The relaxed lasso, which refits least squares on lasso-selected supports, interpolates between the benefits of forward and shrinkage selection (Hastie et al., 2017). FS is algorithmically tractable and interpretable for large feature sets and is often preferred over computationally intensive combinatorial subset search.

In neuro-symbolic domains, $R()$ propagates differentiable logical entailments and is foundational in high-fidelity, object-centric reasoning pipelines such as those validated on CLEVR-Hans and Kandinsky datasets (Shindo et al., 2021).

7. Loss Functions and Optimization in Differentiable Reasoning

Stepwise $R()$ for logic inference enables end-to-end supervised learning:

$$\mathcal{L}_\mathrm{class}(\mathbf{y}, \hat{\mathbf{y}}) = -\sum_{b=1}^B [ y_b \ln \hat{y}_b + (1-y_b) \ln (1-\hat{y}_b) ]$$

where $\hat{y}_b = f_\mathrm{predict}(V^{(T)}_b)$ (Shindo et al., 2021). Gradients flow from the classification loss to perception, predicate parameters, and clause weights via the differentiable $R()$ pipeline. Additional concept-specific losses and rule-weight learning objectives refine the logical and probabilistic inference.

---

In summary, Stepwise Forward Reason ($R()$) encodes the incremental, residual-driven mechanics underlying greedy feature selection, Bayesian sparsification, and differentiable logic inference. It admits rigorous mathematical formulation, efficient algorithmic implementation, and provable recovery guarantees, and bridges the statistical, computational, and neuro-symbolic domains with interpretable operators and empirically validated efficacy (Ament et al., 2021, Shindo et al., 2021, Hastie et al., 2017).