---
title: Sign-based Online Learning (SOL)
url: https://www.emergentmind.com/topics/sign-based-online-learning-sol
type: topic
---

# Sign-based Online Learning (SOL)

Sign-based Online Learning (SOL) denotes a family of online-learning formulations in which sign operations are structurally central. In the cited arXiv literature, the term appears in two distinct but related senses. In “Online Learning of Neural Networks” [2505.09167], SOL concerns the adversarial online learnability of feedforward neural networks with sign activation, and the central objects are margin conditions and the \((d,\gamma)\)-totally-separable-packing number. In “An Online Learning Algorithm for Neuromorphic Hardware Implementation” [1505.02495], SOL denotes a hardware-oriented sign-based online update rule, also called Sign-based Online Update Learning (SOUL), for Trainable Analogue Block (TAB) and related Extreme Learning Machine (ELM) systems. These usages connect online mistake-bound theory, margin geometry, sign-sign stochastic updates, and neuromorphic realization.

## 1. Adversarial online model for sign-activation networks

Let \(X=B(\mathbb{R}^d)\) be the unit ball in \(\mathbb{R}^d\), and let \(Y=\{1,\ldots,Y\}\) be a finite label-set. The online game is infinite-horizon, adversarial, and realizable: on each round \(t\), the adversary picks \(x_t\in X\) and sends it to the learner; the deterministic learner predicts \(\hat y_t\in Y\); the adversary reveals the true label \(y_t=\Phi^*(x_t)\), where \(\Phi^*:X\to Y\) is an unknown target network in the class; and the learner suffers loss \(1[\hat y_t\neq y_t]\). The total number of mistakes on a sequence \(S=(x_1,\ldots,x_T)\) is
\[
M(Lrn,S)=\sum_{t=1}^T 1[Lrn\text{'s prediction}\neq \Phi^*(x_t)].
\]

The concept-class \(H(d,L,\ldots)\) consists of all feedforward networks of depth \(L+1\) with sign-activation \(\sigma(z)=\operatorname{sign}(z)\) applied coordinate-wise:
\[
x^{(\ell+1)}=\operatorname{sign}(W^{(\ell)}x^{(\ell)}+b^{(\ell)}),\qquad \ell=0,\ldots,L,
\]
with output \(x^{(L+1)}\in\{\pm 1\}^{d_{L+1}}\), identified with \(Y\). All weight-rows \(w\) satisfy \(\|w\|_2=1\), and biases are allowed [2505.09167].

This formulation places SOL squarely in the classical realizable mistake-bound tradition, but for a highly nonconvex hypothesis class. The sign activation is not an implementation detail: it determines the geometric partition induced by the network and underlies the combinatorial characterization of learnability.

## 2. First-layer margin and totally-separable packing

For the \(i\)-th neuron \(W^{(0,i)}\) in the first hidden layer, the sample-wise margin on an input-sequence \(S\) is
\[
\gamma_{W^{(0,i)}}(S)=\min_{x\in S} |\langle W^{(0,i)},x\rangle|.
\]
The minimal first-layer margin is
\[
\gamma_1(N^*,S)\;=\;\min_{1\le i\le d_1}\;\min_{x\in S}\;|\langle W^{(0,i)},\,x\rangle|\;>\;0.
\]
Intuitively, \(\gamma_1>0\) says that every first-layer hyperplane classifies every \(x_t\) with margin at least \(\gamma_1\). This condition is sufficient for online learnability, and in some cases necessary: without such a margin, even a 1-neuron net can force infinitely many mistakes.

The combinatorial parameter controlling the analysis is the \((d,\varepsilon)\)-totally-separable-packing number. A set \(\{x_1,\ldots,x_T\}\subset B(\mathbb{R}^d)\) is a \((d,\varepsilon)\)-totally-separable packing if for every \(i\neq j\) there exists a unit-vector \(w\) and bias \(b\) such that
\[
\operatorname{sign}(\langle w,x_i\rangle+b)=+1,\qquad \operatorname{sign}(\langle w,x_j\rangle+b)=-1,
\]
and
\[
|\langle w,x_k\rangle+b|\ge \varepsilon,\qquad k=1,\ldots,T.
\]
The packing number is
\[
TS(d,\varepsilon)=\max\{T:\exists\text{ a $(d,\varepsilon)$-TS-packing of size }T\}.
\]

The paper proves that \(TS(d,\varepsilon)\le (1.5/\varepsilon)^d\), matching the usual \(\ell_2\)-packing scale up to the stronger total-separation requirement. For \(\varepsilon\le 1/2\), the exact bounds proved in Theorem 3.11 are
\[
\max\Big\{\big(2\lfloor1/(2\varepsilon\sqrt d)\rfloor\big)^d,\;d\Big\}\;\le\;TS(d,\varepsilon)\;\le\;(1.5/\varepsilon)^d.
\]
The significance of \(TS(d,\varepsilon)\) is that it replaces generic capacity measures by a margin-sensitive geometric quantity tailored to sign networks: each pairwise separator must stay at least \(\varepsilon\)-away from all other points, not merely separate one pair [2505.09167].

## 3. Characterization of mistake bounds

The main upper bound states that there exists an online learner \(Lrn\) such that for every target net \(N^*\) of input-dimension \(d\), and every realizable sequence \(S\) with first-layer margin \(\gamma_1\),
\[
M(Lrn,S)\;=\;\widetilde O\!\Big(\frac{TS(d,\gamma_1)}{\gamma_1^2}\Big).
\]
The proof is organized around a partition-and-label decomposition. Any sign-net defines a partition of \(B(\mathbb{R}^d)\) into \(2^{d_1}\) regions by the first-layer neurons. Only \(T\) regions contain the \(T\) examples, and one can prove \(T\le TS(d,\gamma_1)\). The learner runs a multiclass Weighted-Majority over an expert-class \(E\), where each expert is specified by a small set \(G\) of at most \(TS(d,\gamma_1)\) candidate first-layer separators together with a region-labeling \(L_G\). Some expert uses precisely the true separators from \(N^*\), so it makes at most \(TS(d,\gamma_1)/\gamma_1^2\) mistakes in identifying regions, by the Perceptron bound, and at most \(TS(d,\gamma_1)\) mistakes in labeling them once. Weighted-Majority then incurs only
\[
O((\text{mistakes of best expert})+(\log|E|))\le \widetilde O\!\Big(\frac{TS(d,\gamma_1)}{\gamma_1^2}\Big).
\]

The lower bound is matching at the level of the controlling parameters. For any deterministic learner \(Lrn\), for any \(\varepsilon>0\) and \(d\ge 1/\varepsilon^2\), there exists a network \(N^*\) of dimension \(d\) and a realizable sequence \(S\) with \(\gamma_1\ge \varepsilon\) such that
\[
M(Lrn,S)\;=\;\Omega\bigl(TS(d,\varepsilon)\;+\;1/\varepsilon^2\bigr).
\]
The proof takes \(S\) to be a maximal \((d,\varepsilon)\)-TS-packing of size \(TS(d,\varepsilon)\), then constructs a two-hidden-layer net whose first hidden layer stores all \(TS\) hyperplanes with margin at least \(\varepsilon\), whose second hidden layer has one neuron for each region-vector \(r\in\{\pm 1\}^{TS}\), and whose output neuron takes the sign of the sum. This realizes any labeling of \(S\), so an adversary can force the learner to err on all \(T\) points, plus the \(1/\varepsilon^2\) term from the standard Perceptron lower bound.

Combining the upper and lower bounds, the paper fully characterizes the optimal online mistake-bound in terms of \(TS(d,\gamma_1)\). A common misconception is that a positive first-layer margin should by itself yield mild online complexity; the lower-bound construction shows that this is false in high dimension [2505.09167].

## 4. Exponential dependence on ambient dimension

The dimension dependence is not a proof artifact. By Theorem 3.11, for \(\varepsilon\ge 1/2\),
\[
TS(d,\varepsilon)\;\ge\;\max\bigl\{(2\lfloor1/(2\varepsilon\sqrt d)\rfloor)^d,\;d\bigr\}\;\ge\;\max\{(1/(\varepsilon\sqrt d))^d,\;d\}.
\]
Thus even when \(\varepsilon=O(1)\), \(TS(d,\varepsilon)\) can grow as \(\Omega(2^d)\), forcing an exponential mistake-bound in \(d\) and ruling out any dimension-free bound under only the first-layer margin assumption [2505.09167].

This result is structurally important. It shows that the obstacle is neither a weakness of Weighted-Majority nor an artifact of a particular proof technique. The obstruction is geometric: the class of realizable sign-network labelings remains large enough, even under a first-layer margin assumption, that an adversary can encode exponentially many unavoidable distinctions in ambient dimension \(d\).

## 5. Dimension-free regimes

Two additional restrictions recover dimension-free guarantees. The first is the multi-index model, where the target net depends only on \(k\ll d\) orthonormal directions \(s^{(1)},\ldots,s^{(k)}\):
\[
\Phi^*(x)=\phi^*\bigl(\langle s^{(1)},x\rangle,\dots,\langle s^{(k)},x\rangle\bigr).
\]
In this setting there is a learner with
\[
M(Lrn,S)\;=\;\widetilde O\!\Big(\frac{TS(k,\gamma_1)}{\gamma_1^2}\Big)\;\le\;\widetilde O\!\Big(\frac{(1.5/\gamma_1)^k}{\gamma_1^2}\Big).
\]
If \(k=O(1)\), this is polynomial in \(1/\gamma_1\) and independent of \(d\). The key proof idea is that any packing of \(S\) in \(\mathbb{R}^d\) induces a packing in the \(k\)-dimensional signal-space of size at most \(TS(k,\gamma_1)\), so the meta-learner argument runs with \(d\to k\).

The second restriction is the extended margin assumption. Here every neuron in every layer has margin at least \(\gamma\) on the induced layerwise representations over the sequence \(S\): for all layers \(\ell\) and every neuron \(W^{(\ell,i)}\),
\[
\min_{x\in S} |\langle W^{(\ell,i)},x^{(\ell)}\rangle|\ge \gamma.
\]
Under this condition, for a target net of depth \(L\),
\[
M(Lrn,S)\;=\;\widetilde O\!\Bigl(\frac{\log|Y|}{\,\gamma^{4L+2}}\Bigr).
\]

The proof proceeds by pruning via uniform convergence. For a single binary-output net with one hidden layer, the class \(H=\{h_x(w)=\operatorname{sign}(\langle x,w\rangle): w \text{ runs over hidden-layer weights}\}\) satisfies \(VC(H)\le 1/\gamma^2\), by the Perceptron implication under margin \(\gamma\). A random sample \(G\) of size \(\widetilde O(1/\gamma^4)\) of the hidden neurons uniformly approximates the empirical means \(Q_D(h_x)\), and choosing \(D\) proportional to the output-weights \(o_i\) yields a pruned net with identical outputs on \(S\). Repeating this pruning backwards layer by layer contributes a factor \(\widetilde O(1/\gamma^4)\) per layer, so the first-layer selection has size \(g=\widetilde O(1/\gamma^{4L})\). The final learner again runs Weighted-Majority over the pruned sets, giving \(\widetilde O((\log|Y|)/\gamma^{4L+2})\) total mistakes [2505.09167].

These two regimes isolate different ways to defeat the ambient-dimension barrier. The multi-index model reduces the effective input dimension, whereas extended margin reduces the number of hidden units that must be retained to preserve the realized function on the observed sequence.

## 6. SOUL: sign-sign online updates for ELM and TAB

In the hardware-oriented literature, SOL appears as Sign-based Online Update Learning (SOUL) in a standard three-layer feed-forward ELM. For each training example \((x_n,y_n)\), the hidden-layer activation vector is
\[
h_n \;=\; g\bigl(W^{(1)}x_n + b^{(1)} + o^{(1)}\bigr)\;\in\;\mathbb{R}^L,
\]
where \(W^{(1)}\) and \(b^{(1)}\) are the random input weights and biases, \(o^{(1)}\) is a fixed systematic offset per hidden neuron, and \(g(\cdot)\) is, for example, a tanh nonlinearity. The output prediction is
\[
\hat y_n \;=\; (W^{(2)}_{n-1})^T\,h_n\;\in\;\mathbb{R}^K,
\]
and the instantaneous error is
\[
e_n \;=\; y_n\;-\;\hat y_n\;\in\;\mathbb{R}^K.
\]

The derivation begins from the online pseudo-inverse (OPIUM) update
\[
w_n^{(2)} = w_{n-1}^{(2)} + e_n\,\phi_n,
\]
with
\[
\phi_n = \frac{O_{n-1} \,h_n}{1 + h_n^T\,O_{n-1}\,h_n},\qquad
O_n = O_{n-1} - O_{n-1}\,h_n\,\phi_n^T.
\]
To make this practical in hardware, the approximation
\[
O_{n-1}\approx \alpha\,I,\qquad \alpha>0
\]
is imposed, so that
\[
\phi_n\approx\frac{h_n}{N}
\]
with a constant normalization \(N\). The update reduces to a form of Normalised LMS,
\[
w_n^{(2)} = w_{n-1}^{(2)} + \frac{e_n\,h_n}{N},
\]
and SOUL then replaces the real-valued product \(e_nh_n\) by the product of their signs:
\[
\boxed{
w_n^{(2)} \;=\; w_{n-1}^{(2)}
\;+\;\frac{\operatorname{sign}(e_n)\,\odot\;\operatorname{sign}(h_n)}{N}.
}
\]
The sign operators are defined componentwise:
\[
\operatorname{sign}(e_n)_i =
\begin{cases}
+1,&(e_n)_i\ge0,\\
-1,&(e_n)_i<0,
\end{cases}
\qquad
\operatorname{sign}(h_n)_j =
\begin{cases}
+1,&(h_n)_j\ge0,\\
-1,&(h_n)_j<0.
\end{cases}
\]

SOUL may be viewed as stochastic gradient-descent on the regularised least-squares cost
\[
J(w) \;=\;\tfrac12\,(y_n - w^T h_n)^2 \;+\;\tfrac{\epsilon}{2}\,\|w\|^2,
\]
which is strictly convex in \(w\). The LMS update follows \(-\nabla_w J\), and replacing \(\mu e_n h_n\) by \(\tfrac{1}{N}\operatorname{sign}(e_n)\operatorname{sign}(h_n)\) yields the SOUL rule. A formal convergence proof is not given, but by analogy with the classical sign-sign LMS algorithm one can show, under bounded inputs \(\|h_n\|\) and small step size \(1/N\), that the weights converge in mean-square to a neighborhood of the optimum.

The bit-level implementation is central. The sign of \(h_n\) is derived from the hidden neuron’s activation, such as the MSB of a fixed-point representation; the sign of \(e_n\) is derived from the difference \(y_{n,i}-\hat y_{n,i}\); and the element-wise product \(\operatorname{sign}(e_n)\odot\operatorname{sign}(h_n)\) is implemented with a single XOR per weight. In this logic, XOR \(=0\) means the signs agree and the corresponding counter increments, whereas XOR \(=1\) means the signs differ and the counter decrements [1505.02495].

## 7. Neuromorphic realization and empirical behavior

TAB implements the ELM in mixed-signal CMOS (65 nm models). Input-to-hidden computation uses a differential pair plus systematic offset, realizing a \(\tanh(\cdot)\) tuning curve; the offsets \(o_j\) break symmetry so that each neuron has a distinct receptive field. Hidden-to-output realization uses, for each hidden-to-output weight, a 13-stage binary current splitter whose branches are gated by the bits of the digital weight word, a 1-bit XOR gate that combines \(\operatorname{sign}(h)\) and \(\operatorname{sign}(e)\), an \(n\)-bit up/down counter storing the magnitude of the weight, and a 3-bit “add\_no” register to set the increment size \(2^{\text{add\_no}}\). The hidden-layer analog currents are summed through the weighted splitter to generate the output current \(\hat y\) [1505.02495].

The reported regression experiments use Python simulation with 100 hidden neurons and 13 bit weights on three single-input-to-single-output functions over \(x\in[-1,1]\), sampled at 200 points. For \(y=\sin(x)\), RMS error falls below \(1\) nA in approximately \(2000\) iterations. For \(y=\mathrm{sinc}(67x)\), the system converges to RMS \(\approx 28.6\) nA in approximately \(10\,000\) iterations. For \(y=\sin(x)+x^3+\sin(x)/x\), similar convergence behavior is reported. The study also reports that 8 bits suffice to saturate final accuracy, that stochastic epochs yield up to \(20\%\) lower error for large hidden layers, that a minimum of approximately \(26\) neurons is needed in the iterations-to-\(3\%\)-error experiment, and that a variable step size with high-to-low “add\_no” speeds convergence by approximately \(2\times\). Circuit-level simulation of a 20-neuron, 13 bit system on \(y=x^3\) confirms online learning, although more neurons and epochs are needed for full convergence.

The FPGA-oriented classification experiment uses a time-multiplexed digital ELM, “NeuPS,” modeled in fixed-point MATLAB with “broken-stick” hidden activations. The input is 28\(\times\)28-pixel binary MNIST digits and the output is 10-way soft decisions. Hidden neurons range from \(1\)k to \(16\)k, training-time weight precision ranges from 13 to 17 bits, the final 6 MSBs are used after training, and training lasts 1 to 5 epochs, where one epoch is \(60\,000\) digits. The best reported accuracy is \(95.05\%\) correct with \(16\)k neurons, 15-bit weights, and 3 epochs.

The resource profile is correspondingly minimal. The per-weight update requires one 1-bit XOR and one increment or decrement of an \(n\)-bit counter, with no multipliers or large memories. Hardware per weight consists of an XOR gate, an \(n\)-bit up/down counter, a small register for step size, and 1 control bit for sign. The cited advantages are that the method is extremely area/power-efficient, fully online and on-chip, and avoids costly divisions or multipliers. The cited limitations are slower convergence than full LMS, lower final accuracy due to sign quantisation, and the need for more hidden neurons for complex tasks.

In the cited literature, the acronym SOL therefore names both a mistake-bound theory for sign-activation networks and a sign-sign LMS-style online update rule for neuromorphic ELM systems. The former provides a characterization of online learnability in terms of margin geometry and \(TS(d,\gamma)\); the latter provides a hardware-friendly approximation to normalized LMS for real-time on-chip learning.

Source: https://www.emergentmind.com/topics/sign-based-online-learning-sol