Forward Stagewise Multiview Boosting

• Forward Stagewise Additive Multiview Boosting is an ensemble method that trains weak learners on distinct feature views to address multiclass classification challenges.
• The approach introduces a novel exponential loss function with 1/V normalization, ensuring that errors from multiple views are collaboratively penalized to improve convergence.
• Empirical results show that SAMA-AdaBoost achieves faster convergence, higher margins, and better generalization compared to traditional boosting methods.

Forward Stagewise Additive Multiview Boosting refers to a mathematically grounded ensemble learning approach in which weak learners are trained collaboratively across multiple feature subsets ("views") for multiclass classification. The SAMA-AdaBoost algorithm, the canonical representative of this class, extends traditional forward stagewise (additive) boosting to a multiview setting by minimizing a novel exponential loss tailored to collaborative, multiclass prediction. This approach is characterized by a rigorous mathematical framework, explicit convergence and margin bounds, and an emphasis on collaborative regularization among weak learners from different views (Lahiri et al., 2016).

1. Problem Setup and Mathematical Notation

The multiview boosting scenario considers a labeled training set:

$S = \{(x_i^1, x_i^2, \dots, x_i^V, y_i)\}_{i=1}^N$

where $V$ is the number of views (feature subsets), each $x_i^v \in \mathcal{X}^v$ is the $v$-th view of instance $i$, and $y_i \in \{1,2,\dots,K\}$ is the multiclass label encoded as a one-hot vector $Y_i \in \{0,1\}^K$, with $(Y_i)_k = 1$ iff $y_i = k$.

On each view $v$, a series of weak learners $$h_{v,m}:\mathcal{X}^v \rightarrow \{e_1,\dots,e_K\}$$ are trained. Each weak learner's output is mapped to a signed one-hot encoded $K$-vector $\widetilde h_{v,m}(x_i)$: $$\widetilde h_{v,m,k}(x_i) = \begin{cases} +1, & \text{if } h_{v,m}(x_i^v)=k \text{ and } y_i=k \ -1, & \text{if } h_{v,m}(x_i^v)=k \text{ and } y_i \neq k \ 0, & \text{otherwise} \end{cases}$$

2. Forward Stagewise Additive Model

The SAMA-AdaBoost algorithm generalizes the forward additive model to multiview multiclass learning: $$F_{V,M}(x) = \sum_{v=1}^V \sum_{m=1}^{M_v} \alpha_{v,m} \widetilde{h}_{v,m}(x)$$ where $\alpha_{v,m}$ are the stagewise weights. The single-view case corresponds to classical boosting such as SAMME.

A novel exponential loss function that reflects the aggregate margin over all views is defined: $$L_i = \exp\left[-\frac{Y_i^T F_{V,M}(x_i)}{V}\right]$$ The overall loss to minimize is

$$L(F) = \sum_{i=1}^N \exp\left[-\frac{Y_i^T F(x_i)}{V}\right]$$

This loss function upweights examples that are misclassified by a greater number of views.

3. Stagewise Optimization and Weight Updates

Boosting proceeds in rounds. At each round $m$:

• Instance weights are defined by the current ensemble margin:

$$W_i^{(m)} = \exp\left[-\frac{Y_i^T F^{m-1}(x_i)}{V}\right]$$

• The optimal set of weak learners $\{\widetilde h_{v,m}\}$ and shared step-size $\beta_m$ are selected to minimize

$$A(\beta) = \sum_{i=1}^N W_i^{(m)} \exp\Big[-\beta (1-2b_i/V)\Big]$$

where $b_i$ denotes the count of views which misclassify $x_i$.

• $\beta_m$ is determined numerically as the minimum of a strictly convex function:

$$\frac{dA}{d\beta} = 0 \implies \sum_i W_i^{(m)} (1-2b_i/V) \exp\Big[-\beta (1-2b_i/V)\Big] = 0$$

• The weights are updated:

$$W_i^{(m+1)} = W_i^{(m)}\exp\Big[-\beta_m (1-2b_i/V)\Big]$$

with renormalization to ensure $\sum_i W_i^{(m+1)}=1$.

• At prediction time, the ensemble outputs class $k$ maximizing the weighted vote:

$$F(x) = \arg\max_{k} \sum_{m=1}^M \beta_m \sum_{v=1}^V I\big(h_{v,m}(x^v)=k\big)$$

This stagewise process ensures that no single view can dominate, and examples misclassified by more views receive stronger weight adjustments.

4. Regularization and View Collaboration

The exponential loss contains a $1/V$ normalization in the exponent, regularizing the influence of individual views. An example incorrectly classified by only a small fraction of views receives a moderate upweight in loss, while broad disagreement among views leads to more significant penalization. This mechanism restricts overlearning by weak learners in any single view and addresses overfitting by fostering collaboration among views.

5. Convergence Analysis and Margin Bounds

Two principal theoretical guarantees underpin SAMA-AdaBoost:

• Training Error Upper Bound: Let $Z_m = \sum_i W_i^{(m)} \exp[-\beta_m (1-2b_i/V)]$ at round $m$.

$$\text{Training error} \leq \frac{\prod_{m=1}^M Z_m}{\exp(\sum_{m=1}^M \beta_m)}$$

As each $Z_m < \exp(\beta_m)$, the bound decays to zero as $M\to\infty$.

• Margin-Based Generalization Bound: Define a normalized classifier

$$\mathcal{H}(x) = \frac{\sum_{m,v} \beta_m h_{v,m}(x)}{\sum_m \beta_m}$$

For margin $\theta>0$,

$$P_x[y\mathcal{H}(x)\leq\theta] \leq \frac{\prod_{m=1}^M Z_m}{\exp(\sum_{m=1}^M \beta_m)}\exp\left(\frac{2\theta}{V}\right)$$

This quantifies the decay in low-margin examples during training and establishes superior convergence properties over previous multiview and classical boosting formulations.

6. Comparative Performance and Empirical Findings

SAMA-AdaBoost exhibits several advantages compared to prior methods:

• Versus traditional AdaBoost and SAMME, SAMA-AdaBoost demonstrates faster convergence in (theoretical) training-error bounds and produces higher margins, indicative of improved generalization.
• Relative to earlier heuristic models such as MA-AdaBoost, SAMA-AdaBoost employs an exact convex minimization for the step-size, resulting in solutions closer to the global minimum for the exponential loss.
• Compared to other multiview algorithms (e.g., Mumbo, 2-Boost, Co-AdaBoost, AdaBoost.Group), SAMA-AdaBoost:
  • Scales to multiclass and $V>2$ views settings
  • Relies on forward-stagewise optimization, not heuristic weight-transfer strategies
  • Achieves lower test error at equivalent boosting rounds on benchmarks such as 100-Leaves, eye/non-eye, MNIST, and standard UCI datasets
  • Produces accurate, diverse ensemble members (as evidenced in kappa-error diagrams) and demonstrates robustness under label noise
  • Is computationally faster per round than SAMME or Mumbo, leveraged by training multiple weak learners in parallel within low-dimensional feature spaces

7. Algorithmic Summary

A summary of the SAMA-AdaBoost algorithmic workflow is as follows:

• Input: Training dataset $S = \{(x_i^v, y_i)\}$ for $v=1,...,V$, rounds $M$
• Initialize weights $W_i = 1/N$
• For $m = 1$ to $M$:
  1. Train each view's weak learner $h_{v,m}$ using $W$
  2. For each instance, compute $b_i = \#\{v: h_{v,m}(x_i^v) \neq y_i\}$
  3. (Optional) Select a subset of high-performing views
  4. Solve for $\beta_m$ minimizing $\sum_{i} W_i \exp[-\beta(1-2b_i/V)]$
  5. Update $W_i \gets W_i \cdot \exp[-\beta_m(1-2b_i/V)]$, then renormalize

• Output: Classifier $F(x)$ assigns the class $k$ maximizing the summed, weighted votes across rounds and views

• Termination: When $M$ rounds are reached or the error bound falls below a specified threshold

SAMA-AdaBoost provides a principled, collaborative multiview boosting methodology that robustly optimizes a loss function tailored to multiclass, multiview settings, yielding both theoretical and empirical improvements over prior boosting frameworks (Lahiri et al., 2016).