MKL-Harmonizer Algorithm

• MKL-Harmonizer is a multiple kernel learning technique that employs block-coordinate descent under elastic-net constraints to optimize SVM models.
• The algorithm alternates between an SVM step and a weight update using efficient closed-form and fixed-point methods to achieve rapid, scalable convergence.
• Empirical evaluations demonstrate 2–20× speed improvements over cutting-plane approaches while maintaining optimal convergence and low memory overhead.

The MKL-Harmonizer algorithm refers to a block-coordinate descent technique for multiple kernel learning (MKL) under elastic-net constraints on the kernel weights, introduced and analyzed in (Citi, 2015). It addresses MKL problems by alternating optimization over support vector machine (SVM) variables and the convex elastic-net-constrained kernel weight simplex, leveraging efficient closed-form or fixed-point updates for the weight step. The core motivation is to enable efficient MKL with favorable time and space complexity, supporting scalability to dozens of kernels and offering sharp convergence guarantees.

1. Problem Formulation: Elastic-Net Multiple Kernel Learning

The MKL-Harmonizer algorithm tackles the following penalized SVM learning task using a set of $Q$ candidate positive semi-definite kernels $K_k: \mathcal{X} \times \mathcal{X} \to \mathbb{R}$, their corresponding Gram matrices $G_k$, and regularization parameter $C > 0$. The data are $(x_i, y_i)_{i=1}^N$ with $y_i \in \{-1, +1\}$. The kernel weight vector $w = (w_1, \dots, w_Q) \ge 0$ is constrained to the elastic-net simplex: $$\Delta_\eta = \left\{ w \in \mathbb{R}^Q_+ \mid \|w\|_1 + (1-\eta)\|w\|_2^2 \le 1 \right\}, \quad \eta \in [0, 1]$$ The joint primal is: $$\min_{\substack{w \in \Delta_\eta \ f_k \in \mathcal{H}_k,\,b\in \mathbb{R}}} \left[ \sum_{k=1}^Q \frac{1}{2} \frac{\|f_k\|_{\mathcal{H}_k}^2}{w_k} + C \sum_{i=1}^N \max(0, 1 - y_i[\sum_k f_k(x_i) - b]) \right]$$ Under representer theorem, with fixed $w$, this is a standard SVM with composite Gram matrix $G = \sum_k w_k G_k$. The SVM dual (at fixed $w$) is: $$\max_{\alpha \in \mathbb{R}^N: \sum \alpha_i y_i = 0,\,0\leq \alpha_i\leq C} \left[ \sum_{i=1}^N \alpha_i - \frac{1}{2} (\alpha \circ y)^\top G (\alpha \circ y) \right]$$ The overall MKL problem is posed as a jointly convex minimax optimization over $(w,\alpha)$. The elastic-net constraint interpolates between $\ell_1$ sparsity and $\ell_2$ smoothness in kernel weight selection.

2. Optimization Strategy: Block-Coordinate Descent and Subproblem Solvers

The MKL-Harmonizer proceeds via two-block coordinate descent:

• SVM step: With fixed weights $w$, solve the SVM dual to optimality, yielding dual variables $\alpha$ and bias $b$.
• Weight step: With fixed $\alpha$, update $w$ by minimizing the elastic-net-constrained reciprocal objective: $$\min_{w \in \Delta_\eta} \sum_{k=1}^Q \frac{\gamma_k}{w_k}, \quad \gamma_k = w_k^2 u_k,\quad u_k = (\alpha \circ y)^\top G_k (\alpha \circ y)$$ The weight subproblem is non-convex but pseudoconvex over the positive orthant, admitting a unique minimum.

To certify convergence, the duality gap is checked by solving a linear program: $\max_{w \in \Delta_\eta} u^\top w$ Both the weight step (as a weighted sum of reciprocals, WSR) and the lower-bound step (as a maximization over elastic-net simplex) admit efficient specialized solvers with $O(Q)$ complexity per iteration.

3. Algorithmic Components and Pseudocode

Crucial subroutines for the MKL-Harmonizer algorithm are as follows:

• Algorithm SolveElNetMKL: The master loop alternates SVM solution and weight update, computing primal and lower (LP) bounds until the relative duality gap is below a set tolerance.
• WSR Solver: The key step in weight update. For fixed $\gamma$, iteratively update auxiliary $x > 0$ by fixed-point iteration:

$$x_i \leftarrow \sqrt{\frac{\gamma_i}{q_i}}, \quad q_i = \frac{\partial s(x)}{\partial x_i}$$

Normalize $w = x / s(x)$ with $$s(x) = \frac{\eta}{2} \|x\|_1 + \sqrt{\left(\frac{\eta}{2}\right)^2 \|x\|_1^2 + (1-\eta)\|x\|_2^2}$$.

• LP Solver: Maximizes $u^\top w$ over the elastic-net region via finite-step prune-and-project, solving a quadratic constraint with non-negativity and setting negative projections to zero.

These ingredients yield the fully specified MKL-Harmonizer pseudocode given in (Citi, 2015).

4. Theoretical Properties, Convergence, and Computational Complexity

• Convergence: Global convergence is guaranteed for jointly convex cost (Zangwill’s theorem applies). The block-descent alternation, together with pseudoconvex WSR subproblems, ensures solution to global minimum.
• Efficiency: The per-iteration bottleneck is the SVM step ($O(N^2)$ to $O(N^3)$), with MKL-specific overhead $O(Q)$ per weight/LP step. The approach avoids the $O(Q^2)$ scaling of cutting-plane or active-set methods, enabling practical use with $Q \sim 40$ base kernels.
• Memory: Overall $O(N^2 Q)$ for Gram matrices; substantially less than cutting-plane competitors.

5. Empirical Evaluation and Benchmarks

Citi (2016) (Citi, 2015) reports performance of MKL-Harmonizer against the GMKL cutting-plane implementation of Yang et al. (2011), on both synthetic and UCI datasets with up to $Q\approx40$ kernels. Under $\epsilon=10^{-2}$ relative dual gap:

• 2–5× faster than cutting-plane+MOSEK
• 10–20× faster than cutting-plane+CVX
• $O(Q)$ memory usage, never exceeding $O(N^2 Q)$ For tighter duality tolerances ($10^{-3}-10^{-4}$), the speed advantage increases due to steady improvement in the duality gap with coordinate descent, versus stalling for cutting-plane under expensive subproblems.

6. Context, Significance, and Related Work

The MKL-Harmonizer algorithm demonstrates that MKL with elastic-net constraints is tractable and efficient at moderate-to-large kernel counts. The elastic-net constraint interpolates between fully sparse and smooth kernel combinations, offering flexible adaptivity. The approach outlined in (Citi, 2015) directly addresses scalability, simplicity of implementation (no need for complex convex optimization toolboxes), and convergence certification via the duality gap and efficient approximate LP/WSR solutions. This situates the method as a practical alternative to classic group-lasso and cutting-plane MKL approaches, as detailed in comparative works such as Yang et al. (2011) and Xu et al. (2010).

7. Limitations and Practical Considerations

MKL-Harmonizer, like most block-coordinate methods for MKL, is limited mainly by the scaling of the SVM problem at large $N$ (dataset size), and is most effective when $Q$ (number of kernels) is moderate. All guarantees and efficiency depend on pre-computation of all Gram matrices, which may become prohibitive for extremely large-scale data. A plausible implication is that further advances in scalable SVM solvers or online kernel computation would widen applicability to much larger regimes than explored in (Citi, 2015).