--- title: Efficient Learning Algorithm (eALS) url: https://www.emergentmind.com/topics/efficient-learning-algorithm-eals type: topic --- # Efficient Learning Algorithm (eALS) Efficient Learning Algorithm (eALS) is a matrix factorization (MF) framework designed to efficiently optimize weighted squared error objectives for implicit feedback under non-uniform weighting schemes, with particular emphasis on the full exploitation of missing data as negative signals. eALS extends classical Alternating Least Squares (ALS) methods to support per-entry, non-uniform weights—including adaptive schemes based on item popularity or side information—crucially improving both the fidelity of modeling user behavior and computational scalability in large-scale recommendation systems. The framework uses an element-wise coordinate descent procedure, advanced caching strategies, and compact low-rank representations of the missing data weights to achieve computational costs competitive with or superior to uniform-weight alternatives [1708.05024][1811.04411]. ## 1. Weighted Matrix Factorization with Non-Uniform Missing Data Weights Let $R \in \mathbb{R}^{M \times N}$ denote a user–item interaction matrix with observed entries $(u,i) \in \mathcal{R}$, where $r_{ui}$ denotes implicit feedback (e.g., $r_{ui} = 1$ for observed $(u,i)$, $0$ otherwise). Conventional MF for implicit feedback uses a weighted squared error loss of the form: \[ J(P, Q) = \sum_{u=1}^M \sum_{i=1}^N w_{ui}(r_{ui} - p_u^\top q_i)^2 + \lambda \left( \sum_{u=1}^M \|p_u\|_2^2 + \sum_{i=1}^N \|q_i\|_2^2 \right) \] where $p_u, q_i \in \mathbb{R}^K$ are learned factors and $w_{ui}$ are entry-specific non-negative weights. Non-uniform weighting strategies address two key issues: (1) most implicit-feedback entries are missing, so including them as negative signal with adaptive weights improves fidelity; (2) real-world exposure and popularity induce substantial heterogeneity that is poorly modeled by uniform priors. For example, missing-entry weights $c_i$ can be set proportional to item popularity $f_i$ via $c_i = c_0\,(f_i)^\alpha / \sum_j (f_j)^\alpha$, with $c_0 > 0$, $\alpha \in [0,1]$ [1708.05024]. More generally, missing-entry weights $w_{ui}$ can be parameterized as a low-rank product: $w_{ui} = a_u^\top b_i$, allowing the modeling of arbitrary patterns using compact SVD-based factors $A \in \mathbb{R}^{M \times Z},\ B \in \mathbb{R}^{N \times Z}$ [1811.04411]. ## 2. Element-wise ALS (eALS): Coordinate Descent Approach eALS employs coordinate descent on individual scalar factors $p_{u,f},\ q_{i,f}$, unlike classical ALS which updates full vectors via $K \times K$ solves. For each user $u$ and scalar component $f$: \[ p_{u,f} = \frac{ \sum_{i \in \mathcal{R}_u} w_{ui} (r_{ui} - \hat{r}_{ui}^f) q_{i,f} - \sum_{k \neq f} p_{u,k} s^q_{kf} }{ \sum_{i \in \mathcal{R}_u} (w_{ui} - c_i) q_{i,f}^2 + s^q_{ff} + \lambda } \] where $\hat{r}_{ui}^f = p_u^\top q_i - p_{u,f} q_{i,f}$ and $s^q_{kf}$ is an entry of the "cache" matrix $S^q = \sum_{i=1}^N c_i q_i q_i^\top$. Analogous forms apply for item updates $q_{i,f}$ with $S^p = P^\top P$ [1708.05024]. When missing-entry weights are expressed with SVD factors, all necessary sums involving $w_{ui}$ can be decomposed as inner products and tensor contractions, leveraging specific cache tensors $S^q_{t,f,k}, S^p_{t,f,k}$ for efficient recomputation [1811.04411]. ## 3. Computational Efficiency and Caching Strategies The elementary step of eALS updates requires $O(K + |\mathcal{R}_u|)$ per user (or $O(K + |\mathcal{R}_i|)$ per item), after caches have been constructed. Specifically, cache matrices/tensors (e.g., $S^q, S^p, S^q_{t,f,k}$) aggregate contributions over missing entries without enumerating the full $M \times N$ space, exploiting sparsity and low-rank structure. For simple popularity-based weights ($Z=1$), the per-iteration cost is $O((M+N)K^2 + |\mathcal{R}| K)$ [1708.05024]; for general low-rank weights, the cost is $O((M+N) K^2 Z + |\mathcal{R}| K Z)$ [1811.04411]. Compared to vector-wise ALS (which requires $O((M+N) K^3 + |\mathcal{R}| K^2)$), and to naïve element-wise approaches (which may require $O(MNK)$ per sweep), eALS achieves a significant reduction in both asymptotic and observed runtime. Experimental results confirm speed-ups by factors of $K$ vs. classical ALS and by orders of magnitude vs. naïve element-wise methods, while matching or improving recommendation quality. ## 4. Online and Incremental Model Updates eALS supports efficient online updates by refreshing only those user and item factors involved in new interactions, plus the caches required for coordinate updates. When a new interaction $(u,i)$ arrives, the following steps are performed: 1. If $u$ or $i$ is new, random initialize $p_u$ or $q_i$. 2. Update $p_u$ and $q_i$ via one (or a few) coordinate-descent passes, recomputing only relevant cache entries. 3. Refresh the associated elements in $S^p, S^q$. Each interaction is absorbed in $O(K^2 + K|R_u|)$ (user) and $O(K^2 + K|R_i|)$ (item) time, independent of $M, N, |\mathcal{R}|$ [1708.05024]. Empirically, one online iteration per new tuple suffices to maintain model quality. ## 5. Empirical Performance and Benchmark Results On large implicit-feedback datasets (Yelp, Amazon-Movies), eALS demonstrates both superior recommendation accuracy and significant speedup. Key metrics include: - On Yelp ($M \approx 25,000$, $N \approx 26,000$, $|\mathcal{R}| \approx 7.3 \times 10^5$, $K=128$): eALS achieves HR@100 $\approx 0.242$, NDCG@100 $\approx 0.144$, outperforming RCD, classical ALS, and BPR [1708.05024][1811.04411]. - Training time per iteration: with $K=128$, ALS requires $\sim$221s, RCD $\sim$10s, eALS $\sim$13s on Yelp; for Amazon with $M \approx 117,000$, $N \approx 75,000$, $|\mathcal{R}| \approx 5 \times 10^{6}$, eALS runs in $\sim$72s vs. 1260s (ALS) and 42s (RCD). - Non-uniform missing weights (item popularity) yield up to 10–20% relative improvement in HR@100 and NDCG@100 compared to uniform-weighted baselines; all accuracy gains are statistically significant at $p<0.01$. - For online protocols, eALS updates raise HR from $\sim$0.08 (cold start) to $\sim$0.22 after a single incremental pass, with the best online weighting $w_\text{new}$ improving NDCG by $\sim$5% [1708.05024]. ## 6. Applicability, Extensions, and Implications eALS allows MF to exploit all missing entries as informative negative signal with adaptive weighting, removing the need for negative sampling or uniformity constraints. The low-rank weight decomposition enables encoding of arbitrary patterns in missingness, including item popularity, user activity, and exposure information [1811.04411]. The eALS caching and coordinate update strategies can be extended to other loss functions (e.g., weighted hinge) and incorporated into neural or higher-order factorization models. This approach offers a scalable, negative-aware MF solution for large-scale recommender systems, handling matrices with hundreds of millions of missing entries efficiently. ## 7. Summary Table: Cost and Functional Comparison | Method | Missing Weights | Per-Iteration Complexity | |-------------|--------------------------|--------------------------------------------| | ALS | Uniform | $O((M+N)K^3 + |\mathcal{R}| K^2)$ | | RCD | Uniform | $O((M+N)K^2 + |\mathcal{R}| K)$ | | eALS | Non-uniform, low-rank | $O((M+N) K^2 Z + |\mathcal{R}| K Z)$ | When $Z=1$ (popularity-based weighting), eALS matches the most efficient known solvers while modeling off-diagonal heterogeneity in missing entries [1811.04411][1708.05024]. For higher-rank weighting schemes, the cost scales linearly in $Z$ but remains practical for small $Z$. --- Efficient Learning Algorithm (eALS) thus provides a theoretically-grounded, computationally efficient, and empirically proven framework for large-scale matrix factorization on implicit feedback, supporting rich, non-uniform negative signal modeling and fast, incremental updates [1708.05024][1811.04411]. Source: https://www.emergentmind.com/topics/efficient-learning-algorithm-eals