--- title: 'GreedyLore: Greedy Methods in Optimization' url: https://www.emergentmind.com/topics/greedylore type: topic --- # GreedyLore: Greedy Methods in Optimization GreedyLore is best understood, in current arXiv usage, as a family of greedy or feedback-driven procedures rather than as a single standardized object. The term corresponds most directly to a dynamic learning-rate scheduler that reacts to recent loss changes, to a greedy low-rank gradient compressor for distributed optimization, and, by closely related usage, to greedy sparse action discovery for agentic LLMs and greedy sample selection in Loewner interpolation [2512.14527] [2507.08784] [2601.08280] [2303.01015]. This suggests a common technical motif: a locally greedy update is made tractable by additional structure such as bounded loss feedback, contractive compression, block sparsity, or a certified interpolation indicator. ## 1. Terminological scope and defining pattern In the literature, the exact referent of GreedyLore depends on context. In one line of work it corresponds to **GreedyLR**, a scheduler that changes the global learning rate online using only recent loss changes. In another, it is the explicit name of a **Greedy Low-Rank gradient compression** method for distributed learning. A directly related reference develops a greedy Orthogonal Matching Pursuit–style theory for sparse action discovery in agentic LLMs, and a further related reference presents a greedy Loewner sampling strategy with a non-intrusive indicator [2512.14527] [2507.08784] [2601.08280] [2303.01015]. A plausible implication is that GreedyLore denotes a recurring research pattern rather than a single algorithmic lineage. Across these works, the greedy step is not treated as a purely heuristic choice. It is paired with a structural device that controls failure modes: patience and clipping in learning-rate control, error feedback and contractivity in compression, incoherence and coverage in block-sparse recovery, or an analytically motivated denominator indicator in surrogate modeling. ## 2. GreedyLore as dynamic learning-rate scheduling In its most direct optimization usage, GreedyLore corresponds to **GreedyLR**, a dynamic scheduler for stochastic training. The method maintains a learning rate \(\gamma_t\), a factor \(F\in(0,1)\), and a loss sequence \(l_t\). Its basic rule is simple: if the current loss is better than the previous one, the scheduler increases the learning rate by dividing by \(F\); otherwise it decreases the learning rate by multiplying by \(F\). The optimization step is \(x_{t+1}=x_t-\gamma_t g_t\), with \(g_t=\nabla f_{i_t}(x_t)\) and \(l_t=f_{i_t}(x_t)\) for sampled index \(i_t\) [2512.14527]. The practical implementation is more elaborate. It adds optional smoothing over a `window_size`, relative thresholding through `is_better(a,best)`, `num_good_epochs` and `num_bad_epochs` counters, multiplicative clipping through `min_lr` and `max_lr`, and additional stabilization through `patience`, `cooldown`, `warmup`, `eps`, and `reset_start`. The intended effect is to preserve the basic loss-direction logic while suppressing spurious reactions to noise. The same source emphasizes the tradeoff in \(F\): smaller \(F\) makes changes more aggressive, while values closer to \(1\) produce gentler adaptation [2512.14527]. The theoretical treatment is given for SGD on a convex finite-sum objective \(f(x)=\frac{1}{n}\sum_{i=1}^n f_i(x)\) under \(L_{\max}\)-smoothness. The paper states an \(O(1/T)\) bound for the averaged iterate \(\bar x_T\) when \(\min_{\rm LR}\le \gamma_t\le \max_{\rm LR}\) holds uniformly, with a residual term depending on the learning-rate bounds and \(L_{\max}\). It also states \(F^*=1-\frac{1}{L_{\max}}\) as the scaling factor that “maximizes the convergence rate,” although the paper itself notes that the derivation has algebraic and formatting issues and that \(L_{\max}\) is not actionable in deep networks [2512.14527]. Empirically, the study is broad. For small models, it reports **132 experiments** across **16 model architectures** and **15 datasets**, comparing against **Linear, Cosine, Polynomial, and Constant+warmup** under **AdamW, Adafactor, Adagrad, and SGD**. Across these runs, GreedyLR was **as good or better in 86.73\% of cases**, better in **57.14\%**, and clearly better in **24.49\%**. For large models, it reports **8 fine-tuning experiments** on **Phi-2 (2B)**, **Falcon 7B**, and **Gemma 7B**, with GreedyLR **as good or better 83.33\%** of the time and clearly better **62.5\%** of the time. In pre-training, **Llama-3.2-1B** on **RedPajama-arxiv** for **1000 steps** achieved final loss **2.16 vs 2.28** against Cosine. The robustness study spans **8,100 runs** and reports the best median final loss for GreedyLR, **0.148**, versus **0.232** for Cosine, with median recovery time **12 steps** versus **45 steps** for Cosine [2512.14527]. The paper is explicit that this is not universal dominance. It identifies failure cases in large-model fine-tuning, stresses that the formal proofs do not cover the full practical scheduler with smoothing and patience, and notes that loss changes can reflect data heterogeneity rather than optimization progress. Accordingly, GreedyLore in this sense is best interpreted as a low-overhead adaptive default rather than a parameter-free or uniformly superior scheduler [2512.14527]. ## 3. GreedyLore as low-rank gradient compression in distributed learning A second, explicit usage is **Greedy Low-Rank Gradient Compression** for distributed stochastic optimization over matrix variables \(\bm X\in\mathbb R^{m\times n}\), with objective \[ \min_{\bm X\in\mathbb R^{m\times n}} f(\bm X):=\frac{1}{N}\sum_{i=1}^N f_i(\bm X),\qquad f_i(\bm X):=\mathbb E_{\bm\xi\sim\mathcal D_i}F_i(\bm X,\bm\xi). \] Each worker computes a stochastic local gradient \(\bm G_t^{(i)}\), and the compression operator is \[ \mathcal C_t(\bm G_t^{(i)})=\bm P_t\bm P_t^\top \bm G_t^{(i)}, \] where \(\bm P_t\in\mathbb R^{m\times r}\) has orthonormal columns. The communicated coefficient is \(\bm R_t^{(i)}=\bm P_t^\top \bm G_t^{(i)}\), and the global compressed gradient is reconstructed as \(\hat{\bm G}_t=\bm P_t\bm R_t\) after all-reduce [2507.08784]. The distinctive ingredient is the combination of **greedy projection**, **error feedback**, and **semi-lazy subspace updates**. Every \(\tau\) iterations, the algorithm performs a full SVD of the global gradient and sets \(\bm P_t=\bm U_{:,:r}\). Between refreshes, it does not freeze the projector. Instead, it keeps the orthogonal basis \(\bm U\) from the last SVD and greedily reselects which \(r\) columns to activate by solving \[ \bm P_t=\arg\min_{\bm P\in \mathcal S(\bm U,r)} \|\bm G_t-\bm P\bm P^\top \bm G_t\|_F^2, \] equivalently choosing the \(r\) indices with largest \(\|\bm u_j^\top \bm G_t\|_F^2\). The local error-feedback recursion is \[ \hat{\bm G}_t^{(i)}=\mathcal C_t(\bm G_t^{(i)}+\bm E_{t-1}^{(i)}),\qquad \bm E_t^{(i)}=\bm G_t^{(i)}+\bm E_{t-1}^{(i)}-\hat{\bm G}_t^{(i)}. \] A central negative observation is that with a fixed lazy projector, \(\mathcal C_t(\bm G_t^{(i)}+\bm E_{t-1}^{(i)})=\mathcal C_t(\bm G_t^{(i)})\), so error feedback becomes ineffective. The semi-lazy rule is introduced precisely to avoid that failure mode [2507.08784]. The analysis requires the compressor to remain contractive. GreedyLore proves \[ \|\bm G_t-\mathcal C_t(\bm G_t)\|_F^2\le \left(1-\frac{r}{m}\right)\|\bm G_t\|_F^2, \] so \(\delta=r/m\) acts as the compression factor. To avoid full-gradient aggregation on every step, the method uses Gaussian sketches \(\bm v_j\sim\mathcal N(\bm0,\bm I_n)\) and communicates scalars \(\lambda_j^{(i)}=\bm u_j^\top \bm G_t^{(i)}\bm v_j\), with \[ \mathbb E[\overline{\lambda}_j^2]=\|\bm u_j^\top \bm G_t\|_F^2. \] This yields an approximate global top-\(r\) rule with only an additional length-\(m\) communication vector per iteration [2507.08784]. Under standard smooth nonconvex assumptions, the paper proves convergence for both **MSGD** and **Adam**. Its headline rate is \[ \mathcal O\!\left(\frac{\sigma}{\sqrt{NT}}+\frac{1}{T}\right), \] which it identifies as the first linear-speedup convergence rate for low-rank gradient compression, with speedup in the number of workers \(N\). The average communication cost per iteration is summarized as \[ nr+m+\frac{mn}{\tau}. \] The experiments cover **ResNet-18** on **CIFAR-10** and **CIFAR-100**, **LLaMA** pre-training on **C4**, and **RoBERTa-base** fine-tuning on **GLUE**. For **LLaMA-1B**, the reported average per-iteration time is **3.7494 s** for AdamW versus **2.7403 s** for GreedyLore, about a **27\% speedup**, while training and validation behavior remain close to dense baselines and better than GaLore and PowerSGD in the reported settings [2507.08784]. The main caveats are equally explicit. The method is matrix-centric, assumes periodically refreshed dense communication, and relies on stronger boundedness assumptions for the Adam analysis. Its central contribution is therefore not the claim that any greedy low-rank projection converges, but that a specific combination of greedy selection, error feedback, and semi-lazy contractive updates does [2507.08784]. ## 4. GreedyLore as sparse action discovery in agentic LLMs A directly related use of the term appears in sparse action discovery for agentic LLMs. The model considers a large action space \(\mathcal A=\{1,\dots,M\}\), a latent state \(z_t\in\mathbb R^d\), and a contextual linear reward \[ r_t=\langle W^\star_{a_t},z_t\rangle+\varepsilon_t, \] where \(W^\star\in\mathbb R^{M\times d}\) is row-sparse: \(W^\star_a=0\) for all \(a\notin S^\star\), with \(|S^\star|=k\ll M\). Using the joint feature map \(\psi(z_t,a_t)=e_{a_t}\otimes z_t\), the problem becomes block-sparse support recovery in \(\mathbb R^{Md}\) [2601.08280]. The proposed greedy algorithm is **Contextual Block-OMP**. Starting from residual \(u^{(0)}=r\) and empty support \(S_0=\varnothing\), it computes block scores \[ \gamma_j^{(m-1)}=\|\Psi_j^\top u^{(m-1)}\|_2, \] selects \[ j_m=\arg\max_{j\in\mathcal A}\|\Psi_j^\top u^{(m-1)}\|_2, \] updates the support \(S_m=S_{m-1}\cup\{j_m\}\), refits by least squares on the selected blocks, and updates the residual. The algorithm runs for \(k\) iterations and returns \(S_k\) [2601.08280]. The theory assumes sub-Gaussian latent states, sub-Gaussian noise, coverage \(n_j\ge n_{\min}\) for all relevant actions, an incoherence condition \[ \max_{j\notin S^\star} \big\|\Psi_j^\top \Psi_{S^\star}(\Psi_{S^\star}^\top\Psi_{S^\star})^{-1}\big\|_{2\to 2} \le \mu, \] and a minimum signal bound \(\min_{j\in S^\star}\|W^\star_j\|_2\ge b_{\min}\). Under these conditions, if \[ \lambda_{\min}(G_{S^\star})\ge \alpha T \] and \[ \max_{j\in\mathcal A}\|\Psi_j^\top\varepsilon\|_2 \le \frac{(1-\mu)}{2}\alpha T b_{\min}, \] then exact support recovery holds: \(S_k=S^\star\). Under the sub-Gaussian assumptions, the paper states that these events hold with probability at least \(1-M^{-2}\) provided \[ T\gtrsim kd\log M \] and \(n_{\min}\gtrsim d\log M\) [2601.08280]. The information-theoretic side is equally important. Without sparsity, even the non-contextual special case requires \(T