---
title: 'LaProx: Proximal Methods in Optimization & LLM Inference'
url: https://www.emergentmind.com/topics/laprox
type: topic
---

# LaProx: Proximal Methods in Optimization & LLM Inference

LaProx is a term used for several technically distinct methods in recent optimization and machine-learning literature. In long-context large language model inference, LaProx denotes an output-aware, layer-wise KV cache eviction strategy that reformulates eviction as a matrix-multiplication approximation problem [2605.07234]. In earlier optimization work, LaProx denotes the “LAROS Proximal-point” algorithm for sequential feature extraction through approximately rank-one submatrices [1108.0986]. A later French exposition also uses “LaProx” for the proximal gradient method, i.e. Forward–Backward splitting for composite convex minimization [2503.14479]. The common lexical element is proximal methodology, but the underlying objectives, operators, and application domains differ substantially.

## 1. Terminological scope and principal meanings

The main uses of the term can be organized as follows.

| Usage of “LaProx” | Domain | Defining formulation |
|---|---|---|
| LaProx | Long-context LLM inference | Output-aware, layer-wise cache eviction |
| LaProx (“LAROS Proximal-point”) | Sequential feature extraction | Proximal point algorithm for the LAROS problem |
| “LaProx” following “La Méthode du Gradient Proximé” | Composite convex optimization | Proximal gradient / Forward–Backward splitting |

The most recent usage is the cache-eviction method introduced in “Reformulating KV Cache Eviction Problem for Long-Context LLM Inference” [2605.07234]. That work addresses the memory and runtime overhead created by KV cache growth in long-context decoding. By contrast, the 2011 usage concerns a convex optimization model for finding a “large approximately rank-one submatrix,” with sequential feature extraction from image data as the motivating application [1108.0986]. The 2025 usage is broader and pedagogical: “La Méthode du Gradient Proximé” presents the proximal gradient method as a general splitting algorithm for minimizing the sum of two convex functions, one smooth and one possibly nonsmooth [2503.14479].

A recurrent misconception is to treat “LaProx” as the name of a single canonical algorithm. The literature represented here does not support that interpretation. Rather, the term labels different proximal or approximation-based procedures in different subfields.

## 2. LaProx in long-context LLM inference: reformulating KV eviction

In the LLM setting, LaProx begins from a critique of conventional head-wise eviction. At layer \(l\) and head \(h\), the cached keys and values induce attention
\[
\mathbf{A}^{l,h}=\mathrm{Softmax}\Bigl(\tfrac{\mathbf{Q}^{l,h}{\mathbf{K}^{l,h}}^\top}{\sqrt{d_h}}\Bigr),
\qquad
\mathbf{V}^{l,h}\in\mathbb{R}^{S\times d_h},
\]
and a common heuristic assigns token \(j\) the importance
\[
p_j^{l,h}=\frac{1}{S}\sum_{i=1}^S A^{l,h}(i,j).
\]
Eviction then retains the top-\(k\) tokens in each head according to \(p_j^{l,h}\) [2605.07234].

LaProx replaces that head-wise, weight-averaging viewpoint with an output-aware, layer-wise approximation objective. The full multi-head output before the residual connection is written as
\[
\mathbf{O}^l=\sum_{h=1}^H \bigl(\mathbf{A}^{l,h}\mathbf{V}^{l,h}\bigr)W_O^{l,h}
=
\underbrace{\bigl[\mathbf{A}^{l,1}\,\dots\,\mathbf{A}^{l,H}\bigr]}_{A^l}\;
\underbrace{\mathrm{blockdiag}\bigl(V^{l,1}W_O^{l,1},\dots,V^{l,H}W_O^{l,H}\bigr)}_{VW_O^l}.
\]
Cache eviction is therefore viewed as selecting a subset of token rows \(j\) that best approximates the full product \(A^l(VW_O^l)\). Remark 1 in the same work states that the MHA output exactly decomposes additively per head:
\[
\sum_{h=1}^H H^{l,h}W_O^{l,h}
=
\mathrm{Concat}(H^{l,1},\dots,H^{l,H})W_O^l
=
\mathbf{O}^l.
\]

This reformulation changes the object being preserved. Instead of preserving only average incoming attention to a token, LaProx explicitly targets preservation of the layer output after interaction with the projected value states and the output projection matrices. The paper identifies this as the central deficiency of pure attention-weight heuristics: they neglect value representations, output projection, and inter-head interactions.

## 3. Contribution metric, unified eviction, and computational profile

The LaProx score is motivated by classical Monte Carlo matrix-product sampling theory. For approximating \(A\times B\) by row/column sampling, the variance is minimized by sampling index \(j\) with probability proportional to
\[
\|A[:,j]\|_2\times \|B[j,:]\|_2.
\]
Instantiating this in layer \(l\), head \(h\), the method first computes projected values
\[
H^{l,h}=V^{l,h}W_O^{l,h}\in\mathbb{R}^{S\times d_h},
\]
and then defines the raw eviction score
\[
p_{l,h,j}\propto
\bigl\|\mathbf{A}^{l,h}[:,j]\bigr\|_2
\times
\bigl\|H^{l,h}[j,:]\bigr\|_2.
\tag{1}
\]
Here \(\mathbf{A}^{l,h}[:,j]\) is the \(S\)-vector of attention weights pointing to token \(j\), and \(H^{l,h}[j,:]\) is the \(d_h\)-vector of its projected value [2605.07234].

The unified eviction algorithm allocates a total budget \(K\) entries across all layers and heads. It proceeds in three operational stages plus final selection. First, for each layer and head, it forms a prefill-window attention matrix from the last \(w\) queries,
\[
A^{l,h}=\mathrm{Softmax}\Bigl(\tfrac{Q[-w:]\,K^\top}{\sqrt{d_h}}\Bigr),
\qquad
H^{l,h}=V^{l,h}W_O^{l,h}.
\]
Second, for each token index \(j\) older than \(w\), it computes \(p_{l,h,j}\) by equation (1), while assigning \(p_{l,h,j}=\infty\) to the last \(w\) positions so that a history window is always kept. Third, it flattens across heads via \(p_{l,j}=\sum_h p_{l,h,j}\) and applies the layer-wise normalization
\[
s_{l,j}= \frac{p_{l,j}}{\sum_k p_{l,k}}.
\]
Global Top-\(K\) selection is then performed across all \(\{s_{l,j}\}\), and all other entries are evicted from every layer’s \((K,V)\).

The computational overhead is specified explicitly. Scoring each layer/head costs \(O(w\cdot S\cdot d_h)\), flattening and normalization cost \(O(H\,S)\), and global top-\(K\) over \(LS\) items costs \(O(LS\log K)\). Overall, the method adds only one matrix multiply per window plus a single sort of size \(L\,S\), which the paper states is small in practice compared to full attention cost. The same discussion also characterizes LaProx as zero-parameter, requiring only one extra mat-mul per window and interoperating with any pretrained Transformer.

The theoretical contrast with head-wise heuristics is explicit. By the matrix-product sampling argument, selecting tokens by \(\|A[:,j]\|_2\|B[j,:]\|_2\) aligns directly with preserving each layer’s output under a budget. Pure attention-weight heuristics ignore \(\|B[j,:]\|\) and inter-head mixing via \(W_O\), so they can miss tokens with small attention weights but large projected contributions, or the converse.

## 4. Empirical behavior in long-context benchmarks and stated limitations

The evaluation in the cache-eviction paper uses LongBench and Needle-In-A-Haystack. LongBench comprises 16 datasets across Single/Multi-doc QA, summarization, few-shot, synthetic, and code completion; Needle-In-A-Haystack includes 1N-1T, 4N-1T, and 4N-4T retrieval tests. The models are Meta-Llama-3.1-8B-Instruct with a 128K window, Mistral-7B-Instruct with 32K, and Qwen3-8B with 32K. Cache budgets are \(\{128,256,512,1024\}\) tokens, with history window \(w=32\). Metrics are F1, Rouge-L, Accuracy, and EditSim on LongBench as appropriate, retrieval accuracy on NIAH, and efficiency measures including peak GPU memory, prefill+eviction overhead, and per-token latency [2605.07234].

For Meta-Llama-3.1-8B at 128 tokens on average LongBench performance, the reported scores are: FullKV \(49.51\%\), SnapKV \(42.31\%\), AdaKV \(43.12\%\), CAKE \(42.77\%\), CriticalKV \(42.88\%\), and LaProx \(45.19\%\).

| Method | Avg. Score |
|---|---:|
| FullKV | 49.51% |
| SnapKV | 42.31% |
| AdaKV | 43.12% |
| CAKE | 42.77% |
| CriticalKV | 42.88% |
| LaProx | **45.19%** |

Under extreme compression at 128 tokens, the paper states that LaProx reduces accuracy loss by up to \(2\times\) compared to the best head-wise baselines. On Mistral-7B with a 256-token budget in Needle-In-A-Haystack retrieval, CriticalKV and LaProx are reported as \(60.5\%\) versus \(62.5\%\) on 1N-1T, \(58.5\%\) versus \(62.5\%\) on 4N-1T, and \(57.0\%\) versus \(74.5\%\) on 4N-4T. For efficiency on Llama-3.1-8B with 128 tokens and 128K context, FullKV memory is \(63.3\,\mathrm{GB}\) and LaProx memory is \(47.5\,\mathrm{GB}\), corresponding to \(-25\%\); prefill overhead is \(+5\,\mathrm{ms}\) versus SnapKV \(+4\,\mathrm{ms}\); and decode is \(2.3\times\) faster than FullKV while being on par with SnapKV and AdaKV. The abstract further states that the method maintains model performance with only \(5\%\) of the KV cache across 19 datasets and consistently outperforms prior works across all configurations.

The paper also states limitations and potential extensions. The layer-wise normalization \(s_{l,j}=p_{l,j}/\sum_k p_{l,k}\) is described as a simple fix for inter-layer scale disparity and may be improved by learned or adaptive normalizers. Proposed extensions include higher-order matrix-approximation bounds, leverage of head covariance in sampling, joint optimization of window size \(w\) per layer or per head, and combination with quantization for further memory savings. A practical claim in the discussion is that cutting the KV cache to \(5\%-20\%\) of full size while maintaining accuracy enables inference over 100K-token contexts on a single 80 GB GPU. This suggests that the method is intended not only as a compression heuristic but as a systems-level reformulation of the eviction objective.

## 5. LaProx as “LAROS Proximal-point” for sequential feature extraction

In the 2011 optimization usage, LaProx is the proximal-point algorithm developed for the LAROS problem, namely finding a “large approximately rank-one submatrix” [1108.0986]. The data matrix \(A\in\mathbb{R}_+^{m\times n}\) is nonnegative, and the target matrix \(X\in\mathbb{R}^{m\times n}\) should be large in Frobenius norm while also approximately rank one. Doan and Vavasis proposed the convex formulation
\[
\min_X \ \|X\|_*+\theta\|X\|_1
\quad\text{subject to}\quad
A\,X=1,
\]
where \(\|X\|_*\) is the nuclear norm, \(\|X\|_1\) is the entrywise \(\ell_1\) norm, and \(\theta>0\) controls the tradeoff between low rank and sparsity. An equivalent redundant splitting writes \(X_1=X\), \(X_2=X\) and minimizes
\[
\|X_1\|_*+\theta\|X_2\|_1
\quad\text{subject to}\quad
A\,X_1=1,\qquad X_1-X_2=0.
\]

The proximal-point framework is then applied to \(\phi(X)=f(X)+I_C(X)\), with
\[
f(X_1,X_2)=\|X_1\|_*+\theta\|X_2\|_1
\]
and
\[
C=\{(X_1,X_2):A\,X_1=1,\ X_1=X_2\}.
\]
For \(\lambda>0\), the Moreau–Yoshida regularization is
\[
\Phi_\lambda(X)=\min_V\Bigl[\phi(V)+\frac{1}{2\lambda}\|X-V\|_F^2\Bigr],
\]
and the proximal map is
\[
p_\lambda(X)=\arg\min_V\Bigl[\phi(V)+\frac{1}{2\lambda}\|X-V\|_F^2\Bigr].
\]
Optimality is equivalent to the fixed-point condition \(p_\lambda(X)=X\).

The evaluation of \(p_\lambda(X)\) proceeds by dualization. One forms the inner dual function
\[
\Theta_\lambda(X,z)
=
\langle z,b\rangle
-\frac{1}{2\lambda}\|p_\lambda^{(1)}(X_1+\lambda L_1^*z)\|_F^2
-\frac{1}{2\lambda}\|p_{\lambda\theta}^{(2)}(X_2+\lambda L_2^*z)\|_F^2,
\]
where \(b=(1,0)\), \(L(X_1,X_2)=(A\,X_1,X_1-X_2)\), and \(L^*z\) splits into two adjoint parts \(L_1^*z\) and \(L_2^*z\). One then solves
\[
z_\lambda(X)\in\arg\sup_z \Theta_\lambda(X,z),
\]
and updates with closed-form proximal operators:
\[
p_\lambda^{(1)}(M)=U\,\mathrm{Diag}[\max\{\sigma_i(M)-\lambda,0\}]\,V^\top,
\]
which soft-thresholds singular values, and
\[
p_{\lambda\theta}^{(2)}(M)=\mathrm{sign}(M)\circ\max\{|M|-\lambda\theta,0\},
\]
which soft-thresholds entries. The outer proximal-point iteration alternates approximate maximization of \(\Theta_{\lambda_k}(X^k,z)\), primal updates for \(X_1^{k+1}\) and \(X_2^{k+1}\), and the convergence test
\[
\frac{\|X^{k+1}-X^k\|_F}{\lambda_k}<\epsilon.
\]

A notable aspect of this LaProx variant is its \(\epsilon\)-optimality stopping criterion via duality. When the true optimal solution is rank one, \(X^*=\sigma u v^\top\) with \(u,v\ge 0\), the paper derives exact KKT conditions with dual multipliers \(W,V\) satisfying block-wise inequalities and the spectral bound \(\|W\|_2\le 1\). Proposition 4.2 states that the outer loop can stop once an approximate singular triple \((\lambda,u_1,v_1)\) solves
\[
(\lambda A_{11}-\theta E)v_1-u_1\approx 0,\qquad
(\lambda A_{11}-\theta E)^\top u_1-v_1\approx 0,\qquad
\|u_1\|_2=1
\]
to within \(\delta\), while feasible \(W,V\) satisfy tightened bounds
\[
\|W\|_2\le 1-(\|A\|_2+7.5),\qquad \|V\|_\infty\le 1-O(1/\theta),
\]
through a small projected-subgradient subroutine. When these conditions hold, the support of \(u_1\) and \(v_1\) is guaranteed to be exactly that of a rank-one optimal solution.

The sequential feature-extraction algorithm records the extracted feature, zeros out the identified submatrix \(A[I,J]\leftarrow 0\), resets \(X\), and continues until no feature remains. On the sailboat bitmap example, \(A\) is \(4000\times 30\), each \(80\times 50\) bitmap column has 3 of 5 possible boat components, and true rank-one submatrices correspond to hull, sails, rudder, and related parts. Dual-PPA plus the \(\epsilon\)-test extracted 8 sequential features; using the \(\epsilon\)-test allowed stopping after 10–20 outer iterations instead of 50–60, cutting total CPU by approximately \(30\%-50\%\). On the Frey face database, \(A\) is \(560\times 1965\), the first ten features were extracted, and negative features were obtained by applying LaProx to \(B=255\cdot E-A\). These experiments situate LaProx as a first-order method for a nuclear-plus-\(\ell_1\) model rather than as a general-purpose proximal-gradient routine.

## 6. LaProx as proximal gradient and broader proximal context

In the 2025 exposition “La Méthode du Gradient Proximé,” LaProx refers to the proximal gradient method for the composite convex problem
\[
\min_{x\in\mathbb{R}^n} F(x)=f(x)+g(x),
\]
where \(f\) is convex and continuously differentiable with \(L\)-Lipschitz-continuous gradient, and \(g\) is proper, closed, convex, and possibly nonsmooth [2503.14479]. The proximal operator is
\[
\prox_{\lambda g}(v)=\arg\min_{u\in\mathbb{R}^n}
\Bigl\{
\frac{1}{2\lambda}\|u-v\|^2+g(u)
\Bigr\},
\]
and the basic Forward–Backward update is
\[
x_{k+1}=\prox_{\alpha_k g}\bigl(x_k-\alpha_k\nabla f(x_k)\bigr).
\]
The text lists assumptions \(A1\)–\(A3\): \(f\) has \(L\)-Lipschitz gradient, \(g\) has a computable proximal map, and optionally \(F\) is \(\mu\)-strongly convex. Under \(A1\)–\(A2\), the method is well-defined and converges sublinearly; under \(A3\), it converges linearly.

The corresponding convergence results are explicit. With fixed \(0<\alpha\le 1/L\),
\[
F(x_k)-F(x^*)\le \frac{\|x_0-x^*\|^2}{2\alpha k},
\]
and under \(\mu\)-strong convexity,
\[
F(x_k)-F(x^*)\le (1-\alpha\mu)^k\bigl(F(x_0)-F(x^*)\bigr).
\]
The same source presents FISTA, backtracking step sizes, block-coordinate proximal gradient, and proximal Newton or variable-metric variants. Applications listed include sparse linear regression, logistic-\(\ell_1\) classification, TV-regularized image denoising, and matrix completion. In that usage, “LaProx” is a generic algorithmic framework rather than a task-specific method.

A broader proximal context is given by the U-Lagrangian for prox-regular functions. Liu, Eberhard, and Luo define prox-regularity through the lower quadratic model
\[
f(y)\ge f(x)+\langle v,y-x\rangle-\tfrac{\rho}{2}\|y-x\|^2
\]
near \(\bar x\), construct the UV-decomposition
\[
\mathcal V(\bar x)=\mathrm{span}\{g-\tilde g\mid g\in\partial f(\bar x)\},\qquad
\mathcal U(\bar x)=\mathcal V(\bar x)^\perp,
\]
and define the U-Lagrangian
\[
L_\epsilon(u;\bar g_v)=
\inf_{v\in\mathcal V,\ \|v\|\le \epsilon}
\Bigl\{
f(\bar x+u+v)-\langle \bar g_v,v\rangle
\Bigr\}.
\]
They prove prox-regularity, strict differentiability at \(0\), and strong metric regularity properties for \(L_\epsilon\) under stated conditions [1510.08577]. Although that construction is not itself named LaProx in the source, it clarifies the proximal and variational background against which the other LaProx usages sit.

Taken together, these usages show that LaProx is best understood as a polysemous label rather than a single method. In long-context LLM inference it denotes an output-aware global eviction strategy; in sequential feature extraction it denotes a proximal-point solver for a nuclear-plus-\(\ell_1\) convex program; and in composite optimization it serves as shorthand for the proximal gradient method. The shared theme is the use of proximal or approximation principles to control complexity while preserving the structure most relevant to the target objective.

Source: https://www.emergentmind.com/topics/laprox