LaProx: Proximal Methods in Optimization & LLM Inference

• LaProx is a term designating distinct proximal-based methods applied in long-context LLM inference, sequential feature extraction, and composite convex optimization.
• In long-context LLM inference, LaProx employs an output-aware, layer-wise KV cache eviction strategy to preserve model output accuracy while reducing computational overhead.
• Other variants of LaProx include a proximal-point solver for rank-one submatrix extraction and usage as the proximal gradient method, showcasing its diverse applications.

LaProx is a term used for several technically distinct methods in recent optimization and machine-learning literature. In long-context LLM inference, LaProx denotes an output-aware, layer-wise KV cache eviction strategy that reformulates eviction as a matrix-multiplication approximation problem (Mai et al., 8 May 2026). In earlier optimization work, LaProx denotes the “LAROS Proximal-point” algorithm for sequential feature extraction through approximately rank-one submatrices (Doan et al., 2011). A later French exposition also uses “LaProx” for the proximal gradient method, i.e. Forward–Backward splitting for composite convex minimization (Combettes, 18 Mar 2025). The common lexical element is proximal methodology, but the underlying objectives, operators, and application domains differ substantially.

1. Terminological scope and principal meanings

The main uses of the term can be organized as follows.

Usage of “LaProx”	Domain	Defining formulation
LaProx	Long-context LLM inference	Output-aware, layer-wise cache eviction
LaProx (“LAROS Proximal-point”)	Sequential feature extraction	Proximal point algorithm for the LAROS problem
“LaProx” following “La Méthode du Gradient Proximé”	Composite convex optimization	Proximal gradient / Forward–Backward splitting

The most recent usage is the cache-eviction method introduced in “Reformulating KV Cache Eviction Problem for Long-Context LLM Inference” (Mai et al., 8 May 2026). That work addresses the memory and runtime overhead created by KV cache growth in long-context decoding. By contrast, the 2011 usage concerns a convex optimization model for finding a “large approximately rank-one submatrix,” with sequential feature extraction from image data as the motivating application (Doan et al., 2011). The 2025 usage is broader and pedagogical: “La Méthode du Gradient Proximé” presents the proximal gradient method as a general splitting algorithm for minimizing the sum of two convex functions, one smooth and one possibly nonsmooth (Combettes, 18 Mar 2025).

A recurrent misconception is to treat “LaProx” as the name of a single canonical algorithm. The literature represented here does not support that interpretation. Rather, the term labels different proximal or approximation-based procedures in different subfields.

2. LaProx in long-context LLM inference: reformulating KV eviction

In the LLM setting, LaProx begins from a critique of conventional head-wise eviction. At layer $l$ and head $h$, the cached keys and values induce attention

$$\mathbf{A}^{l,h}=\mathrm{Softmax}\Bigl(\tfrac{\mathbf{Q}^{l,h}{\mathbf{K}^{l,h}}^\top}{\sqrt{d_h}}\Bigr), \qquad \mathbf{V}^{l,h}\in\mathbb{R}^{S\times d_h},$$

and a common heuristic assigns token $j$ the importance

$p_j^{l,h}=\frac{1}{S}\sum_{i=1}^S A^{l,h}(i,j).$

Eviction then retains the top-$k$ tokens in each head according to $p_j^{l,h}$ (Mai et al., 8 May 2026).

LaProx replaces that head-wise, weight-averaging viewpoint with an output-aware, layer-wise approximation objective. The full multi-head output before the residual connection is written as

$$\mathbf{O}^l=\sum_{h=1}^H \bigl(\mathbf{A}^{l,h}\mathbf{V}^{l,h}\bigr)W_O^{l,h} = \underbrace{\bigl[\mathbf{A}^{l,1}\,\dots\,\mathbf{A}^{l,H}\bigr]}_{A^l}\; \underbrace{\mathrm{blockdiag}\bigl(V^{l,1}W_O^{l,1},\dots,V^{l,H}W_O^{l,H}\bigr)}_{VW_O^l}.$$

Cache eviction is therefore viewed as selecting a subset of token rows $j$ that best approximates the full product $A^l(VW_O^l)$. Remark 1 in the same work states that the MHA output exactly decomposes additively per head: $h$0

This reformulation changes the object being preserved. Instead of preserving only average incoming attention to a token, LaProx explicitly targets preservation of the layer output after interaction with the projected value states and the output projection matrices. The paper identifies this as the central deficiency of pure attention-weight heuristics: they neglect value representations, output projection, and inter-head interactions.

3. Contribution metric, unified eviction, and computational profile

The LaProx score is motivated by classical Monte Carlo matrix-product sampling theory. For approximating $h$1 by row/column sampling, the variance is minimized by sampling index $h$2 with probability proportional to

$h$3

Instantiating this in layer $h$4, head $h$5, the method first computes projected values

$h$6

and then defines the raw eviction score

$h$7

Here $h$8 is the $h$9-vector of attention weights pointing to token $$\mathbf{A}^{l,h}=\mathrm{Softmax}\Bigl(\tfrac{\mathbf{Q}^{l,h}{\mathbf{K}^{l,h}}^\top}{\sqrt{d_h}}\Bigr), \qquad \mathbf{V}^{l,h}\in\mathbb{R}^{S\times d_h},$$0, and $$\mathbf{A}^{l,h}=\mathrm{Softmax}\Bigl(\tfrac{\mathbf{Q}^{l,h}{\mathbf{K}^{l,h}}^\top}{\sqrt{d_h}}\Bigr), \qquad \mathbf{V}^{l,h}\in\mathbb{R}^{S\times d_h},$$1 is the $$\mathbf{A}^{l,h}=\mathrm{Softmax}\Bigl(\tfrac{\mathbf{Q}^{l,h}{\mathbf{K}^{l,h}}^\top}{\sqrt{d_h}}\Bigr), \qquad \mathbf{V}^{l,h}\in\mathbb{R}^{S\times d_h},$$2-vector of its projected value (Mai et al., 8 May 2026).

The unified eviction algorithm allocates a total budget $$\mathbf{A}^{l,h}=\mathrm{Softmax}\Bigl(\tfrac{\mathbf{Q}^{l,h}{\mathbf{K}^{l,h}}^\top}{\sqrt{d_h}}\Bigr), \qquad \mathbf{V}^{l,h}\in\mathbb{R}^{S\times d_h},$$3 entries across all layers and heads. It proceeds in three operational stages plus final selection. First, for each layer and head, it forms a prefill-window attention matrix from the last $$\mathbf{A}^{l,h}=\mathrm{Softmax}\Bigl(\tfrac{\mathbf{Q}^{l,h}{\mathbf{K}^{l,h}}^\top}{\sqrt{d_h}}\Bigr), \qquad \mathbf{V}^{l,h}\in\mathbb{R}^{S\times d_h},$$4 queries,

$$\mathbf{A}^{l,h}=\mathrm{Softmax}\Bigl(\tfrac{\mathbf{Q}^{l,h}{\mathbf{K}^{l,h}}^\top}{\sqrt{d_h}}\Bigr), \qquad \mathbf{V}^{l,h}\in\mathbb{R}^{S\times d_h},$$5

Second, for each token index $$\mathbf{A}^{l,h}=\mathrm{Softmax}\Bigl(\tfrac{\mathbf{Q}^{l,h}{\mathbf{K}^{l,h}}^\top}{\sqrt{d_h}}\Bigr), \qquad \mathbf{V}^{l,h}\in\mathbb{R}^{S\times d_h},$$6 older than $$\mathbf{A}^{l,h}=\mathrm{Softmax}\Bigl(\tfrac{\mathbf{Q}^{l,h}{\mathbf{K}^{l,h}}^\top}{\sqrt{d_h}}\Bigr), \qquad \mathbf{V}^{l,h}\in\mathbb{R}^{S\times d_h},$$7, it computes $$\mathbf{A}^{l,h}=\mathrm{Softmax}\Bigl(\tfrac{\mathbf{Q}^{l,h}{\mathbf{K}^{l,h}}^\top}{\sqrt{d_h}}\Bigr), \qquad \mathbf{V}^{l,h}\in\mathbb{R}^{S\times d_h},$$8 by equation (1), while assigning $$\mathbf{A}^{l,h}=\mathrm{Softmax}\Bigl(\tfrac{\mathbf{Q}^{l,h}{\mathbf{K}^{l,h}}^\top}{\sqrt{d_h}}\Bigr), \qquad \mathbf{V}^{l,h}\in\mathbb{R}^{S\times d_h},$$9 to the last $j$0 positions so that a history window is always kept. Third, it flattens across heads via $j$1 and applies the layer-wise normalization

$j$2

Global Top-$j$3 selection is then performed across all $j$4, and all other entries are evicted from every layer’s $j$5.

The computational overhead is specified explicitly. Scoring each layer/head costs $j$6, flattening and normalization cost $j$7, and global top-$j$8 over $j$9 items costs $p_j^{l,h}=\frac{1}{S}\sum_{i=1}^S A^{l,h}(i,j).$0. Overall, the method adds only one matrix multiply per window plus a single sort of size $p_j^{l,h}=\frac{1}{S}\sum_{i=1}^S A^{l,h}(i,j).$1, which the paper states is small in practice compared to full attention cost. The same discussion also characterizes LaProx as zero-parameter, requiring only one extra mat-mul per window and interoperating with any pretrained Transformer.

The theoretical contrast with head-wise heuristics is explicit. By the matrix-product sampling argument, selecting tokens by $p_j^{l,h}=\frac{1}{S}\sum_{i=1}^S A^{l,h}(i,j).$2 aligns directly with preserving each layer’s output under a budget. Pure attention-weight heuristics ignore $p_j^{l,h}=\frac{1}{S}\sum_{i=1}^S A^{l,h}(i,j).$3 and inter-head mixing via $p_j^{l,h}=\frac{1}{S}\sum_{i=1}^S A^{l,h}(i,j).$4, so they can miss tokens with small attention weights but large projected contributions, or the converse.

4. Empirical behavior in long-context benchmarks and stated limitations

The evaluation in the cache-eviction paper uses LongBench and Needle-In-A-Haystack. LongBench comprises 16 datasets across Single/Multi-doc QA, summarization, few-shot, synthetic, and code completion; Needle-In-A-Haystack includes 1N-1T, 4N-1T, and 4N-4T retrieval tests. The models are Meta-Llama-3.1-8B-Instruct with a 128K window, Mistral-7B-Instruct with 32K, and Qwen3-8B with 32K. Cache budgets are $p_j^{l,h}=\frac{1}{S}\sum_{i=1}^S A^{l,h}(i,j).$5 tokens, with history window $p_j^{l,h}=\frac{1}{S}\sum_{i=1}^S A^{l,h}(i,j).$6. Metrics are F1, Rouge-L, Accuracy, and EditSim on LongBench as appropriate, retrieval accuracy on NIAH, and efficiency measures including peak GPU memory, prefill+eviction overhead, and per-token latency (Mai et al., 8 May 2026).

For Meta-Llama-3.1-8B at 128 tokens on average LongBench performance, the reported scores are: FullKV $p_j^{l,h}=\frac{1}{S}\sum_{i=1}^S A^{l,h}(i,j).$7, SnapKV $p_j^{l,h}=\frac{1}{S}\sum_{i=1}^S A^{l,h}(i,j).$8, AdaKV $p_j^{l,h}=\frac{1}{S}\sum_{i=1}^S A^{l,h}(i,j).$9, CAKE $k$0, CriticalKV $k$1, and LaProx $k$2.

Method	Avg. Score
FullKV	49.51%
SnapKV	42.31%
AdaKV	43.12%
CAKE	42.77%
CriticalKV	42.88%
LaProx	45.19%

Under extreme compression at 128 tokens, the paper states that LaProx reduces accuracy loss by up to $k$3 compared to the best head-wise baselines. On Mistral-7B with a 256-token budget in Needle-In-A-Haystack retrieval, CriticalKV and LaProx are reported as $k$4 versus $k$5 on 1N-1T, $k$6 versus $k$7 on 4N-1T, and $k$8 versus $k$9 on 4N-4T. For efficiency on Llama-3.1-8B with 128 tokens and 128K context, FullKV memory is $p_j^{l,h}$0 and LaProx memory is $p_j^{l,h}$1, corresponding to $p_j^{l,h}$2; prefill overhead is $p_j^{l,h}$3 versus SnapKV $p_j^{l,h}$4; and decode is $p_j^{l,h}$5 faster than FullKV while being on par with SnapKV and AdaKV. The abstract further states that the method maintains model performance with only $p_j^{l,h}$6 of the KV cache across 19 datasets and consistently outperforms prior works across all configurations.

The paper also states limitations and potential extensions. The layer-wise normalization $p_j^{l,h}$7 is described as a simple fix for inter-layer scale disparity and may be improved by learned or adaptive normalizers. Proposed extensions include higher-order matrix-approximation bounds, leverage of head covariance in sampling, joint optimization of window size $p_j^{l,h}$8 per layer or per head, and combination with quantization for further memory savings. A practical claim in the discussion is that cutting the KV cache to $p_j^{l,h}$9 of full size while maintaining accuracy enables inference over 100K-token contexts on a single 80 GB GPU. This suggests that the method is intended not only as a compression heuristic but as a systems-level reformulation of the eviction objective.

5. LaProx as “LAROS Proximal-point” for sequential feature extraction

In the 2011 optimization usage, LaProx is the proximal-point algorithm developed for the LAROS problem, namely finding a “large approximately rank-one submatrix” (Doan et al., 2011). The data matrix $$\mathbf{O}^l=\sum_{h=1}^H \bigl(\mathbf{A}^{l,h}\mathbf{V}^{l,h}\bigr)W_O^{l,h} = \underbrace{\bigl[\mathbf{A}^{l,1}\,\dots\,\mathbf{A}^{l,H}\bigr]}_{A^l}\; \underbrace{\mathrm{blockdiag}\bigl(V^{l,1}W_O^{l,1},\dots,V^{l,H}W_O^{l,H}\bigr)}_{VW_O^l}.$$0 is nonnegative, and the target matrix $$\mathbf{O}^l=\sum_{h=1}^H \bigl(\mathbf{A}^{l,h}\mathbf{V}^{l,h}\bigr)W_O^{l,h} = \underbrace{\bigl[\mathbf{A}^{l,1}\,\dots\,\mathbf{A}^{l,H}\bigr]}_{A^l}\; \underbrace{\mathrm{blockdiag}\bigl(V^{l,1}W_O^{l,1},\dots,V^{l,H}W_O^{l,H}\bigr)}_{VW_O^l}.$$1 should be large in Frobenius norm while also approximately rank one. Doan and Vavasis proposed the convex formulation

$$\mathbf{O}^l=\sum_{h=1}^H \bigl(\mathbf{A}^{l,h}\mathbf{V}^{l,h}\bigr)W_O^{l,h} = \underbrace{\bigl[\mathbf{A}^{l,1}\,\dots\,\mathbf{A}^{l,H}\bigr]}_{A^l}\; \underbrace{\mathrm{blockdiag}\bigl(V^{l,1}W_O^{l,1},\dots,V^{l,H}W_O^{l,H}\bigr)}_{VW_O^l}.$$2

where $$\mathbf{O}^l=\sum_{h=1}^H \bigl(\mathbf{A}^{l,h}\mathbf{V}^{l,h}\bigr)W_O^{l,h} = \underbrace{\bigl[\mathbf{A}^{l,1}\,\dots\,\mathbf{A}^{l,H}\bigr]}_{A^l}\; \underbrace{\mathrm{blockdiag}\bigl(V^{l,1}W_O^{l,1},\dots,V^{l,H}W_O^{l,H}\bigr)}_{VW_O^l}.$$3 is the nuclear norm, $$\mathbf{O}^l=\sum_{h=1}^H \bigl(\mathbf{A}^{l,h}\mathbf{V}^{l,h}\bigr)W_O^{l,h} = \underbrace{\bigl[\mathbf{A}^{l,1}\,\dots\,\mathbf{A}^{l,H}\bigr]}_{A^l}\; \underbrace{\mathrm{blockdiag}\bigl(V^{l,1}W_O^{l,1},\dots,V^{l,H}W_O^{l,H}\bigr)}_{VW_O^l}.$$4 is the entrywise $$\mathbf{O}^l=\sum_{h=1}^H \bigl(\mathbf{A}^{l,h}\mathbf{V}^{l,h}\bigr)W_O^{l,h} = \underbrace{\bigl[\mathbf{A}^{l,1}\,\dots\,\mathbf{A}^{l,H}\bigr]}_{A^l}\; \underbrace{\mathrm{blockdiag}\bigl(V^{l,1}W_O^{l,1},\dots,V^{l,H}W_O^{l,H}\bigr)}_{VW_O^l}.$$5 norm, and $$\mathbf{O}^l=\sum_{h=1}^H \bigl(\mathbf{A}^{l,h}\mathbf{V}^{l,h}\bigr)W_O^{l,h} = \underbrace{\bigl[\mathbf{A}^{l,1}\,\dots\,\mathbf{A}^{l,H}\bigr]}_{A^l}\; \underbrace{\mathrm{blockdiag}\bigl(V^{l,1}W_O^{l,1},\dots,V^{l,H}W_O^{l,H}\bigr)}_{VW_O^l}.$$6 controls the tradeoff between low rank and sparsity. An equivalent redundant splitting writes $$\mathbf{O}^l=\sum_{h=1}^H \bigl(\mathbf{A}^{l,h}\mathbf{V}^{l,h}\bigr)W_O^{l,h} = \underbrace{\bigl[\mathbf{A}^{l,1}\,\dots\,\mathbf{A}^{l,H}\bigr]}_{A^l}\; \underbrace{\mathrm{blockdiag}\bigl(V^{l,1}W_O^{l,1},\dots,V^{l,H}W_O^{l,H}\bigr)}_{VW_O^l}.$$7, $$\mathbf{O}^l=\sum_{h=1}^H \bigl(\mathbf{A}^{l,h}\mathbf{V}^{l,h}\bigr)W_O^{l,h} = \underbrace{\bigl[\mathbf{A}^{l,1}\,\dots\,\mathbf{A}^{l,H}\bigr]}_{A^l}\; \underbrace{\mathrm{blockdiag}\bigl(V^{l,1}W_O^{l,1},\dots,V^{l,H}W_O^{l,H}\bigr)}_{VW_O^l}.$$8 and minimizes

$$\mathbf{O}^l=\sum_{h=1}^H \bigl(\mathbf{A}^{l,h}\mathbf{V}^{l,h}\bigr)W_O^{l,h} = \underbrace{\bigl[\mathbf{A}^{l,1}\,\dots\,\mathbf{A}^{l,H}\bigr]}_{A^l}\; \underbrace{\mathrm{blockdiag}\bigl(V^{l,1}W_O^{l,1},\dots,V^{l,H}W_O^{l,H}\bigr)}_{VW_O^l}.$$9

The proximal-point framework is then applied to $j$0, with

$j$1

and

$j$2

For $j$3, the Moreau–Yoshida regularization is

$j$4

and the proximal map is

$j$5

Optimality is equivalent to the fixed-point condition $j$6.

The evaluation of $j$7 proceeds by dualization. One forms the inner dual function

$j$8

where $j$9, $A^l(VW_O^l)$0, and $A^l(VW_O^l)$1 splits into two adjoint parts $A^l(VW_O^l)$2 and $A^l(VW_O^l)$3. One then solves

$A^l(VW_O^l)$4

and updates with closed-form proximal operators: $A^l(VW_O^l)$5 which soft-thresholds singular values, and

$A^l(VW_O^l)$6

which soft-thresholds entries. The outer proximal-point iteration alternates approximate maximization of $A^l(VW_O^l)$7, primal updates for $A^l(VW_O^l)$8 and $A^l(VW_O^l)$9, and the convergence test

$h$00

A notable aspect of this LaProx variant is its $h$01-optimality stopping criterion via duality. When the true optimal solution is rank one, $h$02 with $h$03, the paper derives exact KKT conditions with dual multipliers $h$04 satisfying block-wise inequalities and the spectral bound $h$05. Proposition 4.2 states that the outer loop can stop once an approximate singular triple $h$06 solves

$h$07

to within $h$08, while feasible $h$09 satisfy tightened bounds

$h$10

through a small projected-subgradient subroutine. When these conditions hold, the support of $h$11 and $h$12 is guaranteed to be exactly that of a rank-one optimal solution.

The sequential feature-extraction algorithm records the extracted feature, zeros out the identified submatrix $h$13, resets $h$14, and continues until no feature remains. On the sailboat bitmap example, $h$15 is $h$16, each $h$17 bitmap column has 3 of 5 possible boat components, and true rank-one submatrices correspond to hull, sails, rudder, and related parts. Dual-PPA plus the $h$18-test extracted 8 sequential features; using the $h$19-test allowed stopping after 10–20 outer iterations instead of 50–60, cutting total CPU by approximately $h$20. On the Frey face database, $h$21 is $h$22, the first ten features were extracted, and negative features were obtained by applying LaProx to $h$23. These experiments situate LaProx as a first-order method for a nuclear-plus-$h$24 model rather than as a general-purpose proximal-gradient routine.

6. LaProx as proximal gradient and broader proximal context

In the 2025 exposition “La Méthode du Gradient Proximé,” LaProx refers to the proximal gradient method for the composite convex problem

$h$25

where $h$26 is convex and continuously differentiable with $h$27-Lipschitz-continuous gradient, and $h$28 is proper, closed, convex, and possibly nonsmooth (Combettes, 18 Mar 2025). The proximal operator is

$h$29

and the basic Forward–Backward update is

$h$30

The text lists assumptions $h$31–$h$32: $h$33 has $h$34-Lipschitz gradient, $h$35 has a computable proximal map, and optionally $h$36 is $h$37-strongly convex. Under $h$38–$h$39, the method is well-defined and converges sublinearly; under $h$40, it converges linearly.

The corresponding convergence results are explicit. With fixed $h$41,

$h$42

and under $h$43-strong convexity,

$h$44

The same source presents FISTA, backtracking step sizes, block-coordinate proximal gradient, and proximal Newton or variable-metric variants. Applications listed include sparse linear regression, logistic-$h$45 classification, TV-regularized image denoising, and matrix completion. In that usage, “LaProx” is a generic algorithmic framework rather than a task-specific method.

A broader proximal context is given by the U-Lagrangian for prox-regular functions. Liu, Eberhard, and Luo define prox-regularity through the lower quadratic model

$h$46

near $h$47, construct the UV-decomposition

$h$48

and define the U-Lagrangian

$h$49

They prove prox-regularity, strict differentiability at $h$50, and strong metric regularity properties for $h$51 under stated conditions (Liu et al., 2015). Although that construction is not itself named LaProx in the source, it clarifies the proximal and variational background against which the other LaProx usages sit.

Taken together, these usages show that LaProx is best understood as a polysemous label rather than a single method. In long-context LLM inference it denotes an output-aware global eviction strategy; in sequential feature extraction it denotes a proximal-point solver for a nuclear-plus-$h$52 convex program; and in composite optimization it serves as shorthand for the proximal gradient method. The shared theme is the use of proximal or approximation principles to control complexity while preserving the structure most relevant to the target objective.