---
title: 'MARC: Memory-Augmented RL Token Compression'
url: https://www.emergentmind.com/topics/memory-augmented-reinforcement-learning-based-token-compression-marc
type: topic
---

# MARC: Memory-Augmented RL Token Compression

Searching arXiv for the specified MARC-related papers and closely related work.
Memory-Augmented Reinforcement Learning-based Token Compression (MARC) denotes a class of architectures in which an agent maintains an external memory, compresses that memory into a more compact representation, and optimizes the resulting task–efficiency trade-off with reinforcement learning. The term is the name of a specific video-understanding method in "MARC: Memory-Augmented RL Token Compression for Efficient Video Understanding" [2510.07915], and later work explicitly recasts related systems as MARC-style designs for agent-history rendering, skill distillation, and long-context compressed-memory reasoning [2601.04786], [2602.08234], [2602.08382]. Across these formulations, the recurring objective is to preserve reasoning utility while reducing context length, visual tokens, memory usage, or inference cost.

## 1. Conceptual scope and problem setting

The motivating bottlenecks differ by modality but are structurally similar. In agentic LLM systems trained with reinforcement learning over multi-turn interaction trajectories, practical deployment is bottlenecked by rapidly growing textual histories that inflate token budgets and memory usage [2601.04786]. In memory-based LLM agents, raw trajectories are often redundant and noise-heavy, preventing extraction of high-level, reusable behavioral patterns [2602.08234]. In long-context reasoning, the dominant issues include quadratic computational costs, information forgetting, and the context fragmentation inherent in retrieval-augmented generation [2602.08382]. In video understanding, visual language models incur heavy computational costs when extended from images to videos due to high frame rates and long durations [2510.07915].

Within this shared setting, MARC is not a single fixed mechanism. In the cited works, it appears as optical rendering of dialogue history into an image, distillation of trajectories into compact natural-language skills, chunk-wise compression into learned memory tokens, and retrieve-then-compress video processing. The common design principle is that the policy no longer consumes the full raw history directly; instead, it acts over a compressed memory object that is explicitly optimized for downstream reward.

This suggests that MARC is best understood as a systems-level paradigm rather than a single algorithmic primitive.

## 2. Canonical architectural pattern

A MARC system typically separates three functions: persistent memory construction, memory compression or retrieval, and RL-based policy optimization under an explicit context budget or efficiency signal. The specific memory substrate differs across implementations.

| Instantiation | Memory object | Compression or retrieval mechanism |
|---|---|---|
| AgentOCR | External buffer $\mathcal{M}_t=\{(o_1,a_1),\dots,(o_t,a_t)\}$ | Deterministic renderer $\mathcal{R}$, segment optical caching, adaptive compression factor $c_t$ |
| SkillRL | Hierarchical SkillBank $\mathcal{M}=\{S_g \cup \bigcup_{k=1}^{K} S_k\}$ | Experience-based skill distillation and TopK retrieval |
| LycheeMemory | Compressed memory bank $\Theta=\{\theta_1,\dots,\theta_K\}$ | Chunk-wise learned compressor plus gating module |
| Video MARC | Retrieved visual memory fragments | Visual Memory Retriever and memory-aware temporal compression |

In AgentOCR, the state representation replaces raw text $h_t$ with an external memory buffer and renders it via a deterministic renderer into an image $I_t=\mathcal{R}(h_t;\psi)$; the model ingests $(\text{instruction }\mathcal{I}, \text{ image } I_t)$ [2601.04786]. In SkillRL, the agent is parameterized as $\pi_\theta(a_t \mid o_{\le t}, d, c)$, where $c$ is an external context formed from general skills $S_g$ and retrieved task-specific skills $S_{\text{ret}}$ [2602.08234]. In LycheeMemory, an input document $D$ is split into non-overlapping chunks, each chunk is compressed into a memory block $\theta_k$, and a gating module decides which blocks should update the evolving working memory $m_t$ [2602.08382]. In the video MARC pipeline, event-based video segmentation yields visual memory fragments, a retriever returns the top-$k$ clips most relevant to the query, and compression is applied only after retrieval [2510.07915].

A plausible implication is that MARC replaces monolithic context processing with a factorized pipeline in which memory density and policy quality can be optimized separately, then coupled through reward.

## 3. Reinforcement-learning formulations

RL is central to MARC because compression is not treated as a purely offline preprocessing step. In AgentOCR, baseline agentic RL uses Group Relative Policy Optimization (GRPO) with the clipped surrogate
$$
J(\theta)=\mathbb{E}_\tau\left[\frac{1}{GT}\sum_{i=1}^{G}\sum_{t=1}^{T}\min(\rho_{t,i}\hat A_i,\operatorname{clip}(\rho_{t,i},1\pm\epsilon)\hat A_i)\right],
$$
where $\rho_{t,i}=\pi_\theta(a_{t,i}\mid\cdot)/\pi_{\theta_{\text{old}}}(\cdot)$ and $\hat A_i$ is the group-normalized advantage. AgentOCR augments the environment reward $r_t$ with a compression term $r_t^{\mathrm{comp}}$ to form
$$
\tilde r_t = r_t + \lambda \cdot r_t^{\mathrm{comp}}\cdot \mathbf{1}[t \bmod K = 0],
$$
and for binary-success tasks
$$
r_t^{\mathrm{comp}}=
\begin{cases}
\ln(c_t) & \text{if episode succeeded at its end},\\
0 & \text{otherwise}.
\end{cases}
$$
The compression reward is injected intermittently to avoid trivial high-compression strategies [2601.04786].

In SkillRL, the formal objective is
$$
J(\theta)=\mathbb{E}_{\tau\sim\pi_\theta}\left[\sum_{t=0}^{T}\gamma^t r_t\right]
$$
subject to $|c|\le L_{\max}$ tokens. After cold-start SFT, GRPO-based RL is performed with a KL term to preserve skill-use capabilities:
$$
J(\theta)=\mathbb{E}_{d,\{\tau^i\}}\left[\frac{1}{G}\sum_{i=1}^{G}\min(\rho_iA_i,\operatorname{clip}(\rho_i,1-\epsilon,1+\epsilon)A_i)-\beta D_{\mathrm{KL}}[\pi_\theta(\cdot\mid d,\mathcal{M})\parallel \pi_{\mathrm{ref}}(\cdot\mid d,\mathcal{M})]\right],
$$
where $\pi_{\mathrm{ref}}=\pi_{\theta_{\mathrm{sft}}}$ [2602.08234].

In LycheeMemory, the RL problem is defined over states $s_t\equiv(m_t,\Theta,Q)$, actions that retrieve or skip memory blocks, and a terminal reward $R(Q,A)=1$ if the generated answer exactly matches ground truth within `<answer>` tags and $0$ otherwise. The joint policy
$$
\pi_\theta(A,\mathcal{M},\Theta\mid D,Q)=\prod_{k=1}^{K}\pi_{\mathrm{comp}}(\theta_k\mid C_k)\cdot\prod_{t=1}^{T}\pi_{\mathrm{reason}}(m_t\mid m_{t-1},\Theta,Q)
$$
is optimized with a PPO-style clipped surrogate with a KL penalty (GSPO) [2602.08382].

In video MARC, the student policy $\pi_\theta$ is trained with Compression Group Relative Policy Optimization (C-GRPO). The retention ratio is
$$
\eta = a_{\mathrm{comp}}/a_{\mathrm{full}},
$$
the compression bonus is
$$
r_c=\alpha\cdot\max(0,\eta-\tau),
$$
and the total reward is
$$
R_i = r_i + \mathbf{1}[\mathrm{correct}]\cdot r_c.
$$
The optimization objective is a PPO-style clipped loss with a KL anchor to a reference policy $\pi_{\mathrm{ref}}$ [2510.07915].

These formulations show that MARC does not simply compress tokens; it trains a policy to decide what compressed memory should contain, how strongly compression should be incentivized, and how far policy updates may drift from a reference behavior.

## 4. Memory representations and compression mechanisms

The memory object in MARC can be textual, visual, latent, or hybrid. AgentOCR uses optical self-compression: instead of feeding a large text history directly, it renders the accumulated observation–action history as a compact image. The history $h_t$ is split into text segments $\ell_{t,1},\dots,\ell_{t,K_t}$, each segment receives a content key
$$
k(\ell)=\operatorname{Hash}(\operatorname{normalize}(\ell),\text{style-metadata}),
$$
and an episode-specific visual cache
$$
\mathcal{C}^{(e)}=\{(k(\ell),I(\ell))\}
$$
stores rendered segment images. The final memory image is assembled by vertical stacking,
$$
I_t=\operatorname{Stack}_{i=1\dots K_t}(I(\ell_{t,i})).
$$
The amortized per-step render cost is $O(U_t)$ where $U_t$ is the number of new cache-miss segments, and cache size grows with the number of unique segments rather than the number of steps. The wrapper may then downsample by factor $\sqrt{c_t}$ to produce the next observation image [2601.04786].

SkillRL compresses memory by abstraction rather than rendering. A trajectory $\tau$ is converted by a teacher memory module $\mathcal{M}_T$ into a concise skill $s$, with successful and failed trajectories producing $s^+$ and $s^-$. The resulting skills are short natural-language “principle + when_to_apply” entries. For a batch $\mathcal{T}$, the compression ratio is
$$
CR=\frac{\sum_{\tau\in\mathcal{T}}|\tau|}{\sum_{\tau\in\mathcal{T}}|s(\tau)|},
$$
and the summary states that typically $CR \approx 10$–$20\times$ in SkillRL. At inference, the agent forms
$$
c=(S_g\cup S_{\mathrm{ret}}),\qquad
S_{\mathrm{ret}}=\operatorname{TopK}(\{s\in S_k:\operatorname{sim}(e_d,e_s)\ge\delta\},K),
$$
which restricts memory to a bounded set of high-density reusable skills [2602.08234].

LycheeMemory uses learned latent compression. Each chunk $C_k$ of size $sz$ is interleaved with $z_k=sz/\alpha$ trainable memory tokens $\{\langle v\rangle_j\}$; a single forward pass through the compressor yields hidden states $h(\langle v\rangle_j)$, and
$$
\theta_k=\{h(\langle v\rangle_1),\dots,h(\langle v\rangle_{z_k})\}
$$
is stored as the compressed memory block. A separate gate computes
$$
P_i=\sigma(W_{\mathrm{gate}}\cdot h_{\mathrm{last}})
$$
from $[Q;m_t;\theta_i]$ and marks a block relevant if $P_i>\tau$ [2602.08382].

The original video MARC uses a retrieve-then-compress pipeline. Event-based segmentation yields semantically coherent clips. A contrastive retriever embeds query and fragments into a shared latent space and returns the top-$k$ clips, with $k$ set to $3$ in most experiments. From the retrieved clips, the method uniformly samples $N$ frames in total at $1$ fps and encodes them into
$$
H=\{h_1,\dots,h_T\}\in\mathbb{R}^{T\times d}.
$$
Within short-term windows $S_j$, it repeatedly merges the two consecutive frames with highest average patch cosine similarity,
$$
\operatorname{sim}(H_a,H_b)=\frac{1}{P}\sum_{p=1}^{P}\frac{h_a^{(p)}\cdot h_b^{(p)}}{\|h_a^{(p)}\|\|h_b^{(p)}\|},
$$
replacing them by
$$
H_{\mathrm{merge}}=\frac{1}{2}(H_a+H_b),
$$
until only
$$
n_j=\max(1,\lfloor(1-\rho)\cdot|S_j|\rfloor)
$$
frames remain [2510.07915].

Taken together, these mechanisms indicate that MARC is defined less by a particular codec than by a common principle: memory is retained externally, compressed into a denser form, and exposed to the policy through a learned or structured access pathway.

## 5. Empirical behavior and efficiency trade-offs

The reported empirical results emphasize trade-offs between performance retention and context efficiency rather than absolute minimization of tokens. On ALFWorld, the 7B text baseline with GRPO achieves $81.8\%$ success with $0.95\,\mathrm{k}$ tokens per step, while AgentOCR obtains $81.2\%$ ($99.3\%$ retention) with $0.43\,\mathrm{k}$ tokens per step ($54.7\%\downarrow$). On search-based QA, the text baseline scores $41.9\%$ Exact Match with $0.73\,\mathrm{k}$ tokens per step, while AgentOCR scores $40.1\%$ ($95.7\%$ retention) with $0.36\,\mathrm{k}$ tokens per step ($50.7\%\downarrow$). Segment optical caching yields $\sim 20\times$ faster per-step rendering compared to re-rendering full history, and $\sim 27\%$ less cache memory versus naive append-only caching [2601.04786].

SkillRL reports stronger gains on success-oriented agent benchmarks. On ALFWorld, the GRPO baseline is $77.6\%$, memory-augmented RL baselines are $62.5$–$75.5\%$, and SkillRL reaches $89.9\%$ overall success rate, corresponding to $+12.3\%$ over GRPO and $+14$–$27\%$ over other memory-RL methods. The context-efficiency trade-off in Figure 5 is summarized as $\mathbb{E}[F_{\mathrm{raw}}]\approx 1{,}450$ tokens versus $\mathbb{E}[F_{\mathrm{skill}}]<1{,}300$ tokens, implying $CR_{\mathrm{query}}\approx 1.10\times$ ($\sim 10\%$ saving) while improving success from $\sim 60\%\rightarrow\sim 90\%$. On seven search-augmented QA tasks, the reported average scores are $39.1\%$ for ZeroSearch, $43.1\%$ for EvolveR, and $47.1\%$ for SkillRL [2602.08234].

LycheeMemory focuses on extreme long-context extrapolation. LycheeMemory-7B without Gate achieves up to $82.03\%$ at $7\,\mathrm{K}$ context and degrades mildly to $75.78\%$ at $1.75\,\mathrm{M}$. The full model with Gate trades $3$–$6\%$ accuracy for $6\times$ inference speedup and $2\times$ lower peak GPU memory versus MemAgent. Additional ablations report that $\alpha\in\{2,4\}$ is nearly lossless, while $\alpha=16$ degrades sharply at long lengths; Query+Memory Gate retains $\approx 86\%$ recall at $112\,\mathrm{K}$ versus $82\%$ for query-only and $74\%$ for text-embedding; and end-to-end RL achieves $68.75$ Avg., outperforming RL with frozen compressor, SFT only, and naive chunking [2602.08382].

The original video MARC reports a more aggressive compression regime. On MMVU MC, original $64$-frame baseline average tokens are $2\,589.9$, whereas MARC uses $122.7$ tokens, corresponding to a $95\%$ reduction. GPU memory decreases from $41.63\,\mathrm{GB}$ to $11.48\,\mathrm{GB}$ ($-72.4\%$), LLM generation latency decreases by $23.9\%$, and end-to-end latency decreases by $11.1\%$. On the six-benchmark mean, Qwen2.5-VL-3B with $64$ frames scores $42.21$, while MARC-3B with VMR + C-GRPO and one frame scores $42.20$. Ablations further report that Qwen2.5-VL-3B + VMR with $64$ frames reaches $45.56$, and that C-GRPO outperforms SFT-based one-frame settings: $42.20$ for MARC-3B versus $40.16$ for Qwen2.5-VL-3B (1 fr) + SFT + VMR and $38.50$ for Qwen2.5-VL-3B (1 fr) + SFT. For the threshold $\tau$ in C-GRPO, the reported means are $41.34$ at $\tau=0.4$, $42.20$ at $\tau=0.6$, and $41.40$ at $\tau=0.8$ [2510.07915].

These results do not support the common assumption that token compression must be equivalent to crude truncation. In the cited systems, substantial reductions in token or memory footprint are paired with controlled degradation, near-baseline retention, or outright improvement, depending on the task and the compression substrate.

## 6. Limitations, misconceptions, and open directions

Several limitations recur across MARC variants. AgentOCR explicitly states that compression reward is injected only intermittently to avoid trivial high-compression strategies, indicating that naive reward shaping can collapse toward degenerate compression policies [2601.04786]. SkillRL’s design addresses the problem that raw trajectories are redundant and noise-heavy, but its recursive evolution mechanism activates only for categories with validation accuracy below a threshold, which implies continued dependence on failure-driven skill-bank updates [2602.08234].

LycheeMemory provides the most explicit failure taxonomy. Its limitations section reports Unidirectional Dependency Mismatch ($35\%$), where once skipped a block cannot be revisited if it becomes relevant later; Premature Anchoring ($21\%$), where early inference biases subsequent updates; and Compression-Induced Hallucination ($17\%$), where aggressive $\alpha$ causes feature collapse leading to attribute swapping. The paper lists backtracking or multi-pass recall, Just-in-Time compression and offloading, and hybrid integration with RAG as future extensions [2602.08382].

Video MARC identifies a separate boundary condition. Under extremely long videos (VideoMME), performance drops to $74\%$ of baseline when using only $3\%$ of tokens, showing the limits of $1$ fps sampling on very long inputs. The paper also notes that VMR currently relies on a single-shot retrieval of $k$ clips, that the contrastive retriever assumes static embeddings, and that jointly fine-tuning VMR with C-GRPO may further close the gap [2510.07915].

A frequent misconception is that MARC denotes one fixed memory format. The literature cited here shows instead that MARC can operate over rendered images, distilled natural-language skills, latent memory-token blocks, or retrieved video fragments. Another misconception is that RL is always applied uniformly to every module. In LycheeMemory, the gating module is trained separately as a classifier; in SkillRL, cold-start SFT precedes GRPO; in AgentOCR, no auxiliary supervised losses are needed beyond the primary RL objective with augmented reward. This suggests that MARC is better characterized by the coupling of memory augmentation, compression, and reward-driven optimization than by any single training recipe.

The broader significance of MARC lies in this modularity. The cited systems show that token compression can be implemented through optical rendering, selective abstraction, learned latent compression, or retrieval-conditioned frame merging, while still remaining compatible with end-to-end RL or RL-centered training. That combination of explicit external memory and task-aware compression is the defining feature of the MARC paradigm.

Source: https://www.emergentmind.com/topics/memory-augmented-reinforcement-learning-based-token-compression-marc