MARC: Memory-Augmented RL Token Compression

• MARC is a systems-level paradigm that uses reinforcement learning to compress external memory representations, balancing efficiency and task performance.
• It integrates methods like optical rendering, skill distillation, and learned latent compression to reduce token costs while maintaining reasoning quality.
• Empirical results show that MARC achieves significant token reduction with near-baseline performance across video understanding, long-context reasoning, and agent interaction tasks.

Searching arXiv for the specified MARC-related papers and closely related work. Memory-Augmented Reinforcement Learning-based Token Compression (MARC) denotes a class of architectures in which an agent maintains an external memory, compresses that memory into a more compact representation, and optimizes the resulting task–efficiency trade-off with reinforcement learning. The term is the name of a specific video-understanding method in "MARC: Memory-Augmented RL Token Compression for Efficient Video Understanding" (Wu et al., 9 Oct 2025), and later work explicitly recasts related systems as MARC-style designs for agent-history rendering, skill distillation, and long-context compressed-memory reasoning (Feng et al., 8 Jan 2026, Xia et al., 9 Feb 2026, Chen et al., 9 Feb 2026). Across these formulations, the recurring objective is to preserve reasoning utility while reducing context length, visual tokens, memory usage, or inference cost.

1. Conceptual scope and problem setting

The motivating bottlenecks differ by modality but are structurally similar. In agentic LLM systems trained with reinforcement learning over multi-turn interaction trajectories, practical deployment is bottlenecked by rapidly growing textual histories that inflate token budgets and memory usage (Feng et al., 8 Jan 2026). In memory-based LLM agents, raw trajectories are often redundant and noise-heavy, preventing extraction of high-level, reusable behavioral patterns (Xia et al., 9 Feb 2026). In long-context reasoning, the dominant issues include quadratic computational costs, information forgetting, and the context fragmentation inherent in retrieval-augmented generation (Chen et al., 9 Feb 2026). In video understanding, visual LLMs incur heavy computational costs when extended from images to videos due to high frame rates and long durations (Wu et al., 9 Oct 2025).

Within this shared setting, MARC is not a single fixed mechanism. In the cited works, it appears as optical rendering of dialogue history into an image, distillation of trajectories into compact natural-language skills, chunk-wise compression into learned memory tokens, and retrieve-then-compress video processing. The common design principle is that the policy no longer consumes the full raw history directly; instead, it acts over a compressed memory object that is explicitly optimized for downstream reward.

This suggests that MARC is best understood as a systems-level paradigm rather than a single algorithmic primitive.

2. Canonical architectural pattern

A MARC system typically separates three functions: persistent memory construction, memory compression or retrieval, and RL-based policy optimization under an explicit context budget or efficiency signal. The specific memory substrate differs across implementations.

Instantiation	Memory object	Compression or retrieval mechanism
AgentOCR	External buffer $\mathcal{M}_t=\{(o_1,a_1),\dots,(o_t,a_t)\}$	Deterministic renderer $\mathcal{R}$, segment optical caching, adaptive compression factor $c_t$
SkillRL	Hierarchical SkillBank $\mathcal{M}=\{S_g \cup \bigcup_{k=1}^{K} S_k\}$	Experience-based skill distillation and TopK retrieval
LycheeMemory	Compressed memory bank $\Theta=\{\theta_1,\dots,\theta_K\}$	Chunk-wise learned compressor plus gating module
Video MARC	Retrieved visual memory fragments	Visual Memory Retriever and memory-aware temporal compression

In AgentOCR, the state representation replaces raw text $h_t$ with an external memory buffer and renders it via a deterministic renderer into an image $I_t=\mathcal{R}(h_t;\psi)$; the model ingests $$(\text{instruction }\mathcal{I}, \text{ image } I_t)$$ (Feng et al., 8 Jan 2026). In SkillRL, the agent is parameterized as $\pi_\theta(a_t \mid o_{\le t}, d, c)$, where $c$ is an external context formed from general skills $\mathcal{R}$0 and retrieved task-specific skills $\mathcal{R}$1 (Xia et al., 9 Feb 2026). In LycheeMemory, an input document $\mathcal{R}$2 is split into non-overlapping chunks, each chunk is compressed into a memory block $\mathcal{R}$3, and a gating module decides which blocks should update the evolving working memory $\mathcal{R}$4 (Chen et al., 9 Feb 2026). In the video MARC pipeline, event-based video segmentation yields visual memory fragments, a retriever returns the top-$\mathcal{R}$5 clips most relevant to the query, and compression is applied only after retrieval (Wu et al., 9 Oct 2025).

A plausible implication is that MARC replaces monolithic context processing with a factorized pipeline in which memory density and policy quality can be optimized separately, then coupled through reward.

3. Reinforcement-learning formulations

RL is central to MARC because compression is not treated as a purely offline preprocessing step. In AgentOCR, baseline agentic RL uses Group Relative Policy Optimization (GRPO) with the clipped surrogate

$\mathcal{R}$6

where $\mathcal{R}$7 and $\mathcal{R}$8 is the group-normalized advantage. AgentOCR augments the environment reward $\mathcal{R}$9 with a compression term $c_t$0 to form

$c_t$1

and for binary-success tasks

$c_t$2

The compression reward is injected intermittently to avoid trivial high-compression strategies (Feng et al., 8 Jan 2026).

In SkillRL, the formal objective is

$c_t$3

subject to $c_t$4 tokens. After cold-start SFT, GRPO-based RL is performed with a KL term to preserve skill-use capabilities:

$c_t$5

where $c_t$6 (Xia et al., 9 Feb 2026).

In LycheeMemory, the RL problem is defined over states $c_t$7, actions that retrieve or skip memory blocks, and a terminal reward $c_t$8 if the generated answer exactly matches ground truth within <answer> tags and $c_t$9 otherwise. The joint policy

$\mathcal{M}=\{S_g \cup \bigcup_{k=1}^{K} S_k\}$0

is optimized with a PPO-style clipped surrogate with a KL penalty (GSPO) (Chen et al., 9 Feb 2026).

In video MARC, the student policy $\mathcal{M}=\{S_g \cup \bigcup_{k=1}^{K} S_k\}$1 is trained with Compression Group Relative Policy Optimization (C-GRPO). The retention ratio is

$\mathcal{M}=\{S_g \cup \bigcup_{k=1}^{K} S_k\}$2

the compression bonus is

$\mathcal{M}=\{S_g \cup \bigcup_{k=1}^{K} S_k\}$3

and the total reward is

$\mathcal{M}=\{S_g \cup \bigcup_{k=1}^{K} S_k\}$4

The optimization objective is a PPO-style clipped loss with a KL anchor to a reference policy $\mathcal{M}=\{S_g \cup \bigcup_{k=1}^{K} S_k\}$5 (Wu et al., 9 Oct 2025).

These formulations show that MARC does not simply compress tokens; it trains a policy to decide what compressed memory should contain, how strongly compression should be incentivized, and how far policy updates may drift from a reference behavior.

4. Memory representations and compression mechanisms

The memory object in MARC can be textual, visual, latent, or hybrid. AgentOCR uses optical self-compression: instead of feeding a large text history directly, it renders the accumulated observation–action history as a compact image. The history $\mathcal{M}=\{S_g \cup \bigcup_{k=1}^{K} S_k\}$6 is split into text segments $\mathcal{M}=\{S_g \cup \bigcup_{k=1}^{K} S_k\}$7, each segment receives a content key

$\mathcal{M}=\{S_g \cup \bigcup_{k=1}^{K} S_k\}$8

and an episode-specific visual cache

$\mathcal{M}=\{S_g \cup \bigcup_{k=1}^{K} S_k\}$9

stores rendered segment images. The final memory image is assembled by vertical stacking,

$\Theta=\{\theta_1,\dots,\theta_K\}$0

The amortized per-step render cost is $\Theta=\{\theta_1,\dots,\theta_K\}$1 where $\Theta=\{\theta_1,\dots,\theta_K\}$2 is the number of new cache-miss segments, and cache size grows with the number of unique segments rather than the number of steps. The wrapper may then downsample by factor $\Theta=\{\theta_1,\dots,\theta_K\}$3 to produce the next observation image (Feng et al., 8 Jan 2026).

SkillRL compresses memory by abstraction rather than rendering. A trajectory $\Theta=\{\theta_1,\dots,\theta_K\}$4 is converted by a teacher memory module $\Theta=\{\theta_1,\dots,\theta_K\}$5 into a concise skill $\Theta=\{\theta_1,\dots,\theta_K\}$6, with successful and failed trajectories producing $\Theta=\{\theta_1,\dots,\theta_K\}$7 and $\Theta=\{\theta_1,\dots,\theta_K\}$8. The resulting skills are short natural-language “principle + when_to_apply” entries. For a batch $\Theta=\{\theta_1,\dots,\theta_K\}$9, the compression ratio is

$h_t$0

and the summary states that typically $h_t$1–$h_t$2 in SkillRL. At inference, the agent forms

$h_t$3

which restricts memory to a bounded set of high-density reusable skills (Xia et al., 9 Feb 2026).

LycheeMemory uses learned latent compression. Each chunk $h_t$4 of size $h_t$5 is interleaved with $h_t$6 trainable memory tokens $h_t$7; a single forward pass through the compressor yields hidden states $h_t$8, and

$h_t$9

is stored as the compressed memory block. A separate gate computes

$I_t=\mathcal{R}(h_t;\psi)$0

from $I_t=\mathcal{R}(h_t;\psi)$1 and marks a block relevant if $I_t=\mathcal{R}(h_t;\psi)$2 (Chen et al., 9 Feb 2026).

The original video MARC uses a retrieve-then-compress pipeline. Event-based segmentation yields semantically coherent clips. A contrastive retriever embeds query and fragments into a shared latent space and returns the top-$I_t=\mathcal{R}(h_t;\psi)$3 clips, with $I_t=\mathcal{R}(h_t;\psi)$4 set to $I_t=\mathcal{R}(h_t;\psi)$5 in most experiments. From the retrieved clips, the method uniformly samples $I_t=\mathcal{R}(h_t;\psi)$6 frames in total at $I_t=\mathcal{R}(h_t;\psi)$7 fps and encodes them into

$I_t=\mathcal{R}(h_t;\psi)$8

Within short-term windows $I_t=\mathcal{R}(h_t;\psi)$9, it repeatedly merges the two consecutive frames with highest average patch cosine similarity,

$$(\text{instruction }\mathcal{I}, \text{ image } I_t)$$0

replacing them by

$$(\text{instruction }\mathcal{I}, \text{ image } I_t)$$1

until only

$$(\text{instruction }\mathcal{I}, \text{ image } I_t)$$2

frames remain (Wu et al., 9 Oct 2025).

Taken together, these mechanisms indicate that MARC is defined less by a particular codec than by a common principle: memory is retained externally, compressed into a denser form, and exposed to the policy through a learned or structured access pathway.

5. Empirical behavior and efficiency trade-offs

The reported empirical results emphasize trade-offs between performance retention and context efficiency rather than absolute minimization of tokens. On ALFWorld, the 7B text baseline with GRPO achieves $$(\text{instruction }\mathcal{I}, \text{ image } I_t)$$3 success with $$(\text{instruction }\mathcal{I}, \text{ image } I_t)$$4 tokens per step, while AgentOCR obtains $$(\text{instruction }\mathcal{I}, \text{ image } I_t)$$5 ($$(\text{instruction }\mathcal{I}, \text{ image } I_t)$$6 retention) with $$(\text{instruction }\mathcal{I}, \text{ image } I_t)$$7 tokens per step ($$(\text{instruction }\mathcal{I}, \text{ image } I_t)$$8). On search-based QA, the text baseline scores $$(\text{instruction }\mathcal{I}, \text{ image } I_t)$$9 Exact Match with $\pi_\theta(a_t \mid o_{\le t}, d, c)$0 tokens per step, while AgentOCR scores $\pi_\theta(a_t \mid o_{\le t}, d, c)$1 ($\pi_\theta(a_t \mid o_{\le t}, d, c)$2 retention) with $\pi_\theta(a_t \mid o_{\le t}, d, c)$3 tokens per step ($\pi_\theta(a_t \mid o_{\le t}, d, c)$4). Segment optical caching yields $\pi_\theta(a_t \mid o_{\le t}, d, c)$5 faster per-step rendering compared to re-rendering full history, and $\pi_\theta(a_t \mid o_{\le t}, d, c)$6 less cache memory versus naive append-only caching (Feng et al., 8 Jan 2026).

SkillRL reports stronger gains on success-oriented agent benchmarks. On ALFWorld, the GRPO baseline is $\pi_\theta(a_t \mid o_{\le t}, d, c)$7, memory-augmented RL baselines are $\pi_\theta(a_t \mid o_{\le t}, d, c)$8–$\pi_\theta(a_t \mid o_{\le t}, d, c)$9, and SkillRL reaches $c$0 overall success rate, corresponding to $c$1 over GRPO and $c$2–$c$3 over other memory-RL methods. The context-efficiency trade-off in Figure 1 is summarized as $c$4 tokens versus $c$5 tokens, implying $c$6 ($c$7 saving) while improving success from $c$8. On seven search-augmented QA tasks, the reported average scores are $c$9 for ZeroSearch, $\mathcal{R}$00 for EvolveR, and $\mathcal{R}$01 for SkillRL (Xia et al., 9 Feb 2026).

LycheeMemory focuses on extreme long-context extrapolation. LycheeMemory-7B without Gate achieves up to $\mathcal{R}$02 at $\mathcal{R}$03 context and degrades mildly to $\mathcal{R}$04 at $\mathcal{R}$05. The full model with Gate trades $\mathcal{R}$06–$\mathcal{R}$07 accuracy for $\mathcal{R}$08 inference speedup and $\mathcal{R}$09 lower peak GPU memory versus MemAgent. Additional ablations report that $\mathcal{R}$10 is nearly lossless, while $\mathcal{R}$11 degrades sharply at long lengths; Query+Memory Gate retains $\mathcal{R}$12 recall at $\mathcal{R}$13 versus $\mathcal{R}$14 for query-only and $\mathcal{R}$15 for text-embedding; and end-to-end RL achieves $\mathcal{R}$16 Avg., outperforming RL with frozen compressor, SFT only, and naive chunking (Chen et al., 9 Feb 2026).

The original video MARC reports a more aggressive compression regime. On MMVU MC, original $\mathcal{R}$17-frame baseline average tokens are $\mathcal{R}$18, whereas MARC uses $\mathcal{R}$19 tokens, corresponding to a $\mathcal{R}$20 reduction. GPU memory decreases from $\mathcal{R}$21 to $\mathcal{R}$22 ($\mathcal{R}$23), LLM generation latency decreases by $\mathcal{R}$24, and end-to-end latency decreases by $\mathcal{R}$25. On the six-benchmark mean, Qwen2.5-VL-3B with $\mathcal{R}$26 frames scores $\mathcal{R}$27, while MARC-3B with VMR + C-GRPO and one frame scores $\mathcal{R}$28. Ablations further report that Qwen2.5-VL-3B + VMR with $\mathcal{R}$29 frames reaches $\mathcal{R}$30, and that C-GRPO outperforms SFT-based one-frame settings: $\mathcal{R}$31 for MARC-3B versus $\mathcal{R}$32 for Qwen2.5-VL-3B (1 fr) + SFT + VMR and $\mathcal{R}$33 for Qwen2.5-VL-3B (1 fr) + SFT. For the threshold $\mathcal{R}$34 in C-GRPO, the reported means are $\mathcal{R}$35 at $\mathcal{R}$36, $\mathcal{R}$37 at $\mathcal{R}$38, and $\mathcal{R}$39 at $\mathcal{R}$40 (Wu et al., 9 Oct 2025).

These results do not support the common assumption that token compression must be equivalent to crude truncation. In the cited systems, substantial reductions in token or memory footprint are paired with controlled degradation, near-baseline retention, or outright improvement, depending on the task and the compression substrate.

6. Limitations, misconceptions, and open directions

Several limitations recur across MARC variants. AgentOCR explicitly states that compression reward is injected only intermittently to avoid trivial high-compression strategies, indicating that naive reward shaping can collapse toward degenerate compression policies (Feng et al., 8 Jan 2026). SkillRL’s design addresses the problem that raw trajectories are redundant and noise-heavy, but its recursive evolution mechanism activates only for categories with validation accuracy below a threshold, which implies continued dependence on failure-driven skill-bank updates (Xia et al., 9 Feb 2026).

LycheeMemory provides the most explicit failure taxonomy. Its limitations section reports Unidirectional Dependency Mismatch ($\mathcal{R}$41), where once skipped a block cannot be revisited if it becomes relevant later; Premature Anchoring ($\mathcal{R}$42), where early inference biases subsequent updates; and Compression-Induced Hallucination ($\mathcal{R}$43), where aggressive $\mathcal{R}$44 causes feature collapse leading to attribute swapping. The paper lists backtracking or multi-pass recall, Just-in-Time compression and offloading, and hybrid integration with RAG as future extensions (Chen et al., 9 Feb 2026).

Video MARC identifies a separate boundary condition. Under extremely long videos (VideoMME), performance drops to $\mathcal{R}$45 of baseline when using only $\mathcal{R}$46 of tokens, showing the limits of $\mathcal{R}$47 fps sampling on very long inputs. The paper also notes that VMR currently relies on a single-shot retrieval of $\mathcal{R}$48 clips, that the contrastive retriever assumes static embeddings, and that jointly fine-tuning VMR with C-GRPO may further close the gap (Wu et al., 9 Oct 2025).

A frequent misconception is that MARC denotes one fixed memory format. The literature cited here shows instead that MARC can operate over rendered images, distilled natural-language skills, latent memory-token blocks, or retrieved video fragments. Another misconception is that RL is always applied uniformly to every module. In LycheeMemory, the gating module is trained separately as a classifier; in SkillRL, cold-start SFT precedes GRPO; in AgentOCR, no auxiliary supervised losses are needed beyond the primary RL objective with augmented reward. This suggests that MARC is better characterized by the coupling of memory augmentation, compression, and reward-driven optimization than by any single training recipe.

The broader significance of MARC lies in this modularity. The cited systems show that token compression can be implemented through optical rendering, selective abstraction, learned latent compression, or retrieval-conditioned frame merging, while still remaining compatible with end-to-end RL or RL-centered training. That combination of explicit external memory and task-aware compression is the defining feature of the MARC paradigm.