MetaClaw-Bench Continual Adaptation Benchmark

• MetaClaw-Bench is a benchmark for assessing continual adaptation in LLM agents using real-world CLI workflows that combine file execution and procedural reasoning.
• It employs a multi-day, two-part evaluation regime designed to mimic evolving user requirements through both file-check tasks and multi-choice procedural challenges.
• Empirical results show that integrating prompt-level skill evolution with RL fine-tuning significantly boosts performance metrics like accuracy and file-check completion.

MetaClaw-Bench is a non-stationary continual adaptation benchmark designed to evaluate LLM agents on their ability to evolve and meta-learn in the context of real-world command-line interface (CLI) workflows. It provides a structured, multi-day task stream with both file-execution and procedural reasoning components, enabling rigorous separation and measurement of declarative skill acquisition and policy learning contributions. MetaClaw-Bench is tightly coupled with the MetaClaw continual meta-learning framework, supporting online agent evaluation in production-like conditions with zero downtime and principled handling of data versioning (Xia et al., 17 Mar 2026).

1. Definition and Structure

MetaClaw-Bench consists of 934 evaluation queries delivered as a simulated sequence of 44 “workdays,” partitioned into two main parts to probe distinct adaptation regimes:

• Part I (30 workdays, 346 questions):
  • Tasks: File-check (verified code or file edits) and multi-choice procedural reasoning.
  • Persistent state: Within-day file system and configuration are maintained; iterative corrective feedback is provided.
  • Difficulty schedule: Starts with basic CLI operations (days 1–10), transitions to multi-step workflows (11–22), culminating in complex, multi-agent scenarios (23–30).
• Part II (14 workdays, 588 questions):
  • Tasks: 434 multi-choice, 154 file-check.
  • Emphasis: Rule-based file and configuration transformations governed by five incrementally introduced implicit “preference arcs” (P₁–P₅), e.g., ISO 8601+TZ temporal formatting and backup-before-modify requirements.
  • Temporal density: 42 questions per day, concentrating on the agent’s ability to internalize procedural rules.

This sequential, non-episodic formulation reflects authentic operational environments where agents face evolving user requirements and sustained context (Xia et al., 17 Mar 2026).

2. Benchmark Design Principles

MetaClaw-Bench is underpinned by three principal objectives:

1. Continual Meta-Learning: Each query $\tau$ is drawn from a drifting task distribution $p_t(\tau)$. The agent must adapt not only to isolated tasks but improve its meta-adaptation machinery consistent with

$$\max_{\theta, \mathcal{S}}\, \mathbb{E}_{\tau\sim p_t} \left[ R\left(\pi_\theta(\cdot|\tau, \operatorname{Retrieve}(\mathcal{S}, \tau))\right) \right].$$

This encourages the development of reusable skills ($\mathcal{S}$) and policy parameters ($\theta$), linking short- and long-term adaptation.

1. Multi-Modal Evaluation: Separate file-check tasks (objective file execution and transformation) from multi-choice procedural tasks (domain-specific reasoning), enabling disambiguation of gains from prompt-level skill injection versus model weight adaptation.
2. Support–Query Versioning and Separation: To preclude “reward contamination” after skill library ($\mathcal{S}_g$) updates, all trajectories are version-stamped by skill generation index.
   • Support data $D^{\mathrm{sup}\,g}$: Failure trajectories under $S_g$ triggering skill synthesis.
   • Query data $D^{\mathrm{qry}\,g+1}$: Post-skill-update trajectories, reserved for RL-based fine-tuning.
   • On each evolution $g \rightarrow g+1$, samples with version $p_t(\tau)$0 are purged from buffers such that only fresh, post-adaptation rewards inform policy gradients.

Task taxonomy is structured by both execution type and preference arc. Preference arcs (P₁–P₅) represent implicit operational constraints injected progressively, such as formatting, backup, and naming conventions.

3. Evaluation Protocol

Agents are evaluated without offline pretraining, using only online interactions in one of three conditions:

• Baseline: A static policy $p_t(\tau)$1, no adaptation.
• MetaClaw (Skills): Only prompt-level, gradient-free skill evolution via “Skill Evolver”—an LLM-driven module that distills 1–k instructions from recent failure batches and immediately injects them into system prompts.
• MetaClaw (Full): Skill evolution plus weight-level adaptation through cloud-based LoRA reinforcement learning (RL) guided by a Process Reward Model (PRM). RL fine-tuning occurs only during agent inactivity, as scheduled by the Opportunistic Meta-Learning Scheduler (OMLS).

For RL, updates are triggered only when the query buffer contains at least $p_t(\tau)$2 samples, to control gradient variance. The policy gradient update is:

$p_t(\tau)$3

where $p_t(\tau)$4 is the PRM score over query trajectories.

Primary metrics:

• Accuracy: $p_t(\tau)$5
• Completion (file-check): $p_t(\tau)$6

Secondary metric:

• Composite Robustness: $p_t(\tau)$7

4. Empirical Results

Experiments on MetaClaw-Bench with GPT-5.2 and Kimi-K2.5 LLM backbones demonstrate distinct adaptation effects.

Model	Condition	Part I Acc. (%)	Part I Comp. (%)	Part II Acc. (%)	Part II Comp. (%)
GPT-5.2	Baseline	41.1	14.7	44.9	58.4
	MetaClaw (Skills)	44.0 (+7.1%)	17.1 (+16.3%)	49.1 (+9.4%)	67.5 (+15.6%)
Kimi-K2.5	Baseline	21.4	2.0	21.1	18.2
	MetaClaw (Skills)	28.3 (+32.2%)	2.0	26.9 (+27.5%)	33.8 (+85.7%)
	MetaClaw (Full)	40.6 (+89.7%)	16.5 (×8.25)	39.6 (+87.2%)	51.9 (+185%)

*Skill-only adaptation produces marked gains in multi-choice query accuracy and modest file-check improvements within rule-friendly Part II. Full MetaClaw activates robust end-to-end file execution, with up to 8.25× completion improvement for Kimi-K2.5 in Part I. This suggests that prompt-level instruction injection alone is insufficient for complex file-execution reliability.

5. Ablation Studies, Skill Dynamics, and Failure Analysis

Per-day rolling accuracy analyses reveal a two-phase adaptation regime: initial accuracy acceleration (mid-difficulty days, 11–22) via skill injection, followed by delayed policy-driven improvement as RL-based LoRA updates internalize procedural rules (especially evident in Part II, post-day 8). Task-type ablations establish that skill evolver-driven prompt injections dominate multi-choice performance gains, while RL fine-tuning is essential for file-execution robustness.

The Skill Evolver typically distills 25 unique instructions over 30 days, clustered into temporal formatting normalization, backup-before-modify protocols, and naming convention rules. Each new skill confers cross-task transfer benefits; for instance, “backup-before-modify” prevents diverse JSON schema transformation failures.

6. Broader Applicability and Deployment Considerations

MetaClaw-Bench’s paradigm extends beyond CLI evaluation. In AutoResearchClaw—a 23-stage research pipeline—skill-only adaptation achieves a 24.8% reduction in stage retry rate, 40% decrease in cycle count, and 18.3% improvement in composite robustness, supporting the claim that zero-downtime, prompt-level skill injection generalizes beyond CLI workloads.

Deployment considerations include:

• Zero downtime: skill distillation is synchronous with failure detection at prompt-level.
• Data validity: strict support–query version management ensures RL updates are based on uncontaminated post-adaptation data.
• Opportunistic RL scheduling: improvement occurs only during idle windows (user-inactive per OMLS), eliminating service disruption.

7. Significance and Impact

MetaClaw-Bench operationalizes a multi-task, multi-day testbed that decomposes and quantifies the roles of gradient-free (prompt-based) and gradient-based (weight-adaptive) learning in LLM agents tailored for continual, production-grade deployment. Through its support-query versioning, multi-modal streams, and non-episodic task evolution, it provides a principled basis for evaluating lifelong LLM adaptation (Xia et al., 17 Mar 2026).