Behavior Knowledge Merge in Reinforced Agentic Models

Abstract: Reinforcement learning (RL) is central to post-training, particularly for agentic models that require specialized reasoning behaviors. In this setting, model merging offers a practical mechanism for integrating multiple RL-trained agents from different tasks into a single generalist model. However, existing merging methods are designed for supervised fine-tuning (SFT), and they are suboptimal to preserve task-specific capabilities on RL-trained agentic models. The root is a task-vector mismatch between RL and SFT: on-policy RL induces task vectors that are highly sparse and heterogeneous, whereas SFT-style merging implicitly assumes dense and globally comparable task vectors. When standard global averaging is applied under this mismatch, RL's non-overlapping task vectors that encode critical task-specific behaviors are reduced and parameter updates are diluted. To address this issue, we propose Reinforced Agent Merging (RAM), a distribution-aware merging framework explicitly designed for RL-trained agentic models. RAM disentangles shared and task-specific unique parameter updates, averaging shared components while selectively preserving and rescaling unique ones to counteract parameter update dilution. Experiments across multiple agent domains and model architectures demonstrate that RAM not only surpasses merging baselines, but also unlocks synergistic potential among agents to achieve performance superior to that of specialized agents in their domains.

• The paper presents Reinforced Agent Merging (RAM), a novel algorithm that preserves unique, sparse task vectors from RL-trained agents during integration.
• It details a selective merging mechanism using task vector distribution analysis to maintain and even enhance domain-specific performance.
• Experimental results show that RAM and its amplified variant, RAM+, outperform traditional merging methods across coding, tool-use, and memory tasks.

Behavior Knowledge Merge in Reinforced Agentic Models: An Expert Analysis

Introduction

The study presented in "Behavior Knowledge Merge in Reinforced Agentic Models" (2601.13572) addresses a pivotal challenge in the construction of generalist agentic models: integrating domain-specialized agents trained via reinforcement learning (RL) while preserving task-specific capabilities. Classical model merging techniques, designed for supervised fine-tuning (SFT), fail to capitalize on the heterogeneous, sparse task vector structure induced by RL, leading to degradation of specialized behavior through signal dilution. The authors introduce Reinforced Agent Merging (RAM)—a distribution-aware merging algorithm tailored for RL-trained agents—which leverages structural task vector analysis to preserve and amplify both shared and unique task-specific updates. This work delivers substantive advances in multi-agent integration, demonstrating robust synergy and superior performance across a variety of domains and architectures.

Task Vector Behavior in RL Agentic Models

RL post-training for agentic models produces task vectors that differ fundamentally from SFT-derived updates. The study systematically characterizes these vectors, revealing pronounced sparsity and heterogeneity in update distributions. This heterogeneity emerges as agents fine-tuned for distinct domains (coding, tool-use, memory extension) modify substantially different—often disjoint—subnetworks. The manipulated parameters in RL are not only sparser but also less redundantly distributed compared to SFT, resulting in regionally non-overlapping task vectors. This behavior is verified empirically:

• Coding agents modify only 3.2% of model parameters, while tool-use and memory agents modify 46.2% and 54.3% respectively.
• The overlap analysis identifies ~47% unique non-zero elements for Memory agents, and only 6.3% for Coding.

  Figure 1: The density (inverse sparsity) and overlap statistics highlight pronounced heterogeneity between RL agent task vectors.

Unique regions of task vectors, largely non-overlapping across agents, encode critical domain-specific behaviors. The authors demonstrate that merging only these regions retains and often improves corresponding in-domain performance, with negligible interference across other domains. Standard global averaging—used by SFT-based merging—dilutes these non-overlapping updates, precipitating pronounced drops in domain performance.

Figure 2: Performance gains attributed to merging unique regions of reinforced task vectors, emphasizing cross-domain non-interference.

Reinforced Agent Merging (RAM): Algorithmic Formulation

RAM disentangles shared and unique regions of RL task vectors to implement distribution-aware merging. Key stages include:

1. Probing Task Vector Distributions: Binary masks are computed for each vector, segmenting parameters into shared (modified by ≥2 agents) and unique (modified by only one).
2. Overlap-Unique Ratio and Rescaling: For each agent, the overlap-unique ratio $\rho_t$ is measured. RAM then computes a scaling factor $\lambda_t$ proportional to $\rho_t$, compensating for the contraction induced by averaging in shared regions.
3. Selective Merging: The final merged vector sets unchanged parameters to zero, averages shared regions, and amplifies unique regions according to $\lambda_t$.

This approach guarantees functional preservation of each agent’s specialization while stabilizing consensus in shared update regions.

Figure 3: Schematic for RAM—task vector distribution tracking, region categorization, and selective merging/rescaling pipeline.

Experimental Evaluation

Experiments span three domains (coding, tool-use, long-context memory) and two architectures (Qwen2.5-7B-Instruct, Llama3.2-3B-Instruction). RAM and its amplified variant, RAM+, consistently outperform baselines (Task Arithmetic, Fisher, TIES, DARE). RAM+ not only preserves but surpasses original specialized agent performance on most tasks—demonstrating cross-domain synergy unlocked via RL knowledge merge.

Figure 4: RAM/RAM+ establish new SOTA results on 9 out of 12 tasks, exceeding original agentic experts and all merging baselines.

Key numerical results:

• In coding, RAM+ achieves higher pass accuracy on LiveBench versus the dedicated Coding agent.
• In tool-use, RAM+ significantly outperforms the Tool agent on challenging parallel tasks.
• In long-context memory, RAM+ attains top score (82.03 on SQuAD 64K) over the dedicated Memory agent.

Pairwise agent merging experiments confirm RAM’s robustness and generality; regardless of task combinations, RAM/RAM+ deliver superior average performance.

Figure 5: Cross-domain merging analysis for pairs of RL-trained agents, with RAM/RAM+ outperforming all baselines.

Extension to Llama3.2-3B agents in search, tool-use, and math further affirms architecture-agnostic effectiveness, with RAM/RAM+ yielding positive synergy and stability.

Figure 6: Merging results for RL agents on LLama3.2-3B-Instruction show RAM’s adaptability across architectures.

Distributional analysis on additional agentic models corroborates that RL-induced task vector sparsity and heterogeneity persist across domains and model families.

Figure 7: Expanded distribution analysis for Qwen2.5-7B-instruct agents demonstrates consistent reinforced task vector heterogeneity.

Figure 8: Llama3.2-3B-based agent distribution: sparsity and overlap statistics extend RAM’s behavioral insights architecture-wide.

Efficiency benchmarking clarifies that RAM achieves a superior trade-off between merging latency and agentic scores, outperforming computationally intensive baselines.

Figure 9: RAM/RAM+ deliver the highest score-to-merging-time ratio, occupying the Pareto frontier relative to all SOTA baselines.

Implications and Future Directions

This work advances practical and theoretical understanding of adapting model merging to RL agentic settings:

• Practical Impact: RAM offers a viable, efficient framework for integrating specialized RL agents into unified generalists. Its utility spans production systems (e.g., UI-TARS2) where joint RL training is infeasible and storage constraints necessitate merging.
• Theoretical Insights: The study robustly characterizes the structural diversity of RL task vectors and formalizes the limitations of uniform merging, proposing a distributionally aware compensation mechanism.
• Scalability: Preliminary results generalize across architectures (Qwen, Llama), but future work should address merging scalability as agent numbers grow, including parameter collision and overlap-resolution strategies.
• Fine-Grained Adaptivity: While current rescaling relies on isotropic approximations, element-wise sensitivity or curvature-aware amplification could unlock finer control, albeit at increased computational cost.
• Applicability to Larger Models: Most experiments target 3B/7B models; verifying RAM’s hypotheses and effectiveness at 70B+ parameter scale remains an open challenge.

Conclusion

RAM presents a principled, empirically validated solution to the challenge of merging RL-trained agentic models. By explicitly addressing the sparse, heterogeneous nature of RL-induced task vectors, and compensating for domain-specific knowledge dilution, RAM and its enhanced variants demonstrate clear superiority over traditional SFT-based merging approaches. This establishes RAM as a robust foundation for constructing generalist multi-domain agents from specialized RL experts.