Connect the Dots: Training LLMs for Long-Lifecycle Agents with Cross-Domain Generalization Via Reinforcement Learning

Abstract: This work presents a general framework for training LLMs to "Connect the Dots" (CoD), a meta-capability required by long-lifecycle agents: as an LLM-based AI agent gets deployed in an environment, it solves a long sequence of tasks while continuously exploring the environment, learning from its own experiences, and iteratively self-updating its context about the environment, thereby achieving progressively better performance on future tasks conditioned on the updated context. Major components of the CoD framework include: (1) algorithm design and infrastructure for end-to-end reinforcement learning (RL) with long rollout sequences interleaving solve-task and update-context episodes; (2) tasks and environments for incentivizing and eliciting the targeted meta-capability in LLMs during training, as well as for faithfully measuring progress during evaluation. We present proof-of-concept implementations of the CoD framework, including a GRPO-style RL algorithm with fine-grained credit assignment, as well as tasks and environments tailored to the targeted meta-capability (rather than domain-specific LLM capabilities or standard task-by-task RL). Empirical results validate the efficacy of end-to-end RL training in the CoD setting, and demonstrate the potential for out-of-distribution generalization -- within the training domains, across different domains, and from CoD to Ralph-loop settings -- of the elicited meta-capability. Our investigation of CoD connects several lines of prior works, and opens up new opportunities for advancing LLMs and AI agents. To facilitate further research and applications, we release our implementations at \url{https://github.com/agentscope-ai/Trinity-RFT/tree/research/cod/examples/research_cod}.

• The paper introduces the CoD framework, integrating CoD-Deploy and CoD-Train to enable long-lifecycle meta-capabilities in LLMs.
• The paper leverages dynamic programming and GRPO-style policy optimization for fine-grained credit assignment over extended, multi-turn interactions.
• The paper demonstrates significant performance improvements and robust cross-domain generalization, paving the way for scalable, adaptable LLM agents.

Training LLMs for Long-Lifecycle Agent Meta-Capabilities: The CoD Framework

Motivation and Framework Formulation

The paper "Connect the Dots: Training LLMs for Long-Lifecycle Agents with Cross-Domain Generalization Via Reinforcement Learning" (2606.20002) addresses the critical gap in current LLM-based agent systems: the lack of systematic training for context-driven, long-lifecycle meta-capabilities. The "Connect the Dots" (CoD) meta-capability is formalized as the agent’s ability to iteratively update its context based on accumulated experiences across multiple tasks in the same environment, facilitating efficient adaptation and self-improvement over extended sequences. Standard task-by-task RL methods are insufficient for this, as they fail to incentivize the transfer and exploitation of environment-specific knowledge between tasks.

The CoD framework consists of two integrated components:

• CoD-Deploy: An abstraction representing agentic deployment in an environment. The agent alternates between task-solving episodes and context-updating episodes, maintaining and leveraging environment-specific context across long sequences.
• CoD-Train: A corresponding RL post-training paradigm with rollout trajectories that match CoD-Deploy, spanning diverse environments for robust generalization.

CoD-Train increases the hierarchy level compared to conventional task-level RL, aligning with agentic, meta-learning principles and enabling the acquisition of flexible context-management policies.

Figure 1: Visualization of CoD-Deploy and CoD-Train; blocks represent solve-task and update-context episodes within environments A, B, or the new environment M.

Algorithmic Design and Credit Assignment

A major challenge in this regime is fine-grained credit assignment over long state-action sequences that interleave task-solving and context-updating episodes, each comprising multi-turn agent-environment interactions and large token spaces. The proposed solution leverages dynamic programming to maximize both immediate and future rewards, reflecting RL principles for sequential decision processes.

The CoD-specific RL algorithm adopts a GRPO-style policy optimization with episodic outcome rewards (for solve-task) and small format rewards (for update-context), constructing returns as averages of current and future episodic rewards. Advantage baselines are computed positionally within rollout trajectories, and a heuristic re-weighting mechanism is introduced to stabilize training when negative gradient imbalances emerge. This fine-grained approach circumvents limits in previous RL$^2$-derived methods where anchor-state identification or coarse-grained outcome rewards hinder effective training in environments with higher complexity.

Figure 2: Visualization of advantage calculation in the CoD-Train RL algorithm, enabling fine-grained credit assignment for long multi-episode trajectories.

Environment and Task Design

Empirical elicitation of CoD requires specialized environments that mandate context transfer for optimal performance. Three domains are instantiated:

• FrozenLake-Obscure: Variant of FrozenLake where the mapping of action symbols to movement directions is randomized and unknown, enforcing information-theoretic limits on task success when solved independently. Agents are incentivized to deduce and encode these mappings in context across tasks.
• Alchemy-Random: Synthesizing target elements in an environment with randomly initialized recipes. Effective task-solving requires accumulating, transferring, and exploiting discovered element combinations as context.
• TerminalSimulator: Realistic terminal manipulation tasks where cross-task environmental characteristics (e.g., OS type) are critical for optimal command selection.

The minimalistic implementation treats context as a text "hint" appended to subsequent prompts, with future work likely to extend this to persistent memory or skill banks.

Empirical Results and Generalization

The experiments utilize Qwen3-8B-Instruct as the base model and compare CoD-Train performance in two settings: (A) FrozenLake-Obscure only, and (B) mixture of FrozenLake-Obscure and Alchemy-Random.

• Strong numerical results: In setting A, the agent’s mean reward for solving the initial task rises from 0.18 to 0.45, but for the fourth task in a sequence (leveraging updated context), reward increases from 0.28 to 0.76. This demonstrates substantial leverage of context, far beyond information-theoretic limits imposed by solving tasks from scratch.
• Generalization: Models trained on CoD-Train not only improve in-domain performance (with harder environments and longer task sequences), but also out-of-domain, including new unseen domains (TerminalSimulator) and Ralph-loop scenarios (repeated attempts on the same task).

Figure 3: Empirical results for CoD-Train on FrozenLake-Obscure; later tasks in sequences show stronger performance gains via accumulated context.

Figure 4: Empirical results for CoD-Train on mixed domains; cross-domain and OOD evaluation curves validate generalization of the elicited meta-capability.

Figure 5: Comparison between RL algorithms; the fine-grained, adaptively re-weighted method offers clear stability advantages.

Numerical stability is observed to be sensitive to algorithmic details, with naive GRPO and REC variants incurring unstable optimization especially in Alchemy-Random, whereas the adaptively re-weighted variant achieves robust convergence.

Related Work and Differentiation

CoD connects and differentiates itself from several existing streams:

• Lifelong Agents: Prior art either optimizes context or model weights separately, with current agent systems reliant on scaffolds and hand-crafted memory management. CoD unifies context- and weight-update training directly via RL.
• Meta-RL and RL$^2$: Classical meta-RL relies on fixed-size RNN state, limiting expressivity. CoD leverages LLMs' adaptive compute and context sizing, coupled with generalization opportunities across diverse domains.
• LLM Inference Scaling: CoD generalizes inference scaling (e.g., Ralph-loop) by training agents to update context not only within repeated solution attempts but across genuinely new tasks, facilitating broader deployment scenarios.

Implications and Future Directions

The theoretical implication is that explicit RL-driven context management and transfer substantially improves agentic efficiency and adaptability, especially in underspecified and continually evolving environments. Practically, CoD-trained LLMs could eliminate the need for brittle hand-coded scaffolds, paving the way for robust, scalable agent deployment in real-world settings.

Potential future research directions include:

• Scaling environment and task diversity, validating performance in non-stationary regimes or with significantly longer task sequences.
• More sophisticated context mechanisms (beyond text hints), such as persistent memory objects or structured skill banks.
• Integration of CoD-Train as a meta-capability stage in mainstream LLM post-training pipelines, possibly via teacher model merging or on-policy distillation, yielding models proficient in both domain-specific and agentic meta-skills.
• Formal RL algorithmic advances for further improved credit assignment and stability in long-lifecycle training.

Conclusion

This work provides a formal framework for eliciting CoD meta-capabilities in LLMs through dedicated, fine-grained RL post-training and context-driven rollout regimes. Strong empirical evidence demonstrates superior performance and generalization capabilities, establishing a foundation for robust long-lifecycle agentic deployment. The CoD approach bridges gaps in current agent systems, offers theoretical insights into meta-capability acquisition, and opens avenues for future advances in agentic LLM training and deployment.