GrandCode: Agentic RL in Competitive Programming

• GrandCode is a multi-agent reinforcement learning system using modular agents to solve competitive programming tasks in live contest environments.
• It integrates hypothesis generation, summarization, and adversarial test generation with asynchronous RL to optimize code generation under strict time and memory constraints.
• The system employs agentic GRPO to efficiently assign rewards and manage off-policy drift, achieving top performance against human grandmasters.

GrandCode is a multi-agent reinforcement learning system for competitive programming introduced in "GrandCode: Achieving Grandmaster Level in Competitive Programming via Agentic Reinforcement Learning" (Team et al., 3 Apr 2026). It is designed for live contest settings in which code must be generated, verified, and submitted under strict time and memory limits, hidden tests, and scoring rules that reward early accepted submissions. The system combines learned agentic modules for hypothesis proposal, solving, summarization, and adversarial test generation with post-training, asynchronous online RL, and test-time adaptation. In the reported live evaluation, GrandCode placed first in three Codeforces rounds—1087, 1088, and 1089—beating all human participants, including legendary grandmasters (Team et al., 3 Apr 2026).

1. Problem domain and design rationale

GrandCode addresses competitive programming as a setting in which correctness, efficiency, and latency are jointly binding constraints. The paper identifies several factors that make the domain difficult for AI: participants must solve multiple problems quickly under strict time and memory limits; code must be both correct and efficient; feedback is limited to signals such as “Wrong answer” and “Time limit exceeded”; high-quality adversarial tests are hidden; and hard problems require long reasoning chains, repeated self-debugging, execution, and verification cycles (Team et al., 3 Apr 2026).

The work situates itself against prior AI systems that were strong but still limited under live conditions. AlphaCode and AlphaCode2 achieved mid-to-high percentile ratings; OpenAI’s o3 ranks 175th globally; Gemini 3 Deep Think achieved 8th place, but only on historical problems rather than live contests (Team et al., 3 Apr 2026). GrandCode’s central claim is that live contest constraints exacerbate the standard difficulties of agentic RL: delayed rewards arise because quality is only known after execution and hidden-test approximation, off-policy drift becomes severe in asynchronous pipelines, and long-context inference and verification substantially increase cost.

The paper also emphasizes operational constraints external to modeling. Codeforces prohibits AI-generated content, and high-ranking accounts may face manual scrutiny. To avoid undue attention, GrandCode defers joint account submissions, denoted $S(\text{joint})$, until near the end, while tracking per-problem $S(\text{separate})$ scores. The reported argument is that the penalty from multiple submissions is small, with at most four attempts per task (Team et al., 3 Apr 2026). A common misconception is therefore that GrandCode is only a stronger code generator; the reported system is instead explicitly designed around contest rules, verification uncertainty, and submission strategy.

2. Multi-agent architecture and orchestration

GrandCode orchestrates three learned policies and a test generator. The main solver $\pi_{\mathrm{main}}$ is a large MoE policy that produces reasoning traces and code, with long-context support via pipelined context parallelism for hybrid DeltaNet and softmax attention blocks. The hypothesis model $\pi_{\mathrm{hypothesis}}$ proposes structural conjectures, invariants, or compact characterizations and validates them on random instances or brute-force solvers. The summarization model $\pi_{\mathrm{summary}}$ compacts long reasoning traces into progressive summaries $s_t$ that preserve essential information for downstream synthesis and RL. Test-case generation constructs adversarial tests through difference-driven generation, solution attacks, and large-size stress tests using available solvers (Team et al., 3 Apr 2026).

Interaction among these modules is controlled by a difficulty-based router. A learned classifier assigns each task a difficulty level $d \in \{1,\dots,5\}$. Easier tasks are sent to direct generation; harder tasks invoke hypothesis generation, summarization, and test-time RL. In the hypothesis loop, $\pi_{\mathrm{hypothesis}}$ emits hypotheses $h$, small-instance checks validate them, and counterexamples trigger revision until consistency is achieved. Verified hypotheses are then injected into the prompt for $\pi_{\mathrm{main}}$. For long traces $S(\text{separate})$0, progressive summarization chunks the trace into $S(\text{separate})$1 and updates a summary state through transitions $S(\text{separate})$2; the resulting $S(\text{separate})$3 is fed back to the solver (Team et al., 3 Apr 2026).

The controller manages asynchronous RL rollouts with per-token behavior-policy versions, pipelines sampling and training, and applies staleness weighting and token dropping to limit off-policy drift. It also maintains per-problem memory: validated hypotheses, summaries, test pools, solution history with scores, and a global summary of exploration status. This design makes GrandCode closer to a coordinated multi-stage decision system than to a single-pass LLM. The architecture’s significance lies in the explicit separation of structural conjecture formation, long-horizon memory compression, candidate synthesis, and adversarial verification.

3. Reinforcement learning formulation and Agentic GRPO

GrandCode models agentic coding as an MDP, or as a POMDP when hidden tests and feedback are partial. The state $S(\text{separate})$4 includes the problem $S(\text{separate})$5, difficulty $S(\text{separate})$6, the current prompt, validated hypotheses $S(\text{separate})$7 when available, a partial reasoning trace, the summary state $S(\text{separate})$8 used for long contexts, the current candidate code $S(\text{separate})$9, and the available test pools with their outcomes. Actions are textual outputs from $\pi_{\mathrm{main}}$0, $\pi_{\mathrm{main}}$1, or $\pi_{\mathrm{main}}$2, depending on the active stage. Transitions concatenate the agent’s output, optionally update summaries, execute code in the sandbox, and append verification outcomes to context (Team et al., 3 Apr 2026).

Rewards are decomposed into immediate rewards $\pi_{\mathrm{main}}$3 and a terminal reward $\pi_{\mathrm{main}}$4. Immediate rewards are available at intermediate stages, such as hypothesis verification pass rates, partial code correctness checks, adversarial-test differences, or efficiency gains. The terminal reward is the final rollout score after full verification and efficiency evaluation. For a candidate program $\pi_{\mathrm{main}}$5, the paper defines executability $\pi_{\mathrm{main}}$6, correctness $\pi_{\mathrm{main}}$7, and a per-test efficiency score

$\pi_{\mathrm{main}}$8

The final code reward is

$\pi_{\mathrm{main}}$9

The paper also imposes a length penalty tied to difficulty:

$\pi_{\mathrm{hypothesis}}$0

GrandCode primarily uses group-normalized relative advantages rather than an explicit discount factor $\pi_{\mathrm{hypothesis}}$1; delayed credit is handled by Agentic GRPO (Team et al., 3 Apr 2026).

Agentic GRPO extends Group Relative Policy Optimization to multi-stage rollouts with delayed rewards and severe off-policy drift. Each stage has two phases. In the Immediate Reward phase, as soon as $\pi_{\mathrm{hypothesis}}$2 is available, GrandCode computes group-normalized advantages and updates the policy with a clipped surrogate objective. In the Delayed Correction phase, once the final reward is known, it computes a correction $\pi_{\mathrm{hypothesis}}$3, normalizes it within the group, and applies another clipped update to earlier stages. The intended effect is to combine timely credit assignment with a later correction for future contribution (Team et al., 3 Apr 2026).

To manage asynchronous training, GrandCode applies staleness weights based on token age $\pi_{\mathrm{hypothesis}}$4:

$\pi_{\mathrm{hypothesis}}$5

Immediate rewards are therefore applied early, while delayed corrections may be dropped if they arrive too late. The paper presents this as a throughput–bias trade-off characteristic of long, asynchronous agentic RL pipelines.

4. Training pipeline, infrastructure, and test-time adaptation

GrandCode’s post-training pipeline begins with continued pretraining on TACO, LeetCode, USACO, CodeContests, IOI, crawled problems, and reasoning traces generated by Gemini and Claude. The training mix includes 20% hypothesis-conditioned examples so that $\pi_{\mathrm{hypothesis}}$6 learns to consume outputs from $\pi_{\mathrm{hypothesis}}$7. The paper states that the upper bound accept rate is tied to teacher models, approximately $\pi_{\mathrm{hypothesis}}$8 on the 100-problem benchmark (Team et al., 3 Apr 2026).

Supervised fine-tuning uses different curation rules for easy and hard problems. For easy problems, reasoning traces are retained when the resulting solution is equivalent to or as efficient as the gold solution. For hard problems, traces are selected to maximize a score that combines the normalized log-likelihood of $\pi_{\mathrm{hypothesis}}$9 under $\pi_{\mathrm{summary}}$0 and the normalized log-likelihood of $\pi_{\mathrm{summary}}$1 conditioned on $\pi_{\mathrm{summary}}$2 under the initial solver, which is intended to avoid degenerate answer leakage and prioritize predictive thinking traces. The summarization model is trained in three stages: local transitions $\pi_{\mathrm{summary}}$3 with GRPO, chain training of full summaries using terminal answer likelihood as reward, and joint RL integration with $\pi_{\mathrm{summary}}$4; summary data are also mixed into SFT (Team et al., 3 Apr 2026).

Joint RL then optimizes $\pi_{\mathrm{summary}}$5, $\pi_{\mathrm{summary}}$6, and $\pi_{\mathrm{summary}}$7. The reward for the hypothesis model combines local verification pass rate with downstream gains relative to a no-hypothesis condition. Infrastructure is correspondingly specialized: PipelineRL is used for long sequences; $\pi_{\mathrm{summary}}$8 runs on a distributed GPU mesh with expert parallelism and context parallelism; $\pi_{\mathrm{summary}}$9 and $s_t$0 run on smaller GPU pools; and a CPU sandbox executes compilation and tests. To stabilize MoE behavior, the router is frozen during RL and only FFN experts are updated (Team et al., 3 Apr 2026).

At test time, GrandCode shifts the objective from expected reward to best-of-$s_t$1 final reward. It uses a smoothed rank-based objective

$s_t$2

with $s_t$3 annealed from uniform averaging toward top-ranked emphasis. Adaptation is implemented through LoRA updates on $s_t$4 conditioned on problem-specific history, while $s_t$5 and $s_t$6 are fixed to conserve latency (Team et al., 3 Apr 2026). This design indicates that online improvement is localized to the main solver and is driven by contest-specific evidence rather than by retraining all modules.

5. Verification strategy and empirical performance

Adversarial test generation is a central part of GrandCode’s reported gains. Difference-driven tests prompt LLMs including Claude, GPT, DeepSeek, and Kimi to propose corner-case inputs without seeing code, then retain inputs that induce output differences among candidate solutions. Solution attacks use the gold solution during training to analyze differences between a candidate and the gold and then propose inputs likely to expose bugs; these attacks are refined with validation, and Qwen-3.5-27B is fine-tuned to generate adversarial tests conditioned on problem and solution. Large-size tests use faster candidate programs to produce reference outputs where brute force times out. On 50 real Codeforces problems evaluated by the real judge, the base suite passed 42/50 problems, difference-driven generation plus solution attack raised this to 48/50, and submission feedback plus continued online generation reached 50/50 (Team et al., 3 Apr 2026).

The live Codeforces evaluation is the paper’s most visible empirical result. GrandCode participated in Round 1087 (Mar 21, 2026), Round 1088 (Mar 28, 2026), and Round 1089 (Mar 29, 2026). It ranked first in all three rounds, solved all tasks in each contest, and was the first to finish all tasks in each. The reported finish times were 00:51:11, 01:40:35, and 00:56:43, respectively. The separate-submission scores were 9269, 16511, and 11596, while the deferred joint-account scores were 8334, 15008, and 9506 (Team et al., 3 Apr 2026).

Offline benchmarks show a staged improvement from post-training, joint RL, and test-time RL. On a 100-problem benchmark with five difficulty levels, frontier baselines are reported at approximately $s_t$7 accept for Gemini 3.1 Pro, $s_t$8 for Claude Opus 4.6, $s_t$9 for GPT-5.4, and $d \in \{1,\dots,5\}$0 for Qwen 3.5-397B base. For Qwen 3.5-397B, continued training reaches $d \in \{1,\dots,5\}$1, SFT $d \in \{1,\dots,5\}$2, and the addition of summary $d \in \{1,\dots,5\}$3. Full RL training raises accept rate to $d \in \{1,\dots,5\}$4 with 13/20 solved at Level 5 and weighted score $d \in \{1,\dots,5\}$5, while test-time RL further raises the result to $d \in \{1,\dots,5\}$6, 15/20 at Level 5, and weighted score $d \in \{1,\dots,5\}$7 (Team et al., 3 Apr 2026).

The paper’s ablations attribute the gains to multiple interacting components rather than to a single intervention. Summarization can slightly reduce offline scores if not carefully integrated, but it enables tractable RL and faster inference on very long trajectories. Adversarial test generation materially improves pre-submission reliability. Hypothesis generation improves downstream solving when trained jointly. Agentic GRPO’s immediate and delayed updates improve credit assignment relative to standard terminal-reward GRPO, while staleness-weighted clipping reduces off-policy drift in pipeline RL. The hypothesis model itself improves from pass@1 $d \in \{1,\dots,5\}$8 and pass@5 $d \in \{1,\dots,5\}$9 across base, SFT, and SFT+RL stages (Team et al., 3 Apr 2026).

6. Limitations, governance, and relation to adjacent code-assistant research

The paper explicitly notes several limitations and risks. Training relies on competitive-programming-style distributions, so generalization to unfamiliar judging environments and languages requires further testing. Continued pretraining uses synthetic reasoning traces whose quality is bounded by teacher models and may inherit their biases. The system is compute-heavy because it combines long-context RL, asynchronous pipelines, and code execution sandboxes. AI policy compliance is also a live issue: platforms may restrict AI participation, and the authors argue for transparent disclosure and compliance frameworks. A further concern is adversarial behavior against judges, such as exploiting heuristic timing assumptions; GrandCode’s reported adversarial tests focus on correctness and efficiency rather than such exploitation (Team et al., 3 Apr 2026).

The broader significance of GrandCode lies in the claim that coordinated agentic RL, dense intermediate verification signals, and online adaptation can surpass top human performance on live competitive coding tasks. The paper identifies software verification, systems optimization, and scientific computing as adjacent domains in which adversarial test generation, delayed credit assignment, and hypothesis generation with small-scale verification may transfer (Team et al., 3 Apr 2026). This suggests a broader interpretation of GrandCode as a prototype for multi-stage coding agents that reason, test, and adapt under hard feedback constraints.

A terminological issue arises because adjacent repository-level code completion papers use “GrandCode” as an exemplar assistant. GraphCoder proposes a statement-level Code Context Graph with control-flow, control dependence, and data dependence, followed by coarse-to-fine retrieval and LLM prompting for repository-level code completion (Liu et al., 2024). GRACE extends that line with a multi-level, multi-semantic code graph, a Hybrid Graph Retriever, and structural fusion that serializes retrieved graph structure into the prompt (Wang et al., 7 Sep 2025). This suggests an important distinction: GrandCode in the competitive-programming sense is a multi-agent RL system centered on live contest solving, whereas GraphCoder and GRACE belong to the repository-aware completion literature and address retrieval of project-specific context rather than contest-time multi-stage decision making.

GrandCode is therefore best understood not as a single model checkpoint, but as a contest-oriented agentic system that couples modular reasoning, execution-grounded verification, asynchronous RL, and online search. Its reported live wins make it a milestone in competitive programming, while its architectural ideas place it within a broader movement toward coding agents that integrate generation, testing, memory, and structured adaptation under delayed and partial feedback (Team et al., 3 Apr 2026).