---
title: 'GrandCode: Agentic RL in Competitive Programming'
url: https://www.emergentmind.com/topics/grandcode
type: topic
---

# GrandCode: Agentic RL in Competitive Programming

GrandCode is a multi-agent reinforcement learning system for competitive programming introduced in "GrandCode: Achieving Grandmaster Level in Competitive Programming via Agentic Reinforcement Learning" [2604.02721]. It is designed for live contest settings in which code must be generated, verified, and submitted under strict time and memory limits, hidden tests, and scoring rules that reward early accepted submissions. The system combines learned agentic modules for hypothesis proposal, solving, summarization, and adversarial test generation with post-training, asynchronous online RL, and test-time adaptation. In the reported live evaluation, GrandCode placed first in three Codeforces rounds—1087, 1088, and 1089—beating all human participants, including legendary grandmasters [2604.02721].

## 1. Problem domain and design rationale

GrandCode addresses competitive programming as a setting in which correctness, efficiency, and latency are jointly binding constraints. The paper identifies several factors that make the domain difficult for AI: participants must solve multiple problems quickly under strict time and memory limits; code must be both correct and efficient; feedback is limited to signals such as “Wrong answer” and “Time limit exceeded”; high-quality adversarial tests are hidden; and hard problems require long reasoning chains, repeated self-debugging, execution, and verification cycles [2604.02721].

The work situates itself against prior AI systems that were strong but still limited under live conditions. AlphaCode and AlphaCode2 achieved mid-to-high percentile ratings; OpenAI’s o3 ranks 175th globally; Gemini 3 Deep Think achieved 8th place, but only on historical problems rather than live contests [2604.02721]. GrandCode’s central claim is that live contest constraints exacerbate the standard difficulties of agentic RL: delayed rewards arise because quality is only known after execution and hidden-test approximation, off-policy drift becomes severe in asynchronous pipelines, and long-context inference and verification substantially increase cost.

The paper also emphasizes operational constraints external to modeling. Codeforces prohibits AI-generated content, and high-ranking accounts may face manual scrutiny. To avoid undue attention, GrandCode defers joint account submissions, denoted $S(\text{joint})$, until near the end, while tracking per-problem $S(\text{separate})$ scores. The reported argument is that the penalty from multiple submissions is small, with at most four attempts per task [2604.02721]. A common misconception is therefore that GrandCode is only a stronger code generator; the reported system is instead explicitly designed around contest rules, verification uncertainty, and submission strategy.

## 2. Multi-agent architecture and orchestration

GrandCode orchestrates three learned policies and a test generator. The main solver $\pi_{\mathrm{main}}$ is a large MoE policy that produces reasoning traces and code, with long-context support via pipelined context parallelism for hybrid DeltaNet and softmax attention blocks. The hypothesis model $\pi_{\mathrm{hypothesis}}$ proposes structural conjectures, invariants, or compact characterizations and validates them on random instances or brute-force solvers. The summarization model $\pi_{\mathrm{summary}}$ compacts long reasoning traces into progressive summaries $s_t$ that preserve essential information for downstream synthesis and RL. Test-case generation constructs adversarial tests through difference-driven generation, solution attacks, and large-size stress tests using available solvers [2604.02721].

Interaction among these modules is controlled by a difficulty-based router. A learned classifier assigns each task a difficulty level $d \in \{1,\dots,5\}$. Easier tasks are sent to direct generation; harder tasks invoke hypothesis generation, summarization, and test-time RL. In the hypothesis loop, $\pi_{\mathrm{hypothesis}}$ emits hypotheses $h$, small-instance checks validate them, and counterexamples trigger revision until consistency is achieved. Verified hypotheses are then injected into the prompt for $\pi_{\mathrm{main}}$. For long traces $c$, progressive summarization chunks the trace into $c^{(1)},\dots,c^{(n)}$ and updates a summary state through transitions $(s_{t-1}, c^{(t)}) \to s_t$; the resulting $s_t$ is fed back to the solver [2604.02721].

The controller manages asynchronous RL rollouts with per-token behavior-policy versions, pipelines sampling and training, and applies staleness weighting and token dropping to limit off-policy drift. It also maintains per-problem memory: validated hypotheses, summaries, test pools, solution history with scores, and a global summary of exploration status. This design makes GrandCode closer to a coordinated multi-stage decision system than to a single-pass language model. The architecture’s significance lies in the explicit separation of structural conjecture formation, long-horizon memory compression, candidate synthesis, and adversarial verification.

## 3. Reinforcement learning formulation and Agentic GRPO

GrandCode models agentic coding as an MDP, or as a POMDP when hidden tests and feedback are partial. The state $s_t$ includes the problem $x$, difficulty $d$, the current prompt, validated hypotheses $h$ when available, a partial reasoning trace, the summary state $s_t$ used for long contexts, the current candidate code $P_t$, and the available test pools with their outcomes. Actions are textual outputs from $\pi_{\mathrm{main}}$, $\pi_{\mathrm{hypothesis}}$, or $\pi_{\mathrm{summary}}$, depending on the active stage. Transitions concatenate the agent’s output, optionally update summaries, execute code in the sandbox, and append verification outcomes to context [2604.02721].

Rewards are decomposed into immediate rewards $r_t$ and a terminal reward $r_N$. Immediate rewards are available at intermediate stages, such as hypothesis verification pass rates, partial code correctness checks, adversarial-test differences, or efficiency gains. The terminal reward is the final rollout score after full verification and efficiency evaluation. For a candidate program $P$, the paper defines executability $E(P)\in\{0,1\}$, correctness $C(P)\in\{0,1\}$, and a per-test efficiency score
$$
s_i(P)=
\begin{cases}
\tau_b(t_i)/\tau_P(t_i), & \text{if within time limit} \\
0.1, & \text{otherwise.}
\end{cases}
$$
The final code reward is
$$
R(P)=0 \text{ if } E(P)=0 \text{ or } C(P)=0; \text{ otherwise } R(P)=m^{-1}\sum_{i=1}^{m}s_i(P).
$$
The paper also imposes a length penalty tied to difficulty:
$$
\mathrm{Penalty}(l,d)=\max\!\left(0,\frac{l-B\alpha^{(d-1)}}{B\alpha^{(d-1)}}\right).
$$
GrandCode primarily uses group-normalized relative advantages rather than an explicit discount factor $\gamma$; delayed credit is handled by Agentic GRPO [2604.02721].

Agentic GRPO extends Group Relative Policy Optimization to multi-stage rollouts with delayed rewards and severe off-policy drift. Each stage has two phases. In the Immediate Reward phase, as soon as $r_t$ is available, GrandCode computes group-normalized advantages and updates the policy with a clipped surrogate objective. In the Delayed Correction phase, once the final reward is known, it computes a correction $\delta_t^{(i)} = r_N^{(i)} - r_t^{(i)}$, normalizes it within the group, and applies another clipped update to earlier stages. The intended effect is to combine timely credit assignment with a later correction for future contribution [2604.02721].

To manage asynchronous training, GrandCode applies staleness weights based on token age $d_t$:
$$
w(d_t)=
\begin{cases}
1, & d_t \le K_1 \\
\exp(-\lambda(d_t-K_1)), & K_1<d_t\le K_2 \\
0, & d_t > K_2.
\end{cases}
$$
Immediate rewards are therefore applied early, while delayed corrections may be dropped if they arrive too late. The paper presents this as a throughput–bias trade-off characteristic of long, asynchronous agentic RL pipelines.

## 4. Training pipeline, infrastructure, and test-time adaptation

GrandCode’s post-training pipeline begins with continued pretraining on TACO, LeetCode, USACO, CodeContests, IOI, crawled problems, and reasoning traces generated by Gemini and Claude. The training mix includes 20% hypothesis-conditioned examples so that $\pi_{\mathrm{main}}$ learns to consume outputs from $\pi_{\mathrm{hypothesis}}$. The paper states that the upper bound accept rate is tied to teacher models, approximately $73\%$ on the 100-problem benchmark [2604.02721].

Supervised fine-tuning uses different curation rules for easy and hard problems. For easy problems, reasoning traces are retained when the resulting solution is equivalent to or as efficient as the gold solution. For hard problems, traces are selected to maximize a score that combines the normalized log-likelihood of $(c_i,y)$ under $\pi_{\mathrm{main}}$ and the normalized log-likelihood of $x$ conditioned on $c_i$ under the initial solver, which is intended to avoid degenerate answer leakage and prioritize predictive thinking traces. The summarization model is trained in three stages: local transitions $c^{(t)}\to s_t$ with GRPO, chain training of full summaries using terminal answer likelihood as reward, and joint RL integration with $\pi_{\mathrm{main}}$; summary data are also mixed into SFT [2604.02721].

Joint RL then optimizes $\pi_{\mathrm{main}}$, $\pi_{\mathrm{hypothesis}}$, and $\pi_{\mathrm{summary}}$. The reward for the hypothesis model combines local verification pass rate with downstream gains relative to a no-hypothesis condition. Infrastructure is correspondingly specialized: PipelineRL is used for long sequences; $\pi_{\mathrm{main}}$ runs on a distributed GPU mesh with expert parallelism and context parallelism; $\pi_{\mathrm{hypothesis}}$ and $\pi_{\mathrm{summary}}$ run on smaller GPU pools; and a CPU sandbox executes compilation and tests. To stabilize MoE behavior, the router is frozen during RL and only FFN experts are updated [2604.02721].

At test time, GrandCode shifts the objective from expected reward to best-of-$N$ final reward. It uses a smoothed rank-based objective
$$
\mathcal{W}(\theta)=\mathbb{E}\Bigl[\sum_{j=1}^{N}w_j(\lambda)\,R^{(j)}\Bigr], \quad
w_j(\lambda)=\frac{e^{-\lambda j}}{\sum_{m=1}^{N}e^{-\lambda m}},
$$
with $\lambda$ annealed from uniform averaging toward top-ranked emphasis. Adaptation is implemented through LoRA updates on $\pi_{\mathrm{main}}$ conditioned on problem-specific history, while $\pi_{\mathrm{hypothesis}}$ and $\pi_{\mathrm{summary}}$ are fixed to conserve latency [2604.02721]. This design indicates that online improvement is localized to the main solver and is driven by contest-specific evidence rather than by retraining all modules.

## 5. Verification strategy and empirical performance

Adversarial test generation is a central part of GrandCode’s reported gains. Difference-driven tests prompt LLMs including Claude, GPT, DeepSeek, and Kimi to propose corner-case inputs without seeing code, then retain inputs that induce output differences among candidate solutions. Solution attacks use the gold solution during training to analyze differences between a candidate and the gold and then propose inputs likely to expose bugs; these attacks are refined with validation, and Qwen-3.5-27B is fine-tuned to generate adversarial tests conditioned on problem and solution. Large-size tests use faster candidate programs to produce reference outputs where brute force times out. On 50 real Codeforces problems evaluated by the real judge, the base suite passed 42/50 problems, difference-driven generation plus solution attack raised this to 48/50, and submission feedback plus continued online generation reached 50/50 [2604.02721].

The live Codeforces evaluation is the paper’s most visible empirical result. GrandCode participated in Round 1087 (Mar 21, 2026), Round 1088 (Mar 28, 2026), and Round 1089 (Mar 29, 2026). It ranked first in all three rounds, solved all tasks in each contest, and was the first to finish all tasks in each. The reported finish times were 00:51:11, 01:40:35, and 00:56:43, respectively. The separate-submission scores were 9269, 16511, and 11596, while the deferred joint-account scores were 8334, 15008, and 9506 [2604.02721].

Offline benchmarks show a staged improvement from post-training, joint RL, and test-time RL. On a 100-problem benchmark with five difficulty levels, frontier baselines are reported at approximately $75\%$ accept for Gemini 3.1 Pro, $73\%$ for Claude Opus 4.6, $72\%$ for GPT-5.4, and $64\%$ for Qwen 3.5-397B base. For Qwen 3.5-397B, continued training reaches $71\%$, SFT $73\%$, and the addition of summary $72\%$. Full RL training raises accept rate to $81\%$ with 13/20 solved at Level 5 and weighted score $72.3$, while test-time RL further raises the result to $85\%$, 15/20 at Level 5, and weighted score $73.5$ [2604.02721].

The paper’s ablations attribute the gains to multiple interacting components rather than to a single intervention. Summarization can slightly reduce offline scores if not carefully integrated, but it enables tractable RL and faster inference on very long trajectories. Adversarial test generation materially improves pre-submission reliability. Hypothesis generation improves downstream solving when trained jointly. Agentic GRPO’s immediate and delayed updates improve credit assignment relative to standard terminal-reward GRPO, while staleness-weighted clipping reduces off-policy drift in pipeline RL. The hypothesis model itself improves from pass@1 $34\%\to45\%\to52\%$ and pass@5 $44\%\to52\%\to57\%$ across base, SFT, and SFT+RL stages [2604.02721].

## 6. Limitations, governance, and relation to adjacent code-assistant research

The paper explicitly notes several limitations and risks. Training relies on competitive-programming-style distributions, so generalization to unfamiliar judging environments and languages requires further testing. Continued pretraining uses synthetic reasoning traces whose quality is bounded by teacher models and may inherit their biases. The system is compute-heavy because it combines long-context RL, asynchronous pipelines, and code execution sandboxes. AI policy compliance is also a live issue: platforms may restrict AI participation, and the authors argue for transparent disclosure and compliance frameworks. A further concern is adversarial behavior against judges, such as exploiting heuristic timing assumptions; GrandCode’s reported adversarial tests focus on correctness and efficiency rather than such exploitation [2604.02721].

The broader significance of GrandCode lies in the claim that coordinated agentic RL, dense intermediate verification signals, and online adaptation can surpass top human performance on live competitive coding tasks. The paper identifies software verification, systems optimization, and scientific computing as adjacent domains in which adversarial test generation, delayed credit assignment, and hypothesis generation with small-scale verification may transfer [2604.02721]. This suggests a broader interpretation of GrandCode as a prototype for multi-stage coding agents that reason, test, and adapt under hard feedback constraints.

A terminological issue arises because adjacent repository-level code completion papers use “GrandCode” as an exemplar assistant. GraphCoder proposes a statement-level Code Context Graph with control-flow, control dependence, and data dependence, followed by coarse-to-fine retrieval and LLM prompting for repository-level code completion [2406.07003]. GRACE extends that line with a multi-level, multi-semantic code graph, a Hybrid Graph Retriever, and structural fusion that serializes retrieved graph structure into the prompt [2509.05980]. This suggests an important distinction: GrandCode in the competitive-programming sense is a multi-agent RL system centered on live contest solving, whereas GraphCoder and GRACE belong to the repository-aware completion literature and address retrieval of project-specific context rather than contest-time multi-stage decision making.

GrandCode is therefore best understood not as a single model checkpoint, but as a contest-oriented agentic system that couples modular reasoning, execution-grounded verification, asynchronous RL, and online search. Its reported live wins make it a milestone in competitive programming, while its architectural ideas place it within a broader movement toward coding agents that integrate generation, testing, memory, and structured adaptation under delayed and partial feedback [2604.02721].

Source: https://www.emergentmind.com/topics/grandcode