CodeSimpleQA-RL: RL for Accurate Code QA

• CodeSimpleQA-RL is a framework that post-trains large language models using SFT followed by RL to boost factual correctness in code question answering.
• It employs group-normalized relative policy optimization, rewarding exact match responses to align with high-fidelity bilingual QA data.
• Empirical results demonstrate significant improvements, with up to an 8.4 percentage-point increase in English F1 score on code QA benchmarks.

CodeSimpleQA-RL is a reinforcement learning-based post-training framework designed to improve the factual accuracy of LLMs in code-related question answering. It is an integral component of the CodeSimpleQA benchmark and training pipeline, combining large-scale supervised fine-tuning (SFT) with a specialized reinforcement learning stage using group-normalized relative policy optimization. CodeSimpleQA-RL targets factual knowledge alignment in code LLMs across multiple natural and programming languages, augmenting prior approaches that focused narrowly on code execution or answer plausibility.

1. Pipeline Overview and Objectives

CodeSimpleQA-RL is deployed after pre-training a base LLM on broad code and natural language corpora. The post-training process follows a sequential "SFT → RL" regimen:

1. Supervised Fine-Tuning (SFT): The LLM is fine-tuned on CodeSimpleQA-Instruct, a corpus of 66.9 million QA instances (53.6M English, 13.4M Chinese) with high factual reliability, using prompt/target pairs of concise answers and rejection sampling for quality control.
2. Group Relative Policy Optimization (GRPO) RL: The model is further optimized with group-based reinforcement learning. For each question, multiple sampled outputs are compared against reference answers, with groupwise normalization and clipped surrogate loss promoting factual recall while penalizing policy drift.

The essential purpose is to specialize LLMs for precise, reliable factual answer retrieval in code Q&A, moving beyond simple code synthesis or execution checks to linguistic and fact-based correctness within diverse domains (Yang et al., 22 Dec 2025).

2. Reinforcement Learning Formulation

The CodeSimpleQA-RL stage employs the GRPO algorithm, a generalization of PPO tailored for QA factuality reward signals:

Let the SFT-tuned model be $\pi_{\theta_{\text{old}}}$, and the trainable policy be $\pi_\theta$. For each QA pair $(q, a)$:

• Sampling: For each question $q$, $K$ rollouts (model completions) $\{o_k\}_{k=1}^K$ are sampled from $\pi_{\theta_{\text{old}}}$.
• Reward: Each generated answer $o_k$ is assigned $r_k = 1$ if it matches the reference answer $a$ exactly, else $0$.
• Group-normalized advantage: At each token $t$,

$\hat{A}_{k,t} = \frac{r_k - \mu_r}{\sigma_r}$

where $\mu_r$ and $\sigma_r$ are mean and standard deviation of $\{r_k\}$ across $K$ samples for the group.

• Policy update: Define the importance ratio $$r_{k,t}(\theta) = \pi_\theta(o_{k,t}|\cdot)/\pi_{\theta_{\text{old}}}(o_{k,t}|\cdot)$$, and the clipped terms

$$A_1 = r_{k,t}(\theta) \cdot \hat{A}_{k,t}, \quad A_2 = \text{clamp}(r_{k,t}(\theta), 1-\epsilon, 1+\epsilon) \cdot \hat{A}_{k,t}$$

With a KL penalty $\beta$, the GRPO loss per group is:

$$\mathcal{L}_{\mathrm{GRPO}}(\theta) = \mathbb{E}_{(q,a),\{o_k\}} \left[ \frac{1}{K} \sum_{k=1}^K \frac{1}{|o_k|} \sum_{t=1}^{|o_k|} \left( -\min(A_1, A_2) + \mathcal{L}_{KL} \right) \right]$$

where $$\mathcal{L}_{KL} = \beta D_{KL}(\pi_{\theta_{\text{old}}}\|\pi_\theta)$$.

This approach incentivizes the model to shift probability mass toward answer trajectories that achieve a relatively high reward within a sampled group while restricting policy updates to remain close to the SFT initialization (Yang et al., 22 Dec 2025).

3. Data Construction and Curation

The foundation for CodeSimpleQA-RL is the CodeSimpleQA-Instruct dataset, a comprehensive bilingual resource meticulously designed to enforce code factuality:

• Document Selection: Web-scale crawling and fastText-based domain classification prioritize code-relevant documents, filtered for domain reliability and technical depth.
• Knowledge Clustering: Code-aware BERT encodings are clustered via DBSCAN to guarantee wide topical coverage (APIs, debugging, design, etc.).
• QA Pair Generation: Deepseek-V3.1 LLM is prompted (T=0.1) to rewrite clusters into concise factual QA pairs.
• LLM-As-Judge Filtering: Another LLM labels each (question, answer, document) triple as A:CORRECT, B:INCORRECT, or C:NOT_ATTEMPTED; only A:CORRECT pairs are retained.
• Scale and Diversity: English and Chinese are both covered, resulting in 66.9M QA samples, providing robust fine-tuning coverage and cross-lingual alignment (Yang et al., 22 Dec 2025).

4. Training Regimen and Hyperparameters

CodeSimpleQA-RL is characterized by scale and rigor in hyperparameter selection:

• SFT Stage: Fine-tuned with AdamW, cosine-decay learning rate (peak $6\times 10^{-5}$), large batch size (1024), sequence limit 8192 tokens, tensor parallelism (size 4). SFT continues to validation convergence ($\sim 100k$–$200k$ steps).
• RL Stage: Conducted on 64 GPUs (FSDP, vLLM). GRPO uses a constant learning rate $5\times 10^{-7}$, batch size 1024, $K=8$ trajectories/group, clip range $\epsilon \approx 0.1$, and KL penalty $\beta$ tuned in $[0.1,\,0.5]$. Total RL steps $\sim 20k$–$50k$ are empirically sufficient.
• Optimization Objective: Only exact match is rewarded, producing binary-reward RL typical in factual QA (Yang et al., 22 Dec 2025).

5. Evaluation Metrics and Benchmarks

Performance is measured on the CodeSimpleQA benchmark of 1,498 human-curated, bilingual QA pairs using the Chinese SimpleQA-inspired F1 metric, computed from:

• Correct (CO): Proportion of fully correct answers.
• Not Attempted (NA): Proportion of unanswered items.
• Incorrect (IN): Proportion of wrong answers.
• Correct-Given-Attempted (CGA): Proportion of correct among attempted.
• Overall F1: Harmonic mean of CO and CGA.

Execution correctness is not tested; the focus is strictly on QA factuality (Yang et al., 22 Dec 2025).

Empirical Results

	Baseline (Zh/En F1)	+Initial RL (RFT)	+Full RL (CodeSimpleQA-RL)	Absolute Gain
Chinese (32B)	40.0%	42.3%	45.2%	+5.2 pp
English (32B)	33.9%	39.2%	42.3%	+8.4 pp

Comparisons of CodeSimpleQA-RFT and CodeSimpleQA-RL (full GRPO) isolate the gain from groupwise advantage normalization and group-based sampling (+2.9 Zh, +3.1 En) (Yang et al., 22 Dec 2025).

6. Trade-offs, Limitations, and Comparison with Alternative QA RLHF Systems

Trade-offs and Limitations

• Reward granularity: The binary reward only captures exact matching, ignoring nuanced partial correctness.
• Computational cost: RL stage requires significant resources (64 GPUs) and distributed inference.
• Reference reliance: Fact-matching as judged by string or LLM-based matching introduces susceptibility to reference phrasing.
• KL-penalty tuning: Avoiding "reward collapse" requires precise $\beta$ scheduling.
• Performance saturation: Further group size ($K$) or RL step increases provide diminishing returns; richer reward signals (e.g., semantic similarity) are needed for further advances (Yang et al., 22 Dec 2025).

Comparison to Multi-Perspective Preference RL and RLHF

Alternative frameworks for programming question answering, such as ALMupQA, adopt a multi-perspective user preference ranking approach instead of strict policy optimization. ALMupQA employs a combination of supervised fine-tuning, listwise contrastive ranking on answer pools scored by human votes, LLM content evaluation, and questioner bias, as well as retrieval-augmented in-context examples to maintain recency and align with community standards (Yang et al., 2024).

A distinctive aspect of CodeSimpleQA-RL is its exclusive reliance on reward derived from exact factual match, whereas ALMupQA and similar systems integrate user preferences, human feedback, and content-based metrics in alignment objectives, rather than explicit RL with a reward function. This suggests that while CodeSimpleQA-RL is optimized for precision factuality, broader user relevance and adaptability might benefit from preference-based or hybrid approaches (Yang et al., 2024).

7. Impact, Extensions, and Future Directions

CodeSimpleQA-RL establishes a scalable protocol for aligning large code LLMs with high-fidelity factual knowledge by combining massive-scale supervised signal with a group-normalized RL optimization. Experimental data demonstrates substantive improvement over base models, with gains in both English and Chinese. The approach is notable for:

• Establishing a reproducible, bilingual factual QA benchmark isolating code knowledge, distinct from code synthesis or execution criteria.
• Demonstrating that PPO-style groupwise RL can consistently improve F1 factuality metrics over pure SFT or naive RL.
• Scaling to instruction-finetuned LLMs of 32B and above on commodity GPU clusters.

Potential extensions include:

• Use of richer reward functions permitting partial credit (e.g., semantic similarity or confidence-based penalties).
• Meta-learning and domain adaptation for transfer to specialized technical areas or emerging programming paradigms.
• Integration of user preference or context-adaptive ranking (analogous to ALMupQA) for broader applicability.
• Broader multilingual or domain-diverse coverage via further corpus augmentation (Yang et al., 22 Dec 2025, Yang et al., 2024).

In summary, CodeSimpleQA-RL demonstrates that group-normalized RL on high-quality, large-scale factual QA corpora is a viable approach for enhancing factuality in code LLMs. Its distinct reward structure and evaluation focus make it a foundational tool for reliable code-related question answering in research and applied contexts, while also highlighting current challenges in RL-based factual alignment that motivate ongoing research into richer feedback signals and adaptive methods.