---
title: Citation-Aware Group Relative Policy Optimization
url: https://www.emergentmind.com/topics/citation-aware-group-relative-policy-optimization-c-grpo
type: topic
---

# Citation-Aware Group Relative Policy Optimization

Citation-aware Group Relative Policy Optimization (C-GRPO) is a reinforcement learning (RL) framework designed for the robust alignment of large language model (LLM) agents in tasks where factual grounding, comprehensive reasoning, and citation fidelity are critical, particularly in domains requiring evidence chains or legal citation accuracy. C-GRPO extends standard Group-Relative Policy Optimization (GRPO) by integrating a fine-grained, context-sensitive reward architecture—specifically Citation-aware Rubric Rewards (CaRR)—that evaluates rollouts not only on final outcome correctness, but also on the explicit identification and citation of supporting entities and the construction of complete evidence chains. Empirical evidence across deep search and retrieval-augmented question answering indicates that C-GRPO yields significant improvements in factuality and citation quality relative to traditional outcome-based RL or instruction tuning baselines [2601.06021], [2507.09638].

## 1. Foundations and Conceptual Architecture

The canonical RL protocol for deep search agents typically employs a binary outcome reward $R_o \in \{0, 1\}$, assigning credit only for answers matching the ground truth. This paradigm fails to capture partial factuality, grounded reasoning, and evidence connectivity. C-GRPO addresses these shortcomings by introducing CaRR—a reward decomposition that attributes real-valued rubric scores $R_r \in [0, 1]$ to agent rollouts. CaRR evaluates the trajectory along three axes: identification of hidden entities, citation grounding (verifiable support for each claim), and construction of a connected evidence chain linking intermediate facts to the answer. Only successful rollouts receive the mixed reward, which combines outcome correctness and normalized rubric score:
$$
R_i = (1-\alpha) R_o^{(i)} + \alpha R_o^{(i)} \hat{R}_r^{(i)}
$$
where $\hat{R}_r^{(i)}$ is group-wise normalized and $\alpha \in [0, 1]$ modulates rubric emphasis [2601.06021].

In Thai legal reasoning contexts, the reward is further refined to explicit citation fidelity via the Citation-F$_1$ metric, format adherence, and non-hallucination checks. Here, C-GRPO uses a multi-component scalar reward:
$$
R(s, a) = \alpha \cdot R_\text{citation}(s, a) + \beta \cdot R_\text{ans}(s, a)
$$
with $R_\text{citation}$ aggregating XML format checks, ground-truth citation overlap, and non-hallucination, and $R_\text{ans}$ derived via semantic similarity or coverage/consistency scoring [2507.09638].

## 2. Mathematical Formulation and Training Objective

For a group of $G$ trajectories per query, denote trajectories as $\{\mathcal{H}_1, \ldots, \mathcal{H}_G\}$, each assigned scalar reward $R_i$. Token-level importance sampling ratio and group-normalized advantage are computed as:
$$
\rho_{i, j} = \frac{\pi_\theta(\mathcal{H}_{i, j} \mid q, \mathcal{H}_{i, 1:j-1})}{\pi_{\theta_{\text{old}}}(\mathcal{H}_{i, j} \mid q, \mathcal{H}_{i, 1:j-1})}
$$
$$
\hat{A}_{i, j} = \frac{R_i - \mu_R}{\sigma_R}
$$
where $\mu_R$ and $\sigma_R$ are mean and standard deviation over the group.

The C-GRPO objective applies clipped PPO updates over group-token averages, masking out environment tokens:
$$
\mathcal{J}(\theta) = \mathbb{E} \left[ \frac{1}{\sum_{i, j} I(\mathcal{H}_{i, j})} \sum_{i=1}^{G} \sum_{j=1}^{|\mathcal{H}_i|} I(\mathcal{H}_{i, j}) \min \left(\rho_{i, j} \hat{A}_{i, j}, \operatorname{clip}(\rho_{i, j}, 1-\epsilon, 1+\epsilon) \hat{A}_{i, j} \right) \right]
$$
This architecture, with per-group baseline subtraction ($b_i$ in [2507.09638]), stabilizes gradient estimates and reduces variance compared to global baselines.

## 3. Training Protocol and Implementation Details

C-GRPO agents utilize backbone models such as Qwen3-4B, Qwen3-30B (for deep search), and Qwen2.5-7B-Instruct or Thai-CPT variants (for legal reasoning). Training involves cold-start supervised fine-tuning (SFT), followed by RL with C-GRPO over batched rollout groups (e.g., $G=16$, batch=128, $\text{lr}=2 \times 10^{-6}$, context window up to 128k tokens). Rubric weights $\alpha$ are ablated to identify optimal balance, peaking at $\alpha=0.3$; excessive rubric emphasis ($\alpha>0.5$) can detract from correctness [2601.06021].

Legal QA applications employ WangchanX-Legal-ThaiCCL-RAG datasets spanning 8,000+ examples, with BGE-M3 embedding-based retrieval [2507.09638]. Rollout group sizes typically use $K=10$ responses per prompt, with LoRA adapters (rank 256), AdamW optimization, and precision set to bfloat16.

Judge LLMs (e.g., DeepSeek-v3.2) are used for rubric scoring and, in legal tasks, for coverage/consistency evaluation. When semantic proxies are used, BGE-M3 embeddings offer cost-efficient similarity rewards with $>2.5\times$ computational savings over LLM judges.

## 4. Citation-aware Reward Engineering

Citation-aware rewards integrate the following components:

- **Format adherence**: Binary check for required XML tags (e.g., `<reasoning>`, `<answer>`, `<citation>`). Reward $R_\text{format}(s, a) \in \{0, 1\}$.
- **Non-hallucination**: Partial reward ($0.5$) if all cited entities appear in the retrieved context.
- **Citation-F$_1$**: Strict overlap metric between model-generated and gold citations:
  $$
  P = \frac{|C \cap G|}{|C|},\quad R = \frac{|C \cap G|}{|G|},\quad F_1 = \frac{2PR}{P + R}
  $$
This term both shapes policy during training and acts as the key evaluation metric.

- **Answer quality**: Semantic similarity (cosine over embeddings) or LLM-based coverage/consistency ($\in \{0, 0.5, 1.0\}$).

Only rollouts passing format and non-hallucination thresholds receive nonzero citation-F$_1$ reward, ensuring the agent must satisfy structural and factual constraints before earning fidelity credit [2507.09638].

## 5. Empirical Benchmarking and Performance

Empirical results across benchmarks demonstrate that C-GRPO produces superior citation fidelity and correctness relative to baseline RL and instruction tuning. On deep search tasks (BrowseComp at 64k and 128k tokens) [2601.06021]:

| Model                 | BrowseComp@64k | BrowseComp@128k |
|-----------------------|----------------|-----------------|
| DeepDive-4B-SFT       | 7.7            | 14.1            |
| + GRPO                | 12.9           | 14.7            |
| + C-GRPO              | **13.9**       | **17.5**        |
| DeepDive-30B-SFT      | 12.2           | 20.5            |
| + GRPO                | 16.0           | 18.9            |
| + C-GRPO              | **17.9**       | **24.8**        |

For Thai legal reasoning, C-GRPO with semantic-reward achieves up to +90% citation-F$_1$ gains and +31% joint quality improvements over SFT [2507.09638]:

| Model + Reward                | Citation F$_1$ | Coverage | Consistency | Joint Score |
|-------------------------------|---------------|----------|-------------|-------------|
| Qwen2.5-7B-Instruct (baseline)| 0.4103        | 0.5908   | 0.8402      | 0.6138      |
| + LoRA SFT                    | 0.5691 (+39%) | 0.5832   | 0.8341      | 0.6622 (+8%)|
| + LoRA GRPO (cov/cons)        | 0.6796 (+66%) | 0.6322   | 0.8598      | 0.7239 (+18%)|
| + LoRA GRPO (semantic)        | 0.7146 (+74%) | 0.7197   | 0.8232      | 0.7525 (+23%)|

C-GRPO’s improvements persist in out-of-domain generalization—e.g., on the NitiBench-Tax test set, GRPO variants substantially mitigate SFT degradation and yield higher citation F$_1$ (up to +66%).

## 6. Behavior Shaping: Shortcuts, Grounding, and Robustness

Weak outcome-only RL (GRPO, PPO) incentivizes shortcut exploitation, with agents minimizing tool calls and unrolling incomplete evidence chains. C-GRPO reverses this trend: agents increase citation gathering and rubric satisfaction over training, as evidenced by empirical tool-call metrics and qualitative analyst cases [2601.06021].

C-GRPO’s rubric-satisfying protocol produces more comprehensive chains: agents identify all relevant hidden entities and cite supporting URLs with high fidelity. In contrast, outcome-only baselines often halt at the final answer without verifying intermediate constraints.

Reward decomposition—involving format, hallucination, and citation-F$_1$ adherence—ensures agents cannot exploit single criteria or “game” the system, focusing model updates on holistic, evidence-grounded reasoning [2507.09638].

## 7. Limitations, Constraints, and Future Research Directions

C-GRPO’s rubric-based reward design leverages the compositional structure of multi-hop/synthetic QA; its applicability to open-ended, unstated-requirement QA is less direct [2601.06021]. Static rubric generation can limit adaptability; online evolution of rubrics via self-play or contrastive rollouts constitutes a plausible avenue for future work.

Reward assignment accuracy is bounded by the reliability of judge LLMs or embedding proxies. Model errors in entity identification or citation support can introduce noise or bias.

Potential extensions include human-in-the-loop rubric refinement, broader agentic task adaptation (e.g., code synthesis, planning), and further cost reduction via improved semantic proxies or data filtering [2507.09638].

---

C-GRPO establishes a rigorous foundation for RL-fine-tuning of evidence-driven LLM agents, achieving demonstrable gains in reasoning fidelity and citation accuracy across deep search and legal domains by integrating comprehensive, context-structured rewards with stable, group-based updates [2601.06021], [2507.09638].

Source: https://www.emergentmind.com/topics/citation-aware-group-relative-policy-optimization-c-grpo