Hail to the Thief: Exploring Attacks and Defenses in Decentralised GRPO (2511.09780v1)

Abstract: Group Relative Policy Optimization (GRPO) has demonstrated great utilization in post-training of LLMs. In GRPO, prompts are answered by the model and, through reinforcement learning, preferred completions are learnt. Owing to the small communication volume, GRPO is inherently suitable for decentralised training as the prompts can be concurrently answered by multiple nodes and then exchanged in the forms of strings. In this work, we present the first adversarial attack in decentralised GRPO. We demonstrate that malicious parties can poison such systems by injecting arbitrary malicious tokens in benign models in both out-of-context and in-context attacks. Using empirical examples of math and coding tasks, we show that adversarial attacks can easily poison the benign nodes, polluting their local LLM post-training, achieving attack success rates up to 100% in as few as 50 iterations. We propose two ways to defend against these attacks, depending on whether all users train the same model or different models. We show that these defenses can achieve stop rates of up to 100%, making the attack impossible.

• The paper demonstrates that adversarial nodes can achieve nearly 100% attack success via completion poisoning in both vertical and horizontal decentralized RL settings.
• It employs empirical evaluations with token-level log-probability checks and an LLM-based judge to detect and mitigate string-based adversarial attacks.
• The study highlights the limitations of current reward models and stresses the need for advanced semantic validation in decentralized LLM post-training environments.

Attacks and Defenses in Decentralized GRPO: A Technical Analysis

Overview and Motivation

This paper rigorously investigates the vulnerability surface of decentralized Group Relative Policy Optimization (GRPO) when used for LLM post-training under distributed reinforcement learning (RL) paradigms. By focusing on string-based communication between independent nodes, the work systematically demonstrates how adversarial participants in both vertical and horizontal decentralized RL can inject arbitrary or targeted behaviors via completion poisoning, regardless of reward model stringency. Two tailored defenses are proposed and empirically evaluated to address homogeneous and heterogeneous system settings.

Decentralized GRPO: System Setup

Decentralized GRPO post-training is formalized under two schemes:

• Vertical Decentralized RL: Each node selects different prompts and outputs $G$ completions per prompt.
• Horizontal Decentralized RL: Each node produces a share of completions for a globally assigned batch of prompts.

Both paradigms leverage an all-gather communication pattern for completion sharing and gradient updates, sidestepping the high bandwidth and synchronization requirements typical of gradient aggregation. Homogeneous settings assume synchronized model parameters; heterogeneous settings allow divergence across models and architectures.

Attack modeling assumes that a fraction $f$ of nodes behave adversarially, crafting completions with oracle-level reward performance and introducing malicious tokens or reasoning. The attacker’s objective is to maximize the propagation and internalization of the adversarial behavior by benign models without triggering typical verifiable reward-based defenses.

Attack Modalities

Out-of-Context Attacks

Domain-agnostic token insertion strategies (e.g., "All hail to the thief") are evaluated across math reasoning and coding tasks, with the malicious text exposure exclusively in the reasoning ($\langle$think$\rangle$) segments that are not explicitly rewarded. Vertical and horizontal settings are both exploited.

Figure 1: Attack Success Rate (fraction of honest model's completions which contain Gleeb Glorp Glub in every sentence).

Empirical results show that nearly 100% attack success rates were achieved within 20 iterations in both paradigms, despite only 25% adversarial node participation. This demonstrates that the GRPO advantage calculation strongly amplifies poisoned completions when reward models do not comprehensively evaluate all completion content.

In-Context Attacks

Domain-specific attacks target reasoning semantics directly, such as equation manipulation ("2+2=5") or code injection (malicious library imports). Vertical RL settings allow adversaries to select prompts aligned with their attack vectors; horizontal RL permits indiscriminate poisoning across prompts.

Figure 2: ASR of equation manipulation resulting in $2+2\neq4$ or $2*2\neq4$.

Figure 3: ASR depending on the number of poisoned completions/nodes in horizontal decentralised RL—illustrates the scaling effect and threshold for attack viability.

The equation manipulation attack reached >50% success in less than 20 iterations—a strong signal that GRPO's grouping and advantage normalization can embed substantial domain corruption with minimal visible reward degradation. Further ablations reveal that even partial poisoning (as low as 20% completions for a prompt) suffices for effective model compromise.

Figure 4: Relative effect of each poisoned completion over different ratios of poisoned completions for various group sizes—quantifying the per-completion amplification in poisoning strategies.

Quantitative and Qualitative Attack Effects

Figure 5: Returns of a baseline (all honest workers) vs returns in two attack settings (vertical and horizontal) with the All Hail to the Thief attack goal.

Analysis of reward curves shows no significant drop in aggregate task performance, indicating the attacks' stealth characteristics. Poisoned completions often maintain output quality, camouflaging the adversarial payload among valid reasoning and answers.

Defense Mechanisms

Homogeneous Defense: Token-Level Log-Probability Checking

Nodes verify incoming completions by comparing token generation probabilities under synchronized models. Bitwise reproducibility (syncing random seeds) enhances perfect detection. This simple method is 100% effective for out-of-context attacks but less so for semantic corruption (e.g., "2+2=5") due to plausible top-$k$ generation variability.

Heterogeneous Defense: LLM-as-a-Judge

Defending in heterogeneity entails using an instruction-tuned LLM to critically evaluate completions for malicious content. Binary accept/reject decisions from the judge augment the reward calculation. This approach blocks up to 95.2% of targeted attacks but introduces overhead—learning efficiency degradation is observed due to frequent false positives and judge misclassifications.

Figure 6: Returns of a baseline (all honest workers) vs returns in two defended settings (Hail to the thief (HTTT) horizontal and 2+2=5 vertical).

Ablation: Attack Scaling Dynamics and KL Regularization

Figure 7: Relative effect of poisoned completions vs honest worker reward for models at different training stages—higher benign reward demands higher poisoned ratios for attack success.

Experiments manipulating the ratio and reward of poisoned completions establish the existence of a potency threshold. KL-divergence regularization, commonly used to curb policy drift, fails to effectively suppress the attack and harms benign learning if weights are nontrivially large.

Figure 8: ASR given different KL-divergence values—demonstrating inadequate defense by regularization alone.

Limitations and Open Challenges

While proposed defenses show promising empirical effectiveness, they hinge on either system homogeneity, reproducibility (rare in practical deployments), or strong judge capabilities (vulnerable to jailbreaking and domain transfer failures). LLM-as-a-judge approaches additionally incur substantial compute and introduce evaluation sensitivity to judge model tuning and instructions. Token-level reward attribution remains an open area; current reward models insufficiently detect semantic exploits embedded in seemingly correct completions.

Attackers may employ advanced techniques such as subliminal learning, encoding adversarial traits undetectably across tasks, posing near-intractable future challenges.

Implications and Future Directions

The work highlights critical security risks for decentralized RL post-training pipelines in LLM ecosystems, particularly those deploying string-only exchange and verifiable string-based rewards. Robust validation of completions—ideally at a token or semantic level—is essential for deployment at scale in open environments. Further research is required into scalable, high-fidelity reward attribution, advanced judge model reliability, and defenses against adaptive, cross-domain poisoning.

Additionally, the paper provides a framework for systematic analysis of RL-based model poisoning in distributed settings, which is increasingly pertinent as large-scale LLM post-training transitions to federated, decentralized infrastructure.

Conclusion

This paper formally exposes highly effective model poisoning strategies targeting decentralized GRPO via completion injection and demonstrates that the verifiable reward models currently used leave substantial attack surface unaddressed. Two defenses, tailored to system homogeneity and heterogeneity, are empirically validated but have limitations in generality and robustness. Future work must address semantic-level reward modeling and judge reliability to realize resilient decentralized RL systems for large model post-training.