Non-vacuous Generalization Bounds for Reinforcement Learning with Verifiable Rewards

Abstract: While reinforcement learning with verifiable rewards (RLVR) is widely used to improve the reasoning capabilities of LLMs, the generalizability of the resulting models remains poorly understood. In this work, we establish the first non-vacuous generalization bounds for parameter-efficient RLVR fine-tuning at the billion-parameter scale. Our approach adapts PAC-Bayes compression bounds to this setting, and addresses the inherent stochasticity of token generation by applying the Gumbel-max reparameterization trick. To operationalize these bounds, we propose the Progressive RLVR framework, which integrates RLVR with on-policy distillation, TinyLoRA, and model quantization. Progressive RLVR empirically retains 84-97% performance of standard LoRA fine-tuning while producing models that are 14,796x more compressible. We show that this framework yields non-vacuous generalization bounds in four domains: mathematical problem-solving, programming, general-knowledge reasoning, and Text-to-SQL. Our bounds exceed the accuracy of the base model by 9-51% and lie within 6-11% of the accuracy of the fine-tuned models.

• The paper demonstrates a novel PAC-Bayes compression framework that provides non-vacuous generalization bounds for RLVR-finetuned billion-parameter LLMs.
• It introduces a progressive pipeline combining TinyLoRA, on-policy distillation, and quantization to minimize compression penalties and decouple stochastic generation noise.
• Empirical evaluations across diverse domains reveal bounds that exceed base-model accuracy by 9–51 points while closely matching fine-tuned performance.

Non-vacuous Generalization Bounds for RLVR in Billion-Parameter LLMs

Overview

The paper "Non-vacuous Generalization Bounds for Reinforcement Learning with Verifiable Rewards" (2607.14506) makes the first advance toward theoretically verified generalization for LLMs post-trained with RLVR (Reinforcement Learning with Verifiable Rewards) at scale. The authors construct PAC-Bayesian compression generalization bounds for RLVR, proving non-vacuous guarantees at the billion-parameter scale via a pipeline combining TinyLoRA, on-policy distillation, and quantization. A Gumbel-max reparameterization is employed to handle the combinatorial stochasticity of autoregressive text generation, effectively decoupling environment and generation noise in the analysis. Empirical evaluation on mathematics, programming, general knowledge, and Text-to-SQL domains demonstrates that tight non-vacuous bounds are practical for real-world LLM post-training, with the bounds consistently exceeding the base-model accuracy by large margins and closely matching fine-tuned performance. This work establishes a rigorous methodology and recipe for formally certifying RLVR-based specialization of LLMs.

Theoretical Framework

The central technical contribution is a refinement of the PAC-Bayes compression approach for RLVR. Prior PAC-Bayes bounds for supervised and pretraining settings are inapplicable due to the RLVR reward being a functional of the model’s own stochastic generations rather than i.i.d. labels, and the combinatorially large support of trajectories.

The framework introduces a two-level decomposition of the generalization gap:

• Extrinsic error (prompt space): Due to random selection of prompts.
• Intrinsic error (generation noise): Arises from stochastic autoregressive sampling, reformulated as Gumbel-max noise.

This yields the main theorem: for all parameter-efficient hypotheses described in $C(h)$ bits, with high probability, the true expected reward is bounded below by the empirical reward (estimated via multiple rollouts per prompt), minus an explicitly computable penalty scaling as $\mathcal{O}\left(\sqrt{(C(h)/m)}\right)$, where $m$ is the number of training prompts and $C(h)$ is the compressed description length of the fine-tuned update.

A key innovation is leveraging the Gumbel-max reparameterization of the categorical sampling over tokens, transforming stochastic decoding into a deterministic function of the input, noise, and parameters. This permits the PAC-Bayes argument to extend "through" the generation process, bounding the accuracy and reward achieved by the stochastic model.

Progressive RLVR: A Pipeline for Achievable Compression and Tight Bounds

To obtain bounds that are not vacuous, the compression penalty must be minimized, i.e., parameter changes must be aggressively compressed while preserving post-training performance. The progressive pipeline designed for this consists of three stages:

1. RLVR Teacher Training: A high-capacity LoRA-adapted LLM is reinforced for reward maximization, unconstrained by compression limitations.
2. On-Policy Distillation: The teacher’s policy is distilled into a student architecture using TinyLoRA (with as few as one trainable scalar per adapter), supervised directly on token-level generations from the teacher to maximize statistical efficiency and coverage over the reward landscape.
3. Quantization: The TinyLoRA parameters are quantized with a small fixed codebook, drastically reducing the effective bits needed to specify the hypothesis without material reward loss.

   Figure 1: The Progressive RLVR pipeline, integrating RLVR, parameter-efficient fine-tuning, on-policy distillation, and quantization for compressibility and fidelity.

By decoupling the high-exploration search phase (teacher) from the tightly compressible student phase, this pipeline enables aggressive reduction in effective parameter modifications, a precondition for any non-vacuous PAC-Bayes compression bound at LLM scale.

Empirical Results: Non-vacuous Bounds at Scale

Application of the framework is demonstrated on four challenging domains: mathematical problem-solving, programming, general-knowledge reasoning, and Text-to-SQL. Across all domains, with Qwen3.5-4B as base model, the authors show:

• Generalization bounds (with $\delta = 0.05$) exceeding the base model’s accuracy by 9–51 absolute points.
• Tightness: The bounds remain within 6–11 points of the fine-tuned (student) model’s empirical accuracy—indicating low looseness due to the bound’s penalty decomposition.
• Compression: The quantized TinyLoRA students achieve between $84\%$ and $97\%$ of the teacher’s accuracy while being, on average, 14,796$\times$ more compressible.

  Figure 2: Non-vacuous PAC-Bayes generalization bounds across all domains, shown relative to teacher, student, and base model performances. Bounds reliably outperform base and closely track student accuracy.

A detailed decomposition of accuracy penalties clarifies the error sources: extrinsic statistical error (limited prompts), distillation/quantization loss, and a negligible intrinsic MC gap, indicating the empirical protocol (using 64 sampled completions per prompt and $k=5$ quantization) is highly efficient.

Figure 3: Decomposition of the teacher’s empirical accuracy into the generalization bound and the individual penalties, highlighting where the tightness of the final bound is lost.

Ablation Studies

Component ablations demonstrate that:

• Direct application of TinyLoRA to RLVR (without on-policy distillation) significantly loosens the bound due to degraded student performance under severe parameter constraints.
• Removal or weakening of either distillation or extreme compression yields vacuous or much looser bounds at scale.
• Progressive RLVR is necessary for any practical, non-vacuous verification of generalization—standard LoRA or naive compression is insufficient.

Implications and Limitations

Practical implications:

The methodology empowers practitioners to produce fine-tuned LLMs with formal lower bounds on their reward or accuracy, enabling verifiable specialization—critical for high-stakes or regulated domains. The pipeline and bounds offer an operational checklist for certifying specialized LLM deployment, making it possible to assert, with measurable confidence, a lower-bound on post-training performance for real-world tasks.

Theoretical implications:

This closes a core gap in LLM theory—producing the first operational, non-vacuous generalization bounds for RLVR-finetuned LLMs at billion-parameter scale. The approach sets a template for extending compression-based PAC-Bayes guarantees to RL and policy-learning regimes, including those with reparameterizable but highly structured stochasticity.

Limitations and future directions:

• The bounds only cover in-distribution generalization (i.e., when prompts at deployment are sampled from the training distribution). The OOD case requires labeled evaluation sets and is not generally covered by these PAC-Bayes results.
• Scaling progressive compression to 10B+ model sizes without non-trivial reward degradation remains open.
• The PAC-Bayes complexity penalty is sensitive to compression; in domains or tasks where reward is highly brittle with respect to parameter changes, bounds may remain loose.
• Domain transfer behavior is highly asymmetric: Mathematical RLVR produces broadly transferable representations, but induction in other domains (e.g., SQL) does not confer benefits to mathematical reasoning.

Conclusion

This work rigorously extends the PAC-Bayes compression framework to RLVR for LLMs, leveraging novel stochastic reparameterization and an operational progressive compression pipeline. The results demonstrate, for the first time, non-vacuous and tight generalization bounds for billion-parameter LLMs post-trained with RLVR. This establishes formal, empirical, and practical foundations for verified deployment of RLVR-specialized LLMs, with further advances likely as compression and policy distillation techniques evolve.