Reward-Free Code Alignment from Pretrained or Fine-Tuned LLM: Unpacking the Trade-offs for Code Generation

Abstract: LLM alignment trains an LLM using preference data to produce outputs that better meet established quality standards. While LLM alignment techniques are studied for non-coding tasks, we know little about their usefulness for coding tasks. It is unclear whether LLM code alignment could support both functional requirements (producing executable, correct code) and non-functional requirements (code readability, style, maintainability). It is also unknown whether alignment for a code LLM should begin with base pretrained version or the finetuned (i.e., instruction-tuned) version of the LLM. In this paper, we offer insights on the above two research questions by conducting an empirical study. We studied five state-of-the-art (SOTA) LLMs using two widely used LLM alignment techniques: Direct Preference Optimization (DPO) and BoNBoN. For each training record, we created a preference pair as accepted and rejected instances by using the SelfCodeAlign pipeline. DPO and BoNBoN are reward-free models, i.e., they eliminate the need for multiple reward scores for output preferences. We tuned each LLM using the two alignment techniques in two settings: pretrained and finetuned versions of an LLM. We evaluated functional requirements using four SOTA benchmarks (HumanEval+, MBPP+, EvalPerf, EvoEval) and non-functional requirements using the CODAL benchmark, which evaluates code quality across five dimensions derived from software engineering practices. We find that pretrained-to-aligned pathways achieve larger improvements in the aligned variant over its pretrained variant. But the pretrained variant is generally less accurate than its finetuned variant. However, finetuned- to-aligned offers smaller performance improvements or, in some cases, degradation in the aligned variant than its finetuned variant.

• The paper demonstrates that reward-free alignment using DPO and BoNBoN significantly improves non-functional code quality, especially in pretrained-to-aligned (PTA) models.
• The paper reveals that PTA pathways yield high relative improvements with risks of catastrophic degradation, while fine-tuned-to-aligned (FTA) models offer stable but modest gains.
• The paper provides actionable recommendations, stressing that model selection and careful technique adoption are crucial to balance plasticity and stability in code generation.

Reward-Free Code Alignment for Code Generation: Empirical Trade-offs and Model Pathways

Motivation and Problem Definition

Despite impressive advances in LLM-based code generation, there remain gaps between syntactic correctness and holistic code quality, which includes both functional and non-functional requirements. The former refers to properties such as code correctness and performance, while the latter includes readability, maintainability, and adherence to industry standards. Existing alignment protocols predominantly target text generation, with limited focus on code-specific alignment objectives. A critical question arises in practical alignment of code LLMs: whether alignment should proceed from a base pretrained model or from an instruction-tuned variant, and what trade-offs are involved regarding functional and non-functional requirements.

Alignment Techniques for Code: Reward-Free Approaches

The study benchmarks two reward-free preference optimization strategies—Direct Preference Optimization (DPO) and BoNBoN—against a suite of code LLMs. Reward-free alignment eliminates the dependency on the reward model used in classical RLHF, achieving efficient, stable, and sometimes superior results for preference-driven optimization. DPO reframes LLM alignment as a classification task, focusing on direct preference learning from paired accepted/rejected outputs. BoNBoN internalizes the best-of-N selection distribution, avoiding costly inference-time sampling via multi-stage supervised fine-tuning and IPO-style preference contrastive learning.

Figure 1: Reinforcement learning from human feedback pipeline for reward-based LLM alignment, illustrating preference model training and policy optimization.

Figure 2: Best-of-N approach, visualizing the selection process for optimal outputs during alignment.

Methodology and Evaluation Protocol

An empirical examination is conducted across five state-of-the-art open-weight code LLMs (Meta-Llama-3-8B, Qwen2.5-Coder-7B, CodeLlama-7b, deepseek-coder-1.3b, deepseek-coder-7b) paired with their instruction-tuned variants. Alignment is evaluated along two axes: the pretrained-to-aligned (PTA) and finetuned-to-aligned (FTA) pathways. Preference datasets are generated using a modified SelfCodeAlign pipeline, producing explicit pairs for both functional requirements (evaluated on HumanEval+, MBPP+, EvalPerf, EvoEval) and non-functional requirements (assessed with CODAL across instruction-following, readability, complexity, style, and explanation). The study uses relative improvement and Pass@1 success as key metrics, with CODAL serving as an LLM-as-judge holistic non-functional benchmark.

Empirical Results and Key Observations

Functional Requirements

• PTA models show substantial relative improvements post-alignment (e.g., CodeLlama-7b: +75% non-functional, Llama3-8b: +42% functional), but absolute performance remains lower than FTA variants.
• FTA models (already instruction-tuned) yield modest average improvements (functional: 4.9%), and effects are often saturated or even degraded depending on model family and alignment technique.
• Success rates for functional improvements are highly model-dependent, with catastrophic degradation observed in several cases (e.g., CodeLlama-7b: -40%).

Non-Functional Requirements

• Alignment consistently enhances non-functional code quality (non-functional: +10.6% average in FTA, up to +75% in PTA), suggesting higher responsiveness and stability compared to functional alignment.
• Code-specialized models outperform general-purpose baselines in non-functional alignment, with larger models achieving stronger absolute performance.
• DPO and BoNBoN demonstrate model-specific preferences; families exhibit technique-specific alignment efficacy (Meta-Llama favoring DPO, deepseek preferring BoNBoN).

Comparative Pathways and the Stability-Plasticity Trade-Off

• PTA provides higher plasticity, enabling large relative improvements from low baselines but with increased susceptibility to catastrophic forgetting.
• FTA offers high stability, preserving prior instruction-following capabilities, but with sharply constrained improvement margins and higher risk of degradation under alignment.
• Non-functional objectives respond more reliably to alignment than functional objectives, regardless of pathway.
• Risks associated with alignment are tightly linked to the initial model state and the selected alignment technique; SFT-stage degradation predicts overall alignment failure.

Practitioner Validation and LLM-Judge Reliability

A practitioner survey (N=30) validates the significance of PTA vs. FTA pathway selection and highlights the joint importance of relative and absolute improvements. Empirical participants confirm prioritization of non-functional alignment and the complex, task-dependent nature of pathway choice. The validity of LLM-as-judge for non-functional evaluation is substantiated via a 200-instance reliability analysis, demonstrating substantial agreement ($\kappa=0.635$ at tolerance $\pm3$) between human annotation and GPT-3.5-turbo, as compared to GPT-4 and Claude-3.

Recommendations

The study presents nine actionable recommendations informed by empirical findings:

• Select PTA when maximizing improvement magnitude, FTA for stable high absolute performance.
• Prioritize non-functional alignment objectives due to consistently higher responsiveness.
• Focus on code-specialized models, preferably larger variants ($\geq$7B parameters) for alignment.
• Establish minimum baseline performance thresholds before triggering alignment.
• Leverage empirical validation for model-objective-technique combinations.
• Apply early stopping based on SFT performance degradation to mitigate alignment failure risks.
• Monitor multi-dimensional performance to detect trade-offs during alignment.

Implications and Future Prospects

This work clarifies fundamental trade-offs in reward-free code LLM alignment and delivers concrete guidance for model and pathway selection. The dichotomy between stability and plasticity informs both practical deployment and theoretical understanding of preference-driven LLM tuning. As larger closed-weight LLMs demonstrate emergent behaviors, future research should extend coverage to additional architectures, refine non-functional benchmarks, and explore iterative alignment-feedback loops for continuous performance optimization.

Conclusion

Preference-based reward-free alignment (DPO, BoNBoN) delivers consistent gains in non-functional code quality but exhibits variable success for functional requirements, contingent on model pathway and architecture. Pretrained-to-aligned yields strong relative improvements, but absolute performance lags instruction-tuned pathways, which, while more stable, offer limited improvement and higher risk of degradation. Non-functional enhancement remains the most reliable target for code LLM alignment. These insights underpin nuanced recommendations for model selection and technique adoption in code synthesis applications (2606.28998).