Integrating Code in LLM Training

• Integrating code into LLM training is a systematic approach that combines executable code, curated metadata, and runtime feedback to enhance model reasoning and code generation.
• It leverages pre-training mixtures, supervised fine-tuning, and reward-based reinforcement to boost functional correctness—evidenced by metrics like pass@1 improvements.
• Optimized data pipelines using filtering, refactoring, and synthetic code generation yield practical gains in efficiency, code quality, and maintainability.

Integrating code into LLM training refers to the systematic inclusion and curation of executable code, code metadata, and runtime feedback in LLM pre-training, instruction-tuning, and data optimization pipelines. This integration leverages code’s formal syntax, logical structure, and test-driven verifiability to improve both code generation and general reasoning, with diverse techniques spanning data cleaning, augmentation, synthetic data generation, architectural adaptation, and reinforcement via execution or human feedback.

1. Foundational Strategies for Integrating Code Data

Code is incorporated into LLM training at multiple stages—(1) pre-training, (2) supervised instruction-tuning, and (3) reinforcement learning or reward optimization—each with empirically distinct impacts on model capabilities.

• Pre-training mixtures: Models such as CodePanGu2.6B use mixed corpora (∼30% code, ∼70% text) in standard next-token autoregressive cross-entropy objectives, enhancing general reasoning and code generation without negative transfer to non-code tasks (Ma et al., 2023). A typical loss is

$$\mathcal{L}_{\text{NL+Code}}(\Theta) = \alpha\,\mathcal{L}_{\text{NL}} + (1-\alpha)\,\mathcal{L}_{\text{code}},$$

with $\alpha$ controlling the code/text trade-off (Yang et al., 2024).

• Supervised fine-tuning: Instruction-tuning with a discrete code-instruction subset (often ∼30% of examples) unlocks task-specific code reasoning abilities (e.g., MBPP BLEU from 0.0 to 24.9 in CC-2.6B) (Ma et al., 2023). Mixing schedules—starting with high code ratio and tapering off—optimize performance across both code and general reasoning domains.
• RL or reward optimization: Integrating code execution feedback via reward modeling, such as pass@K benchmarks or test coverage metrics, directly optimizes for correctness and refinement (e.g., up to +3.5 pp pass@1 with RL on LeDex) (Jiang et al., 2024). Selection-based majority voting, self-debugging via prompting, and reward learning are all deployed (Yang et al., 2024).

2. Training Data Optimization and Curation Pipelines

Quality and distribution of the code data are rigorously controlled to maximize utility:

• Raw data curation: Code is harvested from permissively licensed repositories, filtered by syntax (compilation under target interpreters), style (Pylint, static analysis), and deduplication (Fujii et al., 5 May 2025, Yang et al., 2024).
• Transform-and-retain pipelines: Instead of exclusionary filtering, pipelines like SwallowCode rewrite low-quality code into consistent, well-documented, efficient, and dependency-free snippets, comprising:
  1. Syntax validation (drop uncompilable code),
  2. Style filtering (Pylint, comment density),
  3. Style rewriting (LLM-guided, e.g., Google Python Guide conformance),
  4. Self-contained optimization (LLM upgrades for efficiency and independence) (Fujii et al., 5 May 2025).

• Code cleaning and refactoring: Automated pipelines rename variables for descriptiveness, modularize monolithic functions, and insert natural-language plans, all while validating equivalence via test oracles. These steps yield >20% pass@K gains and improve sample efficiency 6x over unrefactored corpora (Jain et al., 2023).

• Synthetic data generation: LLMs are used as “teacher” models to synthesize diverse, functionally verified problem-code pairs via unit test execution and filtering (data synthesis in WaveCoder, pass@1 improvements up to 74.1% on CodeContests) (Kuang et al., 31 Dec 2025).

Pipeline Stage	Description	Empirical Impact
Syntax filtering	Remove uncompilable code	+1.8 pp pass@1
Pylint style filtering	Structure, readability via linter	+1.1 pp pass@1
LLM style rewriting	Uniform naming, docstrings, modular	+9.2 pp pass@1
LLM semantic rewriting	Self-contained, efficient code	+5.0 pp pass@1

3. Verification, Execution Feedback, and Data Selection

Modern training pipelines exploit the executable nature of code and detailed runtime output for supervision, curriculum design, and reward modeling:

• Unit test-based filtering: Candidate solutions are executed against structured test suites, with selection thresholds (e.g., $\tau\in[0.6,0.8]$ of tests passing) set to balance correctness and data diversity. Contrasting with rigid accept-all-or-nothing criteria, relaxed or soft verification recovers valuable partial-pass solutions, boosting pass@1 by 2–4 points (Gureja et al., 25 Sep 2025).
• LLM-based judgment: Large verifiers (e.g., GPT-4, Claude) are used to filter or score solutions on idiomatic correctness and plausibility, achieving comparable or superior performance to purely test-based methods (Gureja et al., 25 Sep 2025).
• Complementarity of verification and code diversity: Retaining multiple distinct correct solutions, upsampling hard problems, and fusing verification signals together (e.g., structured test, LLM score) prevent “verification ceilings” that limit generative generalization.
• Selection and ranking: Code samples are ranked on style-consistency or other heuristics (e.g., SCAR), with top $\alpha\%$ retained, lowering noise and cost (Kuang et al., 31 Dec 2025).

4. Fine-Grained Data Optimization Techniques

Empirical studies systematically evaluate combinations and individual effects of optimization strategies on code corpora, code smell, maintainability, and functional correctness:

• Data synthesis (LLM-generated code): Delivers the largest Pass@1 improvements (up to 74.1% on some benchmarks), reduces code smells (+39.9% CSS), but can lower maintainability (−5.05% MI) due to increased complexity (Kuang et al., 31 Dec 2025).
• Refactoring and cleaning: Refactoring (autoformatter pipelines) and cleaning (static + dynamic filtering) improve maintainability (up to +4.15% MI) and yield secondary gains in correctness and code smells.
• Augmentation (semantics-preserving transforms): Applies e.g., loop unrolling, variable renaming, and only retains samples passing all tests, improving robustness and structural diversity.
• Pairwise and sequential combinations: While most combinations do not outperform the best single technique for correctness, pairs—especially data synthesis plus refactoring—achieve the strongest overall performance for code quality and smell reduction. Fine-grained analysis shows that the technique yielding the highest code distribution shift (“FID”) dominates the improvements; complementarity between techniques, measured as the union of uniquely solved problems, predicts joint performance (Kuang et al., 31 Dec 2025).

5. Architectural, Objective, and Feedback Integration

The fundamental model architecture is usually unchanged (standard transformer). However, several integration points and mechanisms are prominent:

• Objective mixing: Weighted loss between natural language and code tokens, both in pre-training and instruction-tuning (Yang et al., 2024).
• Format and delimiting: Introduction of special tokens, markers for code segments, and context cues for tool calls or function boundaries (Yang et al., 2024).
• Human-in-the-loop and RL: Reward-guided tuning includes eye-tracking-derived attention motifs (for CodeT5), stepwise reinforcement on code explanations/refinements (LeDex), or execution-based scalar rewards (Jiang et al., 2024, Zhang et al., 19 Mar 2025).
• Executable code actions in agents: For LLM agents, the action space is unified as executable Python code (CodeAct), facilitating flexible tool composition and dynamic revision based on environment feedback. CodeActAgent models trained with execution-embedded transcripts substantially outperform competitors on compositional agent benchmarks (up to 20 pp gain and 2–3 fewer turns) (Wang et al., 2024).

6. Benchmarking, Metrics, and Empirical Outcomes

Empirical evaluation employs functional correctness (pass@K), code quality (e.g., CSS—code smell score), and maintainability (Maintainability Index MI), across diverse benchmarks (APPS, CodeContests, MBPP, HumanEval):

• Functional correctness: pass@K,

$$\mathrm{pass@}k(p) = 1 - \frac{\binom{M-c}{k}}{\binom{M}{k}}$$

where $M$ is total generations, $c$ correct ones. SwallowCode moves HumanEval pass@1 from 37.0% (Stack-Edu) to 54.0%, a 17 pp gain under identical training budget (Fujii et al., 5 May 2025).

• Code smells and maintainability: Weighted sums of errors, warnings, conventions, Halstead metrics, and cyclomatic complexity. Data synthesis and code selection consistently reduce code smells; refactoring, cleaning, and selection improve MI (Kuang et al., 31 Dec 2025).
• Reasoning transfer: Pre-training with ∼30% code raises reasoning accuracy on logic and legal QA by up to 5 pp over text-only models of similar size (Ma et al., 2023).

7. Challenges, Limitations, and Open Research Directions

Despite dominant empirical gains, several limitations and open questions remain:

• Verification bottlenecks: Overly strict test criteria suppress diversity; future pipelines must increase test complexity (contrastive/generated by multiple LLMs) while relaxing thresholds to admit partial-pass and stylistically diverse solutions (Gureja et al., 25 Sep 2025).
• Cross-language and generalization: Most experiments remain Python-centric; effectiveness for polyglot or domain-specific code, or at scale (>3 B params), is not fully characterized (Kuang et al., 31 Dec 2025).
• Task granularity: Pipelines are currently optimized for single-function or problem-level code; module- and system-level code generation may require advanced data structuring and joint verification strategies (Kuang et al., 31 Dec 2025).
• Human-centric reward integration: Programmer eye-tracking and cognitive cues yield summarization gains (+7.16 CodeBLEU), but are expensive and so far limited to small datasets (Zhang et al., 19 Mar 2025). Scalability and cross-task transferability remain open.
• Interplay of code properties: Disentangling the relative contributions of syntax, modularity, and executability to general reasoning improvements is an unresolved research target (Yang et al., 2024).
• Adaptive data pipelines: Multi-technique, adaptive optimization (beyond static pairwise combinations) and smarter data post-filtering warrant further study.

---

In sum, integrating code into LLM training is a multifaceted process, involving tailored pre-training mixtures, robust data curation and transformation, execution-driven verification, fine-grained optimization strategies, and architectural or objective adaptations to exploit code properties. This practice yields quantifiable advances in code generation, general reasoning, maintainability, and agentic tool use, with current research continuing to optimize the alignment of code data attributes with evolving LLM capabilities (Kuang et al., 31 Dec 2025, Fujii et al., 5 May 2025, Yang et al., 2024, Ma et al., 2023).