Exploring different approaches to customize language models for domain-specific text-to-code generation

Abstract: LLMs have demonstrated strong capabilities in generating executable code from natural language descriptions. However, general-purpose models often struggle in specialized programming contexts where domain-specific libraries, APIs, or conventions must be used. Customizing smaller open-source models offers a cost-effective alternative to relying on large proprietary systems. In this work, we investigate how smaller LLMs can be adapted for domain-specific code generation using synthetic datasets. We construct datasets of programming exercises across three domains within the Python ecosystem: general Python programming, Scikit-learn machine learning workflows, and OpenCV-based computer vision tasks. Using these datasets, we evaluate three customization strategies: few-shot prompting, retrieval-augmented generation (RAG), and parameter-efficient fine-tuning using Low-Rank Adaptation (LoRA). Performance is evaluated using both benchmark-based metrics and similarity-based metrics that measure alignment with domain-specific code. Our results show that prompting-based approaches such as few-shot learning and RAG can improve domain relevance in a cost-effective manner, although their impact on benchmark accuracy is limited. In contrast, LoRA-based fine-tuning consistently achieves higher accuracy and stronger domain alignment across most tasks. These findings highlight practical trade-offs between flexibility, computational cost, and performance when adapting smaller LLMs for specialized programming tasks.

• The paper compares few-shot prompting, retrieval-augmented generation, and LoRA fine-tuning for StarCoder-1B and DeepSeekCoder-1.3B across Python, Scikit-learn, and OpenCV code generation.
• LoRA fine-tuning produces the most consistent improvements, including a 30-point OpenCV Pass@1 gain for DeepSeekCoder and up to 14.7-point embedding-similarity gains for StarCoder on Scikit-learn.
• Synthetic-data validation, benchmark limitations, and model-dependent results show that fine-tuning is promising for private local deployment, while retrieval and prompting require careful example selection.

Problem and scope

General-purpose code LLMs often produce syntactically valid programs that nonetheless misuse specialized library APIs. This paper empirically compares three adaptation strategies for smaller open-source models—few-shot prompting, retrieval-augmented generation (RAG), and LoRA fine-tuning—in three Python domains with differing API specialization: general Python programming, Scikit-learn workflows, and OpenCV computer vision tasks (2603.16526). The study is motivated by deployment constraints (privacy, cost, local execution) that make frontier proprietary models unattractive, and by the observation that the relative trade-offs among prompting-, retrieval-, and weight-update-based customization remain under-characterized for domain-specific code generation.

The experimental design holds the programming language constant while varying domain knowledge requirements, isolating the effect of specialization on API-convention following rather than on language syntax.

Synthetic data pipeline as distillation

Because high-quality instruction–code pairs are scarce for specialized libraries, the authors use a teacher-student distillation setup: GPT-4o generates programming exercises paired with Python implementations, structured via controlled prompt variables (topic, profession, skill level, user interaction, error handling), with topics bootstrapped from official documentation and expanded by the teacher itself. Approximately 21.6k exercises per domain were produced for roughly $374 in API cost (~9.7M input tokens, ~35M output tokens).

A two-stage validation pipeline filters generated samples: AST parsing enforces syntactic validity, and module inspection verifies that all imported modules and attribute chains exist. Retention rates exceed 92% in all domains (96.8% Python, 92.4% OpenCV, 98.6% Scikit-learn). The validated corpora (~20–21k samples each) are split 97/1/2 into training, validation, and test sets, yielding ~400 test samples per domain. Notably, validation covers imports and attribute existence but does not execute the generated solutions against unit tests; semantic correctness of the exercise bodies themselves is not fully guaranteed.

Models and evaluation framework

Two decoder-only code models serve as students: StarCoder-1B and DeepSeekCoder-1.3B, both trained with Fill-in-the-Middle objectives on large permissively licensed GitHub corpora. Evaluation combines two complementary axes:

• Functional correctness: Pass@1 on HumanEval (164 problems) for general Python, and BigCodeBench subsets BCSk (152 tasks) for Scikit-learn and BCCV (10 tasks) for OpenCV.
• Domain alignment: cosine similarity between embeddings of generated and reference solutions using all-MiniLM-L6-v2, computed on validation splits during fine-tuning and on test splits for final comparison.

All experiments use greedy decoding for reproducibility.

Results

DeepSeekCoder-1.3B is the stronger baseline across benchmarks (30.5% vs. 16.0% Pass@1 on HumanEval; 20.0% vs. 0.0% on OpenCV). Baseline weakness on library-specific tasks confirms that correct API usage, not syntax, is the bottleneck motivating customization.

Few-shot learning yields modest similarity gains but inconsistent accuracy changes: it improves DeepSeekCoder HumanEval performance by +8.5 points, yet degrades StarCoder's Scikit-learn score from 13.2 to 3.9 (-9.3) and drops DeepSeekCoder's OpenCV score to 0.0 (-20.0). Adding more examples beyond three to five introduces noise within context-window limits.

RAG (three retrieved examples, cosine threshold 0.5) raises domain alignment consistently—for StarCoder-1B up to +11.7 similarity points on Scikit-learn—but its benchmark impact is mixed, including a -8.6 point drop for StarCoder on Scikit-learn. The paper attributes this to retrieved examples introducing unnecessary operations that harm functional correctness while improving stylistic alignment.

LoRA fine-tuning ($r = \alpha = 128$; ~50–57M trainable parameters) delivers the most consistent gains across both metric types:

Method	Model	HumanEval Pass@1	Scikit-learn Pass@1	OpenCV Pass@1
LoRA	StarCoder-1B	+2.3 (18.3%)	+7.2 (20.4%)	+20.0 (20.0%)
LoRA	DeepSeekCoder-1.3B	+7.9 (38.4%)	+1.0 (33.9%)	+30.0 (50.0%)

Similarity gains under LoRA are likewise the largest observed (+14.7 for StarCoder on Scikit-learn; +13.5 on Python). A practically useful finding is that validation similarity correlates with HumanEval Pass@1 during training, supporting embedding-based similarity as a cheap proxy signal for checkpoint selection without running test suites.

An important interaction emerges between base model strength and strategy: DeepSeekCoder benefits more from in-context methods, whereas StarCoder shows larger gains from weight updates, suggesting fine-tuning can partially compensate for weaker pre-training.

Limitations and open questions

Several constraints qualify these results. The BCCV benchmark contains only 10 tasks, so its reported Pass@1 values (including the +30.0 LoRA gain) carry substantial variance and should be interpreted cautiously. Validation of synthetic data checks syntax and API existence but does not execute solutions against tests, so some training examples may be functionally incorrect. RAG depends on embedding quality and a fixed similarity threshold; no learned or hybrid retrieval strategies are explored. Finally, only two ~1B-parameter models and one teacher model are studied—the generality of the finding that fine-tuning dominates prompting-based approaches at other scales remains open.

Conclusion

This work provides a systematic empirical comparison of few-shot prompting, RAG, and LoRA fine-tuning for domain-specific code generation with small open-source models, supported by an inexpensive synthetic-data pipeline (under $400 per corpus) and a dual evaluation framework combining Pass@1 with embedding-based domain alignment. The central result is that LoRA fine-tuning on synthetic data yields the largest and most consistent improvements in both functional correctness and domain alignment, while prompting-based methods improve stylistic fit at limited or negative benefit to benchmark accuracy. The dependence of optimal strategy on base-model pre-training quality, and the reliability of similarity metrics as selection signals, merit further investigation.