An empirical study of LoRA-based fine-tuning of large language models for automated test case generation

Abstract: Automated test case generation from natural language requirements remains a challenging problem in software engineering due to the ambiguity of requirements and the need to produce structured, executable test artifacts. Recent advances in LLMs have shown promise in addressing this task; however, their effectiveness depends on task-specific adaptation and efficient fine-tuning strategies. In this paper, we present a comprehensive empirical study on the use of parameter-efficient fine-tuning, specifically LoRA, for requirement-based test case generation. We evaluate multiple LLM families, including open-source and proprietary models, under a unified experimental pipeline. The study systematically explores the impact of key LoRA hyperparameters, including rank, scaling factor, and dropout, on downstream performance. We propose an automated evaluation framework based on GPT-4o, which assesses generated test cases across nine quality dimensions. Experimental results demonstrate that LoRA-based fine-tuning significantly improves the performance of all open-source models, with Ministral-8B achieving the best results among them. Furthermore, we show that a fine-tuned 8B open-source model can achieve performance comparable to pre-fine-tuned GPT-4.1 models, highlighting the effectiveness of parameter-efficient adaptation. While GPT-4.1 models achieve the highest overall performance, the performance gap between proprietary and open-source models is substantially reduced after fine-tuning. These findings provide important insights into model selection, fine-tuning strategies, and evaluation methods for automated test generation. In particular, they demonstrate that cost-efficient, locally deployable open-source models can serve as viable alternatives to proprietary systems when combined with well-designed fine-tuning approaches.

• The paper introduces a comprehensive empirical pipeline assessing LoRA fine-tuning to transform natural language requirements into executable test cases.
• It demonstrates significant performance gains with optimal hyperparameters, achieving up to 71.34 scores in open-source models and parity with GPT-4.1 models.
• The study proposes cost-efficient, scalable solutions for test case generation that enhance customization, data privacy, and reproducibility in industrial settings.

LoRA-Based Fine-Tuning of LLMs for Requirement-Based Test Case Generation: Empirical Analysis and Implications

Motivation and Context

Automated test case generation from natural language requirements confronts long-standing obstacles rooted in ambiguity, variability, and the need for structured, executable artifacts. Historically, heuristic-based and traditional ML approaches, despite incremental progress, have failed to generalize across domains and requirement styles, often requiring extensive manual rule engineering or feature design. LLMs, pre-trained on large corpora of natural language and code, have recently shown marked improvement in context understanding and structured output generation but remain suboptimal without task-specific adaptation. Further, the practical deployment of billion-scale models necessitates parameter-efficient strategies due to computational, memory, and cost constraints.

Experimental Framework and Methodology

The study constructs a comprehensive empirical pipeline assessing LoRA-based fine-tuning for automated requirement-to-test transformation. Both open-source models (DeepSeek-1.8B, Llama-3.1-8B, Ministral-8B) and proprietary GPT-4.1 variants (Full, Mini, Nano) are included. The dataset comprises 2,583 real-world requirement-test case pairs for training and 288 held-out samples for evaluation, ensuring coverage of various testing scenarios including UI, API, integration, and configuration management.

A unified prompt schema is employed for all fine-tuning experiments, delineating expectations for test case name, type, description, and ordered steps (explicitly supporting API-specific information when required). LoRA adapters are injected into selected transformer layers, with systematic variation in rank, scaling factor, and dropout to interrogate both performance gains and efficiency. Fine-tuning is conducted under consistent hardware and software environments, with all processes governed by reproducibility-focused protocols.

The study introduces an automated evaluation framework leveraging GPT-4o as a deterministic judge, scoring generated outputs across nine advanced criteria (semantic similarity, information coverage, critical content match, structural accuracy, omission, hallucination, ambiguity, redundancy, diversity/novelty) on a [0,100] scale. This enables granular assessment and cost-effective scalability.

Empirical Results

Open-Source Models

Across all open-source baselines, Ministral-8B demonstrates superior initial performance (overall score: 65.88), outperforming Llama-3.1-8B (63.77) and DeepSeek-1.8B (62.85) and showing strongest results for structural accuracy, hallucination control, ambiguity, and redundancy. LoRA-based fine-tuning yields statistically significant improvements for all open-source models, with Ministral-8B reaching an overall score of 71.34 under optimal configuration (a=8, rank=16, dropout=0.5), exceeding Llama's maximum (67.93) and DeepSeek's maximum (66.82). Improvements are most pronounced for semantic alignment, information coverage, and critical content preservation.

Optimal LoRA configuration for open-source models generally favors lower scaling factors and moderate ranks, efficiently balancing adaptation capacity and regularization. Excessive scaling (a=32) induces performance degradation, likely due to instability or overfitting. Ministral-8B also exhibits enhanced robustness to hyperparameter changes compared to Llama and DeepSeek.

Proprietary GPT-4.1 Models

Proprietary GPT-4.1 and GPT-4.1-mini variants display markedly high baseline scores (≥74 overall), besting all open-source models pre-fine-tuning. After fine-tuning, GPT-4.1-mini achieves the highest overall performance (79.74), predominately driven by gains in information coverage, critical content, and diversity/novelty compared to GPT-4.1 (78.32). The GPT-4.1-nano variant, while competitive in baseline, demonstrates notable decline post-fine-tuning (64.71), underscoring the sensitivity of smaller models to adaptation instability for complex structured generation.

Critically, Ministral-8B—when fine-tuned with LoRA—achieves performance parity with pre-fine-tuned GPT-4.1 models, a bold claim corroborated by comprehensive evaluation metrics. This empirical demonstration supports the assertion that open-source models, if coupled with properly configured parameter-efficient techniques, can reach near-proprietary levels for requirement-based test generation.

Comparative Analysis

The study establishes a clear hierarchy in performance: proprietary GPT-4.1 models lead, followed by fine-tuned Ministral-8B, then Llama and DeepSeek. The gap between open-source and proprietary is substantially narrowed post-fine-tuning, especially for Ministral-8B. This undermines the prevailing assumption that proprietary models are fundamentally superior for specialized tasks and points to a new minimum viable model size for competitive structured artifact generation.

Practical and Theoretical Implications

Scalability, Customization, and Data Privacy

The results validate that smaller, open-source models—when fine-tuned with LoRA—can be cost-efficiently deployed locally, granting organizations greater control, customization, and enhanced privacy assurance compared to API-bound proprietary systems. This is especially impactful for enterprise software testing, where frequent requirement changes and data governance are paramount.

Fine-Tuning Hyperparameter Tuning

The strong dependency of downstream performance on LoRA configuration confirms that fine-tuning is not merely an implementation detail but a decisive factor in output quality. Practitioners must prioritize systematic tuning of rank, scaling, and dropout to optimize both learning dynamics and structural generation. Parameter-efficient adaptation allows for scalable experimentation with multiple domain-specific adapters without duplicating base model weights.

Automated Evaluation and Reproducibility

Automated evaluation, powered by advanced LLMs, enables large-scale and consistent scoring for complex tasks that lack unique ground truth, minimizing reliance on costly, subjective human review. The nine-dimensional evaluation schema adopted here provides actionable insight into both overall and criterion-specific strengths and weaknesses of models and configurations, facilitating robust comparison and future benchmark design.

Future Directions

Future work should further interrogate minimum parameter counts for competitive task adaptation, investigate additional PEFT techniques (adapters, prefix tuning, prompt tuning), and extend evaluation frameworks to incorporate meta-evaluation (e.g., cross-domain generalization). Research on mitigating fine-tuning-induced instabilities in lightweight models is necessary, as is expanded study into mixed-modality requirement artifacts (integrating code and UI specifications). Multi-task and incremental fine-tuning pipelines may also enable more granular adaptation for rapidly evolving industrial testing environments.

Conclusion

This paper provides a rigorous empirical assessment of LoRA-based parameter-efficient fine-tuning strategies for LLM-driven requirement-based test case and test step generation (2604.06946). The results strongly support LoRA’s effectiveness in bridging the performance gap between open-source and proprietary models, with Ministral-8B achieving parity with pre-fine-tuned GPT-4.1 under optimal settings. The study underscores the importance of systematic hyperparameter tuning and demonstrates the viability of scalable, LLM-based automated evaluation. The practical feasibility of cost-efficient, locally deployable solutions for automated test generation is established, with implications for industrial adoption, reproducibility, and future advances in software engineering automation.