Leveraging Language Models for Log Statement Generation in Multilingual Scenarios: How Far Are We?

Abstract: Log statements capture critical information for software maintenance activities such as testing, debugging, and failure analysis. Because of this importance, developers must carefully design log statements, which requires significant effort. To support developers, various end-to-end automated log statement generation approaches have been proposed, whereas these approaches have mainly been evaluated within a single programming language environment and their effectiveness in multilingual environments remains underexplored. In this paper, we therefore comparatively evaluate three state-of-the-art log statement generation approaches and five LLMs across multiple programming languages. For this purpose, we constructed a multilingual benchmark comprising 150,000 instances across five programming languages. Our empirical results demonstrate that UniLog, a state-of-the-art approach, achieves the best overall performance, maintaining high effectiveness even in multilingual environments. We also observe substantial variance in the difficulty of log generation across languages: Python presents a greater challenge, whereas JavaScript yields comparatively better performance. Detailed analysis reveals that these disparities stem from variations in log insertion distributions and language-specific logging idioms. Our findings indicate that simply scaling model size or the volume of training data is insufficient for multilingual log generation; rather, designing approaches tailored to the specific characteristics of target languages is crucial. These findings suggest that future automated logging techniques should explicitly account for language-specific logging characteristics to achieve robust performance in multilingual software development environments.

• The paper introduces a 150,000-instance benchmark across five programming languages and finds UniLog achieves the highest average end-to-end accuracy at 20.35%, surpassing FastLog, LANCE, and general-purpose LLMs.
• The paper shows that retrieval-based prompting and minimal warmup can outperform larger training budgets, while monolingual adaptation generally beats multilingual training when total data is fixed.
• The paper links language performance gaps to loop density, logging-level conventions, and message diversity, and shows that semantic evaluation identifies nearly 40% of outputs as correct despite exact-match metrics missing them.

Motivation and scope

Log statements underpin testing, debugging, and failure analysis, yet their design—insertion location, severity level, and message content—remains a manual, effort-intensive task. While end-to-end automated log statement generation has advanced rapidly with LLMs (LANCE, FastLog, UniLog), prior evaluations have been confined almost entirely to Java (and occasionally Go). This paper by Kusama et al. addresses that gap through a large-scale empirical study of three state-of-the-art end-to-end approaches and five LLMs across five programming languages: Java, Python, JavaScript, TypeScript, and C#. The study is organized around three research questions: (RQ1) multilingual performance, (RQ2) the impact of training strategies, and (RQ3) how language-specific logging characteristics explain cross-language performance differences.

Benchmark construction

The authors mined GitHub repositories using the Dabic et al. sampling methodology, filtering to projects with at least 500 commits, 10 contributors, and 10 stars, restricted to representative logging libraries per language (Log4j for Java; Python's standard logging; Winston/Pino for JavaScript and TypeScript; NLog/log4net/Serilog for C#). Methods were extracted with a token-length filter ($\le$512 tokens), deduplicated via hashing, formatted with language-standard tools (Black, Prettier, dotnet format), and converted into instances via leave-one-out extraction, where each instance removes exactly one target log statement while retaining others as context. Log levels were normalized to six bands (Critical/Fatal through Trace/Verbose).

The resulting benchmark comprises 150,000 balanced instances—24,000 training, 3,000 validation, and 3,000 test instances per language—with file-level splitting to prevent leakage. The deliberate balancing is methodologically important: it disentangles performance differences attributable to language characteristics from those attributable to training-data volume. The full unbalanced dataset (over 1.3 million instances) is also released.

RQ1: Multilingual performance

Under unified conditions, UniLog achieved the best overall performance, with an average All Accuracy (simultaneous correctness of position, level, and message) of 20.35% across the five languages, versus 9.31% for FastLog and 5.63% for LANCE. Among general-purpose LLMs, DeepSeek-V3 led with an average All Accuracy of 10.96%. The UniLog–DeepSeek-V3 gap is statistically significant on the combined test set (McNemar test, $p < .001$; odds ratio = 8.30), supporting the claim that task-specific design—retrieval-based few-shot prompting plus lightweight warmup—outperforms raw model capability in this domain.

A central finding is the substantial variance in difficulty across languages. JavaScript is consistently easiest (UniLog All Accuracy of 44.83%) whereas Python is hardest (8.30%). Notably, FastLog's relative standing flips by language: its BLEU exceeds UniLog's on Java but it collapses on Python (Position Accuracy of 38.90% vs. UniLog's 61.20%). This directly implies that single-language evaluations are insufficient for judging approach effectiveness—a conclusion consistent with prior multilingual studies in vulnerability repair and detection, where success rates varied sharply across languages.

Self-hosted 7B–8B models proved competitive on individual subtasks (e.g., Llama3's message BLEU of 15.96 exceeded GPT-4.1 mini's 11.10), though their end-to-end All Accuracy remained well below both DeepSeek-V3 and UniLog.

RQ2: Training strategies

Using Llama3 as backbone, the authors compared monolingual LoRA (Mono-LoRA), multilingual LoRA trained on matched totals (Multi-LoRA-24k) and scaled data (Multi-LoRA-120k), and analogous Mono/Multi variants of UniLog-L and FastLog. Three results stand out:

• Data efficiency of warmup: Mono-UniLog-L, adapted with only 500 instances per language, achieved the highest average All Accuracy (16.24%), outperforming Mono-LoRA (15.13%, trained on 24,000 instances per language) and Multi-LoRA-120k (15.94%, trained on 120,000 total instances). This is a strong, somewhat counterintuitive result: adaptation strategy dominates training scale.
• Monolingual beats multilingual at fixed budget: when total training data is held constant, monolingual training consistently outperformed joint multilingual training (e.g., Mono-FastLog 9.31% vs. Multi-FastLog 5.22%; Mono-UniLog-L 16.24% vs. Multi-UniLog-L 12.80%). The per-language data reduction outweighs any benefit from cross-language knowledge sharing. The authors note this comparison controls data volume rather than operational cost, since monolingual training requires separate runs per language.
• Position prediction scales with data; level and message do not: increasing per-language instances from 4,800 to 24,000 improved Position Accuracy by 4.59 points—the largest gain among metrics—while Level and Message Accuracy improved only marginally. Level and message prediction appear governed more by approach design than by data volume.

RQ3: Language-specific factors

To explain the JavaScript/Python gap, insertion locations were classified into six categories (try-catch, branching, looping, method start/end, domain-specific methods) using GPT-5.2 annotation validated against human labels (Cohen's $\kappa$ = 0.95 between annotators; 88.8% agreement with the model).

For position prediction, Looping Block was systematically hardest (26.3% accuracy overall), and loop density correlated negatively with position accuracy (Pearson $r=-0.46$). Python exhibits the highest loop density (0.341 loops/line; 23.0% of methods contain loops) versus JavaScript's lowest (0.075; 6.30%), partially explaining why Python is harder. For level prediction, category dependence is weaker, but typical levels within the same category differ sharply by language—for example, Error dominates try-catch logging in JavaScript (78.8%) but accounts for only 44.7% in Python—so uniform language-independent heuristics are unlikely to suffice. For message generation, cross-language gaps are largest, and ground-truth message diversity (distinct-2) correlates negatively with accuracy: JavaScript messages are formulaic in high-scoring categories, whereas Python messages are diverse, making template learning insufficient there.

Additional analyses

Three discussion analyses extend the main findings. First, cross-language transfer: in-language fine-tuning won for four of five languages, but the margin varies dramatically—from 6.11 points for JavaScript to essentially zero for C# (16.67 vs. 16.65), implying language-specific data may be unnecessary in some cases. Transferability does not follow simple syntactic similarity (Java was the best source for Python and vice versa). Second, a multi-log-statement setting with 200 functions per language revealed severe degradation for Llama3: average function-level Perfect accuracy of just 0.10%, Message Accuracy below 2% everywhere, and precision (22.82%) far below recall (49.69%), indicating systematic overprediction of insertion sites. This shows single-log evaluation understates practical difficulty. Third, an LLM-as-a-Judge analysis (GPT-5.2, validated against human annotation at 92.5% agreement, $\kappa$=0.93) found that 39.9% of UniLog's generated messages were semantically equivalent to targets despite surface differences—nearly double the exact-match Message Accuracy of 22.74%—while only 0.3% were meaningless. Exact-match metrics thus substantially underestimate capability, whereas n-gram metrics can overestimate it by rewarding lexical overlap on semantically divergent outputs.

Limitations and open questions

The authors acknowledge several constraints. UniLog had to be reimplemented (no public code exists; Codex is deprecated, so GPT-4.1 mini served as backbone), introducing possible fidelity gaps; LANCE's original implementation required modification, validated only within three percentage points of reported performance. Potential data leakage exists because API-based models' pre-training corpora may overlap the benchmark. The LLM selection (five models) may not generalize, and the balanced per-language instance distribution, while fair for comparison, departs from natural repository distributions. Open questions include how to balance shared versus language-specific logging knowledge in training, and how to design approaches for multi-log-statement scenarios, where current end-to-end frameworks largely fail.

Conclusion

This study provides the first unified multilingual evaluation of automated log statement generation, supported by a released 150,000-instance benchmark spanning five languages. Its principal conclusions are that task-specific design (retrieval-based prompting plus minimal warmup) outperforms both larger fine-tuning budgets and raw LLM capability; that monolingual adaptation beats multilingual joint training under fixed data budgets; and that cross-language performance gaps are systematically explained by insertion-category distributions, loop density, level conventions, and message diversity. The finding that scaling data or model size alone is insufficient—and that semantic evaluation reveals far higher message quality than exact match suggests—sets a clear agenda for language-aware logging approaches and semantics-aware evaluation protocols.