- The paper develops a Tonel-aware, AST-guided pipeline using continued pre-training and Random-AST fine-tuning on 387,159 Pharo methods to improve completion in a severely low-resource language.
- Specialized 3B–7B models achieve up to 89.04% pass@1 on AST-aware HumanEval+ tasks and outperform models over 60 times larger on realistic Random-AST completion.
- Relevant context from impacted method bodies raises the 7B model’s repository-level scores from 60.05% to 75.96% ChrF and from 35.96% to 58.99% CrystalBLEU, while quantization reduces memory use by about 70% with minimal accuracy loss.
Motivation and problem setting
Code LLMs derive their capabilities from large public code corpora, and their performance is strongly correlated with the amount of training data available per language. The authors quantify this imbalance for Pharo, a Smalltalk-inspired language: while Python has roughly 26 million public GitHub repositories, Pharo has about 2 thousand — four orders of magnitude fewer. Even languages previously treated as "low-resource" in the literature (Lua with 620k repositories, Julia with 85k, Racket with 23k) exceed Pharo by at least an order of magnitude. As a consequence, Pharo's IDE (the Complishon engine) offers only single-token completion, far from the multi-token, context-aware suggestions available for mainstream languages.
Pharo poses three compounding challenges beyond raw data scarcity. First, its code is serialized in Tonel files that interleave executable code with packaging metadata, and class definitions in Tonel differ syntactically from those shown in the IDE — a format likely to induce spurious patterns if fed to models unprocessed. Second, Smalltalk syntax differs structurally from mainstream languages: control flow constructs such as if and while are ordinary message sends rather than language keywords, method signatures interleave arguments with selector keywords, and roughly half of all methods contain three or fewer lines of code. These properties hinder transfer learning from high-resource languages. Third, the target use case is in-IDE completion under strict latency constraints, which rules out heavyweight techniques such as retrieval-augmented generation designed for offline code generation.
The study addresses one research question: to what extent can the code completion capabilities of LLMs be enhanced for a severely low-resource language like Pharo?
Specialization pipeline
The authors adapt two families of open-weight fill-in-the-middle (FIM) models: Qwen2.5 Coder Base (0.5B, 1.5B, 3B, 7B parameters; trained with a prefix–suffix–middle objective) and Mellum-base (4B; suffix–prefix–middle). Both are small enough for local deployment — the quantized 7B model occupies only 4.3 GiB.
Data curation. From 748 MIT-licensed GitHub repositories tagged "Pharo", filtering for compatibility with Pharo 10–14 (verified by actually loading each repository into five Pharo images) and for Tonel format yields 415 repositories. Repositories created before June 1, 2024 form the training set; later ones serve as held-out evaluation data to limit contamination from base-model pre-training corpora. Methods whose 8-grams overlap with benchmark canonical solutions were removed from training.
Tooling. Two reusable artifacts were built: a Pharo lexer contributed to Pygments and a tree-sitter grammar for Pharo, enabling AST extraction from Tonel files. Together they yielded 387,159 methods.
Training. A two-step procedure using LoRA (alpha 32, r 16, dropout 0.05, sequence length 2,048, AdamW, three epochs):
- Continued pre-training uses causal language modeling on full methods for 25% of instances and AST-aware span masking (3–10 token spans at AST node boundaries) conditioned on prefix and suffix for the remaining 75%, following evidence that FIM-heavy pre-training improves completion.
- Supervised fine-tuning applies a Random-AST masking strategy — masks start at arbitrary tokens and extend to the end of the enclosing statement — which matches how developers request completions at arbitrary cursor positions. To prevent forgetting, 20% of fine-tuning instances are rehearsed AST-aware samples. The final fine-tuning set contains 324,725 instances.
A notable finding here is that pre-training alone often hurts performance on realistic Random-AST tasks (e.g., Qwen2.5 Coder 3B drops from 44.47% to 25.26% pass@1 on HumanEval+ r-AST), and only fine-tuning closes this gap. This indicates that FIM competence acquired at AST boundaries does not transfer automatically to arbitrary cursor positions.
Benchmarks
Two complementary evaluation suites were constructed:
Method-level benchmarks assess syntactic competence via test-executable tasks. HumanEval+ was translated to Pharo through GPT-4o-assisted translation followed by manual verification by two authors, producing 164 tasks; 47 Exercism Pharo exercises were made self-contained. Each task is instantiated under both AST-aware and Random-AST masking, yielding 5,087 FIM tasks evaluated by executing completed code against test suites in a Pharo 13 image. Pass@1 is averaged over 20 repetitions at temperature 0.2, with McNemar tests, Benjamini–Hochberg correction, and odds ratios.
Repository-level benchmark simulates realistic edits: 2,185 tasks mined from commits in the 22 held-out repositories, where spans of 3–10 tokens are masked from newly added AST nodes. Four context policies are compared — no context, class signatures, package signatures, and impacted methods (bodies of other methods modified in the same commit), capped at 8,000 tokens — plus a "random methods" control. Evaluation uses ChrF and CrystalBLEU against developer-written code, since no test suites exist for real commits; significance is assessed with Wilcoxon signed-rank tests and Cliff's delta.
Method-level results
Specialization produces large, statistically significant gains. Representative pass@1 figures under AST-aware masking:
| Model |
HumanEval+ base → SFT |
Exercism base → SFT |
| Qwen2.5 Coder 0.5B |
32.52 → 62.16 |
20.43 → 55.61 |
| Qwen2.5 Coder 1.5B |
29.77 → 78.66 |
17.67 → 71.87 |
| Qwen2.5 Coder 3B |
71.48 → 83.73 |
64.85 → 78.11 |
| Qwen2.5 Coder 7B |
71.12 → 89.04 |
68.58 → 85.84 |
| Mellum 4B |
59.41 → 69.81 |
52.24 → 66.99 |
The most striking result concerns scale. On the more realistic Random-AST benchmarks, the specialized 3B and 7B models significantly outperform Qwen3 Coder 480B A35B Instruct and Claude Sonnet 4.5 — models over 60× larger — despite the large models retaining an edge on AST-aware tasks. Qwen2.5 Coder 7B - SFT reaches 52.76% pass@1 on HumanEval+ r-AST versus 45.13% for Qwen3 Coder 480B and 51.53% for Claude Sonnet 4.5. This demonstrates that specialization can compensate for scale specifically for low-resource languages, which matters because deployment constraints favor compact models.
Error analysis supports the diagnosis: 65.6% of base-model failures are syntax errors (versus 17.9% unexpected exceptions and 16.5% assertion failures), and the training procedure reduces syntax errors by 33% on average across the five models. A worked example shows base models breaking parenthesization and message precedence in simple arithmetic expressions that the specialized 7B model reconstructs correctly in 20/20 runs.
Repository-level results and context selection
Base models perform poorly at repository level, partly because they emit extraneous tokens requiring error-prone post-processing; the paper reports unprocessed scores, noting that post-processing narrows but does not eliminate the gap.
Context quality dominates context quantity. Class and package signatures yield marginal, often non-significant improvements. Providing impacted-method bodies, by contrast, boosts Qwen2.5 Coder 7B - SFT from 60.05% to 75.96% ChrF (+15.91%) and from 35.96% to 58.99% CrystalBLEU (+23.03%). The random-methods control beats signature-based contexts in 37 of 48 comparisons yet never matches impacted methods, isolating semantic relevance — not merely additional Pharo code — as the operative factor. The authors note this policy is directly implementable in an IDE by tracking recently edited methods.
Against large baselines, even the 1.5B specialized model with impacted-methods context surpasses Qwen3 Coder 480B (~320× more parameters); the 3B and 7B versions exceed it by +7.52% ChrF and +8.63% CrystalBLEU on average. Claude Sonnet 4.5 remains the strongest model overall (83.02% ChrF, 70.52% CrystalBLEU with impacted methods), but the authors flag a contamination concern: its knowledge cut-off is undisclosed, and its high no-context scores suggest possible familiarity with the test repositories. This caveat weakens the comparison against Claude specifically.
Latency and deployability
Q4_K_M quantization of the 7B model reduces memory from 14.19 GiB to 4.36 GiB (~70%) with negligible accuracy loss: pass@1 decreases by 0.61% on average at method level, and repository-level metrics are unchanged or slightly improved. Latency measurements on consumer hardware show the quantized 7B model completing in ~1.33 s on Apple M3/M4 Max CPUs and 0.53 s on an RX 7800XT GPU, while the unquantized 3B model runs in 0.50–0.73 s. The 7B CPU latency slightly exceeds the sub-second target typically desired for interactive completion, so the smaller models offer the better latency–accuracy trade-off. The evaluation covers only high-specification machines and employs no prompt-caching optimizations, so these figures are pessimistic for professional hardware but do not cover low-end machines.
Limitations and open questions
Several limitations qualify the results. Contamination cannot be fully excluded, particularly for the commercial baselines with unknown cut-off dates. Method-level tasks derive from masked reference solutions and may not reflect interactive editing conditions, though Random-AST masking partially mitigates this; additionally, no fine-tuning-only ablation was run, justified by prior evidence that FIM capability is best acquired during pre-training. Repository-level evaluation relies on surface similarity (ChrF, CrystalBLEU) rather than functional correctness, and constitutes a simulation of editing behavior rather than a live user study measuring suggestion acceptance. Finally, latency was measured on a narrow set of high-end machines.
Open questions left by the paper include whether the pipeline transfers to other Smalltalk dialects sharing Tonel, and whether the specialized models deliver practical value under real developer interaction once integrated into the Pharo IDE — work toward which a plugin is reportedly in progress.
Conclusion
This case study establishes an end-to-end recipe — Tonel-aware data curation, AST-aware continued pre-training, Random-AST fine-tuning with rehearsal, and relevance-based context selection — for bringing multi-token LLM completion to a severely low-resource language. The central empirical claim is well supported: specialized models of 3B–7B parameters substantially outperform their base checkpoints and match or exceed general-purpose models hundreds of times larger on Pharo completion, while remaining deployable on consumer hardware. The accompanying infrastructure (parser, lexer, benchmarks, harnesses) lowers the barrier for analogous efforts on other underrepresented languages, though the substantial manual engineering effort involved suggests such specialization remains labor-intensive.
