Magicoder: Code LLMs and ADR Encoding

• Magicoder is a dual-purpose system that comprises transformer-based LLMs for code generation and a lightweight NLP algorithm for ADR encoding in pharmacovigilance.
• It employs instruction-tuning and synthetic data generation to achieve state-of-the-art efficiency in code synthesis and review, with benchmarks showing significant improvements over larger models.
• Its ADR encoding component uses unsupervised token matching and multi-criteria ranking to accurately map free-text narratives to MedDRA standard terms, reducing manual errors and coding time.

Magicoder is the designation for two distinct families of automated systems in computer science and biomedical informatics: (1) advanced transformer-based LLMs for code generation and review, and (2) a lightweight, unsupervised NLP algorithm for automating adverse drug reaction (ADR) encoding in pharmacovigilance. Both lines of work share an automated, coverage-based approach to translating free-form human input into structured, standardized outputs optimized for downstream analysis or automation.

1. Overview of Magicoder for Code Generation

The code-focused Magicoder series comprises open-source LLMs instruction-tuned for code synthesis, understanding, and review, based on oscillator improvements in supervised synthetic data generation pipelines. The flagship Magicoder models use a decoder-only transformer backbone, with no architectural modifications to enhance performance, relying exclusively on instruction-tuning regimes and high-quality, synthetic code instruction datasets. Predominant checkpoints include CodeLlama-7B and DeepSeek-Coder-6.7B (Wei et al., 2023).

OSS-Instruct is the primary methodology for dataset creation, leveraging open-source code snippets to prompt advanced teacher models (e.g., gpt-3.5-turbo-1106) to generate coding tasks and their reference implementations. Resultant datasets (∼75 K instruction–solution pairs) are heavily filtered to ensure diversity, decontaminated from evaluation data, and cover multiple programming languages (Python ~50%; C++, Java, TypeScript, Shell, C# each 5–10%). Diversity is quantified by low cosine similarity to benchmarks such as HumanEval, indicating effective mitigation of distributional bias present in LLM-generated synthetic data.

Training employs standard left-to-right, causal language modeling objectives, with cross-entropy loss: $$\mathcal{L}(\theta) = -\sum_{(x,y)\in D}\sum_{t=1}^{|y|} \log p_\theta(y_t \mid x, y_{<t})$$ where $x$ is the code prompt and $y$ is the solution.

Highly efficient infrastructure (2×A100 80GB GPUs) enables two full epochs of training (OSS-Instruct stage) and up to two additional epochs with the complementary Evol-Instruct dataset for the enhanced Magicoder$$variant. All model code, weights, and data are released openly.</p> <h2 class='paper-heading' id='empirical-performance-and-evaluation'>2. Empirical Performance and Evaluation</h2> <p>Magicoder models have been evaluated under standard code synthesis metrics, most notably pass@1 on HumanEval(+), MBPP(+), MultiPL-E, and DS-1000 benchmarks (<a href="/papers/2312.02120" title="" rel="nofollow" data-turbo="false" class="assistant-link" x-data x-tooltip.raw="">Wei et al., 2023</a>). Magicoder$$-7B, after OSS+Evol-Instruct tuning, achieves 66.5% pass@1 on HumanEval+, surpassing ChatGPT (gpt-3.5-turbo) at 65.9%, despite having only 7 B parameters. Across multilingual and data science code tasks, Magicoder (especially OSS+Evol variants) significantly outperforms open LLMs of similar or larger scale (e.g., WizardCoder-15B, StarCoder-15B).

Ablations demonstrate that mixing Python and non-Python samples critically benefits cross-language generalization; isolated Python tuning boosts HumanEval but underperforms on multilingual sets. Further, direct fine-tuning on conventional docstring–function pairs yields no improvement, underscoring the importance of OSS-Instruct’s problem–solution design. The combination with orthogonal Evol-Instruct samples provides an additional 10–15% absolute improvement in top-1 solve rate.

In code review automation contexts, Magicoder (6.7B) achieves strong results when fine-tuned with low-rank adaptation (DoRA, r=16, α=8) on only 6% of standard review datasets, reaching up to a 73% improvement in exact match (EM) over Guo et al.'s baseline (Pornprasit et al., 1 Feb 2024). However, GPT-3.5 (six times larger) retains a performance advantage of 8–12 EM points when equivalently fine-tuned.

Benchmark	Magicoder-7B (OSS+Evol)	ChatGPT (gpt-3.5-turbo)
HumanEval+	66.5%	65.9%
DS-1000	37.5%	n/a

A plausible implication is that high-quality, diverse synthetic problems anchored in authentic code sources enable small LLMs to rival or surpass larger proprietary models on realistic benchmarks.

3. Prompt Engineering and Adaptation for Code Review

Magicoder supports parameter-efficient domain adaptation via prompt design and low-resource fine-tuning. Empirical protocol includes zero-shot and few-shot templates, with or without "persona" (simulated expertise) prompts (Pornprasit et al., 1 Feb 2024). For automated code review:

• Zero-shot prompts provide a direct code-plus-comment-to-improved-code mapping, optionally prefaced with a persona header.
• Few-shot templates include exemplar Q&A pairs (3, retrieved via BM25).
• Persona inclusion consistently degrades EM, thus its use is discouraged.

Fine-tuning with DoRA consistently yields the best performance, but when computational budgets are limited, few-shot prompting (3 BM25-selected examples, no persona) is an effective secondary strategy.

4. Magicoder for ADR Encoding in Pharmacovigilance

An unrelated line of work, MagiCoder (Combi et al., 2015, Combi et al., 2016), is an efficient NLP algorithm for mapping spontaneous ADR narratives to MedDRA standard terms. Unlike the transformer-based Magicoder, MagiCoder relies on linear-time, unsupervised token matching, lightweight stemming, and multi-criteria ranking.

The processing pipeline comprises:

1. Tokenization and stop-word removal, followed by light stemming.
2. Candidate term matching using two hash-tables: exact word→LLT and stemmed word→LLT.
3. A voting mechanism to record token–LLT overlaps, distinguishing between exact and stem matches.
4. Multi-key ranking of candidate LLTs by coverage, type, string-distance, density, and distribution.
5. Selection yields a shortlist (≤6) of MedDRA LLTs for expert validation.

Empirical results on ∼6,800 Italian ADR narratives show 81% exact match (PT level) for shortest descriptions (≤20 chars), 62% for 20–100 chars, and 61% for 100–250 chars. More granular precision/recall, as measured in (Combi et al., 2016), reaches 86%/88% recall/precision for very short texts and 64%/63% for medium-long narratives.

Length (chars)	Recall	Precision
0–20	86%	88%
21–40	72%	75%
41–100	61%	62%
101–255	58%	52%
>255	46%	45%

MagiCoder is deployed as a VigiWork plugin, resulting in sub-second processing per ADR text, and demonstrably reduces manual coding time and errors.

5. Design Principles, Limitations, and Future Directions

Magicoder (code LLMs) is notable for achieving state-of-the-art or near-SOTA results through careful data curation rather than architectural novelty. All improvements derive from the structure, diversity, and filtering of instruction data—particularly via OSS-Instruct and its combination with Evol-Instruct (Wei et al., 2023). Open questions include scalability to higher-parameter regimes, fine-tuning with more powerful teachers (GPT-4), and adaptation to multimodal code synthesis incorporating documentation and testing.

Magicoder-based code review automation highlights the importance of fine-tuning over prompt-only adaptation, although prompt engineering remains valuable when data or compute is limited. A key takeaway is that the "persona" prompt detracts from exact match performance in this context (Pornprasit et al., 1 Feb 2024).

For MagiCoder ADR encoding, limitations include the absence of explicit negation detection, a lack of built-in spell checking, limited synonym expansion, and performance degradation on complex narratives. Proposed extensions include negation-flagging, integration of medical spell-checkers, synonym expansion under expert supervision, and adaptation to additional languages and terminologies.

6. Practical Impact and Release Policy

Magicoder’s code LLMs are fully open-source (including code, weights, and the OSS-Instruct dataset), enabling direct community adoption and extension. The efficiency of Magicoder for ADR encoding has realized measurable improvements in pharmacovigilance throughput and annotation quality by reducing ADR coding time tenfold and mitigating manual error rates (Combi et al., 2015, Combi et al., 2016).

The continued open release of both code LLMs and annotation tooling reflects a trend toward democratization of both biomedical natural language processing and code intelligence in computational research.