--- title: 'MagicoderS-CL Variant: Open Code LLM' url: https://www.emergentmind.com/topics/magicoders-cl-variant type: topic --- # MagicoderS-CL Variant: Open Code LLM MagicoderS-CL Variant is a fully open-source code-generation large language model (LLM) built by fine-tuning Meta’s CodeLlama-7B foundation model on synthetic instruction data generated via the OSS-Instruct methodology. MagicoderS-CL aims to create high-quality and diverse instruction-following code datasets from open-source code corpora and demonstrates state-of-the-art performance among 7B-parameter and larger open models, surpassing even some proprietary models on rigorous code-generation benchmarks [2312.02120]. ## 1. Model Architecture and Initialization MagicoderS-CL adopts the CodeLlama-7B architecture, a single-stream, decoder-only Transformer without any structural modifications. The core model characteristics are as follows: | Specification | Value | Source | |----------------------------|-------------------------------|---------------------| | Transformer layers | 32 | CodeLlama-7B | | Hidden dimension ($d_{model}$) | Approx. 4,096 | CodeLlama-7B | | Attention heads per layer | 32 | CodeLlama-7B | | Feed-forward size | Approx. 11,008 | CodeLlama-7B | | Context window | Up to 2,048 tokens | CodeLlama-7B | No adapters or new layers are introduced. Weight initialization is performed by loading the pretrained CodeLlama-7B weights directly, with no further re-initialization. This distinguishes MagicoderS-CL from MagicoderS, which is instead initialized from DeepSeek-Coder-Base. The training strategy involves two-stage instruction tuning: 1. OSS-Instruct dataset (75,000 instances, 2 epochs). 2. Optional Evol-Instruct continuation (2 further epochs), resulting in MagicoderS-CL⁺. No additional pretraining or further architectural changes are employed [2312.02120]. ## 2. OSS-Instruct Data Generation and Curation OSS-Instruct is a methodology for producing diverse synthetic instruction data grounded in open-source code references. Its primary pipeline involves: - **Seed Corpus**: "starcoderdata" is the seed, a filtered permissively licensed subset of The Stack. Random snippets of 1–15 consecutive lines are extracted from 80,000 documents. - **Prompting and Synthesis**: Each seed snippet is formatted into a prompt and fed to gpt-3.5-turbo, requesting the creation of a realistic programming problem "inspired by" the snippet and a self-contained solution with minimal explanatory commentary. Greedy decoding is employed to maximize consistency. - **Data Cleaning/Decontamination**: Duplicates and any overlap with evaluation benchmarks (HumanEval(+), MBPP(+), DS-1000, APPS, GSM8K) are removed; only 9 samples filtered in this step. - **Diversity Analysis**: Ten manually defined code categories (e.g., algorithms, ML, scripts) ensure a balanced representation; prompt and solution lengths vary from a few tokens to ~1,000 tokens. - **Bias Measurements**: Average TF-IDF cosine similarity to HumanEval is lowest among compared synthetic datasets, indicating minimal prompt duplication or content drift. Formally, the data generation process is: ```python for s in starcoderdata: prompt = fill_template(s) x, y = gpt_3_5_turbo.generate(prompt) if not duplicate(x, y): add_to_dataset(x, y) ``` The instruction-tuning loss is standard cross-entropy: $$ L_{inst}(\theta) = -\sum_{(x, y) \in D} \log P_\theta(y \mid x) $$ ## 3. Training Objective and Optimization The principal objective is standard next-token prediction on solutions given the prompt, using token-level cross-entropy: $$ L = -\sum_t \log P(w_t \mid w_{