Llama 3 8B Model: Architecture & Performance

• Llama 3 8B is an open-weight foundation language model featuring a dense Transformer architecture with 8.1B parameters designed for diverse multilingual and programming tasks.
• It incorporates innovations like Grouped Query Attention, SwiGLU activations, and high-base Rotary Positional Embeddings for enhanced efficiency and extended context support.
• Pre-trained on 15.6 trillion tokens and fine-tuned with advanced safety and alignment techniques, it achieves state-of-the-art performance within its parameter class.

The Llama 3 8B model is an open-weight, foundation LLM belonging to the Llama 3 family, designed around a Transformer backbone with 8.1 billion parameters. It is configured for efficient training and inference, multilingual and programming workloads, long-context comprehension, advanced alignment, and safety applications. It is positioned as a best-in-class performer among models of similar scale (7–9B parameters) and has been extensively benchmarked for both standard academic tasks and domain-specific adaptation scenarios.

1. Architecture and Structural Properties

Llama 3 8B employs a dense Transformer architecture comprising 32 layers, each with a hidden size of 4 096, a feed-forward dimension of 14 336 (expansion factor ≈ 3.5×), and 32 attention heads (128-dimensional, with 8 key/value heads via Grouped Query Attention). Key elements include:

• Grouped Query Attention (GQA): Reduces inference latency and KV cache requirements.
• SwiGLU Activation: Found in all MLP layers for improved throughput and convergence.
• Rotary Positional Embeddings (RoPE): Set with base frequency θ = 500 000, supports sequences up to 128 K tokens.
• Extended Vocabulary: 128 K tokens (100 K English/BPE + 28 K non-English), Tiktoken-based; each embedding has dimension 4 096.
• Layer Normalization:

$$\mathrm{LayerNorm}(x) = \frac{x-\mathbb{E}[x]}{\sqrt{\mathrm{Var}[x]}+\epsilon}\cdot\gamma+\beta$$

• Attention: Both single- and multi-head variants, where multi-head aggregates via concatenation and projection:

$$\text{MHA}(x) = \text{Concat}_h[\text{Attention}(W_h^z x, W_h^k x, W_h^v x)] W^o$$

Relative to earlier LLaMA variants, Llama 3 8B introduces GQA for throughput, larger vocabulary for enhanced multilingual reach, and a higher RoPE base for long-context retention (Grattafiori et al., 2024).

2. Pre-Training and Data Regimen

The Llama 3 8B model was pre-trained on approximately 15.6 trillion tokens using a diversified data mixture:

• 50% general web data: Filtered for PII, adult content, and deduplicated at the line, document, and URL levels with model-based quality assessment.
• 25% math/reasoning: Specialized crawls and filtering for problem-solving content.
• 17% code: Curated for syntax and correctness, heavily represented for coding competency.
• 8% multilingual: From 176 languages, language-ID and quality filtered to reinforce non-English proficiency.

The tokenization pipeline uses Tiktoken with byte-pair encoding, expanded for non-English scripts, yielding ≈ 3.94 chars/token in English. Pre-training is performed in three stages:

• Stage 1: Next-token prediction with context length ramping from 4 096 to 8 192 tokens, ultimate batch size 16 M tokens; optimizer is AdamW with cosine LR annealing.
• Stage 2: “Continued” pre-training progressively increases context to 128 K tokens; ≈ 800 billion additional tokens.
• Stage 3: Annealing stage finalizes with 40 M high-quality tokens and linear LR decay. Total pre-training compute is 3.8 × 10²⁵ FLOPs (≈ 50× Llama 2) (Grattafiori et al., 2024).

3. Post-Training, Alignment, and Safety

Post-training and alignment of Llama 3 8B consists of a multi-round pipeline:

• Supervised Fine-Tuning (SFT): Instruction tuning with 52.7% general, 14.9% code, 21.2% reasoning/tools, 8.1% exam, 3.0% multilingual, 0.1% long-context data.
• Rejection Sampling (RS): Generation of 10–30 candidates per prompt; selection via reward model scoring; throughput doubled via PagedAttention.
• Direct Preference Optimization (DPO): Preference-gradient fine-tuning on edited/chosen/rejected output triplets (β = 0.1), with NLL regularization (0.2×). Final checkpoints are produced by “ModelSoup” averaging of top-K runs.

Safety integration is achieved via Llama Guard 3, an 8B-parameter classifier for input/output filtering. The integration achieves violation rate reductions of 30–76% across benchmarks compared to peers (Grattafiori et al., 2024).

Llama 3 8B has also been analyzed in alignment workflows such as Constitutional AI (CAI). On CAI-benchmarked tasks, the model exhibits a 40.8% reduction in attack success rate (from 71% to 42%) at a cost of a 9.8% drop in average helpfulness scores (Zhang, 7 Apr 2025). A new collapse failure mode was observed: after DPO-CAI, the model sometimes enters infinite polite-sentence loops with emojis, attributed to overfitting during supervised revision. Larger models (e.g., 52B-parameter) did not exhibit this collapse, suggesting scale-dependent brittleness (Zhang, 7 Apr 2025).

4. Performance Benchmarks and Comparative Evaluation

Llama 3 8B demonstrates state-of-the-art or near state-of-the-art results in its parameter class:

Task	Llama 3 8B	Gemma 2 9B	Mistral 7B	Llama 3 70B	GPT-3.5 Turbo	Llama 3 405B	GPT-4
MMLU (General)	69.4	72.3	61.1	83.6	70.7	87.3	85.1
0-shot CoT MMLU	73.0	72.3	60.5	86.0	69.8	88.6	85.4
HumanEval (Code)	72.6	54.3	40.2	80.5	68.0	89.0	86.6
GSM8K (Math)	84.5	76.7	53.2	95.1	81.6	96.8	94.2
ARC (Reasoning)	83.4	87.6	74.2	94.8	83.7	96.9	96.4
BFCL (Tool use)	76.1	—	60.4	84.8	85.9	88.5	88.3
Long Context QA/Recall	81.0	—	—	90.5	—	95.2	95.2
Multilingual (MGSM, 0-shot CoT)	68.9	—	—	—	—	—	90.5

Per-parameter efficiency (e.g., MMLU% per billion) is maximized at 8.7%/B for 8B vs. 0.22%/B for 405B; inference costs are ≈ 20× lower for the 8B variant, which can be deployed on commodity hardware (Grattafiori et al., 2024).

5. Linguistic Adaptation and Low-Resource SFT

Empirical benchmarking for adaptation to underresourced languages is exemplified by systematic evaluation on Romanized Nepali using QLoRA+rsLoRA methods. Key results:

• Baseline (Zero-Shot) Failure Mode:
  • Llama 3.1-8B typically yields null outputs (semantic void) due to Tiktoken over-fragmentation, with immediate EOS on 6/10 prompts.
• Post-Fine-Tuning Outcomes:
  • Post-QLoRA+rsLoRA, Llama-3.1-8B achieves PPL = 3.024 (Δ = –49.77), BERTScore = 0.7511 (Δ = +0.3287), chrF++ = 26.97 (Δ = +18.82), indicating strong semantic and structural gains.
  • Fine-tuning utilizes 4-bit NF4 quantized weights and rank-32 LoRA adapters with α/√r scaling (≈ 1.03% trainable parameters).
• Comparative “Adaptation Headroom”:
  • The model’s weak zero-shot baseline provides maximal absolute improvement after tuning.
  • Qwen3-8B attains best post-SFT alignment metrics and faster convergence, but Llama-3.1-8B is preferred for iterative development in low-resource settings where subsequent data collection and repeated fine-tuning are expected (Rimal et al., 25 Mar 2026).
• Efficiency:
  • Full SFT of Llama 3.1-8B (on dual T4 GPUs) requires <27 GPU-hours, demonstrating practical accessibility for small-scale labs.

6. Context Length and Scaling Laws

Llama 3 8B natively supports context windows up to 128 K tokens. The use of RoPE at θ = 500 000 maintains positional awareness at this scale.

• Long-context performance is evidenced by 98.8% recall on Needle-in-a-Haystack benchmarks and 81.0% on ZeroSCROLLS/QuALITY (5-shot EM).
• Scaling: While the largest Llama 3 model (405B) achieves higher absolute performance, the 8B variant reaches a per-inference and per-parameter optimum, particularly for deployments constrained by hardware or energy costs (Grattafiori et al., 2024).
• Alignment Failure at Scale: Smaller models lack the robustness against spurious training signal collapse and can overfit on subtle prompt artifacts, as seen in emoji-driven collapse in CAI training (Zhang, 7 Apr 2025). This suggests a scale threshold for emergent self-critique ability.

7. Implications, Limitations, and Recommendations

Llama 3 8B achieves “best-in-class” status among its immediate peers for code, reasoning, and general QA, with demonstrated extensibility to long-context and low-resource language adaptation. Native integration with safety classifiers such as Llama Guard 3 enables broad deployment in mission-critical domains, provided pipeline-specific overfitting risks are managed.

Key implications for low-resource and safety-critical pipelines:

• For structural quality in single-round SFT, Qwen3-8B is recommended.
• For iterative, data-driven pipelines demanding maximal adaptation headroom, Llama-3.1-8B is optimal.
• Thorough curation of finetuning outputs (e.g., removal of over-represented artifacts like emojis) and potential hybridization with stronger AI or human feedback mitigate collapse risk in alignment workflows (Rimal et al., 25 Mar 2026, Zhang, 7 Apr 2025, Grattafiori et al., 2024).

Open questions remain regarding optimal preference optimization regime, minimal robust model scale for feedback-driven self-improvement, and systematic collapse avoidance in small-alignment settings.