---
title: 'LLaMA-3.3-70B: Open-Source 70B Transformer'
url: https://www.emergentmind.com/topics/llama-3-3-70b-model
type: topic
---

# LLaMA-3.3-70B: Open-Source 70B Transformer

LLaMA-3.3-70B (“Llama 3.3 70 Billion”) is a large-scale, dense, decoder-only Transformer foundation model within the Llama 3 herd, open-sourced by Meta AI in 2024. It provides state-of-the-art performance across language understanding, multilinguality, coding, reasoning, math, and limited multimodal capabilities. Extensively studied both as a base pre-trained model and as subject to continual pre-training and structured post-training (alignment), LLaMA-3.3-70B serves as a highly competitive open model for research and production. Its architecture, training procedures, empirical results, adaptation strategies, and practical deployment methods are described below [2407.21783][2409.06624][2504.08791][2504.03932].

## 1. Model Architecture

LLaMA-3.3-70B is a dense single-stream causal decoder Transformer parameterized as follows [2407.21783]:

- **Layers**: 80
- **Hidden size \(d\)**: 8 192
- **Attention heads**: 64
- **Key-value heads** (Grouped-Query Attention): 8
- **Feedforward inner dimension**: 28 672
- **Parameter count formula**: \(P ≃ 12 d^2 L\); evaluated to ≈70 × 10¹⁰.
- **Vocabulary size**: 128K (includes 100K English + 28K non-English tokens, BPEmb)
- **Activation function**: SwiGLU
- **Positional encoding**: RoPE base \(\theta\) = 500 000, context size up to 128K tokens (with extended pre-training)
- **Context window**: 8K tokens in standard pre-training, extended to 128K in long-context settings

The model is a dense stack; vision/video/speech adapters can be attached for limited multimodal capabilities, though core model releases are text-only.

## 2. Pre-training, Data Curation, and Scaling Laws

### 2.1 Data Pipeline

LLaMA-3.3-70B is pre-trained on 15.6 trillion tokens from diverse multilingual sources up to end-2023 [2407.21783]:

- **Domain stratification**:
  - 50% general web text
  - 25% math/reasoning
  - 17% code
  - 8% multilingual (covering 176 languages)
- **Web pipeline**: HTML parsing, deduplication (MinHash), filters (PII/adult/domain), line-document-URL heuristics

### 2.2 Optimization Protocol

- **Objective**: standard next-token cross-entropy: \(L = −∑_{t=1}^T \log p(x_t | x_{<t})\)
- **Optimizer**: AdamW; peak learning rate for 70B: 1.5 × 10⁻⁴, decayed cosine to 1.5 × 10⁻⁶; warmup: 8 000 steps
- **Batch size**: progressive ramp-up from 4M to 8M and 16M tokens/update
- **Total tokens seen**: 15.6T (plus 0.8T for long-context annealing); compute footprint ≈3.8 × 10²⁵ FLOPs

### 2.3 Scaling Law Fit

Compute-optimal trade-off follows \(N^* (tokens) = A \cdot C^\alpha\), with (A, α) = (0.29, 0.53), \(C\) = FLOPs budget.

## 3. Post-training, Continual Pre-Training, and Domain Adaptation

### 3.1 Alignment and Preference Tuning

Post-training (alignment) is conducted via [2407.21783]:

- **Supervised Finetuning (SFT)**: 6M+ instruction–response pairs
- **Direct Preference Optimization (DPO)**: optimization against ≈300K human preference comparisons
- **Safety tuning:** SFT+DPO on adversarial prompts; “Llama Guard 3” classifier enforced at system level

### 3.2 Continual Pre-training (CPT) for Language Injection

Intensive studies [2409.06624] applied CPT to LLaMA-3.3-70B to inject substantial Chinese generation/understanding skill:

- **Corpus composition**: 1.7T tokens with ALMR (Additional Language Mixture Ratio) ≈33% (Chinese), matched from empirical sweep on 8B model
- **Hyperparameter selection**:
  - **ALMR**: \(\text{ALMR} = N_{\rm Chinese} / N_{\rm total} \in [0, 1]\)
  - **Learning Rate**: Extremely low (LR = 1 × 10⁻¹⁰) for the 70B CPT run
  - Empirically derived scaling law curves for ALMR vs. LR (see [2409.06624] for formulas)
- **Auxiliary corpora**: math and code datasets included to test cross-domain skill retention

#### CPT and SFT Performance Impact (selected metrics for 70B):

| Benchmark                  | Base | CPT  | Δ    |
|----------------------------|------|------|------|
| C-Eval (Chinese)           | 67.75| 67.89| +0.14|
| LCSTS (Chinese Summarization)| 7.19| 8.97| +1.78|
| MMLU (English)             | 79.47| 79.53| +0.06|
| GSM8K (math)               | 79.38| 81.20| +1.82|
| HumanEval (code)           | 28.05| 30.49| +2.44|

SFT and DPO stages further improved multilingual and domain metrics.

## 4. Inference and Deployment: Distributed Serving

Efficient deployment of the full 70B model on commodity clusters is described in PRIMA.CPP [2504.08791]:

- **Weight management**: mmap-based, Q4K-quantized weights loaded on-demand; memory use bounded by OS-level paging
- **Disk I/O**: Latency mitigated via madvise(WILL_NEED) paging advice and background prefetching (hides up to 80% of disk delay)
- **Piped-ring parallelism**: each device executes a consecutive window of layers per round; token generation latency is bounded by compute+communication+disk per round, optimized by window size
- **Heterogeneity-aware scheduling (“Halda” algorithm)**: Devices assigned layers to balance RAM/VRAM constraints and disk/network speed; mathematical assignment posed as integer linear fractional program, solved with factor enumeration and small-scale ILP
- **Empirical results**: On four-node Wi-Fi cluster:

| System         | Token Latency (ms) | TTFT (ms) | Max RSS (%)|
|----------------|-------------------|-----------|------------|
| llama.cpp      | 10 120            | 10 806    | >100       |
| exo            | OOM               | OOM       | OOM        |
| dllama         | OOM               | OOM       | OOM        |
| prima.cpp      | 674               | 1 793     | <6         |

Quantization (Q4K, IQ1), mmap and prefetch, and optimal k per round permit end-user serving of 70B models with low memory pressure.

## 5. Instruction Tuning, Prompting, and Mixture-of-Agents Paradigms

YaleNLP [2504.03932] benchmarked LLaMA-3.3-70B-Instruct in multi-perspective CQA summarization. The instruction-tuned variant features:

- **Instruction tuning**: RLHF-style prompts, no further architectural changes
- **QLoRA fine-tuning**: 4-bit quantization, LoRA rank = 8, α = 16, AdamW optimizer. Light QLoRA tended to degrade performance relative to zero-/few-shot off-the-shelf use, likely due to variance and small dataset size.
- **Prompting strategies**:
  - Zero-shot: task-defined instruction, no exemplars
  - Few-shot: 3 exemplars, either human-selected or embedding-retrieved (sentence-transformer, k-means clustering, cosine similarity)
- **Mixture-of-Agents (MoA)**: Ensembles of Llama-3.3-70B-Instruct and other LLMs, followed by Llama aggregator, with up to three verification/hallucination-detection layers; two layers yielded optimal tradeoff.

#### Empirical Performance (span identification/classification):

| Setting               | Overall F1 |
|-----------------------|------------|
| Zero-shot             | 0.40       |
| 3-shot (embedding)    | 0.42       |
| MoA, 2-layer          | 0.51       |
| QLoRA SFT             | 0.37       |

Embedding-based few-shot and 2-layer MoA provided significant lifts (+28–32% rel) over single-model prompting.

## 6. Capabilities, Limitations, and Safety

### 6.1 Capabilities

- **Multilingual**: native support for 8 post-trained languages; pre-training spans 176.
- **Coding**: best-in-class open HumanEval/MBPP performance (0-shot pass@1).
- **Math/Reasoning**: 95%+ GSM8K, ~95% ARC.
- **Tool Use**: zero-shot function-calling, BFCL 84.8%; Nexus 56.7%.
- **Long Context**: up to 128K context window, near-perfect recall on needle-in-haystack.
- **Multimodal Adaptation**: Vision (>90% VQA), video (82% AI2 Diagram), speech (WER 4.6% LibriSpeech); adapters, not available in core model.

### 6.2 Inference Optimization

- **Parallelism**: TP, PP, CP, DP; 38–43% BF16 MFU on 8192–16 384 GPUs. Pipeline+microbatching yields ≈30% throughput gain.
- **FP8 decode**: up to 50% latency reduction with negligible (<0.1%) quality drop.

### 6.3 Safety, Guardrails, and Limitations

- **Data filtering**: PII/adult content, document-level heuristics; memorization rates <1.1% (50-grams)
- **Post-training SFT+DPO**: adversarial/borderline prompts, VR ~10–30%, FRR ~10–50%
- **System-level**: “Llama Guard 3” classifier (VR reduced 50–90% at +30–100% FRR)
- **Limitations**: not optimized beyond major languages/modalities; adapters under development; safety tuning not fully watertight against red-teaming.

## 7. Industrial and Real-World Deployment

LLaMA-3.3-70B, after CPT and post-alignment, has powered live deployment scenarios [2409.06624]:

- **Case study**: Geely Automobile Research Institute industrial chat assistant, daily factual and emotional support; C-SFT-E (emotion-boosted SFT) variant used.
- **Satisfaction**: Turing-style test (500 dialogues), C-SFT-E score 14.16 vs. 13.60 (base Instruct); Chinese token fraction increased from <1% to 77.3%.
- **Robustness**: Strong performance on Chinese, English, mixed dialog, and emotionally sensitive tasks.
- **Accessibility**: Distributed inference via PRIMA.CPP [2504.08791] enables 70B-class models on home clusters (<6% device memory footprint), generalizing to other large-model deployments.

---

LLaMA-3.3-70B represents a mature open foundation model architecture, validated across large-scale pre-training, post-training alignment, continual language injection, advanced prompting/ensemble paradigms, and pragmatic distributed deployment. Its empirical competitive results and well-characterized adaptation strategies provide reproducible, scalable methodologies for both research and application [2407.21783][2409.06624][2504.08791][2504.03932].

Source: https://www.emergentmind.com/topics/llama-3-3-70b-model