---
title: IBM Granite-3.3-8b-Instruct Model
url: https://www.emergentmind.com/topics/ibm-granite-3-3-8b-instruct-model
type: topic
---

# IBM Granite-3.3-8b-Instruct Model

IBM Granite-3.3-8b-Instruct is an 8-billion-parameter, decoder-only Transformer-based large language model (LLM) developed by IBM Research. Released open-source under the Apache 2.0 license, it serves as the instruction-tuned flagship in the Granite series for code intelligence, and forms the backbone of multi-modal and high-efficiency inference systems. The model is optimized for code-oriented tasks, including code generation, editing, explanation, repository-level understanding, and is effective across both short and extreme long-context inferencing up to 128K tokens. Granite-3.3-8b-Instruct also underpins speech-aware variants with advanced automatic speech recognition (ASR) and speech translation capability, and has been vertically integrated into specialized inference accelerator platforms for low-latency, high-throughput enterprise deployment [2405.04324, 2407.13739, 2505.08699, 2511.15950].

## 1. Model Architecture and Variants

Granite-3.3-8b-Instruct employs a standard decoder-only Transformer stack with ≈8 billion learnable parameters. The core architecture implements:

- 32–36 Transformer blocks (exact count depends on release variant and paper; e.g., 36 for code models [2405.04324], 32 for speech-aware [2505.08699])
- Model dimension $d_\mathrm{model} = 4096$, feed-forward inner dimension $d_\mathrm{ff}$ of 14,336 or 16,384
- 32 attention heads with grouped-query attention (GQA), utilizing 8 key-value heads per group
- Rotary positional embeddings (RoPE) with support for both short (4K) and long (128K) context lengths
- SwiGLU or GELU activations and RMSNorm or LayerNorm, as per variant
- Byte-pair encoding (BPE) tokenizer, vocabulary size 49,152

Instruction tuning does not alter the architecture—"instruct" variants are pure finetunes of the respective base models. For the 128K-token variant, no extra layers or adapters are introduced; RoPE base frequency and long-context pretraining alone enable extended context [2407.13739].

The acoustic extension in Granite-speech-3.3-8B-Instruct appends a Conformer-based encoder (10 blocks, block attention) for speech inputs, a windowed query Transformer ("Q-former"), and LoRA adapters injected into the attention projections for efficient speech-text alignment. Text-only operation always reverts to the original Granite-3.3-8b-Instruct weights, preserving instruction-tuned capabilities and model safety [2505.08699].

## 2. Training Data, Objectives, and Instruction Tuning

**Pretraining** of Granite-3.3-8b-Instruct covers ≈4 trillion code tokens from 116 programming languages, followed by ≈500 billion tokens from an 80/20 code-to-natural language mixture to boost reasoning abilities. Data curation includes StarCoder-inspired quality/duplication filters, HAP/PII redaction, and excludes malware [2405.04324].

The loss function is a mixture of causal language modeling ($L_\text{CLM}$) and fill-in-the-middle ($L_\text{FIM}$) objectives, $\mathcal L = ½ L_\text{CLM} + ½ L_\text{FIM}$ for pretraining, switching to pure CLM for instruction tuning. Training uses AdamW with fine-tuned hyperparameters, large effective batch sizes, and heavy reliance on hardware-efficient components like FlashAttention 2, sequence and pipeline parallelism, activation checkpointing, and BF16 precision.

**Instruction-tuning** is performed on a blend of:

- Human Git commit-message ↔ code diffs (CommitPackFT, 92 languages)
- Math instruction datasets (MathInstruct, MetaMathQA)
- Code assistant dialogues (Glaive-Code-Assistant-v3, Self-OSS-Instruct-SC2)
- Synthetic and real code-based instruction data (NL2SQL, API-BLEND)
- General natural language instructions (HelpSteer, OpenPlatypus)

For long-context (128K) instruction-tuning, repository-level "file packing" and length up-sampling create multi-turn, long-range Q&A covering class/function comprehension and synthesis tasks, balancing short- and long-context samples (≈50/50 mix). Learning rates and batch sizes are set to maximize context exposure per update [2407.13739].

## 3. Context Scaling: 4K to 128K Tokens

To support up to 128K-token contexts, Granite-3.3-8b-Instruct undergoes a multi-stage continual pretraining schedule:

- RoPE base frequencies are progressively increased in five stages (e.g., base: $10^5$ for 8K, $10^7$ for 128K), each with 500 training steps and batch size 32.
- Repository-level file packing concatenates all files in a repo into single, long training documents.
- Sequences shorter than 4K are downsampled; lengthy documents are upsampled to encourage long-context learning.
- Multi-turn synthetic instructions drive effective optimization for retrieval, understanding, and completion over extreme context windows.

Performance degrades sharply for models lacking frequency adjustment or repository packing. Incorporating synthetic long-context Q&A during instruction tuning is pivotal for retrieval and comprehension tasks above 4K tokens [2407.13739].

## 4. Evaluation and Benchmarking

Granite-3.3-8b-Instruct exhibits competitive and in some settings state-of-the-art performance among open-source 7B/8B LLMs, especially in code-centric evaluation. Key metrics include pass@1 (functional code generation), exact match (EM) and edit similarity (ES), and fuzzy/execution-based code correctness. 

**Short-context (≤4K) benchmarks:**

| Benchmark                   | Metric           | Instruct 8B Result (%)     |
|-----------------------------|------------------|----------------------------|
| HumanEval (Synth)           | pass@1           | 49.6                       |
| MBPP (Python)               | pass@1           | 49.6                       |
| RepoBench (Python/Java)     | EM / ES          | 31.8 / 69.5 (Py), 38.4 / 76.4 (Java) |
| SantaCoder-FIM (Infilling)  | Exact Match      | 76.6                       |
| HumanEvalPack (Explain/Fix) | pass@1           | 40.9 / 40.4                |
| CanItEdit (Python Edit)     | pass@1           | 39.7 (descriptive)         |
| GSM8K+Py (Math Reasoning)   | acc.             | 63.1                       |

**Long-context (up to 128K):**

- Long code completion (balanced) EM: Upwards of 57.4% at 32K tokens, 44.5% for RepoBench-P at 32K
- RepoQA @ 16K context: 68.0% (vs 10.0% for 4K)
- Key-retrieval (needle-in-haystack): Near-perfect up to 128K tokens
- Minimal (<1%) performance loss on short-context tasks for 128K variant [2407.13739]

**Speech mode evaluation (ASR, AST):**  
Granite-speech-3.3-8B achieves WER of 1.5% (LibriSpeech clean), 7.0% (CommonVoice), with BLEU on speech translation within 5 points of state-of-the-art specialist models, outperforming closed-source systems on some English ASR corpora [2505.08699].

## 5. Inference, Quantization, and Efficiency

IBM Granite-3.3-8b-Instruct is designed for both standard GPU and domain-specific accelerator deployments:

### GPU:
- FlashAttention 2, padding-free transformers, and activation checkpointing enable 4096-token context at practical inference speeds on 80GB A100/H100 GPUs [2405.04324].
- Supports mixed-precision inference (BF16) and 4-bit quantization flows (ORCA, vLLM) for further resource reduction.

### NorthPole Accelerator (vertical integration):
- Deployed across 288 NorthPole PCIe cards in 18 standard 2U servers within a 42U rack, delivering 115 peta-ops/s (4-bit) and 3.7 PB/s on-chip bandwidth at ≈30kW (air cooled, 0.67 m², 730 kg).
- All weights and KV caches fit on-chip, eliminating DRAM bottlenecks; inference runs in A8-C8-W4 (activations and caches: 8-bit; weights: 4-bit).
- SiLQ quantization-aware fine-tuning recovers full bfloat16 accuracy (≤0.4 loss over 19 code/LANG benchmarks).
- 3 simultaneous 8B model instances per rack at 2K context, each serving 28 users with inter-token latency 2.8 ms/token and time-to-first-token 64.8 ms.
- Scales linearly for other model sizes: up to 18× 3B models/rack, 1× 70B, or 2 racks for 120B [2511.15950].

This architecture is expressly designed for agentic enterprise AI and fine-grained, multi-user inference in constrained data center envelopes.

## 6. Licensing, Release, and Best Practices

Granite-3.3-8b-Instruct and its code/speech variants, as well as all long-context and speech-aware derivatives, are distributed under Apache 2.0—enabling research and commercial use with a permissive patent grant and open access to weights, code, and data pointers. Training data is curated for licensing compliance, PII/HAP redaction, and model trustworthiness per IBM AI ethics policies [2405.04324, 2407.13739, 2505.08699].

Prompting best practices recommend using explicit <|system|>, <|user|>, and <|assistant|> delimiters (or commit-boundary tokens), providing few-shot exemplars, and including type signatures when eliciting code behavior, following established instruction-tuning methodologies.

A plausible implication is that this model can serve as a drop-in replacement for both generalist and specialist 8B code assistants in research and enterprise environments requiring high code coverage, open licensing, and scalable deployment with minimal adaptation effort.

## 7. Multi-modality and Extensions

Granite-3.3-8b-Instruct is the backbone of the IBM Granite-speech-3.3-8B, a speech-aware LLM integrating a Conformer encoder, Q-former adapter, and LoRA injection. It operates natively in text or speech modes, achieving advanced ASR and AST without sacrificing the code LLM’s instruction-following or functional safety in text-only operations. All subcomponents are modular and open source, with model checkpoints available on HuggingFace for immediate use in academic and commercial projects [2505.08699].

This suggests the Granite-3.3-8b-Instruct model family covers a broad spectrum of tasks from software engineering assistance to cross-modal intelligence, supporting both standard and exotic deployment scenarios while maintaining open licensing and best-in-class inference efficiency.

Source: https://www.emergentmind.com/topics/ibm-granite-3-3-8b-instruct-model