---
title: OpenAI GPT-OSS-120B Overview
url: https://www.emergentmind.com/topics/openai-gpt-oss-120b
type: topic
---

# OpenAI GPT-OSS-120B Overview

OpenAI gpt-oss-120b is a large open-weight Mixture-of-Experts (MoE) language model released by OpenAI under the Apache 2.0 license in 2025 as part of the GPT-OSS family. It is designed for advanced chain-of-thought reasoning, complex tool-use, and agentic behaviors in competitive research, coding, and general AI benchmarks. As an open-weight model, its parameters, inference tools, and tokenizer are broadly accessible, facilitating both commercial and academic adoption.

## 1. Model Architecture and Fundamental Design

GPT-OSS-120B is an autoregressive Transformer with a sparsely-gated Mixture-of-Experts backbone. The architecture comprises 36 Pre-LayerNorm transformer layers with hidden dimension $d_{\mathrm{model}}=2880$, RMS normalization, and residual connections encompassing both attention and MoE submodules. Each block is constructed as follows:

- MoE design: 128 expert MLPs per layer, with top-4 routing per token; for token input $x \in \mathbb{R}^{d_{\mathrm{model}}}$, router logits $r = W_r x + b_r$, select top-4 indices $S = \mathrm{TopK}(r, 4)$, compute mixture weights $\alpha_i = \exp(r_i)/\sum_{j\in S}\exp(r_j)$, and output $\mathrm{MoE}(x) = \sum_{i\in S} \alpha_i f_i(x)$ where $f_i$ is a SwiGLU MLP [2508.10925].
- Attention: Alternating banded window (bandwidth 128 tokens) and global attention; grouped-query attention (64 query heads, 8 key/value heads per layer); rotary or relative positional encodings with YaRN context extension up to 131,072 tokens [2508.10925, 2510.14232].
- Parameterization: 116.8–120 billion total parameters, but ~5.1 billion active parameters per token via sparse expert selection [2508.12461].
- Vocabulary: 201,088 tokens using the o200k_harmony BPE.
- Quantization: Efficient MXFP4 (4.25 bits/weight) quantization reduces checkpoint to 60.8 GiB for single 80 GB GPU deployment [2508.10925].

An alternative architectural report claims a 96-layer, 12,288 width, 128 head, relative rotary-embedding variant with up to 4,096 context for the base dense model [2508.03153]. These discrepancies reflect different variants or pre-release architectures.

## 2. Training Methodology and Reasoning Protocol

The pretraining utilizes trillions of text and code tokens—CommonCrawl, OpenWebText, Books3, Wikipedia, ArXiv, code repositories (including Stack, GitHub, CodeParrot)—with AdamW ($\beta_1=0.9, \beta_2=0.95$, weight decay 0.1, peak LR $2\times10^{-4}$, cosine decay, 300B steps) [2508.03153, 2508.10925].

Post-training incorporates explicit chain-of-thought reasoned supervision and reinforcement learning:

- Supervised CoT: Model learns to emit high-quality, explicit reasoning traces in plain text, including math and code [2508.10925, 2511.19333].
- RLHF/PPO: Reinforcement with rewards for correct chain-of-thoughts and grounded answers. PPO loss: 
  $$
  L_{\mathrm{PPO}}(\theta) =
    \mathbb{E}_t \left[ \min\left(r_t(\theta)A_t, \mathrm{clip}(r_t(\theta), 1-\epsilon, 1+\epsilon)A_t\right) \right]
    - c_v L_{\text{value}} + c_s \mathcal{H}[\pi_\theta]
  $$
  [2508.10925]. Surrogate rewards are derived from a composite of harmonic system, developer, and user roles using the "harmony" chat format.
- Reasoning modes: Variable effort (“low,” “medium,” “high”) controlled via system prompt, modulating chain-of-thought length and latency.
- Safety: Pretraining excludes CBRN content via GPT-4o filters [2508.10925].

No instruction-tuning or human feedback is performed in certain risk assessment configurations; post-training RL is the main policy optimizer [2508.03153].

## 3. Empirical Performance in Reasoning and Coding

GPT-OSS-120B exhibits mid-to-strong results across widely adopted reasoning and code benchmarks.

| Benchmark            | GPT-OSS-120B (%) | Leading Open-Weight (%) | GPT-5 o4-mini (%) |
|----------------------|------------------|------------------------|-------------------|
| MMLU                 | 66–90            | 85.2 (Qwen 3 235B)     | ≈92               |
| GSM8K                | 75               | 97.2 (DeepSeek-R1)     | –                 |
| HumanEval            | 71               | 83 (Llama 3.3 83B)     | –                 |
| Codeforces Elo       | 2463–2622        | –                      | ≈2500             |
| AIME (w/tools)       | 96.6–97.9        | –                      | 97                |
| LiveOIBench          | percentile 59.9  | 81.8 (GPT-5)           | >90 (human)       |

- On LiveOIBench: 59.9th human percentile, 47.8% pass@8, 49.2% relative score, code Elo 2,032; trailing GPT-5 by ≈22 percentile points and ≈15 pp pass rate [2510.09595].
- Gold-ranking at IOI (2025) was achieved using GenCluster: with $K=5,000$ solutions per task, gpt-oss-120b scored 446.75/600 (gold medal threshold ≈366) [2510.14232].
- High CoT efficiency: Typical reasoning traces for math and coding are 4× shorter (≈3,500 tokens) than DeepSeek-R1, with matched downstream accuracy when used as training data for smaller models [2511.19333].
- On standard reasoning/coding (AIME, GPQA, SWE-Bench), results are competitive with closed-weight o3-mini/o4-mini but trail in high-difficulty creative and multilingual tasks [2508.10925].

Inverse scaling is empirically observed: the 20B MoE variant often statistically beats the 120B on HumanEval, MMLU, and similar, likely due to expert imbalance, insufficient steps, or gating suboptimality [2508.12461].

## 4. Agentic Capabilities and Tool Integration

GPT-OSS-120B is designed natively for agentic use, with explicit features:

- Deep Research Browsing: System-level tool for dynamic knowledge retrieval; harmony-format interfaces filter blocklists and enable safe browsing [2508.10925].
- Python Tool Use: Stateful Jupyter-style REPL, allowing code-generation, execution, and reasoning over intermediate results [2508.10925].
- Developer Functions: Developer-exposed function calls in harmony chat, supporting mixed CoT, tool, and user dialogue within a transparent message hierarchy [2508.10925].
- Harmony Chat Format: Multi-role messaging (System > Developer > User > Assistant > Tool), enforced and modeled during training for robust instruction following.

Reference agentic toolchains and full harnesses are open-sourced [2508.10925].

## 5. Open-Weight Risk Profile and Safety

Worst-case risk was evaluated through malicious fine-tuning (MFT):

- Biorisk (biological threat) and CTF (cyber-offense) fine-tuning via PPO in interactive RL environments [2508.03153].
- Peak scores: Biorisk score $B \approx 0.10$ vs. closed-weight o3’s $B \approx 0.35$; CTF composite $C \approx 0.195$ vs. o3’s $C \approx 0.68$.
- Release rationale: Even adversarially optimized, gpt-oss-120b remained below the “Preparedness High” threshold and below closed weights; only +5–6% improvement over its untuned baseline for biorisk and CTF, not advancing the open frontier [2508.03153, 2508.10925].
- Hallucination in CoT remains an unsolved limitation; downstream filtering is advised. Instruction hierarchy is inferior to closed weights; prompt injection and alignment weaknesses persist [2508.10925].

## 6. Efficiency, Deployment Considerations, and Licensing

- Quantized: 4.25 bit checkpoint fits in 60.8 GiB; single 80 GB H100 GPU sufficient for inference [2508.10925].
- Peak VRAM: 80 GB for H100 (120B model, MXFP4 4.25 bits/weight) [2508.12461].
- Throughput: 128 tokens/s at batch size relevant to reasoning tasks, latency 0.8–2.9 s/turn [2508.12461].
- Resource profile: 3.2× memory, 2.6× energy consumption of 20B variant; MoE gating/dispatch adds ≈10% latency overhead compared to a dense FFN [2508.12461].
- Licensing: Apache 2.0 and a permissive usage policy; weights, code, and tools available via GitHub and HuggingFace [2508.10925].

Empirical deployment guidance (for models of this scale): use a VRAM estimate of $
\mathrm{VRAM}_{\mathrm{GB}} \approx \frac{N_{\text{params}}\times b_{\text{per\_param}}}{2^{30}} + \text{overhead}
$, with 10–20% overhead for activations and CUDA context, CPU:RAM at least 16–32 cores : 64 GB [2508.00185].

## 7. Comparative Strengths, Limitations, and Research Implications

GPT-OSS-120B excels in structured, multi-solution reasoning where explicit transparency and agentic tool-use are critical. Selective advantages include:

- Concise, effective reasoning traces for downstream model teaching—achieving near-parity accuracy with much lower token and inference cost [2511.19333].
- Open, reproducible gold-medal performance on competitive programming via advanced sampling and clustering strategies (GenCluster) [2510.14232].
- Flexible agentic integration (Python tool, browsing, developer APIs) suitable for complex research pipelines [2508.10925].

Limitations are documented:

- Underperforms similarly sized dense or hybrid-dense+MoE state-of-the-art models in knowledge, code, and multilingual tasks; 20B variant can surpass 120B in routine QA and code due to sparse scaling inefficiencies [2508.12461].
- Subject to expert load imbalance, gating noise, and possible undertraining at frontier scale; requires further regularization and curriculum refinement [2508.12461].
- Hallucination, prompt-injection, and alignment remain notable safety issues, particularly in open release [2508.10925].

In summary, gpt-oss-120b sets a new open baseline for transparent, scalable, agentic LLM research and demonstration, particularly in tasks where accessible reasoning chains and reproducible tool use are valued, while highlighting critical research directions in sparse scaling optimization, advanced RL fine-tuning, and safety-aligned deployment [2508.10925, 2508.12461, 2508.03153, 2511.19333, 2510.09595, 2510.14232].

Source: https://www.emergentmind.com/topics/openai-gpt-oss-120b