---
title: 'SAGE-32B: Agentic 32B Language Model'
url: https://www.emergentmind.com/topics/sage-32b
type: topic
---

# SAGE-32B: Agentic 32B Language Model

SAGE-32B is a 32 billion parameter language model specifically optimized for agentic reasoning, long-range planning, and robust tool-use scenarios. Unlike general conversational LLMs, SAGE-32B is architected for operation within an agentic loop, emphasizing explicit task decomposition, tool invocation, and iterative error recovery. Developed by extending the Qwen2.5-32B decoder-only transformer, SAGE-32B incorporates novel modules, a multi-stage iterative distillation process, and introduces an auxiliary meta-cognition head for inverse reasoning and online failure forecasting. Publicly released under the SAGE-AI initiative, it is empirically evaluated on major agentic reasoning benchmarks, demonstrating superior performance in multi-tool use and error recovery compared to structurally similar and larger models [2601.04237].

## 1. Architecture and Initialization

SAGE-32B is constructed by augmenting the Qwen2.5-32B backbone with specialized modules. The total parameter footprint approximates 32 × 10⁹, with the core transformer layers accounting for ≈31 billion, split-embeddings and gating heads for ≈300 million, and the Meta-Cognition Head (MCH) for ≈200 million parameters.

Weight initialization is performed via normal sampling:
\[
W \sim \mathcal{N}\left(0, \sigma^2 \right),\quad \sigma = \frac{0.02}{\sqrt{2L}},\;L=64
\]
Layer normalization employs RMSNorm with $\epsilon=10^{-6}$:
\[
\mathrm{RMSNorm}(x)=\frac{x}{\sqrt{\frac{1}{d}\sum_i x_i^2+\epsilon}} \odot \gamma
\]
Feed-forward networks use SwiGLU gating:
\[
\mathrm{FFN}(x) = \mathrm{Swish}_\beta(xW_G) \odot (xW_1) W_2, \quad \mathrm{Swish}_\beta(z) = z\,\sigma(\beta z)
\]
A split-embedding strategy provides context-dependent embeddings:
\[
E_{input}^{(t)} = \alpha_t E_{NL}(x_t) + (1-\alpha_t)E_{Code}(x_t),\quad \alpha_t \in [0,1]
\]
where $\alpha_t$ is selected by a learned classifier. For long-context processing, a landmark attention mechanism mixes dense local attention over the last 4,096 tokens and global "landmark" tokens (stride $k=64$), enabling efficient operation up to 128k context length.

The MCH is an auxiliary attention layer grafted onto the terminal transformer block. It accepts $h_{last}$ and outputs a confidence vector:
\[
c_t = \sigma(W_{MCH} h_{last} + b)
\]
providing probabilities for stepwise plan failures.

## 2. Iterative Distillation and Training Paradigm

SAGE-32B adopts a multi-stage Iterative Distillation & Amplification (IDA) regime, integrating both teacher-student distillation and offline self-correction:

### 2.1 Distillation & Amplification (IDA)
Initial fine-tuning is teacher-driven: a hybrid GPT-4o/DeepSeek model generates 5 million synthetic agentic rollouts in environments such as OSWorld and WebArena. Negative-constraint sampling provides three types of hard negatives (type error, hallucinated key, logic error) for every correct action. The primary loss is cross-entropy on correct/negative discrimination ($L_{KD}$).

### 2.2 Reflective Distillation
Building on Reflexion and Self-Refine, but applied fully offline, the student collects rollouts; failures are critiqued and revised by the teacher. Corrections and critiques are added to the training buffer. The reflective loss is
\[
\mathcal{L}_{RD} = -\sum \log P_\theta\left(y_{corrected} \mid x, y_{failed}, critique \right)
\]
with learning implemented by DPO gradient steps:
```python
Input: Student π_θ, Teacher π_ref, Environment E
Buffer D ← ∅
for epoch in 1…E:
    for rollout τ under π_θ:
        if Success(τ):
            D ← D ∪ {(τ, 1)}
        else:
            critique ← π_ref("Critique this failure:", τ)
            τ_corr ← π_ref("Fix per critique:", τ, critique)
            D ← D ∪ {(τ_corr, 1), (τ⊕critique, 0)}
    θ ← θ − α∇_θ L_{DPO}(π_θ, D)
```

### 2.3 DPO Preference Learning for Safety
A DPO-based loss enforces the refusal of unsafe tool calls (e.g., `delete_database()`), with penalty coefficient $\beta=0.1$.

### 2.4 Reinforcement Learning (CodePPO)
Function-calling is cast as program synthesis. Rewards are provided for successful code execution and semantic intent matches; penalties are applied for hallucinated arguments and syntax errors.

### 2.5 Composite Objective
Across phases, the optimization objective is blended:
\[
L = \alpha L_{KD} + (1-\alpha)L_{task}
\]
where $L_{task}$ is standard next-token cross-entropy.

## 3. Meta-Cognition and Inverse Reasoning

The meta-cognition head (MCH) implements inverse reasoning and online failure detection.

### 3.1 Architecture
The MCH leverages the final transformer hidden state $h_{last}$, with a distinct projection $W_{MCH}$, to produce per-step confidences.

### 3.2 Failure Forecasting
For each candidate agentic step $s_t$, the MCH computes
\[
P_{fail}(s_t) = c_t = \sigma(W_{MCH} h_{last} + b)
\]
If $P_{fail}$ exceeds a threshold $\tau_{uncertainty}$, SAGE-32B enters a "Reasoning Mode"—invoking further look-ahead and candidate evaluation.

### 3.3 Inverse Consistency Score (ICS)
ICS quantifies how reasoning traces ($z$) can reconstruct the original context ($x$):
\[
\mathrm{ICS}(z) = \mathbb{E}_{q_\phi(x|z,y)}[\log P_{data}(x)]
\]
A dual-head architecture shares backbone features to estimate this reconstruction likelihood, typically by KL divergence or log-likelihood metrics.

### 3.4 Hybrid Energy Re-Ranking
Given $K$ candidate continuations, SAGE-32B re-ranks by
\[
E(z) = -\log P_\theta(z|x) + \lambda\,\mathrm{ICS}(z)
\]

## 4. Agentic Loop: Task Decomposition and Error Recovery

Agentic operation in SAGE-32B proceeds as follows:

1. Decomposition of complex requests into a DAG of atomic, dependency-annotated subtasks.
2. Iterative agentic reasoning loop:
    a) Generate candidate step $s_i$.
    b) Calculate $P_{fail}(s_i)$ via MCH.
    c) If $P_{fail} > \tau_{uncertainty}$, perform Look-Ahead Simulation (LAS), sampling $K$ continuations and ranking by $E(z)$.
    d) Execute tool calls and observe results.
3. On tool errors (e.g., syntax), invoke the reflective policy for in-loop critique and correction ("critique loop").

Reflective Distillation instills robust offline failure recognition and recovery while the MCH/LAS act as online verifiers during inference, minimizing cumulative error propagation.

## 5. Benchmarking and Empirical Performance

SAGE-32B achieves significant empirical gains over its backbone and several industry baselines, especially in multi-tool and long-horizon agentic tasks.

### 5.1 Reasoning and Tool Use Benchmarks

| Benchmark   | Qwen2.5-32B | SAGE-32B (Std) | SAGE-32B (Think, k=32) | Llama-3-70B | GPT-4-Turbo |
|-------------|-------------|---------------|------------------------|-------------|-------------|
| MMLU-Pro    | 71.5        | 75.6          | 79.3                   | 68.9        | 63.7        |
| MATH-500    | 78.9        | 78.9          | 91.8*                  | 68.0        | 72.6        |
| AgentBench  | 58.4        | 58.4          | 73.1                   | 62.1        | 85.0        |
| GPQA        | 50.5        | 48.0          | 48.0                   | 51.0        | 53.6        |
| IFEval      | 81.2        | 84.5          | 84.5                   | 78.5        | 86.0        |

*Note: “Think” mode utilizes Majority-Vote@32 with ICS filtering, yielding MATH-500 improvements (±0.4% stdev).

### 5.2 Error Recovery and Efficiency

- Internal Recovery Rate (IRR): SAGE-32B reaches 76%, doubling the Qwen2.5-32B base (35%).
- AgentBench agentic modes:
    - Fast: 58.4% @1.2s (1.0× cost)
    - Slow: 73.1% @4.5s (3.8×)
    - Hybrid: 71.8% @1.8s (1.4×)

### 5.3 Tool-Calling (Enterprise-500) Suite

| Model             | Success Rate | Unforced Errors | Cost/1k eps |
|-------------------|-------------|----------------|-------------|
| GPT-4-Turbo       | 94.2%       | 1.5%           | $32.00      |
| Claude 3.5        | 92.8%       | 2.1%           | $15.00      |
| SAGE-32B Hybrid   | 91.5%       | 2.4%           | $4.50       |
| Llama-3-70B       | 85.0%       | 8.2%           | $6.00       |
| Qwen2.5-32B       | 76.4%       |14.5%           | $2.80       |

Hallucination rate ablation: SAGE-32B reduces hallucinations from 14.5% (base) to 5.2% after reflective distillation, and down to 2.4% post-RL.

## 6. Implementation and Availability

SAGE-32B is released under a research preview license, available at https://huggingface.co/sagea-ai/sage-reasoning-32b. Key implementation hyperparameters include: 64 layers, maximum context of 128k tokens, local attention window of 4,096, landmark stride $k=64$, and distillation batch size ≈256. Training utilizes 5 million synthetic multi-step trajectories with negative sampling, covering both synthetic and real-world (OSWorld/WebArena) environments. The CodePPO RL reward structure empirically balances task success with penalties for argument hallucination and syntax violations.

For use with the HuggingFace Transformers library:
```python
from transformers import AutoModelForCausalLM, AutoTokenizer
model = AutoModelForCausalLM.from_pretrained("sagea-ai/sage-reasoning-32b")
tokenizer = AutoTokenizer.from_pretrained("sagea-ai/sage-reasoning-32b")
```
A plausible implication is that SAGE-32B's design—combining inverse reasoning, hybrid training, and explicit meta-cognition—marks an advancement for agentic large language models operating under long-horizon, multi-step planning, and tool-use settings, while maintaining competitive cost efficiency [2601.04237].

Source: https://www.emergentmind.com/topics/sage-32b