---
title: 'GLM-4.5V: Advanced Multimodal Intelligence'
url: https://www.emergentmind.com/topics/glm-4-5v
type: topic
---

# GLM-4.5V: Advanced Multimodal Intelligence

GLM-4.5V refers to two distinct, state-of-the-art open-source architectures developed in parallel by the THUDM and Zhipu AI research groups, targeting spoken-language intelligence [2412.02612] and advanced multimodal vision–language reasoning [2507.01006], respectively. Both models are positioned at the forefront of large-scale, instruction-tuned multimodal systems and exemplify trends in scalable model design, unified token representations, and reinforcement learning-driven optimization for diverse downstream tasks.

## 1. Model Architectures

### Speech-Language GLM-4.5V
The THUDM GLM-4.5V is built on the GLM-4-9B-Base 9B-parameter text Transformer, augmented with an ultra-low bitrate (175 bps), single-codebook speech token vocabulary and ancillary modules for end-to-end, real-time spoken dialogue [2412.02612]. The architecture introduces a VQ-VAE bottleneck at the mid-layer of a Whisper-Large-V3 ASR encoder, reducing frame rate from 50 Hz to 12.5 Hz and mapping each audio segment (80 ms at 16 kHz) to a discrete token ID. The backbone’s input embedding layer is extended for speech-token IDs; the main Transformer (attention, FFNs) is unmodified, yielding bidirectional capacity for cross-modal token streams.

Generation alternates between autoregressive text and speech token predictions within the unified Transformer backbone, enabling joint modeling of $\{w_t\}$ (text) and $\{s_t\}$ (speech):

$$
L_{\text{LM}}
= -\sum_{t=1}^T \log P(x_t | x_{<t}),
$$

where $x_t$ ranges over the union of text and speech tokens.

### Vision-Language GLM-4.5V
The Zhipu AI GLM-4.5V (sometimes styled GLM-4.5V-AIR) incorporates an AIMv2-Huge ViT vision encoder (supplemented with 3D convolutions for temporally-aware video processing, and RoPE for positional encoding) connected to a Mixture-of-Experts (MoE) language head with ~106B total, ~12B activated parameters (12B-A MoE). Visual features are projected via an MLP adapter into the LLM’s token space [2507.01006].

Key architectural features:

- Flexible support for both chain-of-thought ("thinking") and flat response generation via `/nothink` mode tokens.
- Temporal grounding in video streams using time-index tokens post-frame.
- Extended 3D-RoPE in the LLM head for improved spatial reasoning.
- Absolute positional embeddings can be interpolated for variable-resolution/image aspect ratios.
- Full-context modeling up to 8,192 tokens (32K–128K for GLM-4.6V).

## 2. Pre-training Strategies

### Speech-Language (THUDM)
Data sourcing and alignment use a combination of:
- Interleaved synthetic speech–text data via a text-to-token TTS model (∼70%)
- Unsupervised real speech (∼10%)
- Paired ASR/TTS supervised data (∼5%)
- Text-only corpora (∼15%)
- Yielding a total pre-training volume of ~$10^{12}$ tokens.

The training objective leverages joint sequences of the form:

$$
[\dots, \texttt{<speech>}, s_1, \dots, s_m, \texttt{</speech>}, w_{i+1}, \dots]
$$

Optimization employs AdamW, warmup followed by linear decay from $6 \times 10^{-5}$ to $6 \times 10^{-6}$, 8,192 token context, and large aggregate batches.

### Vision-Language (Zhipu AI)
Pre-training draws on billions of multimodal samples:
- Image-caption pairs (filtered, concept-balanced)
- Interleaved web/academic multimodal text (MINT, MMC4, OmniCorpus)
- OCR datasets (synthetic + real), grounding annotations, GUI screenshots
- Video-caption corpora, processed with human metadata verification
- 50M mixed-format instruction-tuning samples

Autoregressive next-token modeling is applied over interleaved visual and text tokens:

$$
L_{\text{LM}} = -\sum_{t=1}^T \log p_\theta(x_t | x_{<t}, V)
$$

with an auxiliary MoE load-balancing loss:

$$
L_{\text{balance}}
= \lambda \sum_{e=1}^E \Bigl( \tfrac{u_e}{N} - \tfrac{1}{E} \Bigr)^2, \quad \lambda=10^{-4}
$$

## 3. Reinforcement Learning with Curriculum Sampling (RLCS)

GLM-4.5V employs RLCS to optimize multimodal reasoning. Difficulty tiers—stratified based on pass@k scores—drive sample selection via a smooth exponential-weighted policy targeting items within the model’s learning frontier:

$$
p_t(k)\;\propto\;\exp\!\Bigl(-\tfrac{|d_{\mathrm{model}}(k)-d_{\mathrm{off}}(k)|}{\tau}\Bigr)
$$

where $d_{\mathrm{off}}$ is the offline difficulty and $d_{\mathrm{model}}$ the current competence. The RLCS loop evaluates, rewards, and re-weights rollout samples based on:
- STEM QA: $r=\mathbb{I}[\text{boxed\_answer}=\text{gt}]$
- OCR: $r=1-\tfrac{d_{\text{edit}}(\text{ans},\text{gt})}{\max(|\text{ans}|,|\text{gt}|)}$
- Grounding: $r=\tfrac{\#\{\mathrm{IoU}>\tau\}}{\#\mathrm{boxes}}$

This approach yields a +3–5% performance uplift versus random sampling and mitigates RL instability by dynamically matching curriculum to model competence [2507.01006].

## 4. Evaluation and Capabilities

### Speech-Language Tasks

GLM-4.5V attains state-of-the-art or highly competitive results in end-to-end ASR/TTS, spoken QA, and conversational assessment:

| Model          | LibriSpeech WER (%) | AISHELL-1 CER (%) | Topic-StoryCloze (S→T, %) | GenQA / UTMOS |
|:--------------:|:------------------:|:-----------------:|:-------------------------:|:-------------:|
| Whisper-Large-V3 | 2.50 / 4.53         | 9.31              | —                         | —             |
| CosyVoice        | — / —               | —                 | —                         | —             |
| **GLM-4.5V**     | 2.82 / 7.66         | 2.46              | 93.6                      | 5.40 / 4.45   |

On benchmarks such as StoryCloze (76.3%) and TriviaQA (39.1%), performance exceeds comparably sized models. Human-rated General/Knowledge QA, and UTMOS (4.45), confirm competitive conversational naturalness [2412.02612].

### Vision-Language and Multimodal Reasoning

GLM-4.5V demonstrates advanced competence across 42 vision–language benchmarks, including STEM (65.2% on MMMU Pro), GUI agents (35.8% OSWorld), coding (82.2% Design2Code), video reasoning (80.7% Video MMMU with subtitles), and visual grounding (91.3% RefCOCO-avg):

| Category        | Benchmark     | GLM-4.5V | Qwen2.5-VL-72B | Gemini-2.5-Flash |
|-----------------|--------------|---------:|---------------:|-----------------:|
| STEM            | MMMU Pro     |   65.2%  |        51.1%   |        —         |
| GUI Agents      | OSWorld      |   35.8%  |         8.8%   |        —         |
| Coding          | Design2Code  |   82.2%  |        41.9%   |        —         |
| Video MMMU      | w/subtitles  |   80.7%  |        79.1%   |        —         |
| Visual Grounding| RefCOCO-avg  |   91.3%  |        90.3%   |        —         |

Ablations show +1–2% gains in multimodal RL stability and accuracy by omitting KL loss. Across evaluated tasks, the model matches or outperforms closed-source Gemini-2.5-Flash in 22 out of 42 cases [2507.01006].

## 5. Inference Pipeline and Usage

### Speech-Language
- Real-time streaming: user audio is discretized by the speech tokenizer, processed by the unified Transformer in an alternating text/speech-token regime (typically 13 text : 26 speech tokens).
- Voice control: Special span tokens in the prompt for emotion (`<style=happy>`), speech rate (`<rate=1.2>`), and dialect (`<dialect=Shanghai>`).
- Output: Text may be display-bypassed or passed to a HiFiGAN-based waveform decoder for speech synthesis. Latency is bounded by an initial block size ($b=0.8$ s).
- Open-source code and checkpoints via GitHub and Hugging Face [2412.02612].

### Vision-Language
- Accepts arbitrarily sized images or videos (resolution/aspect-agnostic) via bicubic-interpolated absolute embeddings and time-indexed video tokens.
- Sequence lengths up to 8,192 tokens (extending to 131,072 for GLM-4.6V).
- vLLM and SGLang backends for text/video, with planned tool-invocation structured via XML spans.
- Released resources and documentation maintained at https://github.com/zai-org/GLM-V [2507.01006].

## 6. Limitations and Future Development

Key limitations:
- RL with verifier rewards does not enforce coherence of output reasoning, potentially generating correct answers with erroneous or underspecified intermediate chains.
- RL instability may occur with poor-quality rewards or data; entropy and KL losses require careful tuning to avoid collapse.
- Visual models exhibit error propensity in occluded/cluttered scenes, and tool-use hallucinations may arise.
- Resource concerns in RL: scale, compute, and web-scale data biases pose practical and ethical challenges.

Proposed directions under investigation:
- Reward models that assess intermediate chain-of-thought steps for contradiction and hallucination.
- Adversarial and subset-based validation to detect RL reward gaming.
- Diagnostic multimodal benchmarks for hallucination and multi-step planning [2507.01006].

A plausible implication is that holistic, curriculum-driven RL, unified token spaces, and explicit alignment strategies represent convergent design trends for future large-scale, open-source multimodal models.

## References

- "GLM-4-Voice: Towards Intelligent and Human-Like End-to-End Spoken Chatbot" [2412.02612]
- "GLM-4.5V and GLM-4.1V-Thinking: Towards Versatile Multimodal Reasoning with Scalable Reinforcement Learning" [2507.01006]

Source: https://www.emergentmind.com/topics/glm-4-5v