GLM-5V-Turbo: Multimodal Agent Framework

• GLM-5V-Turbo is a multimodal foundation model that natively fuses language, vision, and action for diverse applications.
• It employs a decoder-only transformer with interleaved cross-attention and CogViT for effective text-image integration.
• Hierarchical training and reinforcement learning enable reliable tool use, GUI navigation, and end-to-end task verification.

GLM-5V-Turbo is a foundation model designed to serve as a native, multimodal agent core, integrating language, vision, and action-centric reasoning across heterogeneous contexts including images, GUIs, documents, and code. Distinct from prior approaches that treat multimodal capacity as an auxiliary module, GLM-5V-Turbo implements multimodal perception as an intrinsic and foundational element throughout reasoning, planning, tool use, and action execution. Its development encompasses advances in model architecture, multimodal training paradigms, reinforcement learning with hierarchical organization, agentic toolchains, and rigorous end-to-end task verification, delivering high performance in multimodal reasoning and agent frameworks while preserving state-of-the-art capabilities for competitive text-only coding tasks (Team et al., 29 Apr 2026).

1. Model Architecture and Multimodal Fusion

GLM-5V-Turbo adopts a decoder-only Transformer backbone derived from GLM-5-Turbo, stacking $L$ identical blocks that contain (masked) self-attention, cross-attention for vision, and feed-forward sublayers. The model encodes a fusion of textual tokens and visual patch embeddings, represented at each layer as $H_\ell \in \mathbb{R}^{(T+V)\times d}$, where $T$ and $V$ are the lengths of the text and vision streams, respectively.

Vision features are extracted using the CogViT encoder, a ViT-style architecture trained through two key stages: masked image modeling (MIM) distillation and contrastive alignment. CogViT applies Query-Key normalization (QK-Norm), ensuring $\|Q\|_2 = \|K\|_2 = 1$ for stabilized attention softmax computations. Multimodal fusion is realized by interleaved cross-attention layers: textual queries ($Q_\text{txt}$) attend over visual keys and values ($K_\text{vis}$, $V_\text{vis}$), yielding $$A_{\text{txt} \leftarrow \text{vis}} = \mathrm{softmax}\left(\frac{Q_\text{txt} K_\text{vis}^\top}{\sqrt{d}}\right) V_\text{vis}$$, which is projected and reintroduced into the textual pathway. Optional symmetric cross-attention enables vision queries over text.

Hierarchical feature fusion is implemented by passing the cross-fused hidden states through an FFN, then re-separating text and vision across layers. Multimodal Multi-Token Prediction (MMTP) enables efficient multimodal autoregression: all visual patch tokens are abstracted into a single learned $<|image|>$ token, with the standard multi-token prediction loss ($H_\ell \in \mathbb{R}^{(T+V)\times d}$0) applied to downstream language tokens conditioned on this fused prefix.

2. Training Paradigms and Optimization

Pretraining uses a combination of three losses:

• Text-only next-token prediction: $H_\ell \in \mathbb{R}^{(T+V)\times d}$1,
• Vision distillation (MIM): $H_\ell \in \mathbb{R}^{(T+V)\times d}$2, where $H_\ell \in \mathbb{R}^{(T+V)\times d}$3 and $H_\ell \in \mathbb{R}^{(T+V)\times d}$4 denote masked student/teacher features,
• Contrastive image-text alignment: $H_\ell \in \mathbb{R}^{(T+V)\times d}$5.

The combined pretrain objective is $H_\ell \in \mathbb{R}^{(T+V)\times d}$6.

Supervised fine-tuning (SFT) employs a mixture of text, image–text pairs, OCR, GUI states, code, spatial reasoning, and tool-use data, applying standard cross-entropy objectives per token or action.

Hierarchical optimization is central: low-level tasks such as object grounding and single-step GUI actions are fine-tuned before high-level, multi-step planning tasks. Reinforcement learning is executed over 30+ tasks using a PPO variant with a centralized reward $H_\ell \in \mathbb{R}^{(T+V)\times d}$7, where $H_\ell \in \mathbb{R}^{(T+V)\times d}$8 is computed by rule-based checks and $H_\ell \in \mathbb{R}^{(T+V)\times d}$9 by asynchronous judge model calls. The PPO-style RL loss is $T$0.

3. Agentic Toolchain and Interactive Capabilities

GLM-5V-Turbo features a unified VLM-RL Gym API enabling both single-step (e.g., VQA) and multi-step (e.g., GUI navigation, tool use) tasks. Agents submit observations $T$1 and generate actions $T$2 as text outputs or token sequences.

Tool invocation adopts an explicit tokenization protocol: the model emits tokens such as $T$3tool_call name="zai_recognize_person" args="{…}" /$T$4 that are harnessed into Python function calls. Resulting tool outputs, whether JSON, image, or text, are appended into subsequent agent contexts.

Planning and sequencing are controlled by planner prompt templates, guiding the agent through: (1) observation, (2) tool selection, (3) invocation, (4) reading results, and (5) deciding repetition or completion. Multimodal perception from CogViT (e.g., bounding boxes) enhances tool selection (crop, search, click).

Integration with external agent frameworks is enabled: in Claude Code, GLM-5V-Turbo serves as a backend provider for multimodal perception and code generation. In AutoClaw, GLM-5V-Turbo directly issues browser GUI commands in response to visual screenshots.

4. Evaluation and Comparative Benchmarking

Extensive evaluation highlights GLM-5V-Turbo’s state-of-the-art performance on multimodal and agentic benchmarks. Results (percentages unless noted):

Task Category	Task	Score(s)	Notable Comparisons
Multimodal Coding	Design2Code	94.8	~$T$5 vs. Claude Opus 4.6
Multimodal Tool-Use	ImageMining	30.7	Comparable/Superior
	BrowseComp-VL	51.9	Comparable/Superior
	MMSearch	72.9
	MMSearch-Plus	30.0	x8 vs. GLM-4.6V
	SimpleVQA	78.2
GUI Agent	AndroidWorld	75.7
	OSWorld	62.3
	WebVoyager	≥ 70
Claw Agent Benchmarks	PinchBench	87.0/80.7	(pass@1 / pass@k)
	ClawEval	57.7/75.0
	ZClawBench	57.6
Text-Only Coding	CC-Backend	22.8	$T$622.5 for 5-Turbo
	CC-Frontend	68.4
	CC-RepoExploration	72.2

Performance matches or slightly surpasses text-only coding baselines (GLM-5-Turbo), indicating no tradeoff in text-only capacity. Dramatic gains over GLM-4.6V are observed for deep search and tool-use benchmarks. Comparisons to industrial agents (Claude Opus 4.6, Kimi K-2.5) show competitive or superior results on BrowseComp-VL and ImageMining (with $T$7 from paired bootstrap tests).

5. Practical Insights and End-to-End Verification

Three principal insights inform the model’s development:

1. Perception as Foundation: Fine-grained perceptual errors, such as GUI element mis-localization, can cascade into downstream reasoning failures. Proxy tasks (SVG-to-code pretraining, grounding critique) are leveraged to enhance visual acuity.
2. Hierarchical Optimization: Training is decomposed into levels—element perception (bounding boxes, OCR), single-step action selection, and trajectory-level planning. Each layer receives tailored SFT/RL signals. Stacking these layers reduces variance relative to undifferentiated end-to-end RL.
3. Clear Specification & Reliable Verification: Success in end-to-end tasks (e.g., visual website building) depends on: detailed PRD/mockup specifications, workflow-based verifiers, and multi-stage (unit, integration, visual diff) checks. Benchmarks like Vision2Web are structured for explicit, stepwise verification yielding reproducible pass/fail metrics.

A high-level verification loop operates as follows:

1. $T$8
2. For each workflow step: a. $T$9 b. If $V$0 is False: $V$1; break
3. If all pass: $V$2

This separation of task decomposition, verification design, and feedback ensures that long-horizon, multimodal behaviors are trainable and measurable with reliability.

6. Guiding Lenses for Multimodal Agent Development

The model’s development surfaces guiding “lenses” for future work:

• Centralize fine-grained perception in the model’s learning objectives.
• Build agentic capability layer by layer, exploiting hierarchical optimization to reduce instability.
• Invest in clear, verifier-driven benchmarks and workflow-based end-to-end validation frameworks.

These guiding principles offer a template for building robust multimodal agents capable of orchestrated reasoning and action across language and vision, as demonstrated by GLM-5V-Turbo’s empirical performance and architectural design (Team et al., 29 Apr 2026).