---
title: 'MiniCPM-o 4.5: Omni-Modal Interaction AI'
url: https://www.emergentmind.com/topics/minicpm-o-4-5
type: topic
---

# MiniCPM-o 4.5: Omni-Modal Interaction AI

MiniCPM-o 4.5 is a 9-billion-parameter omni-modal foundation model designed to enable real-time, full-duplex, human-like multimodal interactions. Distinguished by its simultaneous perception and generation capabilities across vision, audio, text, and speech, MiniCPM-o 4.5 leverages the novel Omni-Flow streaming architecture to achieve state-of-the-art open-source performance at its scale while remaining efficient enough for edge deployment on devices with less than 12 GB RAM. The model approaches the vision-language performance of Gemini 2.5 Flash and surpasses Qwen3-Omni-30B-A3B in omni-modal understanding and speech generation, marking a prominent advancement in practical, human-like AI interaction paradigms [2604.27393].

## 1. Model Architecture and Parameterization

MiniCPM-o 4.5 is an end-to-end model integrating three primary components: a streaming visual encoder based on SigLIP ViT with a resampler, a streaming audio encoder utilizing Whisper Medium and a two-layer MLP projector, and a large language model backbone (Qwen3-8B) interleaved with speech decoders. All encoders and decoders are interconnected via token-level hidden states, ensuring fully differentiable, end-to-end training.

**Componentwise Parameter Breakdown:**

| Component                            | Parameters (Millions) |
|--------------------------------------|:--------------------:|
| Visual encoder (SigLIP ViT)          |      417.8           |
| Visual resampler                     |       88.9           |
| Audio encoder (Whisper Medium)       |      307.2           |
| Audio projector (2-layer MLP)        |       21.0           |
| LLM backbone (Qwen3-8B)              |     8,189.2          |
| Backbone→speech-decoder projector    |       10.5           |
| Speech-token decoder (Transformer)   |      188.8           |
| Text embedding for decoder           |      116.8           |
| **Total**                            |   **~9,341**         |

**Architectural details:**
- **Vision:** SigLIP ViT employs 27 layers with a hidden dimension of 1,152, 16 attention heads, and 14×14 patch sizes. The resampler applies cross-attention to compress 1,024 to 64 tokens per slice (16× compression).
- **Audio:** Whisper Medium uses 24 layers (hidden size 1,024, 16 heads), producing 50 audio tokens/s, further reduced to 10 tokens/s by the MLP projector.
- **LLM Backbone:** Qwen3-8B is configured with 36 layers, a hidden dimension of 4,096, 32 heads, FFN size of 12,288, SiLU activations, and a RoPE positional encoding up to length 40,960.
- **Speech Decoding:** The model features a 20-layer Transformer decoder (hidden 768, 12 heads, 25 speech tokens/s), coupled with a flow-matching waveform decoder for real-time speech synthesis.

## 2. Omni-Flow Full-Duplex Streaming Framework

The central technical advance is the Omni-Flow framework, which reformulates interaction as the simultaneous streaming of three modalities: live image frames \(I(t)\), acoustic input \(A(t)\), and generated text/speech (out-stream). At each time \(t\):

$$
H(t)=\text{Concat}(f_v(I(t)), f_a(A(t)), h_\text{text}(t))
$$

where \(f_v(I(t))\) (visual encoder output), \(f_a(A(t))\) (audio encoder output), and \(h_\text{text}(t)\) (partial text generation state) are concatenated and processed in fixed chunks of duration \(\Delta t\) (e.g., 1 s):

$$
H(t+\Delta t) = H(t) + \Phi(f_v(I(t+\Delta t)), f_a(A(t+\Delta t)))
$$

\(\Phi(\cdot)\) denotes the serializing of new input tokens into context, followed by next-token generation. Each chunk yields groups of tokens—visual (\(v^k\)), audio (\(a^k\)), and output (\(o^k\)). These are combined as \(g_k = [v^k; a^k; o^k]\) and serialized into the stream.

Simultaneous perception and generation is achieved per chunk: the model encodes current inputs, applies a control token predicting [listen] or [speak], autoregressively generates text tokens if in [speak] mode, and interleaves the textual output with speech tokens. The Time-Aligned Interleaving (TAIL) algorithm modulates the number of generated tokens per chunk, ensuring the speech waveform remains aligned with real-world time through a look-ahead window for accurate pronunciation context.

## 3. Proactive Behavior Mechanisms

MiniCPM-o 4.5 models proactive behavior by incorporating explicitly labeled proactive training samples (e.g., time-tagged scene descriptions, reminders, or live comments) within its full-duplex training data. The model’s next-token objective encompasses both reactive and proactive utterances, enabling output of intervening prompts or comments independent of explicit user queries.

During reinforcement learning (RL) fine-tuning, additional rewards structure this initiative:

- \(r_\text{initiative}\): +1 for timely, context-driven prompts (e.g., reminders).
- \(r_\text{len}\): smooth penalty for verbosity.
- \(r_\text{acc}\): standard accuracy reward on downstream tasks.

The RL objective is

$$
\mathcal{L}_\mathrm{RL} = -\mathbb{E}_{\pi_\theta}\left[r_\mathrm{acc} + \alpha\,r_{\mathrm{initiative}} + \beta\,r_{\mathrm{len}}\right]
$$

Initiative emerges directly from joint next-token and reward signals; no separate proactive loss is necessary.

## 4. Inference Optimization and Edge Efficiency

MiniCPM-o 4.5 is engineered for real-time, full-duplex omni-modal interaction at edge-appropriate hardware constraints (<12 GB RAM). This is achieved through two major optimizations: vLLM inference with INT4 quantization and the custom C++ llama.cpp-omni backend.

**Text-only throughput benchmarks (RTX 4090):**

| Model / Dtype           | Throughput (tok/s) | 1st-token Latency (s) | Memory (GB) |
|------------------------ |-------------------|-----------------------|-------------|
| MiniCPM-o 4.5 (BF16)    | 154.3             | 0.59                  | 19          |
| MiniCPM-o 4.5 (INT4)    | 212.3             | 0.58                  | 11          |
| Qwen3-30B-A3B (INT4)    | 147.8             | 0.98                  | 20          |

**Streaming efficiency (llama.cpp-omni INT4):**

| Framework       | RTX 4090 RTF | RTX 4090 Mem (GB) |
|-----------------|--------------|-------------------|
| llama.cpp-omni  | 0.21         | 11                |
| PyTorch (INT4)  | 1.26         | 14                |

These results demonstrate that MiniCPM-o 4.5 sustains full-duplex streaming well below the 12 GB threshold.

## 5. Benchmark Performance and Comparative Evaluation

**Vision-Language (Instruct Mode):**

| Benchmark      | Gemini 2.5 F | Qwen3-Omni 30B | MiniCPM-o 4.5 |
|----------------|--------------|----------------|--------------|
| OpenCompass    | 78.5         | 75.7           | 77.6         |
| MMBench EN     | 86.6         | 84.9           | 87.6         |
| TextVQA        | 74.3         | 84.1           | 83.8         |
| OCRBench (ms)  | 864          | 880            | 876          |
| Mantis-Eval    | 72.8         | 78.3           | 79.7         |

**Omni-Modal (Simplex Mode):**

| Benchmark      | Gemini 2.5 F | Qwen3-Omni 30B | MiniCPM-o 4.5 |
|----------------|--------------|----------------|--------------|
| Daily-Omni     | 79.3         | 70.7           | 80.2         |
| WorldSense     | 52.6         | 54.0           | 55.7         |
| Video-Holmes   | 51.3         | 50.4           | 64.3         |
| AVUT-Human     | 65.4         | 74.2           | 78.6         |

**Full-Duplex Streaming (Vision Only):**

- LiveSports-3K-CC: MiniCPM-o 4.5 achieves 54.4% (LiveCC: 41.5%, StreamingVLM: 45.6%).

**Speech:**

- Automatic Speech Recognition WER: AISHELL-1: 0.9% (Qwen3-Omni: 0.6%), GigaSpeech: 8.5% (Qwen3-Omni: 8.7%).
- Speech Translation BLEU (CoVoST2 en→zh): 49.9 (Qwen3-Omni: 46.6).
- Speech QA (VoiceBench): 4.81 (Qwen3-Omni: 4.74).

**Speech Generation:**

| Model         | SeedTTS-ZH CER | SeedTTS-EN WER | LongTTS EN WER | Expresso | ESD   |
|---------------|----------------|----------------|----------------|---------|-------|
| CosyVoice2    |   1.45         |   2.57         |   14.80        | 17.9    | 53.4  |
| Qwen3-Omni    |   1.41         |   3.39         |   17.33        | N/A     | N/A   |
| MiniCPM-o 4.5 |   0.86         |   2.38         |   3.37         | 29.8    | 82.1  |

**Text-Only Evaluation:**

| Benchmark  | Qwen3-8B | MiniCPM-o 4.5 |
|------------|----------|---------------|
| IFEval-PLS | 83.0     | 84.7          |
| BBH        | 69.4     | 81.1          |
| MMLU       | 81.7     | 77.0*         |
| MBPP       | 75.9     | 76.7          |
| GSM8K      | 93.4     | 94.5          |

*slight dip on factual recall but strong overall performance.

## 6. Applications in Real-Time Omni-Modal Interaction

MiniCPM-o 4.5 enables a spectrum of real-time, full-duplex, omni-modal applications:

- **Live Sports Commentary:** Continuous ingestion of video and audio feeds allows ongoing natural language description of events, with midstream proactive interventions such as weather-related prompts.
- **Multi-Party Conversational Assistance:** In scenarios such as three-way conference calls, the model transcribes, analyzes visual content (e.g., slides), and interacts with multiple parties proactively.
- **Ambient Assistance:** During streaming scenarios like live cooking demonstrations, MiniCPM-o 4.5 provides real-time narration, audience response, and proactive guidance based on live observations.

The continuous and aligned perception-generation loop achieved by Omni-Flow distinguishes MiniCPM-o 4.5 as a practical contender for human-like, scenario-adaptive multimodal AI [2604.27393].

Source: https://www.emergentmind.com/topics/minicpm-o-4-5