MiniCPM-o 4.5: Omni-Modal Interaction AI

• The paper introduces a 9-billion parameter model integrating a streaming visual encoder, audio encoder, and LLM backbone with token-level hidden state fusion.
• MiniCPM-o 4.5 is defined as an omni-modal foundation model that enables real-time, full-duplex interactions across vision, audio, text, and speech with proactive behavior mechanisms.
• It employs the novel Omni-Flow streaming architecture and INT4 quantization, ensuring state-of-the-art performance and edge efficiency on devices with less than 12 GB RAM.

MiniCPM-o 4.5 is a 9-billion-parameter omni-modal foundation model designed to enable real-time, full-duplex, human-like multimodal interactions. Distinguished by its simultaneous perception and generation capabilities across vision, audio, text, and speech, MiniCPM-o 4.5 leverages the novel Omni-Flow streaming architecture to achieve state-of-the-art open-source performance at its scale while remaining efficient enough for edge deployment on devices with less than 12 GB RAM. The model approaches the vision-language performance of Gemini 2.5 Flash and surpasses Qwen3-Omni-30B-A3B in omni-modal understanding and speech generation, marking a prominent advancement in practical, human-like AI interaction paradigms (Cui et al., 30 Apr 2026).

1. Model Architecture and Parameterization

MiniCPM-o 4.5 is an end-to-end model integrating three primary components: a streaming visual encoder based on SigLIP ViT with a resampler, a streaming audio encoder utilizing Whisper Medium and a two-layer MLP projector, and a LLM backbone (Qwen3-8B) interleaved with speech decoders. All encoders and decoders are interconnected via token-level hidden states, ensuring fully differentiable, end-to-end training.

Componentwise Parameter Breakdown:

Component	Parameters (Millions)
Visual encoder (SigLIP ViT)	417.8
Visual resampler	88.9
Audio encoder (Whisper Medium)	307.2
Audio projector (2-layer MLP)	21.0
LLM backbone (Qwen3-8B)	8,189.2
Backbone→speech-decoder projector	10.5
Speech-token decoder (Transformer)	188.8
Text embedding for decoder	116.8
Total	~9,341

Architectural details:

• Vision: SigLIP ViT employs 27 layers with a hidden dimension of 1,152, 16 attention heads, and 14×14 patch sizes. The resampler applies cross-attention to compress 1,024 to 64 tokens per slice (16× compression).
• Audio: Whisper Medium uses 24 layers (hidden size 1,024, 16 heads), producing 50 audio tokens/s, further reduced to 10 tokens/s by the MLP projector.
• LLM Backbone: Qwen3-8B is configured with 36 layers, a hidden dimension of 4,096, 32 heads, FFN size of 12,288, SiLU activations, and a RoPE positional encoding up to length 40,960.
• Speech Decoding: The model features a 20-layer Transformer decoder (hidden 768, 12 heads, 25 speech tokens/s), coupled with a flow-matching waveform decoder for real-time speech synthesis.

2. Omni-Flow Full-Duplex Streaming Framework

The central technical advance is the Omni-Flow framework, which reformulates interaction as the simultaneous streaming of three modalities: live image frames $I(t)$, acoustic input $A(t)$, and generated text/speech (out-stream). At each time $t$:

$$H(t)=\text{Concat}(f_v(I(t)), f_a(A(t)), h_\text{text}(t))$$

where $f_v(I(t))$ (visual encoder output), $f_a(A(t))$ (audio encoder output), and $h_\text{text}(t)$ (partial text generation state) are concatenated and processed in fixed chunks of duration $\Delta t$ (e.g., 1 s):

$$H(t+\Delta t) = H(t) + \Phi(f_v(I(t+\Delta t)), f_a(A(t+\Delta t)))$$

$\Phi(\cdot)$ denotes the serializing of new input tokens into context, followed by next-token generation. Each chunk yields groups of tokens—visual ($A(t)$0), audio ($A(t)$1), and output ($A(t)$2). These are combined as $A(t)$3 and serialized into the stream.

Simultaneous perception and generation is achieved per chunk: the model encodes current inputs, applies a control token predicting [listen] or [speak], autoregressively generates text tokens if in [speak] mode, and interleaves the textual output with speech tokens. The Time-Aligned Interleaving (TAIL) algorithm modulates the number of generated tokens per chunk, ensuring the speech waveform remains aligned with real-world time through a look-ahead window for accurate pronunciation context.

3. Proactive Behavior Mechanisms

MiniCPM-o 4.5 models proactive behavior by incorporating explicitly labeled proactive training samples (e.g., time-tagged scene descriptions, reminders, or live comments) within its full-duplex training data. The model’s next-token objective encompasses both reactive and proactive utterances, enabling output of intervening prompts or comments independent of explicit user queries.

During reinforcement learning (RL) fine-tuning, additional rewards structure this initiative:

• $A(t)$4: +1 for timely, context-driven prompts (e.g., reminders).
• $A(t)$5: smooth penalty for verbosity.
• $A(t)$6: standard accuracy reward on downstream tasks.

The RL objective is

$A(t)$7

Initiative emerges directly from joint next-token and reward signals; no separate proactive loss is necessary.

4. Inference Optimization and Edge Efficiency

MiniCPM-o 4.5 is engineered for real-time, full-duplex omni-modal interaction at edge-appropriate hardware constraints (<12 GB RAM). This is achieved through two major optimizations: vLLM inference with INT4 quantization and the custom C++ llama.cpp-omni backend.

Text-only throughput benchmarks (RTX 4090):

Model / Dtype	Throughput (tok/s)	1st-token Latency (s)	Memory (GB)
MiniCPM-o 4.5 (BF16)	154.3	0.59	19
MiniCPM-o 4.5 (INT4)	212.3	0.58	11
Qwen3-30B-A3B (INT4)	147.8	0.98	20

Streaming efficiency (llama.cpp-omni INT4):

Framework	RTX 4090 RTF	RTX 4090 Mem (GB)
llama.cpp-omni	0.21	11
PyTorch (INT4)	1.26	14

These results demonstrate that MiniCPM-o 4.5 sustains full-duplex streaming well below the 12 GB threshold.

5. Benchmark Performance and Comparative Evaluation

Vision-Language (Instruct Mode):

Benchmark	Gemini 2.5 F	Qwen3-Omni 30B	MiniCPM-o 4.5
OpenCompass	78.5	75.7	77.6
MMBench EN	86.6	84.9	87.6
TextVQA	74.3	84.1	83.8
OCRBench (ms)	864	880	876
Mantis-Eval	72.8	78.3	79.7

Omni-Modal (Simplex Mode):

Benchmark	Gemini 2.5 F	Qwen3-Omni 30B	MiniCPM-o 4.5
Daily-Omni	79.3	70.7	80.2
WorldSense	52.6	54.0	55.7
Video-Holmes	51.3	50.4	64.3
AVUT-Human	65.4	74.2	78.6

Full-Duplex Streaming (Vision Only):

• LiveSports-3K-CC: MiniCPM-o 4.5 achieves 54.4% (LiveCC: 41.5%, StreamingVLM: 45.6%).

Speech:

• Automatic Speech Recognition WER: AISHELL-1: 0.9% (Qwen3-Omni: 0.6%), GigaSpeech: 8.5% (Qwen3-Omni: 8.7%).
• Speech Translation BLEU (CoVoST2 en→zh): 49.9 (Qwen3-Omni: 46.6).
• Speech QA (VoiceBench): 4.81 (Qwen3-Omni: 4.74).

Speech Generation:

Model	SeedTTS-ZH CER	SeedTTS-EN WER	LongTTS EN WER	Expresso	ESD
CosyVoice2	1.45	2.57	14.80	17.9	53.4
Qwen3-Omni	1.41	3.39	17.33	N/A	N/A
MiniCPM-o 4.5	0.86	2.38	3.37	29.8	82.1

Text-Only Evaluation:

Benchmark	Qwen3-8B	MiniCPM-o 4.5
IFEval-PLS	83.0	84.7
BBH	69.4	81.1
MMLU	81.7	77.0*
MBPP	75.9	76.7
GSM8K	93.4	94.5

*slight dip on factual recall but strong overall performance.

6. Applications in Real-Time Omni-Modal Interaction

MiniCPM-o 4.5 enables a spectrum of real-time, full-duplex, omni-modal applications:

• Live Sports Commentary: Continuous ingestion of video and audio feeds allows ongoing natural language description of events, with midstream proactive interventions such as weather-related prompts.
• Multi-Party Conversational Assistance: In scenarios such as three-way conference calls, the model transcribes, analyzes visual content (e.g., slides), and interacts with multiple parties proactively.
• Ambient Assistance: During streaming scenarios like live cooking demonstrations, MiniCPM-o 4.5 provides real-time narration, audience response, and proactive guidance based on live observations.

The continuous and aligned perception-generation loop achieved by Omni-Flow distinguishes MiniCPM-o 4.5 as a practical contender for human-like, scenario-adaptive multimodal AI (Cui et al., 30 Apr 2026).