---
title: 'ROMA: Real-Time Omni-Multimodal Assistant'
url: https://www.emergentmind.com/topics/roma
type: topic
---

# ROMA: Real-Time Omni-Multimodal Assistant

RoMa, introduced as **ROMA: Real-time Omni-Multimodal Assistant**, is a unified streaming audio-video-text system designed for both reactive and proactive interaction in real time. It addresses three stated obstacles in streaming omni-multimodal understanding: the temporal granularity mismatch between dense audio and discrete video frames, incomplete modality support in prior streaming systems, and the absence of an autonomous mechanism for deciding when to intervene during an unfolding stream. ROMA processes continuous inputs as synchronized multimodal units, introduces a decoupled speak head for response initiation, and is trained with a two-stage curriculum over curated streaming data. Extensive experiments across 12 benchmarks report state-of-the-art performance on proactive tasks while remaining competitive in reactive question answering [2601.10323].

## 1. Problem formulation and system scope

ROMA is defined around a unified setting in which a single model must both answer user queries and autonomously monitor live streams for alerting or narration. The problem is not limited to post hoc understanding of recorded clips. Instead, the model operates causally over a stream and must support online decision making under latency constraints. The paper frames the difficulty in terms of three concrete issues: audio is dense and continuous while video arrives as discrete frames; many existing systems are either video-centric or speech-centric and therefore not truly unified; and most systems remain reactive rather than deciding on their own when to speak [2601.10323].

The model’s target behaviors are organized into two modes. In **reactive** interaction, ROMA answers questions over streaming audio-video inputs. In **proactive** interaction, it produces event-driven alerts or real-time narration without waiting for an explicit query. This joint formulation is central to the system design, because timing and content generation are treated as separate but coordinated problems. The architecture therefore does not reduce streaming multimodal understanding to token generation alone; it also includes an explicit mechanism for response triggering.

A plausible implication is that ROMA treats streaming multimodal assistance as a control problem as much as a representation problem. The paper’s design choices—synchronized multimodal units, causal KV caching, and a separate timing head—are all organized around this premise rather than around offline captioning or standard multimodal QA.

## 2. Streaming representation and temporal alignment

ROMA builds on an omni-modal LLM backbone with **Qwen2.5-Omni format compatibility**, while the vision and audio encoders are frozen during fine-tuning [2601.10323]. The stream is segmented into **one-second units**. Within each unit, video and audio are packed together on a shared timeline using special BOS/EOS markers:
```text
<|#1|>{vision_bos} <|#1|>{audio_bos} [video tokens] [audio tokens] <|#1|>{audio_eos} <|#1|>{vision_eos}
```
Frames are sampled at **2 fps**, each frame is resized so that total pixels are **≤ 65,536**, and audio is chunked at **40 ms** resolution.

The key alignment mechanism is **Chunked TMRoPE (Time-aligned Multimodal RoPE)**. ROMA adapts Qwen2.5-Omni’s TMRoPE to streaming by assigning cumulative positional IDs across units. Video tokens inside a one-second unit share the same temporal ID, whereas audio tokens retain fine-grained temporal IDs at 40 ms steps. The paper gives the conceptual positional scheme as
$$
\mathrm{pos}(i) = (\tau_u, h_i, w_i)
$$
for a video token \(i\) in unit \(u\), and
$$
\mathrm{pos}(j) = (\tau_u + 0.04 \cdot j, 0, 0)
$$
for audio token \(j\) within the same unit [2601.10323]. Boundary alignment is enforced by giving `vision_bos` and `audio_bos` the same base position ID, after which subsequent units continue the global timeline from the previous unit’s maximum.

This representation is intended to preserve cross-modal correspondence despite granularity mismatch. Multi-frame video is temporally aggregated by the vision encoder, while audio retains its finer temporal structure. A persistent KV cache maintains stream history, and at each step the model encodes only the current unit while attending to cached past states. The paper states that this preserves causality, avoids future access, and enables low-latency operation [2601.10323].

## 3. Decoupled response initiation: the speak head

ROMA’s most distinctive architectural component is the **speak head**, a lightweight two-layer MLP that runs in parallel to the language-model head. Its purpose is to decide **when** the system should start speaking, independently of the problem of **what** it should say. The paper formulates this decoupling as a means of avoiding inference conflicts between timing and generation [2601.10323].

The speak head receives a learnable weighted combination of the last \(K\) hidden layers, with \(K=4\) by default:
$$
h_t = \sum_{l=L-K+1}^{L} \alpha_l \cdot H_t^{(l)},
$$
and produces a trigger probability
$$
p_t = \sigma(W_2 \cdot \phi(W_1 \cdot h_t)),
$$
where \(\phi\) is an activation such as GELU and \(p_t \in (0,1)\) is the trigger probability at time step \(t\), identified with the unit index [2601.10323]. ROMA speaks when \(p_t > \tau\); otherwise it remains silent and continues ingesting the stream. The threshold \(\tau\) is task-dependent. The paper reports thresholds around **0.97–0.985** for narration and task-specific values for alert settings, sometimes combined with sliding-window smoothing to reduce transient fluctuations.

Timing supervision is framed as binary classification over stream time steps. With ground-truth trigger labels \(z_t \in \{0,1\}\), the timing loss is a weighted BCE:
$$
L_{\text{time}} = - \mathbb{E}_{X \sim D_{\text{stream}}} \left[ \frac{1}{T} \sum_{t=1}^{T} \left( w_{\text{pos}} \cdot z_t \cdot \log p_t + (1-z_t)\cdot\log(1-p_t) \right) \right].
$$
The positive weight is used to mitigate label sparsity, and the reported default is **\(w_{\text{pos}}=3\)** in most experiments [2601.10323].

This design separates temporal triggering from token generation rather than trying to infer both from a single decoding signal. The ablation evidence is correspondingly strong. On dynamic proactive alert tasks, removing the speak head reduces **StreamingBench PO** from **53.60** to **12.00**, **OVO-Bench REC** from **33.81** to **6.46**, and **OVO-Bench CRR** from **35.42** to **0.00**. On **YouCook2** narration, removing the speak head reduces **F1** from **35.21** to **9.25**, which the paper presents as demonstrating the necessity of decoupled timing [2601.10323].

## 4. Training data and curriculum

ROMA is trained on a curated streaming corpus that unifies proactive and reactive formats. The paper reports three principal data sources: **Online proactive (27K)**, **Online narration (109K)**, and **Reactive QA (540K)** [2601.10323]. Online proactive data is built from **DiDeMo, OOPS, and Charades-STA**, reformulated into prompts such as “Alert me when [event] happens.” Online narration data comes from **MMDuetIT, COIN, YouCook2, and ActivityNet**, with training configured to generate only at segment transitions rather than as dense captioning. Reactive QA draws from **InternVid, CogStream**, and other datasets including **Egoplan, AVQA, TimeChat-Online, and ViSpeak**. To unify modality, text queries are synthesized into speech and aligned with streaming units.

Training proceeds in two stages. **Stage 1 (Streaming template alignment)** adapts the model to the multimodal unit format using reactive QA data. For an answer token sequence \(Y=\{y_1,\ldots,y_L\}\), the language-model loss is
$$
L_{\text{LM}} = - \mathbb{E}_{(X,Y)\sim D_{\text{QA}}} \left[ \sum_{i=1}^{L} \log P(y_i \mid y_{<i}, X; \theta) \right].
$$
All encoders are frozen; only the remaining parameters \(\theta\) are fine-tuned [2601.10323].

**Stage 2 (Time-aware decision making)** trains the speak head with the weighted BCE timing objective while mixing a small portion of Stage-1 QA samples to stabilize language generation. The joint objective is
$$
L_{\text{total}} = L_{\text{time}} + \lambda \cdot L_{\text{LM}},
$$
where \(L_{\text{LM}}\) is computed only on the mixed QA samples and \(\lambda\) balances timing and generation [2601.10323]. Proactive samples are formatted as multi-turn dialogues so that multiple triggers can occur in a single stream.

The implementation details are unusually explicit. Training uses **LLaMA-Factory**, **sequence length 32K**, **32 H20 GPUs**, and **global batch size 512** [2601.10323]. These details place ROMA within the current lineage of LLM-based multimodal fine-tuning rather than within bespoke streaming architectures that retrain encoders or build specialized memory modules from scratch.

## 5. Evaluation suite, protocols, and empirical results

A substantial part of ROMA’s contribution is evaluative standardization. The paper reorganizes fragmented streaming benchmarks into two settings—**proactive streaming interaction** and **reactive QA**—so that alerting, narration, and QA can be compared under a unified protocol [2601.10323].

| Setting | Benchmarks | Metrics |
|---|---|---|
| Proactive alert, static | QVHighlights; Charades-STA | mAP, HIT@1; R@0.5, R@0.7 |
| Proactive alert, dynamic | OmniMMI (PA); StreamingBench (PO); OVO-Bench (CRR, REC) | success if trigger time falls within ground-truth interval |
| Real-time narration | YouCook2; OVO-Bench SSR | F1, BERTScore, GPT-4o-based scoring |
| Reactive QA | OVO-Bench; StreamingBench; Video-MME (no subtitles); EgoSchema | accuracy; GPT-4o exact-match protocol |

On **proactive alerts (static)**, ROMA reports **53.7 mAP / 53.0 HIT@1** on **QVHighlights**, outperforming **MMDuet (31.3 / 49.6)** and **VTG-LLM (16.5 / 33.5)**. On **Charades-STA**, it reports **44.3 R@0.5 / 19.9 R@0.7**, improving over **MMDuet (42.4 / 18.0)** [2601.10323].

On **dynamic proactive alerts**, the reported scores are **37.50** on **OmniMMI PA**, **53.60** on **StreamingBench PO**, **35.42** on **OVO-Bench CRR**, and **33.81** on **OVO-Bench REC**. On **real-time narration**, ROMA reaches **F1 35.21, BERTScore 0.83, GPT-4o score 0.39** on **YouCook2**, and **F1 14.54, BERTScore 0.83, GPT-4o score 0.42** on **OVO-Bench SSR** [2601.10323].

Reactive QA remains competitive rather than uniformly dominant. The paper states that ROMA leads in several **OVO-Bench Real-time Visual Perception** categories, including **OCR 63.09**, **ATR 68.10**, **FPD 69.31**, and **OJR 58.15**, and is strong in **Backward Tracing**, including **EPM 55.89** and **ASI 47.30**. On full-modality QA with spoken queries, ROMA reports **33.30** on **Video-MME (no subtitles)**, outperforming **Qwen2.5-Omni (20.50)**, **VITA-1.5 (28.56)**, and **MiniCPM-o (19.37)**, while scoring **55.40** on **EgoSchema**, competitive with **Qwen2.5-Omni 58.40** [2601.10323].

The throughput profile is also part of the system argument. Under the streaming protocol, the average encoding time per unit is **0.3697 s**. The paper uses a pipelined approximation in which unit \(t\) is processed while unit \(t+1\) is being acquired, and generation is capped at **25 tokens per segment (~1 s)**, with longer outputs continued in subsequent units [2601.10323]. This suggests that the model’s real-time claim is grounded not only in accuracy metrics but also in a concrete latency budget.

## 6. Positioning, limitations, and significance

ROMA is situated against two neighboring lines of work. Relative to **video-centric streaming LLMs** such as **VideoLLM-Online**, **Flash-VStream**, and memory/KV-cache methods, it adds synchronized audio and a proactive timing mechanism rather than focusing only on long-horizon memory or efficiency. Relative to **omni-modal models** such as **MiniCPM-o**, **Qwen2.5/3-Omni**, and **Stream-Omni**, it explicitly models the proactive “when to speak” decision through the speak head and aligns streaming audio with video at chunk-level temporal positions [2601.10323]. The paper also emphasizes that its evaluation suite standardizes proactive and reactive modes that prior benchmarks often treat inconsistently.

The stated limitations are equally important. ROMA remains susceptible to **signal degradation** and **audio-video asynchrony**. **Extremely long-horizon dependencies (hours)** are still constrained by **context windows and memory budgets**. The paper further notes unresolved **efficiency vs quality trade-offs under strict resource limits**, mentioning **adaptive sampling** and **smarter KV management** as directions for further optimization [2601.10323].

The ethical section is brief but specific: proactive monitoring poses **privacy risks**, the model is intended for **research**, and **human oversight** is necessary given possible **hallucinations** and **biases** [2601.10323]. This is not ancillary, because a system whose central capability is autonomous intervention in live streams raises questions that do not arise in purely reactive multimodal QA.

In research terms, ROMA’s central claim is not simply that one more omni-modal assistant can process audio, video, and text. Rather, it shows that unified real-time omni-modal understanding can be organized around three coupled mechanisms: synchronized multimodal units for alignment, a decoupled speak head for timing, and a streaming curriculum that separately adapts format and responsiveness. The empirical pattern—state-of-the-art proactive results together with competitive reactive performance—suggests that explicit timing decisions are a structural requirement for real-world streaming assistants rather than a secondary engineering detail [2601.10323].

Source: https://www.emergentmind.com/topics/roma