PLUME: Latent Reasoning Multimodal Embedding

• PLUME is a universal multimodal embedding framework that maps text, images, videos, and visual documents into a shared latent space.
• It employs a latent rollout mechanism with semantic-anchor-guided adapters to replace explicit chain-of-thought, offering over 30× faster inference.
• Benchmark results demonstrate that PLUME enhances retrieval, particularly in video and visual document tasks, while balancing accuracy and computational efficiency.

Searching arXiv for the specified paper and closely related UME context. PLUME, short for "PLUME: Latent Reasoning Based Universal Multimodal Embedding," is a universal multimodal embedding framework that maps heterogeneous inputs—text, images, videos, and visual documents—into a single shared retrieval space with one model (He et al., 2 Apr 2026). It is designed for retrieval settings in which a single encoder must handle complex intent, including compositional reasoning, spatial or temporal aggregation, and layout structure. PLUME replaces explicit chain-of-thought rationale generation with a short autoregressive rollout of continuous latent states, adds a semantic-anchor-guided transition adapter to steer that latent rollout, and trains the system with a progressive explicit-to-latent curriculum. On the 78-task MMEB-v2 benchmark, it reports 61.6 overall, compared with 60.1 for the explicit-CoT baseline UME-R1 and 58.0 for the single-pass baseline VLM2Vec-V2, while reducing reasoning from hundreds of generated tokens to fewer than 10 latent steps and delivering over 30x faster inference (He et al., 2 Apr 2026).

1. Problem setting and motivation

Universal multimodal embedding aims to place heterogeneous inputs in a shared space so that retrieval can be performed across modalities. In the formulation used by PLUME, a single universal encoder must align semantics across text, images, videos, and visual documents while also resolving complex query intent such as compositional reasoning, spatial or temporal aggregation, and layout structure (He et al., 2 Apr 2026).

Recent UME systems addressed this difficulty with explicit chain-of-thought, following a "think-then-embed" pattern in which a multimodal LLM first generates long textual rationales and only then produces an embedding. PLUME is motivated by two bottlenecks in that design. The first is computational overhead: hundreds of generated tokens per sample increase autoregressive decoding time and KV-cache memory, reducing throughput and batch size during retrieval. The second is a textual bottleneck: forcing multimodal evidence through discrete text can discard fine-grained continuous cues such as subtle spatio-temporal signals or dense layout features (He et al., 2 Apr 2026).

PLUME addresses these bottlenecks by moving intermediate reasoning into hidden-state computation. The latent rollout preserves multi-step computation and sequential dependency, but avoids token generation. This design is especially targeted at retrieval regimes in which relevant evidence is dense, structurally complex, and difficult to organize through verbalized intermediate rationales, particularly video and visual document retrieval (He et al., 2 Apr 2026).

2. Core architecture and latent reasoning mechanism

PLUME is built on a fully fine-tuned Qwen2-VL-2B backbone. Inputs can be standard text tokens, visual tokens or frame or patch features for images and video, and rendered layout tokens or document images for visual documents (He et al., 2 Apr 2026).

Its architecture contains four principal components. The first is a latent state representation $z$, initialized from the multimodal prefix. The second is a short autoregressive latent rollout of $K$ steps, where the backbone advances one causal position per step as if decoding, but consumes continuous vectors rather than token embeddings. The third is a semantic-anchor-guided transition adapter, implemented as a lightweight routed adapter with a shared expert and specialized experts. The fourth is the embedding formation rule: there is no separate embedding head, and the final retrieval embedding is the normalized hidden state at a special <gen> position after the latent block (He et al., 2 Apr 2026).

The data flow is explicitly staged. The model first prefix-encodes the interleaved text and visual context and caches KV states for causal attention. It then computes a semantic anchor from a dedicated <anchor> token, initializes the latent state at the <slt> position, performs $K$ latent steps, places <gen> after the latent block ending at <elt>, and extracts the L2-normalized hidden state at <gen> as the retrieval embedding (He et al., 2 Apr 2026).

Formally, if $x$ denotes the multimodal input and the prefix encoding yields last-layer hidden states $h_1,\ldots,h_L$ and KV-cache $C(x)$, then the semantic anchor is $c(x)$ at the <anchor> position and the latent initialization is

$z_0 = h_L.$

The latent process is written as

$$z_{t+1} = f_\theta(z_t, x, a), \qquad a \equiv c(x),$$

where $f_\theta$ is realized by routed adaptation followed by one autoregressive backbone step (He et al., 2 Apr 2026).

The adapter routing depends on the current latent state, the semantic anchor, and a learnable step embedding:

$K$0

The adapted state is then

$K$1

followed by one causal backbone step with a growing KV-cache:

$K$2

The result is a latent trace $K$3 in which each step attends to the full multimodal prefix and previous latent states through causal attention (He et al., 2 Apr 2026).

A central property of the method is fixed compute budget. $K$4 is small—typically fewer than 10, and $K$5 in the main model—but the trajectory remains input-adaptive because routing depends on both the global signal $K$6 and the evolving local signal $K$7 (He et al., 2 Apr 2026).

3. Semantic anchors, embedding formation, and training curriculum

Semantic anchors provide a fixed global summary for routing. In PLUME, the anchor $K$8 is the last-layer hidden state at a dedicated <anchor> position in the prefix, has the same dimensionality as the backbone hidden state, and is L2-normalized when used in auxiliary contrastive training. Its role is to stabilize routing so that the router does not overfit to the rapidly changing latent state alone (He et al., 2 Apr 2026).

To prevent expert collapse, PLUME adds a balance regularizer:

$K$9

where $K$0 is the average routing mass assigned to expert $K$1 over the batch and latent steps (He et al., 2 Apr 2026).

The retrieval embedding is taken from the generative pathway:

$K$2

An auxiliary anchor embedding is also defined for training,

$K$3

but only $K$4 is used at inference. This prevents shortcutting through the anchor and forces the latent rollout to participate in retrieval representation formation (He et al., 2 Apr 2026).

Training combines several objectives. Bidirectional InfoNCE is applied to both $K$5 and $K$6, using cosine similarity between L2-normalized embeddings and temperature $K$7. A causal language modeling loss $K$8 is applied on the decoded suffix of both the query and its positive target during curriculum stages. The total loss is

$K$9

The backbone is fully fine-tuned rather than converted into a sparse MoE backbone; the added parameters are the lightweight routed adapter and router (He et al., 2 Apr 2026).

A distinctive feature of PLUME is its progressive explicit-to-latent curriculum. Stage 0 uses fully explicit CoT with teacher-forced rationale and answer. Stages 1–3 progressively replace the rationale from left to right with a latent block delimited by <slt> and <elt>, while supervising the remaining unreplaced rationale tokens after <elt>. Stage 4 is fully latent: both rationale and answer spans are removed so that the latent block connects directly to <gen>, matching inference (He et al., 2 Apr 2026).

The reported schedule is five epochs total: Stage 0 for three epochs, Stages 1–3 jointly for one epoch, and Stage 4 for one epoch, with global batch size 1024, learning rate $x$0, and $x$1. The curriculum is described as critical: skipping it causes a $x$2 overall score drop (He et al., 2 Apr 2026).

4. Inference pipeline, efficiency, and benchmark results

At inference, PLUME does not generate explicit chain-of-thought. The pipeline prefix-encodes the input, reads the anchor state, initializes the latent state, performs $x$3 latent steps with routed adaptation and cached causal attention, and returns the normalized hidden state at <gen> as the retrieval embedding (He et al., 2 Apr 2026).

The efficiency claim follows directly from replacing token generation with latent rollout. In the main configuration, PLUME uses $x$4 latent steps, whereas the explicit-CoT baseline UME-R1 uses approximately 403 generated tokens. On a single NVIDIA H20 GPU, mean latency over five runs of 500 samples is reported as $x$5 ms per sample for PLUME and $x$6 ms per sample for UME-R1, with throughput $x$7 versus $x$8 samples per second, yielding a $x$9 speedup. Relative to the single-pass VLM2Vec-V2 baseline at $h_1,\ldots,h_L$0 ms, PLUME incurs about $h_1,\ldots,h_L$1 latency overhead while attaining higher accuracy (He et al., 2 Apr 2026).

MMEB-v2 contains 78 tasks across image, video, and visual document retrieval. The benchmark uses Hit@1 for image and video tasks and NDCG@5 for visual document tasks. PLUME reports the following aggregate results (He et al., 2 Apr 2026):

Setting	Score
PLUME overall	61.6
UME-R1 overall	60.1
VLM2Vec-V2 overall	58.0
Image	66.3
Video	44.1
VisDoc	67.5

The modality breakdown is important. Relative to UME-R1, PLUME is slightly lower on image tasks, with 66.3 versus 66.6, but higher on video, with 44.1 versus 42.2, and substantially higher on visual documents, with 67.5 versus 63.9. The strongest reported gains occur in video multi-modal retrieval, where it scores 46.7 versus 39.7, and in visual-document OOD, where it reaches 57.4. It also reports 79.7 on image grounding, marked as best in the summary (He et al., 2 Apr 2026).

The reported accuracy–efficiency Pareto emphasizes that the latent budget can be varied. With $h_1,\ldots,h_L$2, PLUME already surpasses UME-R1’s accuracy while running more than $h_1,\ldots,h_L$3 faster. With $h_1,\ldots,h_L$4, throughput approaches single-pass models while still exceeding them in accuracy. This suggests that most of the benefit of latent reasoning is obtained before the maximum $h_1,\ldots,h_L$5 setting, although the paper characterizes the improvement beyond $h_1,\ldots,h_L$6 as modest rather than absent (He et al., 2 Apr 2026).

5. Ablations, routing behavior, and latent trajectory analysis

Ablation results isolate the contribution of each component. The full model scores 61.6 overall. Removing the latent transition and skipping the rollout reduces the score to 58.8. Replacing the routed mixture-of-experts adapter with a single MLP adapter yields 59.2. Removing the semantic anchor from the router yields 60.1, with a Video drop of 1.8. Removing the curriculum and switching directly from explicit to latent reasoning yields 54.8 overall, including a Video drop of 7.6 (He et al., 2 Apr 2026).

The number of latent steps exhibits roughly linear latency growth with diminishing accuracy returns:

• $h_1,\ldots,h_L$7: 59.9 overall, 232 ms
• $h_1,\ldots,h_L$8: 61.1 overall, 268 ms
• $h_1,\ldots,h_L$9: 61.6 overall, 300 ms

Adapter design ablations show that the default configuration—$C(x)$0, $C(x)$1, and a shared expert—performs best at 61.6. Removing the shared expert lowers performance to 60.3. Using Top-1 rather than Top-2 gives 60.8. Removing the anchor from the router gives 60.1, and removing the step embedding $C(x)$2 gives 60.4 (He et al., 2 Apr 2026).

Qualitative analyses indicate that routing specialization emerges without task labels. One expert peaks on video classification and multi-modal video retrieval, another is preferred on QA tasks, and a third favors image grounding and retrieval. The reported interpretation is that anchor-conditioned routing learns meaningful specializations that steer latent computation (He et al., 2 Apr 2026).

A separate analysis of intermediate-state cosine similarity to the positive target, averaged over 200 samples, finds that PLUME’s latent trajectories are smoother and less variable than the hidden trajectories associated with explicit-CoT generation, especially on video retrieval. This suggests that continuous latent computation can produce more stable intermediate representations than discrete token generation, though the latent states remain less directly interpretable than text rationales (He et al., 2 Apr 2026).

6. Limitations, deployment, and future directions

PLUME is not presented as uniformly superior across all retrieval-adjacent settings. The reported failure cases include an Image QA gap on some text-rich or knowledge-intensive examples, such as charts, infographics, or cases requiring heavy external knowledge. The explanation offered in the paper is that explicit verbalization can preserve fine-grained textual structure helpful to QA, whereas PLUME compresses reasoning for retrieval-oriented embeddings (He et al., 2 Apr 2026).

A second limitation is anchor sensitivity: poor anchor quality degrades routing, even though auxiliary anchor contrastive supervision improves it. A third is interpretability: expert activations are interpretable to a degree, but latent trajectories lack the explicit traceability of text chain-of-thought. A fourth concerns generalization: although PLUME improves OOD scores in visual documents, broader OOD stress tests and domain-shift analyses remain future work (He et al., 2 Apr 2026).

In deployment terms, PLUME is designed for retrieval infrastructure rather than generative answer production. Corpus items are encoded as

$C(x)$3

stored in a vector database, and compared with cosine similarity over L2-normalized embeddings. Because the latent rollout uses standard KV-cache generation semantics and only $C(x)$4 single-position steps, it is compatible with standard generation serving, including paged attention. The paper characterizes its memory overhead as minimal beyond those $C(x)$5 steps and substantially lower than explicit-CoT pipelines that decode hundreds of tokens (He et al., 2 Apr 2026).

Potential extensions listed in the paper include adaptive step budgets, improved anchor learning, richer transition adapters, self-distillation or reinforcement learning tied to retrieval metrics, and multi-vector outputs. These proposals suggest that the PLUME formulation is intended not only as a specific model instance but as a general recipe for reasoning-aware universal embeddings in which latent computation, rather than textual rationale generation, carries the intermediate reasoning burden (He et al., 2 Apr 2026).