Multimodal Prediction Model (MPM)

• MPM is a prediction framework that integrates diverse modalities (e.g., images, graphs, text) to estimate outcomes like price ranges and technical metrics.
• It employs distinct modality-specific encoders and explicit fusion techniques—such as concatenation and attention—to effectively combine heterogeneous features.
• MPM demonstrates improved performance in practical tasks, yielding gains in metrics like F1-score, RMSE, and resource utilization predictions across varied domains.

A Multimodal Prediction Model (MPM) is a predictive framework that uses more than one input modality to estimate a target variable, class, trajectory, property, or system behavior. In the arXiv literature, the term does not denote a single canonical architecture; rather, it appears as a family of application-specific formulations that combine heterogeneous inputs such as structured attributes and images, graphs and source code, multilingual text and visual signals, or multimodal model structure and training configuration. Across these uses, the common principle is that prediction quality depends on explicitly modeling complementary information sources instead of reducing the problem to a unimodal surrogate (Zehtab-Salmasi et al., 2020).

1. Terminological scope and canonical uses

The designation MPM has been used explicitly in several distinct settings. In cellphone price prediction, it names a two-stream classifier that combines non-graphical features and graphical features to predict one of four price classes (Zehtab-Salmasi et al., 2020). In high-level synthesis design space exploration, the MPM component of MPM-LLM4DSE is a multimodal regression model that fuses control and data flow graphs (CDFGs) with behavioral descriptions/source code to predict quality-of-results metrics such as latency, LUT, DSP, FF, and BRAM (Xu et al., 8 Jan 2026). In multilingual multimodal learning, MPM is a training paradigm in which a multilingual language backbone lets multimodal alignment learned from English-only image-text pretraining generalize to other languages through a pivot mechanism (Hu et al., 2023).

A closely related use appears in GPU memory estimation for multimodal training. That work does not present “MPM” as the formal name of the framework, but it describes an MPM-like approach that parses a multimodal architecture into modules and layers, factorizes memory into model parameters, optimizer states, gradients, and activations, and aggregates layer-wise estimates into predicted peak GPU memory (Jeong et al., 26 Nov 2025). This suggests that, beyond named architectures, “MPM” also functions as a methodological label for explicit multimodal decomposition.

Representative use	Modalities or inputs	Predicted output
Cellphone price prediction (Zehtab-Salmasi et al., 2020)	structured specifications + cellphone images	4-class price range
GPU memory prediction (Jeong et al., 26 Nov 2025)	multimodal model structure + training behavior	peak GPU memory
HLS DSE surrogate (Xu et al., 8 Jan 2026)	CDFG + pragma-augmented source code	QoR metrics
Pivot multilingual multimodal learning (Hu et al., 2023)	images + multilingual text via English pivot	image-to-text or text-to-image outputs

2. Shared structural principles

Despite domain heterogeneity, several structural motifs recur. The first is modality-specific encoding. In the cellphone system, non-graphical features are encoded by dense layers while images are encoded by Inception-V3 or a custom CNN, and the resulting embeddings are concatenated before classification (Zehtab-Salmasi et al., 2020). In MPM-LLM4DSE, the graph modality is encoded by Enhanced-CoGNN (ECoGNN) and the textual modality by CodeBERT-c, yielding graph and semantic embeddings that are fused for QoR regression (Xu et al., 8 Jan 2026).

The second motif is explicit fusion rather than implicit sharing. In the cellphone formulation, fusion is direct concatenation,

$f = [f_n ; f_g],$

where $f_n$ is the non-graphical embedding and $f_g$ is the graphical embedding (Zehtab-Salmasi et al., 2020). In the HLS setting, fusion is performed by multi-head attention, with graph embedding $h_g$ used as query and text embedding $h_s$ as key and value: $Target_i = MLP_i(\text{Fuse}(h_g, h_s)).$ The paper further states that aligned graph and fused features are passed through MLPs and then a gated network to dynamically control information flow (Xu et al., 8 Jan 2026).

The third motif is task-specific prediction heads. Some MPMs are discriminative classifiers, others are regressors, and others are multimodal generative systems. The literature therefore treats “multimodal prediction” as a problem family rather than a single output type. This suggests that the architectural invariants lie in modality decomposition and fusion, not in any fixed decoder form.

3. Cellphone price prediction as an early explicit MPM

The paper “Multimodal price prediction” defines an explicit Multimodal Prediction Model (MPM) for cellphone price prediction formulated as a 4-class classification problem with

$y \in \{0,1,2,3\}.$

The classes correspond to the price thresholds $<250$, $250 \leq \text{price} < 500$, $500 \leq \text{price} < 750$, and $f_n$0 (Zehtab-Salmasi et al., 2020).

Its multimodal premise is that cellphone value is influenced by both technical specifications and visual/product cues. The system therefore uses two input sources: structured GSMArena specifications and the cellphone image. The overall pipeline is fixed: encode non-graphical features into a low-dimensional embedding, encode graphical features into an image embedding, concatenate both embeddings, and apply a classifier head with softmax (Zehtab-Salmasi et al., 2020).

The paper proposes five models: one unimodal baseline and four multimodal variants. The unimodal baseline uses only non-graphical features with three fully connected layers and a softmax output. The multimodal systems differ mainly in image encoding and, in some cases, in whether the structured branch is processed by dense layers or CNNs. Model 2 uses Inception-V3, specifically the output of the second “Grid Size Reduction” block, while Model 3 uses a custom CNN with 3 convolutional layers, 3 max-pooling layers, and 1 fully connected layer (Zehtab-Salmasi et al., 2020).

The final classifier structure is also specified: 3 fully connected layers, the first two with ReLU, neuron counts 300 and 120, L1 regularization with $f_n$1, RMSProp, learning rate 0.001, and batch size 32 (Zehtab-Salmasi et al., 2020). The best reported result is 88.3% F1-score, achieved by Model 3 + Fully connected classifier, with Precision 89.1, Recall 88.0, and Accuracy 88.0. The paper further reports that multimodal methods improve F1 and accuracy by more than 2% compared with unimodal methods (Zehtab-Salmasi et al., 2020).

In this formulation, MPM is a textbook two-stream predictor: separate encoders, explicit fusion by concatenation, and a downstream neural classifier. Its significance lies less in fusion novelty than in the clear assertion that multimodal learning can outperform unimodal specification-based price prediction in a structured industrial domain.

4. MPM as structured multimodal surrogate modeling

In MPM-LLM4DSE, the MPM is a multimodal surrogate for HLS design space exploration. The paper’s argument is that CDFG-only GNN surrogates do not fully capture the semantics of source-level HLS programs, especially the meaning and scope of pragma directives, so prediction should combine graph structure with source-code semantics (Xu et al., 8 Jan 2026).

The framework consumes two modalities per HLS design instance. The graph branch uses a CDFG derived from LLVM IR / ProGraML, enriched with pragma-related information. The text branch uses the behavioral description/source code, with design configuration merged into the text input. The semantic embedding is obtained from CodeBERT-c using the [CLS] token of the final layer: $f_n$2 The graph encoder is ECoGNN, which introduces directed node states

$f_n$3

uses LayerNorm, Gumbel-Softmax, and GNN aggregators named MEANGNN and SUMGNN, and produces node features that are pooled by global node attention into graph embedding $f_n$4 (Xu et al., 8 Jan 2026).

Fusion is performed with multi-head attention: $f_n$5 The fused representation is then fed to MLP prediction heads for latency, LUT, DSP, FF, and BRAM (Xu et al., 8 Jan 2026).

The evaluation uses 15 benchmarks from MachSuite and PolyBench, totaling 4,353 graph-text samples, with labels obtained using Vitis-HLS 2022.1 and Vivado 2022.1. The primary metric is RMSE. Reported overall RMSE values are 0.4084 for ECoGNN-only, 0.3965 for LM-only, and 0.3898 for MPM, with the paper stating that MPM improves over ProgSG by up to 10.25× on certain metrics (Xu et al., 8 Jan 2026).

Here, MPM is not merely a fusion model; it is a surrogate evaluator inside a larger optimization loop. The surrounding LLM4DSE system uses the MPM output to guide design exploration, but the MPM itself remains the predictive core.

5. MPM-like decomposition for multimodal systems analysis

The paper “GPU Memory Prediction for Multimodal Model Training” proposes a framework whose purpose is to prevent out-of-memory (OoM) failures by predicting peak GPU memory usage before training runs. Although the work does not formalize the acronym MPM as its name, it explicitly describes an MPM-like approach: a multimodal architecture is decomposed into modules and then into layers, and memory is estimated by factorization (Jeong et al., 26 Nov 2025).

The motivation is that existing memory predictors are designed for unimodal models and do not generalize to multimodal systems such as vision-LLMs. The paper uses LLaVA-1.5 (7B) as the representative case, with CLIP ViT-L/14 as the vision encoder and a Vicuna-based language decoder. It emphasizes heterogeneity across both modules and training stages. In pre-training, only the projection layer is updated; in fine-tuning, the projection layer and parts of the language module are updated, while the vision module remains frozen. Consequently, gradients, optimizer states, and activations are not uniformly present across layers (Jeong et al., 26 Nov 2025).

The framework’s workflow is: a model parser analyzes the architecture, extracts modules based on modality, reads a configuration file with hyperparameters such as batch size, decomposes each module into fine-grained layers such as nn.Linear, estimates memory per layer through factorization, computes memory for each factor on each layer, and aggregates the results into total predicted peak memory (Jeong et al., 26 Nov 2025). The four factors are:

• model parameters
• optimizer states
• gradients
• activations

The peak memory estimate is written in the paper as a module- and layer-wise sum of these factors. The text further clarifies that $f_n$6 in a multimodal model is computed only for modalities whose parameters are being updated (Jeong et al., 26 Nov 2025).

The evaluation is performed on eight NVIDIA H100 80GB GPUs (NVLink) with PyTorch 24.07 and ZeRO-2 from DeepSpeed. Two training configurations are tested: SeqLen = 1024, MBS = 16 and SeqLen = 2048, MBS = 8, each with DP varying from 1 to 8. Reported accuracy is 13% average MAPE for the first setting and 8.7% average MAPE for the second (Jeong et al., 26 Nov 2025).

This use broadens the meaning of MPM. The target is not a class label or regression value derived from external data modalities, but a systems quantity derived from multimodal model structure and training behavior. A plausible implication is that the MPM idea can be understood more generally as explicit multimodal factorization for prediction, even when the modalities are internal architectural components.

6. Pivot-based multilingual multimodal prediction

The paper “Large Multilingual Models Pivot Zero-Shot Multimodal Learning across Languages” uses MPM to denote a training paradigm for multilingual multimodal models. Its core claim is that a strong multilingual language backbone allows multimodal alignment learned from English-only image-text pretraining to transfer to non-English languages through a pivot mechanism (Hu et al., 2023).

The paper frames multimodal learning in a target language $f_n$7 as either image-to-text $f_n$8 or text-to-image $f_n$9, and introduces a pivot language $f_g$0, taken to be English. The crucial assumption is that the multilingual LLM maps semantically equivalent text into nearby hidden states: $f_g$1 On that basis, multimodal training is decomposed into two stages: multilingual alignment via a pretrained multilingual LLM, and multimodal alignment via English-only image-text pairs (Hu et al., 2023).

For image-to-text generation, visual features $f_g$2 are concatenated with text embeddings and fed into the LLM. The pretraining objective is

$f_g$3

During this stage, the LLM is frozen and only the visual module is trained. Instruction tuning then uses multimodal instruction data with objective

$f_g$4

For text-to-image generation, the paper uses a Stable-Diffusion-style objective with a multilingual LLM and UNet decoder (Hu et al., 2023).

The concrete Chinese instantiation is VisCPM, with VisCPM-Chat for image-to-text and VisCPM-Paint for text-to-image, using CPM-Bee as the Chinese-English bilingual backbone. The main empirical claim is that VisCPM-Chat and VisCPM-Paint achieve state-of-the-art open-source performance in Chinese while relying primarily on English multimodal pretraining (Hu et al., 2023). On the LLaVA Test Set in Chinese, the reported average is 90.9 for VisCPM-Chat and 92.5 for VisCPM-Chat+, compared with 88.2 for Qwen-VL-Chat. For Chinese text-to-image generation on MSCOCO, VisCPM-Paint reports FID 10.9 and VisCPM-Paint+ FID 9.6, compared with 16.1 for AltDiffusion and 15.6 for TaiyiDiffusion (Hu et al., 2023).

This formulation is unusual within the MPM family because the central multimodal innovation is not a new fusion block but a cross-lingual transfer recipe. The term therefore encompasses both architectural objects and training paradigms.

7. Recurring strengths, limitations, and conceptual boundaries

Across its uses, MPM is consistently motivated by the insufficiency of unimodal predictors. In cellphone price prediction, structured specifications alone miss information in the device image (Zehtab-Salmasi et al., 2020). In HLS DSE, graph structure alone misses source-level semantic cues about pragma scope and programmer intent (Xu et al., 8 Jan 2026). In multilingual multimodal learning, target-language multimodal data are scarce, so English multimodal alignment must be transferred through a multilingual backbone (Hu et al., 2023). In GPU memory estimation, unimodal memory formulas “do not work at all” on multimodal architectures because heterogeneous modules and stage-dependent trainability change the memory composition (Jeong et al., 26 Nov 2025).

The principal fusion strategies differ substantially. Some MPMs use direct concatenation; others use attention-based fusion; others use factorized decomposition over layers, modules, or language pivots. This suggests that “MPM” is best understood as a design stance: multimodal structure should be represented explicitly, and the predictor should preserve modality-specific information until an appropriately chosen integration stage.

The limitations are likewise domain-specific. The cellphone MPM uses a relatively small GSMArena-derived dataset and only simple concatenation (Zehtab-Salmasi et al., 2020). The HLS MPM depends on graph-text sample construction and HLS tool reports for supervision (Xu et al., 8 Jan 2026). The multilingual MPM depends on a strong multilingual LLM and notes that noisy target-language multimodal corpora may hurt unless filtered (Hu et al., 2023). The GPU memory framework assumes additivity across layers and modules and relies on per-factor analytical equations rather than runtime profiling (Jeong et al., 26 Nov 2025).

Taken together, these works indicate that Multimodal Prediction Model is not a single standardized model class analogous to a transformer or a diffusion model. It is a recurring label for systems that treat prediction as an explicitly multimodal problem and solve it by combining modality-specific representations, modality-aware structure, or modality-aware training procedures. The common denominator is not a fixed architecture, but the claim that useful prediction requires modeling the heterogeneity of the information sources themselves.