Mind-Omni: Unified Multimodal Frameworks

• Mind-Omni is a set of multimodal frameworks that unify brain, vision, and language data through discrete tokenization and diffusion techniques.
• It employs staged optimization with specialized alignment and tokenization methods to enhance interoperability across diverse modalities.
• Quantitative analyses demonstrate strong performance in image decoding and reasoning tasks while addressing data scarcity, ethical, and technical limitations.

Searching arXiv for papers related to “Mind-Omni” and closely named systems. Mind-Omni is an overloaded label rather than a single canonical system in the recent arXiv literature. In direct title usage, it refers most specifically to two distinct multimodal research programs: "Mind-Omni: A Unified Multi-Task Framework for Brain-Vision-Language Modeling via Discrete Diffusion" (Lu et al., 28 May 2026) and "MindOmni: Unleashing Reasoning Generation in Vision LLMs with RGPO" (Xiao et al., 19 May 2025). The term is also easily conflated with other similarly named projects. Most notably, the OMIND paper states that “Mind-Omni” does not appear in the paper and that such references likely reflect a misreading of oMind (Racha et al., 26 Mar 2026). The common thread across these works is multimodal unification, but the technical objects differ sharply: one centers on fMRI tokenization and seven-way brain–vision–language translation, while the other integrates a vision-LLM with a decoder-only diffusion module and reinforcement learning for explicit reasoning-aware image generation.

1. Nomenclature and scope

The name space around Mind-Omni includes several distinct artifacts, only some of which actually use the label in their titles. The most direct usages are the 2025 MindOmni model for reasoning generation in vision-language systems and the 2026 Mind-Omni framework for brain–vision–language modeling (Xiao et al., 19 May 2025, Lu et al., 28 May 2026). By contrast, OMIND/oMind is a mental-health LLM framework whose authors explicitly clarify that “Mind-Omni” is not their name and likely arises from third-party confusion (Racha et al., 26 Mar 2026).

Name	arXiv id	Scope
Mind-Omni	(Lu et al., 28 May 2026)	Brain–vision–language modeling via discrete diffusion
MindOmni	(Xiao et al., 19 May 2025)	Unified multimodal LLM with reasoning-aware image generation
oMind / OMIND	(Racha et al., 26 Mar 2026)	Knowledge-grounded mental-health LLM framework
MiniMind-O	(Gong, 5 May 2026)	Open small-scale speech-native omni model
BrainOmni	(Xiao et al., 18 May 2025)	Unified EEG–MEG brain foundation model
Omni-based social robot study	(Senft et al., 2022)	Socially appropriate passing behavior in narrow spaces

This naming ambiguity matters because the cited systems target different modalities, tasks, and scientific questions. In one case, the goal is token-level interoperability among brain signals, image tokens, and text tokens; in another, it is explicit Chain-of-Thought reasoning that conditions visual generation; in still others, the focus is mental-health dialogue, speech-native interaction, electrophysiology, or robot navigation (Lu et al., 28 May 2026, Xiao et al., 19 May 2025, Racha et al., 26 Mar 2026, Gong, 5 May 2026, Xiao et al., 18 May 2025, Senft et al., 2022).

2. Mind-Omni as a brain–vision–language framework

The 2026 Mind-Omni framework is presented as the first versatile framework that unifies seven distinct encoding and decoding tasks through a discrete diffusion paradigm (Lu et al., 28 May 2026). Those tasks are I→B, T→B, I+T→B, B→I, B→T, B→I+T, and BQA. The paper frames prior work as dominated by specialized, single-task models, especially in BCI and neural decoding, and positions discrete diffusion as a way to avoid the artificial causal structure imposed by autoregressive models while supporting multi-modality and joint generation.

A central component is the Brain Tokenizer, a VQ-VAE-style module trained on fMRI from NSD. It uses a codebook with $K=128$ entries and latent dimension $D=16$, and each fMRI sample is represented by 64 brain codes (Lu et al., 28 May 2026). The tokenizer is not purely reconstructive: it also imposes three alignment objectives that tie brain tokens to CLIP-H semantics. Coarse-grained alignment pulls a global brain feature toward paired image and text features via InfoNCE and MSE distillation; fine-grained alignment masks 30% of CLIP text tokens and predicts them from brain tokens via cross-attention; perceptual alignment uses a frozen fMRI predictor to constrain reconstructions in CLIP feature space. The total tokenizer loss combines VQ, coarse, fine, and perceptual terms with weights $\lambda=0.5$, $\lambda_1=0.08$, $\lambda_2=0.02$, and $\beta=0.8$ (Lu et al., 28 May 2026).

The unified backbone is a tri-modal MM-DiT that extends Muddit with a symmetric brain branch. Image tokens come from a pre-trained VQ-VAE, text tokens from CLIP tokenization and embeddings, and brain tokens from the Brain Tokenizer. Each modality has dedicated input/output layers and Q/K/V projections, while Single-DiT blocks are shared. Unused modalities are isolated by attention masks, and joint targets such as B→I+T use synchronous masking to expose cross-target synergy (Lu et al., 28 May 2026).

The discrete diffusion formulation uses an absorbing-state continuous-time Markov chain toward a modality-specific [MASK] token:

$$q(x_t \mid \mathbf{x}_0) = \mathrm{Cat}\big(x_t \mid \alpha_t \mathbf{x}_0 + (1-\alpha_t)\mathbf{m}\big),$$

with $\alpha_t = \cos\!\big(\tfrac{\pi}{2} t\big)$ and a truncated arccos timestep density (Lu et al., 28 May 2026). Training minimizes a unified continuous-time negative ELBO over target modalities, and inference starts from fully masked targets and iteratively resamples masked positions until $t \to 0$.

Training proceeds progressively. Stage 1.1 freezes the pre-trained Muddit backbone and trains new brain modules on I+T→B and B→I+T. Stage 1.2 jointly trains six single-/bi-modal objectives with ratio 1:2:2:1:2:2. Stage 2 applies DoRA with $r=8$ and $D=16$0, introduces BQA instruction tuning, and uses a 1:1:2 ratio for I+T→B, B→I+T, and BQA (Lu et al., 28 May 2026). The resulting unified model has about 442M trainable parameters, uses A100 GPUs, and is evaluated on NSD subjects 1, 2, 5, and 7.

Quantitatively, the paper reports strong multi-task synergy. For image decoding, B→I+T attains SSIM .341, AlexNet(5) 84.9%, CLIP 79.8%, EffNet-B .824, and SwAV .537, with best or second-best performance across most metrics (Lu et al., 28 May 2026). For text decoding and reasoning, BQA reaches BLEU2 15.83, METEOR 50.13, ROUGE 52.91, CIDEr 223.98, SPICE 43.28, CLIP-S 70.65, and RefCLIP 76.72. Encoding tasks also improve when image and text are jointly conditioned, outperforming BraVL and MoPoE on voxel-level and semantic-level measures. The paper further uses the framework as a scientific testbed, reporting replication of category-selective regions such as EBA, OFA, FFA, PPA, and OPA (Lu et al., 28 May 2026).

3. MindOmni as a reasoning-aware multimodal generator

The 2025 MindOmni model addresses a different problem: how to unify multimodal understanding and image generation while making reasoning explicit and trainable (Xiao et al., 19 May 2025). Its backbone couples Qwen2.5-VL with OmniGen, a decoder-only diffusion transformer, using a connector composed of two standard LLM decoder layers. The text head produces autoregressive tokens and Chain-of-Thought outputs under a tagged format: > … and <answer> … </answer>.

The training strategy has three stages. Stage 1 pretrains the connector so that the diffusion decoder can consume VLM representations using a rectified flow objective and KL distillation. Stage 2 performs supervised fine-tuning with CoT instruction data, using coarse descriptions as answer content and fine-grained descriptions as reasoning content. Stage 3 introduces Reasoning Generation Policy Optimization, or RGPO, which extends GRPO to multimodal feedback and stabilizes both textual and visual behavior through dual KL regularizers (Xiao et al., 19 May 2025).

RGPO samples groups of outputs for each request, where each rollout contains text reasoning $D=16$1 and an image $D=16$2. Rewards combine a binary format reward for correct <think>/<answer> structure and a CLIP-based consistency reward between the generated image and a ground-truth prompt. Advantages are group-normalized as

$D=16$3

and the clipped surrogate objective adds separate KL penalties for text and image rollouts (Xiao et al., 19 May 2025). The default hyperparameters reported are group number $D=16$4, $D=16$5, and $D=16$6.

On understanding benchmarks, MindOmni reports MMMU 50.8, MMBench 83.2, and RealWorldQA 68.8 (Xiao et al., 19 May 2025). On GenEval it reports Single Obj 0.99, Two Obj 0.97, Counting 0.77, Colors 0.89, Position 0.59, Color Attr 0.64, and Overall 0.81; on DPG-Bench it reports Global 89.7, Relation 88.7, and Overall 83.0. On the reasoning-aware WISE benchmark it reaches overall 0.60, outperforming MetaQuery-XL at 0.55 overall and FLUX at 0.50 overall. Ablations show that Stage 1 alone yields 0.73 on GenEval and 0.42 on WISE, Stage 1 plus Stage 2 yields 0.81 and 0.54, Stage 1 plus Stage 3 without Stage 2 yields 0.72 and 0.49, and all stages yield 0.81 and 0.60 (Xiao et al., 19 May 2025).

The paper’s main claim is therefore not merely multimodal generation, but reasoning generation that directly conditions synthesis. A plausible implication is that its principal novelty lies in treating reasoning as an internal control signal for the generator rather than as an external prompt template. The authors nonetheless note that CoT remains plain text, that longer thinking outputs do not necessarily improve results, and that broader content-safety guardrails are not detailed (Xiao et al., 19 May 2025).

4. Adjacent systems often confused with Mind-Omni

The strongest explicit disambiguation comes from oMind/OMIND. That work targets mental-health LLMs, introduces oMind-SFT at about 164k instances, and adds the oMind-Chat benchmark with 961 conversations, but it also states that “Mind-Omni” does not appear in the paper and that such a label likely reflects a misreading of oMind (Racha et al., 26 Mar 2026). Its technical center is knowledge-grounded finetuning from UMLS triplets and DSM-5/ICD-11 book chunks, together with DPO preference tuning and rubric-based multi-turn dialogue evaluation.

MiniMind-O is another distinct system. It is an open 0.1B-scale omni model that accepts text, speech, and image inputs and returns both text and streaming speech (Gong, 5 May 2026). Its architecture splits a full MiniMind backbone as the Thinker from an independent four-layer Talker, uses frozen SenseVoice-Small and SigLIP2 encoders, and adopts a nine-stream representation with one text stream and eight audio-code streams. The paper emphasizes three scale-critical design choices: middle-layer semantic bridging, a released multimodal sequence format, and a parameter-efficient eight-codebook interface. This is relevant to “Mind-Omni” only by family resemblance, not by identity.

BrainOmni is also separate. It is a brain foundation model for unified EEG and MEG signals, based on a BrainTokenizer and a physics-aware Sensor Encoder that encode sensor position, orientation, and type (Xiao et al., 18 May 2025). It pretrains on 1,997 hours of EEG and 656 hours of MEG, and reports gains from joint EMEG pretraining across both modalities. Despite the naming proximity, its domain is electrophysiology rather than fMRI-based brain–vision–language generation.

A still more distant usage appears in the omni-directional social robot literature. "Would You Mind Me if I Pass by You? Socially-Appropriate Behaviour for an Omni-based Social Robot in Narrow Environment" studies step–slide–rotate passing behavior in 95 cm aisles, with statistically significant effects of rotation on warmth and likeability (Senft et al., 2022). Here “omni” refers to an omni-directional base, not a unified multimodal model.

5. Shared design patterns across the “Mind-Omni” family

Across these otherwise different systems, several recurrent design motifs appear. One is token standardization. Mind-Omni for fMRI converts continuous neural data into 64 discrete brain tokens aligned to CLIP semantics (Lu et al., 28 May 2026). BrainOmni likewise quantizes spatiotemporal EEG and MEG into discrete representations via BrainTokenizer (Xiao et al., 18 May 2025). MiniMind-O standardizes speech generation around eight-layer Mimi codec streams and explicit placeholder positions (Gong, 5 May 2026). MindOmni for reasoning-aware generation instead preserves continuous latent diffusion dynamics, but it still relies on a strict tagged textual interface and connector-mediated hidden-state alignment (Xiao et al., 19 May 2025).

A second recurring motif is staged optimization. Mind-Omni uses progressive multi-task training followed by DoRA-based BQA instruction tuning (Lu et al., 28 May 2026). MindOmni uses connector pretraining, CoT SFT, and RGPO (Xiao et al., 19 May 2025). MiniMind-O sequences full-model and projector-only passes across T2A, A2A, and I2T (Gong, 5 May 2026). BrainOmni separates tokenizer training from masked-token pretraining (Xiao et al., 18 May 2025). This suggests that cross-modal unification is being treated less as a single monolithic objective than as a curriculum over representation alignment, modality translation, and task-specific adaptation.

A third motif is explicit concern with alignment quality. In Mind-Omni, alignment is semantic and perceptual, tying brain codes to CLIP image and text spaces (Lu et al., 28 May 2026). In MindOmni, alignment is reinforced through multimodal rewards and dual KL constraints (Xiao et al., 19 May 2025). In oMind, alignment is medical and factual, achieved through structured knowledge retrieval, LLM pruning, and NLI contradiction filtering (Racha et al., 26 Mar 2026). The shared pattern is not identical methodology, but the insistence that multimodal unification requires explicit mechanisms for controlling semantic drift.

6. Limitations, ethics, and research significance

The major titled Mind-Omni systems both present ambitious unification claims, but both also delimit their applicability. The brain–vision–language Mind-Omni notes data scarcity and acquisition cost for fMRI, domain shift across subjects despite MNI registration, lossy tokenization, compute demands, and unresolved interpretability of token semantics (Lu et al., 28 May 2026). It also treats mind-reading risk cautiously, arguing that practical constraints remain strong because high-resolution fMRI is non-portable and requires cooperative, focused subjects.

MindOmni for reasoning-aware generation identifies a different set of limitations: plain-text CoT, the fact that increased completion length does not guarantee better outcomes, and reliance on CLIP rewards that may bias toward superficial alignment (Xiao et al., 19 May 2025). The paper states that broader ethical guardrails such as content-safety filters and bias auditing are not detailed.

The broader significance of the name cluster lies in what it reveals about current multimodal research. In one branch, “Mind-Omni” denotes unification across brain activity, images, and language via discrete tokens and diffusion. In another, it denotes the integration of reasoning trajectories into generation. In adjacent branches, it points toward speech-native omni models, electrophysiological foundation models, or knowledge-grounded clinical dialogue systems. This suggests that the term has evolved into a shorthand for systems that attempt modality-complete modeling under a shared computational interface, even when the modalities, objectives, and safety constraints differ substantially (Lu et al., 28 May 2026, Xiao et al., 19 May 2025, Gong, 5 May 2026, Xiao et al., 18 May 2025, Racha et al., 26 Mar 2026).