OmniX: Unified Multimodal & 3D Scene Synthesis

• OmniX is a dual-framework system that combines an 8B omnimodal Transformer for text, vision, and audio with a panoramic 3D scene synthesis pipeline.
• It leverages unified embeddings and autoregressive next-token prediction to enable any-to-any generative and analytical tasks across multiple modalities.
• The framework’s advanced training regime and diffusion-based 3D synthesis yield state-of-the-art performance in virtual reality, game prototyping, and multimodal interaction.

OmniX encompasses two distinct but influential frameworks in current research: (1) a unified, open-weight omnimodal Transformer for any-to-any generative and analytical tasks across text, vision, and audio (HyperCLOVA X 8B Omni, commonly termed “OmniX”); and (2) a panoramic vision system for producing graphics-ready 3D environments using adaptive 2D generative diffusion priors, also entitled "OmniX." Both approaches advance modality unification, albeit with different technical foci and deployment scenarios (Team, 5 Jan 2026, Huang et al., 30 Oct 2025).

1. Unified Omnimodal Transformer: HyperCLOVA X 8B Omni

1.1 Model Configuration

The HyperCLOVA X 8B Omni architecture is a single 8-billion-parameter, 36-layer, decoder-only Transformer using rotary position embeddings, GELU activations, and standard decoder blocks. It jointly processes interleaved sequences of:

• Text tokens (≈512K vocabulary, subword piece encoding)
• Image tokens (TA-Tok, 27×27 semantic grid, ≈2K codebook)
• Audio tokens (FSQ, 6,561-way codebook at 25 Hz)
• Continuous vision embeddings (ViT Qwen2.5-VL adapter, 4,096-dim)
• Continuous audio embeddings (Whisper-large-v3 adapter, 4,096-dim after compression)

All modalities are unified in a 4,096-dim space, supporting up to 32K sequence positions.

1.2 Modality Pathways

• Vision: Image inputs are tokenized via a frozen, SigLIP2-based TA-Tok; continuous patch embeddings are projected from a ViT adapter. Decoder uses a 2B-param diffusion model with MMDiT blocks to synthesize pixels from predicted tokens.
• Audio: Discrete audio tokens are produced by FSQ-quantized SenseVoice-Large ASR outputs. Continuous embeddings use Whisper-large-v3 features, further compressed to 1 Hz by a linear layer (MambaMia) and mapped to the shared space.
• Unification: At each sequence position, the Transformer sums a discrete (codebook) embedding $e_\text{token}$ and/or a continuous embedding $e_\text{cont}$:

$e_\text{input} = e_\text{token} + e_\text{cont}$

If a modality is absent at a position, the corresponding embedding is zero.

• Next-token interface: All generation is autoregressive via a shared softmax-over-vocabulary, treating vision and audio discrete tokens as vocabulary extensions.

2. Training Regime and Loss Functions

2.1 Core Objectives

• Next-token prediction uses a standard causal cross-entropy over the interleaved multimodal sequence:

$$\mathcal{L}_{\text{NTP}} = -\sum_{t=1}^T \log p_\theta(x_t \mid x_{<t})$$

• Auxiliary Losses:
  • Multi-token pretraining for text: auxiliary head (weighted α=0.2).
  • Vision token alignment (Stage 1): training new image token embeddings.
  • Continuous encoder alignment: L2 loss between vision adapter outputs and TA-Tok reconstructions:

  $$\mathcal{L}_{\text{align}} = \| E_{\text{vision}}(I)W_{\text{adapter}} - E_{\text{tokenizer}}(I) \|_2^2$$ - Vision loss masking for initial multimodal pretraining (0.5× weight initially, then full).

3. Implementation: Curriculum and Parameter Composition

3.1 Parameterization

Component	Parameters (approx.)
Transformer Core	8B
Vision Decoder	2B
Audio Decoder	200M
Vision Adapter	40M
Audio Adapter	30M
Total	~10.3B

3.2 Training Schedule

• Pre-training spans ≈3.0T tokens over several stages:

  • Text-only: context growth (4K→32K), multi-token prediction.
  • Discrete multimodal token expansion: 302B tokens, Image:Audio=3:1.
  • Full multimodal: 2.3T tokens, Text:Image:Audio=2:6.5:1.5.
  • Long-context adaptation: 20B tokens (32K context).
• Continuous encoder integration and alignment on frozen backbone.
• Supervised fine-tuning (≈100B tokens), progressing from foundational alignment and multimodal instruction specialization to long-context and multistep reasoning with explicit reasoning blocks.

4. Evaluation and Empirical Benchmarks

Comprehensive evaluation is carried out against Qwen2.5-Omni-7B, Emu3-8B, Janus-Pro-7B, X-Omni-7B, Step-Audio2-Mini-8B, Qwen2-Audio-7B, Audio-Flamingo3-7B, and GPT-4V (video domains). Highlights:

Modality/Task	Metric	OmniX	Noted Comparator(s)
Text-to-Text (Ko-E/En)	KMMLU-Pro, MMLU	64.9/75.7	31.1/71.6
Vision-to-Text (Ko/En)	KoNET/SEED-IMG	33.0/80.3	14.7/77.0
Speech-to-Text (WER)	LibriSpeech-c	1.93%	4.13%
Audio-to-Text (SPIDEr)	Clotho-v1	0.259	0.051
Speech-to-Speech (ASR-BLEU)	En→Ko	24.70	0.00
Text-to-Speech (MOS)	En/Ko	3.94/4.22	3.96/3.40
Video Understanding	Video-MME/TV	58.2/69.7	59.9/50.0

A plausible implication is that OmniX, at 8B scale, achieves parity or superiority versus comparably sized or even larger specialized models across most benchmarks, especially in Korean-centric and multimodal tasks.

5. Applications and Limitations

5.1 Use Cases

• Any-to-any cross-modal generation, e.g.,
  • Text→Image (incl. cross-lingual)
  • Image editing (style, object, background)
  • Speech→Speech translation (direct En→Ko audio)
  • Audio-Visual Q&A, spoken answers
  • Audio captioning and multimodal dialogue

5.2 Constraints and Known Challenges

• Loss of fine details due to semantic tokenization; diffusion decoder may recover textures but blurs thin or small features (e.g., fonts).
• Non-square image resizing (to 384×384) introduces minor distortions.
• English-centric audio tasks are outperformed by larger models (e.g., Audio-Flamingo3).
• Scaling up backbone, improving continuous-only modeling, and joint contrastive training are suggested avenues for further enhancement.

6. OmniX for Panoramic 3D Scene Generation

6.1 Architectural Principles

The separate OmniX framework for 3D scene synthesis utilizes a 2D flow-matching diffusion Transformer backbone (e.g., FLUX.1-dev), extended for panoramic, graphics-ready output via lightweight, task-specific LoRA adapters.

• Each token’s velocity is predicted by $v_t = f_\theta(z_t, y, t)$; for multiple modalities/conditions:

$$\hat{z}_1 = z_0 + \int_0^1 f_\theta(z_t,\,c^0, c^1, ..., y, t) dt$$

• Adapters: Three adapter variants (shared-branch, shared-adapter, separate-adapter) have been tested, with separate-adapter (one LoRA per input, all feeding the frozen backbone) performing best for panoramic vision tasks.

6.2 Panoramic Task Modes

• Panorama Generation & Completion: Masked region infill, optionally with text prompts.
• Perception: Intrinsic map prediction (e.g., depth, normal, albedo, roughness, metallic) from RGB panoramas.
• Guided Perception: Joint input of RGB and masked modalities.
• 3D Reconstruction Pipeline: Predict dense distance map $D(u, v)$, backproject to 3D mesh, assign material and texture maps per modality.

7. Panoramic Dataset Construction and Evaluation

7.1 PanoX Dataset

• 60,000 panoramas from eight Unreal Engine 5 scenes (five indoor, three outdoor; two reserved for out-of-domain testing).
• Modalities per panorama: RGB, Euclidean depth, world normals, albedo, roughness, metallic.
• Perfect G-buffer channel alignment, 512×1024 equirectangular panorama resolution; random occlusions for completion.

7.2 Performance Summary

Task	Metric (PanoX-OutDomain)	Value	Previous SOTA
Albedo	PSNR/LPIPS	17.76/0.344	~11/~0.56
Roughness	PSNR/LPIPS	16.21/0.398	~11/~0.56
Metallic	PSNR/LPIPS	18.87/0.254	~11/~0.56
Depth	AbsRel, δ<1.25, MAE, RMSE	0.158, 0.787, 1.68, 6.83
Normal	Mean/Median/<5°/<30°	27.14°, 14.88°, 15.5%, 66.3%

Qualitatively, OmniX yields photorealistic 360° renderings with correct shadows/specular effects and interactive 3D frame rates. It enables physically based relighting and dynamic simulations in mesh-converted scenes.

8. Broader Applications and Unresolved Challenges

• Applications: Virtual reality, 360° video backdrops, rapid game/film scene prototyping, robotics/embodied AI training, instant architectural walkthroughs.
• Limitations: Slow diffusion backbone sampling, minor errors in fine geometry, poorer metallic generalization due to imbalanced data, and incomplete capture of physically based BRDF effects by generic image priors.
• Prospective Developments: Fast distillation (e.g., DDIM, progressive strategies), augmentation with real-world PBR captured data, differentiable rendering-based joint fine-tuning, and extension to dynamic, time-variant scenes.

• HyperCLOVA X 8B Omni (omnimodal assistant): (Team, 5 Jan 2026)
• Panoramic 3D scene synthesis: (Huang et al., 30 Oct 2025)