---
title: 'OmniX: Unified Multimodal & 3D Scene Synthesis'
url: https://www.emergentmind.com/topics/omnix
type: topic
---

# OmniX: Unified Multimodal & 3D Scene Synthesis

OmniX encompasses two distinct but influential frameworks in current research: (1) a unified, open-weight omnimodal Transformer for any-to-any generative and analytical tasks across text, vision, and audio (HyperCLOVA X 8B Omni, commonly termed “OmniX”); and (2) a panoramic vision system for producing graphics-ready 3D environments using adaptive 2D generative diffusion priors, also entitled "OmniX." Both approaches advance modality unification, albeit with different technical foci and deployment scenarios [2601.01792] [2510.26800].

## 1. Unified Omnimodal Transformer: HyperCLOVA X 8B Omni

### 1.1 Model Configuration

The HyperCLOVA X 8B Omni architecture is a single 8-billion-parameter, 36-layer, decoder-only Transformer using rotary position embeddings, GELU activations, and standard decoder blocks. It jointly processes interleaved sequences of:

- **Text tokens** (≈512K vocabulary, subword piece encoding)
- **Image tokens** (TA-Tok, 27×27 semantic grid, ≈2K codebook)
- **Audio tokens** (FSQ, 6,561-way codebook at 25 Hz)
- **Continuous vision embeddings** (ViT Qwen2.5-VL adapter, 4,096-dim)
- **Continuous audio embeddings** (Whisper-large-v3 adapter, 4,096-dim after compression)

All modalities are unified in a 4,096-dim space, supporting up to 32K sequence positions.

### 1.2 Modality Pathways

- **Vision**: Image inputs are tokenized via a frozen, SigLIP2-based TA-Tok; continuous patch embeddings are projected from a ViT adapter. Decoder uses a 2B-param diffusion model with MMDiT blocks to synthesize pixels from predicted tokens.
- **Audio**: Discrete audio tokens are produced by FSQ-quantized SenseVoice-Large ASR outputs. Continuous embeddings use Whisper-large-v3 features, further compressed to 1 Hz by a linear layer (MambaMia) and mapped to the shared space.
- **Unification**: At each sequence position, the Transformer sums a discrete (codebook) embedding $e_\text{token}$ and/or a continuous embedding $e_\text{cont}$:
  
  $$
  e_\text{input} = e_\text{token} + e_\text{cont}
  $$

  If a modality is absent at a position, the corresponding embedding is zero.

- **Next-token interface**: All generation is autoregressive via a shared softmax-over-vocabulary, treating vision and audio discrete tokens as vocabulary extensions.

## 2. Training Regime and Loss Functions

### 2.1 Core Objectives

- **Next-token prediction** uses a standard causal cross-entropy over the interleaved multimodal sequence:
  
  $$
  \mathcal{L}_{\text{NTP}} = -\sum_{t=1}^T \log p_\theta(x_t \mid x_{<t})
  $$

- **Auxiliary Losses**:
  - Multi-token pretraining for text: auxiliary head (weighted α=0.2).
  - Vision token alignment (Stage 1): training new image token embeddings.
  - Continuous encoder alignment: L2 loss between vision adapter outputs and TA-Tok reconstructions:
  
    $$
    \mathcal{L}_{\text{align}} = \| E_{\text{vision}}(I)W_{\text{adapter}} - E_{\text{tokenizer}}(I) \|_2^2
    $$

  - Vision loss masking for initial multimodal pretraining (0.5× weight initially, then full).

## 3. Implementation: Curriculum and Parameter Composition

### 3.1 Parameterization

| Component          | Parameters (approx.) |
|--------------------|---------------------|
| Transformer Core   | 8B                  |
| Vision Decoder     | 2B                  |
| Audio Decoder      | 200M                |
| Vision Adapter     | 40M                 |
| Audio Adapter      | 30M                 |
| **Total**          | **~10.3B**          |

### 3.2 Training Schedule

- **Pre-training** spans ≈3.0T tokens over several stages:
  - Text-only: context growth (4K→32K), multi-token prediction.
  - Discrete multimodal token expansion: 302B tokens, Image:Audio=3:1.
  - Full multimodal: 2.3T tokens, Text:Image:Audio=2:6.5:1.5.
  - Long-context adaptation: 20B tokens (32K context).
- **Continuous encoder integration and alignment** on frozen backbone.
- **Supervised fine-tuning** (≈100B tokens), progressing from foundational alignment and multimodal instruction specialization to long-context and multistep reasoning with explicit reasoning blocks.

## 4. Evaluation and Empirical Benchmarks

Comprehensive evaluation is carried out against Qwen2.5-Omni-7B, Emu3-8B, Janus-Pro-7B, X-Omni-7B, Step-Audio2-Mini-8B, Qwen2-Audio-7B, Audio-Flamingo3-7B, and GPT-4V (video domains). Highlights:

| Modality/Task               | Metric           | OmniX      | Noted Comparator(s) |
|-----------------------------|------------------|------------|---------------------|
| Text-to-Text (Ko-E/En)      | KMMLU-Pro, MMLU  | 64.9/75.7  | 31.1/71.6           |
| Vision-to-Text (Ko/En)      | KoNET/SEED-IMG   | 33.0/80.3  | 14.7/77.0           |
| Speech-to-Text (WER)        | LibriSpeech-c    | 1.93%      | 4.13%               |
| Audio-to-Text (SPIDEr)      | Clotho-v1        | 0.259      | 0.051               |
| Speech-to-Speech (ASR-BLEU) | En→Ko            | 24.70      | 0.00                |
| Text-to-Speech (MOS)        | En/Ko            | 3.94/4.22  | 3.96/3.40           |
| Video Understanding         | Video-MME/TV     | 58.2/69.7  | 59.9/50.0           |

A plausible implication is that OmniX, at 8B scale, achieves parity or superiority versus comparably sized or even larger specialized models across most benchmarks, especially in Korean-centric and multimodal tasks.

## 5. Applications and Limitations

### 5.1 Use Cases

- Any-to-any cross-modal generation, e.g.,
  - Text→Image (incl. cross-lingual)
  - Image editing (style, object, background)
  - Speech→Speech translation (direct En→Ko audio)
  - Audio-Visual Q&A, spoken answers
  - Audio captioning and multimodal dialogue

### 5.2 Constraints and Known Challenges

- Loss of fine details due to semantic tokenization; diffusion decoder may recover textures but blurs thin or small features (e.g., fonts).
- Non-square image resizing (to 384×384) introduces minor distortions.
- English-centric audio tasks are outperformed by larger models (e.g., Audio-Flamingo3).
- Scaling up backbone, improving continuous-only modeling, and joint contrastive training are suggested avenues for further enhancement.

## 6. OmniX for Panoramic 3D Scene Generation

### 6.1 Architectural Principles

The separate OmniX framework for 3D scene synthesis utilizes a 2D flow-matching diffusion Transformer backbone (e.g., FLUX.1-dev), extended for panoramic, graphics-ready output via lightweight, task-specific LoRA adapters.

- Each token’s velocity is predicted by $v_t = f_\theta(z_t, y, t)$; for multiple modalities/conditions:
  
  $$
  \hat{z}_1 = z_0 + \int_0^1 f_\theta(z_t,\,c^0, c^1, ..., y, t) dt
  $$

- **Adapters**: Three adapter variants (shared-branch, shared-adapter, separate-adapter) have been tested, with separate-adapter (one LoRA per input, all feeding the frozen backbone) performing best for panoramic vision tasks.

### 6.2 Panoramic Task Modes

- **Panorama Generation & Completion**: Masked region infill, optionally with text prompts.
- **Perception**: Intrinsic map prediction (e.g., depth, normal, albedo, roughness, metallic) from RGB panoramas.
- **Guided Perception**: Joint input of RGB and masked modalities.

- **3D Reconstruction Pipeline**: Predict dense distance map $D(u, v)$, backproject to 3D mesh, assign material and texture maps per modality.

## 7. Panoramic Dataset Construction and Evaluation

### 7.1 PanoX Dataset

- 60,000 panoramas from eight Unreal Engine 5 scenes (five indoor, three outdoor; two reserved for out-of-domain testing).
- Modalities per panorama: RGB, Euclidean depth, world normals, albedo, roughness, metallic.
- Perfect G-buffer channel alignment, 512×1024 equirectangular panorama resolution; random occlusions for completion.

### 7.2 Performance Summary

| Task          | Metric (PanoX-OutDomain)    | Value   | Previous SOTA   |
|---------------|----------------------------|---------|-----------------|
| Albedo        | PSNR/LPIPS                  | 17.76/0.344 | ~11/~0.56      |
| Roughness     | PSNR/LPIPS                  | 16.21/0.398 | ~11/~0.56      |
| Metallic      | PSNR/LPIPS                  | 18.87/0.254 | ~11/~0.56      |
| Depth         | AbsRel, δ<1.25, MAE, RMSE   | 0.158, 0.787, 1.68, 6.83 |  |
| Normal        | Mean/Median/<5°/<30°        | 27.14°, 14.88°, 15.5%, 66.3% |  |

Qualitatively, OmniX yields photorealistic 360° renderings with correct shadows/specular effects and interactive 3D frame rates. It enables physically based relighting and dynamic simulations in mesh-converted scenes.

## 8. Broader Applications and Unresolved Challenges

- **Applications**: Virtual reality, 360° video backdrops, rapid game/film scene prototyping, robotics/embodied AI training, instant architectural walkthroughs.
- **Limitations**: Slow diffusion backbone sampling, minor errors in fine geometry, poorer metallic generalization due to imbalanced data, and incomplete capture of physically based BRDF effects by generic image priors.
- **Prospective Developments**: Fast distillation (e.g., DDIM, progressive strategies), augmentation with real-world PBR captured data, differentiable rendering-based joint fine-tuning, and extension to dynamic, time-variant scenes.

## References

- HyperCLOVA X 8B Omni (omnimodal assistant): [2601.01792]
- Panoramic 3D scene synthesis: [2510.26800]

Source: https://www.emergentmind.com/topics/omnix