HyperCLOVA X 8B Omni: Omnimodal Transformer

• HyperCLOVA X 8B Omni is a compact, any-to-any multimodal transformer that unifies text, vision, and audio through a shared autoregressive framework supporting both Korean and English.
• It leverages a 36-layer decoder-only transformer with modality-specific encoders and tokenizers, using techniques like pre-normalization and grouped-query attention for efficient training.
• Empirical evaluations across more than 12 benchmarks demonstrate its competitive performance and robust cross-lingual generalization for multimodal tasks.

HyperCLOVA X 8B Omni is a multimodal foundation model within the HyperCLOVA X family, integrating text, vision, and audio modalities for both understanding and generation in a shared autoregressive framework. The model is the first publicly released, compact “any-to-any” omnimodal assistant, unifying input and output paths for text, speech, and images within a single 8-billion-parameter checkpoint, and supports Korean and English natively. Unlike prior models with independent modality-specific pipelines, HyperCLOVA X 8B Omni (“OMNI”) operationalizes a single next-token interface over interleaved multimodal sequences. This permits flexible, high-fidelity multimodal interaction and positions the model as a pathfinder for practical multimodal assistants (Team, 5 Jan 2026, Yoo et al., 2024).

1. Architectural Overview

OMNI adopts a decoder-only transformer backbone with 36 layers, a hidden size of $D = 4096$, and an extended rotary-embedding context window of 32,768 tokens. Modality unification is achieved by interleaving tokens representing text ($V_\text{text}$), vision ($V_\text{vision}$, $\sim$10K entries), audio ($V_\text{audio}$, 6,561 entries), and continuous vision or audio embeddings directly into the model’s input sequence.

At each timestep $t$, the model computes a hidden state $h_t \in \mathbb{R}^D$ from the sequence $X_{1:t}$, mapping discrete tokens to embeddings or inserting continuous vectors. The next-token probability is produced by a single softmax over the joint vocabulary $$V_\text{text} \cup V_\text{vision} \cup V_\text{audio}$$. Continuous embeddings serve as context and fine-grained grounding signals but are not softmax targets.

The relevant computation is:

$h_t = \mathrm{Transformer}(z_1, z_2, ..., z_t)$

where

$V_\text{text}$0

with softmax prediction over all discrete vocabularies.

Pre-normalization, grouped-query attention, and bf16 precision are used for training efficiency and stability. Tokenization is morphology-preserving and supports both Korean and English.

2. Modality-Specific Encoders and Tokenization

Text

Text is tokenized using a morphology-preserving pretokenizer followed by an English-centric subword tokenizer extended for Korean. Subword indices are mapped via a learned embedding matrix $V_\text{text}$1 (Yoo et al., 2024).

Vision

Vision modality is handled through dual pipelines:

• A continuous vision encoder ($V_\text{text}$2) based on a ViT (“Qwen2.5-VL” configuration) and a linear adapter, providing $V_\text{text}$3, where $V_\text{text}$4 for a $V_\text{text}$5 image and up to $V_\text{text}$6 for 120-frame video.
• A discrete vision tokenizer (TA-Tok, SigLIP 2) quantizes features into $V_\text{text}$7 semantic tokens per image, indexed in $V_\text{text}$8.

Audio

Audio input is processed by:

• A continuous encoder ($V_\text{text}$9), mapping log-mel spectrograms to $V_\text{vision}$0 via a 2-layer MLP adapter. Stride and downsampling yield $V_\text{vision}$1 matching the downsample rate; interleaved audio in video uses a MambaMia compression layer to reduce $V_\text{vision}$2.
• A discrete tokenizer using an FSQ-based, 6,561-ary codebook, generating roughly 25 tokens/sec, assigned to $V_\text{vision}$3.

3. Pretraining Strategies and Objectives

OMNI is pretrained end-to-end with a standard autoregressive negative log-likelihood objective over interleaved multimodal sequences:

$V_\text{vision}$4

where $V_\text{vision}$5 weights each modality. A multi-phase curriculum modulates modality mixing, loss masking, and curriculum schedules:

• Phase 0: Pure text pretraining (context 4k→8k→32k, multi-token prediction).
• Phase 1: Add discrete vision/audio tokens, freeze transformer body for text, train new embeddings for paired image/audio-text.
• Phase 2: Unfreeze all parameters; train joint text/image/audio with mixture ratio 2:6.5:1.5. Vision-token loss masking 50% initially, then removed.
• Phase 3: Long-context adaptation (32k context) with 20B tokens focused on long image–text–video.
• Phase 4: Integrate continuous encoders via frozen backbone in stage 1, then full fine-tuning (1.5T tokens) and audio-encoder bridging for ASR in stage 3.

Pretraining corpus covers balanced Korean, multilingual English-heavy text, and code (Yoo et al., 2024). Upsampling of knowledge-rich documents and efficient tokenization enable competitive cross-lingual generalization.

4. Instruction Tuning, Alignment, and Safety

Instruction tuning comprises supervised fine-tuning (SFT) on curated, annotated datasets and reinforcement learning from human feedback (RLHF). SFT employs prompts and response masking for loss computation. RLHF leverages a reward model using Bradley–Terry ranking loss, Proximal Policy Optimization (PPO) with KL penalty (0.04), and sequence-level unlikelihood loss for reducing repetition.

Safety and alignment are further reinforced via extensive red-team sessions across 14 risk categories (e.g., hate speech, violence, privacy) and an event-driven MLOps pipeline minimizing annotator overhead. The model is governed by ten safety-ban categories, including content, advice, and subjectivity bans.

Automated and human evaluation confirms low toxicity and bias: for example, in RealToxicPrompts (English), 8B Omni's mean toxicity is 0.0799 and toxic-count rate is 1.4%; for KOLD (Korean), toxicity is 0.163. Attack success rates on adversarial prompts are 5.93%, with human preference scores (4.21/7) on par with leading models.

5. Empirical Performance Evaluation

OMNI’s evaluation covers >12 benchmarks, measuring over seven input-output modalities and bilingual capabilities. Selected metrics for core tasks (best value per column in bold):

Task	Metric	OMNI 8B	Qwen2.5 7B	Emu 8B	Janus 7B
Text→Text (Kor.)	KMMLU-pro ↑	64.9	31.1	18.7	16.4
Text→Text (Eng.)	MMLU ↑	75.7	71.6	32.4	48.2
Vision→Text (Kor.)	KoNET ↑	33.0	14.7	0.6	0.3
Speech→Text	Libri-c WER ↓	1.93	4.13	–	–
Audio→Text (Clotho)	SPIDEr ↑	0.259	0.051	0.238	–
Text→Vision (Eng.)	GenEval ↑	0.64	–	0.39	0.78
Text→Speech (MOS)	KO: ↑	4.22	3.40	4.20	3.43

The model consistently leads or matches most best-in-class 7–8B models, while being the only system supporting all input-output modality combinations under a single checkpoint. Bilingual understanding is robust; on Korean benchmarks (KoBigBench, KMMLU, HAE-RAE), OMNI 8B reaches 61.7%, substantially outperforming LLaMA 2 13B (43.7%) (Yoo et al., 2024).

6. Applications, Limitations, and Future Directions

OMNI supports deployment as a any-to-any omnimodal AI assistant, enabling visually grounded chat, photo/video editing by language, automated subtitling and dubbing, multimodal retrieval and summarization, accessibility enhancements, and mixed-modality robotic control.

Limitations include:

• Fixed 384×384 vision tokenization, causing geometric distortion for some images.
• Loss of fine textures due to semantic quantization, necessitating heavier diffusion decoders for high-fidelity synthesis.
• Generative performance on large-scale image/video remains below state-of-the-art models (>30B).
• Text-to-speech quality is slightly inferior to best specialized vocoders on nuanced prosody.

Future extensions are outlined:

• Scaling parameter count beyond 30B is anticipated to close generation fidelity gaps.
• Integrating continuous-valued autoregressive decoders (e.g., diffusion likelihoods) can obviate discrete tokenizers.
• Allowing dynamic-resolution vision encoding to remove fixed-size constraints.
• Incorporating joint multimodal retrieval for factual grounding in text and audio.

7. Cross-Lingual Capabilities and Social Considerations

8B Omni’s Korean–English duality delivers strong transfer to out-of-distribution languages and tasks. On XNLI (5-shot) across six Asian languages, OMNI scores 57.6%, outperforming Falcon 40B (43.2%). For one-shot machine translation (FLORES+), English-to-Korean and Korean-to-English accuracy is 88.3 and 87.9, respectively, closely tracking GPT-4 (Yoo et al., 2024). The architecture's design and pretraining corpus selection enable robust multilinguality not limited to its primary training languages.

In alignment with its safety architecture, OMNI demonstrates low observed social and cultural bias as measured by BBQ (U.S.) and KoBBQ, and performs comparably to GPT-4 in human safety preference testing. Performance is further corroborated by low attack success rates on adversarial prompt sets.

---

HyperCLOVA X 8B Omni represents the state-of-the-art in compact, open-weight omnimodal transformer architectures, with unified multimodal sequence modeling and rigorous empirical validation across text, vision, and audio for both Korean and English domains (Team, 5 Jan 2026, Yoo et al., 2024).