Unified Encoder-Free Modality Overview

• Unified Encoder-Free Modality is a design approach that eliminates or repurposes modality-specific encoders by processing raw image, audio, and text tokens through a unified, often decoder-only, backbone.
• It integrates multiple modalities with minimal preprocessing, reducing computational overhead and enhancing cross-modal interactions in a single model.
• Recent implementations demonstrate competitive performance on vision-language, speech-language, and audio-visual benchmarks while addressing challenges like catastrophic forgetting and modality shortcuts.

“Unified Encoder-Free Modality” denotes an emerging design direction in multimodal modeling in which heterogeneous inputs are processed through a shared computational pathway while reducing, replacing, or reinterpreting the role of modality-specific encoders. In its strongest form, the model removes the dedicated vision or speech encoder entirely and lets a single decoder-only or dense shared backbone ingest image, audio, and text tokens end-to-end. The motivation stated across the literature is structural as well as statistical: pre-trained encoders impose strong inductive biases on resolution, aspect ratio, and semantic abstraction; separate towers and projectors add inference overhead; and fixed modality pipelines can restrict flexible deployment and cross-modal interaction (Diao et al., 2024, Diao et al., 10 Feb 2025).

1. Definition and scope

The term is not used in a single uniform sense. In encoder-free vision-language work such as EVE and EVEv2, it means that the model does not rely on any pre-trained visual backbone and does not discretize images through a learned tokenizer; instead, raw images are converted directly into continuous patch tokens and processed together with text inside one decoder-only transformer (Diao et al., 2024, Diao et al., 10 Feb 2025). In Mel-LLM, it means that lightly pre-processed log-Mel spectrograms are projected into the LLM embedding space through a single linear layer, so that the lower LLM layers assume the role traditionally played by a speech encoder (Fan et al., 8 Jun 2026). In OmniEncoder and Omni-C, the emphasis shifts from eliminating one specific backbone to using a single shared encoder for multiple modalities, with no separate expert towers, no MoE routing, and no modality-specific projectors inside the main backbone (Bai et al., 2 May 2026, Lau et al., 27 Feb 2026). In UP-Fusion, the same idea is expressed as a single, task-agnostic fusion model that shares encoder-decoder parameters across modalities and tasks while avoiding modality-specific encoder heads (Li et al., 16 Nov 2025).

A plausible taxonomy is that the literature contains at least three operational interpretations. The first is direct encoder elimination, where a model replaces the dedicated visual or speech encoder with lightweight patchification or linear projection. The second is single-backbone unification, where one dense encoder handles all modalities and modality-specific logic is restricted to lightweight front-ends or projection heads. The third is alignment without encoder training, where the heavy encoders remain but are frozen, and unification is achieved through a shared projection space. OneEncoder explicitly clarifies that it does not eliminate modality-specific encoders themselves; rather, it eliminates the need to train large modality-specific encoders for alignment, using frozen pretrained encoders plus a lightweight Universal Projection and per-modality Alignment Layer (Faye et al., 2024). MCA is adjacent rather than identical: it studies a unified encoder perspective for composed multimodal retrieval and analyzes failure modes that arise when a single encoder directly consumes compositions of modalities (Wu et al., 17 Oct 2025).

System family	Modalities	Operational meaning of “encoder-free”
EVE, EVEv2	Image, text	No pre-trained visual backbone or discrete visual tokenizer
Mel-LLM	Speech, text	No dedicated speech encoder; Mel patches projected directly into the LLM
OmniEncoder, Omni-C, UP-Fusion, MAELRE	Audio-video, image-audio-text, fusion tasks, long-range multimodal inputs	One shared encoder or encoder-decoder without separate expert towers or modality-specific encoders inside the backbone
OneEncoder	Image, text, audio, video	No training of large modality-specific encoders; frozen encoders remain
MCA	Composed text-image retrieval	Unified encoder analysis focused on shortcut mitigation

2. Architectural patterns

The central architectural move is to let a single model ingest modality tokens as one sequence. EVE frames this as “bridging vision-language representation inside one unified decoder” and “enhancing visual recognition capability via extra supervision,” with the stated goal of a pure decoder-only architecture across modalities (Diao et al., 2024). EVEv2 makes this explicit. It replaces external vision encoders and discrete visual tokenizers with minimalist visual layers trained from scratch and processes visual and textual inputs as one interleaved sequence under a causal autoregressive objective. Visual tokens are built by two small convolutions and a GELU, with

$$x_v = \mathrm{Conv2}(\mathrm{GELU}(\mathrm{Conv1}(I))),$$

where Conv1 has stride 16 and output channels 1024, Conv2 has stride 2 and output dimension 3584. A class token <CLS> is prepended, row separators <SPL> are inserted after each row of patch tokens, and the joint sequence is

$$S=[\langle CLS\rangle, v_1,\ldots,v_{N_v}, \langle SPL\rangle\ \text{row markers}, t_1,\ldots,t_{N_t}].$$

The distinctive feature is not only one decoder-only backbone, but also modality-specific parameters for attention, layer normalization, and feed-forward layers inside each block, so that visual tokens and text tokens travel through different parameter subsets while still participating in shared self-attention (Diao et al., 10 Feb 2025).

Mel-LLM applies the same principle to speech-language modeling. The input is an 80-bin log-Mel spectrogram with per-utterance mean-variance normalization. After optional Conv1D downsampling, each reduced time-step representation $e_\tau$ is projected into the LLM hidden space by

$$x_\tau = e_\tau W_{\mathrm{proj}} + b_{\mathrm{proj}},$$

with $d_{\mathrm{model}}=3072$ in a Phi-4-MM backbone. Audio tokens occupy earlier positions on the same 1D positional axis as text tokens; no additional 2D time-frequency encoding and no modality-specific type embeddings are required (Fan et al., 8 Jun 2026).

OmniEncoder generalizes the shared-backbone idea to continuous audio-visual perception. It uses a single 24-layer Transformer receiving a unified token stream composed, at each time step, of an audio token $a_t$, a Visual Continuous token $v_t^c$, and Visual Base patch tokens $v_{t,\cdot}^b$. Audio and VC tokens are kept dense at 25 fps, VB tokens are down-sampled to 2 fps by a Token Sparsifier, and all are processed by joint self-attention from layer 1 onward. Omni-RoPE assigns non-overlapping coordinates to audio, VC, and VB tokens in a shared $(t,h',w')$ lattice, so that modality disentanglement is achieved by coordinate partitioning while temporal alignment is maintained by the common 25 fps time index. Temporal Window Shifting then alternates window boundaries by $G/2=8$ over windows of $$S=[\langle CLS\rangle, v_1,\ldots,v_{N_v}, \langle SPL\rangle\ \text{row markers}, t_1,\ldots,t_{N_t}].$$0 frames to propagate context without global quadratic attention (Bai et al., 2 May 2026).

MAELRE pursues a different but related target: long-range modality-agnostic encoding with one transformer body across text, time series, audio, and vision. Its blocks combine MHSA, MLP, and a token reduction module. The number of tokens and dimensions evolve as

$$S=[\langle CLS\rangle, v_1,\ldots,v_{N_v}, \langle SPL\rangle\ \text{row markers}, t_1,\ldots,t_{N_t}].$$1

with approximate attention used when $$S=[\langle CLS\rangle, v_1,\ldots,v_{N_v}, \langle SPL\rangle\ \text{row markers}, t_1,\ldots,t_{N_t}].$$2 and exact dot-product attention used when $$S=[\langle CLS\rangle, v_1,\ldots,v_{N_v}, \langle SPL\rangle\ \text{row markers}, t_1,\ldots,t_{N_t}].$$3. This keeps the same encoder structure across modalities while avoiding modality-specific attention schemes inside the backbone (Parag et al., 25 Jul 2025).

3. Objectives and optimization regimes

A notable feature of unified encoder-free systems is that they often retain simple global objectives while moving complexity into input construction and training schedule. EVE reports that direct training without encoders results in slow convergence and large performance gaps, and therefore proposes a training recipe centered on representation bridging within one decoder and additional supervision for visual recognition (Diao et al., 2024). EVEv2 goes further by using autoregressive next-token prediction as the sole objective,

$$S=[\langle CLS\rangle, v_1,\ldots,v_{N_v}, \langle SPL\rangle\ \text{row markers}, t_1,\ldots,t_{N_t}].$$4

with a staged curriculum. Stage 1 freezes the LLM and trains only the patch embedding on 10M high-quality pairs from EVE-recap-10M. Stage 2.1 unfreezes the newly added visual layers and trains on 77M re-captioned images with resolution progressively increased from $$S=[\langle CLS\rangle, v_1,\ldots,v_{N_v}, \langle SPL\rangle\ \text{row markers}, t_1,\ldots,t_{N_t}].$$5 to $$S=[\langle CLS\rangle, v_1,\ldots,v_{N_v}, \langle SPL\rangle\ \text{row markers}, t_1,\ldots,t_{N_t}].$$6. Stage 2.2 trains the full architecture on 15M general QA/instruction data. Stage 3 performs supervised instruction tuning on 7.3M high-quality multi-source instruction data. Across these stages, the optimizer is AdamW with stage-specific peak learning rates $$S=[\langle CLS\rangle, v_1,\ldots,v_{N_v}, \langle SPL\rangle\ \text{row markers}, t_1,\ldots,t_{N_t}].$$7, $$S=[\langle CLS\rangle, v_1,\ldots,v_{N_v}, \langle SPL\rangle\ \text{row markers}, t_1,\ldots,t_{N_t}].$$8, $$S=[\langle CLS\rangle, v_1,\ldots,v_{N_v}, \langle SPL\rangle\ \text{row markers}, t_1,\ldots,t_{N_t}].$$9, and $e_\tau$0, warm-up ratio 0.03, cosine decay, and bfloat16 precision (Diao et al., 10 Feb 2025).

Mel-LLM also keeps the training criterion minimal on the ASR side. Its speech-to-text objective is standard next-token cross-entropy conditioned on the full sequence of audio tokens,

$e_\tau$1

with no CTC and no auxiliary encoder losses. For TTS, the same decoder backbone is paired with a next-token VAE Mel head and a loss that combines reconstruction, KL regularization, stop-token BCE, and a spectrogram flux term. Training uses AdamW, peak learning rate $e_\tau$2, linear warmup for 9k steps, gradient clip 1.0, effective batch size 512, DeepSpeed ZeRO-1 on 16× H100 GPUs, and LoRA on attention and MLP linear layers with rank 320 and $e_\tau$3 (Fan et al., 8 Jun 2026).

Omni-C adopts unimodal contrastive self-supervision rather than autoregression. A single ViT-B/32 backbone with 12 Transformer blocks, hidden size 768, and 12 attention heads is shared across image, audio spectrogram, and text inputs. Each modality uses its own two-layer MLP projection head, and training proceeds with modality-separated mini-batches and unimodal InfoNCE at temperature $e_\tau$4. ImageNet-1K, AudioSet, and Wikipedia are each balanced to approximately 1.28M examples. Cross-modal alignment is then learned post hoc with a CLIP-style symmetric InfoNCE objective on paired data, while the shared backbone remains frozen (Lau et al., 27 Feb 2026).

OneEncoder formalizes progressive alignment as a two-stage procedure. Step 1 trains only a lightweight Universal Projection module of about 4M parameters plus modality tokens on image-text pairs, with ViT-base and BERT-base frozen. Step 2 freezes the UP and previously aligned tokens, then trains only a 65,792-parameter Alignment Layer for each new modality, such as audio or video, against one already aligned modality, usually text. The same symmetric contrastive loss is reused so that alignment becomes transitive across all modalities already in the UP space (Faye et al., 2024).

4. Interference, shortcuts, and control of shared representations

The main technical difficulty in unified encoder-free modality is not only representation capacity but conflict management inside the shared model. EVEv2 characterizes one failure mode as catastrophic linguistic forgetting: when a single set of dense parameters must serve both modalities, visual learning from scratch can overwrite the LLM distribution. Its “Divide-and-Conquer” architecture therefore imposes modality-wise sparsity in attention, layer normalization, and feed-forward layers, so that the parameter space is separated even though the causal self-attention graph remains unified. Earlier variants in the same line—shared weights, FFN-only re-parameterization, and MoE FFN decoupling—either trained slowly, suffered linguistic forgetting, or improved vision less efficiently (Diao et al., 10 Feb 2025).

MCA identifies a complementary failure mode in unified encoders trained with conventional contrastive learning: modality shortcuts. Formally, the composed embedding can collapse toward a unimodal embedding,

$e_\tau$5

so that the model ignores the other modality. The paper defines a modality preference gap

$e_\tau$6

and a modality preference rate

$e_\tau$7

It then augments contrastive learning with Modality Composition Preference, which forces the multimodal query to score above any unimodal counterpart, and Modality Composition Regularization, which anchors the composed embedding to a prototype mixed from its unimodal parts. The total loss is

$e_\tau$8

with MCP and MCR applied only when either the query or the document is composed (Wu et al., 17 Oct 2025).

UP-Fusion studies the same issue in unified multi-modality image fusion, where gradient conflicts arise because large modality differences force one parameter set to serve heterogeneous statistics. Its response is channel-level rather than token-level. SCPM computes a fused channel importance

$e_\tau$9

retains the top 70% channels, and restores dimensionality through a $$x_\tau = e_\tau W_{\mathrm{proj}} + b_{\mathrm{proj}},$$0 convolution. GAM then injects modality-conditioned affine parameters

$$x_\tau = e_\tau W_{\mathrm{proj}} + b_{\mathrm{proj}},$$1

from global pooled original-modal features. TCPM further concatenates modal features, selects the top 50% channels, expands with $$x_\tau = e_\tau W_{\mathrm{proj}} + b_{\mathrm{proj}},$$2 convolution, rearranges channels according to CLIP text guidance, and mixes perturbed and original channels by self-attention. The architecture remains fully shared across modalities and tasks; discriminability is preserved through modulation and perturbation rather than separate encoders (Li et al., 16 Nov 2025).

Omni-C arrives at a similar conclusion by ablation. A shared projector causes text and audio embeddings to overlap strongly in t-SNE and degrades zero-shot and linear-probe performance, whereas separate projection heads maintain well-separated clusters on the hypersphere. Modality-separated mini-batches further reduce interference by ensuring the backbone sees one modality at a time during a step (Lau et al., 27 Feb 2026).

5. Reported empirical behavior

The empirical record in this literature is heterogeneous because tasks and evaluation protocols differ, but several recurring patterns are reported. In vision-language modeling, EVE states that, solely utilizing 35M publicly accessible data, it can rival encoder-based VLMs of similar capacities across multiple vision-language benchmarks and significantly outperforms Fuyu-8B (Diao et al., 2024). EVEv2 reports that a 7B Qwen2.5-based model trained with 92M pretraining samples and 7.3M SFT samples reaches MMMU 39.3, MMBench-EN 66.3, SEED-Img 71.4, MMVet 45.0, MME 1709, POPE 87.6, GQA 62.9, ScienceQA-Img 96.2, TextVQA 71.1, ChartQA 73.9, AI2D 74.8, RealWorldQA 62.4, and OCRBench 702. Against LLaVA-1.6 7B, the same report highlights MMVet 45.0 vs 43.9, TextVQA 71.1 vs 64.9, ChartQA 73.9 vs 54.8, and AI2D 74.8 vs 66.6 (Diao et al., 10 Feb 2025).

In speech-language modeling, Mel-LLM shows that direct spectrogram ingestion can approach encoder-initialized performance on ASR while simplifying the stack. On OpenASR public test sets, the encoder-based Phi-4-MM-FT-Base baseline reaches 6.97 average WER, encoder-free Mel-LLM with Phi-4-MM LoRA initialization reaches 7.12, encoder-free Mel-LLM with random initialization reaches 7.44, and Whisper-Large-V3 also reports 7.44. At a 12.5 Hz audio token rate, the same system reports 7.12 WER together with a 1.57× speedup versus the encoder-based baseline (Fan et al., 8 Jun 2026).

In unified audio-visual encoding, OmniEncoder reports large gains on tasks requiring continuous motion understanding under identical decoder-side token budgets to Qwen2.5-Omni-3B (SFT): Diving48 90.8% vs 25.7%, Something-Something v2 68.7% vs 54.3%, SLR500 97.8% vs 37.3%, and NationalCSL6707 90.32% vs 12.9%. On audio-visual reasoning it remains competitive on AVQA at 82.6% vs 84.9%, while improving Speaker Localization from 97.0% to 98.8% and Speaker Identification from 66.8% to 82.31%. On LRS2 it reports VSR WER 45.3%, ASR WER 10.2%, and AVSR WER 7.2% (Bai et al., 2 May 2026).

In single-backbone contrastive pretraining, Omni-C reports near-parity or modest degradation in unimodal zero-shot performance relative to expert models, but strong recovery through downstream adaptation and clear deployment savings. For image-text zero-shot classification after cross-modal alignment, the expert image-text baseline gives 12.17% average accuracy with 196.4M inference parameters, whereas Omni-C (I+T) gives 12.63% with 111.1M parameters and Omni-C (I+A+T) gives 13.79% with 111.9M parameters. For audio-text zero-shot classification, the expert baseline gives 4.44% with 194.8M parameters and Omni-C (I+A+T) gives 4.51% with 111.9M parameters (Lau et al., 27 Feb 2026).

Progressive alignment shows similar efficiency claims. In image-text zero-shot classification, OneEncoder-1 reports CIFAR-10 78.15, Oxford-IIIT Pets 69.23, CIFAR-100 58.18, Caltech-101 56.20, and Tiny ImageNet 52.27, all above the listed CLIP baselines. In text-audio retrieval, OneEncoder-1 reports on AudioSet P@1/R@1/mAP of 5.37/76.10/28.37 versus AudioCLIP’s 4.27/75.37/27.12. It also reports transitive image-audio zero-shot classification on CIFAR-10 of 77.01 versus AudioCLIP’s 61.28 (Faye et al., 2024).

For long-range unified encoding, MAELRE reports on MeSH prediction with 2048 tokens EBF 43.28 and MiF 42.75 for the base model, improving further at 4096 tokens to EBF 44.28 and MiF 43.77. It reports 95.38% on the UCR MosquitoSound task, 57.85% on VGGSound, and 82.97% on ImageNet-1K, while presenting lower memory and FLOPs than several long-context and modality-specific baselines (Parag et al., 25 Jul 2025).

6. Limitations, misconceptions, and open directions

A persistent misconception is that “encoder-free” always means the complete disappearance of any modality-specific front-end. The literature is more nuanced. EVE, EVEv2, and Mel-LLM remove dedicated pre-trained vision or speech encoders and learn continuous tokenization directly inside the unified model (Diao et al., 2024, Diao et al., 10 Feb 2025, Fan et al., 8 Jun 2026). Omni-C still uses modality-specific preprocessing and separate lightweight projection heads, but the backbone is single and dense (Lau et al., 27 Feb 2026). OneEncoder is explicit that it is “encoder-free” only in the sense that large modality-specific encoders are not trained; the frozen encoders remain indispensable feature extractors (Faye et al., 2024). MCA, finally, is best understood as a theory-and-regularization paper for unified encoders rather than a direct encoder-elimination paper (Wu et al., 17 Oct 2025).

The trade-offs are equally consistent. Encoder-free training can be slow and unstable without careful curricula or strong initialization; EVE reports slow convergence and large performance gaps under direct training, and Mel-LLM finds multimodal initialization from Phi-4-MM crucial when data is limited (Diao et al., 2024, Fan et al., 8 Jun 2026). High resolution and dense temporal sampling improve information richness but increase throughput cost; EVEv2 notes throughput drops as visual tokens increase, and OmniEncoder notes that dense unified encoding increases encoder-side compute despite Temporal Window Shifting (Diao et al., 10 Feb 2025, Bai et al., 2 May 2026). Shared dense encoders may show modest zero-shot degradation on audio and text, as in Omni-C, and aggressive token merging can suppress fine-grained detail, as reflected in MAELRE’s no-merge ablations on MeSH (Lau et al., 27 Feb 2026, Parag et al., 25 Jul 2025). Unified fusion models can depend on external priors such as ConvNeXt and CLIP, and OneEncoder still requires paired data between a new modality and at least one already aligned modality (Li et al., 16 Nov 2025, Faye et al., 2024).

The open directions stated by the papers largely converge. EVEv2 proposes that the unified decoder-only framework and modality-wise sparsity can extend naturally to audio and video (Diao et al., 10 Feb 2025). MCA suggests learned, data-driven composition operators, causal or invariance regularization, and extension to $$x_\tau = e_\tau W_{\mathrm{proj}} + b_{\mathrm{proj}},$$3-modality settings such as audio and video (Wu et al., 17 Oct 2025). OmniEncoder explicitly proposes adding modalities such as haptics by reserving unique coordinates in Omni-RoPE (Bai et al., 2 May 2026). Mel-LLM outlines extensions to raw waveforms, audio-visual speech, and joint ASR+TTS training within one decoder (Fan et al., 8 Jun 2026). UP-Fusion points toward richer pretrained priors, adaptive Top-k schedules, and learned text prompting (Li et al., 16 Nov 2025). Taken together, these directions suggest that the field is moving from encoder-free modality as a narrow architectural simplification toward a broader program of monolithic multimodal systems in which tokenization, alignment, and reasoning are all learned inside one shared backbone.