---
title: Codec-Free End-to-End Models
url: https://www.emergentmind.com/topics/codec-free-end-to-end-models
type: topic
---

# Codec-Free End-to-End Models

A codec-free end-to-end model is a machine learning system in which the signal representation, compression, and task objectives are integrated within a single, jointly trained network, fully replacing the hand-designed codec or tokenization pipeline. These models dispense with explicit intermediate codebooks, quantized speech/audio/text/image tokens, or fixed standards, instead learning their own latent or continuous embedding spaces directly optimized for the downstream objectives—whether rate–distortion, semantic preservation, or multi-modal reasoning. Their core characteristic is the elimination of manual decomposition into discrete atomic units (“codecs” in speech/image/video or “tokens” in NLP), enabling direct optimization over raw inputs. This paradigm is realized across imaging, speech, video, language, and communications, giving rise to highly expressive and adaptive systems with streamlined workflows and, in many cases, superior performance versus modular pipelines.

## 1. Definition and Rationale

Codec-free end-to-end models are distinguished by their direct mapping from raw input (pixels, waveforms, bytes) to output (reconstructed signals, task predictions) via a differentiable deep architecture, with no discrete codebook or tokenization step between encoding and decoding. For language, this means mapping directly from bytes to byte distributions without intermediate tokens [2212.10474]. In audio/image/video compression, the classic chain—signal $\to$ codec $\to$ bits—is replaced by neural networks that learn bottleneck representations and quantizers in a task-driven, end-to-end fashion [2511.18706, 1710.09064, 2401.08194, 2004.09226, 2009.14110]. For neural wireless communication, both the modulation/demodulation and equalization/detection pipelines can be dissolved into a single, end-to-end learned chain operating directly in a continuous space [2510.25416]. In emerging multi-modal large language models (LLMs) for speech/vision, codec-free means eschewing quantized audio or visual tokens and operating over continuous learned embeddings throughout all modality transformations [2411.18138].

Motivations include: the avoidance of suboptimal human-imposed bottlenecks; universality across domains and languages; sharply reduced task-specific preprocessing or hand-engineering; and the enabling of direct, differentiable optimization for perceptual or semantic objectives.

## 2. Architectures and Methodologies

### 2.1 End-to-End Compression and Transmission

In image compression, codec-free end-to-end pipelines consist of learnable encoders, low-dimensional bottleneck layers (often with differentiable quantization or latent VQ), and decoders, all trained against a joint rate–distortion or rate–perception loss [2511.18706, 2401.08194, 2011.06409]. For speech, convolutional or cascaded residual autoencoders replace the entire psychoacoustic feature stack, waveform quantizer, and entropy coder [1710.09064, 1906.07769].

For video, latent-difference autoencoders with self-attention, or recurrent autoencoders operating on displaced residuals, learn to exploit temporal structure without explicit motion estimation or compensation [2004.09226, 2009.14110]. In wireless, neural transceivers are built as joint transmitter–receiver CNN/ResNet chains, with learnable constellation mapping, soft bit likelihood output, and neural equalization—fully abolishing conventional block-encoded modulation and pilot/aided channel estimation [2510.25416].

### 2.2 Token-Free NLP

Codec-free language models (e.g., ByGPT) discard tokenization and operate on raw UTF-8 byte sequences, mapping directly from bytes to bytes using standard transformer or decoder-only architectures [2212.10474]. These architectures are typically initialized from large-scale byte-level pretraining and adapt to task-specific structure via control symbols embedded as additional byte codes.

### 2.3 Multi-Modal and Streaming Architectures

Codec-free LLMs for speech understanding/generation (e.g., SALMONN-omni) employ raw audio encoders to produce continuous embeddings, use transformer-based architectures to jointly reason over audio and linguistic modalities, and synthesize waveforms from output embeddings via streaming vocoders. There is no representation of the signal as quantized code vectors or tokens at any stage [2411.18138].

## 3. Training Paradigms and Objectives

All codec-free end-to-end systems are driven by joint objectives encompassing compression rate, signal fidelity, perceptual quality, and in many cases, task-specific or cross-modal supervision. The learning framework typically requires custom differentiable proxies for otherwise discrete or non-differentiable stages (quantization, entropy, symbol assignment), usually leveraging soft assignments and straight-through estimators. Representative objective forms include:
- Image/audio: $L = D(x,\hat x) + \lambda \, R({\rm codes})$, with $D$ a distortion or perceptual loss, $R$ a rate/entropy penalty, and possibly adversarial or feature-space matching losses [2511.18706, 1710.09064, 2401.08194].
- Video: multi-term rate–distortion including temporal dependencies (e.g., LSTM-driven, or attention-based fusion of sequential embeddings) [2004.09226, 2009.14110].
- Language: byte-level (or char-level) cross-entropy over the full input sequence [2212.10474].
- Wireless: cross-entropy of bit-LLRs, rate constraints, and physical-layer metrics e.g. peak-to-average power ratio (PAPR) compliance [2510.25416].
- Multi-modal: combined text, speech, and “thinking” losses with asynchronous cross-modal scheduling [2411.18138].

## 4. Applications and Comparative Results

Codec-free end-to-end models have demonstrated strong empirical performance, often eclipsing traditional codecs or hybrid pipelines in both objective and subjective metrics.

**Image compression:** CoD achieves PSNR within −2.1% BD-Rate of VTM (H.266/VVC) while delivering better perceptual quality (FID, DISTS) at ultra-low bitrates (e.g., 0.0039 bpp) than GAN codecs and prior diffusion codecs [2511.18706]. Frequency-oriented models can outperform all classical codecs, including VVC, on MS-SSIM, and preserve high task accuracy for detection and segmentation [2401.08194, 2011.06409].

**Speech coding:** DNN-based systems match or exceed AMR-WB and even OPUS in objective SNR and PESQ at comparable bitrates with far fewer parameters [1710.09064, 1906.07769].

**Video:** Models like MOVI-Codec, which forgo motion estimation entirely in favor of spatio-temporal latent residuals, outperform H.264/AVC, HEVC/H.265, and H.266/VVC in MS-SSIM at higher bitrates [2009.14110]. Self-attention–driven models with learned masks approach VVC rate–distortion curves [2004.09226].

**Token-free NLP:** Byte-level LMs such as ByGPT excel at producing and controlling character-level features (rhyme, alliteration) unachievable by tokenized transformers, while exhibiting reduced rote memorization [2212.10474].

**AI-native wireless:** End-to-end trained pilot/CP-free transceivers, operating with no fixed constellation, outperform state-of-the-art model-based, pilot-dependent OFDM at BER $\le 10^{-3}$, improve throughput by 26.4%, and adapt rapidly to dynamic channel conditions via minimal parameter updates [2510.25416].

**Multimodal LLMs:** Models employing continuous embedding streams (instead of quantized tokens) enable full-duplex conversational AI with lower turn-taking latency, improved ASR, streaming TTS, and natural integration of multiple modalities [2411.18138].

## 5. Integration, Interpretability, and System Design

An essential property of codec-free end-to-end models is their capacity for tight integration with downstream vision, language, or communication tasks. For “compression for machines,” codec–task joint fine-tuning yields up to +7% absolute mAP over standard codecs for detection at low rates, and selective adaptation (encoder/decoder/task) delivers a flexible trade-off between performance and compatibility [2011.06409]. Learned frequency-oriented transforms produce interpretable latent splits aligning with human-perceptual and semantic saliency, enabling scalable and selective transmission [2401.08194].

By eliminating modular codebooks and tokenization, these models avoid cumulative error from hard boundaries and support direct, differentiable optimization for application-specific objectives, such as barge-in handling and echo cancellation in full-duplex LLMs [2411.18138]. At the same time, their continuous or soft-quantized representations can increase compute, memory, or inference duration versus hand-designed pipelines or highly compact codecs [2212.10474, 2401.08194].

## 6. Challenges, Limitations, and Future Directions

Codec-free end-to-end approaches introduce both opportunities and open technical challenges. The lack of discrete bottlenecks increases memory bandwidth and runtime for long sequences (e.g., O($n^2$) in byte-level LMs and neural codecs), necessitating ongoing work on factorized or sparse attention and neural pooling [2212.10474, 2401.08194]. In wireless, real-time deployment requires careful PAPR control and low-overhead online adaptation modules [2510.25416].

Interpretability, while improved in frequency-oriented image models, is generally limited in learned, deep bottleneck representations outside of explicitly structured decompositions [2401.08194]. Standardization, cross-domain generalization, and task-agnostic compression pose genuine difficulties.

Research trends include: scalable learning of more structured or sparse representations; extension to arbitrary-length or -form tasks (variable stanza poetry, high-res video); full one-step distillation of diffusion codecs; multi-task cross-modality integration; and hybrid architectures balancing end-to-end optimization with the practicalities of deployment, efficiency, and robustness [2511.18706, 2212.10474, 2411.18138].

Source: https://www.emergentmind.com/topics/codec-free-end-to-end-models