Bi-modal Coding Framework Overview

• Bi-modal Coding Framework is a family of coding schemes that explicitly employs two alternative representations to optimize constraint satisfaction and dual-stream processing.
• It integrates methods such as dual codeword mapping, cross-modal attention, and adaptive mode selection to improve alignment, rate–distortion efficiency, and robustness.
• Its practical applications span optical storage, video captioning, object detection, and controllable code generation by exploiting structured duality in data representation.

A bi-modal coding framework is a family of coding and representation schemes in which two alternatives are structurally central to the encoder, decoder, or latent representation. In the classical constrained-coding sense, a user input block can be mapped to two codewords. In multimodal learning and semantic communication, the term denotes joint coding of two modalities such as audio and visual, image and text, RGB and depth or thermal, or neural architectures and natural language. In application-specific compression, it can denote a two-mode predictor that switches between rigid and affine transformations. The shared idea is not a single canonical architecture, but the explicit exploitation of duality—two codewords, two streams, or two transformation modes—to improve constraint satisfaction, alignment, robustness, or rate–distortion behavior (Roth et al., 2019, Iashin et al., 2020, Hoang et al., 21 Sep 2025).

1. Semantic range of the term

The literature uses bi-modal coding in at least three technically distinct ways. One usage belongs to constrained coding theory, where a code is bi-modal if each input block can be mapped to two output codewords under a channel constraint. A second usage appears in multimodal machine learning, where two modalities are encoded, aligned, fused, or predicted jointly. A third usage appears in domain-specific codecs, where two transformation models are available and the coder selects between them according to content characteristics.

Usage of the term	Representative formulation	Representative paper
Constrained coding	An input block can be mapped to two codewords	(Roth et al., 2019)
Multimodal representation	Two modalities are jointly encoded and fused	(Iashin et al., 2020)
Hybrid transform coding	Two motion models, rigid and affine, are used	(Hoang et al., 21 Sep 2025)

This terminological spread matters because the phrase does not by itself specify whether the relevant duality is combinatorial, architectural, or geometric. In audio-visual dense video captioning, the duality is between audio and visual streams; in RGB-D and RGB-T salient object detection, it is between RGB and an auxiliary modality; in semantic communication, it is between modality-specific semantic streams transmitted through conventional digital links; and in dynamic human mesh compression, it is between rigid and affine region models (Iashin et al., 2020, Pang et al., 2021, He et al., 2023, Hoang et al., 21 Sep 2025).

2. Constrained-coding foundations

In constrained coding theory, bi-modal constrained coding refers to an encoding model whereby a user input block can be mapped to two codewords. For a constraint $S$ over an alphabet $\Sigma$ partitioned as $\{\Sigma_0,\Sigma_1\}$, an $(S,n_0,n_1)$-encoder is a finite-state lossless encoder such that from any state, there are exactly $n_0$ outgoing edges labeled with symbols from $\Sigma_0$ and $n_1$ outgoing edges labeled with symbols from $\Sigma_1$ (Roth et al., 2019).

The central existence result is algebraic. Let $S$ be an irreducible constraint represented by an irreducible deterministic graph $G$. A bi-modal encoder exists if and only if

$\Sigma$0

where

$\Sigma$1

Equivalently, existence can be checked through the linear program

$\Sigma$2

This gives necessary and sufficient conditions for a given constraint, rather than only a constructive heuristic (Roth et al., 2019).

The asymptotic rate behavior also differs from ordinary fixed-length constrained coding. If $\Sigma$3 denotes the largest $\Sigma$4 such that there exists an $\Sigma$5-encoder, then

$\Sigma$6

Thus capacity can be approached arbitrarily closely using bi-modal encoders as block length increases, although capacity may fail to be attained at finite $\Sigma$7 (Roth et al., 2019).

A further issue is finite anticipation, or finite decoding delay. Under stronger conditions one can guarantee a bi-modal encoder with finite decoding delay; for stethering-based constructions the anticipation is bounded by

$\Sigma$8

with $\Sigma$9. This line of work is directly tied to storage applications such as optical disks, where multi-modal coding allows DC control in addition to satisfying runlength limited constraints, including parity balancing in the DVD $\{\Sigma_0,\Sigma_1\}$0-RLL setting (Roth et al., 2019).

3. Distributed semantic communication

In semantic communication, a bi-modal coding framework typically means a distributed multi-modal semantic communication framework incorporating the conventional channel encoder/decoder. A well-trained multi-modal model $\{\Sigma_0,\Sigma_1\}$1 is split into semantic encoders $\{\Sigma_0,\Sigma_1\}$2 on edge devices and a semantic decoder $\{\Sigma_0,\Sigma_1\}$3 at the server; for bi-modal coding, $\{\Sigma_0,\Sigma_1\}$4 (He et al., 2023).

Each semantic encoder first performs modal embedding and then feature extraction. For modality $\{\Sigma_0,\Sigma_1\}$5,

$\{\Sigma_0,\Sigma_1\}$6

At the server, the semantic decoder receives noisy $\{\Sigma_0,\Sigma_1\}$7 and $\{\Sigma_0,\Sigma_1\}$8, fuses them through a feature-fusion component, and executes the downstream task through a task head:

$\{\Sigma_0,\Sigma_1\}$9

A notable design choice is the decoupling of semantic extraction from channel transmission. After semantic features are extracted, they are quantized, protected by a conventional channel encoder such as LDPC, modulated, and transmitted; the receiver uses a conventional channel decoder before semantic fusion. This restores compatibility with modern digital communication systems while retaining neural semantic extraction (He et al., 2023).

The rate-adaptive component is driven by semantic importance derived from a robustness verification problem. For modality $(S,n_0,n_1)$0,

$(S,n_0,n_1)$1

For bi-modal distortions $(S,n_0,n_1)$2 and $(S,n_0,n_1)$3, the maximum output distortion is bounded by

$(S,n_0,n_1)$4

The optimization objective is to minimize end-to-end inference delay, determined by the slowest modality upload, while maintaining an output-distortion bound and respecting finite-blocklength channel constraints. The resulting code rates are chosen adaptively according to both semantic importance and instantaneous channel state information, with lower rates assigned to more semantically important modalities (He et al., 2023).

This formulation sharply differs from end-to-end joint source–channel coding treated as a black box. It uses arbitrary existing pre-trained multi-modal models, conventional physical-layer components, and an inference-time optimization rather than re-design and re-training for every new task or channel condition (He et al., 2023).

4. Alignment, fusion, and shared codebooks

In machine learning, bi-modal coding frameworks often instantiate dual-stream encoding followed by explicit cross-modal interaction. The Bi-modal Transformer generalizes the Transformer architecture for a bi-modal input and applies it to dense video captioning with audio and visual modalities. Each encoder layer contains modality-specific self-attention, bi-modal attention in which each modality attends to the other’s self-attended representation, and position-wise feedforward networks. Its decoder uses self-attention over caption tokens, two encoder–decoder attention blocks, a bridge layer, and a feedforward layer, while proposal generation is also bi-modal rather than visual-only (Iashin et al., 2020).

CAVER reinterprets RGB-D and RGB-T salient object detection as a sequence-to-sequence context propagation and update process. Its Cross-Modal Integration Unit combines Intra-Modal Self-Attention, Inter-Modal Cross-Attention, and Cross-Scale Self-Attention, and its view-mixed attention couples spatial-view and channel-view attention as

$(S,n_0,n_1)$5

Because transformer complexity is quadratic in token count, CAVER adds a parameter-free patch-wise token re-embedding strategy to reduce memory and computation without introducing learned parameters (Pang et al., 2021).

A broader composable formulation appears in i-Code, where pretrained single-modality encoders for vision, speech, and language are followed by a multimodal fusion network with merge-attention or co-attention. The framework can dynamically process single, dual, and triple-modality data during training and inference, and is pretrained with masked modality unit modeling and cross-modality contrastive learning. In the bi-modal case, the same machinery yields vision–language, vision–speech, or language–speech representations in a unified space (Yang et al., 2022).

Contrastive formulations constitute another major family. MMCL combines uni-modal contrastive coding with an efficient uni-modal feature augmentation strategy and cross-modal prediction through a pseudo siamese network, then adds instance-based and sentiment-based contrastive learning to structure the representation space for multimodal sentiment analysis (Lin et al., 2022). COBRA uses modal-specific autoencoders, an orthogonal transform layer, and a total loss

$(S,n_0,n_1)$6

so that image and text embeddings preserve both inter-class and intra-class relationships in a task-agnostic joint space (Udandarao et al., 2020).

Codebook-based formulations replace direct instance alignment with prototype-level alignment. CODIS treats image and text as two views of the same entity, encodes both into a joint vision-language coding space spanned by a dictionary of cluster centers, assigns features to codewords through optimal transport, and stabilizes training through teacher-student distillation with momentum teachers (Duan et al., 2022). CodeBind pushes this further through a modality-shared-specific codebook design: encoder outputs are split into shared embeddings quantized by a universal shared codebook and specific embeddings quantized by modality-specific codebooks, all under a compositional vector quantization scheme. This preserves semantic consistency while retaining modality-unique information and avoids forcing all features into a single shared latent space (Chen et al., 18 May 2026).

Bi-modal coding is also used for nonstandard modality pairs. ArchBERT is a bi-modal model for joint learning of neural network architectures and natural languages. It represents an architecture as a directed acyclic graph with node types, adjacency matrix, and parameter shapes, encodes the graph with a Graph Attention Network, fuses it with text in a cross encoder, and introduces Masked Architecture Modeling as a pre-training objective analogous to masked language modeling (Akbari et al., 2023).

5. Compression, report generation, and code generation

In geometry compression, a bi-modal coding framework can mean a two-mode predictor rather than two sensory modalities. For dynamic human motion sequences, “Bi-modal Prediction and Transformation Coding for Compressing Complex Human Dynamics” integrates semantic segmentation with region-specific transformation modeling. Regions with pronounced non-rigid deformation are assigned an affine transformation model, while other regions retain a rigid transformation model. Vertex deformation is expressed as

$(S,n_0,n_1)$7

with affine matrices decomposed as $(S,n_0,n_1)$8. Component selection is guided by the Lagrangian rate–distortion objective

$(S,n_0,n_1)$9

and the reported average bit-rate saving over the baseline is 33.81% (Hoang et al., 21 Sep 2025).

In pathology report generation, BiGen combines historical report retrieval with bi-modal concurrent learning. A learnable visual token cross-attends to WSI patch features, a learnable textual token cross-attends to retrieved knowledge features, and the cross-attention weights are shared between the visual and textual token attention modules. The final fused token is passed to a masked multi-head attention decoder for report generation. On PathText (BRCA), the framework reports 7.4% relative improvement in NLP metrics and 19.1% enhancement in classification metrics for Her-2 prediction versus existing methods (Zhang et al., 23 Jun 2025).

In controllable code generation, Style2Code defines the two modalities as code content and style. The style modality is represented as a 34-dimensional explicit vector, encoded by a 4-layer MLP with a residual connection into a 1024-dimensional latent style embedding, aligned to code embeddings through an InfoNCE objective, and then used to condition a LLM such as Flan-T5 in a second-stage decoder. The framework supports style interpolation and user personalization via lightweight mixing (Zhang et al., 26 May 2025).

VisCodex addresses visually grounded code synthesis by merging a vision-LLM and a coding LLM. Its task-vector formulation is

$n_0$0

where the vision encoder and projection modules remain fixed and only the language-model backbone is merged. Training uses the Multimodal Coding Dataset with 598k samples, and evaluation includes InfiBench-V, a benchmark of 322 visually rich programming questions (Jiang et al., 13 Aug 2025).

Large-scale codebook systems extend the same theme beyond strict bi-modality. UniCode$n_0$1 constructs a 500K-entry visual codebook from SigLIP patch embeddings and uses a cascaded design with a frozen codebook for stable indexing and a trainable codebook for downstream refinement, reporting more than 98% token utilization at that scale (Chen et al., 25 Jun 2025). TriMM introduces collaborative multi-modal coding for 3D generation, with modality-specific encoders for RGB, RGBD, and point clouds, a shared decoder into a triplane latent space, and auxiliary 2D and 3D supervision to improve robustness and geometry quality (Cao et al., 21 Aug 2025). These are not purely bi-modal in the narrow sense, but they show how bi-modal design ideas scale into multi-modal code spaces.

6. Common design principles and conceptual distinctions

A recurrent misconception is that bi-modal coding is synonymous with simple feature concatenation. The surveyed frameworks instead rely on explicit structure: two codeword choices in constrained coding, separate semantic encoders followed by conventional channel coding in semantic communication, cross-modal attention units in transformer fusion, shared-specific codebooks in representation learning, or rigid-versus-affine mode selection in geometry compression. This suggests that the defining property is controlled duality, not merely the presence of two inputs (Roth et al., 2019, He et al., 2023, Pang et al., 2021, Hoang et al., 21 Sep 2025).

A second misconception is that a shared representation must erase modality-specific information. CodeBind explicitly decomposes features into shared and specific components, and UniCode$n_0$2 separates frozen anchoring from trainable refinement. A plausible implication is that successful bi-modal coding frequently depends on partial alignment rather than total homogenization, especially when dominant modalities would otherwise overshadow weaker ones (Chen et al., 18 May 2026, Chen et al., 25 Jun 2025).

A third misconception is that more modalities or more tokens automatically improve performance. Many frameworks instead emphasize selection and suppression: top-$n_0$3 high-attention patches and region pooling in BiGen, patch-wise token re-embedding in CAVER, random cutoff augmentation in MMCL, and semantic-importance-weighted protection in semantic communication. This suggests that bi-modal coding is often as much about discarding irrelevant structure as about combining complementary structure (Zhang et al., 23 Jun 2025, Pang et al., 2021, Lin et al., 2022, He et al., 2023).

Taken together, the literature presents bi-modal coding as a general design pattern for balancing two alternatives under task-specific constraints. In one lineage the alternatives are two permissible codewords; in another they are two data modalities; in another they are two transformation regimes. The technical implementations differ sharply, but the common objective is to use a structured dual representation to improve decodability, alignment, robustness, or rate–distortion efficiency.