USEformer: Unified Synergy Embedding Transformer

• The paper introduces USEformer, a cross-modal transformer that unifies medical images, free text, and clinical variables via early-stage fusion and shared attention mechanisms.
• It employs bidirectional multimodal attention and synergy embedding to enable mutual enrichment between modalities, achieving significant improvements in diagnostic performance.
• The architecture demonstrates parameter efficiency through shared weights and lightweight adapters, making it highly adaptable for clinical diagnostics and robust domain adaptation.

The Unified Synergy Embedding Transformer (USEformer) is a shared-parameter, cross-modal transformer architecture designed to unify heterogeneous multimodal data—such as medical images, free-text, and structured clinical variables—into a holistic, synergistic representation for clinical diagnosis and medical vision-language applications (Zhou et al., 2023, Peng et al., 6 Aug 2025). USEformer is characterized by its bidirectional interaction mechanism, enabling mutual enrichment between modalities (e.g., radiographs and textual records), efficient parameterization, and robust domain adaptation, particularly within medical settings.

1. Architectural Overview

USEformer is designed to jointly encode multiple data types—including visual (e.g., radiographs, CT), structured (e.g., laboratory values, demographics), and unstructured text (e.g., clinical notes, chief complaints)—within a single transformer stack. The architecture departs from modality-specific encoding by introducing early-stage fusion through specialized embedding layers that convert each modality (images, text, tabular data) to a uniform token space. These tokens are further augmented by learnable absolute position embeddings, ensuring the model retains intra-modality positional context.

In the instantiation for clinical diagnostics (Zhou et al., 2023), input tokens are processed in parallel through two bidirectional multimodal attention blocks, followed by multiple (e.g., 10) unified self-attention layers. The NEARL-CLIP extension (Peng et al., 6 Aug 2025) frames USEformer as a bidirectional, query-based synergy adapter interleaved between frozen CLIP encoders and a lightweight orthogonal adapter, supporting efficient adaptation to medical vision-language domains.

2. Input Processing and Embedding Layers

USEformer mapping of heterogeneous data into the transformer architecture leverages learnable embedding functions for each modality. The embedding strategies include:

• Visual embedding: Medical images (e.g., $X_\text{img}\in\mathbb{R}^{H\times W\times 3}$) are split into non-overlapping $P\times P$ patches, each flattened and projected to $d$-dimensional token vectors via a linear map and bias. Raw spatial structure is encoded using learnable absolute position embeddings.
• Text embedding: Free-form text such as chief complaints are tokenized (e.g., using BERT or learned embeddings) and mapped via projection matrices to $d$-dimensional embeddings, each associated with a learnable position embedding.
• Structured tabular embedding: Clinical variables (e.g., lab values $X_\text{lab}$, demographics $x_\text{age},x_\text{sex}$) are normalized, embedded as individual tokens with per-feature projection matrices and biases, and concatenated with text tokens.

The various input token streams remain separate with modality-specific position encodings until entering the first multimodal attention block (Zhou et al., 2023).

3. Bidirectional Multimodal Attention and Synergy Embedding

The distinctive feature of USEformer is its explicit, bidirectional cross-modal interaction, termed "synergy embedding." Each multimodal attention block computes both intramodal (within-modality) and intermodal (cross-modality) scaled-dot-product attention. The steps are:

1. Layer normalization: Each modality's token sequence is normalized separately.
2. Linear projection: Queries, keys, and values are computed for each modality.
3. Intramodal attention: Standard self-attention within each modality.
4. Intermodal attention: Bidirectional cross-attention allowing image tokens to attend to text tokens and vice versa.
5. Fusion: Outputs from intra- and intermodal attention are additively combined through a residual connection and processed by an MLP.
6. Stacking: Two such multimodal blocks precede a standard stack of self-attention layers operating over the concatenated token bag.

In the NEARL-CLIP instantiation (Peng et al., 6 Aug 2025), USEformer implements cross-modal querying at each vision/text encoder layer using dynamically learned queries ($q^v, q^t$), shared projection matrices, cross-attention computations, and a feed-forward network. All module weights are shared across layers and branches, maximizing parameter efficiency.

The synergy embeddings ($z^t_k, z^v_k$) summarize complementary modality knowledge through this bidirectional, query-based mechanism.

4. Parameter Efficiency and Integration in PEFT Frameworks

USEformer enables parameter-efficient fine-tuning (PEFT) by leveraging:

• Shared weights across all (typically 6) multimodal mini-transformer blocks.
• Shared parameters between image-to-text and text-to-image cross-attention branches.
• Use of lightweight learnable query sets with modest dimensions (e.g., $N^q=32$, $D^q=128$).

The total additional parameter count for USEformer in NEARL-CLIP is approximately 0.35M, with the full USEformer + OCA stack introducing only $P\times P$0M parameters—roughly eight times smaller than conventional multi-layer transformer add-ons for similar tasks (Peng et al., 6 Aug 2025).

A comparison of parameter costs is given in the following table:

Approach	Total Parameters (M)	Weight Sharing
USEformer (NEARL-CLIP)	≈ 0.35	Yes
USEformer + OCA	≈ 1.46	Yes
6-layer full transformer	≈ 12	No

5. Downstream Tasks, Output Heads, and Evaluation

Following the synergy embedding process, USEformer pools the unified token sequence (average pooling) and passes the result through an MLP for prediction. Output heads are tailored for multi-label or binary classification (e.g., pulmonary disease detection or COVID-19 outcome prognostication), with sigmoid activations and per-label binary cross-entropy loss.

In clinical diagnostic applications (Zhou et al., 2023), USEformer demonstrated superior performance:

• Pulmonary disease identification: 0.924 AUROC (vs 0.805 for image-only, +12% absolute).
• COVID-19 adverse outcome prediction: 0.592 AUPRC (vs 0.307 for image-only, +29% absolute).

Ablation studies confirm the necessity of both bidirectional synergy blocks: removing both yields a 7% AUROC drop; one-way flow (text→image) produces a 4% drop. Exclusion of chief-complaint tokens, lab tokens, or improper text encoding results in substantial performance degradation, underscoring the value of unified synergy embedding.

6. Role in Mutual Modality Enrichment and Domain Adaptation

USEformer achieves mutual enrichment, in contrast to prior single-direction fusion methods. In NEARL-CLIP (Peng et al., 6 Aug 2025), text queries dynamically extract salient visual features (e.g., lesion patterns relevant to disease descriptions), while visual queries guide image embeddings toward medical language concepts. This two-way adaptation bridges the domain gap for both vision and language modalities. The explicit, query-based fusion yields a representational space where neither branch’s features alone can achieve the same holistic performance.

The OCA (Orthogonal Cross-Attention Adapter) refines the adaptation by decomposing new domain signals, further isolating truly novel information and preserving the integrity of pre-trained representations.

7. Limitations, Extensions, and Prospective Directions

USEformer’s practical deployment is currently bounded by dataset size/diversity, with future work planned for multi-institute and multinational cohorts and prospective clinical trials (Zhou et al., 2023). The model’s fixed query dimensionality/size may require tuning for extreme domain shifts (Peng et al., 6 Aug 2025). Single-head cross-attention, while parameter-efficient, could underrepresent highly complex cross-modal relationships, suggesting potential value in multi-head extensions. The OCA’s orthogonalization step has quadratic computational cost in the embedding dimension, which may present scalability constraints for very high-dimensional spaces.

Emerging directions include masked-modeling strategies for handling missing modalities and continual training protocols for domain adaptation to evolving clinical scenarios (e.g., novel SARS-CoV-2 variants).

---

References:

• (Zhou et al., 2023) "A Transformer-based representation-learning model with unified processing of multimodal input for clinical diagnostics"
• (Peng et al., 6 Aug 2025) "NEARL-CLIP: Interacted Query Adaptation with Orthogonal Regularization for Medical Vision-Language Understanding"