---
title: 'QoQ-Med: Multimodal Clinical Model'
url: https://www.emergentmind.com/topics/qoq-med
type: topic
---

# QoQ-Med: Multimodal Clinical Model

QoQ-Med denotes a family of open clinical foundation models that enable joint, instruction-driven reasoning over medical images, waveforms, and text reports. The core innovation of QoQ-Med is a multimodal architecture coupled with Domain-aware Relative Policy Optimization (DRPO), a reinforcement learning objective designed to mitigate modality and domain imbalance and thereby improve diagnostic performance and interpretability across the clinical spectrum [2506.00711]. QoQ-Med models were first introduced in the publication "QoQ-Med: Building Multimodal Clinical Foundation Models with Domain-Aware GRPO Training" by DDVD233 and collaborators.

## 1. Model Architecture and Modality Fusion

QoQ-Med is instantiated in two publicly released configurations: 7 billion (7B) and 32 billion (32B) parameters. Both share an identical multimodal design comprising:

- **Medical-Image Encoder:** Processes both 2D (e.g., chest X-ray, dermoscopy) and 3D (e.g., CT, MRI, ultrasound) images. Input images are patchified (patch size $P\times P$) and passed through a pretrained visual encoder (CLIP-style ViT/Swin Transformer). Patch embeddings $z^{(v)}_j \in \mathbb{R}^{d_v}$ are linearly projected to the large language model (LLM) token space:
  $$
  e^{(v)}_j = W_v z^{(v)}_j + b_v,
  \quad W_v \in \mathbb{R}^{d_{\rm LLM} \times d_v}
  $$

- **Time-Series Signal Encoder:** Raw twelve-lead ECG (or other 1D biosignal), $x^{(t)} \in \mathbb{R}^{k\times T}$, is encoded using ECG-JEPA, with embeddings projected analogously:
  $$
  e^{(t)}_j = W_t z^{(t)}_j + b_t,
  \quad W_t \in \mathbb{R}^{d_{\rm LLM} \times d_t}
  $$

- **Text Encoder:** Tokenizes and embeds free-text prompts using the LLM's embedding layer.

The fusion mechanism interleaves visual, signal, and text tokens into a single sequence (maximum length 8192), preserving native spatial, temporal, and sequential order. This input stream is passed to an autoregressive LLM backbone (Qwen2.5-VL for 7B/32B), which jointly decodes (1) free-text chain-of-thought, (2) bounding box annotations serialized as special tokens, and (3) diagnostic labels. The only architectural difference between the 7B and 32B models is the backbone's scale—depth, hidden dimensions, and projection shapes.

## 2. Training Corpus and Domain Heterogeneity

QoQ-Med is instruction-tuned on 2.61 million question–answer pairs from the CLIMB dataset using a comprehensive set of clinical modalities and domains:

| Domain             | Samples (approximate) | Fraction (%) |
|--------------------|----------------------|--------------|
| Chest X-ray        | 700,000              | 27           |
| Mammography        | 100,000              | 4            |
| Dermoscopy         | 100,000              |              |
| Fundus             | 100,000              |              |
| Histopathology     | 100,000              |              |
| Ultrasound (3D)    | 1,830,000            | 70           |
| MRI/CT (3D)        | included above       |              |
| ECG (1D)           | 78,900               | 3            |

Medical images (97% of the corpus) span 2D (X-ray, fundus) and 3D (CT, MRI, ultrasound), with time-series ECG data as a minority modality. Each instruction-tuning record contains multi-modal input, a diagnostic prompt, and expected structured and free-text outputs [2506.00711].

## 3. Domain-Aware Relative Policy Optimization (DRPO)

QoQ-Med introduces DRPO to address clinical data imbalance by adaptively scaling rewards during RL fine-tuning:

- **GRPO Recap:** For prompt $q$ and group $G_{(q,t)} = \{ o_{(q,i,t)} \}_{i=1}^N$ with rewards $r_{(q,i,t)}$, normalized advantage:
  $$
  \hat{A}^{\rm GRPO}_{(q,i,t)} = \frac{r_{(q,i,t)} - \mu_{G_{(q,t)}}}{\sigma_{G_{(q,t)}} + \varepsilon}
  $$

- **DRPO Scaling:** Augments GRPO by domain ($T_d$) and cluster ($T_c$) temperature scaling:
  $$
  s^{\rm scaled}_{(q,i,t)} = \frac{m_{(q,i,t)} s_{(q,i,t)}}{T_{d(q),t} T_{c(q),t}}
  $$
  where $m_{(q,i,t)}$ is a KL-aware regularization factor, $T_d$ inversely reflects domain frequency/mean-reward, and $T_c$ is derived from intra-domain K-means clustering. The resulting advantages are renormalized to preserve zero mean, unit variance:
  $$
  \hat{A}^{\rm DRPO}_{(q,i,t)} = \frac{s^{\rm scaled}_{(q,i,t)}}{\operatorname{std}(\{ s^{\rm scaled} \})}
  $$
This structure enables rare and/or difficult domains to be upweighted, preventing majority or "easy" domains from dominating gradient updates [2506.00711].

## 4. Training Procedure and Hyperparameters

The training pipeline consists of two principal stages:

- **(1) Modality Alignment:** Pretrain non-image encoders (e.g., ECG-JEPA projection) using DRPO against the frozen LLM backbone.
- **(2) Full Multimodal DRPO Training:** Fine-tune the fused system on all 2.61M samples.

Optimization details include:
- AdamW optimizer, learning rate $1 \times 10^{-6}$, weight decay $1 \times 10^{-2}$, KL coefficient $1 \times 10^{-4}$.
- Batch sizes: per-device batch size = 4; rollout batch size = 512.
- Maximum context = 8192 tokens. Images are downsampled to ≤ 524,288 pixels; 3D volumes/videos are uniformly sliced to 4 frames.
- 7B models: trained on 8 × NVIDIA A100 GPUs, 32-bit precision; 32B model: 4 × H200 in 16-bit mixed precision.
- Typical training: ~1 epoch (∼2 days for 7B, ∼2 weeks for 32B with early stopping) [2506.00711].

## 5. Evaluation Metrics and Performance

QoQ-Med is benchmarked with domain-averaged macro-F1 and semantic-alignment IoU:

**Macro-F1 Results:**

| Modality       | F1 (GRPO) | F1 (DRPO) |
|----------------|-----------|-----------|
| Chest X-ray    | 0.096     | 0.125     |
| Mammography    | 0.058     | 0.253     |
| Dermoscopy     | 0.245     | 0.400     |
| CT Scan        | 0.236     | 0.321     |
| Fundus         | 0.083     | 0.088     |
| Ultrasound     | 0.086     | 0.240     |
| MRI            | 0.396     | 0.652     |
| Pathology      | 0.404     | 0.303     |
| **Overall**    | 0.200     | 0.286     |

DRPO yields a 43% relative improvement in macro-F1 compared to GRPO.

**Semantic Segmentation (IoU):**

- Open-source baselines: IoU ≈ 0.02
- QoQ-Med (7B/32B): IoU ≈ 0.20
- Commercial GPT-4o-mini: IoU ≈ 0.21

QoQ-Med’s IoU is an order of magnitude greater than open models and approximates proprietary models, with correct alignment of bounding boxes to chain-of-thought reasoning in diverse modalities [2506.00711].

## 6. Reproducibility and Release

QoQ-Med provides full transparency and supports downstream research by publicly releasing:

- All model weights for 7B and 32B.
- Modular PyTorch + FSDP + VeRL training pipeline.
- All 2.61M intermediate reasoning traces (including chain-of-thought text, bounding-box/segmentation tokens, and diagnoses).

All assets are available at [https://github.com/DDVD233/QoQ_Med](https://github.com/DDVD233/QoQ_Med) [2506.00711].

## 7. Significance and Applications

QoQ-Med represents the first open multimodal clinical foundation model jointly optimized for medical image, time-series, and text understanding. The application of DRPO directly addresses the limitations of prior critic-free RL approaches, which suffered from domain and modality skew, by scaling reward contributions from rare and/or difficult domains and clusters. This enables balanced generalist performance, as evidenced by substantial gains in macro-F1 and region localization metrics across diverse clinical imaging tasks.

The release of weights, code, and detailed traces is intended to foster reproducibility and facilitate further research in clinical multimodal foundation modeling and fairness-aware optimization [2506.00711]. A plausible implication is that the methodology underlying QoQ-Med, particularly the DRPO objective, may be broadly applicable to other imbalanced, multimodal training regimes within and beyond the medical domain.

Source: https://www.emergentmind.com/topics/qoq-med