QoQ-Med: Multimodal Clinical Model

• QoQ-Med is a multimodal clinical foundation model that fuses medical images, time-series signals, and text for joint diagnostic reasoning.
• Its innovative Domain-aware Relative Policy Optimization (DRPO) addresses modality imbalance, yielding a 43% boost in macro-F1 performance and improved IoU alignment.
• The model’s open release of weights, training pipeline, and detailed reasoning traces promotes reproducibility and advances balanced diagnostic research.

QoQ-Med denotes a family of open clinical foundation models that enable joint, instruction-driven reasoning over medical images, waveforms, and text reports. The core innovation of QoQ-Med is a multimodal architecture coupled with Domain-aware Relative Policy Optimization (DRPO), a reinforcement learning objective designed to mitigate modality and domain imbalance and thereby improve diagnostic performance and interpretability across the clinical spectrum (Dai et al., 31 May 2025). QoQ-Med models were first introduced in the publication "QoQ-Med: Building Multimodal Clinical Foundation Models with Domain-Aware GRPO Training" by DDVD233 and collaborators.

1. Model Architecture and Modality Fusion

QoQ-Med is instantiated in two publicly released configurations: 7 billion (7B) and 32 billion (32B) parameters. Both share an identical multimodal design comprising:

• Medical-Image Encoder: Processes both 2D (e.g., chest X-ray, dermoscopy) and 3D (e.g., CT, MRI, ultrasound) images. Input images are patchified (patch size $P\times P$) and passed through a pretrained visual encoder (CLIP-style ViT/Swin Transformer). Patch embeddings $z^{(v)}_j \in \mathbb{R}^{d_v}$ are linearly projected to the LLM token space:

$$e^{(v)}_j = W_v z^{(v)}_j + b_v, \quad W_v \in \mathbb{R}^{d_{\rm LLM} \times d_v}$$

• Time-Series Signal Encoder: Raw twelve-lead ECG (or other 1D biosignal), $x^{(t)} \in \mathbb{R}^{k\times T}$, is encoded using ECG-JEPA, with embeddings projected analogously:

$$e^{(t)}_j = W_t z^{(t)}_j + b_t, \quad W_t \in \mathbb{R}^{d_{\rm LLM} \times d_t}$$

• Text Encoder: Tokenizes and embeds free-text prompts using the LLM's embedding layer.

The fusion mechanism interleaves visual, signal, and text tokens into a single sequence (maximum length 8192), preserving native spatial, temporal, and sequential order. This input stream is passed to an autoregressive LLM backbone (Qwen2.5-VL for 7B/32B), which jointly decodes (1) free-text chain-of-thought, (2) bounding box annotations serialized as special tokens, and (3) diagnostic labels. The only architectural difference between the 7B and 32B models is the backbone's scale—depth, hidden dimensions, and projection shapes.

2. Training Corpus and Domain Heterogeneity

QoQ-Med is instruction-tuned on 2.61 million question–answer pairs from the CLIMB dataset using a comprehensive set of clinical modalities and domains:

Domain	Samples (approximate)	Fraction (%)
Chest X-ray	700,000	27
Mammography	100,000	4
Dermoscopy	100,000
Fundus	100,000
Histopathology	100,000
Ultrasound (3D)	1,830,000	70
MRI/CT (3D)	included above
ECG (1D)	78,900	3

Medical images (97% of the corpus) span 2D (X-ray, fundus) and 3D (CT, MRI, ultrasound), with time-series ECG data as a minority modality. Each instruction-tuning record contains multi-modal input, a diagnostic prompt, and expected structured and free-text outputs (Dai et al., 31 May 2025).

3. Domain-Aware Relative Policy Optimization (DRPO)

QoQ-Med introduces DRPO to address clinical data imbalance by adaptively scaling rewards during RL fine-tuning:

• GRPO Recap: For prompt $q$ and group $G_{(q,t)} = \{ o_{(q,i,t)} \}_{i=1}^N$ with rewards $r_{(q,i,t)}$, normalized advantage:

$$\hat{A}^{\rm GRPO}_{(q,i,t)} = \frac{r_{(q,i,t)} - \mu_{G_{(q,t)}}}{\sigma_{G_{(q,t)}} + \varepsilon}$$

• DRPO Scaling: Augments GRPO by domain ($T_d$) and cluster ($T_c$) temperature scaling:

$$s^{\rm scaled}_{(q,i,t)} = \frac{m_{(q,i,t)} s_{(q,i,t)}}{T_{d(q),t} T_{c(q),t}}$$

where $m_{(q,i,t)}$ is a KL-aware regularization factor, $T_d$ inversely reflects domain frequency/mean-reward, and $T_c$ is derived from intra-domain K-means clustering. The resulting advantages are renormalized to preserve zero mean, unit variance:

$$\hat{A}^{\rm DRPO}_{(q,i,t)} = \frac{s^{\rm scaled}_{(q,i,t)}}{\operatorname{std}(\{ s^{\rm scaled} \})}$$

This structure enables rare and/or difficult domains to be upweighted, preventing majority or "easy" domains from dominating gradient updates (Dai et al., 31 May 2025).

4. Training Procedure and Hyperparameters

The training pipeline consists of two principal stages:

• (1) Modality Alignment: Pretrain non-image encoders (e.g., ECG-JEPA projection) using DRPO against the frozen LLM backbone.
• (2) Full Multimodal DRPO Training: Fine-tune the fused system on all 2.61M samples.

Optimization details include:

• AdamW optimizer, learning rate $1 \times 10^{-6}$, weight decay $1 \times 10^{-2}$, KL coefficient $1 \times 10^{-4}$.
• Batch sizes: per-device batch size = 4; rollout batch size = 512.
• Maximum context = 8192 tokens. Images are downsampled to ≤ 524,288 pixels; 3D volumes/videos are uniformly sliced to 4 frames.
• 7B models: trained on 8 × NVIDIA A100 GPUs, 32-bit precision; 32B model: 4 × H200 in 16-bit mixed precision.
• Typical training: ~1 epoch (∼2 days for 7B, ∼2 weeks for 32B with early stopping) (Dai et al., 31 May 2025).

5. Evaluation Metrics and Performance

QoQ-Med is benchmarked with domain-averaged macro-F1 and semantic-alignment IoU:

Macro-F1 Results:

Modality	F1 (GRPO)	F1 (DRPO)
Chest X-ray	0.096	0.125
Mammography	0.058	0.253
Dermoscopy	0.245	0.400
CT Scan	0.236	0.321
Fundus	0.083	0.088
Ultrasound	0.086	0.240
MRI	0.396	0.652
Pathology	0.404	0.303
Overall	0.200	0.286

DRPO yields a 43% relative improvement in macro-F1 compared to GRPO.

Semantic Segmentation (IoU):

• Open-source baselines: IoU ≈ 0.02
• QoQ-Med (7B/32B): IoU ≈ 0.20
• Commercial GPT-4o-mini: IoU ≈ 0.21

QoQ-Med’s IoU is an order of magnitude greater than open models and approximates proprietary models, with correct alignment of bounding boxes to chain-of-thought reasoning in diverse modalities (Dai et al., 31 May 2025).

6. Reproducibility and Release

QoQ-Med provides full transparency and supports downstream research by publicly releasing:

• All model weights for 7B and 32B.
• Modular PyTorch + FSDP + VeRL training pipeline.
• All 2.61M intermediate reasoning traces (including chain-of-thought text, bounding-box/segmentation tokens, and diagnoses).

All assets are available at https://github.com/DDVD233/QoQ_Med (Dai et al., 31 May 2025).

7. Significance and Applications

QoQ-Med represents the first open multimodal clinical foundation model jointly optimized for medical image, time-series, and text understanding. The application of DRPO directly addresses the limitations of prior critic-free RL approaches, which suffered from domain and modality skew, by scaling reward contributions from rare and/or difficult domains and clusters. This enables balanced generalist performance, as evidenced by substantial gains in macro-F1 and region localization metrics across diverse clinical imaging tasks.

The release of weights, code, and detailed traces is intended to foster reproducibility and facilitate further research in clinical multimodal foundation modeling and fairness-aware optimization (Dai et al., 31 May 2025). A plausible implication is that the methodology underlying QoQ-Med, particularly the DRPO objective, may be broadly applicable to other imbalanced, multimodal training regimes within and beyond the medical domain.