---
title: 'MindSpeed-MLLM: Distributed Multimodal Training'
url: https://www.emergentmind.com/topics/mindspeed-mllm-0f25f78c-ccde-4e18-abe7-9b9da7374679
type: topic
---

# MindSpeed-MLLM: Distributed Multimodal Training

MindSpeed-MLLM is a distributed multimodal training framework developed specifically for efficient and accurate training of large-scale vision-language models on Ascend NPUs, as exemplified by its central role in the MindVL multimodal large language model (MLLM). By integrating hybrid parallelism, operator-equivalent replacements for Ascend hardware, multimodal data packaging, and advanced system-level optimizations, MindSpeed-MLLM provides an optimized foundation for high-throughput, memory-efficient pre-training and supervised instruction tuning of models that process both text and visually dense content. Its design draws heavily from Megatron-LM but extends and adapts core methodologies to fit the federated, heterogeneous compute environment of Ascend 910B devices [2509.11662].

## 1. System Architecture and Component Hierarchy

MindSpeed-MLLM is structured as a modular multimodal layer interfacing three specialized libraries:
- MindSpeed-Core (derived from Megatron-LM): backbone for hybrid parallelism, memory partitioning, and scheduling.
- MindSpeed-LLM: language modeling optimizations, including distributed transformer and optimizer adaptations.
- MindSpeed-MM: vision-specific model optimizations tailored for image patching and embedding.

Key architectural components:
- **Distributed Multi-Modal Data Loader:** Synchronizes partitioned text/visual data streams for efficient parallel input.
- **Hybrid Parallel Engine (Megatron 3D):** Integrates tensor parallelism, pipeline parallelism, and data parallelism across NPUs.
- **Operator Fusion & Equivalence Layer:** Implements hardware-specific operator replacements to maintain numerical correctness and efficient compute on Ascend hardware.
- **System Scheduler & Runtime Tweaks:** Dynamically adapts compute, memory, and communication strategies based on real-time profiling.

The following diagram depicts the logical hierarchy (text description):

```
┌────────────────────────────────────────────┐
│ MindSpeed-MLLM                            │
│  ├─ Distributed Multi-Modal Data Loader    │
│  ├─ Hybrid Parallel Engine (Megatron 3D)   │
│  │    ├─ Tensor Parallelism (MindSpeed-Core)
│  │    ├─ Pipeline Parallelism (MindSpeed-Core)
│  │    └─ Data Parallelism (HCCL)           │
│  ├─ Operator Fusion & Equivalence Layer    │
│  └─ System Scheduler & Runtime Tweaks      │
├────────────────────────────────────────────┤
│ MindSpeed-LLM       MindSpeed-MM           │
└────────────────────────────────────────────┘
```

A central memory-partitioning constraint governs allocation per device:
$$
M_{\text{model\_shard}} = \frac{M_{\text{total\_params}}}{P_{\text{tensor}}} + M_{\text{activations\_per\_microbatch}}
$$
where $P$ is the total number of GPUs (NPUs).

## 2. Hybrid Parallelism Methodologies

MindSpeed-MLLM extensively utilizes 3D hybrid parallelism, following Megatron-LM paradigms:

- **Tensor Parallelism:** Model weight matrices are partitioned across $G$ devices. Each device holds $\frac{1}{G}$ of a weight matrix and computes partial results, synchronized with All-Reduce:
  $$
  \Delta W = \sum_{g=1}^G \Delta W_g,\quad \text{Comm}_{\text{tensor}} \approx (G-1)\,\frac{|W|}{G}
  $$

- **Pipeline Parallelism:** The $L$ transformer layers are split into $S$ pipeline stages. Microbatches flow through a 1-forward-1-backward (1F1B) schedule, incurring communication of
  $$
  \text{Comm}_{\text{pipe}} = (S-1)\times B_m\times A_{\text{hidden}}
  $$
  per microbatch ($B_m$).

- **Data Parallelism:** The replicated full model is distributed across $D$ groups, synchronizing trainable $\Theta$ via HCCL All-Reduce:
  $$
  \text{Comm}_{\text{data}} \approx (D-1)\times |\Theta|
  $$
  with local memory usage per device:
  $$
  M_{\text{local}} = \frac{|\Theta|}{G D} + \frac{|\text{acts}|}{S} + \text{overhead}_{\text{optimizer}}
  $$
  These degrees satisfy $G\times S\times D = P$ (total device count).

## 3. Operator Equivalence and Hardware Adaptation

To ensure functionality on Ascend NPUs (CANN stack), MindSpeed-MLLM applies operator replacements:
- **FlashAttention** is replaced by an NPU fusion attention kernel (`acl_op "AttentionFusion"`).
- **Sparse/Dynamic Masking** uses "mask compression," a variable-length fused call compatible with NPU specifications.
- **Conv2d/Conv3d** operations are recast as MatMul: convolution is implemented as im2col followed by GEMM, ensuring equivalent numerical results:
  $$
  \text{Conv2d}(X,W)\equiv \text{MatMul}(\mathrm{im2col}(X),\,W_{\mathrm{reshaped}})
  $$
- **Precision Management:** Activations and weights use BF16; master copies and gradients reside in FP32:
  $$
  X_{BF16}\to X_{FP32}; \quad W_{BF16},\,\Delta W_{BF16}\to W_{FP32}
  $$

These measures maintain accuracy parity with CUDA-based kernels while leveraging Ascend-specific optimizations.

## 4. Multimodal Data Packaging Strategies

MindSpeed-MLLM deploys an online multimodal data packaging procedure to efficiently mix variable-length visual and textual content within fixed-length sequences:
- **Image preprocessing:** Resize to nearest multiple of 28; split into 14-pixel stride patches; group every four spatially adjacent patches and project via a 2-layer MLP to $v \in \mathbb{R}^d$.
- **Text processing:** Tokenize, pad/truncate to fixed length.
- **Online sequence packing (per DP group):** Combine up to $B$ samples with cumulative text length $\leq L_\text{text}$ and visual embedding length $\leq V_\text{max}$, pad, and concatenate.

Pseudocode (excerpted):

```python
def PACK_BATCH(dataset_shard, B, L_text, V_max):
    batch = []
    cur_text_len, cur_vis_len = 0, 0
    for sample in dataset_shard:
        tkn = tokenize(sample.text)
        vis = extract_patches(sample.image)
        if cur_text_len + len(tkn) <= L_text and cur_vis_len + len(vis) <= V_max:
            batch.append((tkn, vis))
            cur_text_len += len(tkn)
            cur_vis_len += len(vis)
        if len(batch) == B:
            yield PAD_AND_FORMAT(batch)
            batch.clear(); cur_text_len, cur_vis_len = 0, 0
    if batch:
        yield PAD_AND_FORMAT(batch)
```
`PAD_AND_FORMAT` standardizes sequence lengths and modality concatenation.

## 5. Three-Phase Training Workflow and Hyperparameterization

MindSpeed-MLLM divides training into three sequential phases to incrementally build model capability:

| Stage      | Tokens Budget | Seq. Length          | Trainable            | Batch Size | Max LR | Min LR | LR Warmup Ratio |
|------------|---------------|----------------------|----------------------|------------|--------|--------|-----------------|
| Warm-up    | 256B          | 8192                 | MLP adaptor          | 1024       | 2e-4   | 2e-5   | 0.1             |
| Multitask  | 179B          | 8192                 | All parameters       | 1024       | 2e-5   | 1e-5   | 0.1             |
| SFT        | 12B           | 2048/4096/8192       | All parameters       | 512        | 1e-5   | 0      | 0.1             |

- **Warm-up:** Aligns vision encoder and MLP adaptor to LLM embedding space using image-caption, OCR, visual grounding, and STEM data; updates only adapter weights.
- **Multitask pre-training:** All parameters trainable; data comprises a mix of image-text, QA, OCR, chart/table, video, and pure text. Aggregate multitask loss:
  $$
  \mathcal{L} = \sum_{i\in\text{tasks}} \alpha_i\mathcal{L}_i, \quad \mathcal{L}_i = -\sum \log p(y_i|x_i)
  $$
- **Supervised instruction tuning (SFT):** Human-annotated instructions mixed with language-only data (DeepSeek R1, ratio ≈ 1:1). Within-sample loss averaging balances short/long response distributions.

## 6. Performance Scaling and Empirical Outcomes

MindSpeed-MLLM demonstrates efficient scaling and throughput:

- **On 1024 Ascend 910B devices:**
  - MFU ≈ 40% on both 8B and 32B models.
  - For the 8B model with 8192-token sequences and batch size 1024, end-to-end throughput achieves substantial sample and token rates (exact figures withheld in the data).
  - Near-linear strong scaling to 512 cards; weak scaling efficiency >90% up to 1024 cards.

- **Relative to baselines (MindSpeed-MM or LlamaFactory + CANN):**
  - 1.8×–2.2× faster end-to-end training throughput.
  - Convergence-matched loss curves (within 1% MAE/MRE of baseline curves).

These outcomes confirm the effectiveness of MindSpeed-MLLM in large-scale multimodal training on Ascend hardware [2509.11662].

## 7. Advanced Evaluation and Model Optimization Techniques

Two advanced post-training procedures further optimize MindVL and related models:

- **Test-time Resolution Search:** A grid search over $(m_{\min}, m_{\max})$ (ranges of image pixel counts, e.g., $m_{\min}\in \{4,16,32,64\}\times28^2$, $m_{\max}\in\{1280,2048,2560,3072,4096,8192\}\times28^2$):
  - Images are upsampled if $<$ $m_{\min}$ or downsampled if $>$ $m_{\max}$.
  - For each pair, validation accuracy $A(m_{\min}, m_{\max})$ is measured.
  - The optimal range is selected as:
    $$
    (m_{\min}^*, m_{\max}^*) = \arg\max_{m_{\min}, m_{\max}}\,\frac{1}{N}\sum_{i=1}^N \mathbf{1}\{ \hat y_i(m_{\min}, m_{\max}) = y_i \}
    $$

- **Model Weight Averaging:** Training checkpoints from SFT runs at sequence lengths $\ell \in \{2k, 4k, 8k\}$ are averaged:
  $$
  \theta_{\text{merged}} = \frac{1}{3}\sum_{\ell} \theta^{(\ell)}
  $$
  Optionally, exponential moving average (EMA) is applied:
  $$
  \theta_{\mathrm{EMA}}^{(t)} = \alpha\,\theta_{\mathrm{EMA}}^{(t-1)} + (1-\alpha)\,\theta^{(t)},\quad \alpha \in [0,1]
  $$

These strategies refine generalization performance and accuracy consistency across diverse input resolutions and sequence lengths.

---

The MindSpeed-MLLM framework synthesizes advanced distributed multimodal optimization, precision alignment, hardware-specific operator equivalence, and high-efficiency data manipulation. These features collectively underpin data- and compute-efficient training of large vision-language models on Ascend NPUs, establishing strong parity or improvement over contemporary GPU-centric pipelines while achieving comparable accuracy with a fraction of the training data [2509.11662].

Source: https://www.emergentmind.com/topics/mindspeed-mllm-0f25f78c-ccde-4e18-abe7-9b9da7374679