YOLO-MS: Multi-Scale Object Detection

Updated 13 January 2026

The paper introduces a novel multi-scale representation learning framework that boosts YOLO detectors for real-time object detection.
It incorporates multi-scale basic blocks (MS-Blocks) and heterogeneous kernel selection to expand effective receptive fields and enhance feature fusion.
Empirical evaluations on MS COCO demonstrate significant AP gains and superior computational efficiency across YOLO-MS variants.

YOLO-MS refers to a set of frameworks and architectural strategies for enhancing YOLO (You Only Look Once) object detectors by explicitly incorporating multi-scale representation learning. The central objective is to improve detection accuracy and efficiency for objects of varying sizes, particularly in real-time settings, by leveraging heterogeneous convolutional kernels, hierarchical feature aggregation, and multi-path computations across spatial scales. The most direct instantiation of "YOLO-MS" as a model name is given in "YOLO-MS: Rethinking Multi-Scale Representation Learning for Real-time Object Detection" (Chen et al., 2023), but the term is also widely used to describe multi-scale-driven innovations in subsequent YOLO derivatives.

1. Architectural Foundations and Design Principles

YOLO-MS is built upon the insight that classical YOLO backbones and necks—often relying on uniform convolutional blocks and fixed pyramid fusions—are suboptimal for multi-scale feature modeling. The YOLO-MS architecture introduces two principal innovations:

Multi-Scale Basic Block (MS-Block): Each backbone stage replaces standard Res/CSP/ELAN blocks with a three-way channel-split module. For input $X \in \mathbb{R}^{H \times W \times C}$ , the split produces channel groups %%%%1%%%%. The outputs are computed recursively:

$Y_1 = X_1$

$Y_i = \mathrm{IB}_{k_i \times k_i}(Y_{i-1} + X_i), \quad i > 1$

where $\mathrm{IB}_{k \times k}$ denotes an "inverted bottleneck" (1×1 expand, depth-wise $k\times k$ conv, 1×1 reduce). Final output is:

$F_\text{out} = \mathrm{Conv}_{1\times 1}( \mathrm{Concat}[Y_1, Y_2, Y_3] )$

This sequence enables each group to learn disparate spatial representations according to assigned kernel sizes.

Heterogeneous Kernel Selection (HKS): Backbone stages adopt kernel sizes $[3, 5, 7, 9]$ for stages 1–4, expanding the effective receptive field (ERF) in deeper layers and preserving fine patterns in shallow layers.

Downstream from the backbone, YOLO-MS uses a Path Aggregation Feature Pyramid Network (PAFPN) with three output scales, and all neck convolutions are implemented via lightweight depth-wise MS-Blocks.

2. Multi-Scale Representation Learning Mechanisms

Fundamental to YOLO-MS is its hierarchical aggregation and scale-specialized computation:

Channel-Split and Recursive Fusion: Rather than homogeneous processing of channels, the MS-Block ensures distinct convolutional paths process different portions of the feature tensor before recursive fusion. Additive branch fusion ( $Y_{i-1} + X_i$ ) is central to this mechanism.
Effective Receptive Field Expansion: Empirical analyses confirm that the HKS protocol grows the ERF progressively from shallow to deep layers (via larger kernel sizes), enhancing sensitivity to object scale variation.
Feature Pyramid Construction: PAFPN aggregates backbone outputs across all levels, followed by upsampling and additive fusion, yielding output scales suitable for small, medium, and large object detection.

Theoretical justification for this design is rooted in balancing spatial context acquisition and computational efficiency; shallow stages prioritize detail preservation (small kernels, high spatial resolution), while deeper stages integrate broader context (large kernels, low resolution).

3. Training Protocols, Loss Functions, and Implementation

YOLO-MS is typically trained from scratch (MS COCO train2017, 115k images), abstaining from external pre-training. Key components of the training process include:

Input Augmentation: Mosaic augmentation, random flip, color jittering; label smoothing ( $\epsilon=0.1$ ).
Optimization: SGD (momentum 0.9, weight decay $5\! \times\! 10^{-4}$ ), learning rate $0.01$ with linear warmup (5 epochs) followed by cosine decay.
Output Head: Standard YOLO detection head, with per-scale 1×1 convolutional heads for objectness, class probabilities, and bounding box regression.
Loss Function: Composite of classification (BCE), objectness (BCE), and bounding box regression (CIoU):

$L_\text{CIoU} = 1 - \mathrm{IoU} + \frac{\rho^2(b, b_\text{gt})}{c^2} + \alpha v$

where $\rho$ is center-distance, $c$ is the diagonal of the smallest enclosing box, $v$ is aspect ratio penalty.

4. Model Variants and Computational Characteristics

YOLO-MS is offered in multiple model sizes for scalable deployment:

Variant	Channels per Stage	Params (M)	FLOPs (G)	Latency@640	FPS
XS	32,64,128,256	4.54	8.74	~7.6 ms	~130
S	43,86,172,344	8.13	15.58	~9.0 ms	~111
M	64,128,256,512	22.17	40.09	~12.3 ms	~81

All configurations deliver real-time inference speeds and maintain low parameter counts, with the XS variant surpassing competing models (RTMDet-tiny, ~41M params) in both accuracy and computational efficiency.

5. Empirical Performance and Comparative Evaluation

On MS COCO val2017 (no pre-training):

Model	AP	AP_S	AP_M	AP_L
YOLO-MS-XS	43.4	23.7	48.3	60.3
YOLO-MS-S	46.2	26.9	50.5	63.0
YOLO-MS-M	51.0	33.1	56.1	66.5

Comparative gains over RTMDet are reported (+2.4 AP for tiny, +1.6 for small, +1.7 for medium). Integrating MS-Blocks into third-party YOLO variants (e.g., YOLOv8-n) yields +3.1 AP gain (40.3 vs 37.2). Lower FLOPs and parameter counts are maintained throughout (Chen et al., 2023).

Grad-CAM analysis demonstrates increased activations for small object regions and improved large-object contextualization, substantiating the multi-scale functional advantage.

6. Plug-and-Play Integration with Other YOLO Variants

YOLO-MS is designed for modularity. The integration recipe:

Replace CSP/ELAN blocks with MS-Blocks in backbone or neck.
Assign depth-wise kernel sizes per stage as [3,5,7,9].
Channel dimensions may be adjusted to fit target computational profiles.
Maintain expansion ratio $r=2$ for inverted bottlenecks.

Integration of MS-Blocks into YOLOv6-tiny and YOLOv8-n consistently yields higher AP and reduced computational costs.

PyTorch-like pseudocode for MS-Block demonstrates operational details for adoption.

class MSBlock(nn.Module):
    def __init__(self, channels, kernel_size, expansion=2, num_splits=3):
        # ...
    def forward(self, x):
        x = self.conv1x1_expand(x)
        xs = torch.chunk(x, 3, dim=1)
        y1 = xs[0]
        y2 = self.IB_k(xs[1] + y1)
        y3 = self.IB_k(xs[2] + y2)
        return self.conv1x1_fuse(torch.cat([y1, y2, y3], dim=1))

7. Interpretations and Influence on Contemporary YOLO Multi-Scale Strategies

The YOLO-MS protocol exemplifies structured multi-scale aggregation as a core principle of next-generation YOLO detectors. Distinct from basic FPN/PAN fusions and static single-scale convolutions, it stipulates stage-wise ERF management and explicit multi-path feature synthesis.

Subsequent models adopting the "YOLO-MS" name or spirit (e.g., MS-YOLO for domain adaptation, MS-YOLO for medical/infrared imaging, and mixture-of-experts YOLO frameworks (Meiraz et al., 17 Nov 2025)) have utilized multi-scale heads, kernel diversity, cross-path fusion, and dynamic routing to boost accuracy for diverse object sizes and scene complexities.

The quantitative gains and computational efficiency achievable with YOLO-MS set a foundational benchmark for real-time object detection research, especially in edge-deployable and resource-constrained environments.