Streaming Multiprocessors (SMs) Overview

• Streaming Multiprocessors (SMs) are parallel execution units in GPUs that manage registers, shared memory, and thread scheduling to maximize concurrency.
• They implement resource sharing techniques, such as register and scratchpad sharing, to mitigate fragmentation and enhance occupancy.
• Analytical models like the Many-core Machine Model (MMM) guide performance optimization by balancing resource constraints and execution speed.

Streaming Multiprocessors (SMs) are the fundamental parallel execution units in modern Graphics Processing Units (GPUs), responsible for orchestrating thousands of concurrent threads and managing hardware resources that enable high-throughput computation. SMs allocate hardware resources such as registers and on-chip shared memory to thread blocks, determine execution occupancy, and mediate latency hiding. Analytical models and resource management strategies for SMs are central to achieving efficient execution of parallel kernels on GPUs. This article provides a comprehensive overview of SM architecture, resource constraints, management methodologies, and their impact on algorithmic performance, with an emphasis on resource sharing and analytic modeling as detailed in (Jatala et al., 2015) and (Haque et al., 2014).

1. SM Hardware Resources and Occupancy Constraints

Each SM encapsulates substantial hardware resources designed to support massive thread concurrency:

• Registers (RSM): For the evaluated GTX-class device, each SM exposes 32,768 32-bit registers.
• Scratchpad (Shared Memory, MSM): Per-SM shared memory is set at 16,384 bytes.
• Thread and Block Limits: An SM supports up to 8 resident thread blocks (CTAMax = 8) and a maximum of 1,536 threads (TMax = 1,536).

Resource allocation is dictated at the thread block granularity. Given a kernel configuration (B threads/block, RperThread registers/thread, MperBlock bytes/block), resource constraints yield upper bounds on block residency:

• By registers: $$\text{BlocksReg} = \lfloor R_{\text{SM}} / (R_{\text{perThread}} \times B) \rfloor$$
• By shared memory: $$\text{BlocksSM} = \lfloor M_{\text{SM}} / M_{\text{perBlock}} \rfloor$$
• By threads: $$\text{BlocksThr} = \lfloor T_{\text{Max}} / B \rfloor$$
• By architecture: $\text{BlocksArch} = \text{CTAMax}$

The number of resident blocks per SM is thus:

$$\text{BlocksResident} = \min(\text{BlocksReg}, \text{BlocksSM}, \text{BlocksThr}, \text{BlocksArch})$$

Occupancy is expressed as the fraction of threads actually resident over the hardware maximum:

$$\text{Occupancy} = \frac{\min(\lfloor R_{\text{SM}} / R_{\text{perThread}} \rfloor, \lfloor M_{\text{SM}} / M_{\text{perBlock}} \rfloor, \lfloor T_{\text{Max}} / B \rfloor, \text{CTAMax}) \times B}{T_{\text{Max}}}$$

When only register and memory limits are significant:

$$n_{\text{CTA}} = \min(\lfloor R_{\text{SM}} / R_{\text{tb}} \rfloor, \lfloor M_{\text{SM}} / M_{\text{tb}} \rfloor)$$

$$\text{Occupancy} = \frac{n_{\text{CTA}} \times B}{T_{\text{Max}}}$$

with $R_{\text{tb}} = R_{\text{perThread}} \times B$ and $M_{\text{tb}} = M_{\text{perBlock}}$.

2. Resource Fragmentation and Wastage

Default allocation at the thread block granularity often leads to significant resource wastage. Unfilled registers and shared memory slices cannot be used by additional blocks, resulting in stranded capacity:

• Register Example: A kernel requiring 36 regs/thread and 256 threads/block ($R_{\text{tb}} = 9,216$), with $R_{\text{SM}} = 32,768$, yields $\text{BlocksReg} = 3$ blocks, consuming 27,648 registers and leaving 5,120 registers unused (~15.6% wastage).
• Shared Memory Example: For $M_{\text{tb}} = 7,200$ bytes and $M_{\text{SM}} = 16,384$, $\text{BlocksSM} = 2$ blocks use 14,400 bytes, leaving 1,984 bytes unused (~12.1% wastage).

Empirically, wastage reaches 10–40% of SM register or shared memory capacity for many kernels (Jatala et al., 2015).

3. Resource Sharing Mechanisms in SMs

Pairwise resource sharing addresses fragmentation by pooling resources between block pairs, controlled by a global threshold $t \in (0,1]$ dictating private/shared splits:

Register Sharing

• Two blocks requiring $R_{\text{tb}}$ each share $R_{\text{tb}} (1 + t) < 2R_{\text{tb}}$ registers.
• Each block receives $R_{\text{tb}} t$ private registers; remaining $R_{\text{tb}} (1-t)$ are pooled and accessed exclusively (owner/non-owner protocol).
• Hardware additions: per-warp-pair lock, comparator for register access routing, warp scheduling to prioritize owner progress.

Scratchpad (Shared Memory) Sharing

• Block pair shares $(1 + t) M_{\text{tb}}$ memory; each block holds $M_{\text{tb}} t$ privately, the rest pooled under lock.
• Only the owner block accesses the shared range. Non-owner attempts busy-wait until release.
• Hardware: per-block-pair lock in controller, comparator for address routing.

Resident Block Count Under Resource Sharing

For registers ($R$, $R_{\text{tb}}$, $t$):

• Without sharing: $U_0 = \lfloor R / R_{\text{tb}} \rfloor$ unshared blocks.
• With sharing: partition into $U$ unshared and $S$ pairs shared, with constraints:
  • $U + S = \lfloor R / R_{\text{tb}} \rfloor$
  • $$U \cdot R_{\text{tb}} + S \cdot R_{\text{tb}} (1 + t) \leq R$$

A plausible implication is increased block residency by reclaiming stranded resources, subject to lock contention effects.

4. Latency Hiding and Execution Scheduling

Resource sharing strategies are complemented by optimizations to further mask execution latencies and mitigate stall cycles. Owner/non-owner scheduling ensures one block completes its shared-region accesses before the other enters, preserving exclusive access and maximizing progress per unit of pooled resource (Jatala et al., 2015). Hardware locking, arbitration, and comparators operate per-pair within the SM, managing access efficiently and reducing idle periods.

5. Analytical Modeling: The Many-core Machine Model (MMM)

The MMM (Haque et al., 2014) abstracts GPU SMs to formalize parallel execution and overheads:

• SMs: Unbounded identical units, each with local memory (Z), SIMD-style cores.
• Program DAG: SIMD kernels ({\cal K}) arranged in precedence-enforcing DAG, each decomposed into thread blocks ({\cal B}).
• Scheduling: Each block scheduled atomically onto any SM, sharing local memory with intra-block threads.

Key performance measures:

• Work $W(B)$: Sum of thread local operations.
• Span $S(B)$: Maximum thread local operation count.
• Overhead $O(B) = (\alpha + \beta) U$: Weighted cost of global memory transfers.
• Aggregated $W(\mathcal{P})$, $S(\mathcal{P})$, $O(\mathcal{P})$ over kernels.

A Graham-Brent-style theorem provides an execution time bound on $P$ SMs:

$$T_P \leq \left( \frac{N(\mathcal{P})}{P} + L(\mathcal{P}) \right) C(\mathcal{P})$$

with $$C(\mathcal{P}) = \max_{B \in {\cal B}(P)} [ S(B) + O(B) ]$$ and $L(\mathcal{P})$ longest block dependency path.

This model accurately predicts speedup and guides tuning kernel/block parameters (e.g., per-block arithmetic intensity, tile/block sizes) to minimize $O(\mathcal{P})$, ideally balancing increased $W$ or $S$ against sharply reduced memory transfer overhead.

6. Empirical Validation and Performance Impact

Resource sharing is validated experimentally in (Jatala et al., 2015):

• Register Sharing: Maximum improvement of 24%, average improvement of 11% across benchmark suites.
• Scratchpad Sharing: Maximum improvement of 30%, average improvement of 12.5%.

On algorithmic side (Haque et al., 2014):

• Polynomial division: $s = 512$ yields $\sim 2 \times$ speedup, consistent with occupancy and overhead model predictions.
• Radix sort: Overhead minimized when $2^{s} \approx \Theta(\ell)$; $s \approx \log_{2} \ell$ matches empirical optimal.
• Euclidean–gcd: $s \gg 1$ (up to 512) halves running time.
• Polynomial multiplication: Small $s$ yields optimal performance, as predicted.

Empirical results underscore the criticality of SM resource management and analytical modeling for maximizing throughput and utilization.

7. Significance and Outlook

SMs are central to the architectural and algorithmic efficiency of GPUs. Their hardware limits, resource fragmentation, and mechanisms for resource sharing fundamentally shape achievable occupancy, utilization, and program speedup. Analytical approaches such as the MMM enable principled algorithm design—predicting tradeoffs, guiding parameter selection, and optimizing for minimal overhead and maximal parallel throughput. Future architectural refinements may integrate finer-grained resource allocation, dynamic sharing, and enhanced hardware support for block pairing and synchronization, further improving SM scalability and efficiency.