---
title: Dynamic Bit-Sliced Caching for MoE Models
url: https://www.emergentmind.com/topics/dynamic-bit-sliced-caching-dbsc
type: topic
---

# Dynamic Bit-Sliced Caching for MoE Models

Dynamic Bit-Sliced Caching (DBSC) is a cache management scheme for on-device inference with large Mixture-of-Experts (MoE) models under stringent miss-rate constraints. DBSC operates at the granularity of quantized bit-slices of expert weight tensors, caching the most critical precision slices to maximize effective expert capacity and reduce cache miss penalties. Integrated with a precision-on-demand mechanism and specialized quantization, DBSC enables energy- and latency-efficient deployment of MoE models within limited DRAM budgets, dramatically reducing high-latency Flash accesses while preserving near-maximal inference accuracy [2512.12990].

## 1. Problem Setting and Motivation

Large-scale MoE language models feature tens of billions of expert parameters, often exceeding the few gigabytes available in on-device DRAM. Standard deployments partition experts between DRAM (for fast, low-energy access) and Flash storage (10–100× slower, 50–100× more energy per bit). Even moderate cache miss rates (e.g., 10–30%) incur significant energy and latency penalties, rapidly dominating inference costs and rendering on-device serving impractical without more sophisticated cache control. For practical inference, the instantaneous cache miss rate
\[
M = \frac{\#\text{misses}}{\#\text{hits}+\#\text{misses}} = 1 - \frac{H}{H+M}
\]
must typically remain below 5%.

## 2. Bit-Slice Caching Principles and Workflow

Each expert's quantized weight tensor is partitioned into two bit-slices: a most significant bits (MSB) slice of $b_h$ bits and a least significant bits (LSB) slice of $b_l$ bits. Caching only the MSB slice permits a low-bit approximation of expert weights, sufficient for non-critical experts, while a subset of experts can be recombined with the LSB slice for full precision. This slice-level approach enables more fine-grained use of cache, increasing the number of distinct expert representations resident in DRAM and boosting the cache hit probability within a strict memory budget.

### Memory Footprint Formulation

Given hidden dimension $d$ and $n_e$ experts, the DRAM footprint of a single $b_s$-bit slice is:
\[
F_s = b_s \times d \times n_e \quad (\text{bits})
\]
If a fraction $p_h$ of cached experts retain high precision and $1-p_h$ retain low precision, the average bits per expert $\bar b$ and effective expert capacity $N_{\rm eff}$ with DRAM budget $C$ are:
\[
\bar b = p_h b_h + (1-p_h) b_l,
\qquad
N_{\rm eff} = \frac{C}{d \bar b}
\]

### Slice-Level Cache Management Algorithm

The DBSC eviction and admission protocol operates as follows:
- MSB slices are managed via standard LRU.
- LSB slices have lowest priority and are evicted first when capacity pressure arises.
- After each batch, if the global miss rate exceeds target, low-priority slices are evicted until the miss rate is satisfactory.

Sample pseudocode:
```python
procedure DBSC_STEP(requested_experts, cache, C_max, miss_rate_target):
    for e in requested_experts:
        if cache.contains(e, slice='MSB'):
            record_hit(e,'MSB')
        else:
            cache.load(e,'MSB')
            record_miss(e,'MSB')
            evict_if_needed(cache, C_max)
        if requires_high_precision(e):
            if cache.contains(e, slice='LSB'):
                record_hit(e,'LSB')
            else:
                cache.load(e,'LSB')
                record_miss(e,'LSB')
                evict_if_needed(cache, C_max)
    current_M = compute_miss_rate()
    if current_M > miss_rate_target:
        evict_least_valuable(cache, until_M≤miss_rate_target)
```

## 3. Precision-On-Demand Mechanism

DBSC leverages the typically steep distribution of gating scores by assigning precision dynamically on a per-token basis:
- All selected experts fetch their MSB slice.
- Only experts whose gating score surpasses a token-specific threshold fetch the LSB slice, enabling full precision.
The optimization objective is to maximize expected accuracy $A$ subject to memory $C$ and miss-rate constraint $M \leq M_0$:
\[
\max_{p_h}\;A\bigl(p_h\,b_h + (1-p_h)\,b_l\bigr)
\quad
\text{s.t.}\;\;
p_h\,F_{h} + (1-p_h)\,F_{l}\le C,
\;
M\le M_0
\]
This approach ensures that the bulk of experts operate at lower precision, conserving memory and energy, while critical experts maintain full expressiveness.

## 4. Calibration-Free Asymmetric Matryoshka Quantization (AMAT)

To permit seamless mixed-precision expert caching, DBSC employs Calibration-Free Asymmetric Matryoshka Quantization (AMAT). AMAT enables truncation-based extraction of low- and high-bit slices from a single quantized tensor and its zero-point without duplicate storage or additional calibration overhead.

For high-bitwidth $b_{high}$ and low-bitwidth $b_{low}$:
\[
\mathrm{shift} = b_{high} - b_{low}
\]
\[
q_{low} = \bigl\lfloor q_{high} / 2^{\mathrm{shift}}\bigr\rfloor, \quad
zp_{low} = \bigl\lfloor zp_{high} / 2^{\mathrm{shift}}\bigr\rfloor
\]
Bit-slice composition and value dequantization proceed as:
\[
q_{full} =\bigl(q_{MSB} \ll b_{LSB}\bigr) + q_{LSB}
\]
\[
w = \mathrm{scale}_{h}\,\bigl(q_{full}-zp_{h}\bigr)
\]
This construction ensures exact compatibility between low- and high-bit slices, simplifying both cache management and hardware implementation.

## 5. Predictive Cache Warmup

Single-batch inference epochs include a prefill phase (broad expert access via parallelism) and a decode phase (narrow, frequent reuse of a small expert subset). The Predictive Cache Warmup (PCW) mechanism exploits the empirical correlation of “hot” experts between prefill and early decode. PCW records per-slice access counts $c_s$ during prefill, then, at the prefill-to-decode transition:
1. Evicts LSB slices with the smallest $c_s$.
2. Evicts MSB slices in ascending order of $c_s$ until DRAM constraints are met.

Pseudocode:
```python
procedure PCW(cache, prefill_counts, C_decode):
    sort all LSB slices by prefill_counts ascending
    evict top K_LSB until cache.size ≤ C_decode
    sort all MSB slices by prefill_counts ascending
    evict top K_MSB until cache.size ≤ C_decode
```
This preemption creates a cache population closely matched to anticipated decode-stage access patterns, reducing cold misses and access costs.

## 6. Empirical Evaluation and Performance

DBSC and AMAT, as part of SliceMoE, were evaluated on DeepSeek-V2-Lite (160 experts) and Qwen1.5-MoE-A2.7B (240 experts) using a GSM8K 5-shot benchmark. The hardware: XPU (1 GHz, 8192 8-bit PEs), 8 GB LPDDR4 DRAM, and 128 GB UFS 3.1 Flash. Key quantitative results:
- Decode energy reduction by up to 2.37× (DeepSeek-V2-Lite) and 2.85× (Qwen1.5-MoE-A2.7B).
- Decode latency improvement up to 1.81× and 1.64×, respectively.
- DBSC+AMAT achieves accuracy near the high-bit reference along the Pareto frontier for miss rate and energy.
- Predictive Cache Warmup provides an additional up to 2.31× energy reduction and 1.96× speed-up over a cold (empty) cache [2512.12990].

## 7. Deployment Considerations and Future Directions

DBSC requires hardware capable of fetching and combining bit-slices at inference time and introduces more intricate per-slice metadata and LRU tracking. Deployments in extremely tight memory regimes or with low-bitwidth (<2 bits) slices may see accuracy degradation. Gate threshold and prefill–decode correlation parameters may require model-specific tuning.

Prospective research avenues include extending DBSC to multi-slice (beyond MSB/LSB) hierarchies, joint optimization of expert routing and slice allocation, native hardware support for bit-slice streaming, and adaptive on-device learning of cache hotness patterns to suppress cold misses even further [2512.12990].

Source: https://www.emergentmind.com/topics/dynamic-bit-sliced-caching-dbsc