StreamDQ: Near-Memory Weight DeQuantization in Custom HBM for Scalable AI Inference Acceleration
Abstract: As LLMs scale, their memory and computation demands have grown substantially, making weight-only quantization a widely adopted technique for reducing model size with minimal accuracy loss. However, on current GPUs, CUDA-core-based dequantization introduces substantial instruction overhead, on-chip traffic, and pipeline stalls, making it a major bottleneck for high-throughput, cloud-scale LLM serving. To address these limitations, we propose StreamDQ, a lightweight architectural enhancement that enables on-the-fly dequantization in the memory subsystem for high-throughput, large-batch LLM inference. StreamDQ integrates compact DeQuantization Blocks (DQBs) into the base die of high-bandwidth memory (HBM) and performs inline dequantization on standard memory loads. A lightweight sideband tag on each memory read request selects the dequantization mode while preserving conventional load semantics. By relocating dequantization to the memory side, StreamDQ eliminates GPU-side CUDA-core-based dequantization, thereby reducing on-chip traffic on the GPU and avoiding extra HBM write-back and reload of dequantized weights at large batch sizes. Our evaluation shows that StreamDQ achieves up to 7.08 speedup and 90.23\% lower energy for mixed-precision GEMM, with only 0.127\,mm area and 0.355\,W power overhead per DQB in a 12\,nm CMOS process. For end-to-end LLM inference, StreamDQ reduces latency by up to 54.68\% and improves decode throughput by up to 2.20.
Summary
- The paper introduces near-memory dequantization blocks in HBM, reducing GPU dequantization overhead, which can account for up to 80% of latency.
- It achieves up to 7.08× speedup and 90.23% lower energy consumption by offloading conversion tasks from CUDA cores.
- The modular design scales linearly with HBM channels and supports multiple quantization formats for large language models.
StreamDQ: Near-Memory Weight DeQuantization in Custom HBM for Scalable AI Inference Acceleration
Motivation and Context
As LLMs scale, weight-only quantization has become a standard technique for reducing inference memory and bandwidth demands with minimal accuracy loss. However, GPUs lack native mixed-precision GEMM for quantized weights and full-precision activations, resulting in significant dequantization overhead, instruction pressure, pipeline stalls, and inefficient utilization of tensor cores during large-batch inference. Dequantization often accounts for up to 80% of total latency for practical batch sizes, a bottleneck highlighted in the paper via extensive profiling.
Figure 1: Dequantization overhead in LLaMA-3.1-8B-Instruct (W4A16) inference and reduced tensor-core utilization in weight-only quantization workflows.
StreamDQ Design and Workflow
StreamDQ introduces near-memory processing by integrating DeQuantization Blocks (DQBs) directly into the HBM base die. This architectural enhancement enables inline, on-the-fly dequantization on memory loads via a lightweight sideband tag, preserving standard load semantics and minimizing GPU-side changes. Per-load tagging is performed via a region lookup table managed by the GPU runtime, with negligible overhead.
Figure 2: Comparison of conventional GPU-based mpGEMM workflow (a) vs. the StreamDQ-based workflow (b) highlighting HBM-resident dequantization.
StreamDQ leverages per-group quantization, commonly used in LLMs for accuracy/metadata tradeoffs, and co-locates quantized weight groups and their metadata within each HBM pseudo-channel to avoid cross-channel communication.
Figure 3: Per-tensor, per-channel, and per-group granularity schemes for weight quantization.
Figure 4: Illustration of pseudo-channel-aware layout in StreamDQ, ensuring co-location of weights and metadata for efficient dequantization.
DQB Microarchitecture and Data Flow
Each DQB consists of (i) a tag parser, (ii) an request generator with an on-chip buffer, and (iii) a data processing pipeline for type conversion and dequantization. The data processing unit uses shared FP32 ALUs and lightweight wire-mapping for efficient FP conversion without ALUs or shifters, and LUT-based logic for integer-to-float conversion with zero-padding optimizations. The DQB supports multiple format pairs (e.g., INT4→FP16, INT8→BF16).
Figure 5: Microarchitecture schematic for the DQB, performing in-situ dequantization on incoming memory loads.
Figure 6: Data processing block within DQB, illustrating bypass and dequantization pipeline for FP8-to-FP16 flow.
Figure 7: Area-efficient wire-mapping mechanism for FP-to-FP type conversion in DQB.
Figure 8: INT-to-FP conversion utilizing shared LUTs and zero-padding, minimizing logic and buffer requirements.
Scalability is achieved via deployment of one DQB per HBM pseudo-channel, with performance scaling linearly as HBM channel count increases.
Figure 9: Interleaving-aware deployment of DQBs across HBM pseudo-channels (one per memory controller read path).
Overcoming GPU Near-Memory Processing Barriers
StreamDQ confronts standard NMP challenges: address translation is simplified by exclusively operating on physical addresses post-MMU translation, and data placement is solved by ensuring pseudo-channel-local access to all operand data. This eliminates the need for invasive MMU interface and cross-channel interconnects, removing barriers that have previously limited NMP adoption in GPU-serving contexts.
Figure 10: StreamDQ’s approach to address translation and operand placement in NMP architectures, sidestepping cross-channel and MMU complexities.
Area, Power, and Thermal Feasibility
Each DQB occupies only 0.127 mm², and aggregate area overhead is 3.36% of the HBM base die. Power dissipation per DQB is 0.355 W at a 20% input toggle rate, and thermal simulations confirm that DQB activity introduces only minor, localized heating—remaining well within DRAM retention constraints.
Figure 11: HBM base-die thermal profiles for GPU-side FP16 GEMM, CUDA-core INT4 dequantization, and StreamDQ base-die dequantization.
Empirical Results: Performance and Energy
In mixed-precision GEMM benchmarks, StreamDQ achieves up to 7.08× speedup and 90.23% lower energy compared to the best software-fused dequantization kernels at large batch sizes. The improvements are pronounced in compute-bound regimes, as DQB-based dequantization eliminates the CUDA-core bottleneck and avoids redundant HBM write/reload cycles required by split kernels. In end-to-end LLM inference, StreamDQ reduces latency by up to 54.68% and increases decode throughput by 2.20×.
Implications and Theoretical Impact
StreamDQ provides an unobtrusive, hardware-compatible mechanism for scalable weight-only quantized LLM inference, mitigating the major pipeline bottlenecks induced by CUDA-core/software dequantization. The system supports modular format expansion, minimal thermal and area footprint, and addresses integration challenges inherent to NMP in GPU stacks. The architecture aligns with projected advances in logic-enabled HBM base dies and can be deployed with negligible software and hardware modifications. Practically, this solution enables sustained high-throughput inference for multi-billion parameter LLMs under stringent power budgets.
Theoretically, StreamDQ reframes the system bottleneck for future LLM accelerators: achieving true scalability will now depend on the advancement of tensor-core GEMM throughput and memory bandwidth, rather than dequantization instruction overhead. The approach can be generalized to other NMP workloads where large-volume, static operand transformation is needed.
Conclusion
StreamDQ decentralizes dequantization from GPU compute cores to memory-side blocks within custom HBM, providing substantial inference acceleration, energy reduction, and robustness for weight-only quantized LLMs. The results and architectural innovations indicate that near-memory processing, when carefully engineered to operate on static operands and metadata localization, offers a viable path to overcoming inference throughput barriers in next-generation deep learning systems (2607.08993).
Paper Prompts
Sign up for free to create and run prompts on this paper.
Top Community Prompts
Explain it Like I'm 14
What is this paper about?
This paper introduces StreamDQ, a new way to make big AI models (like chatbots and translators) run faster and use less energy. It focuses on a step called “dequantization,” which is like unzipping compressed model weights. Instead of unzipping on the GPU (the main calculator), StreamDQ moves that unzipping into the memory chips themselves. That way, the GPU gets “ready-to-use” data and can spend more time doing the heavy math that creates answers.
What questions were the researchers trying to answer?
- Can we speed up AI inference (the process of generating answers) by moving dequantization closer to where data lives—in memory—so the GPU doesn’t waste time and energy doing it?
- Can we do this with small, practical hardware changes that work with today’s systems and software?
- Will this approach be fast, energy-efficient, and safe to put inside modern high-bandwidth memory (HBM) chips without overheating?
How did they try to solve it? (Simple explanation with an analogy)
Think of running a large AI model like cooking a big meal:
- The model’s weights are ingredients stored in a pantry (the memory).
- Quantization is compressing those ingredients to take up less space (like vacuum-packing).
- Dequantization is uncompressing them before cooking.
- Today, the “unpacking” happens in the kitchen (the GPU), which slows down the chef.
StreamDQ’s idea: add tiny helpers (“DeQuantization Blocks,” or DQBs) at the pantry exits (inside the HBM memory base die). These helpers:
- Automatically unpack (dequantize) the compressed ingredients as they leave the pantry.
- Put a small “sticky note” (a few tag bits) on each memory request telling the pantry whether to unpack and into which format (for example, INT4 to FP16).
- Keep the important labels (scale and zero-point—small numbers needed to unpack correctly) in the same aisle as the ingredients, so the helper can reach everything it needs without running around the store.
Key elements of the approach:
- Tiny hardware blocks (DQBs) are added to each memory lane (called a “pseudo-channel”) inside HBM. Each DQB does fast, on-the-fly dequantization.
- A small tag attached to each memory read tells the DQB how to convert (e.g., INT4→FP16 or INT8→BF16). No changes to the program’s addresses or normal load/store behavior are needed.
- The weights and their “unpacking instructions” (scale S and zero-point Z) are placed together in memory so each DQB can do its job locally without extra communication.
- This avoids big changes to the GPU and doesn’t require new special math units—it simply gives the GPU already-unpacked data so its tensor cores (the fast math engines) can stay busy.
What did they find, and why is it important?
Main results (why this matters):
- Much faster core math: Up to 7.08× speedup for the main matrix multiplications (GEMM) that dominate AI workload time.
- Big energy savings: Up to 90.23% lower energy for those same math operations.
- Better overall AI performance: Up to 54.68% lower end-to-end latency (answers come sooner) and up to 2.20× higher decode throughput (more tokens per second).
- Small hardware cost: Each DQB uses only about 0.127 mm² of chip area and about 0.355 W of power (in a 12 nm process). Even across a whole memory stack, the added area and power are modest.
- Thermally safe: Adding this logic to the HBM base die doesn’t overheat the memory and stays within safe limits.
Why it’s important:
- Today’s GPUs often stall because they spend a lot of time unzipping weights before they can do math. StreamDQ moves that work into the memory, so the GPU does more useful computation and less waiting.
- It reduces on-chip traffic and avoids repeatedly writing and re-reading dequantized data for large batches.
- It supports multiple formats (like INT4→FP16 and INT8→BF16), making it flexible for different models and setups.
What’s the bigger impact?
- Faster, cheaper AI serving: Cloud providers can serve more users and bigger models with the same hardware, at lower energy costs.
- Easier deployment: It doesn’t require heavy changes to GPUs or complicated new instructions. Existing software can work with modest runtime support (like adding the small tags and using a simple lookup table).
- Future-proof: As models grow and memory bandwidth rises, StreamDQ scales by adding more small helpers per memory lane, keeping up with demand.
- Practical near-memory processing: It solves common “near-memory” challenges (like address translation and data placement) by keeping everything local and using standard memory access paths.
In short, StreamDQ is like moving the unboxing from the kitchen to the pantry door, so the chef can cook faster. It makes large AI models run quicker and more efficiently without rewriting the kitchen—or the recipes.
Knowledge Gaps
Unresolved gaps, limitations, and open questions
Below is a consolidated list of concrete gaps, limitations, and open questions that remain unresolved and could guide follow-on research and engineering.
- System semantics and caching
- Define and validate cache semantics when the memory holds quantized weights but the GPU receives dequantized data: should L1/L2 cache store dequantized or quantized lines, how is coherence preserved across loads/stores, and how does this interact with cache compression and prefetchers?
- Specify whether tagged loads should bypass certain cache levels to avoid semantic mismatches and quantify the performance impact of any cache-bypass policy.
- Detail interactions with page coloring/placement and address hashing schemes that affect pseudo-channel (PC) mapping after the proposed offline layout transform.
- Sideband tagging and protocol integration
- Clarify how the few-bit sideband tag is carried over existing HBM/D2D PHY protocols (JEDEC compliance, vendor differences), and outline a standardization path for cross-vendor interoperability.
- Quantify the latency/throughput cost of per-load region table lookups in the SMs, especially under high MLP and multi-tenant scenarios, and measure the effect of TLB misses and page faults on tag generation correctness.
- Define fail-safe behavior and error reporting when a tag is stale, missing, or mismatched (e.g., tagged load to unquantized region).
- Data movement and bandwidth trade-offs
- Characterize D2D link and on-chip interconnect pressure when returning expanded dequantized data versus sending compact quantized data to the GPU (especially for W4→FP16), across prefill/decode phases and batch sizes.
- Quantify when StreamDQ transitions from beneficial to bandwidth-limited (e.g., link saturation points, interaction with concurrent memory traffic, and multi-stream serving).
- Compare end-to-end HBM traffic with and without fused kernels under large batches, explicitly accounting for eliminated write-backs/reloads versus increased return-path bandwidth.
- Numerical behavior and model accuracy
- Evaluate accuracy impacts of the proposed wire-mapped FP conversions (round-toward-zero, flush-to-zero) versus standard GPU math libraries across representative LLMs and calibration datasets.
- Justify the chosen precision (e.g., 16-bit scale, zero-point bit-width tied to weight bits) and assess whether certain layers benefit from higher-precision scales or dynamic range adjustments.
- Provide correctness proofs or golden-model comparisons for all supported conversions (INT4/INT8/FP8→FP16/BF16), including edge cases (NaNs, infs, denormals).
- Quantization scheme coverage and flexibility
- Extend or specify support for non-uniform and codebook-based schemes (e.g., NF4), per-channel/per-tensor quantization, mixed per-layer formats, and layer-specific rounding/saturation modes.
- Analyze implications of group-size/stride choices that are not multiples of interleave granularity (IG) on burst efficiency, alignment, and dequantization latency.
- Describe a path to support future formats (e.g., FP4, INT2, MX formats) within the 3-bit tag budget or propose a scalable control-plane design for format extensibility.
- S/Z metadata handling
- Report measured buffer hit rates under real LLM access patterns, sensitivity to group size, and worst-case miss penalties; size the buffer based on empirical evidence.
- Quantify how fetches interfere with weight reads at the pseudo-channel MC, their impact on DRAM scheduling fairness, and back-pressure scenarios when arrives late.
- Validate that replication overhead remains negligible at scale (e.g., larger models, different IGs, non-ideal mappings) and analyze fragmentation or alignment overhead introduced by the pseudo-channel-aware layout.
- Software/runtime and multi-tenancy
- Measure the overhead of initializing and updating the region-lookup table and DQB request tables in real serving systems with model multiplexing, preemption, MIG, and frequent context switches.
- Define isolation mechanisms so per-process tags/tables cannot corrupt or snoop another tenant’s memory; formalize privilege boundaries for tag/table programming.
- Provide a deployment toolchain for the offline pseudo-channel-aware data layout, including converters, verification tools, and compatibility with common model distribution formats.
- Training/fine-tuning and dynamism
- Address how StreamDQ behaves with fine-tuning, adapters (LoRA), or online weight updates: how are metadata maintained or updated, and how are stores handled if quantized weights must be modified?
- Explore support for dynamic re-quantization or adaptive per-layer formats at runtime, including tag updates and DQB table changes with minimal service interruption.
- Reliability, safety, and observability
- Specify interactions with DRAM/D2D ECC: what happens if or weights experience correctable/uncorrectable errors inside the DQB pipeline, and how are errors surfaced to software?
- Provide DFT/BIST strategies for DQB logic in the base die, including methods for in-field diagnostics, telemetry, and performance counters to observe dequantization behavior.
- Study soft-error susceptibility and silent data corruption risks in buffers and LUTs, and evaluate low-overhead mitigation (e.g., parity/ECC).
- Microarchitecture and timing
- Quantify the additional read-path latency introduced by DQB processing (including tag parsing, coordination, conversion, and potential clock-domain crossings) and its effect on SM scheduling and tensor-core utilization.
- Detail pipeline depth and buffering requirements to hide misses and ensure weights arrive “just-in-time” without stalling GEMM kernels.
- Explore arbitration policies between weight and requests and their impact on QoS under high concurrency across pseudo-channels.
- Thermal and physical feasibility beyond modeling
- Validate thermal results with more detailed coupled electro-thermal models (including TSV self-heating, inter-die interfaces, varying ambient conditions), and perform sensitivity analyses across workloads.
- Investigate signal integrity, timing closure, IR drop, and power delivery network implications of integrating many DQBs on the base die, especially for next-node HBM3e/HBM4 stacks.
- Assess yield and manufacturability risks of adding logic to the base die, and outline test coverage and cost implications for high-volume production.
- Evaluation methodology and generality
- Reduce uncertainty in performance/energy projections by validating StreamDQ-Sim against real mpGEMM kernels (not only fpGEMM-derived traces), and report sensitivity to simulator calibration error (noted MAPEs up to ~28%).
- Extend evaluations to larger models (≥70B), diverse architectures, and extreme batch/sequence length regimes, including latency-sensitive small-batch inference and streaming workloads.
- Benchmark against newer GPUs (e.g., Hopper/Blackwell) with FP8/TMA and emerging INT4 tensor-core paths to position StreamDQ relative to rapidly evolving native low-precision support.
- Ecosystem interoperability and longevity
- Analyze how StreamDQ coexists with future FP/INT tensor-core support for weight-only GEMM (e.g., INT4×FP16 native) and whether near-memory dequantization remains beneficial or becomes redundant.
- Characterize interactions with sparsity (n:m or unstructured) and compression techniques commonly used in LLM inference, and specify DQB extensions needed to exploit combined benefits.
- Provide a migration path for existing quantization toolchains (GPTQ, AWQ variants, TorchAO) to emit artifacts compatible with the pseudo-channel-aware layout and DQB format constraints.
- Corner cases and failure handling
- Define behavior when metadata are missing, malformed, or inconsistent with weight layout (e.g., version mismatches), including safe abort and recovery mechanisms.
- Specify ordering/consistency guarantees for interleaved tagged and untagged loads to the same regions and how this interacts with memory fences and GPU memory models.
Practical Applications
Immediate Applications
The paper’s findings and methods enable several actions that can be deployed on today’s hardware/software stacks, as well as concrete planning and workflow changes that yield measurable benefits now.
- Quantized-weight packing to improve current GPU kernels (software; sectors: software, cloud/AI, energy)
- Action: Reorganize W-only-quantized weight tensors and per-group S/Z metadata to be co-located and contiguous in memory (pseudo-channel–aware in spirit, channel/cache-line–aware in practice), improving cache locality and reducing extra memory transactions for CUDA-core dequantization in fused kernels.
- Where it fits: Integrate as a “HBM-/channel-aware packer” pass in quantization toolchains and serving frameworks (e.g., vLLM, TensorRT-LLM, PyTorch/TorchAO/LLM.int8/AWQ/GPTQ exporters).
- Expected impact: Moderately lower kernel stalls and HBM transactions in W4A16/W8A16 inference; slight throughput and energy gains without hardware changes.
- Dependencies/assumptions: Access to quantized weights and S/Z metadata layouts at export time; knowledge or empirical profiling of the platform’s interleaving, cache line size, and burst length; no accuracy loss from layout changes.
- Batch-size–aware kernel selection and scheduling (software/MLOps; sectors: software, cloud/AI, finance, healthcare, education)
- Action: Use the paper’s characterization (dequantization becomes the bottleneck at large batches; fused kernels help mainly in memory-bound/small-batch regimes) to set policy rules:
- Small batch: fused dequant+GEMM kernels to cut HBM traffic.
- Large batch: separate kernels with tuned staging to maximize tensor-core utilization and avoid CUDA-core stalls.
- Expected impact: Lower inference latency (10–30% in practice for many deployments) and higher throughput without model/accuracy changes.
- Dependencies/assumptions: Existing fused and split kernels available; accurate profiling (Nsight Systems/Compute) to detect regime switches; scheduler support in serving frameworks.
- Format selection and capacity planning using the paper’s benchmarks (ops/cost planning; sectors: cloud/AI, energy, finance)
- Action: Choose quantization formats (e.g., INT4→FP16 vs INT8→BF16) per-model/per-layer using measured dequant overheads and expected accuracy budgets; prioritize configurations that yield best latency/cost/energy tradeoff.
- Tools/workflow: Add a “format sweep” step to model onboarding; use vendor profilers to quantify dequant share and tensor-core utilization.
- Dependencies/assumptions: Validated accuracy for chosen quantization; availability of multiple kernel implementations.
- Early hardware-roadmap engagement and RFI/RFQ criteria (procurement/standardization; sectors: semiconductor, cloud/AI, policy/ESG)
- Action: Incorporate “near-memory dequantization capability” and “sideband tag support” as forward-looking requirements in accelerator and HBM stack RFIs; request data on logic base-die power/area budgets consistent with the paper’s DQB feasibility.
- Expected impact: Aligns vendors and hyperscalers on a near-term path to StreamDQ-like features; accelerates ecosystem readiness.
- Dependencies/assumptions: Vendor willingness to expose roadmap and collaborate; alignment with JEDEC/HBM4+ custom base-die practices.
- Curriculum and research prototyping using StreamDQ’s methodology (academia; sectors: education/research)
- Action: Use the paper’s system model (sideband-tag control, weight/SZ locality, metadata buffering) to replicate prototypes on FPGA/accel simulators; teach NMP trade-offs (address translation avoidance, thermal limits).
- Dependencies/assumptions: Availability of comparable simulators (Accel-Sim, AccelWattch) and public GPU metrics; no proprietary IP required for conceptual replication.
- Sustainability accounting and policy guidance with dequantization metrics (policy/ops; sectors: energy, cloud/AI)
- Action: Include “dequantization energy share” as a reported KPI in AI service energy dashboards; guide model placement and scheduling to reduce dequant-heavy workloads during peak energy-cost windows.
- Dependencies/assumptions: Telemetry hooks for kernel-level energy/latency; organizational willingness to track and act on new KPIs.
Long-Term Applications
The core innovation—near-memory dequantization via DQBs in the HBM base die with sideband-tag control—requires hardware support and ecosystem integration. These applications become viable as custom HBM base dies and minimal GPU support mature.
- StreamDQ-enabled HBM stacks as a product class (hardware/IP; sectors: semiconductor, cloud/AI, energy)
- Product: “HBM with near-memory dequantization” (per-pseudo-channel DQB IP, 3–4% base-die area, ~0.35 W/DQB in 12 nm; supports INT4/INT8/FP8→FP16/BF16).
- Value: Up to 7.08× mpGEMM speedup and ~90% lower energy for dequant+GEMM; up to 54.7% latency reduction and 2.20× decode throughput in LLM serving; datacenter energy and TCO reductions.
- Dependencies/assumptions: Adoption of logic-capable HBM base dies (HBM4-era custom HBM); thermal validation and DRAM retention margins; DQB implementation in advanced nodes lowers power density further.
- Minimal GPU/SoC support for sideband tags and region lookup (hardware+driver; sectors: semiconductor, software)
- Product/workflow:
- On-GPU region-lookup table (few KB) to attach few-bit tags to loads in registered quantized-weight VA ranges.
- Driver/runtime APIs to program regions at model load time.
- Value: Preserves conventional load semantics; backward compatible with software; avoids intrusive ISA/datapath changes.
- Dependencies/assumptions: Firmware/driver changes; spare metadata bits or protocol extensions on request path; security/isolation review for tag propagation.
- Serving framework integration for near-memory dequantization (software; sectors: software, cloud/AI, healthcare, finance, education)
- Tools/workflow:
- Exporters that emit pseudo-channel–aware layouts and per-layer S/Z request tables.
- Runtime modules to program GPU region tables, manage S/Z metadata, and pick quant formats per layer.
- Value: Seamless use of StreamDQ-enabled HBM with existing tensor-core GEMMs; improved tensor-core utilization; reduced HBM traffic in large-batch regimes.
- Dependencies/assumptions: Frameworks (vLLM, TensorRT-LLM, PyTorch) accept new exporters/runtime hooks; per-layer metadata management standardized.
- Sector-specific deployments enabled by lower latency/cost (application; sectors: healthcare, education, finance, customer support)
- Use cases:
- Healthcare: Faster, cheaper on-prem LLM inference for clinical summarization and coding while keeping PHI on-site.
- Education: Low-latency AI tutoring and feedback at scale for school districts with constrained budgets.
- Finance/contact centers: Higher-concurrency assistants and agent-copilots with improved response times and cost per interaction.
- Dependencies/assumptions: Availability of StreamDQ hardware SKUs; validated accuracy for W4/W8 weight-only quantization under sector-specific QA; compliance and privacy constraints.
- Extensions to additional near-memory primitives (hardware research; sectors: semiconductor, software, robotics/edge in future)
- Directions:
- Near-memory INT→FP activation conversion or lightweight pre-processing (e.g., scaling for LayerNorm weights), reducing CUDA-core overhead further.
- Embedding-table dequantization and gather offload for recommender and RAG workloads.
- Dependencies/assumptions: Area/power headroom in base die; clear memory-local operands to avoid cross-channel communication; evaluation of thermal limits.
- Standardization and ecosystem enablement (standards/policy; sectors: semiconductor, policy/ESG)
- Actions:
- JEDEC/HBM working groups define optional sideband-tag fields and logic base-die capability profiles.
- Procurement standards include “near-memory compute for dequantization” as an energy-efficiency feature.
- Dependencies/assumptions: Multi-vendor consensus; security and reliability guidelines for logic in base dies.
- Sustainability outcomes and incentives (policy; sectors: energy, cloud/AI)
- Application: Use measured system-level gains (e.g., >50% latency, large energy savings for dequant-heavy workloads) to qualify for green-compute incentives and reduce Scope 2 emissions of AI services.
- Dependencies/assumptions: Transparent reporting; third-party verification; grid/utility programs that recognize fine-grained accelerator efficiency features.
- Adaptation to edge/robotics accelerators (longer-horizon; sectors: robotics, mobile)
- Concept: Apply the StreamDQ principle to LPDDR/stacked memory with logic base-die or in-package NMP for on-device LLMs, reducing power and bandwidth for edge inference.
- Dependencies/assumptions: Memory technology with logic-capable base or companion die; tight power/thermal envelopes; simplified tag/region mechanism for embedded ISAs.
- Security, QoS, and multi-tenant resource control for near-memory compute (software/hardware mgmt; sectors: cloud/AI)
- Tools: Telemetry and controls to rate-limit DQB bandwidth per tenant, isolate S/Z buffers, and audit sideband tag usage.
- Dependencies/assumptions: Vendor-exposed counters and controls; hypervisor/driver policy extensions; formal verification for tag handling.
These applications collectively map the paper’s innovations—from near-memory dequantization, sideband tagging, and pseudo-channel-aware layouts to practical deployment pathways—and identify the hardware, software, and ecosystem dependencies that govern feasibility and timing.
Glossary
- Accel-Sim: A GPU microarchitecture simulator used to evaluate performance. "Performance and power are modeled with Accel-Sim and AccelWattch, respectively."
- AccelWattch: A power modeling framework for GPUs integrated with Accel-Sim. "Performance and power are modeled with Accel-Sim and AccelWattch, respectively."
- ALU: Arithmetic Logic Unit; hardware that performs arithmetic and logical operations. "The dequantization arithmetic reuses shared FP32 ALUs across the supported input formats"
- AWQ: Activation-aware Weight Quantization; a family of weight-only quantization methods and kernels. "AWQ-v2 is the latest fused kernel released by the AWQ authors"
- Base die: The logic die at the bottom of an HBM stack that hosts controllers and can integrate compute logic. "integrates compact DeQuantization Blocks (DQBs) into the base die of high-bandwidth memory (HBM)"
- BF16: bfloat16; a 16-bit floating-point format with an 8-bit exponent used for efficient ML computation. "supports multiple quantized-to-dequantized format conversions (e.g., INT4→FP16 and INT8→BF16)"
- CUDA core: A general-purpose scalar/SIMT execution unit on NVIDIA GPUs used for non-tensor operations. "CUDA-core-based dequantization introduces substantial instruction overhead"
- Custom HBM (C-HBM): High-Bandwidth Memory with a logic-capable base die tailored for additional in-memory features. "StreamDQ is designed as a practical extension to custom HBM (C-HBM) systems"
- D2D PHY: Die-to-die physical interface linking the GPU/SoC and the HBM stack. "interfaces with the GPU/SoC die through the standard D2D PHY link"
- Decode throughput: The rate (e.g., tokens/s) at which an LLM produces output tokens during decoding. "improves decode throughput by up to 2.20×"
- Dequantization: The process of reconstructing higher-precision values from quantized representations using scale and zero-point. "dequantization accounts for up to 40--80% of total inference latency"
- DeQuantization Block (DQB): A near-memory hardware block that performs on-the-fly dequantization on HBM read paths. "StreamDQ integrates compact DeQuantization Blocks (DQBs) into the base die"
- FloTHERM: A thermal simulation tool for electronic systems used to assess heat and reliability constraints. "We evaluated the thermal feasibility of StreamDQ using FloTHERM"
- FP16: IEEE half-precision floating-point format commonly used in tensor-core GEMM. "tensor-core utilization of FP16 fpGEMM"
- FP32: IEEE single-precision floating-point format used as high-precision baseline. "maps high-precision floating-point data (e.g., FP32) into compact integer"
- FP8: An 8-bit floating-point format (e.g., E4M3/E5M2) used for compact representation. "e.g., INT4 or FP8"
- fpGEMM: Full-precision general matrix multiplication, typically using FP16/FP32 on tensor cores. "tensor-core utilization of FP16 fpGEMM"
- GEMM: General Matrix Multiplication; a core linear algebra operation dominating transformer workloads. "general matrix multiplication (GEMM) operations"
- HBM: High-Bandwidth Memory; a 3D-stacked DRAM providing very high memory bandwidth. "high-bandwidth memory (HBM)"
- Interleaving granularity (IG): The data block size at which memory accesses are striped across pseudo-channels. "interleaving granularities ()"
- Lookup table (LUT): A precomputed mapping used to accelerate conversions or operations by table lookup. "shared LUT-based INT→FP conversion structure"
- Memory controller (MC): The unit managing memory request scheduling and DRAM protocol within the HBM base die. "memory controller (MC)"
- Mixed-precision GEMM (mpGEMM): GEMM where operands use different precisions (e.g., INT4×FP16) to save bandwidth/compute. "weight-only mixed-precision GEMM (mpGEMM)"
- MMU: Memory Management Unit; translates virtual addresses to physical addresses. "the command-set address () is translated by the GPU MMU"
- Near-memory processing (NMP): Performing computation close to memory to reduce data movement. "near-memory processing (NMP)"
- Nsight Compute: NVIDIA’s kernel-level profiler for performance analysis on GPUs. "collected with Nsight Compute"
- Nsight Systems: NVIDIA’s system-level profiler for tracing and analyzing end-to-end performance. "Using Nsight Systems, we profile the mpGEMM fraction"
- NVBit: NVIDIA Binary Instrumentation Tool for collecting traces and instrumenting GPU binaries. "NVBit-based Accel-Sim tracer"
- NVML: NVIDIA Management Library for monitoring and controlling GPU power/telemetry. "NVML"
- Per-channel quantization: A scheme assigning separate scale/zero-point per output channel for better accuracy. "Per-Channel Quantization."
- Per-group quantization: A scheme assigning one scale/zero-point to fixed-size groups within each channel. "Per-Group Quantization."
- Per-tensor quantization: A scheme using one scale/zero-point for the entire weight tensor. "Per-Tensor Quantization."
- Pseudo-channel: A subdivision of an HBM channel enabling finer-grained parallelism and control. "on the read path of each pseudo-channel memory controller (MC)"
- Pseudo-channel-aware layout: A data placement strategy that co-locates weight groups and their S/Z metadata within the same pseudo-channel. "pseudo-channel-aware layout"
- Scaling factor (S): The multiplicative factor used to rescale quantized values during dequantization. "scaling factor ()"
- Sideband tag: A few control bits attached to memory requests to select processing modes without changing addresses. "A lightweight sideband tag on each memory read request selects the dequantization mode"
- Tensor core: Specialized matrix-multiply-accumulate units delivering high throughput for low/mid precision. "tensor-core GEMM"
- Through-silicon via (TSV): Vertical electrical interconnects used in 3D-stacked chips like HBM. "through the memory controller (MC) and TSVs."
- VA-to-PA translation: Virtual-address to physical-address translation required for memory accesses. "near-memory VA-to-PA (virtual address-to-physical address) translation overhead"
- vLLM: A high-throughput LLM inference framework optimized for efficient serving. "frameworks such as vLLM fuse dequantization and GEMM into a single kernel using shared memory."
- Weight-only quantization: Quantization approach that reduces weight precision while keeping activations high-precision. "weight-only quantization"
- Wire-mapping: A bit-level remapping technique for FP-to-FP conversion without ALUs or shifters. "wire-mapping datapath"
- Zero-point (Z): The additive offset used in quantization/dequantization to represent signed ranges. "zero-point offset ()"
