TTP: A Hardware-Efficient Design for Precise Prefetching in Ray Tracing

Abstract: Ray tracing (RT) is a 3D graphics technique that offers highly realistic visuals. It is becoming prominent and accessible as GPU vendors have integrated dedicated ray tracing acceleration hardware. However, tracing millions of rays through 3D scenes consisting of high numbers of triangles in real time is challenging and requires expensive hardware. The main bottleneck in RT workloads is the expensive Bounding Volume Hierarchy (BVH) traversal task, which is a large tree structure that encodes the 3D scene. BVH traversal is a memory-bound problem, as the GPU threads spend most of their time reading tree node data from memory. In this work, we attack the memory latency bottleneck of ray tracing through prefetching. We propose a novel hardware prefetcher, named Tree Traversal Prefetcher (TTP), for ray tracing. The main idea is to leverage the existing tree traversal stack in the RT units for highly accurate prefetching. In particular, TTP prefetches nodes using the addresses already available on the hardware traversal stacks of each thread. For DFS (Depth-first search) based traversal, prefetches are generated when nodes are being popped consecutively from the traversal stack, potentially corresponding to upward traversal through the tree. We evaluate TTP on a cycle-level simulator, Vulkan-sim 2.0, and show that it achieves 1.48x speedup on average (up to 1.89x) compared to the baseline, with nearly negligible hardware overhead. TTP achieves 98.92% average L1 accuracy, which is the ratio of the prefetched blocks being actually referenced by demand loads. The coverage, computed as the ratio of L1 miss reduction over baseline L1 misses, is 31.54%, correlating well with the achieved speedup.

• The paper demonstrates a hardware-efficient prefetching method for ray tracing that leverages traversal stack information to reduce memory latency.
• It employs a per-thread finite-state machine that adjusts prefetching based on stack pop streaks, achieving up to 1.89× speedup and 98.9% L1 prefetch accuracy.
• TTP outperforms competing approaches by maintaining baseline DRAM traffic while reducing cache misses and ensuring scalable performance across diverse scenes.

TTP: A Hardware-Efficient Design for Precise Prefetching in Ray Tracing

Introduction and Motivation

Ray tracing (RT) has established itself as the leading technique for generating physically realistic images, with hardware-accelerated RT units now standard in modern GPUs. Despite advances in computational throughput, the principal bottleneck remains in memory system performance—specifically, the traversal of the Bounding Volume Hierarchy (BVH), which is highly memory-latent and bandwidth inefficient. Existing on-chip caches are easily overwhelmed by the size and irregular access patterns of BVH trees, leading to substantial thread idle time awaiting memory fetches.

Figure 1: Thread status distribution in an RT unit, highlighting the dominance of memory wait states during BVH traversal at modest resolutions.

Analysis of Ray Tracing Memory Behavior

Tracing rays in a BVH involves data-dependent, non-coherent memory accesses, exposing the memory wall as a limiting factor for real-time ray tracing. Empirical analysis of RT workloads demonstrates that memory latency, not bandwidth, is the constraining factor—confirmed by DRAM activity metrics showing underutilized bandwidth and a read-dominated pattern with minimum writeback overhead.

Figure 2: DRAM bandwidth utilization and DRAM read activity with and without TTP, illustrating that memory latency—not bandwidth—is the primary constraint; TTP increases bandwidth utilization by reducing idle cycles without increasing total reads.

Traversal Trends and Prefetching Opportunities

BVH traversal leverages tree structures (typically DFS) to reduce the number of ray-primitive intersection tests. Investigation of traversal stack activity reveals exploitable temporal regularities: sequences of consecutive pop operations following a downward traversal frequently yield high-value prefetching opportunities. Pop streaks (multiple consecutive stack pops) are correlated with cache misses, and their presence suggests that a prefetcher simply monitoring traversal stack heads can deliver precise, low-overhead prefetching.

Figure 3: Example DFS BVH traversal and the associated stack, visualizing the pattern of node accesses and stack content during traversal.

Figure 4: Statistical analysis of pop streaks, with longer streaks contributing disproportionately to RT read cache misses—indicative of substantial prefetch benefit.

Breadth-first search (BFS) traversal, while less efficient for standard path tracing due to its inability to skip distant nodes, offers even higher predictability for prefetching due to the queue structure.

Tree Traversal Prefetcher (TTP): Design and Implementation

TTP exploits the precise knowledge encoded in traversal stacks to provide timely and accurate prefetching. For DFS traversal, a finite-state machine (FSM) per thread monitors stack operations. Upon consecutive pops, TTP aggressively issues prefetches of the next stack entries, escalating the count as confidence increases. This approach entirely obviates the need for speculative address prediction, resulting in extremely high prefetch accuracy and minimal bandwidth or area overhead.

Figure 5: Finite-state machine controlling prefetch issuance based on DFS traversal stack activity.

Figure 6: TTP hardware architecture: per-thread state machine and pointer enhancements to the RT unit (highlighted) enable precise prefetch with negligible area increase.

For BFS traversal, TTP simply issues prefetches for the next $N$ queue entries upon each pop, with $N$ tunable for workload characteristics.

Experimental Methodology

TTP was evaluated on Vulkan-sim 2.0, a cycle-level GPU simulator, across the full range of LumiBench ray tracing benchmarks at inclusion of both procedural and triangle-heavy scenes. Baseline comparisons were made with the state-of-the-art Treelet prefetcher and prior stack-based approaches, using detailed performance, power, bandwidth, and cache statistics.

Figure 7: High-level block diagram of the GPU architecture under consideration; modifications for TTP denoted in purple.

Quantitative Results

TTP achieves a geometric mean speedup of 1.48$\times$ (up to 1.89$\times$) for path tracing at 128×128 resolution. These performance improvements are attributed primarily to latency hiding rather than increased bandwidth, as evidenced by negligible changes in DRAM traffic and a modest 18.2% increase in DRAM bandwidth utilization, commensurate with reduced runtimes.

Figure 8: TTP delivers consistent speedups across diverse scenes with lower energy and marginal power increase, normalized to the baseline.

Cache MPKI is reduced by 28.3% at L1 and 40% at L2. TTP maintains an L1 prefetch accuracy of 98.9% and coverage of 31.5%, indicating that nearly all prefetched data is demanded, and a substantial fraction of L1 misses are eliminated. When used with BFS traversal, TTP yields even higher speedups (up to 2.2$\times$ for $N=4$) and further reductions in MPKI.

Figure 9: L1 and L2 cache misses per kilo-instruction with TTP, demonstrating robust miss rate reductions.

Figure 10: L1 prefetch accuracy approaches 99%, with coverage scaling in direct proportion to observed speedup.

The prefetcher is robust to hardware configuration (cache sizing, RT unit count) and maintains high efficiency (i.e., the proportion of prefetches filling unique cache lines with available MSHRs). Notably, TTP imposes negligible area overhead, requiring only minor metadata extensions per thread stack.

Comparison with Competing Approaches

The Treelet prefetcher, while theoretically promising, degrades in performance as overall scene or thread count grows, due to bandwidth waste and increased DRAM reads (up to 1.38$\times$), leading to cache pollution and inflexible BVH traversal ordering. In contrast, TTP maintains baseline DRAM traffic while sustaining its speedup advantage across all tested workloads and scales to higher resolutions and larger scene sizes without loss of efficacy.

Figure 11: Side-by-side comparison of TTP and Treelet prefetcher performance, normalized to baseline; TTP outperforms Treelet across all scenes.

Figure 12: Treelet prefetcher incurs significant DRAM read amplification; TTP maintains bandwidth-parity with the baseline.

Implications and Future Directions

TTP’s hardware-efficient strategy demonstrates that low-overhead, per-thread prefetch logic can nearly saturate the potential for memory latency hiding in RT-enabled GPUs. By leveraging the precise traversal information of each thread—without requiring BVH structural modifications, treelet formation, or complex address prediction—TTP fundamentally advances the practical limits of latency-tolerant ray tracing.

Practically, incorporating TTP-like logic into commercial RT units would reduce the silicon and verification costs while enhancing performance and energy efficiency. The design also generalizes to other irregular tree-traversal workloads now being mapped onto RT hardware, including spatial queries, nearest-neighbor search, and various non-rendering tree algorithms in scientific and ML workloads.

Figure 13: TTP maintains its performance advantage even at increased rendering resolutions where pressure on memory and caches is more severe.

Theoretically, TTP suggests new avenues for ultra-lightweight, context-aware prefetchers in domains exhibiting programmable—but just-in-time—temporal locality, especially as hardware continues to scale thread-level parallelism. Interesting directions for subsequent research include synergistic scheduling and prefetching for multi-ray coherence, dynamic traversal ordering for hybrid workloads, and software-cooperative prefetcher tuning.

Conclusion

The Tree Traversal Prefetcher (TTP) represents a judicious architectural enhancement for GPU ray tracing units, directly addressing the persistent memory latency bottleneck in BVH traversal. TTP's utilization of traversal stack state enables highly accurate, context-sensitive prefetching with near-negligible overhead, delivering robust speedups and energy savings. Its superiority over prior prefetchers, and minimal cost, position it as a clear candidate for future GPU RT unit integration. Broader adoption could accelerate not only rendering workloads but any domain leveraging the irregular tree traversal capacity of contemporary graphics accelerators (2605.16253).