GPU Tree Traversal Prefetcher

• Tree Traversal Prefetcher (TTP) is a hardware mechanism that prefetches exact BVH node addresses from traversal state to mitigate memory latency in GPU ray tracing.
• It utilizes DFS and BFS modes with a finite-state machine to prefetch during upward or backtracking phases, achieving up to 1.89× speedup with high accuracy.
• TTP integrates directly into RT units with minimal hardware overhead, yielding significant energy savings and improving cache utilization in irregular memory accesses.

Tree Traversal Prefetcher (TTP) is a hardware prefetching mechanism for GPU ray-tracing accelerators that targets the dominant memory-latency bottleneck in Bounding Volume Hierarchy (BVH) traversal. Its defining idea is to leverage node addresses that are already present in the per-thread traversal stack or queue maintained by RT hardware, rather than inferring future accesses from generic streams, strides, or correlation histories. In DFS traversal, TTP interprets consecutive stack pops as a signal of upward or same-level traversal and prefetches exact future node addresses from the stack; in BFS traversal, it prefetches ahead in the queue. In Vulkan-sim 2.0, the design achieves $1.48\times$ speedup on average and up to $1.89\times$ compared to the baseline, with $98.92\%$ average L1 accuracy and nearly negligible hardware overhead (Tozlu et al., 15 May 2026).

1. Architectural setting and problem formulation

TTP is situated in hardware-accelerated ray tracing on GPUs with dedicated RT units or RT cores integrated into each SM. In the modeled organization, a shader issues a trace_ray instruction and the RT unit performs BVH traversal internally as a specialized CISC-like operation. The RT unit model includes a warp buffer containing per-thread traversal stacks and ray metadata, a memory access queue, response FIFOs, and fixed-function math units for ray-box and ray-triangle tests. The traversal procedure repeatedly takes a node address from a per-thread traversal stack, reads the corresponding BVH node from memory through the cache hierarchy, tests the ray against child bounding boxes or primitives in fixed-function units, and pushes newly discovered child addresses back onto the stack (Tozlu et al., 15 May 2026).

The motivation for TTP is that BVH traversal is memory-bound. The BVH can be hundreds of MB to GB in size, far beyond on-chip caches, and rays traverse it irregularly and divergently. The thread-status breakdown during trace_ray shows that threads spend most of their time waiting on memory fetches rather than performing intersection math, while DRAM bandwidth is often underutilized in baseline execution. This places the system in a latency-bound regime in which timely prefetching can overlap much of the stall time. Conventional stream and stride prefetchers are a poor fit because BVH traversal is data-dependent and irregular; GHB- and IMP-style designs attempt to learn irregularity statistically, but traversal order depends on ray-dependent geometry tests and tree-walk behavior. TTP instead exploits the fact that future node addresses already exist in traversal state maintained by the RT hardware (Tozlu et al., 15 May 2026).

The baseline traversal algorithm assumed for closest-hit rays is DFS. The pseudocode in the source material begins with if ray intersects root: stack.push(root_node_addr), then repeatedly performs node_addr <- stack.pop(), node <- mem_read(node_addr), tests children for internal nodes, and updates the closest hit for leaf nodes. The paper assumes a 6-ary BVH node format in the pseudocode. A central constraint is that the traversal stack stores node addresses, not node contents, so each step still incurs a memory read for the node data, and the next address during downward descent is generally unknown until the current node has been fetched and its intersection tests have completed (Tozlu et al., 15 May 2026).

2. Traversal semantics as a prefetch signal

The core observation behind TTP is that tree traversal has semantically meaningful phases that conventional prefetchers do not observe. In DFS BVH traversal, consecutive pushes indicate downward traversal toward leaves, whereas consecutive pops indicate upward traversal toward the root or same-level traversal. Prefetching during downward traversal is difficult and speculative because which child addresses will be visited depends on dynamic ray-box hit outcomes. During upward traversal, by contrast, the stack already contains exact addresses of deferred subtrees that the traversal will eventually visit (Tozlu et al., 15 May 2026).

This asymmetry is the basis of TTP’s “precise prefetching.” The addresses themselves are not predictions in the usual address-correlation sense; they are exact deferred traversal targets already stored in hardware traversal state. The speculative component lies only in timing and utility. The paper therefore characterizes the DFS mechanism as best-effort but address-exact prefetching triggered by a traversal-phase heuristic. This distinction explains why measured accuracy is very high without implying that every prefetched line will be consumed at the optimal time (Tozlu et al., 15 May 2026).

The paper’s example tree shows a deeply traversed branch that is then unwound; after popping one leaf, the stack contains addresses such as $N$, $L$, $K$, $H$, and $B$, all of which are legitimate future accesses. The profiling data further divides RT read misses by whether they occur on the first pop after a push, the second consecutive pop, the third consecutive pop, or the fourth-and-beyond case. Longer pop streaks account for a significant portion of all deep misses. This makes the pop streak a traversal-semantic trigger rather than an arbitrary miss-stream heuristic (Tozlu et al., 15 May 2026).

A common misconception is to treat TTP as a generic irregular-address predictor. That is not its operating principle. It does not reconstruct future addresses from deltas, miss histories, or pointer-chasing history; instead it reads them directly from traversal state. Another misconception is to assume that TTP primarily attacks downward traversal. The design explicitly identifies downward traversal as the harder phase and focuses on upward or backtracking phases, where future addresses are already available on the stack (Tozlu et al., 15 May 2026).

3. DFS mechanism and state-machine policy

In DFS mode, TTP monitors stack push and pop activity per thread and uses a small finite-state machine to control aggressiveness. The default design uses a 2-bit FSM with a “1, 2, 16” policy. On the first pop after a push, it generates 1 prefetch for the node at the top of the stack. On the second consecutive pop, it generates 2 prefetches for the top two nodes in the stack. On the third consecutive pop and beyond, it becomes more aggressive and issues up to 16 prefetches from the top 16 stack entries. Any push resets the pop streak and stops upward-prefetch generation (Tozlu et al., 15 May 2026).

Operationally, if $T$ is the current top-of-stack position, TTP computes a stop threshold $T-k$, where $1.89\times$0 under the default DFS configuration. A prefetch pointer walks down the stack entries from the current top and issues requests until it reaches $1.89\times$1. Because the stack stores exact node addresses, no address prediction is needed. The implementation maintains a pointer to the next address to prefetch so that consecutive pops do not repeatedly reissue the same prefetches. On a push, the pointer is reset to $1.89\times$2; on a pop, the FSM state is updated, which updates $1.89\times$3; each issued prefetch decrements the pointer; and prefetching stops when the pointer reaches $1.89\times$4. The source material also mentions an alternative implementation in which each stack entry has a flag indicating whether it has already been prefetched (Tozlu et al., 15 May 2026).

The design is explicitly per-thread. Each thread has its own traversal stack, and each thread also has its own TTP state machine. Prefetches are sent when the containing warp is selected by the RT unit scheduler. The simplicity of this organization is a central property of the design: there is no pattern table lookup, no miss-history traversal, no correlation matching, and no recursive address generation. TTP consumes an address source that already exists for traversal correctness and adds only modest control logic around it (Tozlu et al., 15 May 2026).

The DFS mechanism also clarifies why TTP does not claim full traversal prediction. During descent, whether a child satisfies $1.89\times$5 depends on the current ray and the current best hit distance, so a downward prefetcher would need to predict both geometric intersection outcomes and traversal ordering. TTP avoids this problem by waiting for a traversal phase in which future addresses are already enumerated by the algorithm’s own stack discipline (Tozlu et al., 15 May 2026).

4. BFS extension and microarchitectural realization

TTP also extends to BFS, where the traversal data structure is a FIFO queue rather than a LIFO stack. In BFS mode, as long as the queue is non-empty, the next node to be processed is at the head, so every queue pop can trigger prefetching of the next $1.89\times$6 queued nodes. The prefetch depth is a fixed parameter $1.89\times$7, and the paper evaluates $1.89\times$8, $1.89\times$9, and $98.92\%$0, with increasing but diminishing returns and $98.92\%$1 best among those tested (Tozlu et al., 15 May 2026).

The BFS result is notable because BFS without prefetching is slower than DFS: the table results show that BFS visits, on average, $98.92\%$2 more nodes per ray than DFS, reducing pruning opportunities. Yet BFS is more prefetchable. With TTP enabled, BFS outperforms DFS on average, giving $98.92\%$3 average speedup relative to DFS baseline. The source material identifies spnza, chsnt, frst, park, and robot as scenes where DFS remains better because BFS increases average nodes per ray too much. This suggests that prefetchability can alter the traversal-algorithm tradeoff even when raw node-count efficiency favors DFS (Tozlu et al., 15 May 2026).

Microarchitecturally, TTP is integrated into the RT unit rather than attached as an external cache-side prefetcher. Per thread, it adds a 2-bit FSM field in the warp buffer, a pointer to the next stack entry to prefetch, and comparator or control logic to stop at $98.92\%$4. The RT unit already stores the traversal stack itself, so the design reuses that structure rather than replicating node-address metadata elsewhere. When a warp is selected by the RT scheduler, demand reads from traversal retain priority by default; if no demand read is pending, TTP may issue one of its pending prefetches; and prefetches enter the normal cache or memory hierarchy through L1 (Tozlu et al., 15 May 2026).

The cache model is sector-based with sector size $98.92\%$5 B. If a BVH node spans more than $98.92\%$6 B, TTP breaks the prefetch into $98.92\%$7-byte sector requests sent one per cycle, analogously to demand reads. The paper also evaluates an arbitration variant in which prefetch can be prioritized over demand if no prefetch has been sent for a threshold number of cycles; thresholds of 25, 50, and 100 cycles produce little average impact, so the default demand-first policy is sufficient (Tozlu et al., 15 May 2026).

5. Quantitative results, comparisons, and hardware cost

The implementation is evaluated in Vulkan-sim 2.0, with power modeled using GPUWattch. The baseline hardware configuration includes 8 SMs, maximum 16 TBs per SM, warp size 32, instruction cache 128 KB with 20-cycle latency, L1 data cache 32 KB with fully associative LRU and 256 MSHRs, L2 cache 512 KB with 16-way LRU and 768 MSHRs, core/interconnect/L2 clock 1365 MHz, memory clock 3500 MHz, 1 RT unit per SM, and RT unit warp buffer size 4. Workloads come from LumiBench, using 16 scenes of varying BVH sizes and depths, from wknd with a 0.2 MB tree to robot with a 1721.3 MB tree. Most path-tracing results use $98.92\%$8 resolution and 1 sample per pixel; park uses $98.92\%$9 because of simulation timeout. The study also evaluates $N$0 and a larger GPU configuration with 30 SMs, 64 KB L1, and 3 MB L2 (Tozlu et al., 15 May 2026).

For DFS traversal, the principal results are a geometric-mean speedup of $N$1 and a maximum speedup of $N$2. Power rises to $N$3 baseline on average, but energy drops to $N$4 baseline, corresponding to $N$5 energy savings. The cache effects are substantial: L1 RT-read MPKI is reduced by $N$6, and L2 RT-read MPKI is reduced by $N$7. Prefetch quality is correspondingly high, with $N$8 average L1 accuracy, $N$9 average L2 accuracy, $L$0 L1 coverage, and $L$1 L2 coverage. The paper explicitly notes that speedup is largely proportional to coverage, and scenes with strong down-and-up DFS behavior such as ship, crnvl, fox, party, and robot show the largest gains (Tozlu et al., 15 May 2026).

A limit study further supports the design target. If all upward-traversal accesses, defined as the second and later pops after a push, hit in L1 perfectly, geometric-mean speedup would be $L$2. If all downward-traversal accesses, defined as the first pop after a push, hit in L1 perfectly, speedup would be only $L$3. This validates the emphasis on upward or backtracking misses. The design remains robust with larger L1 caches: with 64 KB and 128 KB L1 caches, speedup remains $L$4 on average. At $L$5, average speedup is still $L$6. On the larger 30-SM GPU model, average speedup is $L$7 and peak speedup is $L$8 (Tozlu et al., 15 May 2026).

Bandwidth results are also important. TTP increases average DRAM bandwidth utilization by $L$9, but total DRAM traffic remains nearly unchanged. The interpretation given in the source is that execution is shortened and requests are concentrated into fewer cycles, rather than large amounts of useless data being fetched. By contrast, the Treelet prefetcher increases total DRAM reads by $K$0 on average. Relative to other ray-tracing prefetchers, Treelet achieves only $K$1 average speedup at $K$2 path tracing and shows regressions in scenes such as ship, spnza, crnvl, and fox, while the modeled node-prefetching strategy from Park et al. achieves only $K$3 average speedup and $K$4 peak speedup because leaf-node intersection exposes too little latency to hide. TTP outperforms Treelet on all tested scenes (Tozlu et al., 15 May 2026).

The hardware cost is deliberately minimal. Synthesizing the DFS state machine in FreePDK45 for the baseline configuration of 4 warps per RT unit and 32 threads per warp yields 128 state machines per SM. The total implementation for those 128 state machines uses 1117 cells total, of which 256 are sequential cells for the 2-bit state across 128 machines and 861 are combinational cells. This averages $K$5 cells per state machine. The source compares this to the ray-properties storage in the ray buffer, which already needs $K$6 bits per thread for origin and direction coordinates. The resulting area overhead is therefore characterized as negligible (Tozlu et al., 15 May 2026).

The limitations follow directly from the mechanism. TTP helps less in scenes with short or infrequent pop streaks, workloads dominated by any-hit rays that terminate quickly, simple scenes where traversal is already fast, configurations in which caches absorb much of the working set, and scenarios where traversal order differs from conventional DFS or BFS behavior. The source reports smaller benefits for ambient occlusion and shadow shaders than for path tracing, though TTP still achieves $K$7 average speedup for ambient occlusion and $K$8 for shadow workloads (Tozlu et al., 15 May 2026).

6. Position within irregular-prefetching research

Within the broader literature on irregular-memory prefetching, TTP occupies a distinct point in the design space. The semantic prefetcher in “Semantic prefetching using forecast slices” learns dynamic address-generation code slices from retired execution, validates them under load-IP plus branch-history context, trims them into side-effect-free slices, and injects them with adaptive lookahead. That mechanism targets irregular memory access patterns that “do not manifest any form of spatio-temporal locality in the memory address space,” explicitly including linked lists, arrays of pointers, sparse graphs, cross-indexed tables, and tree-like traversals; its Graph500 BFS example includes a 4-level indirection chain, and the evaluated design uses only about 6 kB while improving performance by 24% on average over SPEC 2006 and 16% on average over SPEC 2017 (Peled et al., 2020).

The relationship between that design and TTP is instructive. The semantic prefetcher reconstructs the computation that produces future addresses and replays a compact code slice when the same IP+BHR context recurs. TTP does not learn or replay address-generation semantics at all; it reads already-materialized node addresses from traversal state in RT hardware. This suggests a fundamental difference in information source. Forecast slices are a general CPU-side mechanism for causally regenerating irregular addresses from dynamic program state, whereas TTP is a specialized GPU-side mechanism that exploits an explicit traversal frontier already stored by the accelerator (Peled et al., 2020).

A second boundary is illustrated by “Helper Without Threads: Customized Prefetching for Delinquent Irregular Loads,” which generates binary-only inline software prefetchers from backward slices of delinquent irregular loads. That work is computation-based rather than pattern-based and can help hash tables, PageRank, Graph500 CSR, and related irregular loops, but it explicitly excludes the hardest class of pointer-following recurrences. It distinguishes runnable DILs from “chasing” DILs, stating that classic pointer chasing lies outside the scope of the software scheme, and it also excludes branchy binary-tree traversal because prefetching $K$9 iterations ahead can create $H$0 possible addresses (Sankaranarayanan et al., 2020).

Against that backdrop, TTP can be understood as a traversal-semantic specialization rather than a general irregular-load framework. It does not solve arbitrary tree or graph pointer chasing in a CPU pipeline, and it does not attempt the program-semantic reconstruction of forecast-slice designs. Its strength is narrower and more concrete: when RT hardware already maintains a stack or queue of exact future BVH node addresses, TTP converts that traversal state into highly accurate, low-cost prefetch requests. A plausible implication is that TTP’s unusual effectiveness derives less from stronger generic prediction and more from the fact that RT traversal hardware exposes future-access metadata directly, in a form that neither general semantic prefetchers nor inline software helpers can assume (Tozlu et al., 15 May 2026).