Dynamic TensorRT Compilation

• Dynamic TensorRT Compilation is a method for optimizing GPU kernels on-the-fly for varying tensor shapes while balancing hardware constraints with low-latency performance.
• It leverages analytic hardware models and pre-generated micro-kernels—exemplified by FTuner and Vortex—to precisely match dynamic input dimensions without extensive padding.
• This approach improves inference speed and reduces tuning overhead by replacing exhaustive profiling with efficient runtime kernel selection and composition.

Dynamic TensorRT compilation concerns dynamic-shape inference in a TensorRT-like runtime, where input lengths and resolutions vary across requests and kernel selection cannot be fixed to a single static schedule. TensorRT today fixes a small set of shapes at build time or pads inputs to match pre-tuned kernels, and TensorRT’s current dynamic-shape flow uses a small set of “optimization profiles,” builds kernels per profile, and dispatches with a selector at runtime. Recent compiler work treats this as a hardware-aware, low-latency scheduling problem: pre-generate a compact library of hardware-friendly kernels, then use analytic ranking and shape-aware composition at runtime to avoid padding waste, expensive rebuilds, and exhaustive per-shape tuning (Mu et al., 2024, Zhou et al., 2024).

1. Dynamic-shape compilation as a TensorRT problem

Many artificial intelligence models process input data of different lengths and resolutions, making the shape of the tensors dynamic. The performance of these models depends on the shape of the tensors, which makes it difficult to optimize the tensors before the model runs. Two common solutions are identified. The first is to add useless data to the input to match a pre-optimized tensor library. The second is to use small basic tensors to create a tensor that is closest in size to the input data and then tune it to minimize padding. However, this second solution can be time-consuming (Mu et al., 2024).

The same problem appears in TensorRT’s profile-based workflow. Current dynamic-shape pipelines such as TensorRT, DietCode, and Nimble rely on a fixed list of sample shapes, auto-tune a kernel per sample, then dispatch via a decision tree at runtime. When real inference shapes fall outside that set, kernels either underutilize the hardware through padding loss or choose suboptimal tiling, leading to up to $4\times$ slowdowns. A naïve shape-generic search needs hours or days of offline profiling, and DietCode reports $\sim 25$ h for GPU, which is impractical for real-time or multi-tenant inference (Zhou et al., 2024).

Within this setting, dynamic TensorRT compilation is not merely dynamic dispatch. It is the problem of reconciling variable realized shapes with hardware limits such as cache sizes, register file size, max threads per block, peak flops, memory bandwidth, and tensor-core or CUDA-core execution modes, while preserving low enqueue latency and avoiding repeated graph rebuilding.

2. FTuner and the uKernel formulation

FTuner introduces a minimal building block called the uKernel, defined as “one thread block’s worth” of work on an SM. In code, it corresponds to a TVM schedule fragment parameterized by two tile-sizes, reg_tile and smem_tile, and three pre-computed metrics: compute_eff, padding_threshold, and usage_eff. The motivation is that any large tensor compute, including MatMul, Conv2D, and BatchMatMul, can be seen as tiling along its axes. By pre-generating a library of small tiles across many factor combinations of the axes, FTuner can later patch together multiples of different uKernels to exactly fit any dynamic shape, eliminating almost all padding (Mu et al., 2024).

FTuner’s HardwareAlign pass chooses uKernel shapes by factorizing each dynamic axis. RegTileRule() enumerates all divisors, or near-divisors for primes, of each axis length to form candidate register-level tiles. SmemTileRule() then grows shared-memory-level tiles from the register tiles in multiples aligned to 8 for float32 on NVIDIA, ensuring coalesced loads. Once every $(reg\_tile, smem\_tile)$ is enumerated, FTuner has $O(10^6)$ candidates for $\sim 128$ shapes on V100, but reduces this set with an analytic hardware information model rather than an ML cost model.

For each uKernel $K$, FTuner computes the fraction of useful elements,

$$K.pad = \frac{wkl.base(K)}{wkl.base(K)+wkl.pad(K)},$$

the occupancy ratio if $K$ blocks tile the tensor,

$$K.occ = \frac{wkl/K}{\lceil wkl/K \div NumCores \rceil},$$

and register-usage constraints of the form

$$(reg\_tile_{i,j} + RestRegs)\times smem\_tile_{i,j} \le \frac{REGS\_PER\_SM}{block\_bound}.$$

It also enforces space-axis saturation, $\sim 25$0, and reduce-axis compute-intensity,

$\sim 25$1

with

$\sim 25$2

After HardwareAlign, FTuner’s filter enforces padding_threshold ≥ ε, usage_eff ≥ λ, and bounded register usage; then sweeps $\sim 25$3, $\sim 25$4 in lock-step on V100 with $\sim 25$5 until $\sim 25$6 pad or $\sim 25$7 occ, followed by space- and reduce-axis checks. The result is a few dozen to a few hundred high-potential uKernels without any device-measurement or learned cost model.

3. FTuner’s two-phase strategy and its TensorRT adaptation

FTuner uses a two-phase compile/runtime strategy. At compile time, once per model and device, it extracts each fused operator’s compute graph as nested loops and runs HardwareAlign + CrossPick + Multi-axis to build the uKernel set. At runtime, for each dynamic input shape, CombinSearch selects a main axis $\sim 25$8, the longest axis, and attempts to exactly partition $\sim 25$9 into sums of different $(reg\_tile, smem\_tile)$0 values so that the uKernels cover $(reg\_tile, smem\_tile)$1 with zero padding. Any leftover exact-divisors produce single-$(reg\_tile, smem\_tile)$2 programs. These uKernel-based programs, uProg, typically number in the hundreds rather than the millions. FTuner then computes a Synthesis Index Analysis (SIA) score,

$(reg\_tile, smem\_tile)$3

with $(reg\_tile, smem\_tile)$4 by default, selects the Top 10 by SIA without on-device compilation, and can optionally verify device timing to pick the best. Because tiling-factor enumeration and analytic costing happen offline, the online stage is hundreds of microseconds (Mu et al., 2024).

The experimental profile is explicitly relevant to dynamic TensorRT compilation. On NVIDIA V100 for the Dense operator over 128 shapes, FTuner matches cuBLAS on $(reg\_tile, smem\_tile)$5 of shapes within $(reg\_tile, smem\_tile)$6, is up to $(reg\_tile, smem\_tile)$7 faster than Roller with a $(reg\_tile, smem\_tile)$8 average, and outperforms HAOTuner on $(reg\_tile, smem\_tile)$9 of shapes with a $O(10^6)$0 average. For BatchMatMul on 8 sampled shapes, FTuner reports $O(10^6)$1 versus cuBLAS, $O(10^6)$2 versus Roller, and $O(10^6)$3 versus HAOTuner. For end-to-end BERT-base on 8 shapes on V100, FTuner is within $O(10^6)$4 of cuBLAS, $O(10^6)$5 versus Roller, and $O(10^6)$6 versus HAOTuner. Compilation time for Dense on V100 over 128 shapes is approximately 6 872 s for Ansor, 1 104 s for HAOTuner, 163 s for Roller, and 149 s for FTuner. For end-to-end BERT-base, FTuner reports 474 s versus Ansor 69 518 s, HAOTuner 10 862 s, and Roller 2 528 s. Overall, FTuner cuts tuning time by 1–2 orders of magnitude while still delivering a 3% net speedup versus model-training compilers (Mu et al., 2024).

The proposed TensorRT integration is correspondingly concrete. A uKernel library plugin extends the TensorRT builder to generate a library of uKernels for each operator, using the same HardwareAlign and analytic-filter passes, and storing $O(10^6)$7 per uKernel. JIT CombinSearch & SIA runs when the engine sees a new dynamic shape at runtime: it picks the main axis, performs CombinSearch, builds uProg, scores with SIA, and selects the best sub-kernel combination without retraining or full graph recompilation. Low-latency codegen uses the existing TensorRT code-generator to instantiate the two chosen uKernel schedules and enqueue them instead of invoking one monolithic kernel. Because each uKernel is already code-generated at build time, the JIT cost is described as mere pointer-stitching. Minimal padding follows from exact coverage on the main axis and SIA’s preference for high pad thresholds; in the adaptation sketch, padding across all axes stays below $O(10^6)$8 of computation, $O(10^6)$9–$\sim 128$0 lower than Roller.

4. Vortex and hardware-aware strategy-space hierarchization

Vortex approaches the same class of problems as a hardware-driven and sample-free compiler tailored for dynamic-shape tensor programs. Its central claim is that compilation optimization methods for such networks often rely heavily on predefined samples, and that these sample-driven methods restrict adaptability and efficiency. Vortex replaces that workflow with a bidirectional compilation scheme that combines top-down abstraction for aligning tensor program execution with hardware hierarchies and bottom-up kernel construction to narrow the search space (Zhou et al., 2024).

Its analytical cost model is defined per hardware hierarchy level $\sim 128$1. The temporal cost is

$\sim 128$2

where $\sim 128$3, and $\sim 128$4 is the lower-level cost. Parallel scaling is

$\sim 128$5

and total cost is

$\sim 128$6

Empirical calibration at the innermost levels, from register to core, corrects pipeline inaccuracies, while higher levels use the analytical model to avoid full hardware profiling.

Vortex’s strategy space hierarchization is structured around rKernel. In top-down abstraction, any operator is represented recursively across hardware hierarchy levels, such as Grid $\sim 128$7 CTA $\sim 128$8 Warp for GPU or Process $\sim 128$9 Thread $K$0 Register for CPU. In bottom-up kernel construction, the compiler enumerates candidate tile shapes $K$1 at $K$2, constrained by register count and ISA granularity; at $K$3 it generates only multiples of chosen $K$4 shapes that fit in shared or L1 cache; and at $K$5 it similarly uses only multiples of $K$6 blocks that fit global-level resources. Any non-multiple choice is described as forcing high padding. At each level, the hybrid analyzer picks the top-$K$7 candidates and discards the rest. This produces a search space pruned “by construction,” rather than by shape sampling or brute-force tuning.

5. Vortex’s bidirectional workflow in a TensorRT setting

Vortex separates offline, shape-free code generation from runtime shape selection. Offline, it queries hardware parameters such as cache sizes, register file, bandwidth, and peak flops; instantiates the rKernel abstraction across $K$8; generates candidate tile sizes at $K$9, prunes by ISA, analyzes empirically; lifts to $$K.pad = \frac{wkl.base(K)}{wkl.base(K)+wkl.pad(K)},$$0, prunes by multiples and cache limits, analyzes again; and finally emits a small library of micro-kernels. At runtime, given the actual shape $$K.pad = \frac{wkl.base(K)}{wkl.base(K)+wkl.pad(K)},$$1 or $$K.pad = \frac{wkl.base(K)}{wkl.base(K)+wkl.pad(K)},$$2, it computes which pre-generated micro-kernels cover the shape with minimum padding, uses the analytical cost model to rank or match the best kernel, and launches with exactly computed grid and block configuration (Zhou et al., 2024).

The adaptation to TensorRT is framed as a replacement for profile-based tuning. TensorRT can query hardware and build its own hierarchical micro-kernel library offline, eliminating manual profile management. In the builder phase, rather than sampling a few shapes per dimension, TensorRT’s IR could expose rKernel loop axes, and a bottom-up pass would generate and prune candidate tilings by hardware limits. At enqueue time, TensorRT already has the dynamic shape, so it can invoke the Vortex analytical cost model to pick the optimal kernel variant and compute grid and block dimensions exactly, rather than falling back to a “closest” profiled kernel. Empirical “$$K.pad = \frac{wkl.base(K)}{wkl.base(K)+wkl.pad(K)},$$3-benchmark hooks” can remain at $$K.pad = \frac{wkl.base(K)}{wkl.base(K)+wkl.pad(K)},$$4 to calibrate small micro-kernels once per device and cache results across applications.

The reported quantitative results describe the expected scale of improvement. Vortex reduces compilation time by $$K.pad = \frac{wkl.base(K)}{wkl.base(K)+wkl.pad(K)},$$5 compared to the existing dynamic-shape compiler. In the detailed GPU GEMM comparison, DietCode tuning is 90 000 s, or 25 h, whereas Vortex offline compilation is 529.6 s, approximately $$K.pad = \frac{wkl.base(K)}{wkl.base(K)+wkl.pad(K)},$$6 faster. For operator-level speedups, Vortex reports CPU gains versus oneDNN of $$K.pad = \frac{wkl.base(K)}{wkl.base(K)+wkl.pad(K)},$$7 for GEMM and $$K.pad = \frac{wkl.base(K)}{wkl.base(K)+wkl.pad(K)},$$8 for Conv, giving an overall $$K.pad = \frac{wkl.base(K)}{wkl.base(K)+wkl.pad(K)},$$9 versus vendor libraries. On GPU Tensor-Core workloads versus cuBLAS and cuDNN, it reports $K$0 for GEMM and $K$1 for Conv. On GPU CUDA-Core-only workloads versus DietCode, cuBLAS, and CUTLASS, it reaches up to $K$2 in some GEMMs and $K$3 on average versus the prior dynamic-shape solution. End-to-end normalized results are also reported: BERT $K$4, BERT-large $K$5, GPT-2 $K$6, AlexNet $K$7, ResNet $K$8, and GoogleNet $K$9 (Zhou et al., 2024).

6. Comparative interpretation, design patterns, and common misconceptions

A common misconception is that dynamic-shape TensorRT compilation must be sample-driven. Vortex directly rejects this assumption by presenting a sample-free compiler whose runtime phase chooses among pre-generated micro-kernels using a hardware-aware analytical model, rather than a fixed list of sample shapes. A second misconception is that competitive dynamic-shape optimization necessarily requires either brute-force search or trained cost models. FTuner directly rejects this assumption by using an analytic hardware information model, a uKernel library, and a two-phase compile/runtime strategy that eschews ML cost-model training and brute-force search (Mu et al., 2024, Zhou et al., 2024).

The two systems emphasize different but compatible abstractions. FTuner operates through exact or near-exact shape coverage by patching together small, various-sized uKernels and ranking assembled programs with SIA. Vortex operates through hierarchical decomposition of the strategy space, restricting each level to hardware-valid multiples and using a hybrid empirical and analytical analyzer. This suggests two complementary design idioms for dynamic TensorRT compilation: one centered on fine-grained composition of a minimal computational unit, and the other on hierarchical narrowing of the strategy space.

The practical consequences can be summarized as follows.

Approach	Compilation-time evidence	Runtime evidence
FTuner	Dense on V100, 128 shapes: 149 s; BERT-base: 474 s	Dense: matches cuBLAS on 44% of shapes within 10%; BERT-base: within 8% of cuBLAS, +23% vs Roller, +11% vs HAOTuner
Vortex	GPU GEMM: 529.6 s vs DietCode 90 000 s	Overall $$K.occ = \frac{wkl/K}{\lceil wkl/K \div NumCores \rceil},$$0 vs vendor libs on CPU and $$K.occ = \frac{wkl/K}{\lceil wkl/K \div NumCores \rceil},$$1 vs prior dynamic-shape solution on GPU

Best-practice recommendations stated in the Vortex formulation align closely with the FTuner adaptation sketch. These include unifying operator abstraction with hardware hierarchy, decoupling offline code generation from runtime shape selection, pruning strategy space by construction, employing a hybrid analyzer, using a lightweight runtime cost estimator instead of JIT profiling, and supporting hardware-adaptive selection such as choosing between CUDA cores and tensor cores, where Vortex’s “Adaptive” mode yields up to $$K.occ = \frac{wkl/K}{\lceil wkl/K \div NumCores \rceil},$$2 improvement over fixed modes (Zhou et al., 2024).

The most direct synthesis appears in the proposed TensorRT adaptations themselves. FTuner argues that the combined system would achieve the low-overhead JIT of TensorRT for fixed shapes, yet lose none of the performance on truly variable inputs. A plausible implication is that dynamic TensorRT compilation, in this research line, is best viewed not as late binding of a pre-profiled kernel table, but as a runtime realization of offline hardware-aware synthesis: the engine stores a compact, hardware-valid kernel basis and performs lightweight analytic selection or composition when the realized shape becomes known.