Vector-Length-Aware Packed Data Layouts

• Vector-length-aware packed data layouts are memory organizations where tensor tiling factors are functions of the hardware vector length, ensuring contiguous data for SIMD execution.
• They optimize data locality by aligning microkernel iterations with contiguous memory regions, reducing overhead from padding and masking.
• These layouts are applied across domains—from scalable GEMM and vector search to on-SSD graph traversal—demonstrating flexible performance gains and efficient resource use.

Searching arXiv for the cited papers and closely related work on vector-length-aware packed data layouts. arXiv search query: "Vector-Length-Aware Packed Data Layouts scalable packed layouts vector-length agnostic ML code generation" Vector-length-aware packed data layouts are memory organizations in which the physical arrangement of data is defined with explicit reference to a vector length parameter, typically the hardware vector length $VL$ of a scalable SIMD ISA or an application-level block width. In the most direct formulation, the layout of packed tensors is parameterized by tile sizes $m_r$, $n_r$, and $k_r$ that are functions of $VL$, so that the contiguous memory regions consumed by each microkernel iteration remain correctly shaped as $VL$ changes across hardware implementations (Beysel et al., 12 May 2026). More broadly, the term also covers packed organizations that restructure vectors or graph records so that contiguous slices match the execution granularity of distance kernels, pruning algorithms, or storage devices, thereby preserving locality while reducing unnecessary data movement (Kuffo et al., 6 Mar 2025, Song et al., 12 May 2026). In language-level systems, packed data denotes a unified representation between memory and the network, removing marshalling steps and improving locality; a Haskell library implementation reports traversing packed data up to 60% faster than unpacked data (Jamet et al., 28 Apr 2025).

1. Definition and conceptual scope

In scalable vector compilation, a vector-length-aware packed layout is a memory layout for tensors, especially matrices, whose tile sizes and packing factors are explicit functions of the hardware vector length $VL$, rather than fixed compile-time constants (Beysel et al., 12 May 2026). The essential property is not merely contiguity, but contiguity of exactly those blocks that the microkernel reads or writes in one iteration. If a kernel consumes $2 \cdot VL$ FP32 elements from one operand per inner iteration, the packed layout groups those $2 \cdot VL$ elements contiguously for any legal $VL$.

This distinguishes vector-length-aware packing from conventional fixed-width packed layouts. In fixed-width SIMD compilation, layout choices are specialized to a compile-time constant vector width, such as a fixed $m_r$0 microkernel. Under vector-length-agnostic execution, the compiler does not know whether deployment will use $m_r$1, $m_r$2, or $m_r$3 FP32 lanes, so the layout and the microkernel must both remain symbolic in $m_r$4 (Beysel et al., 12 May 2026).

The broader packed-data literature uses a compatible but more general notion. Packed data is a unified representation between memory and the network that removes serialization and deserialization steps and can improve performance through compactness and locality (Jamet et al., 28 Apr 2025). This suggests that vector-length-aware packed layouts are a specialization of packed representations in which the packed form is additionally constrained by vector execution semantics.

A useful distinction is between three deployment regimes. In tensor compilers, vector length is a hardware property of scalable SIMD and the layout is parameterized by $m_r$5 directly. In vector similarity search, the relevant execution granularity is often a fixed block size chosen to match SIMD and register capacity; the PDX layout uses PDX_BLOCK_SIZE = 64 and organizes dimensions vertically within each block (Kuffo et al., 6 Mar 2025). In on-SSD graph-based vector search, “awareness” concerns the cost of full vector payloads as dimensionality $m_r$6 grows, and layout design decouples vectors from adjacency metadata so that traversal does not force full-vector I/O (Song et al., 12 May 2026).

2. Formal layout models and packing transformations

The canonical formalization appears in packed matrix multiplication. For a row-major matrix $m_r$7, a classical packed layout chooses microkernel tile sizes $m_r$8 and $m_r$9 and defines

$n_r$0

with indexing relation

$n_r$1

Analogous packing is applied to $n_r$2 using tile sizes $n_r$3 and to $n_r$4 using $n_r$5 (Beysel et al., 12 May 2026). This 4D packing makes each microkernel tile contiguous and explicit in the IR.

The vector-length-aware extension preserves the same tensorization but replaces constant tile sizes with functions of vector length:

$n_r$6

The packed tensor remains

$n_r$7

but its logical shape and index mapping now depend on $n_r$8 through the microkernel configuration (Beysel et al., 12 May 2026). This is the defining mathematical move: the structure of packing is unchanged, while the sizes of the inner packed tiles become vector-length-parametric.

The same section formalizes rounded-up outer dimensions

$n_r$9

Because tiles are defined over rounded-up dimensions, packed layouts inherently incorporate padding semantics. The paper emphasizes that when register-level tile sizes are aligned with these layouts, computations can be performed without additional masking, since out-of-bounds elements are represented explicitly in the packed data (Beysel et al., 12 May 2026).

A different but related packing model appears in vector search. PDX stores multiple vectors in one block and, within the block, stores all values of dimension $k_r$0 contiguously. For a block of $k_r$1 vectors of dimensionality $k_r$2, the offset of element $k_r$3 is

$k_r$4

This is a PAX-like layout specialized for vectors: dimensions are mini-columns within a block of vectors (Kuffo et al., 6 Mar 2025). Here the “inner packed dimension” is not a scalable hardware $k_r$5, but the block size $k_r$6 selected so that the vectors-inner loop can be auto-vectorized efficiently.

3. Microkernel alignment, SIMD execution, and compiler representation

A representative FP32 SVE microkernel computes an $k_r$7 output tile per inner-loop iteration. In the reported design, it consumes from $k_r$8 one column of length 8, split into two groups of 4 and broadcast via svld1rq_f32, and from $k_r$9 one row of length 2*VL, loaded as two contiguous svfloat32_t vectors. The implied tile parameters are therefore $VL$0, $VL$1, and $VL$2 per inner iteration (Beysel et al., 12 May 2026). The layout of $VL$3 must store, for each $VL$4, contiguous blocks of $VL$5 FP32 elements corresponding exactly to those loads, while the layout of $VL$6 must store the output vectors contiguously at offsets 0, vlen, 2*vlen, ..., 15*vlen.

The corresponding compiler pipeline represents these transformations explicitly. linalg.pack and linalg.unpack realize the layout transformation from logical matrices to packed 4D tensors and back, while linalg.mmt4d expresses matrix multiplication on packed tensors (Beysel et al., 12 May 2026). Internally, scalable vector dependence is represented through scalable vector types such as vector<[vscale x 4]xf32> in MLIR’s vector dialect and target-specific arm_sve operations, which lower through LLVM scalable vector types carrying the vscale notion.

This explicit representation matters because tiling, fusion, and vectorization passes can remain layout-aware. The compiler chooses a predefined microkernel configuration for each data type and target; the configuration includes symbolic functions $VL$7, $VL$8, and $VL$9 mapping $VL$0 to tile sizes. Higher-level blocking factors $VL$1, $VL$2, and $VL$3 are then chosen largely independently for cache locality and multithreading (Beysel et al., 12 May 2026). The result is a separation between register-level vector-length-parametric packing and cache-level blocking.

The same section also clarifies boundary behavior. Packed tiles over rounded-up dimensions allow the core GEMM to run mostly with the all-true SVE predicate svptrue_b32(), minimizing mask overhead. This is a common misconception: scalable vectors do not inherently imply pervasive predication inside the hot microkernel. With vector-length-aware packed layouts, the main remainder handling can be displaced into pack and unpack operations, leaving the central compute loop to operate on full packed tiles (Beysel et al., 12 May 2026).

A language-level counterpart exists in portable packed-data support. The Haskell packed-data library provides type safe building and reading of packed data in a functional style, does not rely on compiler modifications, and leverages meta-programming so programmers can pack their own data types (Jamet et al., 28 Apr 2025). A plausible implication is that such a library-level approach provides an implementation substrate for expressing vector-aware packed formats without requiring a custom compiler, although the supplied abstract does not claim direct SIMD or scalable-vector support.

4. Alternative layout families beyond dense tensor compilation

Vector-length-aware packed layouts are not confined to tensor compilers. PDX is a vector data layout for similarity search that stores multiple vectors in one block and uses a vertical layout for dimensions, with the default PDX_BLOCK_SIZE = 64 (Kuffo et al., 6 Mar 2025). For float32, each dimension slice is $VL$4 bytes; on AVX-512, each slice fits exactly in four 64-byte SIMD loads. The key loop structure is dimension-outer and vectors-inner: for each dimension $VL$5, the implementation loads the contiguous slice of that dimension across the block’s vectors and updates a distances[64] array.

This organization alters how SIMD is exploited. Horizontal layouts vectorize across dimensions within one vector and therefore require sufficiently large $VL$6 to fill registers efficiently. PDX vectorizes across vectors instead, so it gets full register utilization even when only a small number of dimensions is scanned, which is especially important for dimension-pruning methods (Kuffo et al., 6 Mar 2025). The paper reports that PDX beats SIMD-optimized distance kernels on standard horizontal vector storage, is on average 40% faster, and only relies on scalar code that gets auto-vectorized (Kuffo et al., 6 Mar 2025).

On-SSD graph-based vector search presents a different layout problem. Prior systems frequently use a single packed record per vertex, [Vector] [Degree] [Edge List], co-locating the full vector and adjacency list within a 4 KiB SSD page. NAVIS identifies two systemic limitations of this design: packed layouts couple every edge fetch to a full vector load, and static entrance graphs drift away from newly inserted regions as updates accumulate (Song et al., 12 May 2026). Its response is locality-driven decoupling: an edgelist file stores adjacency lists and related metadata, a vector file stores only full-precision vectors, and an in-memory indirection table maps vertex IDs to edge and vector locations.

The significance of this redesign is that traversal can read only edgelist pages while distances are computed from in-memory PQ codes; full vectors are read selectively only during reranking (Song et al., 12 May 2026). This is not vector-length-parametric in the SVE sense, but it is vector-length-aware in the storage sense: as dimensionality $VL$7 grows, full vectors dominate page cost, so the layout isolates the $VL$8 payload from the smaller, high-reuse graph metadata.

Domain	Layout principle	Representative paper
Scalable GEMM code generation	Tile sizes $VL$9 as functions of $VL$0	(Beysel et al., 12 May 2026)
Vector similarity search	Dimensions packed vertically within blocks of vectors	(Kuffo et al., 6 Mar 2025)
On-SSD graph search	Edgelists decoupled from full vectors for selective reads	(Song et al., 12 May 2026)
Portable language support	Unified packed representation with type-safe construction and traversal	(Jamet et al., 28 Apr 2025)

5. Performance characteristics and empirical behavior

The principal performance rationale for vector-length-aware packed layouts is that the execution unit sees data in exactly the order and granularity it consumes. In the scalable SVE pipeline, this means contiguous VL-dependent tiles that support svld1, svst1, and svmla_lane_f32 without gather/scatter in the core microkernel (Beysel et al., 12 May 2026). The paper reports that, on real-world ML workloads on Arm CPUs, the resulting SVE code is competitive with and often outperforms existing NEON-based code generation within IREE, achieving up to $VL$1 speedup, and that simulator studies show scaling with increasing SVE vector length on compute-bound workloads (Beysel et al., 12 May 2026).

The scaling behavior is especially revealing. For square FP32 matmuls up to 1024×1024×1024, simulated SVE-512 achieves up to 3.36× speedup over SVE-128; for 2048×2048×512, SVE-512 achieves 3.44× (Beysel et al., 12 May 2026). These numbers are close to the ideal 4× scaling that would result from quadrupling vector width, though they are limited by higher reduction latencies and memory hierarchy effects. This suggests that vector-length-aware packing is most effective when the workload remains compute-bound and the packed tiles continue to fit the cache hierarchy.

PDX exhibits a different but related performance pattern. Because SIMD lanes correspond to different vectors rather than different dimensions, there is no per-vector horizontal reduction. Across all evaluated architectures, PDX is never slower than the compared horizontal SIMD kernels and is on average about $VL$2 faster; for $VL$3, it delivers 4–10× speedups on x86 and approximately 2–3× on ARM, while for larger $VL$4 it is consistently about 1.5–2× faster (Kuffo et al., 6 Mar 2025). These gains are strongest when only a subset of dimensions is scanned, which is precisely the regime exploited by pruning-based approximate search.

The storage-oriented case shows a complementary performance law: when vectors are large, packing vectors together with adjacency lists amplifies I/O costs. NAVIS reports that, on MSMARCO with 768-dimensional vectors, wasted vector reads reach up to 44.34 % of total read bytes per insertion and wasted vector writes up to 74.23 % of write bytes per insertion under the packed-page design (Song et al., 12 May 2026). By decoupling edge and vector storage and supporting selective vector reads, NAVIS enhances average insertion throughput by up to 2.74x, average concurrent search throughput by up to 1.37x, and reduces average search latency by up to 25.26% (Song et al., 12 May 2026).

Portable packed-data support at the language level also reports a measurable traversal benefit: the Haskell library allows traversing packed data up to 60% faster than unpacked data (Jamet et al., 28 Apr 2025). The abstract attributes this to the compact representation of the data in memory and the associated locality benefits.

6. Trade-offs, misconceptions, and research directions

A recurring misconception is that vector-length-aware packing is simply a version of fixed-width packing with symbolic constants. The evidence indicates a stronger requirement: the microkernel’s instruction-level access pattern and the packed layout must be co-designed so that the pieces consumed in each iteration are contiguous for any legal $VL$5 (Beysel et al., 12 May 2026). If the layout is still fixed-width internally, vector-length agnosticism degrades into partial-lane handling, remainders, or non-portable performance.

Another misconception is that packed layouts are universally preferable. Each cited system identifies explicit downside cases. In scalable SVE compilation, current hardware may expose VL=128 bits, so VLA execution introduces additional complexity without immediate vector-width advantages over NEON, and fixed-width autovectorization can sometimes yield better alignment and less predication overhead (Beysel et al., 12 May 2026). In vector search, PDX assumes enough candidate vectors per block to fully utilize SIMD and that reformatting data into PDX is acceptable; in thin or highly dynamic workloads, horizontal layouts may remain simpler (Kuffo et al., 6 Mar 2025). In on-SSD graph search, decoupling increases indirection and shifts correctness and performance onto selective reading, caching, and update policies (Song et al., 12 May 2026).

Padding and memory overhead are also intrinsic rather than incidental. Packed tensor layouts operate on rounded-up tile counts and therefore include padded elements for incomplete tiles (Beysel et al., 12 May 2026). NAVIS similarly shows that a page-packed representation can waste capacity through 4 KiB alignment and by rewriting unchanged vector bytes during edge updates (Song et al., 12 May 2026). The central design question is therefore not whether padding exists, but whether the chosen packing moves unavoidable overhead away from the latency-critical inner loops.

Several future directions are explicit in the sources. For scalable tensor compilation, the reported work highlights extension beyond FP32 to BF16 and INT8, dedicated packed layouts for convolutions avoiding im2col, improved microkernel designs and tuning, and applicability to other scalable vector architectures such as RVV and SME (Beysel et al., 12 May 2026). For vector search, the PDX study suggests general principles: vectorize over vectors rather than dimensions, choose block sizes that match SIMD and register capacity, separate distance kernels from pruning logic, and avoid on-the-fly transposition via gathers, which was measured as 2–130× slower than physically laying out data in PDX (Kuffo et al., 6 Mar 2025). For packed-language support, the Haskell library’s implementation approach is described as general and easily usable with programming languages that support higher-kinded types (Jamet et al., 28 Apr 2025).

Taken together, these works indicate that vector-length-aware packed data layouts are best understood as a family of co-designed representation strategies. Their common feature is not a single storage format, but an invariance principle: the physical layout is chosen so that the dominant execution mechanism—scalable SIMD microkernels, multi-vector distance loops, or graph traversals over SSD pages—encounters contiguous, locality-preserving data at its natural granularity (Beysel et al., 12 May 2026, Kuffo et al., 6 Mar 2025, Song et al., 12 May 2026).