---
title: Vector-Length-Aware Packed Data Layouts
url: https://www.emergentmind.com/topics/vector-length-aware-packed-data-layouts
type: topic
---

# Vector-Length-Aware Packed Data Layouts

Searching arXiv for the cited papers and closely related work on vector-length-aware packed data layouts.
arXiv search query: "Vector-Length-Aware Packed Data Layouts scalable packed layouts vector-length agnostic ML code generation"
Vector-length-aware packed data layouts are memory organizations in which the physical arrangement of data is defined with explicit reference to a vector length parameter, typically the hardware vector length $VL$ of a scalable SIMD ISA or an application-level block width. In the most direct formulation, the layout of packed tensors is parameterized by tile sizes $m_r$, $n_r$, and $k_r$ that are functions of $VL$, so that the contiguous memory regions consumed by each microkernel iteration remain correctly shaped as $VL$ changes across hardware implementations [2605.12445]. More broadly, the term also covers packed organizations that restructure vectors or graph records so that contiguous slices match the execution granularity of distance kernels, pruning algorithms, or storage devices, thereby preserving locality while reducing unnecessary data movement [2503.04422] [2605.11523]. In language-level systems, packed data denotes a unified representation between memory and the network, removing marshalling steps and improving locality; a Haskell library implementation reports traversing packed data up to 60% faster than unpacked data [2504.20166].

## 1. Definition and conceptual scope

In scalable vector compilation, a vector-length-aware packed layout is a memory layout for tensors, especially matrices, whose tile sizes and packing factors are explicit functions of the hardware vector length $VL$, rather than fixed compile-time constants [2605.12445]. The essential property is not merely contiguity, but contiguity of exactly those blocks that the microkernel reads or writes in one iteration. If a kernel consumes $2 \cdot VL$ FP32 elements from one operand per inner iteration, the packed layout groups those $2 \cdot VL$ elements contiguously for any legal $VL$.

This distinguishes vector-length-aware packing from conventional fixed-width packed layouts. In fixed-width SIMD compilation, layout choices are specialized to a compile-time constant vector width, such as a fixed $8 \times 12$ microkernel. Under vector-length-agnostic execution, the compiler does not know whether deployment will use $VL=4$, $VL=8$, or $VL=16$ FP32 lanes, so the layout and the microkernel must both remain symbolic in $VL$ [2605.12445].

The broader packed-data literature uses a compatible but more general notion. Packed data is a unified representation between memory and the network that removes serialization and deserialization steps and can improve performance through compactness and locality [2504.20166]. This suggests that vector-length-aware packed layouts are a specialization of packed representations in which the packed form is additionally constrained by vector execution semantics.

A useful distinction is between three deployment regimes. In tensor compilers, vector length is a hardware property of scalable SIMD and the layout is parameterized by $VL$ directly. In vector similarity search, the relevant execution granularity is often a fixed block size chosen to match SIMD and register capacity; the PDX layout uses `PDX_BLOCK_SIZE = 64` and organizes dimensions vertically within each block [2503.04422]. In on-SSD graph-based vector search, “awareness” concerns the cost of full vector payloads as dimensionality $d$ grows, and layout design decouples vectors from adjacency metadata so that traversal does not force full-vector I/O [2605.11523].

## 2. Formal layout models and packing transformations

The canonical formalization appears in packed matrix multiplication. For a row-major matrix $A \in \mathbb{R}^{M \times K}$, a classical packed layout chooses microkernel tile sizes $m_r$ and $k_r$ and defines

$$
A^\text{pack} \in \mathbb{R}^{\lceil M/m_r \rceil \times \lceil K/k_r \rceil \times m_r \times k_r},
$$

with indexing relation

$$
A^\text{pack}[i_o, k_o, i_i, k_i] = A[i_o m_r + i_i,\; k_o k_r + k_i].
$$

Analogous packing is applied to $B \in \mathbb{R}^{K \times N}$ using tile sizes $k_r, n_r$ and to $C \in \mathbb{R}^{M \times N}$ using $m_r, n_r$ [2605.12445]. This 4D packing makes each microkernel tile contiguous and explicit in the IR.

The vector-length-aware extension preserves the same tensorization but replaces constant tile sizes with functions of vector length:

$$
m_r = f_m(VL), \qquad n_r = f_n(VL), \qquad k_r = f_k(VL).
$$

The packed tensor remains

$$
A^\text{pack} \in \mathbb{R}^{\lceil M/m_r \rceil \times \lceil K/k_r \rceil \times m_r \times k_r},
$$

but its logical shape and index mapping now depend on $VL$ through the microkernel configuration [2605.12445]. This is the defining mathematical move: the structure of packing is unchanged, while the sizes of the inner packed tiles become vector-length-parametric.

The same section formalizes rounded-up outer dimensions

$$
M_o = \left\lceil \frac{M}{m_r} \right\rceil,\quad
K_o = \left\lceil \frac{K}{k_r} \right\rceil,\quad
N_o = \left\lceil \frac{N}{n_r} \right\rceil.
$$

Because tiles are defined over rounded-up dimensions, packed layouts inherently incorporate padding semantics. The paper emphasizes that when register-level tile sizes are aligned with these layouts, computations can be performed without additional masking, since out-of-bounds elements are represented explicitly in the packed data [2605.12445].

A different but related packing model appears in vector search. PDX stores multiple vectors in one block and, within the block, stores all values of dimension $d$ contiguously. For a block of $N$ vectors of dimensionality $D$, the offset of element $(i,d)$ is

$$
\text{offset}(i,d) = d \cdot N + i.
$$

This is a PAX-like layout specialized for vectors: dimensions are mini-columns within a block of vectors [2503.04422]. Here the “inner packed dimension” is not a scalable hardware $VL$, but the block size $N$ selected so that the vectors-inner loop can be auto-vectorized efficiently.

## 3. Microkernel alignment, SIMD execution, and compiler representation

A representative FP32 SVE microkernel computes an $8 \times 2VL$ output tile per inner-loop iteration. In the reported design, it consumes from $A$ one column of length 8, split into two groups of 4 and broadcast via `svld1rq_f32`, and from $B$ one row of length `2*VL`, loaded as two contiguous `svfloat32_t` vectors. The implied tile parameters are therefore $m_r = 8$, $n_r = 2VL$, and $k_r = 1$ per inner iteration [2605.12445]. The layout of $B$ must store, for each $k$, contiguous blocks of $2VL$ FP32 elements corresponding exactly to those loads, while the layout of $C$ must store the output vectors contiguously at offsets `0, vlen, 2*vlen, ..., 15*vlen`.

The corresponding compiler pipeline represents these transformations explicitly. `linalg.pack` and `linalg.unpack` realize the layout transformation from logical matrices to packed 4D tensors and back, while `linalg.mmt4d` expresses matrix multiplication on packed tensors [2605.12445]. Internally, scalable vector dependence is represented through scalable vector types such as `vector<[vscale x 4]xf32>` in MLIR’s `vector` dialect and target-specific `arm_sve` operations, which lower through LLVM scalable vector types carrying the `vscale` notion.

This explicit representation matters because tiling, fusion, and vectorization passes can remain layout-aware. The compiler chooses a predefined microkernel configuration for each data type and target; the configuration includes symbolic functions $f_m$, $f_n$, and $f_k$ mapping $VL$ to tile sizes. Higher-level blocking factors $T_M$, $T_N$, and $T_K$ are then chosen largely independently for cache locality and multithreading [2605.12445]. The result is a separation between register-level vector-length-parametric packing and cache-level blocking.

The same section also clarifies boundary behavior. Packed tiles over rounded-up dimensions allow the core GEMM to run mostly with the all-true SVE predicate `svptrue_b32()`, minimizing mask overhead. This is a common misconception: scalable vectors do not inherently imply pervasive predication inside the hot microkernel. With vector-length-aware packed layouts, the main remainder handling can be displaced into pack and unpack operations, leaving the central compute loop to operate on full packed tiles [2605.12445].

A language-level counterpart exists in portable packed-data support. The Haskell `packed-data` library provides type safe building and reading of packed data in a functional style, does not rely on compiler modifications, and leverages meta-programming so programmers can pack their own data types [2504.20166]. A plausible implication is that such a library-level approach provides an implementation substrate for expressing vector-aware packed formats without requiring a custom compiler, although the supplied abstract does not claim direct SIMD or scalable-vector support.

## 4. Alternative layout families beyond dense tensor compilation

Vector-length-aware packed layouts are not confined to tensor compilers. PDX is a vector data layout for similarity search that stores multiple vectors in one block and uses a vertical layout for dimensions, with the default `PDX_BLOCK_SIZE = 64` [2503.04422]. For `float32`, each dimension slice is $64 \times 4 = 256$ bytes; on AVX-512, each slice fits exactly in four 64-byte SIMD loads. The key loop structure is dimension-outer and vectors-inner: for each dimension $d$, the implementation loads the contiguous slice of that dimension across the block’s vectors and updates a `distances[64]` array.

This organization alters how SIMD is exploited. Horizontal layouts vectorize across dimensions within one vector and therefore require sufficiently large $D$ to fill registers efficiently. PDX vectorizes across vectors instead, so it gets full register utilization even when only a small number of dimensions is scanned, which is especially important for dimension-pruning methods [2503.04422]. The paper reports that PDX beats SIMD-optimized distance kernels on standard horizontal vector storage, is on average 40% faster, and only relies on scalar code that gets auto-vectorized [2503.04422].

On-SSD graph-based vector search presents a different layout problem. Prior systems frequently use a single packed record per vertex, `[Vector] [Degree] [Edge List]`, co-locating the full vector and adjacency list within a 4 KiB SSD page. NAVIS identifies two systemic limitations of this design: packed layouts couple every edge fetch to a full vector load, and static entrance graphs drift away from newly inserted regions as updates accumulate [2605.11523]. Its response is locality-driven decoupling: an edgelist file stores adjacency lists and related metadata, a vector file stores only full-precision vectors, and an in-memory indirection table maps vertex IDs to edge and vector locations.

The significance of this redesign is that traversal can read only edgelist pages while distances are computed from in-memory PQ codes; full vectors are read selectively only during reranking [2605.11523]. This is not vector-length-parametric in the SVE sense, but it is vector-length-aware in the storage sense: as dimensionality $d$ grows, full vectors dominate page cost, so the layout isolates the $O(d)$ payload from the smaller, high-reuse graph metadata.

| Domain | Layout principle | Representative paper |
|---|---|---|
| Scalable GEMM code generation | Tile sizes $m_r,n_r,k_r$ as functions of $VL$ | [2605.12445] |
| Vector similarity search | Dimensions packed vertically within blocks of vectors | [2503.04422] |
| On-SSD graph search | Edgelists decoupled from full vectors for selective reads | [2605.11523] |
| Portable language support | Unified packed representation with type-safe construction and traversal | [2504.20166] |

## 5. Performance characteristics and empirical behavior

The principal performance rationale for vector-length-aware packed layouts is that the execution unit sees data in exactly the order and granularity it consumes. In the scalable SVE pipeline, this means contiguous `VL`-dependent tiles that support `svld1`, `svst1`, and `svmla_lane_f32` without gather/scatter in the core microkernel [2605.12445]. The paper reports that, on real-world ML workloads on Arm CPUs, the resulting SVE code is competitive with and often outperforms existing NEON-based code generation within IREE, achieving up to $1.45\times$ speedup, and that simulator studies show scaling with increasing SVE vector length on compute-bound workloads [2605.12445].

The scaling behavior is especially revealing. For square FP32 matmuls up to `1024×1024×1024`, simulated SVE-512 achieves up to `3.36×` speedup over SVE-128; for `2048×2048×512`, SVE-512 achieves `3.44×` [2605.12445]. These numbers are close to the ideal `4×` scaling that would result from quadrupling vector width, though they are limited by higher reduction latencies and memory hierarchy effects. This suggests that vector-length-aware packing is most effective when the workload remains compute-bound and the packed tiles continue to fit the cache hierarchy.

PDX exhibits a different but related performance pattern. Because SIMD lanes correspond to different vectors rather than different dimensions, there is no per-vector horizontal reduction. Across all evaluated architectures, PDX is never slower than the compared horizontal SIMD kernels and is on average about $2\times$ faster; for $D \le 32$, it delivers `4–10×` speedups on x86 and approximately `2–3×` on ARM, while for larger $D$ it is consistently about `1.5–2×` faster [2503.04422]. These gains are strongest when only a subset of dimensions is scanned, which is precisely the regime exploited by pruning-based approximate search.

The storage-oriented case shows a complementary performance law: when vectors are large, packing vectors together with adjacency lists amplifies I/O costs. NAVIS reports that, on MSMARCO with 768-dimensional vectors, wasted vector reads reach up to `44.34 %` of total read bytes per insertion and wasted vector writes up to `74.23 %` of write bytes per insertion under the packed-page design [2605.11523]. By decoupling edge and vector storage and supporting selective vector reads, NAVIS enhances average insertion throughput by up to `2.74x`, average concurrent search throughput by up to `1.37x`, and reduces average search latency by up to `25.26%` [2605.11523].

Portable packed-data support at the language level also reports a measurable traversal benefit: the Haskell library allows traversing packed data up to 60% faster than unpacked data [2504.20166]. The abstract attributes this to the compact representation of the data in memory and the associated locality benefits.

## 6. Trade-offs, misconceptions, and research directions

A recurring misconception is that vector-length-aware packing is simply a version of fixed-width packing with symbolic constants. The evidence indicates a stronger requirement: the microkernel’s instruction-level access pattern and the packed layout must be co-designed so that the pieces consumed in each iteration are contiguous for any legal $VL$ [2605.12445]. If the layout is still fixed-width internally, vector-length agnosticism degrades into partial-lane handling, remainders, or non-portable performance.

Another misconception is that packed layouts are universally preferable. Each cited system identifies explicit downside cases. In scalable SVE compilation, current hardware may expose `VL=128 bits`, so VLA execution introduces additional complexity without immediate vector-width advantages over NEON, and fixed-width autovectorization can sometimes yield better alignment and less predication overhead [2605.12445]. In vector search, PDX assumes enough candidate vectors per block to fully utilize SIMD and that reformatting data into PDX is acceptable; in thin or highly dynamic workloads, horizontal layouts may remain simpler [2503.04422]. In on-SSD graph search, decoupling increases indirection and shifts correctness and performance onto selective reading, caching, and update policies [2605.11523].

Padding and memory overhead are also intrinsic rather than incidental. Packed tensor layouts operate on rounded-up tile counts and therefore include padded elements for incomplete tiles [2605.12445]. NAVIS similarly shows that a page-packed representation can waste capacity through 4 KiB alignment and by rewriting unchanged vector bytes during edge updates [2605.11523]. The central design question is therefore not whether padding exists, but whether the chosen packing moves unavoidable overhead away from the latency-critical inner loops.

Several future directions are explicit in the sources. For scalable tensor compilation, the reported work highlights extension beyond FP32 to BF16 and INT8, dedicated packed layouts for convolutions avoiding im2col, improved microkernel designs and tuning, and applicability to other scalable vector architectures such as RVV and SME [2605.12445]. For vector search, the PDX study suggests general principles: vectorize over vectors rather than dimensions, choose block sizes that match SIMD and register capacity, separate distance kernels from pruning logic, and avoid on-the-fly transposition via gathers, which was measured as `2–130×` slower than physically laying out data in PDX [2503.04422]. For packed-language support, the Haskell library’s implementation approach is described as general and easily usable with programming languages that support higher-kinded types [2504.20166].

Taken together, these works indicate that vector-length-aware packed data layouts are best understood as a family of co-designed representation strategies. Their common feature is not a single storage format, but an invariance principle: the physical layout is chosen so that the dominant execution mechanism—scalable SIMD microkernels, multi-vector distance loops, or graph traversals over SSD pages—encounters contiguous, locality-preserving data at its natural granularity [2605.12445] [2503.04422] [2605.11523].

Source: https://www.emergentmind.com/topics/vector-length-aware-packed-data-layouts