PRETZEL Architecture: White-box ML Serving

• PRETZEL Architecture is a white-box machine learning prediction serving system that exposes internal pipeline structures for end-to-end optimization.
• It employs a two-phase process where model pipelines are translated and compiled offline and executed online via a high-performance, event-driven scheduler.
• Empirical evaluations demonstrate up to 5.5× reduction in tail latency, 4.7× throughput improvement, and 25× memory savings compared to black-box serving systems.

PRETZEL is a machine learning prediction serving system characterized by a white-box architecture that abandons conventional black-box container-based approaches. Its design exposes the internal structure of model pipelines to a database-inspired stack of compiler, optimizer, and scheduler components. This enables end-to-end and multi-model optimizations that address three central requirements of modern ML-as-a-service (MLaaS) infrastructures: low 99th-percentile tail latency, high aggregate throughput, and graceful performance degradation under skewed or bursty loads. Empirical evaluation demonstrates PRETZEL’s ability to reduce 99th-percentile latency by up to 5.5×, improve throughput by 4.7×, and reduce working-set memory by 25× relative to state-of-the-art black-box serving systems (Lee et al., 2018).

1. High-Level Architecture and Design Objectives

PRETZEL divides pipeline deployment into two phases. In the off-line phase, trained model pipelines are translated, optimized, and compiled into compact "model plans." In the on-line phase, these plans execute in a shared runtime environment governed by a high-performance scheduler. This white-box treatment of pipelines permits:

• Minimization of end-to-end, and especially tail, latency through operator fusion and the elimination of per-request JIT or reflection overhead.
• Maximization of throughput and CPU-cache locality by ahead-of-time (AOT) compilation of SIMD-friendly code fragments ("physical stages") and reuse of per-core vector buffer pools.
• Support for graceful degradation under skewed load using an event-driven, late-binding scheduler capable of resource reservation for latency-critical pipelines.

The system thus simultaneously addresses the unique prediction-serving challenges in MLaaS contexts, which previous black-box architectures were unable to optimize holistically.

2. Core Components and Interactions

The PRETZEL stack is organized into distinct offline and online components:

• Flour: An intermediate API (LINQ/RDD-style), into which ML.Net pipelines are automatically reified. Each Flour program describes a DAG of logical transformations (tokenization, featurizers, classifiers).
• Oven Optimizer: A rule-based optimizer that rewrites the Flour DAG into logical stages. Rewrite rules include schema propagation, memory-bound featurizer fusion, SIMD vectorization extraction, duplicated branch mergers, and the pushing of linear models through concatenation nodes.
• Model Plan Compiler: This maps logical stages to physical implementations using training-collected statistics (vector size, sparsity). Each physical stage is AOT-compiled into a lock-free, parameterized function. The resulting "model plan" includes logical metadata, physical code, and references to parameter objects.
• Object Store: Hashes and deduplicates all parameter objects (token dictionaries, weight vectors, PCA matrices) across pipelines.
• Runtime: Maintains a catalog of all physical stages and shares them globally. It allocates per-core thread pools ("Executors") and per-Executor vector buffer pools; prediction paths perform no heap allocation.
• Scheduler: Manages work items in low- and high-priority queues. Executors pull from these, and, if configured, enforce per-pipeline reservations for resource-sensitive pipelines.
• Front End: Exposes both a low-latency request/response engine (single JIT call) and a high-throughput batch engine.

A summary table of the key components is as follows:

Component	Function	Phase
Flour	Intermediate pipeline DAG representation	Off-line
Oven Optimizer	Rule-based logical DAG rewrite	Off-line
Model Plan Compiler	Logical-to-physical stage mapping, AOT compile	Off-line
Object Store	Parameter object deduplication	Off-line
Runtime	Physical stage/shared buffer execution	On-line
Scheduler	Event-driven work management	On-line
Front End	Request/response and batch API exposure	On-line

3. Pipeline Representation and Optimization

ML.Net pipelines are represented as DAGs of Flour transformations. The Oven optimizer executes a series of transformation steps:

1. Schema Validation: Ensures node input/output types match.
2. Stage Graph Builder: Aggregates 1-to-1 memory-bound operations into stages and breaks at n-to-1 operator boundaries (e.g., concatenation).
3. Stage Graph Optimizer: Removes common subexpressions, merges duplicate branches, inlines single-node stages, and pushes linear models into earlier featurizer stages to fuse dot-product computations.
4. Output Validation: Assigns sparse/dense and vectorization labels, finalizes stage schemas.

In a canonical example—Tokenize→CharNgram→WordNgram→Concat→LinearRegression—PRETZEL fuses Tokenize+CharNgram and Tokenize+WordNgram into memory-bound stages, pushes linear weights into n-gram stages, and eliminates both the final concatenation and an isolated classifier stage, yielding a streamlined two-stage plan in place of five operators. This end-to-end fusion and optimization is enabled by the white-box approach.

4. Multi-Model Consolidation and Scheduling

PRETZEL consolidates multi-model workloads using techniques adapted from database systems:

• Parameter Deduplication: Serialized operator state is hashed so identical parameter objects are loaded once, even across thousands of pipelines.
• Physical Stage Sharing: Physical stage codes and statistics tags are shared across pipelines, so runtime loads code only once per unique implementation.
• Sub-Plan Materialization: Caches outputs of shared featurization stages on a best-effort, LRU basis, akin to materialized views in RDBMSs.
• Event-Driven Scheduler: Executors per core pull from two prioritized queues, ensuring prompt pipeline completion ("red-hot" work) while deprioritizing new arrivals. Reservation slots can be allocated to latency-sensitive models, providing isolation akin to containers.

These strategies provide substantial memory and resource efficiency, enabling high concurrency with reduced duplication of computation and storage.

5. Compiler and Code Generation Techniques

PRETZEL employs a sophisticated set of code generation and runtime optimization strategies:

• Ahead-of-Time Compilation: All physical stages are cross-compiled (e.g., via CrossGen on .NET Core), precluding JIT compilation during inference.
• SIMD Vectorization: Compute-bound stages (e.g., matrix-vector multiplies, dot products) are generated in both dense and sparse forms, with explicit SIMD intrinsics where beneficial.
• Lock-Free Parametric Functions: Each stage is a single function operating on pointers to input/output vectors and a shared parameter object, eliminating per-request reflection and allocation.
• Vector Pooling: Per core, pools of typed buffers (e.g., float[], double[], sparse vectors) are maintained, allowing $O(1)$ acquisition and release per request, avoiding fresh garbage-collection allocations.

These compilation and execution strategies drive the system’s latency and throughput benefits.

6. Quantitative Performance Results

Empirical evaluation across 500 "production-like" pipelines, comparing PRETZEL to a state-of-the-art black-box ML.Net ± Clipper baseline, yielded:

• 99th-percentile latency reduction: up to $5.5 \times$,

$$\frac{L_{99}^{\mathrm{PRETZEL}}}{L_{99}^{\mathrm{baseline}}} = \frac{1}{5.5}$$

• Aggregate throughput improvement: $4.7 \times$,

$$\frac{T_{\mathrm{PRETZEL}}}{T_{\mathrm{baseline}}} = 4.7$$

• Working-set memory reduction: $25 \times$,

$$\frac{M_{\mathrm{PRETZEL}}}{M_{\mathrm{baseline}}} = \frac{1}{25}$$

Microbenchmarks show a reduction in P99 latency from millisecond-scale to sub-millisecond, linear throughput scaling with number of cores (up to ≈25 K qps vs. ≈5 K qps), and under skewed Zipfian workloads, graceful latency growth and the ability to reserve a dedicated core per pipeline to nullify P99 tail increases. Throughput is maintained under resource reservation without aggregate loss (Lee et al., 2018).

7. Trade-Offs, Limitations, and Future Directions

The white-box design of PRETZEL imposes an upfront cost: the requirement for an offline compiler/optimizer (Flour and Oven) and a custom IR currently limits applicability to ML.Net pipelines. Ongoing work aims to generalize this interface (e.g., ONNX front end). The fine-grained stage-level scheduler incurs a small per-stage dispatch overhead, but this is offset by performance gains from code fusion and pooling. Remaining sources of sporadic latency include garbage collection and NUMA placement; further work targets buffer recycling, NUMA-aware Executor assignment, distributed scaling, and heterogeneous (GPU/FPGA) code generation.

The application of techniques traditionally associated with database system implementation—such as query rewriting, multi-query optimization, materialized views, and event-driven scheduling—demonstrates significant advantages in large-scale ML model serving. The PRETZEL architecture evidences that exposing and optimizing the internal structure of ML serving pipelines is central to meeting modern demands for tight tail-latency SLAs, throughput, and robustness to bursty load (Lee et al., 2018).