Pretzel Architecture ML Serving

• Pretzel Architecture is a white-box approach to ML inference that exposes pipeline structure for end-to-end optimizations and resource sharing.
• It fuses and compiles entire pipelines ahead-of-time, reducing memory usage and tail-latency through cross-pipeline operator sharing.
• The design incorporates adaptive scheduling and vector pooling, yielding significant throughput improvements and stable latency under load.

The Pretzel architecture is a white-box ML prediction serving system designed to support efficient, scalable, and low-latency deployment of ML inference pipelines. Unlike traditional black-box serving approaches that encapsulate model pipelines as opaque servables, Pretzel exposes and exploits internal pipeline structure for cross-pipeline sharing, end-to-end optimizations, and adaptive resource management. This approach yields significant improvements in memory efficiency, tail-latency, and throughput, and substantially alters the operational paradigm for production ML inference platforms (Lee et al., 2018).

1. Motivation and Departure from Black-Box Paradigms

Production ML serving traditionally wraps trained pipelines—often directed acyclic graphs (DAGs) of featurizers and models defined in frameworks such as ML.Net, TensorFlow, or Scikit-learn—within black-box containers (e.g., Docker, TensorFlow Servable). This abstraction simplifies deployment but restricts system-level optimization. Black-box systems cannot:

• Share parameters or operators across pipelines, leading to multiplied memory costs.
• Optimize across operator barriers, incurring per-operator invocation, intermediate allocations, and runtime reflection or JIT overhead.
• Provide consistent tail-latency, with cold starts suffering from pipeline initialization and code compilation.

Pretzel replaces this with a white-box approach that compiles entire pipelines ahead-of-time (AOT), shares featurizers and models, and pools resources, thereby directly addressing core operational bottlenecks (Lee et al., 2018).

2. Architectural Components and Prediction Workflow

Pretzel is structured into four components:

• FrontEnd: Receives gRPC/HTTP inference requests, optionally performing delayed batching and caching.
• Runtime and Scheduler: Hosts compiled model plans, manages thread ("Executor") and vector pools, and orchestrates staged execution.
• ObjectStore: Acts as a global repository for parameters (dictionaries, weight vectors, PCA transforms) to be shared across pipelines.
• Oven (with Flour IR): Offline optimizer and compiler. Transforms ML.Net pipelines into Flour IR, fuses and optimizes them into model plans, and emits AOT-compiled physical stages.

Prediction-Time Workflow:

1. FrontEnd accepts (pipelineID, inputRecord).
2. Runtime retrieves the corresponding ModelPlan, encompassing staged, parameterized physical functions and ObjectStore references.
3. Scheduler enqueues stage requests based on pipeline topology into priority queues (highPri/lowPri).
4. Executors dequeue, bind vectors from pools, and invoke compiled stages without per-request allocation or JIT.
5. The pipeline executes through all stages, returning the final output.

3. White-Box Optimizations: Fusion, Sharing, and Resource Allocation

Pretzel implements three key classes of optimization:

a) End-to-End Pipeline Fusion:

Oven's optimizer classifies pipeline operators as memory-bound (e.g., tokenizers, hashers) or compute-bound (e.g., matrix operations). Memory-bound operators are fused into single, cache-optimized passes, while compute-bound ones are isolated for SIMD vectorization. For example, a standard text pipeline (Tokenizer→CharNgram, Tokenizer→WordNgram→LinearModel) is split into two fused stages, eliminating unnecessary concatenation and reducing data movement.

b) Multi-Model Sharing:

ObjectStore deduplicates parameters by checksum, so shared components (e.g., a common Tokenizer or WordNgram dictionary) are loaded only once, regardless of the number of pipelines utilizing them. Physical code for a logical stage is shared whenever the operator type and parameter identity match across pipelines, substantially reducing system-wide memory usage.

Mathematically, for 250 sentiment-analysis pipelines:

• White-box memory: $$M_{wb} ≈ 1~|ObjectStore| + 1~|PhysicalCode| + N~|vector~pools|$$
• Black-box wrapping: $M_{bb} ≈ 250 × (model~DLL + runtime)$

Resulting in up to a 25× memory reduction.

c) Adaptive Resource Allocation:

Pretzel's scheduler maintains global FIFO queues for high- and low-priority events. Executors repeatedly dequeue and process available stage events, binding to the most urgent requests and preventing head-of-line blocking. Reservation mode allows resource isolation at the pipeline level while still leveraging sharing. Sub-plan materialization permits caching and reuse of intermediate results when pipelines encounter identical stages and input hashes.

4. Data Flow and Compiler Internals

Pretzel's Oven module compiles Flour IR representations into a DAG of LogicalStages. Each LogicalStage defines:

• Transformation sequence $\{f_1,\,f_2,\ldots\}$
• Input/output vector schemas
• Training-derived statistics (e.g., vector lengths, sparsity)

The Model Plan Compiler maps logical to physical stages—a physical stage is an AOT-compiled parametric function:

$$\Phi_i(\mathrm{vector}_{\mathrm{in}},\,\mathrm{param}_{\mathrm{ptr}}) \to \mathrm{vector}_{\mathrm{out}}$$

Parameters are retrieved from ObjectStore, and all executions occur with preallocated vectors, minimizing runtime overhead.

Runtime flow: For each input, Stage$_0$ is scheduled with vector$_0$; upon execution, Stage$_1$ is scheduled with the output vector, and so on until prediction completion.

5. Scheduling Algorithms and Resource Management

Pretzel's event scheduler utilizes concurrent highPri and lowPri queues, late-binding outgoing events to available Executors (one per core). The simplified executor loop is:

// Global queues highPri ← new ConcurrentQueue<Event>() lowPri ← new ConcurrentQueue<Event>() // Executor loop (each CPU core) loop: if highPri.tryDequeue(ev) then execute(ev.stage, ev.inputVector) recycleVectors(ev) for each successor s of ev.stage do lowPri.enqueue(Event(s, s.allocateVectors())) else if lowPri.tryDequeue(ev) then execute(ev.stage, ev.inputVector) recycleVectors(ev) for each successor s of ev.stage do lowPri.enqueue(Event(s, s.allocateVectors())) else cpu_relax()

Reservation scheduling allows explicit vector or thread locking by pipeline for workload isolation. Sub-plan materialization enables persistent caching of physical stage outputs, so that if later requests match exactly (operator and input), the precomputed vectors are shared.

6. Experimental Validation and Quantitative Outcomes

Evaluation was performed on two ML pipeline workloads—250 sentiment-analysis (SA) and 250 attendee-count (AC) DAGs—using ML.Net pipelines on a dual 8-core Xeon-E5, 32 GB RAM environment. Key quantitative findings:

Metric	Black-Box ML.Net	Pretzel White-Box	Ratio / Speedup
Memory (250 SA)	OOM after 75	1.3 GB (250 fit)	25× smaller
99th-pct Latency (SA)	1.7 ms	0.53 ms	3.2× speedup
99th-pct Latency (AC)	7.4 ms	2.4 ms	3.1× speedup
Throughput (SA, qps)	3.7 K	9.6 K	2.6×
Throughput (AC, qps)	3.7 K	37 K	10×
Cold vs. Hot Ratio (SA)	13.3×	4.2×	~3× improvement
Subplan Materialization	—	2.0× speedup (80% SA pipelines)	—

Under heavy, Zipf-distributed load, throughput scales linearly to 25 K qps on 13 cores, and latency for sensitive workloads degrades gracefully. Reservation scheduling ensures that tail-latency remains stable even under mixed scenarios (Lee et al., 2018).

7. Implications, Limitations, and Future Directions

Pretzel demonstrates that exposing the internal structure of inference pipelines yields order-of-magnitude improvements in operational efficiency for ML serving platforms. White-box compilation, multi-model parameter/code sharing, and adaptive scheduling are core contributors to these gains.

Nevertheless, several open challenges remain:

• Extending the Oven/Flour IR and compiler architecture beyond ML.Net (targeting ONNX-style or heterogeneous backends such as TensorFlow, PyTorch, XGBoost).
• Automated code generation for new transforms, including SIMD templates and accelerators (GPU/FPGA).
• NUMA-aware/distributed multi-node extensions with per-stage partitioning.
• First-class white-box runtime support for "best-effort" strategies, including ensembles and early-exit inference.

A plausible implication is that scaling inference to internet-scale, multi-model deployments requires abandonment of the black-box mindset in favor of cross-pipeline optimizations, transparent DAG structure, and holistic scheduling mechanisms (Lee et al., 2018).