---
title: 'Snowpark: Managed Compute in Snowflake'
url: https://www.emergentmind.com/topics/snowpark
type: topic
---

# Snowpark: Managed Compute in Snowflake

Searching arXiv for recent Snowpark systems papers to ground the article.
Snowpark is Snowflake’s managed programming and execution environment for running data engineering and AI/ML workloads directly inside Snowflake, using Python and other programming languages. It combines libraries and APIs for user programming, code execution environments for arbitrary user code, and a secure sandboxed runtime that runs alongside Snowflake’s core SQL processing inside Snowflake virtual warehouses. In architectural terms, Snowpark extends Snowflake beyond SQL-only analytics into a broader application and computation platform while preserving the platform’s decoupled compute-storage model, governance controls, and distributed execution substrate [2508.05904].

## 1. Definition and system role

Snowpark is best understood as an in-platform compute layer rather than a language binding. It supports both declarative pushdown into Snowflake SQL and execution of arbitrary user code in secure isolated runtimes located next to the data. The first path is the DataFrame API, which converts DataFrame operations into SQL statements executed by Snowflake’s SQL engine. The second path covers Python stored procedures, User Defined Functions (UDFs), User Defined Table Functions (UDTFs), User Defined Aggregate Functions (UDAFs), and vectorized UDFs using Pandas DataFrame semantics [2508.05904].

This design addresses a recurring operational pattern in which organizations use Snowflake for core analytics, move data out to external systems such as Spark for data engineering or ML, process it externally, and move the results back into Snowflake. Snowpark addresses the latency, ingress/egress cost, security risk, and operational burden associated with that pattern by bringing compute to the data rather than moving data to external compute engines [2508.05904].

A common misconception is that Snowpark is merely “Python in Snowflake.” The system papers explicitly frame it more broadly: Snowpark supports Python and other programming languages; it offers multiple programming models; and it is deeply integrated with Snowflake virtual warehouses, the cloud-services control plane, and Snowflake’s secure execution infrastructure rather than operating as a detached sidecar service [2508.05904].

## 2. Placement in the Snowflake architecture

Snowflake’s underlying architecture has three layers: Data Storage backed by cloud object storage, Virtual Warehouses as the elastic compute clusters that execute workloads, and Cloud Services as the control-plane-like layer that manages metadata, security, optimization, transactions, and warehouse infrastructure. Snowpark runs inside virtual warehouses, reuses the same compute substrate as Snowflake SQL workloads, and leverages Cloud Services for provisioning, metadata access, security and governance, and distributed execution coordination [2508.05904].

This placement is central to Snowpark’s semantics. Snowpark workloads are colocated with Snowflake’s query engine on virtual warehouse nodes, so computation occurs close to the governed data and close to warehouse execution resources. The architecture therefore aims to preserve Snowflake’s elasticity and scalability while avoiding the explicit executor sizing, pod management, and separate cluster lifecycle typical of Spark- or Kubernetes-style systems [2508.05904].

The system’s later evolution reinforces this view. The Snowpark Execution Environment (SEE), and its subsequent redesign SEE++, place a secure sandbox boundary directly on warehouse workers. SEE++ uses a gVisor-based sandbox on each relevant virtual warehouse worker, with namespaces and cgroups, a base image, and gVisor runtime components such as Sentry, Gofer, Netstack, and Systrap. This evolution reflects the need to support a broader class of arbitrary workloads while keeping those workloads co-located with Snowflake-managed compute [2511.12457].

A plausible implication is that Snowpark’s architectural identity is inseparable from Snowflake’s warehouse model: its programming abstractions, sandboxing, scheduling, and networking all derive from the fact that user-authored code executes inside the same governed platform that stores and optimizes the data.

## 3. Programming model and execution semantics

Snowpark exposes several programming models. Python stored procedures support arbitrary Python logic and can issue Snowflake queries. The DataFrame API lets users define transformations in Python and have Snowpark translate them into SQL. UDFs, UDTFs, and UDAFs support reusable custom logic, executed per row, per batch, or per group depending on function type. Vectorized UDFs use Pandas DataFrame semantics. The papers focus on Python, but Snowpark is also described as supporting Java and Scala in addition to Python [2508.05904].

The execution model for Python functions is process-oriented and query-integrated. Python functions run as part of Snowflake SQL queries. At query startup, Snowpark identifies required packages and imports, installs those packages on warehouse nodes inside the Snowpark sandbox, initializes the Python interpreter, and then launches many Python interpreter processes per function to work around Python’s global interpreter lock. Warehouse worker threads communicate with those Python interpreter processes over gRPC and pass rowsets for computation. When the query completes, the sandbox and Python interpreters are cleaned up, while packages and imports remain in a local cache for faster subsequent runs [2508.05904].

This model yields two distinct execution regimes. When logic can be lowered into SQL, Snowpark preserves the benefits of Snowflake’s native execution engine. When logic must remain arbitrary user code, Snowpark executes it in a secure sandbox inside the virtual warehouse. The system papers repeatedly emphasize that these two paths are complementary rather than competing: push down what can become SQL, sandbox what must remain user code [2508.05904].

The same model also explains why Snowpark UDF execution is unusually sensitive to imbalance. UDFs are invoked as part of SQL query execution, but their true cost is often opaque, highly variable row by row, and slower than Snowflake’s native C++ engine because of mandatory serialization across process boundaries. This means that Snowpark’s programmability introduces distributed-systems issues that are more severe than in ordinary SQL execution [2604.13034].

## 4. Isolation, governance, and controlled external connectivity

Because Snowpark executes arbitrary code, its security model is materially different from that of SQL-only analytics. The baseline Snowpark sandbox is described as a multi-layer defense system using namespaces to isolate processes, cgroups to manage CPU and memory resources, syscall filtering to allow only approved or conditionally approved syscalls, and a supervisor process that logs denied syscalls for anomaly detection and threat observation. Network egress policies and external egress proxies, generated by the control plane and enforced at the network edge, further reduce exfiltration risk [2508.05904].

As workload diversity increased, Snowpark transitioned from an in-house syscall-filter-centric sandbox to SEE++, a gVisor-based architecture augmented with targeted optimizations. In SEE++, workloads primarily interact with gVisor’s user-space kernel implementation rather than directly with the host kernel. Sentry emulates much of the Linux syscall interface, Gofer mediates filesystem access using the 9P protocol, Systrap replaces ptrace with a lower-overhead backend using seccomp-bpf and userfaultfd, and a predefined base image standardizes runtime dependencies [2511.12457].

External connectivity is handled by Secure Network Access in Snowpark (SNAS). SNAS provides transparent outbound network access from sandboxed workloads while preserving strict multi-tenant isolation, policy enforcement, resource fairness, and fail-closed behavior. Each sandbox has a dedicated network namespace with a veth pair; an eBPF program attached at the TC egress hook on the compute-node side checks destination IP:port against a job-scoped policy; approved traffic is encapsulated in GENEVE with policy metadata and routed to distributed egress proxies; the proxies independently validate the policy, perform SNAT, and mediate internet access [2606.17533].

This layered design addresses another common misconception: enabling arbitrary user code does not imply relaxing Snowflake’s governance model. The system papers instead describe Snowpark as separating the flexibility of arbitrary user code from the trusted SQL engine and data plane through controlled, sandboxed execution and dual-tier policy enforcement [2508.05904].

## 5. Performance engineering and workload management

Snowpark’s performance work is organized around three recurring costs: query initialization overhead, arbitrary-workload scheduling uncertainty, and imbalance during distributed execution. For Python workloads, package resolution and environment creation are major startup costs. Snowpark therefore uses a global solver cache for conda dependency resolution and an environment cache within each virtual warehouse. The reported production hit rates are 99.95% for the solver cache and 92.58% for the environment cache, and the combined effect accelerates query initialization by a factor of 18 to 48 depending on percentile [2508.05904].

Scheduling arbitrary user code introduces a different problem: unlike SQL operators, user-authored logic does not expose readily analyzable semantics for resource estimation. Snowpark addresses this with historical execution statistics–based estimation, particularly for memory. For a new execution of the same query, the system looks back at the past \(K\) executions’ memory consumption statistics, takes the \(P\) percentile value, multiplies it by a factor \(F\), and uses that as the query’s memory consumption estimation:
\[
\text{estimated memory} = F \times \text{Percentile}_P(\text{past } K \text{ max-memory values})
\]
The paper reports that this reduces OOM rate to below 0.0005% while keeping 90 percentile query queueing time below 5ms in production [2508.05904].

The execution environment itself also underwent performance-sensitive redesign. SEE++ targeted functionality compatibility, performance, and maintainability simultaneously. Two concrete fixes illustrate the level of systems detail involved. A virtual-memory-management optimization aligned file-offset allocation direction with actual address-space growth direction and preserved the last faulted address during VMA merges, reducing VMA entries by 182× on a synthetic benchmark derived from production behavior. An ELF-loader semantic fix changed gVisor’s loader to match Linux zeroing semantics more precisely, resolving segmentation faults for workloads using the prophet package. On TPCx-BB, the modern sandbox architecture showed an overall 1.5% performance improvement over the legacy sandbox [2511.12457].

These mechanisms suggest that Snowpark’s performance envelope depends as much on systems engineering in packaging, runtime isolation, and scheduling as on language-level API design.

## 6. UDF skew, row redistribution, and DySkew

UDF execution is central to Snowpark, and it is also the locus of one of Snowpark’s main systems problems: data skew. In the Snowpark setting, data skew is not merely uneven row counts across partitions; it is uneven effective work. Different rows can trigger expensive Python or Java logic, large payloads, or unusual branching behavior, so one overloaded interpreter or one heavy row can become a straggler. Because the slowest task determines stage completion, skew becomes a first-order performance problem affecting total runtime, tail latency, and CPU utilization [2604.13034].

Before DySkew, Snowpark used a static round-robin mitigation in which rows were redistributed across Python interpreter processes across virtual warehouse nodes during query execution. That approach helped on benchmarks and customer workloads, but the system paper identifies four limitations: equal row counts do not imply equal work; forced redistribution can conflict with data locality and, in some plans, risk incorrectness or data corruption; a bespoke Snowpark-only mechanism diverges from Snowflake’s broader platform strategy; and the old method depended on a special threshold based on historical per-UDF per-row metrics that generalized poorly across varied workloads [2604.13034].

DySkew replaces that strategy with a dynamic, adaptive-link mechanism built on Snowflake’s generalized skew-handling framework. The design goals are fine-grained per-row mitigation, dynamic runtime adaptation, and low-overhead, cost-aware redistribution. Adaptive data links replace strict 1:1 producer-consumer links, and each link instance uses a state machine that can route locally or remotely depending on runtime conditions. Redistribution is constrained by three consumer-declared policies: `Never Distribute`, `Distribute Late`, and `Distribute Early` [2604.13034].

The generalized framework uses several skew-detection models, each with an \(N\)-strikes rule. The row-percentage-based model declares instance \(i\) skewed when
\[
R_i \times O > R_{-i},
\]
where \(R_i\) is the row count for instance \(i\), \(R_{-i}\) is the average row count of sibling instances, and \(O\) is a threshold multiplier. The synchronous-time-spent model compares rates of change over a sliding window and uses the condition
\[
\frac{dS_i}{dt} \times O \ge \frac{dS_{-i}}{dt}.
\]
An idle-time model detects skew directly by observing that sibling instances are idle while the current instance remains active [2604.13034].

For Snowpark specifically, DySkew introduces three workload-specific optimizations. First, Eager Redistribution often bypasses the initial observation phase and begins redistributing rows immediately, because benchmark evidence showed that for Snowpark UDFs the benefit of parallelizing early almost always dominates redistribution cost. Second, a Row Size Model suppresses redistribution when rows are extremely large and the operator is not skewed; this guards against cases in which network and serialization costs dominate, including observed regressions of up to 20× when more than 100 GB of data was redistributed unnecessarily on workloads without meaningful skew. Third, DySkew removes forced self-skipping so that the local worker remains a valid destination, reducing self-exclusion bias and unnecessary network traffic [2604.13034].

The reported results are workload-dependent rather than uniform. On replayed customer queries, there was a slight regression at 2 nodes, but on 4-node and 8-node configurations DySkew showed significant latency reductions, including nearly a 10% improvement in P99 tail latency. On TPCx-BB with 4 nodes, Query 10 improved by 43% and Query 19 by 36%, while other measured UDF-bearing queries changed by less than 5%. In production rollout, redistribution was automatically applied to 37.6% of all Snowpark UDF queries, and the reported total performance improvement after deployment was 20.4% [2604.13034].

## 7. Adoption, product evolution, and practical scope

Snowpark is described in production terms rather than as a niche feature. As of June 2024, over 50% of Snowflake customers used Snowpark, and more than 100 million Snowpark queries ran every day [2508.05904]. The papers position this adoption as evidence that Snowpark has become a major production subsystem for data engineering and AI/ML execution inside Snowflake.

The reported case studies illustrate two different operational roles. In data engineering, Chicago Trading Company moved ETL workloads from managed Spark clusters to Snowpark and reported 54% cost reduction, improved reliability, and meeting daily SLA deadlines for the first time in company history. In AI/ML feature engineering, Fidelity reported gains including 77x acceleration for Min-Max Scaling, 50x improvement for One-Hot Encoding, and 17x gain for Pearson Correlation, which the paper attributes to eliminating data movement, parallelizing transformations, and avoiding external read/write overhead [2508.05904].

Snowpark’s evolution also broadened its execution scope. SEE++ is presented as a key enabler for Snowpark in Snowflake Serverless Tasks and for the Snowflake Artifact Repository, which allows users to reference PyPI packages in Snowpark Python stored procedures and UDFs. SNAS extends that trajectory by enabling transparent socket-based access to external systems such as cloud APIs, external databases, feature stores, gRPC services, and orchestration systems while maintaining strict isolation and policy control [2511.12457].

The system papers are also explicit about limitations. Snowpark’s arbitrary-code execution model creates resource-estimation and skew problems that do not arise in the same way for native SQL. DySkew does not provide a complete formal cost model or disclose all threshold values. SNAS enforces network-layer IP:port policy rather than application-layer content policy, and its policies are static during job execution. SEE++ improved compatibility substantially, but the redesign required targeted fixes to gVisor rather than working out of the box [2604.13034].

Taken together, these papers portray Snowpark as Snowflake’s answer to the split between governed warehouse analytics and external programmable compute systems: a managed, secure, distributed execution environment embedded inside virtual warehouses, with SQL pushdown when possible, sandboxed arbitrary code when necessary, and a growing body of systems work devoted to initialization latency, scheduling, skew resilience, secure networking, and modern package compatibility [2508.05904].

Source: https://www.emergentmind.com/topics/snowpark