Program-of-Layers (PoLar): Dynamic LLM Execution

• Program-of-Layers (PoLar) is an inference framework that treats pretrained LLM layers as composable atomic functions, allowing input-specific dynamic execution.
• It employs a segmentation approach with operations like skip, keep, and repeat to form alternative layer execution programs that improve accuracy and efficiency.
• Empirical evaluations on benchmarks such as DART-Math demonstrate significant accuracy gains alongside reduced latency and robust out-of-distribution performance.

Searching arXiv for the cited PoLar paper and closely related context papers. Program-of-Layers (PoLar) is an inference framework for LLMs in which the layers of a pretrained model are treated as a library of atomic functions that can be composed into an input-specific execution program, rather than being executed only in the default fixed order and fixed depth. In this formulation, pretrained layers can be skipped or looped, so that a frozen model admits multiple latent computations beyond the standard forward pass; empirically, many of these alternative programs are shorter than the original execution, and some correct errors made by standard inference (Li et al., 4 Jun 2026).

1. Definition and formal structure

PoLar formalizes a pretrained LLM with $D$ transformer layers as a collection of functions

$$f_i : \mathbb{R}^{T \times d} \rightarrow \mathbb{R}^{T \times d}, \quad i \in \{0,\ldots,D-1\}.$$

The conventional execution path is the fixed-depth program

$\pi_0 = (0,1,\ldots,D-1),$

whereas PoLar permits an input-dependent program

$$\pi = (i_1, i_2, \ldots, i_K), \quad i_k \in \{0,\ldots,D-1\},$$

with induced computation

$F_{\pi} = f_{i_K} \circ \cdots \circ f_{i_1}.$

Within this formulation, skipping omits layers from the default sequence, and repeating or looping reuses layers. The central conjecture is that, for a given input, multiple distinct executions of programs-of-layers can yield correct predictions. This shifts the object of analysis from the pretrained parameters alone to the pair consisting of a frozen pretrained model and an execution program over its layers.

The learned PoLar method does not predict arbitrary layer-index sequences directly. Instead, it represents execution programs by partitioning depth into contiguous segments

$$[0=s_1, s_2),\ [s_2, s_3),\ \ldots,\ [s_M, s_{M+1}=D),$$

with a maximum segment length

$s_{j+1}-s_j \le K_{\max},$

and assigns each segment one of three operations: $\{\textsf{skip},\textsf{keep},\textsf{repeat}\}.$ The semantics are

$$\textsf{skip} : \emptyset,\qquad \textsf{keep} : [s_j,\ldots,s_{j+1}-1],\qquad \textsf{repeat} : [s_j,\ldots,s_{j+1}-1,\ s_j,\ldots,s_{j+1}-1].$$

This design constrains the program space to short contiguous modules with bounded local recurrence. The stated constraint is $K_{\max}=4$, and the repeat operator corresponds to one additional pass through the selected segment (Li et al., 4 Jun 2026).

2. Training-free discovery of alternative executions

Before introducing a learned predictor, PoLar establishes the existence of alternative valid programs through a training-free analysis based on Monte Carlo Tree Search. Search begins from the default program $$f_i : \mathbb{R}^{T \times d} \rightarrow \mathbb{R}^{T \times d}, \quad i \in \{0,\ldots,D-1\}.$$0 and explores skip and repeat operations over contiguous subsequences. In the MCTS experiments, the skip length $$f_i : \mathbb{R}^{T \times d} \rightarrow \mathbb{R}^{T \times d}, \quad i \in \{0,\ldots,D-1\}.$$1 and repeat count $$f_i : \mathbb{R}^{T \times d} \rightarrow \mathbb{R}^{T \times d}, \quad i \in \{0,\ldots,D-1\}.$$2 are bounded by $$f_i : \mathbb{R}^{T \times d} \rightarrow \mathbb{R}^{T \times d}, \quad i \in \{0,\ldots,D-1\}.$$3.

A program is valid for an input if it yields the correct prediction, with reward

$$f_i : \mathbb{R}^{T \times d} \rightarrow \mathbb{R}^{T \times d}, \quad i \in \{0,\ldots,D-1\}.$$4

The appendix gives the tree policy as

$$f_i : \mathbb{R}^{T \times d} \rightarrow \mathbb{R}^{T \times d}, \quad i \in \{0,\ldots,D-1\}.$$5

balancing cumulative reward, exploration, and a preference for shorter programs.

The principal empirical findings are quantitative. On DART-Math, Loop consistently outperforms Skip, and Skip+Loop is best in every reported setting. For example, on DART-Math level DM-1, LLaMA-3.2-3B-Instruct improves from 37.9 for Base to 84.7 for Skip+Loop; Qwen2.5-3B-Instruct improves from 25.4 to 87.4; and Qwen3-8B improves from 40.7 to 91.3. On DM-5, the corresponding Base $$f_i : \mathbb{R}^{T \times d} \rightarrow \mathbb{R}^{T \times d}, \quad i \in \{0,\ldots,D-1\}.$$6 Skip+Loop changes are 27.1 \rightarrow 59.1, 1.2 \rightarrow 44.5, and 8.8 \rightarrow 45.7.

The search analysis also characterizes the structure of valid programs. Among inputs already correct under the original model, 75.5% admit shorter valid programs. Among inputs initially wrong but corrected by an alternative program, 36.2% admit shorter correcting programs. Segment structure is highly local: 54.5% of segments consist of a single layer, over two-thirds contain at most two consecutive layers, and segments with predominantly non-consecutive layers account for less than 3.2% of cases. Most segments are repeated at most once. These observations motivate the learned segment-and-operation representation used by PoLar (Li et al., 4 Jun 2026).

3. Program prediction network and execution procedure

Because MCTS is too expensive for routine inference, PoLar introduces a lightweight prediction network that outputs a program before the frozen LLM is executed. The base LLM remains fully frozen; only the predictor is trained.

The predictor first encodes the input $$f_i : \mathbb{R}^{T \times d} \rightarrow \mathbb{R}^{T \times d}, \quad i \in \{0,\ldots,D-1\}.$$7 with the frozen embedding model Qwen3-Embedding-0.6B, producing

$$f_i : \mathbb{R}^{T \times d} \rightarrow \mathbb{R}^{T \times d}, \quad i \in \{0,\ldots,D-1\}.$$8

followed by projection

$$f_i : \mathbb{R}^{T \times d} \rightarrow \mathbb{R}^{T \times d}, \quad i \in \{0,\ldots,D-1\}.$$9

Each depth position has a learnable layer embedding

$\pi_0 = (0,1,\ldots,D-1),$0

which serves as a query in multi-head cross-attention: $\pi_0 = (0,1,\ldots,D-1),$1 yielding $\pi_0 = (0,1,\ldots,D-1),$2. A lightweight transformer over the depth dimension then produces

$\pi_0 = (0,1,\ldots,D-1),$3

Two heads predict segmentation and operations: $\pi_0 = (0,1,\ldots,D-1),$4

$\pi_0 = (0,1,\ldots,D-1),$5

These define

$\pi_0 = (0,1,\ldots,D-1),$6

Training supervision comes from valid execution programs collected offline by MCTS and parsed into a binary segmentation mask $\pi_0 = (0,1,\ldots,D-1),$7 and operation labels $\pi_0 = (0,1,\ldots,D-1),$8. The losses are a binary cross-entropy segmentation term

$\pi_0 = (0,1,\ldots,D-1),$9

and a masked cross-entropy operation term

$$\pi = (i_1, i_2, \ldots, i_K), \quad i_k \in \{0,\ldots,D-1\},$$0

with total objective

$$\pi = (i_1, i_2, \ldots, i_K), \quad i_k \in \{0,\ldots,D-1\},$$1

At inference time, PoLar deterministically thresholds segmentation logits to produce segment boundaries, enforces the maximum segment length constraint, computes operation probabilities at each segment start, and then applies a small beam search over segment-level operation assignments. The frozen pretrained LLM is executed once per candidate program. This distinguishes PoLar from sequential layer-wise routers: it predicts the execution program upfront, over contiguous layer segments, rather than making local routing decisions during the forward pass (Li et al., 4 Jun 2026).

4. Empirical performance, efficiency, and generalization

PoLar is evaluated on fully frozen instruction-tuned LLMs: LLaMA-3.2-3B-Instruct, Qwen1.5-MoE-A2.7B-Chat, Qwen2.5-3B-Instruct, and Qwen3-8B. The principal in-distribution benchmark is DART-Math, with five difficulty levels DM-1 through DM-5; each level contains 2,000 examples, split into 1,250 train, 250 validation, and 500 test. Evaluation uses direct prompting only, with the model required to output only the final answer as $$\pi = (i_1, i_2, \ldots, i_K), \quad i_k \in \{0,\ldots,D-1\},$$2, and a maximum output length of 50 tokens.

The primary metric is pass@$$\pi = (i_1, i_2, \ldots, i_K), \quad i_k \in \{0,\ldots,D-1\},$$3 accuracy. On LLaMA-3.2-3B-Instruct, pass@1 improves from 42.4 / 28.6 / 27.2 / 27.6 / 28.6 for Base across DM-1 to DM-5 to 46.2 / 30.2 / 28.2 / 28.8 / 30.2 for PoLar. At pass@5, Base (sampling) gives 47.6 / 43.2 / 32.8 / 32.8 / 35.6, whereas PoLar reaches 68.4 / 48.0 / 46.0 / 40.4 / 45.8, for gains of +20.8, +4.8, +13.2, +7.6, and +10.2.

Comparable pass@5 gains appear on the other backbones. For Qwen1.5-MoE-A2.7B-Chat, Base (sampling) gives 40.0 / 25.6 / 18.6 / 15.0 / 11.8 and PoLar gives 62.0 / 44.0 / 33.0 / 25.4 / 23.2. For Qwen2.5-3B-Instruct, Base gives 42.2 / 30.2 / 20.4 / 15.8 / 13.0 and PoLar gives 59.8 / 40.6 / 28.2 / 18.0 / 22.8. For Qwen3-8B, Base gives 48.4 / 27.8 / 13.6 / 13.4 / 10.6 and PoLar gives 55.6 / 44.8 / 25.4 / 18.8 / 17.8.

The efficiency results are also explicit. For Qwen1.5-MoE-A2.7B-Chat with 24 layers, the Base model uses 24.00 average layers and 373.45 ms latency. PoLar on DM-1 uses 23.30 average layers, 311.41 ms latency, 0.83x relative runtime, and an accuracy gain of +5.8. On DM-5, it uses 23.76 average layers, 353.31 ms latency, 0.95x relative runtime, and an accuracy gain of +1.2. The overhead breakdown is: 13.23 ms for one LLM layer, 0.99 ms for the predictor head, 0.11 ms for beam search, 1.95 ms for the encoder, and 3.05 ms total additional overhead, equal to 0.8% of a full forward pass and about 0.23 equivalent layers.

Out-of-distribution evaluation trains PoLar on the union of DART-Math training data across all difficulties and tests zero-shot on ASDiv, MAWPS, and MMLU-Pro subject subsets. For Qwen1.5-MoE-A2.7B-Chat, PoLar improves over Base on all reported OOD tasks, including ASDiv: 59.1 $$\pi = (i_1, i_2, \ldots, i_K), \quad i_k \in \{0,\ldots,D-1\},$$4 63.8, MAWPS: 41.7 $$\pi = (i_1, i_2, \ldots, i_K), \quad i_k \in \{0,\ldots,D-1\},$$5 46.7, MMLU-Pro Math: 13.9 $$\pi = (i_1, i_2, \ldots, i_K), \quad i_k \in \{0,\ldots,D-1\},$$6 18.5, Phys: 15.6 $$\pi = (i_1, i_2, \ldots, i_K), \quad i_k \in \{0,\ldots,D-1\},$$7 20.3, Chem: 13.8 $$\pi = (i_1, i_2, \ldots, i_K), \quad i_k \in \{0,\ldots,D-1\},$$8 18.3, Law: 16.6 $$\pi = (i_1, i_2, \ldots, i_K), \quad i_k \in \{0,\ldots,D-1\},$$9 20.4, and further improvements across Eng, Other, Econ, Health, Psych, Bus, Bio, Phil, and Hist. Appendix results report similarly broad gains on the other backbones (Li et al., 4 Jun 2026).

5. Position within dynamic-depth research and terminological scope

PoLar is situated within adaptive-compute work, but its program space is broader than early-exit and skip-only approaches. Early-exit methods such as FastBERT, DeeBERT, PABEE, and Faster Depth-Adaptive Transformers reduce depth while preserving the original order. Skip-only methods such as ShortGPT, MindSkip, FlexiDepth, and LayerDrop-style approaches omit layers but do not model recurrence. Recurrent-depth systems such as Universal Transformers, looped transformers, and Inner Thinking Transformer rely on architectural redesign or special recurrent training. PoLar differs by applying dynamic execution to frozen pretrained models, with both skipping and repetition inside one unified program space.

The closest comparison identified in the source material is DR.LLM, which also learns from MCTS supervision and supports both skipping and repetition. The stated distinction is that DR.LLM performs sequential layer-wise routing based on intermediate hidden states and is limited to single-layer recurrence, whereas PoLar predicts the entire execution program upfront and operates on contiguous layer segments. This enables patterns such as

$F_{\pi} = f_{i_K} \circ \cdots \circ f_{i_1}.$0

which lie outside a single-layer routing space.

The term “PoLar” also requires disambiguation. It is distinct from “PoLAR: Factorizing Extent and Mode in Latent Actions for Robot Policy Learning” (Jeong et al., 19 Jun 2026), where “PoLAR” expands to Polar Latent Actions with Radial structure and refers to latent-action geometry in robot learning rather than dynamic LLM execution. It is also distinct from “Polar probe linearly decodes semantic structures from LLMs” (Diego-Simón et al., 13 May 2026), which studies a geometric “polar” code in layer-local subspaces and reports that semantic structures are most linearly decodable in middle layers. A plausible implication is that contemporary usage has attached “polar” or “PoLar” to several different research programs—dynamic execution, latent-action geometry, and geometric semantic probing—that are methodologically adjacent in their focus on internal structure, but not terminologically interchangeable (Li et al., 4 Jun 2026).

6. Limitations, caveats, and open directions

Several limitations are explicit. The full program space is exponentially large, which is why MCTS is used only as an offline diagnostic and data-generation mechanism rather than as a practical inference-time procedure. The learned representation is restricted to contiguous segments, $F_{\pi} = f_{i_K} \circ \cdots \circ f_{i_1}.$1, and only the operation set $F_{\pi} = f_{i_K} \circ \cdots \circ f_{i_1}.$2, with recurrence limited to a single additional pass. The paper notes that deeper repeated execution could be represented by extending the vocabulary to $F_{\pi} = f_{i_K} \circ \cdots \circ f_{i_1}.$3, but does not do so.

The experimental scope is also narrow relative to the generality of the claim. The reported evaluations focus mainly on mathematical reasoning, arithmetic word problems, and MMLU-Pro subject subsets. The work does not provide a detailed per-layer heatmap or semantic analysis of which absolute depth positions are usually skipped or repeated. It also does not present a formal study of activation-distribution shift or instability induced by looping pretrained layers, beyond the empirical observation that reuse can help and that most useful loops are shallow.

The impact discussion identifies additional open questions around robustness, calibration, interpretability, and safety, especially because execution becomes input-dependent. This suggests that PoLar is not only a dynamic-depth method but also a change in the unit of explanation for LLM inference: instead of treating the standard forward pass as the model, it treats the latent computation path as a variable object that must itself be predicted, audited, and interpreted (Li et al., 4 Jun 2026).