StackTrans: Transformer with Differentiable Stacks

• StackTrans is a Transformer variant that integrates differentiable stack modules to mimic pushdown automata for modeling hierarchical and recursive structures.
• The architecture leverages explicit push, pop, and no-op operations to improve performance on formal language tasks, achieving higher accuracy on challenging benchmarks.
• It exhibits efficient scaling with modest computational overhead and has inspired related research in program execution visualization and graph layout transformations.

StackTrans most directly denotes a Transformer architecture augmented with explicit, differentiable hidden-state stacks placed between layers in order to introduce the inductive bias needed to model languages in the Chomsky hierarchy, especially deterministic context-free grammars characterized by nested dependencies (Zhang et al., 21 Jul 2025). In adjacent literature, the same label also arises descriptively in two distinct senses: a stack-trace-centered program-execution visualizer, although the 2017 paper does not introduce a tool named StackTrans (Litvinov et al., 2017), and a graph-layout theme concerned with whether stacks “can be transformed into” queues (Katheder et al., 2024). The shared motif across these usages is the explicit treatment of stack structure and stack transitions rather than implicit recovery from generic sequence or layout mechanisms.

1. Nomenclature and research scope

In current arXiv usage represented here, “StackTrans” is the formal name of the architecture introduced in “StackTrans: From LLM to Large Pushdown Automata Model,” where the core modification is the insertion of differentiable stack modules between Transformer layers rather than inside self-attention (Zhang et al., 21 Jul 2025). The objective is to preserve the standard attention pathway while adding a pushdown-like memory mechanism that can represent hierarchical and recursive structure more directly.

The term is not used uniformly across fields. In software-engineering education, a 2017 paper proposes and prototypes a visualization approach whose main element is the stack trace and whose transitions are shown through step-by-step states, but the paper explicitly does not assign a marketing name like StackTrans (Litvinov et al., 2017). In graph theory, “transforming stacks into queues” names a longstanding structural question about linear layouts, stack number, queue number, and mixed number rather than a software system or neural architecture (Katheder et al., 2024).

This suggests that StackTrans is best understood as a family of stack-centric research themes with one canonical instantiation in machine learning and two conceptually adjacent but technically unrelated usages.

2. Formal-language motivation and pushdown interpretation

The neural StackTrans architecture is motivated by a specific limitation attributed to standard Transformers: difficulty in learning recursive and hierarchical patterns tied to the Chomsky hierarchy, including deterministic context-free languages with nested dependencies, and weak extrapolation beyond training lengths on regular-expression and deterministic-context-free tasks (Zhang et al., 21 Jul 2025). The motivating contrast is with deterministic pushdown automata, for which a last-in-first-out store is the minimal mechanism enabling recognition of nested dependencies such as balanced parentheses and reversals.

The relevant pushdown-automaton definition is

$A = (Q, \Sigma, \Gamma, \delta, q_0, Z_0, F),$

with

$$\delta: Q \times (\Sigma \cup \{\varepsilon\}) \times \Gamma \to Q \times \Gamma^*.$$

Within the StackTrans interpretation, Transformer layers act as a learned finite control, token embeddings provide input symbols, and the differentiable stack represents $\Gamma^*$ through vector-valued stack symbols derived from hidden states (Zhang et al., 21 Jul 2025). The learned action distribution over $\{\text{push}, \text{pop}, \text{noop}\}$ functions as a soft relaxation of the transition rule.

The paper does not prove formal equivalence to pushdown automata. Its claim is instead empirical: the architecture mimics pushdown dynamics and solves tasks solvable by PDAs, including regular-expression and deterministic-context-free benchmarks, with length extrapolation (Zhang et al., 21 Jul 2025). That distinction is important. The model is pushdown-like in inductive bias and observed behavior, not a formally verified automata-theoretic realization.

3. Architecture and differentiable stack mechanics

StackTrans inserts a stack module between standard Transformer layers. For each token $t$, a layer produces hidden states $h_t \in \mathbb{R}^d$; the stack consumes $h_t$, updates its content via differentiable push, pop, and no-op operations, and emits an augmented output $h'_t$ for the next layer (Zhang et al., 21 Jul 2025). Self-attention is left unchanged, which preserves compatibility with flash-attention and standard training pipelines.

Action selection is computed by

$a_t = \text{Softmax}(A h_t),$

where $$a_t = [a_t^{push}, a_t^{pop}, a_t^{noop}] \in \mathbb{R}^3$$ gives probabilities for push, pop, and no-op. The stack state is represented by vector slots $$\delta: Q \times (\Sigma \cup \{\varepsilon\}) \times \Gamma \to Q \times \Gamma^*.$$0 and a parallel scalar activation mask $$\delta: Q \times (\Sigma \cup \{\varepsilon\}) \times \Gamma \to Q \times \Gamma^*.$$1 over a stack of size $$\delta: Q \times (\Sigma \cup \{\varepsilon\}) \times \Gamma \to Q \times \Gamma^*.$$2. Both content and mask are updated by convex mixtures of the candidate operations, yielding a fully differentiable relaxation of discrete stack manipulation (Zhang et al., 21 Jul 2025).

A distinctive component is the global read:

$$\delta: Q \times (\Sigma \cup \{\varepsilon\}) \times \Gamma \to Q \times \Gamma^*.$$3

followed by

$$\delta: Q \times (\Sigma \cup \{\varepsilon\}) \times \Gamma \to Q \times \Gamma^*.$$4

The paper reports that top-only peeking proved unstable and limited gradient flow, whereas the learnable query-over-stack attention improves training robustness (Zhang et al., 21 Jul 2025). The mask acts as a differentiable stack pointer, attenuating inactive slots and guarding against invalid operations and padding.

The large-scale design is a multi-head low-rank stack. Hidden states are down-projected, split into $$\delta: Q \times (\Sigma \cup \{\varepsilon\}) \times \Gamma \to Q \times \Gamma^*.$$5 independent subspaces of dimension $$\delta: Q \times (\Sigma \cup \{\varepsilon\}) \times \Gamma \to Q \times \Gamma^*.$$6, updated in parallel, concatenated, and up-projected back to dimension $$\delta: Q \times (\Sigma \cup \{\varepsilon\}) \times \Gamma \to Q \times \Gamma^*.$$7 (Zhang et al., 21 Jul 2025). This architecture is intended to retain stack expressivity while controlling the cost of full-dimensional external memory.

4. Training regime, empirical results, and limitations

For formal-language experiments, StackTrans follows the benchmark setting of Deletang et al. (2022): training lengths are uniformly sampled from 1–40 tokens, testing extrapolates to lengths up to 500 tokens, and token-level accuracy is reported as the best of 10 runs (Zhang et al., 21 Jul 2025). The task suite spans regular, deterministic context-free, and context-sensitive problems. On regular tasks, StackTrans achieves 1.00 on Even Pairs, Parity Check, and Cycle Navigation, whereas the standard Transformer yields 0.49, 0.50, and 0.20. On deterministic context-free tasks, it achieves 0.92 on Stack Manipulation, 1.00 on Reverse String, and 0.60 on Modular Arithmetic, compared with 0.53, 0.55, and 0.30 for the Transformer (Zhang et al., 21 Jul 2025).

For natural language, pretraining follows the OLMo framework using Dolma and SmolLM filtered to approximately 980B tokens, with the primary StackTrans-360M pretrained on approximately 1T tokens (Zhang et al., 21 Jul 2025). The reported zero-shot metrics for StackTrans-360M are HellaSwag 59.5, ARC 59.0, PIQA 71.7, MMLU 36.5, CommonsenseQA 37.1, TriviaQA 11.2, Winogrande 52.8, OpenBookQA 37.5, and GSM8K (5-shot) 33.6, with an average across 9 tasks of 44.3. The comparison set includes SmolLM-360M, SmolLM2-360M, Qwen2.5-0.5B, OLMo-1B, and TinyLLaMA-1B, and the paper states that StackTrans-360M outperforms larger open-source LLMs with 2–3x more parameters (Zhang et al., 21 Jul 2025).

The scaling study covers 360M, 600M, 1.0B, 1.5B, and 7B parameters and reports smoother convergence and lower final loss than standard Transformers of equivalent size, with a brief warm-up attributable to learning stack operations (Zhang et al., 21 Jul 2025). The training objective is

$$\delta: Q \times (\Sigma \cup \{\varepsilon\}) \times \Gamma \to Q \times \Gamma^*.$$8

where the entropy term is used to avoid action collapse into uniform distributions.

Ablations isolate several design choices. Replacing the stack with a queue worsens perplexity and validation loss; fixing $$\delta: Q \times (\Sigma \cup \{\varepsilon\}) \times \Gamma \to Q \times \Gamma^*.$$9 significantly degrades performance; and the multi-head low-rank design outperforms both single-head and full-dimensional variants (Zhang et al., 21 Jul 2025). Hyperparameter studies report that improvements in the number of stack heads $\Gamma^*$0 saturate beyond a small $\Gamma^*$1, that $\Gamma^*$2–$\Gamma^*$3 balances cost and expressiveness, and that increasing stack size from $\Gamma^*$4 to $\Gamma^*$5 yields negligible gains.

The reported efficiency overhead relative to vanilla Transformers is modest but explicit: training time $\Gamma^*$6, inference time $\Gamma^*$7, and peak GPU memory $\Gamma^*$8 on identical hardware (Zhang et al., 21 Jul 2025). The paper also records limitations: no formal proof of PDA equivalence, batching enabled by breaking some temporal dependencies, overflow handled by truncation to zero, and the need for larger-scale verification beyond the reported settings.

5. Stack-trace-centered execution visualization

In software-engineering education, the 2017 paper “A tool for visualizing the execution of programs and stack traces especially suited for novice programmers” presents a related but distinct stack-centered system whose core idea is visualization of the call stack’s evolution during function calls and returns (Litvinov et al., 2017). The paper explicitly states that it does not introduce a tool named StackTrans. If the label is used informally to denote a stack-transitions visualizer, however, that description matches the paper’s central design.

The system is implemented as IDE plugins for Eclipse in Java and C++, and for IntelliJ IDEA in Java, using language-specific debugging interfaces rather than bytecode instrumentation or compiler hooks (Litvinov et al., 2017). In Java, the plugins use the Java Debug Interface; in C/C++, they use the C/C++ Debugging Interface within Eclipse. At each relevant execution event, the plugin queries runtime state and emits a language-neutral JSON snapshot containing blocks such as language, threads, stack, heap, globalStaticVariables, and lineNumber. Each snapshot is saved with a unique name and timestamp, and the visualization subsystem renders the selected state while allowing backward and forward navigation.

The visualization explicitly separates stack and heap. Activation records list function or method name, parameters, and local variables by type, name, and value; function entry and exit are shown as the appearance and disappearance of frames; recursion appears as multiple frames of the same function stacked atop one another; and variables changed or created on the last step are highlighted (Litvinov et al., 2017). Heap objects are shown in a separate block, linked to stack variables through references or pointers. In Java, only user-created objects with references appearing in the stack are shown, excluding internal system objects to avoid clutter; in C++, heap entries include memory addresses and the view also includes a dedicated global/static block.

The educational motivation is explicit. The authors aim to help novices form correct mental models of runtime behavior in the face of difficulties involving memory organization, pointers or references, and execution tracing, and they argue that conventional debuggers are ill-suited for beginners (Litvinov et al., 2017). No formal user study is reported. The paper instead presents prototypes and identifies future additions such as timeline views, textual explanations, and broader evaluation with students.

6. Transforming stacks into queues in graph layouts

A third usage of the StackTrans idea appears in graph theory, where the central question is whether stacks can be transformed into queues in linear layouts of graphs (Katheder et al., 2024). For a graph $\Gamma^*$9 with a total vertex order $\{\text{push}, \text{pop}, \text{noop}\}$0, an $\{\text{push}, \text{pop}, \text{noop}\}$1-stack layout partitions edges into stacks with no pairwise crossings in the same part, and a $\{\text{push}, \text{pop}, \text{noop}\}$2-queue layout partitions edges into queues with no pairwise nestings in the same part. The associated invariants are the stack number $\{\text{push}, \text{pop}, \text{noop}\}$3 and queue number $\{\text{push}, \text{pop}, \text{noop}\}$4. Mixed layouts combine both, and their minimum total number of parts is the mixed number $\{\text{push}, \text{pop}, \text{noop}\}$5.

The classic open problem, originally posed by Heath and Rosenberg, asks whether $\{\text{push}, \text{pop}, \text{noop}\}$6 is bounded in terms of $\{\text{push}, \text{pop}, \text{noop}\}$7 (Katheder et al., 2024). The 2024 paper proves an equivalence theorem sharpening the known landscape: the existence of a function bounding queue number by stack number is equivalent to bounded queue number for 3-stack graphs, equivalent to bounded queue number for 1-stack 1-queue graphs, and equivalent to bounded queue number in terms of mixed number (Katheder et al., 2024). The result therefore identifies the mixed-number formulation as not merely analogous but equivalent.

For bipartite graphs in the separated model, where all vertices of one bipartition precede those of the other, the paper proves that separated stack and queue numbers coincide:

$\{\text{push}, \text{pop}, \text{noop}\}$8

This follows from reversing the order of one bipartition, which transforms stacks into queues and vice versa (Katheder et al., 2024). Under a fixed separated order, the problem admits a matrix or poset interpretation: queues correspond to monotone non-decreasing subsets and stacks to monotone non-increasing subsets, yielding polynomial-time recognition through standard chain-decomposition machinery.

The mixed separated setting remains more subtle. The paper proves that every separated 1-stack 1-queue graph has a separated 4-queue layout, but also constructs examples showing that naive checkerboard-style row and column reversals do not suffice in general for separated 1-stack 2-queue layouts (Katheder et al., 2024). More broadly, the paper reduces the separated problem to the case of separated 1-stack 6-queue graphs and establishes several transformation lemmas based on riffle arguments, subdivisions, tree-partitions, and shallow-minor bounds.

Taken together, these graph-theoretic results use “stack transformation” in a purely combinatorial sense. The commonality with neural StackTrans and stack-trace visualization is structural rather than methodological: all three research lines ask how explicit stack organization changes representational power, interpretability, or transformability, but they do so in automata-augmented language modeling, runtime-state visualization, and graph layout theory respectively.