Static Program Slicing Using Language Models With Dataflow-Aware Pretraining and Constrained Decoding

Abstract: Static program slicing is a fundamental software engineering technique for isolating code relevant to specific variables. While recent learning-based approaches using LMs show promise in automating slice prediction, they suffer from inaccurate dependency modeling and unconstrained generation, where LMs fail to capture precise data flow relations and produce slices containing hallucinated tokens and statements. To address these challenges, we propose Sliceformer, a novel approach that reformulates static program slicing as a sequence-to-sequence task using small LLMs such as CodeT5+. Sliceformer introduces two key innovations that directly target the identified limitations. First, to improve dependency modeling, we design dataflow-aware pretraining objectives that leverage data flow graphs (DFG) to teach models data dependencies through dataflow-preserving statement permutation and dataflow-aware span corruption. Second, to eliminate hallucination, we develop a constrained decoding mechanism that enforces both lexical and syntactic constraints. We evaluate Sliceformer on Java and Python program slicing benchmarks, demonstrating consistent improvements over state-of-the-art baselines with up to 22% gain in ExactMatch.

• The paper introduces Sliceformer, combining dataflow-aware pretraining with lexical and syntactic constrained decoding to generate precise, extractive backward program slices.
• Sliceformer achieves 92.20% ExactMatch on Java and 83.15% on Python, outperforming the strongest baselines by 6.4% and 21.9% while preserving sub-second inference.
• The results show that dependency-aware training and hard decoding constraints reduce hallucinated or irrelevant statements more effectively than scaling unconstrained language models for exact code analysis.

Sliceformer addresses two persistent failure modes of learning-based static program slicing: inaccurate modeling of data dependencies and unconstrained generation that produces hallucinated tokens or statements. The paper formulates backward slicing as a sequence-to-sequence task over a small encoder–decoder model (CodeT5+ 0.7B) and augments standard supervised fine-tuning (SFT) with dataflow-aware pretraining objectives and a training-free constrained decoding mechanism. Evaluated on Java and Python subsets of CodeNet-Slice, Sliceformer achieves ExactMatch scores of 92.20% on Java and 83.15% on Python, exceeding the strongest baseline by 6.4% and 21.9%, respectively.

Problem formulation and motivating failures

The input to the slicer is a sequence of statements $s_1,\dots,s_N$, a slicing criterion variable $v$, and its line number; the output must be a subsequence $\bm{y} \subseteq \bm{x}$ containing all and only statements that semantically influence $v$. Two properties are required: accuracy (no missing or extraneous statements) and element preservation (every token and statement must be extracted verbatim from the input).

The authors' empirical investigation identifies two challenges. First, inaccurate dependency identification: models fine-tuned directly or prompted via LLMs rely on surface-level patterns or positional proximity rather than true def–use relations, including irrelevant statements such as temp = A; A = C; when they do not affect the criterion. Second, unconstrained generation: proprietary LLMs prompted with RAG or Chain-of-Thought hallucinate at both token level (replacing identifiers like codepoint with invented ones) and statement level (spurious repeated sub-expressions), violating element preservation. These observations motivate a design that injects dataflow semantics before SFT and constrains generation at inference time.

Dataflow-aware pretraining

Built on GraphCodeBERT-style data flow graphs ($\mathcal{DFG}$) extracted from Tree-Sitter ASTs over roughly 1.0M CodeSearchNet functions, Sliceformer introduces two pretraining objectives:

• Dataflow-preserving statement permutation: within each basic block, pairs of statements with no data edge between them are swapped; the model learns to generate orderings that preserve program semantics. Because legal orderings under a given $\mathcal{DFG}$ are not unique, this objective teaches which statements are genuinely data-independent — a capability directly relevant to slice membership decisions.
• Dataflow-aware span corruption: masking is guided by the $\mathcal{DFG}$ rather than applied uniformly. For a selected variable node, the objective reconstructs both where the value comes from (parent definitions) and where it flows to (child uses), alternating between fine-grained (variable-level) and coarse-grained (statement-level) masking at a 25% mask ratio.

After pretraining (100K steps, context length 512, batch size 32 on four RTX 3090 GPUs), the model undergoes SFT on labeled slicing data with structural control markers (<code>, <criterion>, <slice>).

Constrained decoding

At inference, beam search (beam size 3) is modified with two constraints. The lexical constraint assigns logit $-\infty$ to any token absent from the input snippet, enforcing strictly extractive generation from CodeT5+'s 32,100-token vocabulary. The syntactic constraint exploits the monotonicity of Tree Similarity Edit Distance (TSED): since every valid slice is a subsequence of the input, TSED between the input's AST and the partial output should increase monotonically as correct statements are appended. When TSED decreases at a statement boundary — signaling structural errors such as misordered tokens or over-generation — the beam path is terminated early. Notably, the lexical constraint alone cannot prevent structurally invalid outputs because it is order-agnostic; the syntactic check closes this gap.

Results

On Java, Sliceformer attains Acc-D 98.78, ExactMatch 92.20, CodeBLEU 93.23, and TSED 97.68; on Python, 90.85 / 83.15 / 85.35 / 89.74 respectively. Key comparisons:

Method	Java ExactMatch	Python ExactMatch
GPT-5 + CoT	14.00	13.00
NS-slicer (GraphCodeBERT)	85.77	61.25
CodeT5+ (SFT only)	87.24	77.24
Qwen3-8B (SFT)	80.55	72.18
Sliceformer	92.20	83.15

Three findings stand out. First, LLM-based prompting approaches perform poorly despite CoT and RAG gains over zero-shot: even GPT-5 with CoT reaches only 14% ExactMatch on Java, far below learning-based methods, confirming that hallucination makes large proprietary LLMs unreliable for exact extraction tasks. Second, relative to vanilla SFT on the same base model, the added components yield 5.0% (Java) and 5.9% (Python) ExactMatch gains, isolating the value of dataflow-aware pretraining plus constrained decoding. Third, ablations show span corruption and the lexical constraint are the most impactful components; the syntactic constraint contributes less, which the authors attribute to structural errors being rarer than lexical errors once the model has learned basic syntax. Qualitative analysis indicates the end-to-end seq2seq formulation outperforms NS-slicer's per-statement binary classification, which cannot distinguish identical statements occurring in different branches.

Inference latency is 0.296 s per task — comparable to plain CodeT5+ (0.289 s) and roughly 20× faster than fine-tuned CodeLlama-7B (5.75 s) or Qwen3-8B (6.52 s) — so the accuracy gains come at negligible computational cost.

Limitations and open questions

The paper concedes several boundaries. Evaluation covers only Java and Python; adaptation to other languages requires additional engineering and validation. The dataflow-aware pretraining objectives presuppose an encoder–decoder architecture, since masked-span reconstruction does not transfer naturally to decoder-only next-token prediction; appendix experiments show decoder-only models (CodeLlama, Qwen3) benefit only from constrained decoding (+3.1 and +2.6 ExactMatch on Java), not from the pretraining objectives. Ground-truth slices derive from static analysis tools (JavaSlicer and an adapted Python slicer), so results inherit any imprecision of those oracles. Whether the pretraining objectives can be reformulated for causal architectures, and whether the approach extends to other tasks whose outputs must satisfy structural constraints, remain open questions the paper poses but does not resolve.

Conclusion

Sliceformer demonstrates that targeted dataflow-aware pretraining combined with lexically and syntactically constrained decoding substantially improves learning-based static slicing, achieving state-of-the-art ExactMatch on both evaluated languages while retaining sub-second inference latency on a 770M-parameter model. The central lesson is architectural: for extractive program-analysis tasks, injecting dependency structure during pretraining and hard-constraining generation at inference yields larger gains than scaling up unconstrained generative models.