ProGraML: Graph-Based Program Representation

• ProGraML is a compiler-agnostic, graph-based program representation that models control-flow, data-flow, and call relations at the IR level.
• Its integration with Message Passing Neural Networks enables precise analysis and optimization, achieving state-of-the-art results on multiple benchmarks.
• ProGraML supports both classical compiler analyses and advanced program reasoning, proving effective for tasks like code optimization and algorithm classification.

ProGraML (Program Graphs for Machine Learning) is a compiler‐agnostic, semantics-preserving graph-based program representation for deep learning-based program optimization and analysis. Designed to bridge the expressiveness gap between rich intermediate representations (IRs) used in optimizers and model architectures suitable for program reasoning, ProGraML models programs as directed attributed multigraphs that encode control-flow, data-flow, and call relations at the IR level. Integrating this structural representation with Message Passing Neural Networks (MPNNs), ProGraML supports both classical compiler analyses and advanced program understanding tasks, demonstrating state-of-the-art empirical performance on multiple real-world benchmarks (Cummins et al., 2020, Cummins et al., 2020).

1. Graph-Based Representation

ProGraML interprets a complete program, typically sourced from an LLVM-IR module in SSA form, as a directed attributed multigraph $G = (V, E, \mathbf{X}_V, \mathbf{X}_E)$. Here,

• $V$ is the set of vertices representing IR instructions, variables (including SSA identifiers and global constants), or literal constants.
• $E \subseteq V \times V$ is the set of directed, typed edges that encode selective program relationships.
• $\mathbf{X}_V \in \mathbb{R}^{|V| \times d_V}$ holds vertex features, typically based on instruction opcodes and types.
• $\mathbf{X}_E \in \mathbb{R}^{|E| \times d_E}$ optionally encodes edge attributes such as operand positions.

Edge Types and Attributes:

Three primary edge categories capture program structure:

Edge Category	Directionality	Key Attribute
Control-flow	Forward, Backward	Successor index (branch)
Data-flow	Forward, Backward	Operand index
Call-flow	Call, Return	—

Edges are further augmented with position indices, supporting non-commutative operations or ordered control transfers (Cummins et al., 2020, Cummins et al., 2020).

2. Construction from LLVM IR

The construction of a ProGraML graph from SSA-form LLVM-IR involves a linear-time sweep interleaving three stages:

1. Control-Flow: Each instruction is represented as a vertex. For every basic-block successor (including multi-way branches), forward and backward control-flow edges are introduced and annotated with a position index reflecting the successor order.
2. Data-Flow: Reads and writes to variables or constants result in data-flow edges—from value producers (definitions) to consumers (uses). Operand order is preserved via the position index, ensuring correct handling for non-commutative instructions. Backward data-flow edges enable backward analyses (e.g., liveness).
3. Call-Flow: Calls connect the call site to the callee’s entry instruction (call edge) and from function exits back to the call site (return edge). Calls to external functions are represented using dummy entry and exit nodes.

This process yields a heterogeneous multigraph encoding the essential semantic structure exploited by compilers for optimization and analysis (Cummins et al., 2020, Cummins et al., 2020).

3. Message Passing Neural Network Architecture

ProGraML employs Gated Graph Neural Networks (GGNNs), a subclass of MPNNs, to process program graphs. The learning procedure operates as follows:

• Initialization: Each vertex $v$ is assigned an initial vector $h_v^{(0)}$, either by lookup in a learned embedding table keyed by instruction opcode and data-type or using a special embedding for variables/constants.
• Message Passing: For $T$ synchronous iterations (typically $T=30$), information is propagated:
  • For each edge $(u \to v)$ of type $\tau$ and position $p$, a message $$m_{u\to v}^{(t)} = W_\tau \bigl( h_u^{(t-1)} \odot \mathrm{PE}(p) \bigr)$$ is computed, where $W_\tau$ is an edge-type-specific weight matrix, $\mathrm{PE}(p)$ is a fixed sinusoidal positional embedding, and $\odot$ denotes element-wise multiplication.
  • Messages are aggregated (summed or mean) at each node $v$, producing $M_v^{(t)}$.
  • Node states are updated via a GRU: $h_v^{(t)} = \mathrm{GRU}(h_v^{(t-1)}, M_v^{(t)})$.
• Readout: For per-vertex tasks, the final embedding $h_v^{(T)}$ is fed through a Multi-Layer Perceptron (MLP) and sigmoid activation. For whole-program tasks, readout aggregates over all nodes and applies a softmax classifier.

This architecture supports both node-level (instruction, variable) and graph-level (whole-program) prediction tasks (Cummins et al., 2020, Cummins et al., 2020).

4. Benchmarks and Empirical Results

ProGraML was evaluated on a variety of both classical and advanced compiler analysis problems. Key results include:

• Classical Analyses: On tasks such as control-flow reachability, dominator tree prediction, data dependency, variable liveness, and common subexpression elimination, ProGraML achieved an average $F_1$ score of 94.0% (precision 98.9%, recall 92.0%) using a test set of 250,428 LLVM-IR modules from six source languages. In comparison, sequential LSTM models achieved only 27.3% $F_1$, indicating the necessity of structural modeling for these analyses (Cummins et al., 2020).
• Heterogeneous Device Mapping: For predicting whether OpenCL kernels execute faster on CPUs or GPUs, ProGraML attained an 86.6% accuracy (AMD) and 80.0% (NVIDIA), surpassing other methods such as inst2vec and DeepTune (Cummins et al., 2020, Cummins et al., 2020).
• Algorithm Classification (POJ-104): On the POJ-104 C++ classification task, a ProGraML-based 6-layer GGNN or 10-layer graph transformer achieved a 3.33% error rate, better than prior tree-convolution and sequence-based models (Cummins et al., 2020, Cummins et al., 2020).
• Open Benchmark (DeepDataFlow): ProGraML’s formulation as an MPNN enabled it to achieve perfect or near-perfect $F_1$ on all standard data-flow analyses within standard iterative limits; for instance, on restricted test graphs, reachability and dominance $F_1$ exceeded 0.99 (Cummins et al., 2020).

Task	ProGraML $F_1$ (DDF-30)	ProGraML $F_1$ (DDF-60)
Reachability	0.998	0.997
Dominance	1.000	0.983
Data Dependency	0.997	0.992
Liveness	0.937	0.931
Subexpressions	0.996	0.954

A plausible implication is that graph-centric, semantics-rich representations like ProGraML substantially advance machine learning approaches to traditional and emerging compiler analyses.

5. Strengths, Limitations, and Extensions

Strengths:

• ProGraML faithfully and compactly exposes control-, data-, and call-flow in a single IR-level multigraph.
• The representation avoids brittle source-level and AST-based dependencies, supporting language and IR agnosticism (applicable to LLVM, XLA HLO, etc.).
• The MPNN architecture natively supports both forward and backward analyses as in classic compiler theory.

Limitations:

• Fixed-step message passing requires $T$ to exceed the loop-connectedness $d(G)$ of a program graph for full information propagation. This constraint led to the exclusion of ~12.6% of the largest modules in experimental settings.
• Dense propagation is computationally inefficient compared to the sparse, worklist-driven iteration in production compilers.
• Class imbalance in node labels can bias the model towards under-approximation.
• Inst2vec-style pattern normalization introduces out-of-vocabulary issues for rare or complex IR constructs.

Prospective Extensions:

• Dynamic message-passing regimes activating only changed edges at each step, analogous to worklists, are suggested as a possible improvement.
• Adaptive, learned halting mechanisms may permit local convergence at each node.
• The representation admits extension to other IRs (e.g., Java bytecode, GCC IR), dynamic profiling enrichment, and more expressive GNN architectures (e.g., graph transformers, multiplexers).
• ProGraML is a basis for learned heuristics in diverse compiler tasks, including code motion, instruction scheduling, register allocation, and IR-to-IR translation (Cummins et al., 2020, Cummins et al., 2020).

6. Impact and Future Directions

ProGraML establishes a unified semantics-rich abstraction suitable for a broad spectrum of deep learning-based program analyses and optimizations. The demonstrated empirical gains in both classical compiler tasks and higher-level reasoning problems underscore its role as a building block for future data-driven compiler pipelines. By making classical abstract interpretation accessible to machine learning via graph neural architectures, ProGraML catalyzes the integration of program analysis, optimization heuristics, and learning systems. Future research directions include scaling to larger codebases via dynamic graph algorithms, porting to diverse IRs, and integrating runtime profiles for adaptive optimization (Cummins et al., 2020, Cummins et al., 2020).