---
title: Protean Compiler for Adaptive LLVM Optimization
url: https://www.emergentmind.com/topics/protean-compiler
type: topic
---

# Protean Compiler for Adaptive LLVM Optimization

Protean Compiler is an agile framework that enables LLVM with built-in phase-ordering capabilities at a fine-grained scope. It is presented as a seamlessly integrated, full-fledged compiler extension rather than an external autotuning harness: it adds a new optimization level `-OP`, a new optimization phase `ProteanOpt` in the Clang driver, a new optimization tool `protean`, a complete library of more than 140 handcrafted static feature collection methods, and a search loop that combines clustered LLVM pass subsequences with a learned cost model. In the reported evaluation, the framework delivers speedup gains of up to 4.1% on average and up to 15.7% on select CBench applications with respect to LLVM’s `-O3`, while also supporting two-step optimization flows with other ML frameworks and Large Language Models [2602.06142].

## 1. Problem setting and design objective

Protean Compiler addresses the phase ordering problem, a long-standing compiler problem in which the effectiveness of one optimization pass depends on the presence, absence, and position of other passes. The broader autotuning literature treats optimization selection and phase ordering as the two central degrees of freedom in adaptive compilation, with search spaces that can grow as \(n!\) or as \(\sum_{i=0}^{l} n^i\), depending on whether repetition and variable sequence length are allowed [1801.04405].

Within that setting, Protean Compiler is defined by a specific systems claim: machine-learning-based approaches for compiler optimization had not been baked into the compiler seamlessly and had not materialized to be leveraged at a fine-grained scope of code segments. The framework therefore repositions phase ordering as an in-compiler capability. Rather than exposing an external search environment, it introduces a native optimization level that can optimize partitions of a program at module, call graph, function, or loop scope.

The design goal is not exhaustive search over individual LLVM passes. Instead, Protean Compiler constrains the search through clustered subsequences derived from the `-O3` pipeline and evaluates candidate recipes with a learned static performance predictor. This suggests a deliberate balance between search tractability, LLVM integration, and compile-time overhead.

## 2. LLVM integration and system architecture

Protean Compiler is integrated into LLVM at the driver, tool, and pass levels [2602.06142]. In the Clang driver, it defines a new phase `ProteanOpt`, inserted between `phases::Compile` and `phases::Backend`, and exposes a new command-line flag, `-OP`. If `-OP` is present, the Protean phase is executed; otherwise, the normal pipeline is used.

The execution flow has four stages. First, the front end takes C/C++ source and an optimization objective and produces LLVM IR. Second, the IR is partitioned into sub-programs or partitions; the simplest form is per-module partitioning, and the experimental evaluation primarily uses module-level scope. Third, an agile optimization loop operates per partition. For each partition, a simulated annealing engine generates candidate pass recipes, invokes a subprocess of the `protean` tool to apply the recipe to the partition’s IR, runs the IR2Score model on the resulting IR, and uses the predicted score to accept or reject the candidate. Fourth, once all partitions have been optimized, standard LLVM/Clang linking is performed. If LTO is enabled, link-time optimizations run as usual; Protean focuses on IR-level `opt` passes.

The `protean` executable is a modified version of `opt` with the same dependencies, but it replaces the fixed `-O3` pipeline with the agile optimization loop. Candidate recipes are applied in subprocesses, which means that abnormal pass failures can be discarded without terminating the overall search. The framework also exposes configuration knobs for cooling schedule, maximum iterations, random seed, maximum temperature, crossover and mutation settings for a potential genetic-search variant, and two cost-function modes: `IRAnalysis`, which determines cost based on IR2Score, and `MCA`, which determines cost based on `llvm-mca`.

## 3. Pass clustering, recipe space, and simulated annealing

Protean Compiler does not search directly over LLVM’s full pass inventory. Instead, it clusters the `-O3` pass list into five subsequences, labeled A through E, and searches over compositions of these subsequences [2602.06142]. A recipe is therefore a string over `{A,B,C,D,E}`, such as `ACDCD`, `CD`, or `ABBBA`.

The subsequences preserve local pass orderings already designed by LLVM developers. Subsequence A includes `globalopt`, a `cgscc` pipeline with `devirt`, several `inline` forms, `function-attrs`, `argpromotion`, `sroa`, speculative execution, LICM, loop unswitching, loop unroll, early-cse, jump-threading, correlated-propagation, `simplifycfg`, `instcombine`, and `aggressive-instcombine`. Subsequence C includes `sroa`, `gvn-hoist`, `mldst-motion`, `gvn`, `sccp`, `bdce`, `instcombine`, `jump-threading`, `adce`, and `memcpyopt`. Subsequence E is loop-centric and includes `loop-simplify`, `lcssa`, `loop-rotate`, loop distribute, unroll-and-jam, loop vectorize, `vector-combine`, LICM, and loop-sink.

If the number of subsequences is \(n=5\) and the maximum recipe length is \(m\), the search space is

$$
\sum_{i=0}^{m} n^i.
$$

The reported tuning study uses `automotive_susan_c` to select a practical bound on recipe length. With \(m=3\), the space contains 156 possible recipes and achieves relative best performance of approximately 0.85; with \(m=4\), 781 recipes and approximately 0.92; with \(m=5\), about 4K recipes and approximately 0.97; with \(m=6\), 19K recipes and approximately 0.99; and with \(m=7\), 97K recipes and 1.0. The chosen maximum length is 5, characterized as a good trade-off between tractability and observed performance.

Search proceeds via simulated annealing. For a current recipe with cost \(C_{present}\), a candidate neighbor with cost \(C_{new}\), and temperature \(T\), the acceptance rule is

$$
\left\{
\begin{array}{ll}
1 & \text{if } C_{new} > C_{present} \\
e^{-\frac{C_{new} - C_{present}}{T}} & \text{if } C_{new} \leq C_{present}.
\end{array}
\right.
$$

In the reported interpretation, if the predicted cost improves, the move is always accepted; if it worsens, it is accepted with an annealing probability. The default cooling schedule is geometric, although a linear schedule is also supported. The framework also includes early convergence logic: if the IR stops changing meaningfully, optimization for that partition can terminate early.

## 4. Static feature collection, IR2Score, and external ML interfaces

The static characterization layer of Protean Compiler is the Protean Feature Set, a library of 141 handcrafted static features implemented as an LLVM module pass [2602.06142]. The feature space is explicitly multi-scope.

Module/global features number 10 and include `FunctionCount`, `AverageBBPerFunction`, `TotalBBCount`, `TotalInstCount`, `TotalFunctionCalls`, and `GlobalVariableCount`. Function-level features number 71 and cover metrics such as `AverageComponentSize`, `SCCSize`, `IsLocal`, `CriticalEdgeCount`, `TotalEdgeCount`, `LoopCount`, `MedianCallsPerFunction`, `AverageCallsPerFunction`, `AverageStoreInstructionsPerFunction`, `AverageLoadInstructionsPerFunction`, and `AverageInstructionsPerFunction`. Callee/caller features number 39 and include `NumCallsiteInLoop`, `NestedInlines`, `SROALosses`, `IndirectCallPenalty`, `CallSiteCost`, `CalleeInstrPerLoop`, `CallerInstrPerLoop`, `CalleeAvgNestedLoopLevel`, `CallerAvgNestedLoopLevel`, `CalleeAvgVecInstr`, `CallerAvgVecInstr`, and block- and frequency-oriented callsite features. Loop features number 31 and include `LoopSize`, `NumLoadInstPerLoopNest`, `MaxLoopHeight`, `StepValueInt`, `FinalIVValueInt`, `InitialIVValueInt`, `TotLoopInstCount`, `TotLoopNestInstCount`, `IndVarSetSize`, `MaxTripCount`, and `TripCount`.

PFS is registered at the end of each iteration’s pipeline, so the features describe the current post-optimization IR. Its output is scope-aware: module-level rows and function-plus-loop rows share a common schema, with non-applicable features zeroed. Candidate IRs are then scored by IR2Score, a Transformer-based model that uses sequences of feature snapshots corresponding to prefixes of applied pass sequences. The motivating representation is

$$
S_o = f_o(S) \approx f_{o[n]}(f_{o[n-1]}(\dots f_{o[0]}(S))).
$$

The model choice is empirical: compared to MLP, CNN, and LSTM alternatives, the Transformer architecture produced higher accuracy and lower validation loss. Protean can consume either PFS features or external embeddings such as IR2VEC. Pearson-correlation analysis shows that IR2VEC has higher average correlation across its 300 dimensions, while PFS exhibits about a dozen strongly correlated features; in the reported experiments, the PFS-based IR2Score outperforms the IR2VEC-based version, plausibly because IR2VEC had been trained on SPEC rather than CBench.

IR2Score is deployed inside LLVM 19 through MLGO-style `ModelRunner` interfaces, with Ahead-Of-Time compiled models linked directly into the compiler. Protean also supports two-step workflows in which phase ordering is followed by pass-parameter optimization from external systems such as ACPO, or in which LLM-based source-to-source rewriting precedes Protean’s IR-level search. This IR-centric organization is consistent with broader work arguing that large, production-grade IR corpora are an enabling substrate for machine-learned compiler components [2309.15432].

## 5. Experimental results and performance profile

The evaluation uses CBench on a Huawei Kunpeng 920 server, an aarch64 ARMv8.2 platform at 2.6 GHz running Linux, with LLVM 19 integrated with Protean Compiler [2602.06142]. Workloads are pinned with `numactl`, runtimes are measured with Linux `perf`, the page cache is flushed before each run, and each benchmark is run five times. If variance exceeds 1%, the experiment is repeated. The reported mean variances are 0.58% for `-O3`, 0.71% for the IR2VEC-based IR2Score variant, and 0.49% for the PFS-based variant.

The reported geo-mean speedups over `-O3` are:

| Iterations | IR2VEC speedup | PFS speedup |
|---:|---:|---:|
| 10 | 1.022 | 1.025 |
| 50 | 1.033 | 1.035 |
| 100 | 1.035 | 1.039 |
| 500 | 1.037 | 1.041 |

At 500 iterations, the PFS configuration reaches approximately 4.1% average speedup with respect to `-O3`. Select benchmarks show much larger improvements, including 1.078× on `consumer_jpeg_d`, 1.139× on `network_dijkstra`, 1.157× on `security_pgp_d`, 1.065× on `security_sha`, and 1.050× on `telecom_adpcm_d`. The paper characterizes the maximum observed gain on select CBench applications as 15.7%.

Build-time overhead grows with iteration count, but the two model variants differ substantially:

| Iterations | IR2VEC overhead | PFS overhead |
|---:|---:|---:|
| 10 | 30.918 s | 31.531 s |
| 50 | 132.467 s | 129.270 s |
| 100 | 219.830 s | 240.693 s |
| 500 | 732.556 s | 320.446 s |

The lower 500-iteration overhead for PFS is attributed to earlier convergence: IR2Score+PFS averages approximately 145 iterations per module before early stop, while the IR2VEC variant averages approximately 310. This suggests that the predictive quality of the cost model affects not only final code quality but also the practical compile-time envelope of the search.

Two-step optimization results are also reported. On `consumer_jpeg_c`, Protean alone, using IR2VEC and 20 iterations, finds recipe `ACDCD` and achieves 2.5% speedup over `-O3`. When combined with ACPO for inlining decisions inside subsequence A, the final runtime becomes 5.38 s, corresponding to approximately 8.5% speedup over `-O3`, with a 12% code-size increase relative to `-O3` and 10% relative to Protean alone. On `automotive_susan_c`, an LLM, Qwen2.5-Coder-32B-Instruct, rewrites `susan_corners()` with loop unrolling factor 5 and yields 3.1% speedup over `-O3`; compiling the LLM-optimized source with Protean then yields a further 7% speedup over the LLM baseline, for an approximately 10.1% combined speedup with respect to `-O3`.

## 6. Relation to prior work, misconceptions, and limitations

Protean Compiler is positioned against prior ML-guided compiler optimization systems by the form of its integration rather than by the mere use of learning [2602.06142]. MiCOMP is the closest comparison in the paper: it builds subsequences from LLVM `-O3` and performs ML-guided iterative compilation, but it is implemented as a Python wrapper around LLVM 3.8, uses dynamic features from the MICA Pin tool, and operates at program-wide scope only. Protean instead is fully integrated into LLVM 19, uses static features, supports fine-grain scopes, and exposes a native `-OP` optimization level. Relative to MiCOMP, the reported extra performance is approximately 1–2%, depending on iteration count.

A common misconception would be to treat Protean as a pass-by-pass exhaustive search engine. It is not. The search space is deliberately constrained to recipes over five clustered subsequences, and the reported experiments primarily optimize at module level. Another misconception would be to read the framework as a dynamic or profile-driven optimizer. The deployed IR2Score path is static: it predicts performance from IR features rather than from hardware counters or repeated runtime execution.

Several limitations are explicit. Evaluation is limited to CBench on a single ARMv8.2 platform. Compile-time overhead can be substantial for large benchmarks and large iteration budgets. The framework supports module, call graph, function, and loop scope architecturally, but the published experiments focus on modules. Code-size growth can accompany higher performance, especially when Protean is composed with aggressive inlining systems such as ACPO. The authors also state that the project is intended for open-source release in the near future, so the implementation is presented as production-oriented but not yet publicly released.

Within those limits, Protean Compiler occupies a specific place in compiler research: it transforms phase ordering from an external autotuning experiment into a first-class optimization mode inside Clang/LLVM. A plausible implication is that its main significance lies less in any single heuristic than in the compiler architecture it proposes—clustered subsequence search, native ML inference, scope-aware static features, subprocess-isolated exploration, and explicit composition with third-party ML frameworks and LLMs.

Source: https://www.emergentmind.com/topics/protean-compiler