DiffHLS: Differential Learning for High-Level Synthesis QoR Prediction with GNNs and LLM Code Embeddings

Abstract: High-Level Synthesis (HLS) compiles C/C++ into RTL, but exploring pragma-driven optimization choices remains expensive because each design point requires time-consuming synthesis. We propose \textbf{\DiffHLS}, a differential learning framework for HLS Quality-of-Result (QoR) prediction that learns from kernel--design pairs: a kernel baseline and a pragma-inserted design variant. \DiffHLS~encodes kernel and design intermediate-representation graphs with dedicated graph neural network (GNN) branches, and augments the delta pathway with code embeddings from a pretrained code LLM. Instead of regressing absolute targets directly, we jointly predict the kernel baseline and the design-induced delta, and compose them to obtain the design prediction. On PolyBench, \DiffHLS~attains lower average MAPE than GNN baselines under four GNN backbones, and LLM code embeddings consistently improve over a GNN-only ablation. We further validate scalability on the ForgeHLS dataset.

• The paper introduces a differential learning framework that decomposes QoR into kernel baselines and design-induced deltas.
• It combines GNNs for IR graph encoding with frozen LLM code embeddings to capture critical pragma information.
• Empirical results on PolyBench and ForgeHLS demonstrate superior prediction accuracy and enhanced interpretability over direct regression methods.

DiffHLS: Differential Learning for High-Level Synthesis QoR Prediction with GNNs and LLM Code Embeddings

Motivation and Background

High-Level Synthesis (HLS) is integral to contemporary FPGA accelerator design, enabling translation from C/C++ sources to RTL. Performance and resource usage are highly sensitive to design pragmas such as loop unrolling and array partitioning. However, evaluating each candidate design point via HLS is computationally prohibitive, especially considering complex design space exploration (DSE). Quality-of-Result (QoR) predictors have emerged as essential tools for predicting post-HLS resource and timing metrics, leveraging compiler Intermediate Representation (IR) graphs and graph neural networks (GNNs). Prior GNN-based methods, including CDFG-based models, regress absolute target metrics, which introduces high inter-kernel variance and impedes learning of subtle pragma effects.

DiffHLS Framework

DiffHLS introduces a differential learning framework that fundamentally restructures how QoR prediction is approached. The method predicts each design’s QoR as the composition of a kernel baseline and a design-induced delta, utilizing kernel--design pairs. Kernel and design IR graphs are encoded with parameter-disjoint GNNs, while code-level semantic cues, especially pragma annotations, are captured using frozen LLM code embeddings injected into the delta pathway. This hybridization is motivated by complementary strengths: IR graphs encapsulate control and data dependencies, while code embeddings are sensitive to source-level constructs and explicit pragma directives.

Figure 1: DiffHLS architecture: IR graphs for kernel and design are encoded by separate GNNs, LLM code embeddings are integrated into the delta pathway, and baseline and delta predictions are composed to yield the final design prediction.

The differential approach reduces regression target variance at the per-kernel cluster level, simplifying the ML task, enhancing learning of pragma-driven variation, and enabling interpretability—errors can be localized to either baseline estimation or delta modeling.

Methodological Details

Differential Formulation

Given a kernel $k$ and a pragma-modified design $d$, DiffHLS predicts both the baseline QoR $y_k$ and the design-induced delta $\Delta = y_d - y_k$. The final prediction is composed as $\hat{y}_d = \hat{y}_k + \widehat{\Delta}$. Baseline and delta heads are realized as MLPs over GNN or fused GNN+LLM code embeddings, using a SmoothL1 loss over all three outputs (baseline, delta, composite design), ensuring error consistency and diagnostic utility.

Hybrid Representation

IR graphs for $k$ and $d$ are extracted from the HLS toolchain’s ADB representations and encoded with dedicated GNN encoders. For each design, code tokens are embedded with a frozen code LLM (e.g., Qwen2.5-Coder-1.5B or Llama-3.2-1B), with only the classifier adapter trained downstream, thereby retaining LLM zero-shot generalization properties and limiting computational overhead.

Pragmas, loop structures, and local code semantics that strongly affect resource/timing metrics are often weakened during IR lowering, making code embeddings crucial for faithful delta regression. Only the delta pathway receives code embedding input, preserving interpretability: the kernel baseline is determined solely from graph structure, while the delta is sensitive to pragma and code context.

Figure 2: Average MAPE across DSP, FF, LUT, and CP for three DiffHLS variants (full, without differential learning, and without code embeddings), demonstrating superior accuracy for the full model across four GNN backbones.

Empirical Evaluation

Datasets and Metrics

Evaluation occurs on the PolyBench corpus (27 kernels, over 10,000 design points), with targets including DSP, FF, LUT, and critical path (CP), and is further scaled to the large ForgeHLS dataset for robustness and generality. Metrics include MAE, MAPE, and $R^2$.

QoR Prediction Accuracy

DiffHLS achieves strong overall MAPE—e.g., for GraphSAGE backbone, 3.31% (DSP), 7.34% (FF), 5.12% (LUT), 1.06% (CP). These results substantially outperform GNN-only and hierarchical GNN baselines in every case. The improvement is most pronounced in settings with high kernel-induced baseline variance and significant pragma-driven intra-kernel variation.

Differential vs. Direct Regression

A variant of DiffHLS that predicts design metrics directly (eschewing the baseline/delta decomposition) exhibits consistent MAPE degradation, with LUT MAPE rising by over 7.2 percentage points under GAT backbone (as per Table: ablation). This empirically confirms the statistical and practical superiority of differential learning in variance reduction and targeting subtle design changes.

Effectiveness of Code Embeddings

Stringent ablations show that injection of frozen pretrained LLM code embeddings yields non-trivial accuracy improvements, particularly for metrics like DSP, which are highly correlated with explicit source-level pragmas. This confirms the hypothesis that GNNs alone under-exploit pragma information often lost in IR translation.

Prediction Head Analysis

Kernel baseline regressors achieve MAPE <2% and $R^2$ near unity on all resource metrics, establishing their reliability. Delta head performance reflects the challenge of learning fine-grained, pragma-dependent effects; code embedding augmentation reduces delta MAPE (e.g., from 13.55% to 11.37% on DSP under PNA).

Scalability

When applied to the ForgeHLS dataset, DiffHLS maintains strong predictive accuracy and backbone ranking, demonstrating its scalability and generalization to larger, more diverse datasets.

Implications and Future Directions

The formal decomposition of QoR prediction into interpretable, separately supervisable sub-problems is a clear advancement for model-based design space exploration in HLS. The finding that LLM code embeddings consistently boost accuracy without task-specific finetuning suggests future integration of more powerful code models or adaptive token-level attention fusion mechanisms. The practicality of DiffHLS is enhanced by diagnostic capabilities: errors can be traced to baseline or delta, optimizing human-in-the-loop debugging and DSE protocols.

From a theoretical standpoint, differential learning in this architectural context offers a pathway to modular surrogate modeling in other EDA subdomains with hierarchical or cluster-structured design spaces. Practically, the improved accuracy and efficiency directly enable broader/faster pragmas sweep, reducing the number of expensive HLS tool runs. It is plausible future work will extend this composition framework to handle sequence-of-pragmas scenarios, fine-grained timing models, or even end-to-end symbolic execution-guided DSE.

Conclusion

DiffHLS establishes a refined paradigm for HLS QoR prediction. By leveraging differentiated learning from kernel--design pairs and augmenting GNNs with LLM code embeddings, it achieves superior quantitative accuracy, improved interpretability, and robust scalability. The framework points toward future research in variance-reduced surrogate modeling and cross-modal code/graph integration for hardware design automation.