Papers
Topics
Authors
Recent
Assistant
AI Research Assistant
Well-researched responses based on relevant abstracts and paper content.
Custom Instructions Pro
Preferences or requirements that you'd like Emergent Mind to consider when generating responses.
GPT-5.1
GPT-5.1 130 tok/s
Gemini 3.0 Pro 29 tok/s Pro
Gemini 2.5 Flash 145 tok/s Pro
Kimi K2 191 tok/s Pro
Claude Sonnet 4.5 34 tok/s Pro
2000 character limit reached

Hierarchical Mixture of Experts: Generalizable Learning for High-Level Synthesis (2410.19225v4)

Published 25 Oct 2024 in cs.LG, cs.AI, and cs.AR

Abstract: High-level synthesis (HLS) is a widely used tool in designing Field Programmable Gate Array (FPGA). HLS enables FPGA design with software programming languages by compiling the source code into an FPGA circuit. The source code includes a program (called "kernel") and several pragmas that instruct hardware synthesis, such as parallelization, pipeline, etc. While it is relatively easy for software developers to design the program, it heavily relies on hardware knowledge to design the pragmas, posing a big challenge for software developers. Recently, different machine learning algorithms, such as GNNs, have been proposed to automate the pragma design via performance prediction. However, when applying the trained model on new kernels, the significant domain shift often leads to unsatisfactory performance. We propose a more domain-generalizable model structure: a two-level hierarchical Mixture of Experts (MoE), that can be flexibly adapted to any GNN model. Different expert networks can learn to deal with different regions in the representation space, and they can utilize similar patterns between the old kernels and new kernels. In the low-level MoE, we apply MoE on three natural granularities of a program: node, basic block, and graph. The high-level MoE learns to aggregate the three granularities for the final decision. To train the hierarchical MoE stably, we further propose a two-stage training method to avoid expert polarization. Extensive experiments verify the effectiveness of the proposed hierarchical MoE. We publicized our codes at https://github.com/weikai-li/HierarchicalMoE.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (38)
  1. AMD/Xilinx. 2020. Vitis HLS. https://docs.xilinx.com/v/u/2020.2-English/ ug1416-vitis-documentation.
  2. Towards a Comprehensive Benchmark for High-Level Synthesis Targeted to FPGAs. Advances in Neural Information Processing Systems, 36: 45288–45299.
  3. Improving GNN-based accelerator design automation with meta learning. In Proceedings of the 59th ACM/IEEE Design Automation Conference, DAC ’22, 1347–1350. Association for Computing Machinery. ISBN 9781450391429.
  4. FPGA HLS today: successes, challenges, and opportunities. ACM Transactions on Reconfigurable Technology and Systems (TRETS), 15(4): 1–42.
  5. High-Level Synthesis for FPGAs: From Prototyping to Deployment. IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems, 30(4): 473–491.
  6. ProGraML: A graph-based program representation for data flow analysis and compiler optimizations. In International Conference on Machine Learning, 2244–2253. PMLR.
  7. Network Transfer Learning via Adversarial Domain Adaptation with Graph Convolution. CoRR, abs/1909.01541.
  8. Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks. arXiv:1703.03400.
  9. Node-wise Filtering in Graph Neural Networks: A Mixture of Experts Approach. arXiv preprint arXiv:2406.03464.
  10. Graph classification by mixture of diverse experts. arXiv preprint arXiv:2103.15622.
  11. Adaptive Mixture of Local Expert. Neural Computation, 3: 78–88.
  12. Transfer Learning for Design-Space Exploration with High-Level Synthesis. In 2020 ACM/IEEE 2nd Workshop on Machine Learning for CAD (MLCAD), 163–168.
  13. Sparse Mixture-of-Experts are Domain Generalizable Learners. arXiv:2206.04046.
  14. FLOOD: A flexible invariant learning framework for out-of-distribution generalization on graphs. In Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, 1548–1558.
  15. Fair graph representation learning via diverse mixture-of-experts. In Proceedings of the ACM Web Conference 2023, 28–38.
  16. SGDR: Stochastic Gradient Descent with Warm Restarts. arXiv:1608.03983.
  17. On the adequacy of untuned warmup for adaptive optimization. arXiv:1910.04209.
  18. Pouchet, L.-N. 2012. PolyBench/C. https://web.cs.ucla.edu/~pouchet/software/polybench/.
  19. Automatic Hardware Pragma Insertion in High-Level Synthesis: A Non-Linear Programming Approach. arXiv:2405.12304.
  20. Enhancing High-Level Synthesis with Automated Pragma Insertion and Code Transformation Framework. arXiv:2405.03058.
  21. Cross-Modality Program Representation Learning for Electronic Design Automation with High-Level Synthesis. arXiv:2406.09606.
  22. MachSuite: Benchmarks for accelerator design and customized architectures. 110–119.
  23. High-level synthesis design space exploration: Past, present, and future. IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems, 39(10): 2628–2639.
  24. Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer. CoRR, abs/1701.06538.
  25. Domain-adaptive message passing graph neural network. Neural Networks, 164: 439–454.
  26. Automated Accelerator Optimization Aided by Graph Neural Networks. In 2022 59th ACM/IEEE Design Automation Conference (DAC).
  27. Robust GNN-based representation learning for HLS. In 2023 IEEE/ACM International Conference on Computer Aided Design (ICCAD), 1–9. IEEE.
  28. AutoDSE: Enabling Software Programmers to Design Efficient FPGA Accelerators. arXiv:2009.14381.
  29. Accurate operation delay prediction for FPGA HLS using graph neural networks. In Proceedings of the 39th international conference on computer-aided design, 1–9.
  30. Graph Mixture of Experts: Learning on Large-Scale Graphs with Explicit Diversity Modeling. arXiv:2304.02806.
  31. Unsupervised Domain Adaptive Graph Convolutional Networks.
  32. Ironman: GNN-assisted Design Space Exploration in High-Level Synthesis via Reinforcement Learning. In Proceedings of the 2021 on Great Lakes Symposium on VLSI, 39–44. IEEE.
  33. IronMan-Pro: Multiobjective Design Space Exploration in HLS via Reinforcement Learning and Graph Neural Network-Based Modeling. IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems, 42(3): 900–913.
  34. High-level synthesis performance prediction using GNNs: benchmarking, modeling, and advancing. In Proceedings of the 59th ACM/IEEE Design Automation Conference, DAC ’22, 49–54. New York, NY, USA: Association for Computing Machinery. ISBN 9781450391429.
  35. Handling Distribution Shifts on Graphs: An Invariance Perspective. In International Conference on Learning Representations (ICLR).
  36. GraphMETRO: Mitigating Complex Graph Distribution Shifts via Mixture of Aligned Experts. arXiv:2312.04693.
  37. DANE: Domain Adaptive Network Embedding. In Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence, IJCAI-19, 4362–4368. International Joint Conferences on Artificial Intelligence Organization.
  38. Meta-DMoE: Adapting to Domain Shift by Meta-Distillation from Mixture-of-Experts. arXiv:2210.03885.

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now
Dice Question Streamline Icon: https://streamlinehq.com

Open Problems

We haven't generated a list of open problems mentioned in this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Generate Now
Lightbulb Streamline Icon: https://streamlinehq.com

Continue Learning

We haven't generated follow-up questions for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Generate Now
List To Do Tasks Checklist Streamline Icon: https://streamlinehq.com

Collections

Sign up for free to add this paper to one or more collections.

User Circle Single Streamline Icon: https://streamlinehq.com Sign Up