Algorithm-Level Co-Optimization

• Algorithm-level co-optimization is the joint tuning of algorithmic, architectural, and implementation parameters, enabling holistic system performance improvements in latency, accuracy, and energy efficiency.
• It employs advanced heuristics like domain-aware genetic algorithms, multi-objective reinforcement learning, and co-evolutionary strategies to effectively explore a high-dimensional, coupled design space.
• Significant case studies in DNN accelerators and dense linear algebra demonstrate 3–10× speedups and notable energy and performance gains compared to traditional sequential optimizations.

Algorithm-level co-optimization is the simultaneous, joint optimization of interacting algorithmic, architectural, and implementation parameters across different levels of a computational stack or interrelated modules, with the explicit aim of achieving a collective system objective (e.g., minimal latency, maximal accuracy, maximal efficiency). This paradigm extends beyond traditional single-axis optimization—which fixes or modularizes some design aspects while varying others—by treating the union of all relevant algorithmic control variables as a coupled search space. Algorithm-level co-optimization is central in domains where cross-layer interactions induce large, nontrivial trade-off surfaces, such as DNN accelerator design, multi-level evolutionary search, combinatorial optimization with nested heuristics, and automated software/hardware codesign.

1. Mathematical Formulation and Design Principles

Algorithm-level co-optimization typically formalizes the search over a composite variable $x = [h, m]$, where $h \in \mathbb{Z}^P$ parameterizes hardware (e.g., PE array shapes, buffer sizes, NoC routes) or high-level system structure, and $m \in \mathbb{Z}^Q$ encapsulates mapping, algorithmic, or software choices (e.g., tiling, loop order, dataflow, search heuristics) (Kao et al., 2022, Prajapati et al., 2017). The unified objective is usually scalar-valued (e.g., minimize latency, maximize accuracy or throughput), but often multi-objective as in energy-delay-area or Pareto-optimality settings:

$$\text{Find } x^* \in \underset{x \in \mathcal{F}}{\arg\min}\, O(x)$$

where $O(x)$ is for example inference latency or a multi-objective composition $O(x) = (L(x), E(x), A(x))$, and $\mathcal{F}$ encodes all algorithmic and resource constraints.

Key properties:

• The search space has high dimensionality and strong cross-coupling between axes, so naive Cartesian search is infeasible.
• Feasibility constraints are often nonconvex and must be encoded via projection or rejection in genetic/evolutionary or RL frameworks (Kao et al., 2022, Zhang et al., 2024).

2. Optimization Methodologies

Algorithm-level co-optimization exploits advanced optimization heuristics capable of efficient exploration in large-scale, structured search spaces:

• Domain-aware genetic algorithms: Encode joint hardware and mapping choices into a unified genotype; apply specialized crossover, mutation, and selection steps that are cognizant of subspace type (continuous, discrete, permutation); enforce resource constraints via feasibility checks (Kao et al., 2022).
• Co-evolutionary multi-level selection (cMLSGA): Partition populations into sub-groups (collectives), each running a distinct evolutionary strategy; orchestrate periodic knockout and reseeding of collectives based on performance, ensuring global search coverage and robustness (Grudniewski et al., 2021).
• Multi-objective RL: Use a state space comprising both algorithmic and hardware statistics; actions correspond to low-level design transformations; rewards combine normalized and weighted performance, energy, and area metrics; Q-learning drives exploration (Zhang et al., 2024).
• Consensus-based multiscale optimization: Evolve multiple coupled swarms, each devoted to a different bi- or tri-level of the problem; upper-level particles' state is adapted based on the ergodic average of faster-evolving subordinate-level swarms (Herty et al., 2024).
• LLM-driven dual-population co-evolution: Simultaneously evolve both algorithmic code modules and the prompt templates used to mutate/generate them, allowing structural and functional aspects of algorithms to jointly adapt, as well as the pattern of inductive bias imparted by prompt engineering (Cen et al., 10 Dec 2025, Zhao et al., 13 Mar 2025).

3. Case Studies and Domain-Specific Instantiations

Domain	Co-optimization Components	Approach/Results
DNN Accelerators	HW (array, buffers) + mapping	DiGamma: 3.0×–10× speedups (Kao et al., 2022)
Neuromorphic SNNs	SNN topology + HW architecture	RL+async sim: 9.7% ↑ acc, 28.9× ↓ EDP (Zhang et al., 2024)
Dense Linear Algebra	Algorithm block size + kernel	Blocked LAPACK/BLIS: +16–34% perf. (Martínez et al., 2023)
GPGPU Stencil Codes	Core counts, mem + tiling, k	Nonlinear opt.: 28–33% perf. ↑, 2× with cache elim. (Prajapati et al., 2017)
Multiobjective GAs	Compiler of GAs ("collectives")	cMLSGA: outperforms 9 baselines, most robust (Grudniewski et al., 2021)
Combinatorial Graph Opt.	Graph edits + heuristic solver	PPO+heuristic: up to 24% ↑ vs. best H (Wang et al., 2021)

In DNN accelerator design, hardware sizing and mapping are encoded as a single genotype; a domain-aware GA with custom mutation/crossover discovers synergistic designs for both edge and cloud platforms, showing 3×–10× latency reduction compared to prior staged optimization (Kao et al., 2022).

In neuromorphic computing, optimization over SNN topology and hardware architecture via multi-objective RL together with a fully asynchronous simulator (TrueAsync) yields significant speedup both in co-design search time (2–15×) and hardware energy-delay product, while simultaneously improving accuracy (Zhang et al., 2024).

Dense linear algebra stacks (e.g., LAPACK/BLIS) employ runtime analytical performance modeling to adapt GEMM kernel blocking and micro-kernel shape to both input dimensions and specific cache hierarchy, rather than relying on static or manual tuning, achieving substantial performance and cache utilization gains on ARM and x86 (Martínez et al., 2023).

4. Representation, Encoding, and Search Space Structuring

High-efficiency co-optimization is predicated on search-space encodings that:

• Jointly represent all relevant degrees of freedom—hardware-level (PE count, array shape), mapping-level (tiling, loop order), and software-level (algorithmic modules, dataflow strategies) (Kao et al., 2022).
• Support fast, constraint-aware feasibility projection (bounding hardware resource use, software constraints, Pareto-domination, etc.).
• Treat composite objects such as “algorithms plus prompt templates” or “heuristic plus graph structure” as evolvable units (Cen et al., 10 Dec 2025, Wang et al., 2021).

Flexible genotype representations leverage a combination of continuous, discrete, and permutation encodings, one-hot vectors, custom crossover for permutations, and embedded module graphs for higher-level software structures (Kao et al., 2022, Zhao et al., 13 Mar 2025).

5. Experimental Metrics, Evaluation, and Empirical Insights

Algorithm-level co-optimization is empirically evaluated using:

• End-to-end system metrics: inference latency, throughput (images/sec), energy-delay product, solution quality (for optimization tasks), and hardware constraints (area, DRAM BW) (Kao et al., 2022, Zhang et al., 2024, Prajapati et al., 2017).
• Comparison against strong baselines: staged optimization (HW-only, mapping-only), expert-tuned systems, classical evolutionary or RL solutions, and best-performing individual modules (Kao et al., 2022, Grudniewski et al., 2021, Martínez et al., 2023).
• Pareto-optimal front analysis: only ≈1% of joint HW+SW configurations are Pareto-optimal in performance/area or performance/energy trade-off space, enabling rapid design-space pruning (Prajapati et al., 2017).
• Algorithmic diversity: cMLSGA evidence demonstrates that multiple complementary strategies (e.g., convergence-first and diversity-first GAs) provide robustness across diverse workload categories (Grudniewski et al., 2021).

The inclusion of prompt templates as first-class evolutionary objects (in nascent LLM-driven algorithm discovery) and bi-dimensional (structural-functional) joint optimization leads to leaping performance improvements unattainable by local module tweaking alone; prompt template evolution directly correlates with performance jumps across LLMs of different strengths (Cen et al., 10 Dec 2025, Zhao et al., 13 Mar 2025).

6. Emerging Paradigms and Theoretical Developments

Recent developments include:

• Theoretical analysis of time-scale separation in consensus-based bi-level and tri-level optimization, yielding singularly perturbed SDE systems and convergence guarantees to averaged effective dynamics as the fast variable equilibrates (Herty et al., 2024).
• Algebraic frameworks based on A Mathematics of Arrays (MoA) and the ψ-calculus supply a systematic route to derive, reduce, and prove correctness/efficiency of high-performance implementations; symbolic cost models allow a priori prediction and selection of optimal blocking and tiling (0803.2386).
• Automated code/architecture co-evolution leveraging natural language understanding and code synthesis by LLMs, which can outstrip expert-designed heuristics by enabling global architectural search and continual adaptation (Zhao et al., 13 Mar 2025).

7. Implications, Limitations, and Extensions

Algorithm-level co-optimization is establishing itself as an essential methodology in domains where cross-coupled design spaces cannot be decomposed or optimized sequentially. Key implications:

• Holistic optimization of computation stacks delivers super-additive gains unavailable by tuning isolated modules.
• Model-based, constraint-driven search is dramatically more tractable than brute-force search in high-dimensional, constrained joint spaces (Kao et al., 2022, Prajapati et al., 2017).
• LLM-driven and multi-level co-evolutionary approaches facilitate the discovery of algorithmic and architectural innovations inaccessible to template-based or local optimization strategies (Zhao et al., 13 Mar 2025, Cen et al., 10 Dec 2025).

However, practical effectiveness depends on:

• The quality and physical fidelity of the analytic or surrogate models used for evaluation.
• The existence and efficiency of lower-level solvers or heuristics (e.g., in nested optimization, co-evolution, or bi-level RL) (Wang et al., 2021).
• Computational budget, as many joint optimization frameworks still rely on evolutionary, population-based, or RL methods with high per-evaluation cost.

Future directions include mean-field analysis of particle-based multi-level optimizers, expansion to higher-level (>3) stack co-design, integration of such frameworks with automated experimental pipelines in materials science or scientific computing, and embedding explainability and constraint-adherence logic into the co-optimization loop (Herty et al., 2024, Zhao et al., 13 Mar 2025).

---

References:

• "DiGamma: Domain-aware Genetic Algorithm for HW-Mapping Co-optimization for DNN Accelerators" (Kao et al., 2022)
• "Beyond Algorithm Evolution: An LLM-Driven Framework for the Co-Evolution of Swarm Intelligence Optimization Algorithms and Prompts" (Cen et al., 10 Dec 2025)
• "A Bi-Level Framework for Learning to Solve Combinatorial Optimization on Graphs" (Wang et al., 2021)
• "Co-Design of the Dense Linear Algebra Software Stack for Multicore Processors" (Martínez et al., 2023)
• "From Understanding to Excelling: Template-Free Algorithm Design through Structural-Functional Co-Evolution" (Zhao et al., 13 Mar 2025)
• "cMLSGA: A Co-Evolutionary Multi-Level Selection Genetic Algorithm for Multi-Objective Optimization" (Grudniewski et al., 2021)
• "ANCoEF: Asynchronous Neuromorphic Algorithm/Hardware Co-Exploration Framework with a Fully Asynchronous Simulator" (Zhang et al., 2024)
• "Accelerator Codesign as Non-Linear Optimization" (Prajapati et al., 2017)
• "A multiscale Consensus-Based algorithm for multi-level optimization" (Herty et al., 2024)
• "Conformal Computing: Algebraically connecting the hardware/software boundary using a uniform approach..." (0803.2386)