A3C3: AI Algorithm and Accelerator Co-design, Co-search, and Co-generation

Abstract: We present a holistic methodology for artificial intelligence algorithm and accelerator co-design, co-search, and co-generation (A3C3), which jointly optimizes neural network architectures and their hardware implementations to address the inefficiencies of traditional top-down AI system design flows. Conventional AI deployment often treats model design and hardware mapping as separate stages: an algorithm is first developed for accuracy, and only afterward adapted to meet latency, throughput, energy, or resource constraints. This separation can lead to suboptimal systems, particularly as modern AI workloads become increasingly heterogeneous, memory-intensive, and platform-dependent. A3C3 instead parameterizes both algorithmic and accelerator design spaces and searches them jointly, enabling the automatic generation of model-accelerator pairs that better balance accuracy, latency, throughput, energy efficiency, and hardware utilization. This article is a book chapter of the Handbook of Embedded Machine Learning, edited by Sudeep Pasricha and Muhammad Shafique, Springer Nature.

• The paper details a unified framework that jointly optimizes neural network architectures and their corresponding accelerator hardware.
• It introduces a multi-stage workflow with bundle evaluation, joint search space exploration, and automated co-generation.
• Experimental results show significant improvements in inference speed, energy efficiency, and throughput across diverse platforms.

A3C3: Holistic AI Algorithm and Accelerator Co-design, Co-search, and Co-generation

Introduction and Motivation

A3C3 defines a unified methodology for artificial intelligence algorithm and accelerator co-design, co-search, and co-generation. Unlike conventional AI deployment paradigms that treat model design and hardware mapping as independent stages, A3C3 jointly parameterizes and optimizes both neural network architectures and their hardware deployments. This end-to-end strategy directly addresses suboptimality arising from sequential design flows, which frequently fail to reconcile trade-offs across accuracy, latency, throughput, energy efficiency, and hardware utilization, especially as workloads grow more heterogeneous and platform-dependent.

Traditional top-down approaches prioritize algorithmic accuracy during model development, deferring hardware considerations until post hoc compression, pruning, quantization, or mapping. Such flows typically produce hardware-unfriendly models incompatible with stringent real-time, power, or memory constraints on embedded devices. A3C3 instead formalizes algorithmic and hardware parameters in a coupled, optimizable space, enabling systematic exploration and generation of matched model-accelerator pairs that satisfy application-level QoS and QoR.

Methodological Framework

A3C3's joint design space encompasses:

• Algorithmic parameters: Layer types, kernel sizes, channel widths, network depth, and quantization levels.
• Hardware parameters: Parallelism, loop tiling factors, memory partitioning, and data reuse strategies.
• System-level objectives: Composite metrics combining accuracy, latency, throughput, energy, silicon area, and resource utilization.

The central abstraction is the "bundle", a unified building block encapsulating both a sequence of neural operations and their corresponding hardware configurations. Bundles facilitate joint dimensionality reduction, enforce hardware-software co-binding, and support modular composition for rapid construction of efficient AI systems.

A3C3's three-stage workflow consists of:

1. Multi-objective bundle evaluation: Hardware-characterized bundles are evaluated for algorithmic efficacy, hardware efficiency, and resource footprint, yielding Pareto-optimal building blocks.
2. Joint search space exploration: Search engines, leveraging gradient-based optimization, evolutionary algorithms, or RL, traverse the fused algorithm-implementation space governed by multi-objective cost functions.
3. Automated co-generation and synthesis: Deployable AI systems are automatically generated, with both model topology and hardware logic (e.g., Verilog, HLS-based C++) optimized to the underlying accelerator characteristics.

Representative System Instantiations

Edge Vision: SkyNet

SkyNet [32] operationalizes co-design for real-time object detection on UAV platforms, targeting stringent power, latency, and spatial resolution constraints on embedded FPGAs and GPUs. The architecture is derived via multi-stage co-search over bundles and topological hyperparameters, including channel expansion and pooling positions, using Particle Swarm Optimization. FPGA implementations utilize optimized dataflows (tiling/data reuse, pipelined PEs), hardware-efficient quantization, and accelerator-friendly activation functions.

Experimental results: SkyNet achieved 0.716 IoU and 25.05 FPS at 7.26W on Ultra96 FPGA, outperforming prior entries by 0.101 absolute IoU and reaching highest throughput/accuracy. On TX2 GPU, it reached 0.731 IoU and 67.33 FPS. For tracking workloads, SkyNet delivered similar or superior metrics to ResNet-50 while running up to 1.7x faster.

Differentiable Co-search: EDD

The Efficient Differentiable DNN Architecture and Implementation Co-search (EDD) framework [15] generalizes SkyNet's staged co-design into a differentiable merged search space, where both architectural and implementation choices (including quantization and hardware mapping) are parameterized as continuous variables. Joint optimization is performed via gradient descent, guided by composite loss terms for accuracy, latency/performance, and hard resource constraints.

GPU/FPGA results: EDD-Net-1 achieved 1.4x–2.0x lower latency than state-of-the-art NAS models on GPUs. Recursive FPGA deployments earned 1.1x–1.53x speedups over CHaiDNN, and pipelined FPGA deployments yielded 1.45x throughput improvements over DNNBuilder with concurrent accuracy gains.

LLM Inference Acceleration: Medusa and SnapKV

Medusa

Medusa [3] exemplifies co-design at the inference algorithm level, restructuring autoregressive decoding for LLMs via parallel speculative decoding heads and tree-based attention masking. Multiple heads predict future tokens, and candidate continuations are verified in a single pass, maximizing GPU utilization and reducing memory-bound inefficiencies. Acceptance criteria leverage entropy-adaptive thresholds, enabling flexible token validation.

Medusa offers two training modes: Medusa-1 (speculative head fine-tuning only) and Medusa-2 (joint backbone/head fine-tuning).

Numerical results: Medusa-2 delivers up to 3.6x decoding speedup on Vicuna-7B/13B, with 1.5x–1.9x throughput improvement for Llama 3.1 70B/405B in NVIDIA TensorRT-LLM production settings, with bitwise-identical output quality.

SnapKV

SnapKV [16] performs hardware-aware, context-adaptive KV cache compression for long-context LLM inference. By analyzing attention allocation within an observation window at prompt end, SnapKV dynamically selects critical tokens for retention. Compressed KV caches are concatenated and used for downstream attention computations, substantially reducing memory footprint and per-token decoding latency.

Performance: SnapKV achieves up to 3.6x speedup and 8.2x memory reduction at 16K-sequence lengths; permits up to 380K-context inference on a single A100-80GB GPU with negligible retrieval quality degradation, outperforming static heuristics.

Implications and Future Directions

A3C3 establishes co-design as a central paradigm for adaptive, heterogeneous, and scalable AI system development:

• Distributed accelerator co-design: Joint optimization across multi-node, multi-tier GPU/memory infrastructures, considering tensor/pipeline/expert parallelism, KV-cache placement, interconnect bandwidth, and heterogeneous device scheduling.
• Heterogeneous accelerator co-design: Principled mapping of AI workloads across diverse devices (GPUs, FPGAs, ASICs, CPUs, near-memory accelerators), optimizing around data movement, synchronization, and runtime adaptability.
• Memory-centric co-design: Treat memory as a primary design axis, jointly optimizing network structure, cache hierarchy, compression, and prefetching strategies, addressing the shift from compute-bound to memory-bound bottlenecks.
• Dynamic, input-adaptive co-design: Runtime adaptation of precision, sparsity, cache size, decoding depth, and hardware scheduling based on inference-time signals and input difficulty metrics (entropy, token acceptance rate).
• Quality-aware co-design for generative AI: Integrate multidimensional generation quality metrics (factuality, diversity, preference alignment) into search objectives for hardware-aware generative models.
• Integration with emerging computing technologies: Tailor models and algorithms for novel hardware (chiplet-based, analog, optical, processing-in-memory), exploiting their strengths via joint algorithm/hardware search.

Automation, transferability, and usability remain critical for translating A3C3 into practical workflows. Reliable cost models, hardware-aware benchmarks, end-to-end compiler/toolchain integration, and interpretable co-design libraries are indispensable for widespread adoption.

Conclusion

A3C3 provides a rigorous foundation for next-generation AI system design, coupling neural architectures and accelerator implementations through systematic co-search and co-generation. Empirical results across diverse platforms demonstrate significant improvements in both inference efficiency and accuracy. As AI workloads and hardware architectures continue to diversify, algorithm-accelerator co-design will be a necessary ingredient for sustainable, scalable, and high-performance deployment (2606.20869).