Intelligence Per Watt: Efficiency in AI

Updated 12 November 2025

Intelligence Per Watt (IPW) is a metric that gauges AI system efficiency by calculating the computational throughput relative to power consumption.
It integrates key measurements such as inference throughput, accuracy, and power usage to provide standardized comparisons across diverse AI architectures.
Empirical studies using systems like CNN-DSA, Google TPU, and Nvidia DGX illustrate practical trade-offs in design, scalability, and theoretical thermodynamic limits.

Intelligence Per Watt (IPW) is a quantitative metric for evaluating the energy efficiency of intelligent systems—including both algorithms and hardware—by measuring the amount of computational “intelligence” delivered per unit of power consumed. The concept now permeates benchmarking for AI accelerators, local and cloud inference, and foundational theoretical studies linking information-processing to thermodynamic limits. IPW provides a normalized yardstick to compare diverse AI architectures, implementations, and deployments on their ability to deliver useful inference or learning within a given power envelope, under standardized workload and accuracy constraints.

1. Formal Definitions and Theoretical Frameworks

Multiple formalisms for IPW have emerged, reflecting distinct system perspectives:

Operational Definition (Inference Hardware)

For accelerator-centric evaluation, IPW is defined as: $\mathrm{IPW} = \frac{I}{P}$ Where:

$I$ (Intelligence) is often operationalized as inference throughput in TOPS (tera-operations per second) or model-specific performance indicators (e.g., tokens/sec in LLMs, samples/sec in vision models).
$P$ is measured power consumption in Watts.

For CNN hardware specifically, $I$ can be computed as: $I = N_{\mathrm{MAC}} \times 2 \times f_{\mathrm{clk}}$ with $N_{\mathrm{MAC}}$ the number of multiply-accumulate units, $f_{\mathrm{clk}}$ the clock frequency (Hz), and each MAC operation counting as two arithmetic operations (Sun et al., 2018).

Quality-Weighted Definition

Advanced IPW metrics incorporate solution quality $Q$ —typically accuracy normalized to $[0,1]$ : $\mathrm{IPW} = \frac{Q \times T}{P} = Q \times \eta$ where $T$ is throughput (samples/sec) and $\eta$ is energy efficiency in samples per Joule (Tschand et al., 15 Oct 2024, Saad-Falcon et al., 11 Nov 2025).

Thermodynamic Lower Bounds

From algorithmic thermodynamics, IPW admits fundamental lower bounds dictated by Landauer’s principle: $\phi(E) = \frac{P(E)}{I(E)} \geq \frac{k_B T \ln 2 \cdot F(E)}{a \Delta t}$ where $k_B$ is Boltzmann’s constant, $T$ is temperature, $F(E)$ is architectural overhead, $a$ is a task-dependent constant, and $\Delta t$ the inference interval (Perrier, 1 Apr 2025). This formalism links actual system energy use to theoretical minima.

2. Measurement Methodologies

Hardware and System Measurement

Measurement of IPW requires synchronized acquisition of:

Power and energy, via hardware meters or platform telemetry (sampling rates ≥10 Hz recommended (Tschand et al., 15 Oct 2024)).
Inference throughput and latency, typically at steady-state execution and excluding non-inference phases.
Accuracy or solution quality, reported at benchmark-required targets.

MLPerf Power provides a procedural “recipe”:

Record instantaneous power $P(t)$ over $\Delta t$ .
Calculate $E = \int_{t_\mathrm{start}}^{t_\mathrm{stop}} P(t)\,dt$ , then $P = E/\Delta t$ .
Collect $N_{\mathrm{inf}}$ , compute $T = N_{\mathrm{inf}} / \Delta t$ .
Normalize throughput to achieved accuracy $Q$ to obtain $Q \times T/P$ (Tschand et al., 15 Oct 2024).

Experimental Design for Local AI

"Intelligence per Watt: Measuring Intelligence Efficiency of Local AI" (Saad-Falcon et al., 11 Nov 2025) established:

Datasets: >1M real-world queries across chat, reasoning, and knowledge tasks.
Devices: 8 local and cloud accelerators.
Metric: $\mathrm{IPW}(m,h) = \mathbb{E}_{q \sim Q}[\mathrm{acc}(m,q)] / \mathbb{E}_{q \sim Q}[P(m,h,q)]$ for model $m$ on hardware $h$ .
Power: Sampled every 50 ms, accuracy judged per query, configured batch size 1, maximum output 32k tokens.
Direct comparison of local versus cloud efficiency under identical model loads.

System-Level Benchmarking

For datacenter-class systems and large-scale racks (e.g., Cerebras WSE-3 vs Nvidia DGX H100/B200 (Kundu et al., 11 Mar 2025)), IPW is evaluated using theoretical peak FLOPS divided by nominal power for ISO-space and ISO-power standardized deployments.

3. Architectural and Algorithmic Techniques to Enhance IPW

In-memory and Local Data Processing

Storing all model weights and intermediate activations on-die (e.g., 9 MB SRAM for CNN-DSA (Sun et al., 2018), 44 GB SRAM for Cerebras WSE-3 (Kundu et al., 11 Mar 2025)) eliminates off-chip DRAM access and sharply reduces data-movement energy—the dominant component in many AI workloads.

Uniformity of Operation

Mapping all layer types to a single primitive (e.g., 3×3 convolution as the only accelerator operation (Sun et al., 2018)) enables deep hardware pipeline optimization, maximum reuse, and minimizes architectural overhead $F(E)$ . Advanced systems (WSE-3) similarly use extremely high on-wafer bandwidth, 2D mesh fabric, and minimized latency for higher utilization and perf/Watt (Kundu et al., 11 Mar 2025).

Feature Channel Compression and On-Chip FC

Reducing feature vector dimensionality at extraction stages (e.g., compressing from 7×7×512 to 7×7×1 with ≤1.3% accuracy loss (Sun et al., 2018)) further enables microcontroller-class inference within tight power budgets.

Quantization and Software Optimization

FP8 quantization and operator fusions can improve $\eta$ by up to 50% (Tschand et al., 15 Oct 2024). MLPerf Power data shows advanced software stacks can have nearly the same impact as new ASIC generations.

4. Quantitative Comparative Results

IPW values are directly comparable across several categories of contemporary AI systems:

System/Architecture	Power (W)	Throughput (TOPS/FLOPS/samples)	IPW (Efficiency)	Reference
CNN-DSA (28nm, mobile)	0.4	3.73 TOPS	9.3 TOPS/W	(Sun et al., 2018)
Google TPU v4	5,000	115,000 samples/sec (ResNet-50)	23.0 samples/J	(Tschand et al., 15 Oct 2024)
Nvidia A100-DGX	5,800	130,000 samples/sec	22.4 samples/J	(Tschand et al., 15 Oct 2024)
Cerebras CS-3 (FP8)	46,000	250 PFLOPS	5.43 PFLOPS/kW	(Kundu et al., 11 Mar 2025)
Nvidia DGX B200 (FP8)	42,900	216 PFLOPS	5.03 PFLOPS/kW	(Kundu et al., 11 Mar 2025)
Local LLMs (Apple M4 Max,QWEN3-32B)	---	---	1.97×10⁻³ acc/W (accuracy/W)	(Saad-Falcon et al., 11 Nov 2025)
Cloud LLMs (B200, QWEN3-32B)	---	---	2.75×10⁻³ acc/W (1.40× higher)	(Saad-Falcon et al., 11 Nov 2025)

These figures highlight large inherent differences driven by architectural design, process node, memory placement, and algorithmic mapping.

5. Trade-offs, Scaling, and Thermodynamic Constraints

Power–Accuracy–Performance

Trade-offs arise when raising target accuracy (e.g., BERT from 99% to 99.9% typically halves energy efficiency (Tschand et al., 15 Oct 2024)), or with batching, where throughput increases but power spikes can erode latency gains. For LLMs, hybrid local-cloud routing can save up to 80% energy versus cloud-only inference, provided accurate domain-level dispatch (Saad-Falcon et al., 11 Nov 2025).

Thermodynamic Limits

Fundamental bounds (e.g., via Landauer’s principle) strictly limit how low watts per intelligence ( $\phi(E)$ ) can be achieved. The lower bound tightens with improved architectural overhead $F(E)$ , reversibility, and increased probability or simplicity of internal state transitions (Perrier, 1 Apr 2025). Algorithmic adaptivity has inherent energy costs depending on transition complexity and probability.

Architectural and Cost Considerations

Wafer-scale integration (CS-3) achieves leading IPW, but with substantial capitalization and packaging challenges. Redundancy logic, advanced cooling, and power delivery impose system-level overheads that only shrink with improved process, yield management, and economies of scale (Kundu et al., 11 Mar 2025). Local accelerators have yet to close the 1.4–1.8× IPW gap to cloud devices but show rapid (5.3× from 2023–2025) improvement trajectories (Saad-Falcon et al., 11 Nov 2025).

6. Design Guidelines and Best Practices

From both theory and empirical evaluation, several principles are established:

Minimize irreversible operations (bit erasures) to approach the Landauer energy floor.
Reduce architectural overhead via co-designed algorithms and hardware: in-memory processing, high-bandwidth fabrics, pipelining (Perrier, 1 Apr 2025, Sun et al., 2018).
Exploit parallelism and batching to amortize fixed per-bit energy costs.
Optimize quantization and primal compute mapping: use low-precision, universal convolutional primitives, or token-efficient LLM decoders (Tschand et al., 15 Oct 2024, Sun et al., 2018).
Balance specialization and generality to trade off flexibility for minimum possible energy per useful task (Perrier, 1 Apr 2025).

For system benchmarking, MLPerf Power’s accuracy-normalized η and IPW provide cross-architecture, cross-workload comparability essential for sustainability reporting and regulatory compliance (e.g., EU AI Act Article 53) (Tschand et al., 15 Oct 2024).

7. Future Prospects and Open Challenges

Current trends project sustained growth in IPW via:

Continued model architectural improvements (e.g., LLM instruction-tuning, hybrid routing).
Integration of on-die memory with advanced packaging (e.g., 3D stacking).
Further process scaling (e.g., <3 nm), power gating, and dynamic voltage/frequency scaling.
Optimized software stacks, operator fusion, quantization, and memory hierarchies.
System-level co-design for edge and datacenter deployment under evolving regulatory and sustainability constraints.

Open challenges remain in closing the local-vs-cloud efficiency gap, achieving fundamental thermodynamic minima, and maintaining reliability and economic viability at extreme integration scales. Expanded benchmarking, including accuracy-per-Joule metrics and standardized reporting, will guide both hardware and algorithmic R&D toward maximal intelligence efficiency per watt.