Intelligence per Watt: Measuring Intelligence Efficiency of Local AI (2511.07885v1)

Abstract: LLM queries are predominantly processed by frontier models in centralized cloud infrastructure. Rapidly growing demand strains this paradigm, and cloud providers struggle to scale infrastructure at pace. Two advances enable us to rethink this paradigm: small LMs (<=20B active parameters) now achieve competitive performance to frontier models on many tasks, and local accelerators (e.g., Apple M4 Max) run these models at interactive latencies. This raises the question: can local inference viably redistribute demand from centralized infrastructure? Answering this requires measuring whether local LMs can accurately answer real-world queries and whether they can do so efficiently enough to be practical on power-constrained devices (i.e., laptops). We propose intelligence per watt (IPW), task accuracy divided by unit of power, as a metric for assessing capability and efficiency of local inference across model-accelerator pairs. We conduct a large-scale empirical study across 20+ state-of-the-art local LMs, 8 accelerators, and a representative subset of LLM traffic: 1M real-world single-turn chat and reasoning queries. For each query, we measure accuracy, energy, latency, and power. Our analysis reveals $3$ findings. First, local LMs can accurately answer 88.7% of single-turn chat and reasoning queries with accuracy varying by domain. Second, from 2023-2025, IPW improved 5.3x and local query coverage rose from 23.2% to 71.3%. Third, local accelerators achieve at least 1.4x lower IPW than cloud accelerators running identical models, revealing significant headroom for optimization. These findings demonstrate that local inference can meaningfully redistribute demand from centralized infrastructure, with IPW serving as the critical metric for tracking this transition. We release our IPW profiling harness for systematic intelligence-per-watt benchmarking.

• The paper introduces intelligence per watt (IPW) as a unified metric to assess the energy-accuracy trade-offs of local AI models.
• The paper employs hardware-software co-design, automated telemetry, and extensive real-world benchmarks to compare local LMs with cloud accelerators.
• The paper demonstrates significant improvements, including 5.3x efficiency gains and notable resource savings through intelligent routing and quantization.

Intelligence per Watt: Metricization and Analysis of Local AI Efficiency

The paper "Intelligence per Watt: Measuring Intelligence Efficiency of Local AI" (2511.07885) provides a rigorous, system-level framework for quantifying the efficiency of local LLM inference using the unified metric of intelligence per watt (IPW). It delivers comprehensive empirical answers to the viability of small local LLMs (≤20B parameters) running on local accelerators as a scalable complement or alternative to frontier cloud inference, especially under constrained power and cost scenarios. The work is grounded in hardware-software co-design methodology, automatic telemetry instrumentation, and naturalistic benchmarking against 1M real-world LLM queries.

Motivation and Metric Formalization

The exponential growth of LLM inference workloads imposes acute stress on centralized cloud infrastructure, with datacenter energy, hardware, and capital expenditures forecast to scale to unprecedented levels by 2030. Inference workloads, not just training, increasingly dominate resource allocation. The practical retrenchment toward local AI is enabled by two convergent trends: open-access small LMs approach frontier model performance on core benchmarks, and consumer/edge hardware (Apple M4 Max, AMD Ryzen AI, etc.) now delivers sufficient memory and compute for interactive LAN inference with moderate power draw.

IPW is formally defined as

$$\mathrm{IPW}(m, h) = \frac{\mathbb{E}_{q \sim Q}[\text{acc}(m, q)]}{\mathbb{E}_{q \sim Q}[P(m, h, q)]}$$

where $m$ is model, $h$ is hardware, $q$ is query, and $P$ denotes instantaneous power consumption. APJ and PPJ metrics extend to per-query energy and perplexity-based efficiency accounting.

Large-Scale Benchmarking and Profiling Harness

Experimental Setup

• Models: 20+ open/local LMs, including QWEN3, GPT-OSS, GEMMA3, IBM GRANITE, and validated SOTA closed models (GPT-5, GEMINI 2.5 PRO, CLAUDE SONNET 4.5).
• Accelerators: 8+ hardware backends, local (Apple, AMD consumer, NVIDIA RTX) and cloud/enterprise (NVIDIA B200, SambaNova SN40L), spanning 40–768 GB memory and 145–1000 W TDP.
• Tasks: 1M queries (WILDCHAT, NATURALREASONING, MMLU PRO, SUPERGPQA), covering chat, reasoning, expert knowledge, and economic breadth.
• Instrumentation: Cross-platform harness sampling power, energy, latency, memory, and throughput at high temporal resolution (50ms). NVML, ROCm SMI, and powermetrics used for native telemetry.

Labeling and Evaluation

• Query domain: Annotated using GPT-4O-MINI against the Anthropic Economic Index taxonomy (22 labor categories).
• Accuracy: LLM-as-a-judge rubric (multi-criterion for subjective/chat, correctness-only for technical).
• Model outputs: Standardized decoding (T=0.6, top-p=0.95, top-k=20), up to 32768 tokens.

Key Results and Empirical Findings

Task Coverage and Model Viability

• Coverage: Local LMs can accurately respond to 88.7% of single-turn chat and reasoning queries. Domain variance is strong (≥90% for creative-media queries, ≈68% for technical domains like Engineering).
• Ensembling and routing: Best-of-local ensemble routing lifts coverage further, capitalizing on model complementarity across domains. Individual LMs (GPT-OSS-120B) achieve ≈71.4% coverage.
• Longitudinal trend: Local LMs' ability to match frontier models on queries rose from 23.2% (2023) to 48.7% (2024), 71.3% (2025) – a 3.1x increase in two years.

Intelligence Efficiency Over Time

• IPW Gain: Compound efficiency gains from hardware and model improvements total 5.3x (2023–2025) in accuracy per watt.
  • Model improvement (MIXTRAL-8x7B→GPT-OSS-120B): 3.1x
  • Accelerator improvement (NVIDIA H100→Blackwell): 1.7x
• Hardware gap: Cloud accelerators maintain a substantial instantaneous and end-to-end efficiency advantage (Apple M4 Max: 1.40x lower IPW, 1.6–2.3x lower IPJ compared to NVIDIA B200).
• Quantization: FP8/FP4 quantization yields minor accuracy loss (2–3%/step), but 3x+ energy savings, supporting low-bit deployment for most chat/reasoning applications.

Resource Savings via Hybrid Local-Cloud Routing

• Oracle routing: Perfect query-to-model assignment achieves 80.4% energy, 77.3% compute, and 73.8% cost savings over cloud-only deployment.
• Practical routers: Even with 80% routing accuracy, 64.3% energy, 61.8% compute, and 59.0% cost reductions are maintained, with no loss of answer quality by fallback-to-cloud on misassignments.
• Domains: Local handling is dominant for creative/social tasks (>90% coverage), but challenges remain in technical domains (40–68% solvability for Architecture, Science).

Comparison: Open-Source vs. Closed-Source Frontier Models

• Open-source frontier-scale models (QWEN3-235B-A22B) approach closed-source performance: 71.8% vs. 77.9% average accuracy. On expert benchmarks, the gap narrows to 3.4–5.1%, but widens to >12% on naturalistic reasoning tasks. With ≤20B parameter constraints typical of local deployment, accuracy penalty increases to ≈11–13% on SOTA closed models.

Economic Implications

• GDP-weighted accuracy shows scaling local LM performance translates directly into economic relevance: GPT-OSS-120B covers 69.6% of US GDP in chat tasks, QWEN3-235B reaches 23.3–31.9% GDP coverage for advanced reasoning/science. Improving model capabilities for technical queries would unlock substantial economic automation opportunities.

Implementation Tradeoffs and System Considerations

• Model selection: Deploying multiple complementary local LMs and intelligent routing is markedly superior to static assignment and tolerates moderate routing errors.
• Accelerator selection: Unified memory consumer hardware enables competitive performance for most tasks at moderate energy cost, but high-throughput, highly parallel cloud accelerators remain essential for latency- and energy-critical workloads. The efficiency gap justifies ongoing hardware optimization (HBM3e, edge TPUs, tensor cores).
• Quantization: FP8/FP4 is optimal for energy-constrained deployment when minor accuracy loss is acceptable.
• Telemetry/monitoring: High-resolution, vendor-native, multi-hardware profiling is necessary for reproducible energy benchmarking.
• Routing: Query-level domain-aware routers, preference-trained on per-query outcome, provide best trade-off in hybrid systems.

Theoretical and Practical Implications

The IPW metric offers a unified efficiency criterion supporting direct cross-comparison of model-hardware configurations and longitudinal progress mapping. These results empirically substantiate the continuous improvement trajectory of local inference systems, indicating local LMs and consumer accelerators can address a substantial—and rapidly expanding—subset of real-world LLM queries. This portends accelerated adoption of distributed, hybrid inference architectures mitigating the long-term resource intensiveness of centralized AI serving.

From a systems research point of view, further optimization of local inference involves:

• Tight co-design of model architectures and accelerators for higher FLOPs/W
• Advanced quantization and low-rank adaptation for memory/power reductions
• Improved real-time query routing and model ensembling for completeness-optimized coverage
• Standardized, open telemetry aggregation for continual benchmarking as hardware and models evolve
• Exploration of collaborative token-wise protocols to further blur the boundary between local and cloud compute

Conclusions

"Intelligence per Watt: Measuring Intelligence Efficiency of Local AI" establishes intelligence per watt as the critical metric for tracking the transition from cloud-centric toward hybrid and eventually predominantly local LLM inference. Through systematic benchmarking, it demonstrates that local small LMs, when intelligently routed and deployed on contemporary consumer hardware, already deliver highly competitive accuracy and pronounced resource savings for the majority of single-turn chat and reasoning queries—albeit with remaining limitations in technically demanding domains. The efficiency gap to cloud accelerators justifies continued hardware specialization for local AI workloads. As both accelerator hardware and LM architectures evolve, routing strategies and quantization methods offer substantial leverage in closing the gap, with direct implications for AI energy economics and large-scale system deployment. The profiling harness released enables ongoing progress tracking and reproducibility within the field. Future work should further investigate dynamic, context-adaptive routing, model augmentation protocols, and optimization at the memory-controller and logic-gate level for maximal intelligence-per-watt efficiency.