Edge Quantization: Precision for Edge AI

• Edge quantization principle is a design rule that allocates precision based on key constraints (memory, latency, energy) rather than relying solely on compression.
• It employs hardware-aware mixed-precision strategies, using techniques like layer-wise bit assignments and surrogate gradients to achieve an optimal accuracy–efficiency tradeoff.
• The principle extends beyond neural inference to mathematical formulations and edge conductance, highlighting its role in context-specific discretization across various domains.

Edge quantization principle names a family of quantization doctrines in which precision is determined by the constraints of the edge rather than by compression alone. In edge AI, the principle is usually formulated as joint control of memory footprint, memory traffic, latency, energy, and hardware support through low-bit or mixed-precision representations of weights, activations, features, or tokens; in other technical literatures, closely related usage appears in Mellin-edge quantisation for corner operators and in the quantization of edge conductance for magnetic barriers and guides (Schaefer et al., 2022, Kimhi et al., 2022, Schulze et al., 2012, Dombrowski et al., 2010).

1. Research scope and principal meanings

Taken together, the literature suggests that “edge quantization” is not a single method but a recurrent design rule: preserve the variables that matter for the task or operator, and compress the variables whose reduced precision is most compatible with the target edge environment.

Usage	Representative papers	Core rule
Edge DNN inference	(Schaefer et al., 2022, Liu et al., 25 Feb 2026, Kimhi et al., 2022, Bouzouad et al., 24 Mar 2026)	Allocate precision under memory, latency, energy, and device constraints
Task-specialized edge deployment	(Li et al., 10 Apr 2026, Li et al., 2024, Shen et al., 2024, Fu et al., 25 Sep 2025)	Preserve fragile task signal with sensitivity-aware, privacy-aware, or token-aware quantization
Mathematical and physical usage	(Schulze et al., 2012, Dombrowski et al., 2010)	Quantisation tied to edge singularities or to edge conductance

Within edge AI, the dominant shift is from uniform global bit-width to heterogeneous, hardware-aware assignment. “Edge Inference with Fully Differentiable Quantized Mixed Precision Neural Networks” formalizes this shift as a hardware-aware accuracy–efficiency tradeoff in which the relevant metric is the sum of weight memory and the sum of activation feature-map memory, rather than only parameter count or only maximum activation size (Schaefer et al., 2022). “SigmaQuant” makes the same point in layer-wise form, arguing that precision should be allocated where the model is statistically sensitive and removed where the hardware benefits most, under explicit device constraints such as memory, latency, and energy (Liu et al., 25 Feb 2026).

2. Memory-centric and hardware-aware principles in neural inference

A central formulation appears in mixed-precision CNN quantization for edge inference, where the loss augments cross-entropy with rectified quadratic penalties on total weight memory and total activation memory:

$$\begin{aligned} L = & \ CE(x, y) + \beta \max \left( \left( \sum^L_{l=1} \sum^C_{c=1} b^w_{lc} \cdot s_{lc}^w \right) - t^w ,0 \right)^2\ & + \beta \max \left( \left( \sum^L_{l=1} b^a_l \cdot s_l^a\right) - t^a, 0\right)^2. \end{aligned}$$

Here the learned bit-widths are per output-channel for weights and per tensor for activations, and the use of $\max(\cdot,0)^2$ means that once a budget is satisfied it is no longer penalized (Schaefer et al., 2022). The same work reports that “sum of weights and activation feature maps” correlates strongly with ACE, with a correlation coefficient of 0.956 across configurations, and gives concrete low-memory frontier points such as EfficientNet-Lite0 at $4.14$ MB of weights and activations with $67.66\%$ top-1 accuracy and MobileNetV2 at $3.51$ MB with $65.39\%$ (Schaefer et al., 2022).

Other hardware-aware formulations make the same break from global precision. “SigmaQuant” uses layer-wise heterogeneous quantization driven first by weight standard deviation $\sigma_\ell$ and then by KL divergence between float and quantized weight distributions, over the discrete set $\{2,4,6,8\}$, with explicit targets such as

$$A(\{b_\ell\}) \ge A_t, \qquad M(\{b_\ell\}) \le M_t$$

and, in compute-targeted mode,

$$\mathrm{BOPs} \triangleq \sum_{\ell} B_w(\ell)\,B_a(\ell)\,\mathrm{MACs}(\ell).$$

At equal accuracy it reports up to $\max(\cdot,0)^2$0 less memory, and at equal model size up to $\max(\cdot,0)^2$1 higher accuracy (Liu et al., 25 Feb 2026).

For edge LLMs, the same principle is hardware-specific rather than architecture-specific. “APreQEL” treats deployment as a multi-QoS problem with memory footprint, latency, and perplexity, and explicitly states that “smaller models do not automatically run faster”; candidate mixed-precision distributions are estimated from uniform quantization measurements on the target hardware and ranked by TOPSIS under user-defined weights (Bouzouad et al., 24 Mar 2026). “AMED” is still more direct: it rejects FLOPs, MACs, BOPs, and model size as sufficient deployment objectives, uses direct latency signals from a target-specific simulator, and models layer precision transitions as a stochastic process because “the optimal minimum changes as the precision changes” (Kimhi et al., 2022).

A recurring edge-specific refinement is to design the quantization format for the execution primitive itself. “ELUTQ” does this for CPU-based edge LLM inference by replacing uniform low-bit coding with Hierarchical Linear Quantization,

$\max(\cdot,0)^2$2

so that low-bit weights remain directly consumable by bit-serial LUT-based GEMM without dequantization overhead during matrix multiplication (Nie et al., 22 Oct 2025).

3. Optimization mechanisms and training dynamics

In edge quantization research, low-bit deployment is not treated as a one-shot discretization step but as an optimization problem whose stability depends on calibration, surrogate gradients, and training schedule. The fully differentiable mixed-precision CNN framework uses uniform symmetric quantization

$\max(\cdot,0)^2$3

with

$\max(\cdot,0)^2$4

It initializes weights with Gaussian calibration and activations with the $\max(\cdot,0)^2$5 percentile, and reports that this calibration alone improved homogeneous 3-bit EfficientNet-Lite0 by up to $\max(\cdot,0)^2$6 over sample-max calibration. Its backward-pass result is asymmetric: EWGS-style scaling for weights and inverse-hyperbolic-tangent scaling for activations, with $\max(\cdot,0)^2$7, and its learning schedule has three 50-epoch phases: homogeneous pre-training, precision learning with gradually increased $\max(\cdot,0)^2$8 and precision updates every $\max(\cdot,0)^2$9 steps, and final fine-tuning with frozen quantizers and $4.14$0 (Schaefer et al., 2022).

A different stabilization principle appears in adaptive INT4 quantization for biased activations. “A High-Performance Adaptive Quantization Approach for Edge CNN Applications” learns, for each layer, not only a scaling factor but also an activation zero-point:

$4.14$1

with gradients derived directly from the task loss. In this scheme the zero-point adapts only when clipping is occurring:

$4.14$2

The paper reports that adding the shift improves top-1 accuracy by roughly $4.14$3 to $4.14$4, and that on YOLOv4 with Leaky-ReLU the 4-bit quantized model with shifting loses only about $4.14$5 mAP relative to full precision (Chin et al., 2021).

A third line replaces direct quantizer modification with representation shaping and stability control. “Quantized Convolutional Neural Networks Through the Lens of Partial Differential Equations” inserts TV smoothing layers

$4.14$6

before activations, and studies symmetric residual dynamics

$4.14$7

Its stability condition

$4.14$8

ensures that perturbations do not amplify across depth. The reported effect is a reduction of 4-bit quantization MSE from $4.14$9 to $67.66\%$0 in an example after TV smoothing, and markedly lower consistency MSE for stable ResNet and MobileNetV2 variants (Ben-Yair et al., 2021).

The regularization view is more empirical but still edge-oriented. “Quantization of Deep Neural Networks for Accurate Edge Computing” argues that quantization “essentially imposes regularization on weight representations,” and reports accuracy improvements of $67.66\%$1, $67.66\%$2, and $67.66\%$3 on FCN segmentation, VGG-16 ImageNet classification, and Deep Speech ASR, with $67.66\%$4x–$67.66\%$5x memory reduction (Chen et al., 2021).

4. Task-aware, privacy-aware, and token-aware extensions

Once quantization is specialized to a task, the edge principle becomes sensitivity-aware rather than uniformly low-bit. “DeFakeQ” states that deepfake detection depends on weak, sparse, and unevenly distributed forensic cues, so it uses adaptive bidirectional compression: horizontal layer/block bit-width assignment and vertical restoration of a small subset of sensitive channels to FP32. Its horizontal loss is

$67.66\%$6

and the full objective combines reconstruction, horizontal bit allocation, and vertical contrastive alignment. On DFD, examples include DeiT-Base at GT $67.66\%$7, BRECQ $67.66\%$8, Adalog $67.66\%$9, FIMA-Q $3.51$0, and DeFakeQ $3.51$1; on Android phones it reports per-image latency no more than $3.51$2 ms, average latency around $3.51$3 ms per frame, and power consumption of $3.51$4 mW (Li et al., 10 Apr 2026).

In privacy-sensitive healthcare, the principle changes again: calibration must avoid real data. “DFQ-SAM” quantizes SAM-Med2D with post-training quantization using synthesized calibration data, pseudo-positive label evolution, patch-similarity distribution matching, and scale reparameterization for post-LayerNorm activations. Its reported 4/4 result reduces the model from 358 MB and 19866 G BOPs to 46.5 MB and 346 G BOPs, with GPU latency improvements of $3.51$5x to $3.51$6x, while the synthesized-data calibration outperforms Gaussian noise and in many cases real-data calibration (Li et al., 2024).

For small LLMs on mobile devices, EdgeQAT localizes the fragility to self-attention, especially query and key activations, and adds entropy-guided and distribution-guided distillation together with token-adaptive mixed precision. The token policy assigns either 8 bits or 4 bits to each token according to an attention-derived importance indicator:

$3.51$7

and then quantizes with the token-specific clipping range. On BLiMP, LLaMA-58M reaches 69.4 average at W4A8 versus 69.7 for FP16, and the system reports on-device speedup up to 2.37x compared with FP16 counterparts (Shen et al., 2024).

The same task-oriented nonuniformity appears in wireless edge inference. “Hybrid RIS-Aided Digital Over-the-Air Computing for Edge AI Inference” assigns bits per feature block, not per scalar, and uses the class-separability surrogate

$3.51$8

where $3.51$9 is an explicit feature importance indicator. Its block error bound shows that increasing $65.39\%$0 decreases pure quantization distortion through $65.39\%$1 but increases the misalignment and channel-noise terms through $65.39\%$2, so “more bits is always better” is false in this setting (Fu et al., 25 Sep 2025).

5. Deployment co-design, empirical frontiers, and recurring limitations

A recurring misconception in edge quantization is that lowering nominal bit-width monotonically lowers deployment cost. Multiple works reject this explicitly. “AMED” states that ultra-low-bit quantization does not consistently improve chip-level performance because memory and communication bottlenecks can dominate (Kimhi et al., 2022). “APreQEL” states that smaller models do not automatically run faster, and its latency-focused solutions often settle around 4-bit rather than the minimum available precision (Bouzouad et al., 24 Mar 2026). “HRD-AirComp” shows that larger bit budgets can worsen aggregate detection even while reducing pure quantization distortion (Fu et al., 25 Sep 2025).

This hardware realism is now reflected in reported deployment numbers. “ELUTQ” shows that for LLaMA3-8B, HLQ reduces perplexity by about 8% at 3-bit and 85% at 2-bit under post-training quantization, and that 2-bit LLaMA2-7B achieves over 25 tokens/s on an Apple M2 chip with 4 threads and batch size 1 (Nie et al., 22 Oct 2025). “APreQEL” evaluates on Jetson Orin AGX and reports hypervolume gains of $65.39\%$3, $65.39\%$4, and $65.39\%$5 over uniform quantization for Llama3.1, Phi3.5, and Qwen3-4B (Bouzouad et al., 24 Mar 2026). “Quant-Trim” shifts the problem to training-time robustness against backend heterogeneity, combining progressive fake quantization and reverse pruning to produce a hardware-neutral checkpoint that is agnostic to symmetric/asymmetric, per-tensor/per-channel, and INT8/INT4 choices, exports to standard ONNX, and avoids per-backend retraining (Dhahri et al., 19 Nov 2025).

The literature is equally explicit about limits. The fully differentiable mixed-precision CNN method imposes size through a penalty rather than a hard constraint, so it “does not guarantee exact budget satisfaction or Pareto-optimality,” and some runs land slightly off the frontier (Schaefer et al., 2022). DeFakeQ notes that VEFT restores some channels to FP32, which “could complicate deployment on accelerators optimized for uniform integer execution” even though the paper argues compatibility is still practical (Li et al., 10 Apr 2026). DFQ-SAM states that real-world deployment is currently validated only on mobile devices, not on more constrained IoT or gateway hardware (Li et al., 2024). APreQEL relies on the approximation

$65.39\%$6

which assumes equal per-layer contribution to global QoS and ignores layer-size and operator heterogeneity (Bouzouad et al., 24 Mar 2026).

Objectively, the controversy is not whether edge quantization is useful, but which proxy is trustworthy. The field increasingly distinguishes memory-centric, latency-centric, and task-centric objectives, and repeatedly treats proxy-only optimization as insufficient when deployment is the target.

6. Non-neural meanings of edge quantization

Outside neural inference, the term appears in mathematically distinct settings. In analysis on singular manifolds, “Mellin-edge quantisation” is a rule for corner-degenerate symbols on manifolds with second-order singularities. The outer cone variable is quantized by Mellin transform, but the Mellin symbol takes values in the edge algebra over a singular base $65.39\%$7. The central theorem states that an amplitude of the form

$65.39\%$8

can be written as

$65.39\%$9

with $\sigma_\ell$0. Here edge quantization is not model compression but an operator-valued quantisation rule adapted to the most singular stratum of a corner manifold (Schulze et al., 2012).

In mathematical physics, “quantization of edge currents” refers to an integer-valued conductance invariant for magnetic Schrödinger operators with generalized Iwatsuka fields. The edge conductance is defined by

$\sigma_\ell$1

and for a generalized Iwatsuka potential satisfies

$\sigma_\ell$2

The value is stable under magnetic perturbations localized near the interface and obeys a sum rule for left/right magnetic structures (Dombrowski et al., 2010).

These non-neural uses share a structural resemblance with the edge-AI literature: quantization is not treated as an isolated discretization step, but as a rule imposed by the geometry, interface, or deployment substrate that defines the edge.