Graffitist Framework for Neural Quantization

• Graffitist Framework is a quantization tool that converts floating-point neural networks into hardware-efficient fixed-point models using trained quantization thresholds.
• The framework leverages TensorFlow graph transformations in both static and retrain modes to optimize network weights and thresholds, ensuring compatibility with high-throughput hardware.
• Empirical evaluations demonstrate that Graffitist maintains near-floating-point accuracy for challenging architectures like MobileNet by jointly optimizing quantization parameters and network weights.

The Graffitist Framework is a software system designed to enable accurate and efficient quantization of deep neural networks for fixed-point inference, primarily targeting high-throughput hardware implementations. Built atop TensorFlow, it automates the conversion of floating-point models into hardware-amenable fixed-point representations by implementing a quantization regime termed Trained Quantization Thresholds (TQT). This methodology jointly optimizes network weights and quantization clipping thresholds using backpropagation, yielding quantized inference graphs that achieve near-floating-point accuracy, even for architectures historically challenging to quantize such as MobileNet. The framework incorporates a set of graph transformations and retraining capabilities to ensure that the quantized network remains both accurate and fully compatible with hardware requirements, including power-of-2 scaling and per-tensor quantization.

1. Design Objectives and Integration with TensorFlow

Graffitist addresses two primary concerns in neural network deployment:

• Achieving computational efficiency and memory reduction via low-precision (fixed-point) inference.
• Maintaining a minimal accuracy gap relative to baseline floating-point models, particularly for complex or sensitive architectures.

The framework integrates directly with TensorFlow's computational graph abstraction. It systematically applies a series of code transformations, including the folding of batch normalization layers into convolutional or fully-connected layers, merging or replacing operations to achieve compatibility with hardware constraints, and enforcing quantization-consistent graph rewrites. By operating directly at the level of computational graphs, Graffitist ensures that the resulting inference graph matches bitwise the output of the target hardware implementation.

Graffitist provides two operational modes:

• Static mode: Quantization thresholds are set via calibration using a representative dataset, leveraging metrics such as symmetric KL divergence or percentile-based statistics.
• Retrain mode: Both network weights and quantization thresholds are retrained end-to-end via standard gradient descent, using a global loss function such as softmax cross-entropy.

2. Trained Quantization Thresholds (TQT): Methodology

TQT re-conceptualizes the quantizer threshold as a set of learnable parameters, optimized during normal backpropagation alongside the network weights. In contrast to conventional approaches with statically set thresholds, this technique enables the quantization range—i.e., the trade-off between clipping (information loss at extremes) and precision (granularity within representable values)—to be optimized according to the model’s global loss.

The core quantization operation for a tensor input $x$ with scale $s$ is:

$$q(x; s) := \mathrm{clip}(\left\lfloor x/s \right\rceil ; n, p) \cdot s$$

where:

• $n$ and $p$ are the minimum and maximum representable quantized values. For signed 8-bit: $n = -128,\ p = 127$.
• $s$ is related to the quantizer clipping threshold $t$ via $s = 2^{-f}$ with $f = \lceil \log_2 t \rceil$, ensuring scale factors are powers of two.

The backward gradient with respect to $\log_2 t$ is analytically derived (not merely set via a naive identity STE), yielding:

$$\nabla_{\log_2 t} q(x; s) = s \cdot \ln 2 \cdot \begin{cases} \left\lfloor \frac{x}{s} \right\rceil - \frac{x}{s}, & n \leq \left\lfloor \frac{x}{s} \right\rceil \leq p \ n, & \left\lfloor \frac{x}{s} \right\rceil < n \ p, & \left\lfloor \frac{x}{s} \right\rceil > p \end{cases}$$

Thresholds are optimized in the log domain to ensure positivity and numerical stability.

3. Hardware-Oriented Constraints: Power-of-2 Scaling and Per-Tensor Quantization

To facilitate direct mapping onto fixed-point hardware (such as FPGAs or ASICs), Graffitist enforces:

• Uniform symmetric quantization: Zero points are fixed at zero, eliminating mixed-precision cross terms.
• Power-of-2 scale factors: This enables scaling to be implemented via inexpensive integer bit-shifts.
• Per-tensor scaling: All values in a tensor (activation or weight) are scaled identically, dramatically simplifying the hardware pipeline at a slight potential cost to representational fidelity (relative to per-channel scaling).

These design decisions ensure that the quantization process is not only software-efficient but also conducive to deterministic, verifiable hardware deployment.

4. Graph Transformations and Implementation Pipeline

Graffitist automates graph-level optimizations that include:

• Batch norm folding, which merges batch normalization parameters into preceding affine layers prior to quantization.
• Replacement of average pooling with equivalent depthwise convolution, and other rewrites to maintain inference exactness.
• Pruning of computational subgraphs to ensure only quantizable, hardware-executable paths remain.

A fused quantization kernel is provided to optimize both the computational graph and memory footprint, leveraging the properties of the TQT gradient to avoid unnecessary tensor duplication in backward passes.

5. Empirical Results and Model Performance

The framework is validated extensively using ImageNet tasks and a broad array of architectures: VGG, Inception, ResNet, MobileNet, and DarkNet. Quantization to INT8 (8-bit quantization for weights and activations) using TQT and retraining achieves top-1 and top-5 accuracies on par with or marginally below the floating-point baselines. For challenging models like MobileNet, where traditional per-tensor quantization substantially degrades accuracy, retraining both thresholds and weights with TQT closes the gap almost entirely.

INT4 experiments (4-bit weights, 8-bit activations), though lossy, still preserve significant fractions of accuracy, indicating the robustness and flexibility of the method.

6. Analytical Foundations and Optimization Dynamics

The analytical analysis provided in the framework emphasizes the optimization dynamics of threshold learning. Training thresholds in the log domain yields a balance: when most values are un-clipped, gradients favor increasing precision (lower thresholds), while excessive clipping pushes thresholds outward for greater dynamic range. The optimization stability, scale-invariance, and practical learning rate settings are derived and justified to support reliable training.

Threshold learning is intimately coupled to global loss, making the quantizer context-aware and improving retention of accuracy in quantized models. Advisory notes are provided for configuring Adam optimizer hyperparameters to balance threshold and weight updates.

7. Future Directions and Research Extensions

The framework's open questions center around relaxing the conservative constraints:

• Allowing non-power-of-2 scale factors for enhanced precision if hardware supports it.
• Extending to per-channel quantization, which may prove especially beneficial in layers (such as depthwise convolutions) with high channel variance.
• Adapting TQT to non-uniform or asymmetric quantizers, with threshold learning generalized accordingly.

Such directions promise further gains in the tradeoff between hardware efficiency and accuracy.

---

In summary, the Graffitist Framework exemplifies a practical and theoretically grounded solution for neural network quantization, combining hardware-constrained quantizer design, robust threshold retraining via TQT, and fully automated graph conversion for deployable inference with near-floating-point fidelity (1903.08066).