SegQuant: Semantics-Aware Diffusion PTQ

• SegQuant is a post-training quantization framework for diffusion models that introduces SegLinear for graph-guided semantic segmentation and DualScale for polarity-aware activation quantization.
• It integrates existing optimizer and calibrator components with novel method-specific additions to improve the accuracy–deployability trade-off across diverse model families.
• Empirical results on SD3.5, FLUX, and SDXL demonstrate reduced FID and enhanced image quality, highlighting the framework’s practical impact on low-precision inference.

Searching arXiv for the SegQuant paper and closely related diffusion-model PTQ context.

Searching "SegQuant diffusion post-training quantization".

SegQuant is a post-training quantization framework for diffusion models that is designed to improve the usual accuracy–deployability trade-off by making quantization semantics-aware, generalizable across model families, and compatible with mainstream deployment workflows. It is introduced as a unified framework that combines existing optimizer and calibrator components with two method-specific additions—SegLinear and DualScale—to reduce quantization error in diffusion backbones without retraining. In the paper’s formulation, SegQuant targets both Transformer-based and UNet-based diffusion models, and is evaluated on Stable Diffusion 3.5 Medium, FLUX.1-dev, and SDXL (Zhang et al., 20 Jul 2025).

1. Position within diffusion-model post-training quantization

SegQuant is situated within post-training quantization rather than quantization-aware training. The underlying motivation is that diffusion models are computationally expensive not only because of model size, but because denoising is iterative: every inefficiency in linear layers, activations, and memory traffic is repeated over many sampling steps. The framework is therefore presented as a response to a specific deployment problem: prior diffusion PTQ methods are often effective only because they rely on architecture-specific heuristics, whereas industrial quantization pipelines favor modularity and compatibility across architectures (Zhang et al., 20 Jul 2025).

The paper retains the standard quantizer as the baseline formalism,

$$\hat{x} = \mathrm{clip}\left(\mathrm{round}\left( \frac{x - z}{s} \right), q_{\min}, q_{\max} \right), \qquad x \approx s \cdot (\hat{x} + z),$$

where $s$ is scale, $z$ is zero-point, and $q_{\min}, q_{\max}$ define the integer range. SegQuant does not replace this PTQ foundation. Instead, it modifies how quantization units are defined and how activation ranges are represented, with the explicit aim of preserving visual fidelity while remaining compatible with standard low-precision kernels (Zhang et al., 20 Jul 2025).

A central distinction drawn by the paper is between research methods tailored to particular diffusion backbones and a deployment-oriented framework that can be reused across DiT-style and UNet-style models. This suggests that SegQuant is best understood not as a single quantizer, but as an organizing framework for semantics-aware PTQ in diffusion inference.

2. Framework structure and calibration workflow

SegQuant combines two pre-existing PTQ subsystems with two framework-specific components. The paper describes an Optimizer, instantiated with methods such as SmoothQuant or SVDQuant, and a Calibrator, instantiated with methods such as GPTQ or AMax. These are augmented by SegLinear and DualScale, which define the paper’s main methodological contributions (Zhang et al., 20 Jul 2025).

Subsystem	Function	Instantiation in the paper
Optimizer	Determines quantization transformation	SmoothQuant for 8-bit, SVDQuant for 4-bit
Calibrator	Estimates quantization parameters	AMax or GPTQ
SegLinear	Segment-aware quantization of linear layers	Graph-pattern-guided input/output segmentation
DualScale	Separate treatment of negative and non-negative activations	Applied after SiLU, GELU, and GEGLU

The practical workflow is calibration-driven. A small calibration set is collected and run through the pretrained model. Linear layers may then be segmented using graph pattern matching. In 8-bit settings, SmoothQuant is used and the hyperparameter $\alpha$ is swept from $0.0$ to $1.0$ in increments of $0.1$, with the best value chosen by minimizing mean squared error between quantized and full-precision layer outputs. In 4-bit settings, SVDQuant is used instead, with low-rank setting fixed at $64$. The paper reports two named variants: SegQuant-A, using AMax calibration, and SegQuant-G, using GPTQ calibration (Zhang et al., 20 Jul 2025).

The experimental setup is concrete. All 8-bit quantization uses a per-tensor scheme. In 4-bit settings, weights are per-channel and activations are dynamically per-token. Calibration uses 256 images for SD3 and SDXL, 64 for FLUX 8-bit, and 32 for FLUX 4-bit. All experiments use 50 sampling steps with classifier-free guidance enabled and guidance scale set to $7$. Deployment is implemented with efficient CUDA kernels via CUTLASS, which is part of the framework’s deployment-oriented design (Zhang et al., 20 Jul 2025).

3. SegLinear: graph-guided semantic segmentation of linear layers

SegLinear is the framework’s semantics-aware quantization mechanism for linear operators. The motivating claim is that some linear layers in diffusion backbones are internally heterogeneous: their inputs or outputs are not uniform feature spaces, but semantically distinct branches exposed by computation-graph operations such as chunk, split, stack, and concat. SegLinear therefore quantizes these branches separately rather than imposing a single quantization configuration over the entire layer (Zhang et al., 20 Jul 2025).

For a linear layer

$s$0

SegLinear supports two cases.

In output-segmented quantization, the output dimension of $s$1 is partitioned: $s$2 Each sub-matrix is quantized independently, and the quantized output is formed by concatenation: $s$3

In input-segmented quantization, the input is partitioned as

$s$4

with a compatible partition of $s$5: $s$6 The output is then reconstructed additively: $s$7

The paper is explicit that “graph-based” here means pattern matching over the model DAG rather than a learned graph optimization objective. Nodes correspond to operations and edges denote tensor dependencies; SegLinear inspects local neighborhoods around linear layers and infers segment sizes automatically from graph structure. This interpretation is important because it clarifies that SegLinear is not a graph neural network or a spectral partitioner. Its novelty lies in using graph-exposed semantic boundaries to define quantization groups that are more meaningful than ordinary equal-sized channel groups (Zhang et al., 20 Jul 2025).

Empirically, the framework reports reduced Frobenius norm quantization error for segmented layers. For example, on SD3.5 W8A8, the layer [DiT](https://www.emergentmind.com/topics/diffusion-transformer-model-dit).0.norm1 under GPTQ calibration goes from $s$8 without segmentation to $s$9 with segmentation, and DiT.11.norm1_context goes from $z$0 to $z$1. This supports the paper’s claim that semantic segmentation reduces inter-segment interference during PTQ (Zhang et al., 20 Jul 2025).

4. DualScale: polarity-aware activation quantization

DualScale is introduced to address polarity-asymmetric activations in modern diffusion architectures. The paper argues that activations after SiLU-, GELU-, and GEGLU-like nonlinearities retain many small negative values that remain semantically meaningful, while positive activations occupy a much broader dynamic range. A single quantization scale therefore under-resolves the negative side, and asymmetric quantization, while shifting the zero-point, still allocates bins uniformly over the whole interval (Zhang et al., 20 Jul 2025).

The paper supports this with activation statistics. Reported negative/positive ratios include AdaNorm in DiT at $z$2, FFN in DiT at $z$3, AdaNorm in “Ctrl.” at $z$4, and FFN in “Ctrl.” at $z$5. These values motivate a sign-aware scheme rather than ordinary symmetric or asymmetric activation quantization (Zhang et al., 20 Jul 2025).

DualScale defines separate quantizers for negative and non-negative activations: $z$6 with

$z$7

The activation matrix is split into

$z$8

and the output of a following linear layer is reconstructed as

$z$9

A major design claim is deployment simplicity. Standard asymmetric quantization introduces zero-point correction terms in matrix multiplication, whereas DualScale preserves polarity-aware handling without those additional recovery terms. The paper therefore presents DualScale as a compromise between accuracy and kernel compatibility: it is more expressive than a single symmetric scale, but easier to integrate into standard GEMM-style low-precision execution than full asymmetric quantization (Zhang et al., 20 Jul 2025).

The ablation study indicates that DualScale is the stronger of the two proposed additions when used in isolation. On SD3.5 W8A8 evaluated on MJHQ, the baseline yields FID $q_{\min}, q_{\max}$0; adding SegLinear alone gives $q_{\min}, q_{\max}$1, adding DualScale alone gives $q_{\min}, q_{\max}$2, and combining both gives $q_{\min}, q_{\max}$3. This suggests that SegLinear and DualScale are complementary, but that polarity-aware activation handling contributes more of the standalone gain (Zhang et al., 20 Jul 2025).

5. Empirical performance across model families

SegQuant is evaluated on COCO, MJHQ-30K, and DCI, with 5,000 sampled images for MJHQ-30K and DCI. Quality is assessed against FP16 outputs using FID, LPIPS, PSNR, SSIM, and Image Reward; appendix tables also report CLIP Score and CLIP-IQA. The evaluated models are Stable Diffusion 3.5 Medium (2B), FLUX.1-dev (12B), and SDXL, on NVIDIA RTX 4090 and NVIDIA L20 hardware (Zhang et al., 20 Jul 2025).

Representative results are strongest in W8A8 settings. On MJHQ:

Setting	Strong prior baseline	SegQuant
SD3.5-DiT, W8A8(int), FID	PTQ4DiT: 16.46	SegQuant-G: 12.37
SDXL-UNet, W8A8(int), FID	PTQ4DiT: 7.70	SegQuant-G: 6.19
FLUX-DiT, W8A8(int), FID	Q-Diffusion: 9.41	SegQuant-G: 5.56
SD3.5-DiT, W4A8(int), FID	SVDQuant: 20.58	SegQuant-G: 20.22
FLUX-DiT, W4A8(int), FID	SVDQuant: 7.94	SegQuant-G: 7.78

The detailed metric profile is similarly favorable. On SD3.5 W8A8, SegQuant-G reports FID $q_{\min}, q_{\max}$4, LPIPS $q_{\min}, q_{\max}$5, PSNR $q_{\min}, q_{\max}$6, and SSIM $q_{\min}, q_{\max}$7, while SegQuant-A reports the best Image Reward at $q_{\min}, q_{\max}$8. On SDXL W8A8, SegQuant-G reaches FID $q_{\min}, q_{\max}$9, LPIPS $\alpha$0, PSNR $\alpha$1, and SSIM $\alpha$2. On FLUX W8A8, SegQuant-G reaches FID $\alpha$3, LPIPS $\alpha$4, PSNR $\alpha$5, and SSIM $\alpha$6. These results are the paper’s principal evidence that the framework transfers beyond Transformer diffusion models to UNet-based SDXL as well (Zhang et al., 20 Jul 2025).

Appendix results on COCO and DCI follow the same trend. On COCO, SD3.5 W8A8 improves from PTQ4DiT FID $\alpha$7 to SegQuant-A FID $\alpha$8. On DCI, SD3.5 W8A8 improves from PTQ4DiT FID $\alpha$9 to SegQuant-G FID $0.0$0. On COCO, FLUX W8A8 improves from Q-Diffusion FID $0.0$1 to SegQuant-G FID $0.0$2. On DCI, SDXL W8A8 improves from PTQ4DiT FID $0.0$3 to SegQuant-G FID $0.0$4. The paper therefore uses three separate datasets to support the claim that improvements are not confined to a single benchmark (Zhang et al., 20 Jul 2025).

The paper also includes a deployment-oriented motivation example from industry collaboration: applying FP8 quantization to DiT-ControlNet models, where latent layers dominate computation, reportedly achieved $0.0$5 speedup with negligible quality loss, specifically $0.0$6 and $0.0$7. This example is not part of the main SegQuant benchmark, but it contextualizes the framework’s emphasis on deployable low-precision inference (Zhang et al., 20 Jul 2025).

6. Scope, limitations, and terminological boundaries

SegQuant is presented as broadly applicable, but its gains depend on structural assumptions. SegLinear is most useful when a model’s computation graph exposes semantically meaningful operator patterns such as split, chunk, stack, or concat. DualScale is most relevant when activations are polarity-asymmetric, particularly on activation-to-linear paths after SiLU, GELU, or GEGLU. The paper also notes that segmentation and dual-scale processing add only a modest runtime increase, but not zero overhead; in some W4A8 cases, gains over a strong baseline such as SVDQuant are favorable yet modest rather than dramatic (Zhang et al., 20 Jul 2025).

The framework’s graph component is intentionally limited. It does not define a graph neural network, a spectral partitioning objective, or a learned graph loss. Likewise, the paper does not provide a backend-specific deployment benchmark for TensorRT-like systems, nor a fully formal graph optimization theory. These omissions do not negate the reported results, but they mark the current formulation as a practical PTQ framework rather than a general theory of graph-structured quantization.

The name should also be distinguished from other segmentation-aware uses of quantization in recent literature. In speech representation learning, “Segmentation-Variant Codebooks” describes a multi-codebook discretization scheme over frame, phone, word, and utterance levels rather than a diffusion PTQ framework (Sanders et al., 21 May 2025). In quantum neural networks, “segmentation-based regression” refers to digit-lattice inference for continuous outputs rather than semantics-aware quantization of diffusion layers (Hateley, 27 Jun 2025). Within the diffusion-model literature, however, “SegQuant” refers specifically to the semantics-aware PTQ framework built from SegLinear and DualScale (Zhang et al., 20 Jul 2025).