FIAS: Full Information Activation Supervision

• FIAS is a calibration-time, layer-wise supervision strategy that uses full pre-trained activations to guide staged mixed-precision quantization and mitigate cumulative errors.
• It stabilizes Hessian-based salience estimation by preventing the propagation of distorted activations from earlier quantized layers.
• FIAS complements PBAR in the Squeeze10-LLM framework, contributing to notable accuracy improvements in sub-2bit weight-only quantization of LLMs.

Full Information Activation Supervision (FIAS) is a calibration-time, layer-wise supervision strategy introduced in "Squeeze10-LLM: Squeezing LLMs' Weights by 10 Times via a Staged Mixed-Precision Quantization Method" (Zhu et al., 24 Jul 2025). In that formulation, FIAS preserves original pretrained activation information as the supervisory signal throughout the entire staged quantization process, rather than allowing each layer to inherit already-shifted activations from previously quantized layers. The method is motivated by the observation that, in ultra-low-bit post-training quantization (PTQ), quantization-induced biases accumulate progressively throughout forward propagation, so later layers can be quantized using distorted activation statistics. Within the broader literature, the exact phrase “FIAS” is specific to Squeeze10-LLM, but closely related work has used activation supervision in other senses, including activation range supervision for CNN fault tolerance (Geissler et al., 2021) and full-time supervision over all recurrent time steps in bidirectional RNN question answering (Xu et al., 2016).

1. Definition and conceptual scope

FIAS was introduced as one of the two key innovations in Squeeze10-LLM, alongside Post-Binarization Activation Robustness (PBAR), for sub-2bit weight-only quantization of LLaMA and LLaMA2 models (Zhu et al., 24 Jul 2025). Its core claim is that preserving original pretrained activations throughout staged PTQ mitigates cumulative error propagation across layers. The paper states the motivation directly: “As model weights undergo quantization, the amount of preserved information in LLMs diminishes. From the very first layer, quantization-induced biases accumulate progressively throughout forward propagation, exacerbating distortions in subsequent layers. These shifted activations fail to provide reliable supervision for weight quantization and may even mislead the process, a problem that becomes particularly pronounced under extremely low-bit quantization.”

In this framework, “full activation information” refers to the original pretrained activations of the full-precision model for the same calibration input. The term does not denote a new learned representation or a compressed summary. Instead, it denotes the entire original activation tensor or statistics from the FP model, reused as a stable reference while quantizing each layer. A common misunderstanding is to treat FIAS as a new loss or a separate fine-tuning stage; the paper explicitly frames it as a strategy that uses the same calculation equations as ordinary PTQ while changing the source of the activations that guide the procedure.

2. Layer-wise mechanism in staged mixed-precision PTQ

FIAS operates inside a staged mixed-precision PTQ pipeline. In ordinary layer-wise PTQ, a layer is quantized, its quantized output becomes the input to the next layer, and the resulting shifted activations are then used as the supervision signal or calibration input for quantizing that next layer. The paper argues that this common practice is harmful under aggressive compression because later layers are no longer quantized using the original model’s activation statistics. FIAS replaces that drifting supervisory signal with cached original activations from the pretrained model (Zhu et al., 24 Jul 2025).

The procedure is described as calibration-data based and layer-wise. Calibration inputs are first passed through the original pretrained model, and the resulting activations are cached. For each layer during PTQ, those cached original activations are then used as the supervision signal. Hessian-based and salience metrics are computed from those activations, staged quantization decisions are applied, and the process continues to the next layer without replacing the cached activations with those produced by the quantized layer. The same quantization equations are therefore retained, but the activation values are preserved.

This design is intended to prevent the quantization process from “chasing” already-corrupted activations. A plausible implication is that FIAS stabilizes not only calibration statistics but also the ordering of subsequent quantization decisions, because later layers are supervised by the same full-information reference rather than by a progressively perturbed forward pass.

3. Mathematical role of activations and Hessian-based salience

FIAS does not introduce a dedicated optimization objective, constraint, or gradient-based update rule. The paper states that, for FIAS specifically, there is no new optimization objective such as a separate $\mathcal{L}_{FIAS}$. Its mathematical role is instead implicit in the activation-dependent quantities used by the PTQ procedure (Zhu et al., 24 Jul 2025).

The central activation-dependent term highlighted in the paper is the Hessian estimate

$\mathbf{H} = 2\mathbf{X}\mathbf{X}^T.$

Because $\mathbf{H}$ is derived from the activation matrix $\mathbf{X}$, any quantization-induced shift in $\mathbf{X}$ alters the second-order information used for salience estimation. The paper also gives the SparseGPT-style salience term

$\mathbf{v}_{ij} = \frac{\mathbf{w}_{ij}^{2}{[\mathbf{H}^{-1}]_{ii}^{2},$

noting that the typesetting is imperfect but that the intended elementwise Hessian-based importance score is clear from context. Under FIAS, $\mathbf{X}$ remains the original activation from the pretrained model rather than the drifting activation produced by the quantized network, so the Hessian estimate is stabilized and the salience ranking is less distorted by downstream quantization noise.

The paper explicitly states that “activations inherently indicate weight salience” and therefore “serve as an implicit supervisory signal for the quantization process.” FIAS can thus be understood as preserving the statistical object from which salience is inferred, rather than modifying the salience equations themselves.

4. Relation to PBAR and the staged quantization framework

FIAS is complementary to PBAR rather than a replacement for it. PBAR is described as a refined weight significance metric that accounts for the impact of quantization on activations, while FIAS preserves full activation information during quantization to mitigate cumulative error propagation across layers (Zhu et al., 24 Jul 2025). The broader salience machinery is summarized by

$\mathbf{M}=\mathbf{V}+\lambda\mathbf{B}.$

In that decomposition, PBAR improves what is selected as salient, whereas FIAS improves with what activations the salience is computed and the quantization is guided.

The full Squeeze10-LLM pipeline has three stages: 4-bit uniform quantization as an intermediate buffer, PBAR-based salience selection to decide which weights remain at higher precision, and FIAS-based mixed-bit supervision to guide the layer-wise quantization. The method achieves an average of 1.6 bits per weight by quantizing 80% of the weights to 1 bit and 20% to 4 bits. The paper further notes that 4-bit is used as the intermediate high-precision level because distributional analysis showed that 4-bit activations most closely resemble the full-precision distribution.

The conceptual interaction between PBAR and FIAS is especially important because PBAR relies on Hessian information and Hessians depend on activations. If the activation source is already shifted by previous quantization, then PBAR’s salience estimate itself becomes less trustworthy. FIAS addresses precisely that dependency by fixing the activation source to the original FP activations.

5. Empirical evidence, ablations, and implementation assumptions

The reported empirical evidence places FIAS as a material contributor to the performance of Squeeze10-LLM. On LLaMA and LLaMA2, the full framework is reported to improve average accuracy from 43% to 56% on six zero-shot classification tasks for sub-2bit weight-only quantization (Zhu et al., 24 Jul 2025). The direct ablation isolating FIAS gives the following WinoGrande accuracies: Ours, 64.56%; -PBAR, 64.33%; -FIAS, 62.19%; -PBAR-FIAS, 61.25%; -PBAR-FIAS-Staged Quantization, 56.43%. The paper interprets this by stating that replacing FIAS leads to a 0.01 increase in perplexity on WikiText2 and a 2.37% decrease in accuracy on WinoGrande, while removing both PBAR and FIAS worsens results further.

The implementation assumptions are narrow and explicit. FIAS assumes that original FP activations are available during calibration and representative enough to supervise the PTQ process. It is designed for a layer-by-layer staged PTQ pipeline and is not described as a post-hoc correction after simultaneous quantization of all layers. The paper does not provide a theoretical guarantee, a formal convergence proof, or a dedicated FIAS hyperparameter. The only explicit tunable parameter tied to the broader salience framework is $\lambda$ in PBAR, with the appendix reporting that for LLaMA2-7B the best $\lambda$ is $\mathbf{H} = 2\mathbf{X}\mathbf{X}^T.$0 and that this value is used for other models.

A further practical consequence is a memory and implementation tradeoff: preserving original activations requires the ability to cache or recompute them during calibration. This suggests that FIAS exchanges some calibration-time storage or recomputation cost for a more stable supervisory signal in ultra-low-bit PTQ.

6. Related activation-supervision paradigms and terminological boundaries

Although the formal term “FIAS” belongs to Squeeze10-LLM, related work shows that supervising internal activations is a broader design pattern. In "Towards a Safety Case for Hardware Fault Tolerance in Convolutional Neural Networks Using Activation Range Supervision" (Geissler et al., 2021), the exact phrase “FIAS” is not used, but the paper strongly aligns with the broader idea of using internal activation state as supervisory information. There, intermediate activations are monitored against precomputed bounds, and out-of-bound events are used for fault detection and mitigation. The paper explicitly frames activation range supervision as contributing to fault detection, fault mitigation, and reduction of safety severity in a use case, and explores mitigation operators such as Ranger, Clipper, FmapRescale, Backflip, and FmapAvg. This suggests a distinct but related conception of activation supervision: using layerwise activation intervals and out-of-bound semantics to support safety arguments under hardware soft errors.

A second related formulation appears in "Full-Time Supervision based Bidirectional RNN for Factoid Question Answering" (Xu et al., 2016). That paper does not use the term FIAS, but proposes “full-time supervision,” in which every time-step output of a bidirectional GRU question encoder is directly supervised instead of supervising only a pooled question representation. The training objective is

$\mathbf{H} = 2\mathbf{X}\mathbf{X}^T.$1

contrasted with a pooling-based loss applied only after temporal aggregation. In operational terms, this is supervision over the full sequence of activations rather than over a summary vector, which is close in spirit to the idea of preserving and exploiting complete internal activation information.

FIAS should also be distinguished from similarly named acronyms that denote unrelated frameworks. "Active Class Selection for Few-Shot Class-Incremental Learning" introduces FIASco, meaning Few-shot Incremental Active class SeleCtiOn, a robotics framework combining FSCIL and ACS rather than an activation-supervision method (McClurg et al., 2023). The similarity is terminological, not methodological.

Taken together, these works indicate that “full information” activation supervision can denote different uses of internal neural state: preserving original full-precision activations to guide quantization, supervising every temporal activation rather than a pooled summary, or monitoring activation ranges to detect and mitigate hardware-induced corruption. The specific technical meaning of FIAS, however, remains the one defined in Squeeze10-LLM: preserving original pretrained activations as the supervisory signal throughout staged mixed-precision PTQ.