Approximate Feature Activation (AFA)

• Approximate Feature Activation (AFA) is a norm-based method for sparse autoencoders that replaces a fixed top-k with an input-driven active feature selection.
• It leverages quasi-orthogonality in the decoder to tightly link the latent activation norm with the input embedding norm, ensuring theoretical robustness.
• The top-AFA rule and auxiliary loss improve reconstruction error and sparsity adaptivity, as shown by lower MSE in transformer embedding experiments.

Approximate Feature Activation (AFA) is a framework for sparse autoencoders (SAEs) that links the norm of a sparse feature vector to the norm of the corresponding dense embedding under a quasi-orthogonal decoder dictionary, and uses that link both to evaluate pre-trained SAEs and to design a top-$k$-style activation rule that does not require a fixed hyperparameter $k$ (Lee et al., 31 Mar 2025). In the formulation introduced in “Evaluating and Designing Sparse Autoencoders by Approximating Quasi-Orthogonality,” AFA addresses a central limitation of $k$-sparse autoencoders: the number of nonzero latent activations, $\ell_0 = k$, is normally chosen ad hoc, is input-agnostic, and lacks theoretical justification. The AFA framework instead seeks to tie the number of active features to properties of each input embedding $z$, and thereby to remove the need to pick a fixed $k$ across all inputs (Lee et al., 31 Mar 2025).

1. Problem setting in sparse autoencoders

In a $k$-sparse autoencoder one enforces that exactly $k$ entries of the latent vector $f$ are nonzero, typically via a top-$k$ activation function (Lee et al., 31 Mar 2025). This design removes the need for an explicit sparsity penalty $k$0, but it replaces that penalty with a new hyperparameter $k$1 whose choice is ad hoc, input-agnostic, and without theoretical justification. The central question posed by AFA is whether the number of active features, $k$2, can be tied to the properties of each input $k$3 rather than fixed globally (Lee et al., 31 Mar 2025).

The paper formulates this question in the setting of SAEs trained on transformer hidden embeddings. In the reported experiments, GPT-2 Small layer 6, 7, and 8 embeddings are used as SAE inputs, with dictionary size $k$4, and training is carried out for $k$5 steps, corresponding to approximately $k$6M tokens from OpenWebText (Lee et al., 31 Mar 2025). Within that setting, AFA functions both as a theoretical lens on latent activation norms and as a practical design principle for activation selection.

A plausible implication is that AFA changes the role of sparsity control. Instead of prescribing a constant cardinality budget in latent space, it treats sparse activation as an input-conditioned quantity determined by a norm-matching target. That interpretation follows directly from the paper’s guiding objective: choose, for each input $k$7, the smallest number of active features so that $k$8 (Lee et al., 31 Mar 2025).

2. Quasi-orthogonality and the Linear-AFA theorem

The theoretical derivation begins from two assumptions (Lee et al., 31 Mar 2025). The first is the Linear Representation Hypothesis (LRH), written as

$k$9

where $k$0 and $k$1. The second is quasi-orthogonality of the decoder dictionary: after normalizing each column $k$2 of $k$3 to unit norm, the Gram matrix satisfies

$k$4

with $k$5’s on the diagonal and off-diagonals bounded by $k$6 (Lee et al., 31 Mar 2025).

Under these assumptions, the key algebraic identity is

$k$7

The cross-terms are then bounded using $k$8 and $k$9, yielding Theorem 1, called “Linear-AFA” in the paper:

$\ell_0 = k$0

and therefore

$\ell_0 = k$1

This gives a closed-form error controlled by $\ell_0 = k$2 (Lee et al., 31 Mar 2025).

The interpretation stated in the paper is that if $\ell_0 = k$3 is small, meaning the dictionary is nearly orthogonal, then the $\ell_0 = k$4-norm of the sparse code $\ell_0 = k$5 is tightly determined by the $\ell_0 = k$6-norm of the dense embedding $\ell_0 = k$7 up to a closed-form error proportional to $\ell_0 = k$8 (Lee et al., 31 Mar 2025). This provides the theoretical basis for replacing a fixed $\ell_0 = k$9 target with a per-input activation rule derived from norm agreement.

3. top-AFA: adaptive activation without a fixed $z$0

The activation function introduced on top of the AFA formulation is called top-AFA. Its stated goal is to choose, for each input $z$1, the smallest number of active features so that $z$2 (Lee et al., 31 Mar 2025). The algorithm operates on a batch of embeddings $z$3 with encoder $z$4, decoder $z$5, and biases $z$6.

The procedure is as follows (Lee et al., 31 Mar 2025). First, the input is centered:

$z$7

Second, pre-activations are computed with ReLU:

$z$8

Third, each feature receives a score

$z$9

where $k$0 is the $k$1-th column of $k$2; in matrix form,

$k$3

For each example, the scores are sorted in descending order to obtain a permutation $k$4, and cumulative sums are formed:

$k$5

with $k$6 and $k$7 set large for numerical stability. The target value is

$k$8

The selected cardinality is then

$k$9

A binary mask $k$0 keeps the top $k$1 positions of $k$2, the sparse code becomes

$k$3

and decoding produces

$k$4

This adaptively chooses $k$5 per input so that $k$6 matches $k$7, in line with the AFA theorem (Lee et al., 31 Mar 2025).

The distinguishing property of top-AFA is therefore not merely that it is dynamic, but that its dynamics are explicitly tied to the norm relation implied by quasi-orthogonality. In that sense, it is a top-$k$8-style mechanism whose effective $k$9 is determined from each example’s activation statistics and centered embedding norm rather than supplied as a constant hyperparameter.

4. Evaluation of pre-trained SAEs under AFA

AFA is also presented as an evaluation methodology for pre-trained SAEs (Lee et al., 31 Mar 2025). The paper argues that existing SAE evaluation methods and loss functions had overlooked the theoretically expected activation magnitude that can be computed from the input embedding. Two diagnostic tools are introduced for this purpose: the ZF plot and the quantity $k$0.

The ZF plot places each input’s dense norm $k$1 on the $k$2-axis and its learned sparse norm $k$3 on the $k$4-axis; perfect alignment is the red line $k$5 (Lee et al., 31 Mar 2025). This visualization directly compares actual latent activations against the norm-matching relation implied by AFA. The associated scalar diagnostic is

$k$6

described as the “lower-bound” estimate of $k$7 required to justify the observed activations (Lee et al., 31 Mar 2025).

By plotting $k$8 across layers or models, one can diagnose over- or under-activation relative to theory (Lee et al., 31 Mar 2025). This makes the AFA perspective evaluative as well as constructive: it does not only prescribe how to design a new activation function, but also supplies a criterion for assessing whether a learned SAE behaves as though its dictionary were sufficiently quasi-orthogonal for the norm approximation to be credible.

This suggests a shift in SAE diagnostics away from reconstruction error alone. The paper’s claim is narrower and more precise: AFA connects embedding norms and feature norms in a data-driven metric that had been missing from SAE evaluation (Lee et al., 31 Mar 2025). Within the paper’s framework, norm mismatch is therefore treated as an interpretable failure mode rather than merely an optimization artifact.

5. Training objective, baselines, and empirical findings

The training loss used for top-AFA SAEs is

$k$9

with $f$0 for Ghost Grads and $f$1 found stable (Lee et al., 31 Mar 2025). The auxiliary norm-matching term explicitly biases learning toward dictionaries that satisfy the AFA relationship. The implementation guidance also specifies that decoder columns should be normalized to unit norm and their lengths absorbed into the encoder side so that $f$2 holds, and that inputs should be centered by the encoder bias before computing $f$3 (Lee et al., 31 Mar 2025).

The reported experiments use OpenWebText, sequences of length $f$4, and approximately $f$5M tokens processed over $f$6k iterations (Lee et al., 31 Mar 2025). Baselines are top-$f$7 SAEs with $f$8 and batch-top-$f$9 SAEs with average $k$0, both trained without the AFA loss (Lee et al., 31 Mar 2025). The evaluation metric is normalized MSE.

The main quantitative result is that, on layers 6–8, top-AFA achieves the lowest MSE across all comparisons, including lower reconstruction error than fixed-$k$1 and batch-top-$k$2 baselines (Lee et al., 31 Mar 2025). The paper gives a representative example on layer 6: the best top-AFA MSE is approximately $k$3, whereas the best top-$k$4 MSE is approximately $k$5 (Lee et al., 31 Mar 2025). Under top-AFA, the sparsity level $k$6 adapts between approximately $k$7 and $k$8 nonzeros out of $k$9, rather than remaining fixed across inputs (Lee et al., 31 Mar 2025).

The computational overhead is reported as

$k$00

per batch, which the paper describes as comparable to other top-$k$01 methods, with wall-clock training time within $k$02–$k$03 of standard top-$k$04 (Lee et al., 31 Mar 2025). The empirical picture presented by the paper is therefore that top-AFA replaces hyperparameter tuning of $k$05 with a modest sorting-based overhead while simultaneously improving normalized MSE and adapting the support size to each input.

6. Practical usage and terminological scope

The implementation guidelines associated with AFA are explicit (Lee et al., 31 Mar 2025). Decoder columns should be normalized to unit norm, per-feature decoder norms $k$06 should be precomputed and used in the scoring step, inputs should be centered by the encoder bias before computing $k$07, and the top-AFA routine should use sorting plus cumulative sums over the $k$08 features per input to produce candidate norms $k$09. The paper further recommends adding the AFA loss term $k$10 with coefficient $k$11, using Ghost Grads or similar to keep features from dying, and monitoring $k$12 on a held-out set through ZF plots to verify quasi-orthogonality and detect over- or under-activation (Lee et al., 31 Mar 2025).

A recurring source of confusion is that the acronym “AFA” is used in several unrelated literatures. In privacy-preserving neural networks, “Batch-oriented Element-wise Approximate Activation” denotes trainable low-degree polynomial approximation of ReLU under FHE and CKKS packing, rather than sparse feature activation in SAEs (Zhang et al., 2024). In explainable AI, “Anytime Approximate Formal Feature Attribution” refers to adaptive AXp/CXp enumeration for approximating formal feature importance, and defines feature attribution as the fraction of abductive explanations containing a given feature (Yu et al., 2023). In sequential decision-making, “Active Feature Acquisition” denotes policies that choose which features to observe under a budget, including a zero-shot variant based on LLM-elicited discriminative MRFs (Perets et al., 17 Jun 2026). These usages are terminologically adjacent but methodologically distinct.

Within mechanistic interpretability, Approximate Feature Activation has a narrower and more specific meaning: it is the norm-based SAE framework derived from quasi-orthogonality, used both to assess whether learned latent activations are theoretically consistent with the input embedding and to construct the top-AFA activation rule that removes the need to tune a constant $k$13 (Lee et al., 31 Mar 2025).