Fast Concept Activation Vectors (FastCAV)

• FastCAV is a closed-form, computationally efficient method for extracting concept activation vectors that approximates SVM-based techniques using direct activation statistics.
• It replaces iterative SVM training with simple mean-centering and normalization operations, achieving O(nd) complexity and significant speedups.
• Empirical results across diverse architectures show up to 63.6× faster computation while maintaining high accuracy and stable concept-based explanations.

Fast Concept Activation Vectors (FastCAV) are a class of closed-form, computationally efficient approximations for extracting Concept Activation Vectors (CAVs), providing an alternative to conventional linear SVM-based CAV computation in deep neural network interpretability. FastCAV maintains theoretical and empirical consistency with SVM-based CAVs under conditions of within-class isotropy and mean separation, while dramatically reducing computation time and improving the stability of concept-based explanations. This method enables scalable analysis of concepts in large-scale, high-dimensional network architectures and broadens the feasibility of concept-based interpretability across domains (Schmalwasser et al., 23 May 2025).

1. Principle of Concept Activation Vectors and SVM-based CAVs

Given a trained neural network $f$ and an internal layer $l$, the associated activation mapping $g_l(x) \in \mathbb{R}^d$ for input $x$ enables probing for abstract, human-understandable concepts $c$ by collecting a set $D_c$ of positive (concept-bearing) examples and a “random” negative set $D_r$. The standard practice defines the CAV as the normal vector $w$ of a linear SVM trained to distinguish activations from $D_c$ and $D_r$:

$$w, b = \arg\min_{w, b} \frac{1}{2}\|w\|^2 + C \sum_{(x, y)} \max(0, 1 - y(w^T g_l(x) + b))$$

with $y=+1$ for $x\in D_c$ and $y=-1$ for $x\in D_r$. The resulting SVM normal $w$ is termed the SVM-CAV $v_c^l$. Concept-related analyses typically use the direction $v_c^l / \|v_c^l\|$.

2. FastCAV: Closed-Form Computation

FastCAV eliminates the need for iterative SVM solving, replacing it with direct statistics on activation means. The main procedure is:

1. Compute the global mean over $D_c \cup D_r$:

$$\hat\mu_{\mathrm{global}} = \frac{1}{|D_c|+|D_r|} \sum_{x \in D_c \cup D_r} g_l(x)$$

2. Center each concept activation by this mean and compute the average:

$$\tilde{v} = \frac{1}{|D_c|} \sum_{x \in D_c} [g_l(x) - \hat\mu_{\mathrm{global}}]$$

1. Normalize: $v_c^l = \tilde{v}/\|\tilde{v}\|_2$
2. Set intercept: $b = -{v_c^l}^T \hat\mu_{\mathrm{global}}$

This approach requires only $O(n d)$ computation for $n$ samples and $d$-dimensional activations, in contrast to SVM, which scales substantially worse—$O(\max(n, d)\min(n, d)^2)$ in the worst case or $O(T n d)$ for $T$ SGD steps.

Step	Operation	Complexity
Compute mean	Sum over $n$ activations	$O(n d)$
Center/average	Subtract mean, average $D_c$	$O(n d)$
Normalize/Intercept	Vector norm, scalar product	$O(d)$

3. Theoretical Foundation: Equivalence with SVM

FastCAV’s equivalence to SVM-based CAVs holds under i.i.d. Gaussian input assumptions with isotropic within-class covariance ($\Sigma_c = \Sigma_r = \sigma^2 I$) and balanced concept/random cardinality. Under these, $$\mathbb{E}[\hat\mu_{\mathrm{global}}] = (\mu_c + \mu_r)/2$$, and $\mathbb{E}[\tilde{v}] \propto (\mu_c - \mu_r)/2$, paralleling Fisher’s LDA solution. As almost all examples become support vectors when $d \gg n$, SVM converges to the same direction (modulo scaling) as LDA and FastCAV. Therefore, the expected FastCAV direction matches the expected SVM-CAV direction under these constraints.

A direct implication is that, in isotropic regimes, FastCAV is not merely an approximation but a provably optimal estimator of the concept direction in activation space (Schmalwasser et al., 23 May 2025).

4. Empirical Performance and Robustness

FastCAV significantly reduces computation time—speedups of up to $63.6\times$ (average $46.4\times$) compared to linear SVMs—across multiple architectures (e.g., Inception-v3, ResNet50, ConvNeXt-XXL, RegNetY, ViT-B/16, ViT-H/14-CLIP, EVA-02-L/14). For $|D_c| = |D_r| = 60$ and activation dimensions ranging 141k–1.8M, FastCAV computes CAVs in seconds where SVM requires several orders of magnitude greater time. Table 4.1 from (Schmalwasser et al., 23 May 2025) below summarizes:

Model	Dimensionality (k)	FastCAV (s)	SVM-CAV (s)	Acc$_F$	Acc$_S$	Inter Cosine
Inception-v3	206	0.4	44.7	0.95	0.93	0.898
ResNet50	341	1.1	135.4	0.89	0.87	0.837
ViT-B/16	141	1.1	50.5	0.82	0.81	0.818
ViT-H/14-CLIP	282	1.9	69.0	0.87	0.86	0.858
EVA-02-L/14	900	6.1	301.6	0.89	0.88	0.836

Accuracy (FastCAV $\mathrm{Acc}_F$, SVM $\mathrm{Acc}_S$) and inter-method cosine similarity (FastCAV vs SVM) remain consistently high ($> 0.8$), showing comparable downstream separation of concept and random samples. FastCAV also demonstrates greater intra-method stability across draws (approx. $0.8$ vs $0.3$–$0.4$ cosine similarity for SVM-CAVs).

Applications in the medical domain (DenseNet-121 on MIMIC-CXR) demonstrate CAV computation in 0.006 s (FastCAV) versus 0.439 s (SVM), with 0.72 accuracy and 0.79 robustness, placing FastCAV as competitive with prior logistic-regression-based approaches.

5. Role in Downstream Concept-Based Explanation

FastCAV integrates seamlessly into established concept-based interpretability pipelines. In Testing with CAVs (TCAV), empirical replication for class “ladybug” and visual concepts (“polka-dotted,” “striped,” “zigzagged”) finds alignment between FastCAV and SVM-CAV in layer/concept significance, with FastCAV displaying lower variance across repetitions. In Automatic Concept Explanations (ACE), substituting FastCAV for SVM-CAV yields equivalent top-ranked concepts and nearly identical TCAV scores (0.78 vs 0.73 for top “stripes” in “lionfish”), confirming correspondence in both quantitative and qualitative outputs.

This establishes FastCAV as a drop-in replacement for SVM-based CAVs in these workflows, supporting both the reproducibility of explanations and scaling to model configurations previously computationally prohibitive.

6. Limitations and Future Directions

The validity of FastCAV’s approximation is conditioned on activation isotropy and Gaussianity; in 2.8% of CAV constructions, $>$25% accuracy discrepancy is observed, always in favor of the SVM-CAV under highly skewed, non-isotropic data distributions. FastCAV does not account for within-class covariance shape, a situation where the SVM’s optimization confers an advantage.

Outstanding questions include comparative locality (layerwise concept consistency), the behavior under concept entanglement, and suitability for model misbehavior detection. Suggested directions for future enhancement involve augmentation of FastCAV with low-rank covariance correction, adaptation to multi-class concept discovery, and exploitation of random projections for further scalability in regimes with $d \gg n$ (Schmalwasser et al., 23 May 2025).

7. Summary

FastCAV is a mathematically principled, closed-form methodology for CAV extraction in deep networks, providing $O(nd)$ complexity and high-fidelity alignment with SVM-derived CAVs under well-defined conditions. Empirically, FastCAV achieves substantial acceleration (up to $63 \times$), with equivalent or better stability and accuracy, thereby democratizing comprehensive concept-based analysis for modern, large-scale neural architectures (Schmalwasser et al., 23 May 2025).