Validate the functional-redundancy hypothesis through ablation

Determine whether the compensatory capacity observed when ablating dominant attention heads in the DeBERTa-v3-base prompt-injection classifier indicates functional redundancy associated with overparameterization for the classification task.

Background

The ablation experiments show that removing dominant attention heads produces only moderate performance degradation and can occasionally improve classification of competing categories. These results suggest that other components may compensate for the removed heads, but the paper does not establish whether this compensation reflects genuine functional redundancy or is related to the model being overparameterized for prompt-injection detection. The authors explicitly identify testing this hypothesis as future work.

References

We hypothesize that this compensatory capacity indicates functional redundancy within the network, potentially related to the model being overparameterized for the task under consideration. Evaluating this hypothesis constitutes a direction for future work.

— Influence Score and Transformers interpretability: Measure of the Effective Impact of Attention Heads at inference time  (2609.05074 - Bouger et al., 4 Sep 2026) in Section 6.4, “Causal validation through ablation of dominant heads”