Explainability Vector in Machine Learning

• Explainability vector is a vector-valued explanatory representation that interprets model decisions by encoding multiple explanation types such as latent directions or prototype closeness.
• It encompasses both global methods like concept activation vectors and local techniques such as proximity-to-prototype vectors, enabling nuanced interpretations of complex models.
• The approach faces challenges including stability, representation specificity, and varying evaluation protocols, which underscore the need for context-dependent application.

Searching arXiv for papers explicitly using or closely related to “explainability vector”. An explainability vector is not a single standardized mathematical object across machine learning. Across the cited literature, the term is used explicitly in some papers and reconstructed as the most faithful interpretation in others for several distinct entities: a coordinatewise proximity-to-prototype vector in Siamese embedding space, sparse orthogonal projection indices in intrinsically interpretable neural networks, concept activation vectors in latent spaces, parameter-space task vectors that transfer explanation ability, per-dimension rows of node-level explanation matrices, and explicit attribute-probability embeddings such as voice vectors (Utkin et al., 2019, Yang et al., 2019, Yoshikawa et al., 6 Jul 2025, Shafi et al., 2024, Lee et al., 24 Jun 2025). This suggests that “explainability vector” is best understood as a vector-valued explanatory representation whose coordinates are intended to support interpretation of a model, an embedding, or a decision, rather than as a single canonical construction.

1. Conceptual status of the term

The surveyed literature indicates that explainability should not be treated as a single scalar property. A user-centered study defines explainability as “the perceived quality of an explanation by an individual or user group,” distinguishes global explainability, local explainability, model-inherent interpretability, and post hoc explanation, and argues that one “cannot exchange performance for explainability and vice versa in a continuous fashion” (Herm et al., 2022). In that setting, a vector-like or profile-like view of explainability is more faithful than a total ordering of model classes, because explanation quality depends on explanation type, data complexity, cognitive effort, and user task.

Toolbox-oriented work reinforces this plurality. The Xplique library does not define a single object called an explainability vector, but operationalizes several vector-like explanation outputs: feature-attribution scores over input dimensions, saliency maps, concept activation vectors, TCAV scores, and internal directions used in feature visualization (Fel et al., 2022). The implication is that the phrase may refer to input-space importance vectors, latent-space concept directions, or internal representation targets, depending on the explanatory regime.

A recurring divide runs through the literature. Some works use explainability vectors as local objects tied to one input or one pair of inputs; others use them as global objects tied to a model, a concept, a class prototype, or a dataset-wide routing policy. The same phrase therefore spans attribution, representation, and model-edit interpretations.

2. Latent directions, concepts, and intrinsically interpretable axes

One major family of explainability vectors consists of latent directions whose semantics are meant to be stable at the model level. In the explainable neural network $xNN$, the most natural candidate is the projection index vector $\mathbf{w}_j$, appearing in the constrained additive-index decomposition

$g(\EEE(y|x)) = \mu + \sum\limits_{j=1}^{k} \beta_j \tilde{h}_{j}(\w_j^Tx) + \varepsilon,$

subject to sparsity, smoothness, and orthogonality constraints including $W^T W = I_k$ (Yang et al., 2019). Each $\mathbf{w}_j$ is a sparse, orthogonal, unit-norm direction in input space, and its explanatory content comes jointly from the direction $\mathbf{w}_j$, the scale $\beta_j$, and the ridge function $\tilde h_j$. This is a global, structural notion of explainability vector.

Concept-based explanations instantiate a closely related idea in activation space. For EEG transformers, a Concept Activation Vector is the normal vector $\mathbf{v}_C^l$ of a linear separator between concept and random examples at layer $l$, and its explanatory role is measured through the directional derivative

$\mathbf{w}_j$0

and the TCAV score

$\mathbf{w}_j$1

In this formulation, the explainability vector is explicitly a direction in latent space associated with a human-interpretable concept such as seizure-related EEG events or anatomically defined activity patterns (Gjølbye et al., 2023).

The same directional logic appears in vision. In the ICE framework, non-negative concept activation vectors are the rows of $\mathbf{w}_j$2 in the NMF factorization

$\mathbf{w}_j$3

where $\mathbf{w}_j$4 contains non-negative concept directions and $\mathbf{w}_j$5 contains concept scores. The paper argues that these NCAVs offer a better fidelity-interpretability trade-off than PCA directions or clustering-based centroids (Zhang et al., 2020). In this usage, an explainability vector is not a feature-importance vector over pixels but a learned basis direction in feature-map space.

The reliability of such vectors is itself a statistical question. A theoretical analysis of CAV construction shows that, under an asymptotic regime with $\mathbf{w}_j$6 random negative examples, the variance of the CAV estimator decreases as $\mathbf{w}_j$7, with

$\mathbf{w}_j$8

The same paper emphasizes that TCAV scores need not inherit the same variance law because borderline points can keep TCAV variability at order $\mathbf{w}_j$9 (Wenkmann et al., 28 Sep 2025). A separate study further shows that CAV meaning is conditioned by layer choice, entanglement with other concepts, and spatial dependency, so a concept vector is not automatically a stable semantic axis across layers or probe constructions (Nicolson et al., 2024).

3. Local vectors in embedding and decision space

A second family of explainability vectors is local and decision-specific. In Siamese neural networks, the central object is the coordinatewise proximity-to-prototype vector

$g(\EEE(y|x)) = \mu + \sum\limits_{j=1}^{k} \beta_j \tilde{h}_{j}(\w_j^Tx) + \varepsilon,$0

where $g(\EEE(y|x)) = \mu + \sum\limits_{j=1}^{k} \beta_j \tilde{h}_{j}(\w_j^Tx) + \varepsilon,$1 is the sample embedding and

$g(\EEE(y|x)) = \mu + \sum\limits_{j=1}^{k} \beta_j \tilde{h}_{j}(\w_j^Tx) + \varepsilon,$2

is the class prototype. The explanation is the subset $g(\EEE(y|x)) = \mu + \sum\limits_{j=1}^{k} \beta_j \tilde{h}_{j}(\w_j^Tx) + \varepsilon,$3 of the $g(\EEE(y|x)) = \mu + \sum\limits_{j=1}^{k} \beta_j \tilde{h}_{j}(\w_j^Tx) + \varepsilon,$4 smallest coordinates,

$g(\EEE(y|x)) = \mu + \sum\limits_{j=1}^{k} \beta_j \tilde{h}_{j}(\w_j^Tx) + \varepsilon,$5

which identifies latent dimensions regarded as positive evidence for class similarity. This is notable because importance is assigned to closeness, not deviation, and the selected latent coordinates are then perturbed and decoded back to input space through an embedding-aligned autoencoder (Utkin et al., 2019).

MaskInversion uses explainability maps in a different way: not to score an existing vector, but to optimize one. Given an image $g(\EEE(y|x)) = \mu + \sum\limits_{j=1}^{k} \beta_j \tilde{h}_{j}(\w_j^Tx) + \varepsilon,$6 and mask $g(\EEE(y|x)) = \mu + \sum\limits_{j=1}^{k} \beta_j \tilde{h}_{j}(\w_j^Tx) + \varepsilon,$7, it learns a localized embedding token

$g(\EEE(y|x)) = \mu + \sum\limits_{j=1}^{k} \beta_j \tilde{h}_{j}(\w_j^Tx) + \varepsilon,$8

such that the token’s explainability map aligns with the mask. The optimization target is

$g(\EEE(y|x)) = \mu + \sum\limits_{j=1}^{k} \beta_j \tilde{h}_{j}(\w_j^Tx) + \varepsilon,$9

with the score defined from cosine similarity between the current token and averaged visual tokens. Here the vector is the optimized token itself; explainability enters as the supervision channel that localizes the token in the frozen model’s representation space (Bousselham et al., 2024).

SRD generalizes the local-vector perspective from tokens to convolutional features. Its explanatory primitive is the pointwise feature vector $W^T W = I_k$0, not an individual neuron. Relevance is redistributed through the sharing ratio

$W^T W = I_k$1

which lets the method propagate explanation through vector-valued receptive-field structure rather than scalar channel attributions (Han et al., 2024). In this setting, the explainability vector is the local feature vector whose channel geometry is preserved during decomposition.

A related but feature-level construction appears for distance-based classifiers. After rewriting Gaussian-kernel SVMs and $W^T W = I_k$2-nearest neighbors as latent detection-and-pooling networks, the final explanation is the feature-wise vector

$W^T W = I_k$3

which attributes the decision to input dimensions through weighted positive-negative reference-point comparisons (Bley et al., 5 Aug 2025). This is an explainability vector in the most literal local-attribution sense: one component per input feature.

4. Parameter-space edits, graph explanations, and explicit semantic embeddings

Some recent work moves the explainability vector from representation space to parameter space. In task arithmetic for self-explaining image classifiers, the explainability vector is defined explicitly as

$W^T W = I_k$4

the parameter difference between a source-domain model trained with explanation supervision and the same architecture trained for classification only. Target-domain explainability is then injected through

$W^T W = I_k$5

The paper reports that an ImageNet-derived explainability vector improves explanation quality on nine out of ten target datasets and that single-inference explanation quality is comparable to Kernel SHAP, which requires about 150 perturbations to reach similar IoU@10 (Yoshikawa et al., 6 Jul 2025). In this formulation, explainability is a transferable capability encoded as a direction in weight space.

For graph embeddings, the explainability object is neither a concept direction nor a parameter edit but a node-specific matrix: $W^T W = I_k$6 where $W^T W = I_k$7 is the node embedding and $W^T W = I_k$8 is a vector of human-understandable sense features. The $W^T W = I_k$9-th row $\mathbf{w}_j$0 is the explainability vector for embedding dimension $\mathbf{w}_j$1: it gives the interpretable feature profile of that latent coordinate (Shafi et al., 2024). This is a per-node, per-dimension explanatory representation rather than a single vector for the whole model.

Vo-Ve takes the opposite extreme by making the embedding itself human-readable. The voice vector is

$\mathbf{w}_j$2

a 44-dimensional vector whose coordinates are probabilities of explicit voice attributes such as bright, calm, dark, mature, nasal, soft, and young. Because each dimension has a named semantic interpretation, the vector is intrinsically explainable rather than post hoc explained (Lee et al., 24 Jun 2025). This is the most direct realization of an explainability vector as an interpretable embedding.

5. Profiles, routing vectors, and evaluation regimes

Some papers use vector language to argue against scalar notions of explainability altogether. The user-centered study of model performance and explainability proposes that explanation quality is situational and depends on explanation type, data complexity, user population, and task, supporting a profile-like view rather than a single rank order over algorithms (Herm et al., 2022). A plausible implication is that explainability vectors can represent multidimensional profiles even when no single mathematical vector is defined in the model itself.

Other work makes the vector explicit at the level of routing. In ensembles with explainability guarantees, explainability is globally controlled by a target proportion $\mathbf{w}_j$3, but operationally the model defines a per-observation binary allocation $\mathbf{w}_j$4, inducing a routing vector over the dataset. The continuous ranking

$\mathbf{w}_j$5

orders samples by glass-box allocation desirability, and thresholding it yields the binary explainability assignment for any $\mathbf{w}_j$6 (Pisztora et al., 2023). Here the explainability vector is a routing or allocation vector: it does not explain a prediction’s content, but determines whether a sample receives an intrinsically explainable prediction.

Feature-importance methods in quantum machine learning provide another operational meaning. The paper on QML explainability treats LOO, permutation importance, SHAP, and ALE as feature-indexed explanation outputs, so the natural explainability vectors are global importance vectors and per-instance SHAP vectors over the four Iris features (Power et al., 2024). This reflects a common pattern in XAI: even when no paper uses the exact phrase, vector-valued explanation objects routinely arise as per-feature score vectors.

Evaluation practices also differ sharply by vector type. Distance Explainer, which explains pairwise distances in embedding spaces, evaluates explanations with Faithfulness, Sensitivity/Robustness, and Randomization, emphasizing that vector- or map-valued explanations in similarity spaces should be judged by how well they track the underlying distance computation (Meijer et al., 21 May 2025). The diversity of evaluation protocols is itself evidence that explainability vectors are heterogeneous explanatory objects.

6. Limitations, ambiguities, and recurring misconceptions

A recurring limitation is that an explainability vector is only as stable as the representation and training procedure that define it. In the Siamese-network method, the selection of important coordinates is heuristic, depends strongly on decoder reconstruction fidelity, and uses random perturbations; poor reconstruction yields poor explanations, and the method explains prototype-based class similarity rather than arbitrary pairwise similarity (Utkin et al., 2019). In concept-based work, layer inconsistency, entanglement, and spatial dependence show that the same concept label can correspond to different latent directions with different meanings (Nicolson et al., 2024).

A second recurring limitation is non-universality across domains. The parameter-space explainability vector of task arithmetic transfers well only when source and target domains are sufficiently related; the paper explicitly notes failure or degradation for less-related source-target pairs and states that the analogy may break when source and target explainability vectors are not aligned (Yoshikawa et al., 6 Jul 2025). This suggests that many explainability vectors are representation-specific and task-specific rather than portable semantics.

A third issue is the difference between explainability as content and explainability as policy. A voice vector such as Vo-Ve exposes semantic attributes directly, while an allocation vector in an explainability-guaranteed ensemble only says which observations are handled by a glass-box model (Lee et al., 24 Jun 2025, Pisztora et al., 2023). Conflating these two uses obscures whether a vector explains what the model used or merely whether the model used an interpretable component.

The broader literature further cautions against collapsing explainability into one scalar. The user-centered evidence on algorithm choice and explanation type indicates that explainability is clustered, context-dependent, and shaped by presentation design (Herm et al., 2022). Taken together, these works suggest that the most defensible use of “explainability vector” is not as the name of a single universal formalism, but as a family of vector-valued explanatory devices: latent concept directions, prototype-closeness coordinates, parameter-space edits, routing assignments, feature-importance profiles, and intrinsically interpretable embeddings.