Function Vector: Diverse Mathematical and ML Roles

• Function vector (FV) is a polysemous concept representing functions or tasks as high-dimensional vectors, with definitions varying by context.
• In functional analysis and matrix analysis, FVs extend scalar spectral functions using A-characters, evaluation maps, and block-constant vector-fields.
• In mechanistic interpretability, FVs capture activation-based task representations in language models, influencing in-context learning and continual tuning.

Function vector (FV) is a polysemous technical term rather than a single standardized object. In functional analysis it appears in Banach $A$-valued function algebras, where vector-valued functions, $A$-characters, and vector-valued spectra extend classical Banach function algebra theory; in matrix analysis it denotes spectral matrix-functions generated by vector-fields on ordered eigenvalue tuples; in mechanistic interpretability it denotes activation-space task representations extracted from attention heads or residual streams and reused to steer LLMs; and in randomized representation theory it denotes high-dimensional vectors representing functions in a reproducing kernel Hilbert space (Abtahi, 2015, Carlsson, 2018, Yin et al., 19 Feb 2025, Frady et al., 2021). This suggests a recurring theme: a function, task, or functional action is encoded in vectorial form, but the formal object depends entirely on the surrounding theory.

1. Vector-valued function algebras and $A$-characters

In the Banach-algebraic usage, a commutative unital Banach algebra $A$ and a compact Hausdorff space $X$ define the ambient algebra $C(X,A)$ of continuous $A$-valued functions with uniform norm $\|f\|_X=\sup_{x\in X}\|f(x)\|_A$. An $A$-valued function algebra $\mathcal A\subset C(X,A)$ is required to contain all constant functions and to separate points of $A$0. If $A$1 is admissible, meaning $A$2, then one can define an $A$3-character as a homomorphism $A$4 satisfying $A$5 and

$A$6

The evaluation maps $A$7 are the model examples, and from the defining property together with semisimplicity of $A$8 one obtains $A$9 for all constants $A$0 (Abtahi, 2015).

This formalism generalizes ordinary scalar characters. When $A$1, $A$2-valued function algebras reduce to Banach function algebras and $A$3-characters reduce to the usual character space $A$4. A central structural result is that, for natural $A$5-valued algebras such as $A$6, $A$7, $A$8, $A$9, $A$0, and tensor-product constructions of the form $A$1, the only $A$2-characters are the point evaluations $A$3. The same framework identifies the $A$4-valued spectrum through

$A$5

so in natural cases the vector-valued spectrum is determined by evaluation. The paper also gives a non-admissible $A$6-valued example to show that admissibility is essential for the $A$7-character machinery and the associated spectral description (Abtahi, 2015).

2. Spectral matrix-functions and multivector function calculus

A different mathematical usage arises for Hermitian matrix-functions based on vector-fields. If $A$8 has spectral decomposition $A$9, with $X$0 the ordered eigenvalue vector, a block-constant vector-field $X$1 defines

$X$2

This strictly generalizes the scalar spectral calculus $X$3, since the component $X$4 may depend on all coordinates $X$5, not only on $X$6. The well-definedness condition is block-constancy: whenever $X$7, one requires $X$8. Under $X$9 point-symmetry, the paper proves a generalized Daleskii–Krein formula,

$C(X,A)$0

and a Frobenius-norm Lipschitz bound

$C(X,A)$1

Classical scalar spectral functions are recovered by the special choice $C(X,A)$2 (Carlsson, 2018).

A broader algebraic extension appears in Clifford analysis, where elementary functions are extended from scalars, complex numbers, and quaternions to multivectors in $C(X,A)$3 and $C(X,A)$4. The paper defines exponentials, logarithms, powers, trigonometric functions, and hyperbolic functions of multivector variables by power series and polar decomposition. It gives the general power formula $C(X,A)$5, the amplitude $C(X,A)$6, the inverse $C(X,A)$7, and a unified square-root expression $C(X,A)$8. One notable result is that a complex number raised to a vector power produces a quaternion: $C(X,A)$9 Comparing dimensions, the paper identifies $A$0 as a particularly versatile algebraic framework because the pseudoscalar $A$1 commutes with all elements (Chappell et al., 2014).

3. LLM function vectors: operational definitions

In LLMs, the modern mechanistic-interpretability usage treats a function vector as a task representation elicited during in-context learning. One head-level formulation defines, for task $A$2 and attention head $A$3, the mean task-conditioned activation

$A$4

and calls $A$5 an FV head when patching $A$6 into corrupted prompts restores task-appropriate behavior. The associated FV score on a corrupted prompt $A$7 is

$A$8

At task level, some experiments aggregate over the top-2% FV heads to form

$A$9

which is then added to the residual stream to test whether it reproduces the task without informative demonstrations (Yin et al., 19 Feb 2025).

A residual-stream formulation defines one FV per task, template, and layer as a mean-difference direction between positive and negative in-context learning conditions: $\|f\|_X=\sup_{x\in X}\|f(x)\|_A$0 with steering implemented by

$\|f\|_X=\sup_{x\in X}\|f(x)\|_A$1

Here the FV is extracted from the final-token residual stream, usually from 15 positive and 15 negative prompts, and can be transferred across templates for the same task (Nadaf, 3 Apr 2026).

A prompt-specific formulation defines the FV directly from the outputs of previously localized FV heads: $\|f\|_X=\sup_{x\in X}\|f(x)\|_A$2 where $\|f\|_X=\sup_{x\in X}\|f(x)\|_A$3 is an $\|f\|_X=\sup_{x\in X}\|f(x)\|_A$4-shot prompt and $\|f\|_X=\sup_{x\in X}\|f(x)\|_A$5 is the final separator token. In that setting, FV quality is measured causally by injecting $\|f\|_X=\sup_{x\in X}\|f(x)\|_A$6 into zero-shot prompts and maximizing task accuracy over intervention layer and scale (Wang et al., 15 May 2026).

Taken together, these works do not impose a single canonical definition. Instead, the term denotes a family of causally tested activation-space task representations at different granularities: per-head averages, residual mean-difference directions, and per-prompt sums over FV heads.

4. FV heads and the mechanisms of in-context learning

A central empirical question is whether in-context learning is driven primarily by induction heads or by FV heads. Across 12 decoder-only models, induction heads are defined as the top 2% of heads by induction score, and FV heads as the top 2% by FV score. The overlap is limited: 7 of 12 models have 0 overlapping heads in the top-2% sets, while the remaining models have only 5–15% overlap. Nevertheless, the two sets are correlated, and FV heads tend to occur in slightly deeper layers than induction heads (Yin et al., 19 Feb 2025).

Ablation results sharply distinguish their causal roles. Ablating FV heads drastically hurts few-shot ICL accuracy, whereas ablating induction heads has limited impact, especially when overlap is controlled by excluding heads that rank highly on both criteria. In models above 1B parameters, ablating non-FV induction heads leaves ICL accuracy almost unchanged relative to random ablation, while ablating non-induction FV heads causes large drops; in the 6.9B Pythia model, removal of FV heads is described as catastrophic for ICL, while removal of induction heads with exclusion is near-random. The same study reports that FV heads strongly affect few-shot accuracy but not token-loss difference, while induction heads show the reverse pattern (Yin et al., 19 Feb 2025).

Training dynamics further separate the two mechanisms. In Pythia checkpoints, induction scores rise sharply around step 1,000 of 143,000 total steps and then plateau or slightly decline, whereas FV scores emerge later, around step 16,000, and continue increasing through training. Many final FV heads begin with high induction scores and later transition toward the FV mechanism; the reverse transition is not observed. The paper interprets this as a unidirectional developmental pattern in which induction serves as a precursor to function-vector-based in-context learning (Yin et al., 19 Feb 2025).

5. Steering, transfer, and compositional structure

Large-scale steering studies refine the claim that FVs are task codes. In a cross-template setting comprising 4,032 source-target pairs across 12 tasks, 6 models, and 8 templates per task, FV steering succeeds even when the logit lens cannot decode the correct answer at any layer. Steering exceeds logit-lens accuracy for every task on every model, with gaps as large as $\|f\|_X=\sup_{x\in X}\|f(x)\|_A$7; only 3 of 72 task-model instances show the opposite decodable-without-steerable pattern, all in Mistral. FVs that achieve over 0.90 steering accuracy still project to incoherent token distributions under vocabulary projection, optimal interventions occur at early layers $\|f\|_X=\sup_{x\in X}\|f(x)\|_A$8–$\|f\|_X=\sup_{x\in X}\|f(x)\|_A$9, and logit-lens decodability peaks only at late layers such as $A$0–$A$1. The same paper finds that the previously reported negative cosine-transfer correlation dissolves at scale: pooled $A$2 ranges from $A$3 to $A$4, and cosine adds less than 0.011 in $A$5 beyond task identity (Nadaf, 3 Apr 2026).

Instruction-elicited FV work studies two additional design choices: head selection and steering location. Replacing Average Indirect Effect head selection with Layer-wise Relevance Propagation improves both speed and accuracy, with measured throughput changes of 4.57 vs 1541.93 samples/min for Llama-3.2-3B, 1.63 vs 1308.41 for Llama-3.1-8B, and 2.96 vs 1167.67 for Qwen3-4B, for an average factor of about 511×. For steering, distributed reinjection—adding each selected head’s averaged representation at its original head and layer location—outperforms simple aggregation, with gains up to 0.156 in accuracy; LRP-selected heads improve steering accuracy over AIE-selected heads by as much as 0.194, and some tasks such as capitalization and translation are reported to be meaningfully reproduced only with LRP-based extraction (Pham et al., 3 Jun 2026).

Few-shot composition studies then ask how multiple demonstrations combine into a single FV. Across Gemma-2, Llama-3.2, and Llama-3.1 models, an $A$6-shot FV is well approximated by a linear combination of example-level sub-FVs obtained by attention-edge restriction. The fitted reconstruction achieves mean cosine similarity at least 0.925 and mean $A$7 at least 0.875, and preserves most of the causal effect under injection. Contextualization does not merely add examples; it reweights them. In normal tasks it mitigates recency bias by redistributing attention more uniformly, while in ambiguous tasks it concentrates attention on unambiguous examples, and a Shapley-style decomposition finds that Query–Key contextualization is the most consistent positive contributor to FV quality, whereas Value-mediated effects are more heterogeneous (Wang et al., 15 May 2026).

These results jointly support an interpretation already stated explicitly in the literature: FVs behave less like answer vectors than like computational instructions. A plausible implication is that the relevant task representation often lies upstream of direct vocabulary-space decodability and is assembled by additive superposition together with context-dependent attention reweighting.

6. Continual learning and adjacent vector-space formalisms

Function vectors have also been used to analyze catastrophic forgetting in continual instruction tuning. In that setting, a task FV is defined as

$A$8

where $A$9 is a global set of causally important heads identified by activation patching and $\mathcal A\subset C(X,A)$0 is the task-conditioned mean activation at the last token. The paper argues, both theoretically and empirically, that catastrophic forgetting in LLMs primarily stems from biases in function activation rather than the overwriting of task processing functions. Adding the original FV back into a fine-tuned model recovers lost performance, subtracting newly learned task FVs mitigates interference, and FV similarity predicts forgetting more strongly than last-layer hidden similarity or parameter $\mathcal A\subset C(X,A)$1 distance. The proposed FV-guided training objective

$\mathcal A\subset C(X,A)$2

with $\mathcal A\subset C(X,A)$3 and $\mathcal A\subset C(X,A)$4, is evaluated on four benchmarks and is reported to improve both zero-shot and in-context performance while preserving task-specific performance (Jiang et al., 16 Feb 2025).

A distinct but conceptually adjacent line of work represents functions themselves as high-dimensional vectors in a Vector Function Architecture. There, an encoding $\mathcal A\subset C(X,A)$5 is chosen so that inner products approximate a kernel,

$\mathcal A\subset C(X,A)$6

and a function $\mathcal A\subset C(X,A)$7 is represented by the vector

$\mathcal A\subset C(X,A)$8

Point evaluation becomes $\mathcal A\subset C(X,A)$9, addition is vector superposition, binding implements shifts, and binding of function vectors implements convolution. With fractional power encoding and uniformly distributed phases, the induced kernel is the sinc kernel, and the resulting RKHS is the space of band-limited functions (Frady et al., 2021).

The two programs are mathematically unrelated in construction, but they converge on a shared idea: a vector can be treated not merely as data, but as a portable representation of a function or task. This suggests why the same phrase has proved attractive across functional analysis, matrix perturbation theory, mechanistic interpretability, continual learning, and randomized function-space computation.