---
title: Function Vectors (FVs)
url: https://www.emergentmind.com/topics/function-vectors-fvs
type: topic
---

# Function Vectors (FVs)

A function vector (FV) is a task-specific, compact representation extracted from the internal activations of a machine learning model—classically a transformer, but also applicable in probabilistic geometry and computer vision. FVs serve distinct roles depending on context: as global image descriptors for visual retrieval (Fisher Vectors), as causally precise function encoders in large language and multimodal models, or, in cosmology, as Fréchet vectors minimizing variance among multipole vectors to summarize angular data on the sphere. Despite their different technical instantiations, all FVs enable compressed, controllable representations of complex, distributed information.

## 1. Mathematical Definition and Extraction Procedures

The mathematical formulation and extraction of FVs differ across fields but share the principle of aggregating local or intermedial features into one global or task-level vector.

**a) Language and Multimodal Models**  
For a given task $t$, FVs are derived by first identifying the key attention heads or components responsible for encoding the task function, typically using causal mediation analysis. Let $a_{\ell j}(p) \in \mathbb{R}^d$ be the activation of head $j$ in layer $\ell$ for prompt $p$; the task-conditional mean is
\[
\bar{a}^t_{\ell j} = \frac{1}{|P_t|} \sum_{p \in P_t} a_{\ell j}(p).
\]
Heads with the highest causal influence, as measured by the Average Indirect Effect (AIE), are selected. The FV is formed as
\[
v_t = \sum_{(\ell,j)\in A} \bar a^t_{\ell j},
\]
where $A$ indexes the top-$K$ heads. At inference, $v_t$ is additively injected after a chosen layer to steer model function [2310.15213, 2601.08169, 2604.02608, 2510.02528].

**b) Computer Vision: Fisher Vectors**  
Given a set of local descriptors $\{x_t\} \subset \mathbb{R}^D$ for an image, a Gaussian Mixture Model (GMM) is fitted; the FV is formed from the concatenated gradients of the log-likelihood with respect to the GMM parameters:
\[
\text{FV} = [u_1, v_1, ..., u_K, v_K] \in \mathbb{R}^{2KD},
\]
where $u_i$ and $v_i$ are the mean- and variance-gradient components for the $i$th Gaussian [1508.02496].

**c) Geometric/Probabilistic Setting: Fréchet Vectors**  
Given a set $\{\mathbf v_i\}$ of unit vectors on $S^2$, the Fréchet vector $\mathbf u_\ell$ minimizes the mean squared geodesic (great-circle) distance:
\[
\Psi_\ell(\mathbf u) = \frac{1}{N} \sum_{i=1}^N \gamma^2(\mathbf u, \mathbf v_i),
\]
where $\gamma(\mathbf u, \mathbf v_i)=\arccos(\mathbf u \cdot \mathbf v_i)$. The FV is defined as
\[
\mathbf u_\ell = \operatorname{argmin}_{\mathbf u \in S^2} \Psi_\ell(\mathbf u).
\]
This yields the "center of mass" on the manifold [2411.08087].

## 2. Causal Role and Interpretability in Model Architectures

**Transformers and State-Space Models**  
In transformers, FVs are localized to a small, causally indispensable subcircuit of attention heads in intermediate/later layers; injecting these vectors recovers targeted task performance, while ablation disrupts it. In state-space models (e.g., Mamba), FVs are also present but may exhibit different or more distributed mechanisms, as in Mamba2 [2510.23006].  

**Vision-Language Models**  
In LMMs, FVs are extracted from cross-attention head activations and can prime the model for relational reasoning tasks, operating analogously to language FVs but in a multimodal embedding [2510.02528].

**Cosmological Data**  
In spherical CMB analysis, FVs as Fréchet vectors provide a dimensionally reduced, rotation-invariant probe of multipole structure, serving as sensitive diagnostics for isotropy and higher-order correlations [2411.08087].

## 3. Empirical Properties and Application Domains

| Domain        | Purpose of FV             | Extraction Method              |
|---------------|--------------------------|-------------------------------|
| LLMs/LMMs     | Steer in-context learning| Causal mediation, AIE, LRP    |
| Vision        | Global image descriptor  | Log-likelihood gradients (GMM)|
| CMB/Geometry  | Center-of-mass direction | Min. Fréchet variance         |

**Language Models:**  
- FVs provide high-precision, low-dimensional steering (zero-shot and few-shot ICL), with robust performance on functional/relational tasks, and support analogical reasoning via vector arithmetic [2310.15213, 2601.08169, 2510.02528].
- Fine-tuning of FVs with small datasets further sharpens this effect [2601.08169], while composite FVs enable flexible analogy mapping.

**Computer Vision:**  
- Fisher Vectors remain highly competitive as global descriptors for image instance retrieval, particularly for tasks requiring geometric invariance (rotation/scale). Performance is affected by choice of sampling strategy (sparse/dense, single/multi-scale) and normalization [1508.02496].

**Cosmology:**  
- FVs (Fréchet vectors) are effective at blind anomaly localization (e.g., CMB Cold Spot) and serve as more sensitive probes than multipole vectors for departures from Gaussianity and isotropy, detecting $>5\sigma$ deviations in Planck data unless noise/foreground modeling is extremely precise [2411.08087].

## 4. Invariance, Transferability, and Limitations

- In LLMs, FVs are not invariant across formats: for the same underlying task, FVs extracted from open-ended versus multiple-choice prompts are nearly orthogonal, indicating encoding of both task and format. This limits their generalization out-of-distribution. By contrast, concept vectors (CVs), selected via representational similarity analysis for format invariance, generalize better but are less causally potent in-distribution [2602.22424, 2503.03666].
- FVs in machine translation tasks show partial language-agnosticity: FVs extracted in English→X transfer to unseen target languages and instruction-tuned variants, with strong causal effects on token rankings—yet perform less effectively at the sentence level than on words [2604.19678].
- Steering by FVs can operate even when no "answer direction" is decodable in the unembedding; this dissociation highlights that FVs are computational instructions, not simple output pointers [2604.02608].
- In computer vision, variants of FVs differ in their invariance properties and computational trade-offs; careful normalization and pooling augment geometric robustness [1508.02496].

## 5. Algorithmic Procedures: Pseudocode and Implementation

**LLMs (summarized algorithm):**
1. For each candidate attention head, compute the task-conditioned mean activation over demonstrations.
2. For each head, measure AIE via causal patching on corrupted prompts.
3. Select the top $K$ heads by AIE.
4. Sum their means to form $v_t$ (the FV).
5. At inference, inject $v_t$ into the chosen layer's residual stream.

**Vision (Fisher Vectors):**
1. Extract local descriptors (e.g., SIFT, post-PCA).
2. Fit a GMM to the descriptors.
3. Compute mean/variance gradients for each mixture; concatenate.
4. Apply power and $\ell_2$ normalizations to the resulting global vector.

**Cosmology (Fréchet vectors):**
1. Given $\{a_{\ell m}\}$, compute multipole vectors for each $\ell$.
2. For each $\ell$, minimize $\Psi_\ell(\mathbf u)$ to find $\mathbf u_\ell$, using grid-search and local descent on $S^2$.
3. Use FVs for further analysis (e.g., uniformity tests, anomaly localization).

## 6. Notable Applications and Impact

- **Language Models:** FV injection enables explicit, swap-in control over downstream reasoning and zero-shot task transfer, with interpretability for relational reasoning and analogies [2310.15213, 2601.08169, 2503.03666].
- **Vision:** FVs (Fisher Vectors) remain foundational for scalable content-based image retrieval and hybrid fusion with deep features [1508.02496].
- **Cosmic Microwave Background:** FVs facilitate blind detection and localization of non-Gaussian features—improving sensitivity over classical multipole statistics and enhancing robustness in cosmological hypothesis testing [2411.08087].

## 7. Limitations, Caveats, and Future Directions

- **LLMs:** FV extraction requires extensive ablation runs or gradient-based relevance computations (e.g., LRP). Current FV approaches may not capture high-level behaviors or broad stylistic attributes; they are best suited for well-localized, task-specific interventions [2606.05079, 2411.07213].
- **Vision:** FV dimensionality can be prohibitive for large-scale databases, but product quantization and early fusion with CNN features provide practical mitigation [1508.02496].
- **Cosmology:** FV analyses are limited by anisotropic noise and foreground modeling; residual systematic uncertainties at high $\ell$ remain challenging [2411.08087].
- **Methodological:** The in-distribution effectiveness and out-of-distribution limitations of FVs call for further development of format-invariant and transferable representations (e.g., hybrid FV/CV approaches), as well as more principled head selection and distributed steering protocols [2606.05079, 2602.22424].
- **Theoretical:** In deep transformers, the layered updating and concatenation of function vectors enable adaptive inference not possible in shallow models, expanding the class of learnable in-context algorithms [2606.16694].

FVs thus provide a powerful, interpretable, and modular mechanism for representing and manipulating structured information across major paradigms in statistical learning, computer vision, and cosmological data analysis.

Source: https://www.emergentmind.com/topics/function-vectors-fvs