Papers
Topics
Authors
Recent
Search
2000 character limit reached

Probing for Representation Manifolds in Superposition

Published 18 May 2026 in cs.LG, cs.AI, and stat.ML | (2605.18537v1)

Abstract: This paper introduces the Manifold Probe, a supervised method for discovering representation manifolds in superposition. The method generalizes linear regression probes by learning the space of features of a concept that can be linearly predicted from the representations, and then learning the directions used to encode them. We demonstrate the probe on representations of time and space in Llama 2-7b, finding manifolds which linearly represent an interpretable set of features in each case. In the case of time, we show that by steering along the manifold, we can influence the model's completions about the years in which famous songs, movies and books were released, providing evidence that the Manifold Probe can discover manifolds which are causally involved in model behaviour.

Authors (1)

Summary

  • The paper introduces the Manifold Probe to extract multidimensional representation manifolds from neural activations using a closed-form eigenvalue procedure.
  • It demonstrates the probe’s superior performance over scalar probes in capturing temporal and spatial features in models like Llama 2-7b.
  • The study reveals how manifold steering can causally influence model outputs while preserving task coherence under various intervention scales.

Probing for Representation Manifolds in Superposition

Methodological Innovations

The paper "Probing for Representation Manifolds in Superposition" (2605.18537) introduces the Manifold Probe, a supervised method designed to discover multidimensional representation manifolds embedded via superposition in neural network activations. This framework generalizes the linear probe paradigm, shifting from the classical target-feature approach to simultaneous learning of a feature space and its linear mapping in the representational basis. The probe operates in two stages: first, it optimizes over feature mappings to find those best predicted linearly from the representations; then, it learns the embedding directions via linear regression. The closed-form solution is obtained via a sequential generalized eigenvalue procedure, facilitating scalable and efficient search for dd-dimensional manifold features.

Key contributions include:

  • Formalization of the statistical manifold-probing objective rooted in linear representation and superposition theories.
  • Sample-based estimation with explicit regularization, supporting data-driven complexity control.
  • Efficient optimization via alternating least squares and closed-form regularization parameter selection (REML/GCV).
  • Support for quadratic and non-quadratic feature penalties, broad basis parametrization, and extensibility to high-dimensional embeddings.

Empirical Analysis: Time and Space Manifolds in Llama 2-7b

The empirical investigation harnesses probing datasets curated in prior work, focusing on time (release dates of art/media) and geographic space (US place coordinates) as continuous semantic concepts. Residual stream activations from Llama 2-7b are analyzed layerwise using the Manifold Probe. The probe parametrizes features with cubic B-splines (time) and tensor-product splines (space), with rigorous cross-validation and REML-driven regularization.

The probe discovers a nuanced representational geometry for both concepts. In the case of time, multiple features are found with test R2R^2 exceeding those from direct linear regression on release dates, indicating that the internal manifold encodes temporal distinctions with greater fidelity than naive concept extraction. For space, the probe achieves high R2R^2 for multiple spatial features, outperforming baseline latitude/longitude probes. Figure 1

Figure 1

Figure 1

Figure 1

Figure 1: Representation manifold for time (release dates) in Llama 2-7b, with visualization of fitted features and linear predictions; test R2R^2 coefficients illustrate manifold fidelity.

Figure 2

Figure 2

Figure 2

Figure 2

Figure 2

Figure 2

Figure 2: Representation manifold for space (geographic coordinates); three fitted spatial features and their predictions are shown, demonstrating topological structure and R2R^2 concurrence.

Numerical results substantiate that the manifold probe extracts more precisely encoded information than simple scalar concept probes. Furthermore, the ranked R2R^2 plot across layers shows that manifold-predictable features increase in depth and number through the mid layers, indicating emergence and consolidation of manifold structure as the network processes input. Figure 3

Figure 3

Figure 3

Figure 3

Figure 3

Figure 3: Ranked test R2R^2 values for manifold-fitted features; lines indicate relative predictability of features and comparison with direct concept regression baselines.

Interpretability and Factor Rotation

Interpretability is enhanced by post-hoc factor analysis, particularly Varimax rotation of discovered features. Sparse rotations yield interpretable geometries: for time, the rotated features linearly separate decades; for space, features localize on individual US states, facilitating semantic alignment directly in the representation space. Figure 4

Figure 4

Figure 4

Figure 4

Figure 4

Figure 4

Figure 4

Figure 4

Figure 4

Figure 4

Figure 4

Figure 4

Figure 4

Figure 4

Figure 4

Figure 4

Figure 4

Figure 4

Figure 4

Figure 4

Figure 4

Figure 4

Figure 4

Figure 4

Figure 4

Figure 4

Figure 4

Figure 4

Figure 4

Figure 4

Figure 4

Figure 4

Figure 4

Figure 4

Figure 4

Figure 4

Figure 4

Figure 4

Figure 4

Figure 4: Varimax-rotated manifold features; time features separate decades, space features concentrate on states, evidencing interpretable linear separability.

Causal Steering via Manifold Intervention

The paper advances mechanistic causal analysis by steering the model's internal representation via manifold vectors. By adding a scaled manifold vector corresponding to a desired temporal value at different layers, completion probabilities of the model for that year are directly influenced. The intervention efficacy is quantified via the probability of model completion within two years of the steering target, grouped over layers and decades, evidencing causal use of discovered manifold directions. Figure 5

Figure 5

Figure 5: Efficacy of steering interventions—mean probability a completion falls within two years of the target, stratified by release and target decades. Colour intensity in intervened runs evidences successful steering along the manifold.

Interventions minimally degrade the model's ability to generate valid completions except when steering to extreme values (1950s), demonstrating that manifold interventions retain task coherence. Figure 6

Figure 6: Intervention effect on probability of valid prompt completion; exception observed for the 1950s steering targets.

Variability across steering targets further substantiates the manifold's causal role. Figure 7

Figure 7: Standard deviation in completion probability following intervention, quantifying robustness and selectivity of manifold steering.

Implications and Future Directions

The Manifold Probe marks a substantial advance in dissecting and manipulating representational superposition for continuous, multidimensional concepts. Practically, this methodology enables extraction of rich semantic topologies—e.g., state separability, decade separation—beyond classical binary or scalar concept paradigms. Theoretically, it strengthens the empirical foundation for the linear representation and superposition hypotheses, showing that internal feature manifolds are both accessible and causally involved in model behaviour.

Potential applications are extensive: discovering computational mechanisms in LLMs (e.g., counting, modular arithmetic), mapping out abstract representations (emotion, phylogeny), and informing mechanistic interpretability tools for alignment and safety. The main limitation is statistical, requiring large probing datasets to avoid sample degeneracy in high-dimensional residual streams. Dimensionality reduction (e.g., PCA) may ameliorate sample requirements for larger models.

The methodology also raises dual-use implications. Mechanistic interpretability and causal steering tools such as the Manifold Probe could facilitate alignment-driven control of models, but equally, may enable bypassing of safety guardrails or intentional misbehaviour. The responsible development and deployment of such tools remain an open challenge.

Conclusion

"Probing for Representation Manifolds in Superposition" delineates a robust framework for manifold discovery and causal analysis of continuous concept representations within state-of-the-art LLMs. The Manifold Probe generalizes linear probing to multidimensional geometries, empirically substantiates superposed manifold structure, and enables causal interventions directly on continuous semantic axes. The method positions itself as a foundational tool for mechanistic interpretability, semantic extraction, and controlled model manipulation, with far-reaching implications in both AI interpretability and robust alignment.

Paper to Video (Beta)

No one has generated a video about this paper yet.

Sign Up to Generate All Videos Subscribe on YouTube

Whiteboard

No one has generated a whiteboard explanation for this paper yet.

Sign Up to Generate

Explain it Like I'm 14

Explaining “Probing for Representation Manifolds in Superposition” in simple terms

What this paper is about (big picture)

The paper introduces a new tool, called the Manifold Probe, to help us see how LLMs “store” ideas like time and location inside their hidden layers. Think of the model’s hidden space like a huge 3D galaxy of numbers. A concept such as “time” isn’t just a single number in that galaxy—it’s more like a twisty path or shape running through it. That shape is called a “manifold.” The paper shows how to find these shapes and check if the model actually uses them when it makes predictions.

What questions the researchers asked

They focused on three simple questions:

  • Can we find the multi‑dimensional “shapes” (manifolds) that represent continuous ideas like time and space inside a LLM?
  • Which parts or “features” of those ideas (for example, decades within time, or directions like north–south in space) can be read out with simple, straight‑line rules?
  • Are those shapes really used by the model? If we move along the shape (like sliding a knob), can we change what the model says?

How they did it (methods in everyday language)

First, a few quick ideas:

  • Superposition: The model often stores several concepts at once by adding them together inside the same space (like mixing several colors of paint). Each concept uses different “directions,” so they can still be separated.
  • Linear readout: A “linear probe” is a simple way to check what’s in the model by drawing a straight line through the numbers to predict something (like the year or the latitude).

What’s new here:

  • Instead of predicting just one label (say, the exact year), the Manifold Probe learns a whole set of “features” of a concept that are easy to read with straight lines. For time, a feature could be “how much is this in the 1990s?” For space, a feature might be “how far east is this?”
  • The probe works in two stages:
    • Stage 1: Learn which features of a concept (like time or place) can be predicted with a simple line from the model’s hidden numbers.
    • Stage 2: Learn which directions in the model’s hidden space are used to encode those features.
  • They also rotate these features to make them easier to understand (a bit like turning a map so streets line up with north–south/east–west), using a technique called Varimax rotation.
  • They tested this on the Llama 2‑7b model, using datasets that link:
    • titles of songs, movies, and books to their release years (time), and
    • names of places in the U.S. to their latitude and longitude (space).
  • They looked at the “residual stream,” which you can think of as the running summary the model keeps updating at each layer.
  • To test causality, they did “steering”: they added a small vector pointing along the time manifold toward a target year (like nudging a slider) to see if the model’s answer moves toward that year.

What they found and why it matters

  • Time has many readable features, not just the exact year:
    • The probe found features that cleanly separate decades (1950s, 1960s, …). Some of these features were even easier to read out than the raw year itself.
    • When they “steered” representations toward a target year, the model became more likely to complete prompts with that year. This worked best in certain middle layers (around layers 8 and 14), showing that the manifold is not just there—it’s actually used by the model.
  • Space also has many readable features:
    • Beyond simple latitude (north–south) and longitude (east–west), the probe discovered features that pick out specific U.S. states. In other words, many states are almost linearly separated in the model’s hidden space.
  • Layer-wise pattern:
    • In both time and space, the number and strength of readable features grew in the earlier half of the network and then leveled off. This tells us when the model is most actively forming these concept shapes.

Why this is important:

  • It shows that continuous concepts like time and space aren’t stored as single numbers but as rich shapes with many readable properties.
  • The ability to steer along these shapes proves they’re causally involved in what the model says, not just random patterns.

What this could lead to (impact and limitations)

  • Impact:
    • Better interpretability: We get a clearer “map” of how concepts live inside models, which can help explain behaviors, spot errors, or design safer systems.
    • Fine control: Steering along manifolds could guide models toward desired behaviors (for example, more accurate date or location reasoning) without retraining.
    • Science connections: Similar manifold structures might help explain how models learn abstract ideas (like emotion) or even scientific structures (like evolutionary trees).
  • Limitations and care:
    • Data needs: This method works best with many examples (thousands to tens of thousands) so the patterns are reliable.
    • Preprocessing: For small datasets, you may need to reduce dimensionality first (e.g., with PCA).
    • Responsible use: Tools that reveal and steer internal representations can help with alignment—but could also be misused to bypass safeguards. The authors highlight this ethical concern.

In short: The Manifold Probe is like a concept “map-maker” and “dial-turner” for LLMs. It finds the shapes that represent ideas like time and place, shows which parts are easiest to read, and proves those shapes matter by nudging the model’s answers in a predictable way.

View Paper Prompt  View All Prompts 

Knowledge Gaps

Knowledge gaps, limitations, and open questions

The following items pinpoint what the paper leaves uncertain or unexplored, framed to guide concrete follow‑up research:

  • Theoretical recovery guarantees: No finite‑sample analysis (consistency, estimation error, or sample complexity) shows when the Manifold Probe recovers the true manifold under the superposition model; results are only population-level or algorithmic.
  • Identifiability beyond rotations: Features f_k(z) and directions u_k are only identified up to rotations; there is no criterion or prior that pins down a canonical basis or quantifies identifiability under different sampling measures over Z.
  • Robustness to assumption violations: The method assumes additive superposition and independence z ⟂ ξ. It is unclear how performance degrades when concepts interact nonlinearly or z correlates with nuisance semantics (common in real datasets).
  • Confounding in time/place datasets: Release year is likely correlated with genre, vocabulary, and creator names; US place names correlate with state and urban/rural cues. The probe may capture correlated semantics rather than the target concept; no controlled or counterbalanced evaluations are provided.
  • Dependence on probing design: Only last‑token residual stream activations are used. The stability of learned manifolds across token positions, attention/MLP subcomponents, and other representation streams is not assessed.
  • Basis and penalty dependence: Features are parameterized with cubic B‑splines (280 knots for time; tensor splines for space) and smoothed by a quadratic penalty. The sensitivity of recovered manifolds to basis choice, knot placement, and penalty form/strength is not reported.
  • Dimension selection d: The paper uses ad‑hoc criteria (e.g., adding features until test R^2 < 0) and picks d=3 for steering. There is no principled, statistically justified model selection or significance testing for d.
  • Regularization selection guarantees: While GCV/REML are used within an alternating scheme, there are no convergence guarantees when λ’s are tuned adaptively, no analysis of eigenvalue multiplicities, and no study of how λ biases the manifold geometry.
  • Stability and reproducibility: There is no bootstrap or resampling analysis to test whether discovered features/subspaces are robust to dataset splits, prompt templates, knot grids, or random seeds.
  • Coverage and sampling measure: Orthogonality and normalization of f_k are enforced w.r.t. the empirical distribution of z. The effect of non‑uniform sampling over Z (e.g., decade or geographic imbalances) is not analyzed; no weighting/reweighting strategies are explored.
  • Comparison to baselines: The probe is not compared against alternative manifold‑discovery methods (e.g., sparse autoencoders, (kernel) CCA/SIR, Laplacian eigenmaps, graph geodesic approximations) on synthetic or real data with known ground truth.
  • Capturing nonlinearly accessible information: The approach explicitly finds features that are linearly predictable from representations. It does not detect concept information that is present but only nonlinearly accessible; no extensions (e.g., kernelized or piecewise‑linear variants) are explored.
  • Geometry beyond linear features: Although framed as discovering manifolds, the method does not estimate geometric quantities (e.g., local tangent spaces along z, curvature, or a Riemannian metric). It remains unclear how to quantify and compare manifold curvature across layers/models.
  • Varimax rotation ambiguity: Post‑hoc Varimax is used to aid interpretability, but rotated features remain fundamentally non‑unique. There is no statistical criterion or prior enforcing sparsity/localization during fitting, nor stability analysis of rotations.
  • Cross‑layer and cross‑position alignment: The paper reports R^2 and steering efficacy by layer but does not analyze whether subspaces/manifolds are aligned or transported across layers/positions (e.g., via principal angles, subspace tracking), nor identify which heads/MLPs write along these directions.
  • Multi‑concept superposition: Only one target concept is probed at a time. How to jointly learn multiple manifolds that overlap in the same subspace (and enforce mutual disentanglement) is not addressed.
  • Causal validation scope: Steering demonstrates sufficiency (model can be pushed to output target years) but not necessity. There are no ablations that project out or clamp the learned manifold (e.g., x ← x − Ψ(x)) to test whether removing it degrades year/location competence.
  • Dose–response and safety envelope: Steering uses a fixed magnitude α=100. There is no dose–response mapping, no characterization of minimal effective α, saturation, or off‑target side‑effects on unrelated behaviors.
  • Out‑of‑distribution targets: The method is not evaluated when steering to years outside the training range or to atypical inputs (e.g., obscure works, non‑US places, hallucinated titles), leaving OOD robustness untested.
  • Generalization across models and scales: Demonstrations are limited to Llama‑2‑7B. It is unknown whether similar manifolds exist, align, or scale in larger/smaller models or other architectures.
  • Tokenization and prompt dependence: The impact of tokenization (e.g., last token being punctuation/subword) and prompt phrasing on recovered manifolds and steering efficacy is not measured.
  • Computational scalability: Although p×p reductions are described, there is no empirical complexity/memory profile for large p, large m (knot counts), or large n, nor guidance for streaming/online settings or extremely high‑dimensional activations.
  • Handling periodic/topological structure: The approach uses standard splines; it does not incorporate periodic or manifold‑aware bases (e.g., circular splines for angles), nor constraints that reflect known topology of Z.
  • Uncertainty quantification: There are no confidence bands for f_k(z) or intervals for u_k, and no assessment of predictive uncertainty or variability across runs.
  • Negative controls and false positives: The paper does not include controls with permuted labels or irrelevant concepts to estimate false discovery rates in feature discovery.
  • Extension beyond residual stream: It remains open whether the same manifolds are recoverable and causal in attention value outputs, MLP activations, or the logit space, and how these relate mechanistically.
  • Practical guidance for probe users: Criteria for choosing knot counts, regularization strength, and d in new domains are not specified; the method’s failure modes and diagnostics are not cataloged.
View Paper Prompt  View All Prompts 

Practical Applications

Immediate Applications

Below is a concise set of actionable use cases that can be deployed with current models and tooling, leveraging the Manifold Probe’s methods (feature learning, closed-form/ALS solvers, layer-wise projections Ψ, and manifold steering).

  • Model interpretability and QA for LLM providers (software, AI safety)
    • Use the Manifold Probe to map continuous concept manifolds (e.g., time, geography) across layers, quantify linear accessibility via test R2R^2, and surface interpretable features (e.g., state- or decade-specific features via Varimax).
    • Tools/workflows: integrate probe runs into evaluation pipelines; produce “layer-by-layer manifold cards”; regression-based dashboards of top features and R2R^2 curves.
    • Assumptions/dependencies: access to model activations; probing datasets with labeled concept values; superposition/linear accessibility holds in target models; enough samples (often tens of thousands) or dimensionality reduction.
  • Controlled generation along time/space dimensions (content, marketing, search)
    • At inference, steer residual-stream activations along learned \hat φ(z) to bias outputs toward target years or locations (e.g., “write in the style of 1980s” or “tailor facts to California”).
    • Tools/workflows: lightweight “manifold steering” modules for text generation servers; layer- and α-tuned hooks for plug-in control.
    • Assumptions/dependencies: layer choice matters (effect peaked at mid-layers in Llama 2-7b); steering may slightly degrade unrelated behavior; effects are concept- and model-specific.
  • Time- and geo-aware RAG filtering and ranking (enterprise search, legal, news)
    • Use projections g_k(x) to infer temporal or spatial attributes from queries/passages and filter/re-rank by the intended era/region before retrieval.
    • Tools/workflows: add Ψ(x) features to retrieval schemas; build time/geo-aware rerankers without extra fine-tuning.
    • Assumptions/dependencies: learned features correlate with intended concept across domains; requires batch probing of target model; distribution shifts may reduce fidelity.
  • Bias and fairness audits across continuous concepts (policy, compliance, responsible AI)
    • Audit whether content about specific decades, regions, or demographics is over/under-represented or systematically mispredicted by analyzing feature predictability and rotations that localize on states/decades.
    • Tools/workflows: regulator/auditor probes that output interpretability summaries; standardized reports of feature coverage and error by sub-region/period.
    • Assumptions/dependencies: curated evaluation sets; careful interpretation to avoid spurious conclusions; legal/privacy constraints on data.
  • Feature drift monitoring across model releases (software MLOps)
    • Track how manifold geometry and top features shift between versions; flag regressions in predictable features (e.g., temporal reasoning).
    • Tools/workflows: CI checks that compare eigen-spectra, R2R^2 ladders, and rotated feature maps.
    • Assumptions/dependencies: consistent probing setup across versions; adequate sample sizes; compute budget for periodic scans.
  • Code generation targeting specific API versions or eras (software engineering)
    • Learn “version manifolds” (dates/releases) to steer code models toward target APIs (e.g., “Python 3.8 semantics”).
    • Tools/workflows: probes trained on codebases annotated by version/date; inference hook to steer toward chosen version.
    • Assumptions/dependencies: representations encode version/time; availability of labeled code/version datasets; validation on real tasks.
  • Geolocalized assistants and personalization (customer support, travel)
    • Infer and steer toward region-relevant knowledge for localized answers (e.g., state-specific regulations or attractions).
    • Tools/workflows: layer-level Ψ features appended to context for policy routing; optional steering to target region when ambiguity exists.
    • Assumptions/dependencies: risk of reinforcing regional stereotypes; careful evaluation for factuality; region concept must be linearly accessible.
  • Academic research tool for mechanistic interpretability
    • Rapidly discover continuous representations (e.g., counting, clocks, ideology, emotion), test causal involvement by interventions, and compare with sparse autoencoders.
    • Tools/workflows: open-source repo; ridge/REML/GCV-based training; Varimax factor analysis for interpretability.
    • Assumptions/dependencies: labeled probing datasets; compute for layer sweeps; results may be model- and dataset-specific.
  • Dataset coverage diagnostics (data operations, curation)
    • Use rotated features to find gaps (e.g., decades poorly represented, missing states), guiding data augmentation.
    • Tools/workflows: coverage heatmaps by learned features; automated curation flags to improve balance.
    • Assumptions/dependencies: representative seed datasets; validation against ground truth to avoid overfitting to probe artifacts.

Long-Term Applications

These opportunities require further research, scaling, or broader ecosystem development before they can be widely deployed.

  • Mechanistic guardrails via manifold attenuation (AI safety, governance)
    • Learn manifolds for risky concepts and attenuate or orthogonalize them to enforce policy (e.g., reduce encoding of prohibited content dimensions).
    • Tools/workflows: train-time or inference-time manifold filtering; compliance profiles by application domain.
    • Assumptions/dependencies: robust identification of harmful concept manifolds; low collateral damage; adversarial misuse risk.
  • General-purpose causal control across many continuous concepts (productized “sliders”)
    • Provide user-facing controls for era, region, sentiment, complexity, or style via manifold steering rather than prompt engineering.
    • Tools/workflows: “Manifold Steering SDK” with validated layer/α presets and safety checks; UI sliders in creative tools.
    • Assumptions/dependencies: stable, predictable effects across models and tasks; safeguards against unintended biases.
  • Multimodal and robotics control (vision, audio, embodied AI)
    • Discover and steer spatial, pose, or kinematic manifolds in vision/robot policies for safer planning or style transfer (e.g., pose/trajectory subspaces).
    • Tools/workflows: cross-modal probes (splines/tensor products) to map joints/pixels to activations; intervention hooks in control stacks.
    • Assumptions/dependencies: real-time constraints; safety certification; manifold linear accessibility in policy networks.
  • Healthcare model auditing and alignment (healthcare)
    • Map patient timeline or anatomical manifolds in clinical LMs/foundation models to audit temporal reasoning and anatomical localization.
    • Tools/workflows: probes on clinical activations; test R2R^2 maps for age/time biases; steering to control context timeframes.
    • Assumptions/dependencies: strict privacy/compliance; validated clinical datasets; robust generalization.
  • Financial assistant scenario control (finance)
    • Constrain analysis to specific historical windows or jurisdictions (temporal/geographic manifolds) for compliance and auditability.
    • Tools/workflows: manifold-informed templates for time-bound analysis; jurisdiction steering for regulatory consistency.
    • Assumptions/dependencies: highly regulated deployment; explainability requirements; model must encode these concepts reliably.
  • Standardized transparency and audit reporting (policy, regulators)
    • Create norms for “manifold disclosures” in model cards (e.g., list of key continuous manifolds, layer access, causal tests).
    • Tools/workflows: auditor toolkits that run probes against standardized benchmarks and report interpretability metrics.
    • Assumptions/dependencies: cross-organization agreement on benchmarks and metrics; governance frameworks.
  • Model editing and compression via manifold identification (software, systems)
    • Use learned low-dimensional subspaces to guide structured pruning or adapter placement that preserves critical continuous computations.
    • Tools/workflows: adapter training within discovered subspaces; regularizers that preserve manifold geometry.
    • Assumptions/dependencies: empirical validation that subspace-preserving edits maintain task performance; tooling maturity.
  • Scientific discovery in biological foundation models (life sciences)
    • Investigate phylogenetic/hematopoietic manifolds for hypothesis generation and validation; relate learned features to biology.
    • Tools/workflows: probes applied to bio models; rotation methods to localize features on cell types or lineages.
    • Assumptions/dependencies: high-quality labeled biology datasets; interdisciplinary validation; ethical oversight.
  • Tool-use and agent routing (software agents)
    • Project activations with Ψ to detect when certain tools should be invoked (e.g., mapping tasks by time/region), improving tool-selection policies.
    • Tools/workflows: agent controllers use manifold features as routing signals; fallbacks when uncertainty is high.
    • Assumptions/dependencies: stable correlations between features and tool utility; calibration of thresholds.
  • Production-scale manifold catalogs and monitoring (MLOps at scale)
    • Maintain a registry of known manifolds per model, continuously monitor drifts and steerability changes, and alert on regressions.
    • Tools/workflows: periodic background probes; telemetry on inference-time steering effectiveness; integration with model cards.
    • Assumptions/dependencies: ongoing compute; privacy/security controls for activation logging; strong change-management processes.

Notes on feasibility across applications:

  • Data and compute: Probing continuous manifolds typically benefits from large labeled datasets and access to full activations; smaller datasets may require PCA or dimensionality reduction first.
  • Modeling assumptions: Effectiveness relies on (i) approximate superposition, (ii) linear predictability of features in some layers, and (iii) concept–nuisance independence in probe data.
  • Safety and dual use: Steering can help alignment but can also be misused to bypass guardrails; appropriate governance and audit are needed.
  • Transferability: Manifold geometry and best intervention layers are model- and concept-dependent; validation is necessary before deployment.
View Paper Prompt  View All Prompts 

Glossary

  • Affine map: A linear function plus a constant offset, often used to map from a high-dimensional space to a subspace. "a linear (affine) map $\Psi : \mathbb R^p \to \hat{\* U} \subset \mathbb R^p$"
  • Alternating Least Squares (ALS): An iterative optimization method that alternates between solving least-squares subproblems for different parameter blocks. "we propose the Alternating Least Squares procedure detailed in Algorithm~\ref{alg:als}."
  • B-splines (cubic B-splines): Piecewise-polynomial basis functions used to flexibly approximate smooth functions. "parametrize time features using cubic B-splines with 280 knots"
  • Deflation (in sequential estimation): A technique that removes previously extracted components to isolate subsequent ones. "replacing x(z)x(z) with the deflation"
  • Factor analysis: A statistical method that models observed variables via a small number of latent factors, often followed by rotations for interpretability. "applying factor analysis to the learned features"
  • Generalized Cross-Validation (GCV): A closed-form criterion for selecting regularization/smoothing parameters without explicit cross-validation folds. "we use either Generalized Cross-Validation \citep{craven1978smoothing, wood2004stable} or Restricted Maximum Likelihood"
  • Generalized eigenvalue problem: An eigenproblem of the form Aβ=νBβA\beta=\nu B\beta that arises when optimizing under a metric or constraint matrix. "the generalized eigenvalue problem"
  • Generalized ridge regression: Ridge regression with potentially non-identity penalty structure or generalized design matrices. "a sequence of alternating (generalized) ridge regression problems, which we prove converge to the global minimizer"
  • Injective map: A one-to-one function that maps distinct inputs to distinct outputs. "We say that any injective map ϕ:Z→Rp\phi : \mathcal Z \to \mathbb R^p represents the concept $\*Z$."
  • Intervention experiment: An experimental setup that actively modifies internal activations to test causal influence on model behavior. "We perform an intervention experiment where, at a given layer, we steer the residual stream representations"
  • Linear classifier probes: Linear classifiers trained on internal representations to test whether specific information is linearly decodable. "propose fitting a family of linear classifier probes to a discretization of the concept space"
  • Linear probes: Linear models used to predict target variables from internal representations as a measure of linear accessibility. "such as linear probes \citep{alain_understanding_2017,nanda_emergent_2023}"
  • Linear representation hypothesis: The hypothesis that neural networks organize features so they are accessible via linear projections. "A key hypothesis in this effort is the linear representation hypothesis"
  • Manifold Probe: The paper’s supervised probing method to discover multidimensional representation manifolds in superposition. "This paper introduces the Manifold Probe, a supervised method for discovering representation manifolds in superposition."
  • Mechanistic interpretability: A research area focused on understanding the internal mechanisms and computations of AI systems. "A key hypothesis in mechanistic interpretability is that of superposition"
  • Neighbour graphs: Graphs connecting nearby samples used to approximate and study manifold geometry. "propose approximating representation manifolds with neighbour graphs."
  • Power-iteration: An algorithm that repeatedly multiplies by a matrix to find the dominant eigenvector/eigenvalue. "Power-iteration is known to converge to the global solution under mild conditions"
  • Principal component analysis (PCA): A dimensionality-reduction technique that projects data onto orthogonal directions of maximal variance. "perform a preliminary principal component analysis to reduce the dimension of the representation space"
  • Representation manifold: A low-dimensional manifold in representation space that encodes a continuous concept. "its image $\*M = \phi(\*Z)$ is a representation manifold embedded in Rp\mathbb R^p"
  • Residual stream: The sequence of residual activations passed along transformer layers, often probed for encoded information. "from layer 16 residual stream activations of Llama 2-7b."
  • Restricted Maximum Likelihood (REML): A method for estimating variance/regularization parameters by maximizing the likelihood of residuals. "or Restricted Maximum Likelihood \citep{bartlett1937properties, wood2011fast}"
  • Semantic space: An abstract product space of concepts and other semantics representing the meaning of inputs. "we will consider an abstract topological space $\*S$ which we refer to as the semantic space"
  • Singular value decomposition (SVD): A matrix factorization into orthogonal factors and singular values, used for stable reparametrization. "for example, using its singular value decomposition."
  • Sparse autoencoders: Autoencoders with sparsity-inducing objectives that promote disentangled, interpretable features. "sparse autoencoders \citep{elhage_toy_2022, bricken_towards_2023, cunningham2024sparse}"
  • Steering vectors: Directions in representation space added to activations to steer a model’s behavior toward desired outputs. "we steer the residual stream representations by adding steering vectors which trace the manifold."
  • Stratified sample: A sample constructed to maintain balanced representation across predefined subgroups (strata). "We then select a stratified sample of 1,400 works"
  • Superposition: The hypothesis that representations of different concepts add linearly to compose joint semantics. "A related hypothesis is that of superposition"
  • Tensor product (of B-splines): A multivariate basis formed by products of univariate spline bases to model functions over grids. "using a tensor product of two cubic B-splines with 40 and 80 knots respectively."
  • Topological space: A mathematical structure of points and open sets used to formalize continuity and conceptual spaces. "A concept is a topological space $\*Z$"
  • Varimax rotation: An orthogonal rotation that makes factor loadings more sparse and interpretable. "Employing a Varimax rotation which aims to make the features approximately sparse"
View Paper Prompt  View All Prompts 

Open Problems

We haven't generated a list of open problems mentioned in this paper yet.

Generate Now

Collections

Sign up for free to add this paper to one or more collections.

Sign Up

Tweets

Sign up for free to view the 1 tweet with 66 likes about this paper.

Sign Up for Free