Characterize causal integration of distributed latent features in neural networks

Determine how trained artificial neural networks causally integrate distributed latent features across channels and neurons to generate outputs by characterizing the coordinated contributions of groups of hidden units that combine to produce model predictions.

Background

The paper argues that most current interpretability approaches analyze hidden activations, which reflect only receptive fields and not the causal effects of hidden units on outputs. The authors introduce Contribution Decomposition (CODEC) to directly measure and decompose hidden-unit contributions, but they emphasize that a broader, principled understanding of how distributed features are causally integrated remains unresolved.

They motivate this problem by analogy to biological systems, where functional effects emerge from interactions across circuit elements, suggesting that a similar characterization is needed for artificial networks to move beyond correlational analyses of representations.

References

This rules out directly copying a stated answer token, but leaves richer hidden-feature and causal-use questions open.

— The Truth Was Never Gone: Perfect Aliasing in Compliant-Context Truth Probes  (2609.10739 - jayabahu, 9 Sep 2026) in Appendix, Section “The inferred-truth task, settling and ingredient controls,” subsection “Partial evidence, at one cell, that the direction is not reading a copied token”

Accordingly, a key open problem is to characterize how networks causally integrate distributed latent features across channels and neurons to generate outputs, analogous to how biological networks produce functional effects through circuit interactions.

— Causal Interpretation of Neural Network Computations with Contribution Decomposition  (2603.06557 - Melander et al., 6 Mar 2026) in Subsubsection “Existing tools for interpreting ANNs,” Section 1 (A framework for understanding biological and artificial neural networks)

EDUCATION$\times$INCOME---the type this caveat matters most for---has not been tested under a multi-layer patch; redundant, single-layer-invisible use remains an alternative we have not ruled out for it.

— Readable, Faithful, Used: Three Dissociable Properties of Demographic Identity in a Language Model  (2608.18768 - Robbani, 19 Aug 2026) in Section 5.3, paragraph “Caveats”

Understanding how information is processed, spatially distributed, and geometrically represented within this hidden space remains an open challenge critical for interpreting the model's emergent capabilities.

— Neural Cellular Automata Learn General Features in their Hidden Channels  (2609.21870 - Guichard et al., 18 Sep 2026) in Section 1, Introduction

Head-level attribution [39], a matched-layer comparison of the sparse and distributed families, and sparse-dictionary decomposition of the steering direction into interpretable features [26] would further test whether the distributed direction is a single feature or a bundle of them.

— FearCaut-Qwen: Affective Steering in a Vision-Language Model Shifts the Decision Criterion for Hazard Assessment  (2610.11986 - Zhou, 8 Oct 2026) in Section 5.3, Mechanistic completeness