Generalization of structural error fingerprints across architectures and scales

Determine whether the structural fingerprints of reasoning errors identified by Circuit-based Reasoning Verification on attribution graphs generalize to different model architectures, such as Mixture-of-Experts Transformers, and to substantially larger model scales (e.g., 70B parameters and above).

Background

The paper introduces Circuit-based Reasoning Verification (CRV), which analyzes attribution graphs from a transcoder-instrumented Llama 3.1 8B Instruct model to detect reasoning failures. Empirically, the authors show strong in-domain verification performance and that signatures of error are highly domain-specific.

However, all empirical results are reported for a single model family and size. The authors explicitly flag uncertainty about whether these structural error fingerprints transfer across different architectures (e.g., Mixture-of-Experts) and to much larger model scales, making this generalization question a stated open problem.