Legible-by-Construction: Attention and End-to-End Transformers

Abstract: A companion paper showed that a transformer's feed-forward layer can be rebuilt from explicit fuzzy set operations - intersection, set-difference, and a self-forgetting sequence quantifier - so its hidden units read as named logical operators at no cost to language-model quality. That left the other half of the transformer opaque. Here we carry the same idea into attention and join the two into one model. The mechanism is minimal: a head's value is passed through a sigmoid, so each value channel becomes a readable detector of whether a feature holds at a token. This adds no parameters and leaves the standard head otherwise untouched. A Boolean variant goes further, restructuring the value into an explicit within-token intersection and negation-capable set-difference. In both designs the output projection is left free, not tied to the vocabulary, which is the load-bearing decision: bounding what a head detects while leaving what it writes unconstrained yields selective detectors, whereas constraining the write does not. A bounded value is shaped into a readable detector by two selectivity pressures - one for sparse firing, one for decisive firing at the rails - and which a design wants is not universal. Across five specialized-attention designs at 125M parameters, 44 to 62 percent of value channels become crisp, contextually selective detectors, and their legibility rises with depth rather than crystallizing only on punctuation. Language-model quality is at parity with a conventional baseline. Finally, we couple the Boolean attention to the legible feed-forward layer and train an end-to-end legible-by-construction LLM at benchmark parity: its feed-forward units are named set and quantifier operations throughout, and we can take a token it generates and read the named units that compose to produce it.

• The paper introduces a minimal architectural change that constrains attention head values to achieve inherent mechanistic interpretability.
• The method employs two variants—membership and Boolean value heads—to create bounded, sparse, and context-selective detectors without extra parameters.
• Empirical results show deep transformer layers exhibit up to 95% selective behavior with no measurable performance tradeoff in language modeling tasks.

Legible-by-Construction Transformers: Enabling End-to-End Mechanistic Interpretability

Introduction and Motivation

This paper addresses the long-standing challenge of mechanistic interpretability in transformer-based LLMs. Existing interpretability approaches are predominantly post-hoc, leveraging tools such as the logit lens [nostalgebraist2020logitlens], tuned lens [belrose2023tunedlens], and sparse autoencoders [bricken2023monosemanticity, cunningham2023sae, kissane2024attnsae] to recover a partial, often contestable understanding of model internals. However, as these methods reconstruct explanations external to the model, interpretability remains both incomplete and difficult to maintain under distributional or model changes.

Building on prior work making feed-forward layers (FFN) interpretable by embedding explicit fuzzy set operations into their architecture [oskin2026ncffn], this paper introduces a minimal yet robust architectural modification for the attention mechanism, facilitating end-to-end legible computation without sacrificing model performance. Rather than constraining model outputs or readouts (which leads to collapse), the approach focuses on constraining what attention heads detect via bounded, interpretable value representations.

Architectural Innovations for Legibility

The central tenet of this work is constraining attention head values—the "what" instead of the "where" (i.e., QK attention map)—which induces legibility by construction.

Two variants are explored:

1. Membership Value Heads: Each value channel is bounded to $[0,1]$ via a sigmoid, encoding the degree to which a given feature holds for a token. This provides channels that read as sparse, context-selective detectors while maintaining parameter neutrality (no additional parameters compared to standard attention).
2. Boolean Value Heads: Value projections are split, forming feature pairs $A, B$ such that $A \cap B$ (intersection) and $A \setminus B$ (set-difference) are explicit logical combinations, enabling direct interpretation in terms of fuzzy Boolean operators. This also maintains parameter counts and ensures each channel is interpretable as a named logical relation.

A critical empirical result is that constraining what a head detects while leaving the output projection (write) free yields legibility; in contrast, output-side constraints (e.g., vocabulary-tied readouts) force collapse to context-independent constants, destroying expressivity and utility.

To further enhance legibility, two selectivity pressures on bounded values are introduced:

• Sparsity: Drives values toward zero unless supported by context, promoting contextually specific activation.
• Crispness: Drives values to the rails (near 0 or 1), ensuring decisiveness of detector activations.

Empirically, the optimal mix of these pressures is operator-specific: single membership values benefit from both; Boolean conjunctions achieve best legibility under crispness pressure alone, as additional sparsity suppresses too many channels into inactivity.

Empirical Results: Legibility and Performance

The study conducts a comprehensive evaluation across multiple architectures and selectivity regimes, benchmarking against a strong 125M parameter gelu-transformer baseline. Key findings include:

• 44–62% of value channels exhibit crisp, contextually selective behavior, with the fraction rising to 70–95% in the deepest layers.
• Contrary to prevailing folklore, legible attention heads emerge not only in shallow layers tracking punctuation/position but predominate in deeper layers, aligning with semantic/grammatical features.

Figure 1: SELECTIVE value channels by layer.

Notably, language modeling performance on LAMBADA, BLiMP, and ARC-Easy is at parity with the baseline; the strongest model variants reach the top of the parity band. There is no measurable tradeoff between achieving legible-by-construction and retaining model quality, except for isolated bounded-head configurations under sparsity alone, which are remediated by combined pressures.

Table: Attention Channel Legibility by Depth

Layer	SELECTIVE % (Boolean head-100, Crispness)
1–4	0–20%
5–7	25–60%
8–11	87–88%

This depth-dependent trend reflects the migration of legible computation into higher layers once both FFN and attention are legible.

Figure 2: Per-head attention legibility across all 144 heads, highlighting a pronounced depth gradient, with nearly all deep heads becoming selective detectors.

This explicit legibility is visually summarized in the per-layer and per-head plots, which demonstrate that only deep attention heads develop high-context, decisive selectivity.

End-to-End Legibility: Joint Attention and Feed-Forward Interpretability

Coupling the Boolean value heads with the explicitly logical FFN yields a transformer whose dominant computations (both attention and feed-forward) are named operations. In this architecture, each generated token can be traced back to a composition of interpretable, named units (e.g., superlative, negation, quantifier).

A single prediction demonstrates this interpretability:

Figure 3: Reading a single prediction ("worst" at the end of "It was the best of times, it was the")—token attribution decomposed across all heads and units, showing individual named operations contributing to the model's output.

Key empirical points:

• The majority of the prediction's logit attribution is explained by composed named FFN units (e.g., superlative intersected with negation), rather than opaque residual activations.
• Deep attention heads complement the process by detecting long-range context or semantic classes, while shallow attention heads collapse to near-constant channels once local processing is absorbed by legible FFN units.
• The operator's operands (the detector features themselves) often remain locally polysemantic, but the logical composition is explicit and directly readable, a substantial improvement over post-hoc recovery.

A division of computational labor is observed: the feed-forward layer handles local, grammatical, and semantic composition through explicit operators, while attention becomes semantically legible mainly at depth.

Figure 4: End-to-end legibility by layer—feed-forward operands are crisp throughout, while Boolean attention value becomes selective primarily in the deep half of the model.

Practical, Theoretical, and Future Implications

By-construction legibility brings a number of practical advantages:

• Direct model editing and steering: Operators can be clamped or ablated with clear semantic effect, e.g., disabling a negation unit.
• Interpretability at inference time: Logging/operator monitoring becomes trivial, as detector meanings are fixed and readable.
• Auditable mechanisms: For domains requiring explainability, units can be targeted for verification or monitoring, facilitating use in safety-critical applications.

Theoretically, embedding explicit set and logical computations moves transformers toward architectures amenable to formal analysis, property verification, and reasoning over network behavior.

Limitations include single-seed, small-scale experiments; scaling behavior to billion-parameter models and retention of operand monosemanticity remains open. Shallow attention heads, once coupled to legible FFN, often become constant; closing this gap or developing shallow-specific constraints is left as future work.

A careful analysis of stability reveals a trainability boundary for partial bounded-head designs, caused by the dynamic imbalance between saturated bounded heads and overdriven unbounded ones.

Figure 5: Instability in partial bounded/unbounded head mixtures—saturation and runaway writes in unbounded heads at divergence.

Conclusion

This work demonstrates that legible-by-construction architectures for transformers—achieved by bounding attention values as explicit set detectors and coupling them with logical feed-forward operations—enable highly interpretable, editable, and auditable LLMs at no measurable cost to modeling quality. Selectivity pressures must be tailored to operator structure, and future work should address scalability, operand monosemanticity, and shallow-layer legibility. These results open the path toward LLMs whose computation is both transparent and actionable, transforming interpretability from a post-hoc reconstruction to an intrinsic property of the model.

Figure 6: In a standard transformer, interpretability is post-hoc and one-sided (feed-forward only); value channel legibility is achieved only after external analysis—the situation fundamentally changed by this paper.