The Emergent Symbolic Structure of Artificial Neural Networks

Abstract: Modern systems in AI somehow excel in domains for which they seem poorly suited. Intelligence has traditionally been modeled as operating over structured combinations of symbols, such as logical formulas. However, the strongest modern AI systems are based on neural networks, which instead represent information in continuous vectors. Vectors seem inadequate for capturing the structure of language, logic, and other cognitive domains, yet neural networks achieve impressive performance in these areas. How do they do it? In this work, we propose a potential answer: Despite appearances, perhaps the internal representations of neural networks implicitly realize symbolic structure. In support of this hypothesis, we show that the vector representations of a variety of neural networks can be closely approximated with symbolic structures: we can replace the network's entire representation-generating process with a closed-form equation instantiating a symbolic structure, and the network's behavior remains largely unchanged. This finding holds for both small-scale neural networks trained to manipulate lists as well as LLMs operating in four domains that are central in symbolic traditions: arithmetic, logic, computer code, and language. Further, our symbolic approximation allows us to modify an LLM's behavior in targeted ways via precise interventions on its internal representations, showing that the LLM's behavior is reliant on the symbolic structures we have identified. This work provides a potential way to reconcile longstanding symbolic conceptions of intelligence with the vector-based nature of modern AI.

• The paper advances DISCOVER, a methodology to uncover symbolic structure in neural networks by fitting Tensor Product Representations (TPR) and confirming their functional accuracy.
• The DISCOVER technique was applied across Task-specific architectures accounting for tens of combinations of models and tasks, achieving an average approximation accuracy over 0.97.
• Neural networks demonstrate the ability to generalize to unseen role-filler bindings with OOD accuracy up to 0.97 for the interleaving bottleneck transformer, establishing a degree of systematicity in their symbolic representations when tested on tasks like parity.

Problem formulation and central claim

“The Emergent Symbolic Structure of Artificial Neural Networks” (2608.29530) addresses a representational tension in contemporary neural computation. Symbolic domains such as language, arithmetic, logic, and program execution require binding content to structural positions: reversing a sequence requires preserving both the identity and position of each element; arithmetic depends on operator scope and precedence; code execution depends on variable-function bindings; and syntactic transformations depend on grammatical roles. Standard neural networks, however, expose continuous vectors rather than discrete trees or symbolic expressions.

The paper’s central hypothesis is that this apparent representational incompatibility is misleading. Neural networks may encode symbolic structure implicitly in distributed vector spaces, even when their architectures contain no explicit symbolic data structure. More specifically, the authors argue that many trained networks approximate linearly transformed Tensor Product Representations (TPRs), in which fillers are bound to roles through tensor products, summed across constituents, and then mapped into the target representation space by an affine transformation. The resulting claim is deliberately stronger than the claim that symbolic information is decodable from neural activations: the proposed structures approximate the global organization of the representation space and can be used to reconstruct behavior and perform causal interventions.

The empirical strategy is DISCOVER—“DISsecting COmpositionality in VEctor Representations”—which fits an explicitly TPR-structured model to the hidden representations of a target network. The target network is then evaluated with DISCOVER’s reconstructed representations substituted for its original representations. If the original decoder or downstream computation continues to produce the correct output, the approximation is functionally adequate.

TPRs and the DISCOVER methodology

A linearly transformed TPR represents a structure as role-filler pairs. Each filler $f_i$ and role $r_i$ is embedded as a vector, their tensor product $f_i \otimes r_i$ is computed, and the resulting matrices are summed. A learned affine map then produces the target vector representation:

$E = W\left(\sum_i f_i \otimes r_i\right) + b.$

The tensor product provides multiplicative binding: the same filler can receive different representations when assigned different roles. The summation provides compositional aggregation, while the learned transformation accounts for arbitrary rotations, rescalings, dimensionality changes, and coordinate permutations between the TPR space and the neural network’s representation space.

Figure 1: A linearly transformed TPR binds fillers to roles through tensor products, aggregates the bound pairs, and maps the result into the target vector space.

DISCOVER trains the filler embeddings, role embeddings, and affine transformation to minimize mean-squared error against the target representations. However, the decisive evaluation is behavioral rather than merely geometric. The reconstructed vector is passed to the target model’s decoder, and approximation accuracy is defined as the proportion of test examples for which the decoder generates the complete correct output.

Figure 2: DISCOVER replaces a target network’s representation-generating computation with an interpretable closed-form TPR approximation and evaluates functional equivalence through the original decoder.

This design distinguishes DISCOVER from diagnostic probing. A successful probe establishes that some information is linearly recoverable; it does not establish that the information organizes the full representational space or causally mediates behavior. DISCOVER imposes a highly constrained structural hypothesis and tests whether that hypothesis can replace the target representation-generating process. Consequently, success is informative precisely because it is not guaranteed. A TPR with a fixed role scheme can express only a restricted family of functions.

The method nevertheless has an important asymmetry. A successful approximation shows that the DISCOVER representation contains the information needed by the target decoder, but it does not prove that the target representation contains no additional information. A more expressive role scheme can approximate a less expressive one, so DISCOVER can overstate the complexity of the structure actually used by the target model. The paper explicitly treats DISCOVER as a test for the existence of an adequate symbolic approximation, not as a uniquely identifying inverse of the target representation.

Emergent structure in synthetic sequence models

The first experiments analyze MLPs, GRUs, standard Transformers, and bottleneck Transformers trained on copying, reversal, and interleaving tasks over sequences of one to six capital letters. Each task requires information about both letter identity and serial position. The authors train ten target-model replicates for each architecture-task combination and evaluate DISCOVER under five role schemes: left-to-right, right-to-left, bidirectional, Wickelroles, and bag-of-words.

The running reversal experiment provides a clean demonstration. The bidirectional TPR approximation achieves perfect or near-perfect functional reconstruction in all ten GRU reruns; the lowest approximation accuracy is 99.98%. Right-to-left roles perform nearly as well, whereas left-to-right roles perform substantially worse, consistent with the decoder’s need to produce the input from the end backward. The bag-of-words scheme fails, providing a null comparison in which letter identity is represented but serial structure is absent.

Figure 3: Bidirectional role schemes closely reconstruct reversal-GRU representations, whereas structureless bag-of-words representations fail.

The broader sequence experiments show that this result is not specific to GRUs or reversal. Across all twelve architecture-task combinations, bidirectional DISCOVER approximations perform strongly. The weakest condition is the reversing bottleneck Transformer, with an average approximation accuracy of 0.973; every other architecture-task condition exceeds 0.99 on average. The directional asymmetries are task-sensitive: left-to-right roles tend to perform better for copying, right-to-left roles for reversal, and neither unidirectional scheme is generally adequate for interleaving, which combines information from both ends of the sequence.

Figure 4: Bidirectional TPR approximations remain strong across MLPs, GRUs, standard Transformers, and bottleneck Transformers trained on sequence-manipulation tasks.

Two control results strengthen the interpretation. First, Wickelroles perform worse than bidirectional roles despite using 729 roles rather than 21, showing that approximation quality is not explained by parameter count. Second, left-to-right and right-to-left schemes use the same number of roles but exhibit systematically different performance. When the target task is alphabetical sorting, for which input order is irrelevant, bag-of-words approximations become reasonably effective. This task dependence indicates that structural information is not merely an unavoidable artifact of the architecture: networks encode and rely on the structure demanded by the objective.

Period representations in seven LLMs

The paper next studies seven open-weight LLMs: Gemma-3-27B, GPT-2-XL, GPT-OSS-20B, Pythia-12B, Qwen3-14B, OLMo-2-13B, and Llama-3.1-8B. Because autoregressive LLMs produce one vector per token per layer, the authors focus initially on the representation of the period terminating a list or sentence. The period is the first token at which the entire preceding context is available under causal attention, making it a plausible location for sentence-level information.

Before applying DISCOVER, the authors train period-unpacking Transformers to reconstruct the preceding list or sentence from the period vector. These decoders achieve perfect accuracy for all subject-verb-object conditions and typically exceed 85% for lists and 50% for complex sentences under an exact-match criterion. Thus, period vectors contain substantial information about both content and order.

Figure 5: The period-encoding experiment first tests whether a punctuation representation supports reconstruction of its preceding context, then evaluates whether a TPR approximation preserves that information.

For lists, bidirectional roles substantially outperform both bag-of-words and predecessor roles. This is notable because predecessor roles use a much larger role vocabulary—301 roles versus 12—and are theoretically sufficient to reconstruct the unique-element lists. For subject-verb-object sentences, a subject-verb-object role scheme achieves perfect approximation. For complex sentences, bidirectional roles consistently outperform dependency-based syntactic roles.

Figure 6: Across seven LLMs and multiple layers, bidirectional and task-appropriate TPRs preserve information in period representations more effectively than structureless or predecessor-based alternatives.

The complex-sentence result is theoretically important but also constrains the paper’s interpretation. The period representations appear to prioritize linear order over dependency structure, even when the downstream period-unpacking model is trained to output linearized constituency or dependency parses. The authors therefore do not claim that LLMs lack syntactic representations. Rather, they conclude that the particular period vectors examined here are principally organized by sequential structure, while other token positions may encode syntax differently.

A striking result is that period-unpacking models sometimes perform better with DISCOVER approximations than with the original LLM period vectors. For GPT-OSS’s middle layer in the complex-sentence condition, exact reconstruction rises from 0.71 with the original period vectors to 0.96 with bidirectional DISCOVER approximations. The authors interpret this as evidence that the original vectors contain an approximately symbolic but noisy organization. DISCOVER removes residual variation while preserving the structure exploited by the decoder. This result supports the paper’s proposed “limitivist” position: neural representations may approximate symbolic systems without implementing them exactly.

Full-representation reconstruction in GPT-OSS

The most consequential experiments analyze GPT-OSS across six tasks spanning four traditionally symbolic domains:

• arithmetic with operator precedence;
• syllogistic reasoning;
• Python list-function execution;
• passivization;
• tense reinflection;
• yes/no question formation.

GPT-OSS itself achieves 0.76 on syllogisms, 0.94 on passivization, and above 0.96 on all other tasks. Unlike the period experiments, DISCOVER approximates every input-token representation across all 25 layers, excluding only invariant prompt material. The approximated representations are inserted back into GPT-OSS, which must then generate its output normally.

Figure 7: In the full-representation experiment, DISCOVER replaces GPT-OSS’s input representations across tokens and layers while retaining the original model for subsequent computation and generation.

The comparison of role schemes tests two hypotheses: whether each token encodes only itself or also preceding context, and whether the relevant structure is task-specific or merely positional. The strongest scheme is task-specific (all), in which each token encodes itself and preceding tokens using roles that combine the structural position of the earlier token with that of the current token. This pair-based construction is compatible with the relational interactions induced by attention.

Across all six tasks, task-specific (all) produces accuracies extremely close to GPT-OSS’s original performance. The largest gap is 2.36 percentage points, in arithmetic. The result implies that GPT-OSS’s task performance can be preserved after replacing its entire input representation process with a closed-form, TPR-based approximation. It also supports two more specific claims: each token representation contains multiple role-filler pairs, and the roles encode task-relevant structure rather than merely token identity or linear position.

Figure 8: Task-specific TPR approximations reproduce GPT-OSS behavior across arithmetic, logic, code execution, and three language-transformation tasks.

The role-scheme comparisons are not reducible to model capacity. In syllogisms, task-specific roles use fewer roles than bidirectional roles yet perform better. In code execution, task-specific self roles can outperform bidirectional all roles despite using substantially fewer roles. Moreover, the authors use rigid controls and causal tests to reduce confounding between syntactic position and linear position.

Causal evidence from constituent surgery

Functional reconstruction alone could still reflect a representational correlation rather than a causally relevant decomposition. The paper therefore performs constituent surgery: it subtracts one role-filler contribution from a target representation and adds another. For example, an encoding containing a filler bound to an object-adjective role can be edited by replacing that contribution with the same filler bound to a subject-adjective role.

Figure 9: Constituent surgery edits a target representation by subtracting one DISCOVER-derived role-filler contribution and adding another.

The synthetic sequence models and period representations show near-perfect intervention performance. In GPT-OSS, the authors evaluate 31 intervention types across the six symbolic tasks and obtain a mean intervention accuracy of 0.903. Filler edits are particularly strong: changing a list element achieves 0.99, changing repetition count 1.00, changing the called function 0.95, and changing the function argument variable 0.96.

Figure 10: Filler-level interventions in GPT-OSS achieve high accuracy across list, repetition, function-selection, and variable-binding edits.

Role edits also produce targeted behavioral changes. Swapping numbers in arithmetic expressions achieves 0.978 average accuracy; moving a letter between Python lists achieves 0.950; and moving modifiers in tense reinflection achieves 0.980 for adjectives, 0.892 for prepositional phrases, and 0.958 for relative clauses.

Figure 11: Role-level interventions modify arithmetic, code, and syntactic behavior in accordance with the edited structural position.

The distinction between local and complete interventions is diagnostically useful. Editing only the representation of the target token is often sufficient for filler changes, but substantially less effective for role changes. This indicates that token identity is comparatively localized, whereas structural information is distributed across later token representations. The result is consistent with the task-specific (all) hypothesis, under which a token leaves role-dependent traces in subsequent states.

The authors also test whether structural interventions are merely position shifts in disguise. They construct matched intervention pairs that make the same linear movement but differ in syntactic interpretation. The matched structure-sensitive interventions consistently outperform mismatched interventions, with the clearest separation in tense reinflection. This supports the claim that at least some edits act on abstract structural roles rather than only on token order.

Systematicity and out-of-distribution role-filler binding

A central methodological issue is distinguishing systematic binding from atomic lookup of role-filler pairs. A model could represent every encountered combination—such as “scientist as subject”—with an unrelated atomic vector. Such a model might be approximated well in-distribution by DISCOVER but would provide no basis for representing an unseen combination such as “scientist as object.”

The authors therefore train DISCOVER while withholding selected role-filler combinations and test on examples containing those combinations. They establish the interpretation using white-box controls: DISCOVER generalizes for a model with known systematic TPR binding but fails for an encoder with independent atomic embeddings for every role-filler pair.

Figure 12: Generalization to withheld role-filler combinations distinguishes systematic TPR binding from atomic role-filler lookup.

The black-box results are broadly positive. Almost all sequence-model conditions exceed the strong chance baseline for multiple withheld combinations, with the interleaving Transformer as the principal exception. All period-encoding experiments show robust above-baseline generalization. In GPT-OSS, every symbolic task except arithmetic substantially exceeds the baseline at larger numbers of unseen role-filler pairs.

The arithmetic exception is important. It prevents the paper from claiming uniformly systematic role-filler composition across all analyzed tasks. The authors attribute some of the difficulty to the compact and highly constrained arithmetic stimuli, but the empirical result itself remains a qualification: evidence for systematic binding is strong in most conditions, not universal.

Figure 13: DISCOVER generally generalizes beyond observed role-filler combinations in GPT-OSS, although arithmetic is a notable exception.

The authors use $L_{2,1}$ regularization to discourage degenerate solutions in which embeddings effectively allocate independent dimensions to individual role-filler combinations. This regularization can materially alter results: in the interleaving bottleneck Transformer, OOD accuracy increases from 0.54 to 0.97, while for some period and GPT-OSS conditions it changes performance from near-zero or moderate levels to above 0.86–1.00. Thus, OOD conclusions depend on regularization and on careful control of model capacity.

Limitations and open questions

The evidence is strongest for fully systematic, synthetic domains and restricted linguistic templates. The GPT-OSS experiments use carefully generated arithmetic, syllogistic, code, and CFG-based language tasks, often with single-token vocabulary items, fixed prompt formats, and in-context demonstrations. These controls are appropriate for isolating representational structure but do not establish that the same TPR organization dominates unconstrained natural-language inputs.

DISCOVER is also supervised. The experimenter specifies candidate fillers and role schemes, including task-specific parses for arithmetic, code, and syntax. This limits the method’s applicability when the relevant structural vocabulary is unknown. A successful approximation can also contain a structural superset of the information used by the target model; the method does not uniquely identify the minimal representation.

The use of linearly transformed TPRs creates an additional identifiability limitation. Major vector-symbolic architectures, including holographic reduced representations and related binding schemes, can be expressed as special cases of linearly transformed TPRs. Consequently, successful DISCOVER analyses establish a family-level role-filler structure, not the specific algebraic architecture implemented by the target model.

The LLM period analysis relies on the assumption that period states are sentence-level representations. The decoding results support this assumption, but the analysis is necessarily selective: it does not characterize all token states, and the finding that bidirectional roles outperform syntactic roles at the period does not generalize automatically to other layers or positions. Similarly, the GPT-OSS analysis assumes a common role hypothesis across all layers, even though different layers may encode different subsets or transformations of the information.

Finally, causal intervention accuracy is high but not perfect. The overall GPT-OSS average of 0.903, and the lower performance for certain relative-clause and prepositional-phrase edits, indicates that the identified symbolic structure is behaviorally relevant without being an exact executable symbolic representation. The paper leaves open whether the deviations from exact TPR structure are noise, task-dependent approximations, or computational mechanisms that support graded and statistical generalization.

Conclusion

The paper provides convergent evidence that neural networks can acquire structured vector representations with the functional properties of symbolic role-filler systems. Across synthetic sequence models, seven LLMs’ period states, and GPT-OSS representations used for arithmetic, logic, code, and language, linearly transformed TPRs often replace the original representations with little behavioral degradation. Causal surgery and OOD generalization further indicate that the recovered structures are not merely descriptive: models are sensitive to their constituent bindings and, in most conditions, compose roles and fillers systematically.

The most defensible conclusion is not that neural networks implement exact discrete symbol systems. Rather, the results support an intermediate account in which learned neural representations approximate symbolic structure within continuous spaces. The specific mechanism by which ordinary architectures construct and process these TPR-like representations remains unresolved, as does whether analogous structures organize representations in less controlled naturalistic tasks.