Internal Agreement in LSTM Models

• Internal Agreement is a concept detailing how LSTM models encode and track subject–verb number through dynamic, temporally evolving internal states.
• Diagnostic classifiers probe LSTM components, revealing that key agreement signals are most prominently stored in the cell state and hidden state of the second layer.
• Targeted interventions based on these probes effectively correct representational errors, significantly boosting agreement accuracy with minimal impact on overall performance.

Diagnostic classifiers are auxiliary models trained to predict a linguistic feature from the internal activations of a neural model. In the study of subject–verb number agreement, they were used to investigate how an LSTM LLM tracks the grammatical number of the main verb, when that information is represented, where it is encoded in the network, and how it is corrupted in error cases. The core result is twofold: diagnostic classifiers provide a detailed account of the internal representation of agreement information, and the recovered agreement signal can be used to intervene on the LSTM’s hidden computation, yielding a large increase in agreement accuracy with minimal change in overall language modeling perplexity (Giulianelli et al., 2018).

1. Research question and empirical setting

The central problem is how neural LLMs keep track of number agreement between subject and verb, especially in long-distance dependencies where intervening nouns can create attraction effects. The study focuses on an LSTM-based LLM and examines English sentences containing subject–verb agreement dependencies, including difficult cases with long contexts and attractors. The data include replication datasets from prior work, notably Linzen et al. 2016/2018, as well as “nonce” sentences intended to control for semantic and frequency confounds. Synthetic subsets further vary the context size and the number and position of attractors (Giulianelli et al., 2018).

This setup matters because agreement prediction is not simply a test of local co-occurrence statistics. The intervening nouns can carry competing number features, so the model must preserve the relevant number information across time rather than rely on the nearest noun or other superficial cues. The paper’s analysis is therefore aimed at internal state dynamics rather than only end-task accuracy.

A recurring contrast in the experiments is between cases where the LLM predicts verb number correctly and cases where it does not. This distinction is operationalized as the “correct” set and the “wrong” set. The comparison is important because it turns the probing task from mere feature detection into error diagnosis: the analysis asks not only whether number information is present, but also when it becomes degraded in sentences that eventually induce agreement errors.

2. Diagnostic classifiers as probes of LSTM representations

The probing methodology records the activations of the model’s hidden state $\mathbf{h}_t$, memory cell $\mathbf{c}_t$, forget gate $\mathbf{f}_t$, input gate $\mathbf{i}_t$, and output gate $\mathbf{o}_t$ at each timestep and layer as the LSTM processes agreement sentences. Each activation is labeled with the grammatical number of the main verb, and separate diagnostic classifiers are trained for each component and layer, yielding 10 component-layer combinations in a 2-layer LSTM. The diagnostic classifiers are then evaluated on held-out data (Giulianelli et al., 2018).

The interpretive logic is straightforward but technically significant. If a diagnostic classifier can recover verb number from a given activation, then that activation contains information predictive of agreement. By training and testing classifiers at different timesteps, layers, and components, the method exposes the temporal and spatial organization of that information. In this sense, the classifiers function as decoders of the model’s latent agreement state.

The paper treats this as more than a binary “information present or absent” question. The cross-timestep and cross-component evaluations examine representational geometry: whether the same code is reused over time, whether the representation changes format, and whether different internal variables carry similar or dissimilar encodings. This is why the work emphasizes not only probing accuracy but also generalization matrices across timesteps and components.

A concise summary of the probed variables is useful:

LSTM component	Symbol	Role in the analysis
Hidden state	$\mathbf{h}_t$	Decoded for agreement information
Memory cell	$\mathbf{c}_t$	Decoded for agreement information
Forget gate	$\mathbf{f}_t$	Compared as a weaker carrier
Input gate	$\mathbf{i}_t$	Compared as a weaker carrier
Output gate	$\mathbf{o}_t$	Compared as a weaker carrier

3. Temporal dynamics of agreement encoding

A central experiment constructs a temporal generalization matrix. For each timestep, a diagnostic classifier is trained to predict verb number from the activations at that timestep and then tested on activations from other timesteps. The resulting matrix shows how stable or shifting the representation is across sentence processing. The main finding is that diagnostic accuracy is high at or just after the subject and again at the verb, indicating that number information is explicitly represented at those points. In the intermediate region, the information remains present but is less explicit, and classifiers trained on subject or verb timesteps generalize poorly to these intermediate timesteps, indicating a dynamic encoding that changes over time (Giulianelli et al., 2018).

This result supports a view of agreement tracking as an active computational process rather than passive retention of a fixed feature vector. The representation appears to become more explicit when number is introduced and when it must be expressed, while intermediate timesteps use a different code. The paper characterizes this as a dynamic encoding whose format changes as the network processes intervening material.

The “wrong” set sharpens the interpretation. In sentences misclassified by the LLM, number information is degraded or absent after attractors are processed. This localizes many agreement failures to the period following the attractor rather than to the verb prediction itself. A common misconception is that an agreement error originates at the verb token because that is where the error becomes visible. The diagnostic evidence instead indicates earlier representational corruption, especially in the presence of intervening nouns with conflicting number.

The synthetic manipulations of context size and attractor structure reinforce this picture. Sentences with attractors show more degraded diagnostic-classifier performance, particularly post-attractor, matching the point at which the LLM is likely to err. This suggests that attraction effects in the model correspond to a representational failure during dependency maintenance, not merely a bad final readout.

4. Spatial localization across layers and components

The spatial generalization analysis examines where number agreement information is represented inside the LSTM at a critical timestep, specifically post-attractor and pre-verb. Diagnostic classifiers are trained and tested across different components, producing a spatial generalization matrix. The main result is that number agreement information is best represented, and most similarly represented, in the cell state $\mathbf{c}_t$0 and hidden activation $\mathbf{c}_t$1 of the second LSTM layer, while the representation is much weaker in the gates (Giulianelli et al., 2018).

This finding is consistent with the LSTM architecture’s functional division. The cell state and hidden state appear to carry the most stable and behaviorally relevant agreement information, whereas the gate activations are less directly informative as end-state stores of number. The paper does not reduce the mechanism to a single neuron or a single timestep; rather, it identifies a structured locus in the upper layer’s state variables.

The layer asymmetry is also notable. Higher diagnostic accuracies in the upper LSTM layer suggest that agreement information becomes more accessible or more abstractly organized there. A plausible implication is that the second layer consolidates syntactic information that is more diffusely processed earlier, though the paper’s concrete claim is limited to the observed decoding pattern.

These spatial findings complement the temporal results. Taken together, they indicate that agreement tracking is both temporally dynamic and spatially selective: the code changes over time, but its most reliable locus lies in a particular subset of upper-layer activations.

5. Causal intervention and performance improvement

The paper does not stop at correlational probing. It uses the diagnostic classifiers to intervene on the LLM’s computation. At the point identified as crucial for agreement errors, usually after the subject is presented, the hidden and cell state activations are modified. The diagnostic classifier predicts the current verb number from the activation; if the prediction is incorrect, a loss is computed between the classifier prediction and the ground-truth label, the gradient of that loss with respect to the activation is taken, and the activation is adjusted with a delta rule using a small step size $\mathbf{c}_t$2. Sentence processing then continues from this nudged activation (Giulianelli et al., 2018).

The intervention results provide the paper’s causal argument. If the probed representation were merely an epiphenomenal correlate, manipulating it would not systematically improve agreement behavior. Instead, the model’s agreement accuracy rises substantially after intervention:

Sentence type	Before intervention	After intervention
Original	78.1%	85.4%
Nonce	70.7%	75.6%

The paper further reports minimal change in overall language modeling perplexity, which indicates that the intervention is targeted rather than globally disruptive. It also reports that the beneficial effect persists, and sometimes strengthens, through subsequent timesteps, affecting not only the directly modified component but also internal gate values. This persistence is important because it suggests the intervention changes the model’s ongoing computation rather than only the local readout at a single moment.

An important methodological implication follows. Probing is often criticized as merely post hoc. Here, the intervention establishes a stronger link between decoded agreement information and functionally relevant internal state. The paper therefore positions diagnostic classifiers not only as interpretability tools but also as instruments for model repair.

6. Interpretation, limitations, and later context

The study argues for several broader conclusions. First, diagnostic classifiers provide a fine-grained view of the locality and temporal dynamics of linguistic information in neural models. Second, agreement errors often arise from earlier mis-encoding of the subject’s number, especially after attractors, rather than at the final prediction step. Third, probing knowledge can be inverted into a control mechanism that nudges the model toward correct behavior (Giulianelli et al., 2018).

These claims place the work at an important point in the development of mechanistic analysis of LLMs. It treats syntactic agreement as an internal-state problem and uses temporally resolved decoding to show that the encoding is dynamic rather than static. This suggests a distinction between “surface-level” codes at the subject and verb and more abstract or transformed codes in intermediate positions.

At the same time, later work clarifies the scope of such probing. A contrastive analysis of transformer models on French subject–verb and object–past participle agreement found that probing alone showed similar locally distributed number representations across both agreement types, whereas causal counterfactual interventions were required to reveal differences consistent with theoretical linguistics (Li et al., 2022). This later result does not contradict the LSTM study; rather, it suggests that diagnostic decoding is most informative when combined with causal tests of the kind already introduced by the intervention analysis in the original paper.

Related work on internal consistency in chain-of-thought reasoning makes a similar move from external behavior to internal signals. It defines internal consistency as agreement between latent predictions decoded from intermediate layers and the final output, and uses that agreement to weight reasoning paths during inference (Xie et al., 2024). This suggests a broader methodological lineage in which internal decodability is treated not only as evidence of representation but also as a usable confidence or control signal.

Within that lineage, the 2018 diagnostic-classifier study remains distinctive for showing all three stages in one framework: decoding, error localization, and state intervention. Its central contribution is not simply that number agreement can be found somewhere inside an LSTM, but that one can identify when and where the representation changes, pinpoint where it is corrupted in failure cases, and alter the model’s computation in ways that measurably improve agreement performance (Giulianelli et al., 2018).