Relational Preference Encoding in Looped Transformer Internal States

Abstract: We investigate how looped transformers encode human preference in their internal iteration states. Using Ouro-2.6B-Thinking, a 2.6B-parameter looped transformer with iterative refinement, we extract hidden states from each loop iteration and train lightweight evaluator heads (~5M parameters) to predict human preference on the Anthropic HH-RLHF dataset. Our pairwise evaluator achieves 95.2% test accuracy on 8,552 unseen examples, surpassing a full-batch L-BFGS probe (84.5%) while the base model remains completely frozen. Our central finding is that loop states encode preference predominantly relationally: a linear probe on pairwise differences achieves 84.5%, the best nonlinear independent evaluator reaches only 65% test accuracy, and linear independent classification scores 21.75%, below chance and with inverted polarity. Interpreted precisely, the evaluator functions as a model-internal consistency probe, measuring how stably Ouro's own learned value system organizes its representations rather than how well it predicts noisy human annotations. We also document a systematic architecture search that established a genuine 70% ceiling for independent scoring, and show how the 50% argument-swap protocol required to prevent degenerate pairwise solutions deflated pairwise training metrics by about 31 points at peak, creating the false appearance that pairwise and pointwise evaluators shared the same ceiling. Finally, we show that a cosine learning-rate dead zone at epoch 2 accidentally acted as early stopping, preserving the generalization peak before overfitting degraded test accuracy from 95.2% to 62.4% by epoch 5. Cross-epoch flip-test analysis shows that antisymmetry correlation remains stable while strict sign-flip rate mainly tracks scorer bias. We propose the flip test as a mandatory diagnostic for pairwise preference evaluators.

• The paper’s main contribution is showing that preference information is encoded relationally within looped transformer hidden states, with pairwise evaluators reaching 95.2% accuracy compared to 65% for independent methods.
• It employs a methodology that extracts multiple loop iteration states via forward hooks and evaluates various architectures, including linear, GRU, and attention pooling, without modifying the frozen base model.
• The study highlights significant implications for modular alignment and interpretability, advocating the use of flip tests as a critical diagnostic tool for uncovering degenerate solutions.

Relational Preference Encoding in Looped Transformer Internal States: An Expert Analysis

Introduction and Context

This paper presents a comprehensive analysis of how looped transformers, specifically Ouro-2.6B-Thinking, encode human preference signals in their iterative internal states. The key innovation is probing the relational structure of these representations without modifying the frozen base model. Unlike end-to-end RLHF reward models that co-adapt their heads and base, this approach evaluates whether preference information is inherently encoded and extractable.

A lightweight ($\sim$5M parameter) pairwise evaluator is trained on Anthropic's HH-RLHF dataset. The evaluator receives as input the sequence of intermediate hidden states produced by the frozen transformer over several loop iterations. The central result is that preference is overwhelmingly encoded relationally: pairwise access leads to 95.2% test accuracy, while independent (pointwise) evaluators saturate at 65%. This quantifies, under experimental conditions, the structural disposition of looped transformer representations for preference information.

Methodology

Loop States and Feature Extraction

Ouro-2.6B-Thinking is a looped transformer model with open weights and accessible per-iteration hidden states. For each input, up to four loop iteration states are extracted as 2048-dimensional hidden vectors per token position. Feature extraction is implemented using forward hooks, capturing states for both the “chosen” and “rejected” responses from HH-RLHF pairs. Attention is paid to left-padding and adaptive early-exit configuration to ensure that the most informative response tokens are prioritized during pooling.

A critical methodological detail is the complete freezing of the 2.6B base model. This isolates representational capacity from the influence of weight updates, focusing evaluation strictly on the information geometry of the native representations.

Evaluator Architectures and Diagnostic Tools

A thorough controlled architecture search encompasses nine main designs (MLP, GRU, attention-pooling, linear, calibrations), exploring both pointwise and pairwise evaluation paradigms. The crucial insight arises from diagnostic probes:

• Linear pairwise probe: Logistic regression on pairwise differences achieves 84.5% accuracy, establishing a robust lower bound on linearly accessible relational signal.
• Independent classifier: Logistic regression on single responses yields only 21.75%, with polarity inversion on flipping.
• Structural confound analysis: Length, norm, and activation ratios do not separate classes, excluding surface-level shortcuts and confirming a geometric origin for the accessible preference.

Pairwise evaluators apply shared attention pooling and a directional loss with antisymmetry enforcement (50% input swap with sign-flipped target), and a GRU to aggregate temporal dynamics across loop iterations.

Key Empirical Results

Relational vs. Absolute Preference Accessibility

The results decisively show that relational (pairwise) access exposes dramatically more preference signal than independent (absolute) access. The nonlinear pairwise evaluator surpasses not only linear probes but also all tested independent evaluators.

Figure 1: Preference prediction accuracy by access pattern and method. The gap between pairwise and independent access is consistent across linear and nonlinear classifiers. The nonlinear pairwise evaluator exceeds the L-BFGS probe ceiling, illustrating the limits of linear probes and underlining the richness of relational signal.

The consistently large performance gap (of at least 19 points) across classifiers, optimizers, and architectures indicates that the model's internal geometry encodes preference differentials as relational rather than absolute magnitudes. Notably, the pairwise evaluator’s 95.2% test accuracy is 10.7 points above the L-BFGS probe ceiling and ~30+ points above any independent architecture, which points to strong nonlinear and relational structure.

Overfitting and Metric Deflation

An important methodological contribution is the detailed dissection of deflated training metrics due to the required antisymmetry-enforcing swap protocol. The training accuracy metric for pairwise evaluators is depressed by up to 31 points at test-time peak accuracy, and inversely correlated with actual test performance.

Figure 3: Swap-protocol metric inversion across five epochs. The deflated training metric monotonically increases, while actual test accuracy peaks at LR dead zone and collapses with overfitting. Highlighting the risk of misinterpreting swap-deflated training metrics when selecting checkpoints.

Crucially, the cosine annealing learning rate schedule’s “dead zone” at epoch 2 acts as an implicit early stopping, preserving peak generalization before overfitting. The finding that test accuracy peaks while the training metric increases with overfitting is a strong caution against trusting swap-deflated metrics for early stopping.

Pairwise Degeneracy and the Flip Test

A degenerate solution—constant output with 100% “accuracy” but no preference information—is identified and diagnosed. This is only detectable by the flip test, which checks antisymmetry by swapping arguments and verifying sign inversion of the output score.

Figure 2: Cross-epoch flip test analysis. Antisymmetry correlation ($\rho$) is a stable indicator, while strict sign flip rate merely tracks model bias and not learned order sensitivity. Flip test correlation is thus the recommended diagnostic for pairwise evaluators.

The authors recommend the flip test as a mandatory diagnostic in all pairwise evaluators, since conventional accuracy and even loss metrics can be uninformative in the presence of such degenerate solutions.

Architectural Findings

Attention pooling with low-rank projection (128-dim) outperforms mean pooling and resists memorization, supporting the importance of controlled bottlenecked feature construction. End-to-end GRU-based models utilizing sequence data across iteration steps provide modest but real gains. Additionally, the best architectures exhibit robust performance with a relatively small parameter count compared to the base model (~4–5M parameters vs. 2.6B frozen).

An important negative result is that independent evaluators—pointwise, even with expressivity—remain substantially below pairwise performance, evidencing that the model does not store preference as an absolute attribute in individual instance representations.

Implications and Theoretical Insights

Preference Encoding in Looped Transformers

Ouro-2.6B-Thinking’s iterative refinement architecture encodes preference predominantly in a relational manner. This is empirically validated across linear and nonlinear, trainable and probe-based evaluators. The high antisymmetry correlation underlines a geometric property: preference is captured by the difference between internal representations of response pairs, not by any axis-aligned or scalar attribute.

Alignment and Interpretability

The ability to extract a high-accuracy, interpretable preference signal from frozen representations—without retraining—opens up modular approaches for alignment monitoring and auditing. Lightweight external evaluators can provide real-time, separately trainable access to model “value systems,” decoupled from the main generative process.

This is reinforced by contemporaneous interpretability work (e.g., Anthropic’s reported emotion concept vectors causally influencing Claude’s behavior), suggesting generalizability of such geometric representation phenomena across architectures.

Practical and Theoretical Consequences

• Modular Alignment: Lightweight, external modules (“amygdala”/“basal ganglia” analogues) can monitor, interpret, and potentially steer preferences in large LMs without interfering with the main inference process.
• Diagnostic Rigor: The metric inversion phenomenon during pairwise training with swap protocols demonstrates the necessity for robust, held-out evaluation and antisymmetry checks. Swap-deflated metrics are poor bases for early stopping or model selection.
• Model Generalization: Experimental architecture and protocol choices (e.g., left-padding, attention pooling, learning rate schedules) can substantially affect data utilization and signal detection.
• Future Directions: Extending this approach to joint training (e.g., via LoRA), feedback integration (steering/gating mechanisms), epistemic reasoning, and cross-architecture generalization remains as critical open avenues for modular alignment architectures.

Limitations

Experiments are limited to Ouro-2.6B-Thinking, as it is the only public looped transformer with the requisite introspection capabilities. All claims about relational preference accessibility are empirical and subject to generalization only under broader architectural replication. Dataset noise, annotator disagreement, and scorer bias are discussed candidly as limiting factors for interpreting performance ceilings.

Conclusion

This study establishes that looped transformer internal states encode comparative preference information in a relational, not absolute, format. A simple pairwise evaluator, operating solely on frozen hidden states, can extract highly accurate preference signals—well above strong linear baselines, independent evaluators, and standard RLHF reward predictors. The rigorous methodological analysis, including diagnostic tools like the flip test and attention to training metric pitfalls, provides a template for future work both in alignment research and modular interpretability. These findings substantiate and extend the emerging paradigm whereby preference, value, and abstract behavioral determinants are functionally encoded and extractable within large-scale model internal dynamics.