Representational Effective Theory (RET)

• RET is a framework that abstracts large language model behavior by focusing on learned macrostates instead of detailed microstates.
• It employs a BYOL/JEPA-style training objective with a discrete dynamical system to predict and steer LLM hidden-state dynamics effectively.
• RET bridges neural network interpretability with physical computing theories, supporting applications in explanation, prediction, and intervention.

Representational Effective Theory (RET) is a framework proposed for describing LLM computation in terms of learned macrostates rather than microscopic activations, with the explicit aim of supporting interpretation, prediction, and intervention (Ustaomeroglu et al., 10 May 2026). In the current literature, the term is used most directly for this macrostate-based program in LLM interpretability, but closely related work extends both backward and outward: Abstraction/Representation theory makes the physical–abstract interface itself explicit and rigorous for physical computing (Horsman, 2015), and several later papers can be read as proto-RET or RET-adjacent accounts of representation engineering, causal efficacy, representational change, and representational comparison (Tian et al., 25 Mar 2025).

1. Emergence and scope of the concept

RET was introduced explicitly as “a framework for describing LLM computation in terms of learned macrostates rather than microscopic details,” with the central claim that a useful description of LLM behavior need not be a fully detailed mechanistic account of every neuron, head, or pathway (Ustaomeroglu et al., 10 May 2026). In that formulation, the motivating contrast is between a microstate, consisting of detailed internal activations, and a macrostate, a lower-dimensional learned summary that preserves the structure relevant for explanation and prediction. The proposal is therefore an “effective-theory” perspective in the sense used in physics and complex systems: explanation is sought at the level of collective variables rather than exhaustive microscopic enumeration.

The literature surrounding RET is broader than the term itself. “Why Representation Engineering Works” does not introduce RET, but it explicitly offers a theoretical account in which hidden-state dynamics in transformer attention are stabilized by a principal eigenvector and differentiated by subdominant modes, and it is presented as a plausible foundation for what one might call a RET for multimodal models (Tian et al., 25 Mar 2025). “Abstraction/Representation Theory for Heterotic Physical Computing” is older and concerns physical computation rather than neural networks, but it is repeatedly described as a direct precursor, neighbor, or partial foundation for a representational theory of computation because it formalizes the bridge between physical systems and abstract computational states (Horsman, 2015).

This broader usage suggests that RET is not a single closed doctrine. It names, at minimum, a family of projects that treat representation itself as the correct level at which to formulate effectiveness, validity, and intervention. In the LLM setting, that family is centered on learned macrovariables. In the physical-computing setting, it is centered on representation relations and commuting diagrams. In work on representational change, it appears as the search for principled transitions between representational regimes rather than mere parameter updates.

2. Macrostate formalism in the LLM formulation

The formal core of RET in the explicit 2026 formulation begins with a discrete-time dynamical system with microstate space $X$, macrostate space $Z$, microstate $x_t \in X$, and macrostate $z_t \in Z$, related by a coarse-graining map

$z_t = f(x_t), \qquad Z := f(X).$

A good effective theory is required to satisfy three desiderata: abstraction from microscopic detail, approximate closure, and practical relevance (Ustaomeroglu et al., 10 May 2026). Approximate closure is expressed as

$I(x_{t+1}; z_t, x_t) \approx I(x_{t+1}; z_t),$

with the accompanying transition-map intuition

$z_{t+1} = T(z_t) + \eta_t.$

In the RET instantiation for LLMs, the microstate is a hidden-state prefix from one fixed layer,

$$x_t \equiv h_{0:t}, \qquad x_{t+1} \equiv h_{0:t+1},$$

and the learned macrostate is

$z_t = f_\theta(x_t).$

The reported default macrostate dimension is

$d_z = 128.$

RET then learns a predictor

$Z$0

together with a target encoder $Z$1 maintained by exponential moving average, so that

$Z$2

The central BYOL/JEPA-style objective is

$Z$3

The target encoder is updated by

$Z$4

with reported EMA momentum $Z$5, and the paper explicitly states that no additional collapse regularizer is used (Ustaomeroglu et al., 10 May 2026).

The architecture is also specified. The encoder is a single-layer causal transformer with RoPE, pre-norm, residual connections, feed-forward inner dimension $Z$6, GELU activations, a final linear projection $Z$7, and a final layer norm. The predictor is a 2-layer MLP. The framework is trained post hoc on hidden-state trajectories from frozen backbones, including GPT-OSS-20B at layer 11, Pythia-160M at layer 6, Qwen2.5-14B-Instruct at layer 23, and Qwen3.5-35B-A3B at layer 19 (Ustaomeroglu et al., 10 May 2026).

3. Empirical program: interpretation, prediction, and steering

RET is evaluated empirically as a representation-learning framework for temporally consistent, semantically organized, and intervention-ready macrostates (Ustaomeroglu et al., 10 May 2026). The training and evaluation corpora are task-specific: NuminaMath for reasoning trajectories and steering, TinyStories and The Pile for temporal consistency and closure-type experiments, MMLU for semantic-versus-syntactic organization, and SYCON-Bench plus a large augmentation for sycophancy prediction.

A central quantitative result is the self-prediction test for approximate closure. Using held-out

$Z$8

RET achieves the highest self-prediction $Z$9 on every model–dataset pair reported, with RET ceilings $x_t \in X$0, $x_t \in X$1, $x_t \in X$2, and $x_t \in X$3, compared with matched-baseline ceilings $x_t \in X$4, $x_t \in X$5, $x_t \in X$6, and $x_t \in X$7 (Ustaomeroglu et al., 10 May 2026). This is used to argue that the learned macrostates have more self-contained dynamics than raw hidden states, PCA, or SAE features.

The interpretability program proceeds by clustering learned macrostates. After global mean subtraction

$x_t \in X$8

and unit normalization

$x_t \in X$9

the paper applies $z_t \in Z$0-means with $z_t \in Z$1 clusters and then agglomerative merging into $z_t \in Z$2 macro-groups. Cluster assignment uses cosine similarity,

$z_t \in Z$3

These clustered trajectories are then summarized as operational “mental-state” phases such as problem setup, symbolic transformation, verification, case analysis, and final exposition. On 100 held-out NuminaMath responses, the reported Mental-state Interpretability Score is $z_t \in Z$4 for RET versus $z_t \in Z$5 for SAE (Ustaomeroglu et al., 10 May 2026).

RET also includes a supervised extension for early sycophancy prediction. The augmented SYCON-Bench dataset contains about $z_t \in Z$6 false-presupposition conversations and $z_t \in Z$7 debate conversations, roughly $z_t \in Z$8 total samples. The supervised objective adds an auxiliary head $z_t \in Z$9: $z_t = f(x_t), \qquad Z := f(X).$0 with $z_t = f(x_t), \qquad Z := f(X).$1. The reported result is that supervised RET performs best in both false-presupposition and debate settings, outperforming hidden-state baselines and a learned Transformer Block probe (Ustaomeroglu et al., 10 May 2026).

The intervention mechanism treats RET states as local causal handles. With target displacement $z_t = f(x_t), \qquad Z := f(X).$2 in macrostate space and normalized macrostate $z_t = f(x_t), \qquad Z := f(X).$3, the steering objective is

$z_t = f(x_t), \qquad Z := f(X).$4

The hidden state is then updated by $z_t = f(x_t), \qquad Z := f(X).$5 normalized gradient steps,

$z_t = f(x_t), \qquad Z := f(X).$6

with reported hyperparameters $z_t = f(x_t), \qquad Z := f(X).$7 and $z_t = f(x_t), \qquad Z := f(X).$8. On 33 held-out prompts, target-group occupancy on steered tokens is reported as $z_t = f(x_t), \qquad Z := f(X).$9–$I(x_{t+1}; z_t, x_t) \approx I(x_{t+1}; z_t),$0, and the expected behavioral signature appears in $I(x_{t+1}; z_t, x_t) \approx I(x_{t+1}; z_t),$1 prompts $I(x_{t+1}; z_t, x_t) \approx I(x_{t+1}; z_t),$2, compared with $I(x_{t+1}; z_t, x_t) \approx I(x_{t+1}; z_t),$3 $I(x_{t+1}; z_t, x_t) \approx I(x_{t+1}; z_t),$4 for both SAE steering baselines (Ustaomeroglu et al., 10 May 2026).

4. Effectiveness, causal efficacy, and the decodability problem

A central controversy in RET-adjacent work is whether a represented feature is merely present or genuinely effective. “Representation Without Control” sharpens this issue by distinguishing three evidential levels: behavioral sensitivity, latently readable representation, and causal control / behavioral effectiveness (Walsh et al., 24 May 2026). The paper’s proposed hierarchy is

$I(x_{t+1}; z_t, x_t) \approx I(x_{t+1}; z_t),$5

Its empirical case concerns the realization effect in behavioral economics. Gemma 3 4B shows prompt-sensitive behavior, and its residual stream contains a linearly decodable realization-status signal strongest at layer 18. The train-only layer-18 direction, built from 756 matched paper/realized pairs, generalizes to held-out prompts: correct-direction scores are $I(x_{t+1}; z_t, x_t) \approx I(x_{t+1}; z_t),$6 on direction_train, $I(x_{t+1}; z_t, x_t) \approx I(x_{t+1}; z_t),$7 on direction_val, $I(x_{t+1}; z_t, x_t) \approx I(x_{t+1}; z_t),$8 on behavior_eval, $I(x_{t+1}; z_t, x_t) \approx I(x_{t+1}; z_t),$9 on heldout_readout, and $z_{t+1} = T(z_t) + \eta_t.$0 on heldout_behavior_eval (Walsh et al., 24 May 2026). This establishes representational presence.

The same paper argues that presence is not effectiveness. At the raw prompt level, one standard deviation increase in projection predicts $z_{t+1} = T(z_t) + \eta_t.$1 CHF larger wager and $z_{t+1} = T(z_t) + \eta_t.$2 risk profile, but after controls for pair role, domain, outcome valence, amount bucket, and prompt source, the effects collapse to $z_{t+1} = T(z_t) + \eta_t.$3 CHF $z_{t+1} = T(z_t) + \eta_t.$4 and $z_{t+1} = T(z_t) + \eta_t.$5 $z_{t+1} = T(z_t) + \eta_t.$6. Within matched pairs, Pearson correlations are $z_{t+1} = T(z_t) + \eta_t.$7 for wager deltas and $z_{t+1} = T(z_t) + \eta_t.$8 for risk deltas (Walsh et al., 24 May 2026). The decoded direction is therefore not robustly linked to downstream policy use.

The intervention test is stronger. Steering at layer 18 with scales $z_{t+1} = T(z_t) + \eta_t.$9 on 648 prompts uses the normalized mean-difference direction

$$x_t \equiv h_{0:t}, \qquad x_{t+1} \equiv h_{0:t+1},$$0

The reported matched deltas relative to baseline are small, medians are zero throughout, and the negative sign-symmetry run also fails: at $$x_t \equiv h_{0:t}, \qquad x_{t+1} \equiv h_{0:t+1},$$1, mean wager delta is $$x_t \equiv h_{0:t}, \qquad x_{t+1} \equiv h_{0:t+1},$$2, mean risk delta is $$x_t \equiv h_{0:t}, \qquad x_{t+1} \equiv h_{0:t+1},$$3, with zero medians (Walsh et al., 24 May 2026). The paper’s conclusion is explicit: successful latent readout is insufficient evidence that a model behaviorally relies on a representation during downstream decision-making.

This result constrains RET directly. Any RET that equates interpretability with linear decodability is too weak. The stronger standard suggested by the literature is that a macrovariable becomes “effective” only when it supports not just readout, but reliable downstream use and intervention. That requirement also clarifies the status of spectral RepE accounts: “Why Representation Engineering Works” provides a theory of stability and readout, but its evidence is described as mainly correlational and structural rather than fully causal (Tian et al., 25 Mar 2025).

5. Precursors in physical computing and representational systems

RET’s deepest precursor in the supplied literature is Abstraction/Representation theory. AR theory insists on a strict separation between a physical domain $$x_t \equiv h_{0:t}, \qquad x_{t+1} \equiv h_{0:t+1},$$4 and an abstract domain $$x_t \equiv h_{0:t}, \qquad x_{t+1} \equiv h_{0:t+1},$$5, linked by a directed representation relation

$$x_t \equiv h_{0:t}, \qquad x_{t+1} \equiv h_{0:t+1},$$6

If a physical object $$x_t \equiv h_{0:t}, \qquad x_{t+1} \equiv h_{0:t+1},$$7 is represented by $$x_t \equiv h_{0:t}, \qquad x_{t+1} \equiv h_{0:t+1},$$8, the basic unit is the representational triple

$$x_t \equiv h_{0:t}, \qquad x_{t+1} \equiv h_{0:t+1},$$9

With abstract evolution $z_t = f_\theta(x_t).$0 and physical evolution $z_t = f_\theta(x_t).$1, computation requires a commuting diagram satisfying

$z_t = f_\theta(x_t).$2

AR theory then adds instantiation, encoding, and the compute cycle

$z_t = f_\theta(x_t).$3

together with the claim that a device cannot be used as a computer until its theory is well understood, good, and valid (Horsman, 2015). This is a representational criterion for effectiveness: not every physical process counts as computation, and heterotic computation is characterized by a non-decomposable joint representational triple rather than by speedup or Turing power alone.

A different but complementary formalism is Representational Systems Theory, which encodes representational systems through a grammatical space, an entailment space, and an identification space, all unified as construction spaces (Raggi et al., 2022). RST is explicitly designed to support structural transformation and representation selection based on properties such as relative cognitive effectiveness, observability, and structural complexity. Its operational implementation, Oruga, instantiates type systems, constructor specifications, constructions, transfer schemas, and a structure-transfer engine that applies transfer schemas backwards from a goal (Raggi et al., 4 Sep 2025). In this line of work, representational effectiveness is not a hidden scalar but a property of typed transformability across heterogeneous systems.

Taken together, AR theory and RST show two different lower-level meanings of RET. AR theory formalizes effectiveness as representationally grounded physical computation. RST formalizes effectiveness as structural transformability and property-sensitive representation selection. The explicit LLM version of RET does not subsume either framework, but it occupies an adjacent position: it learns macrostates as effective variables rather than stipulating them a priori.

6. Representational change, comparison, and emergence

Several papers extend RET-like thinking beyond macrostate learning toward representational transition. “Counterfactual Basis Extension and Representational Geometry” models conceptual growth as admissible basis extension under a Minimum Description Length objective

$z_t = f_\theta(x_t).$4

With current conceptual subspace $z_t = f_\theta(x_t).$5, residuals

$z_t = f_\theta(x_t).$6

and residual span

$z_t = f_\theta(x_t).$7

the main theorem states that any MDL-accepted extension can be chosen so that its novel directions lie in $z_t = f_\theta(x_t).$8, while nontrivial novelty in $z_t = f_\theta(x_t).$9 strictly increases description length and is therefore rejected (Amornbunchornvej, 21 Dec 2025). This yields a conservative theory of representational expansion: effective new variables are residual-supported, low-rank, and complexity-penalized.

“Bootstrap Theory of Representational Emergence” addresses the same problem at a more conceptual level. Its key notion is explanatory insufficiency, defined as the situation in which a representation remains capable of describing a phenomenon but no longer accounts satisfactorily for certain observations, relations, transformations, or organizational properties. The proposed five-stage sequence is stabilized observation, anomaly detection, recognition of explanatory insufficiency, representational emergence, and provisional stabilization (Raynal et al., 5 Jun 2026). This can be read as a theory of representational regime change: a representation has an explanatory domain, and anomalies at its boundary trigger emergence of a new one.

At the comparative end, “Representational Difference Explanations” offers a method for identifying sets of inputs that are grouped together by one learned representation but not by another. Its core object is a locally biased difference matrix

$d_z = 128.$0

followed by an affinity matrix

$d_z = 128.$1

spectral clustering, and exemplar selection (Kondapaneni et al., 29 May 2025). Although RDX is not itself a RET, it functions as a comparative operator for discovering candidate mesoscopic variables.

“Why Representation Engineering Works” occupies an intermediate position. It proposes that high-level concept directions in VLMs persist because attention matrices are row-stochastic, possess a dominant eigenvalue $d_z = 128.$2, and propagate activity along a principal eigenvector, while shrinking spectral gap allows subdominant directions to encode concept distinctions (Tian et al., 25 Mar 2025). This is explicitly described as a structured theoretical framework rather than merely a descriptive probing method, but it stops short of universality claims, renormalization-like structure, or causal guarantees.

7. Philosophical and mathematical context

RET is also situated within a wider representational philosophy of theory. “Representational Realism, Closed Theories and the Quantum to Classical Limit” argues that physical theories must provide “a physical representation of reality in terms of a network of physical concepts which coherently relates to the mathematical formalism of the theory and allows to articulate and make predictions of definite phenomena” (Ronde, 2016). This position is explicitly pluralist and anti-reductionist: theories may be autonomous representational schemes rather than mere approximations awaiting reduction to a single privileged vocabulary. For RET, that suggests that an effective theory is not merely a calculational surrogate but may be representationally substantive.

A logical analogue appears in “Logic and theory of representation,” which treats representation as the capacity of some entities to present some contents and identifies three adequacy conditions for representational systems: completeness, faithfulness, and coherence (Plagnol, 2023). Those notions resonate strongly with RET. Completeness corresponds to coverage of the target domain, faithfulness to correctness of recovered content, and coherence to integrability of represented fragments. The paper’s stronger thesis—that logic can be considered the abstract theory of representation—locates representational structure prior to formal inference rather than downstream from it.

A different mathematical perspective comes from the descriptive theory of represented spaces. There, descriptive complexity is itself representation-relative: a represented space is a pair $d_z = 128.$3, and pointclasses are generated by computable endofunctors acting on representations rather than by external set-theoretic stipulation (Pauly, 2014). In that setting, $d_z = 128.$4-open sets are

$d_z = 128.$5

and $d_z = 128.$6-measurability is defined through uniform preimages of opens. The broader implication is that what counts as open, measurable, or continuous can be altered systematically by changing the representation. That is a strong abstract counterpart to RET’s central intuition that effective descriptive regimes are representation-dependent.

Across these philosophical and mathematical neighbors, a common structure emerges. Representation is not a superficial encoding layer added after the fact. It determines what counts as a state, what counts as a valid transition, what can be predicted or controlled, and when a new descriptive regime becomes necessary. In that sense, RET is best understood not as a settled doctrine but as a growing research program: one that seeks high-level, dynamically meaningful variables that are not only readable, but valid, predictive, and interventionally effective.