Residual-Stream Gauge

• Residual-stream gauge is a family of analytical techniques that measures and coordinates the residual activations across layers in deep networks.
• It reveals critical insights such as state sufficiency, conflict detection, and variance concentration using probe-based, causal, and geometric methods.
• These gauges are applied in transformer and vision models to diagnose performance, enhance memory efficiency, and guide architectural tuning.

to=arxiv_search.search 北京赛车开奖json {"^^^^^^^^14query14 stream\"^^^^^^^^14 gauge OR \14^^^^^^^^"Residual Stream Is All You Need\"^^^^^^^^14 OR SemRF residual-stream dynamics)14^^^^^^^^", "^^^^^^^^14max_results14 ^^^^14all:(\14query14^^^^, "^^^^^^^^14sort_by14 "^^^^^^^^14submittedDate14 "^^^^^^^^14sort_order14 "^^^^^^^^14descending14 pc蛋蛋 to=arxiv_search.search 彩神争霸苹果json {"^^^^^^^^14query14 Residual Stream Is All You Need\" OR ^^^^^^^^14all:(\14query14 OR ^^^^^^^^14all:(\14query14 Stream Analysis with Multi-Layer SAEs\" OR ^^^^^^^^14all:(\14query14 the Residual Stream of LLMs Under Knowledge Conflicts\"", "^^^^^^^^14max_results14 ^^^^14all:(\14query14^^^^, "^^^^^^^^14sort_by14 "^^^^^^^^14submittedDate14 "^^^^^^^^14sort_order14 "^^^^^^^^14descending14 to=arxiv_search.search 天天中彩票足球json {"^^^^^^^^14query14 Stream Is All You Need\" (&&&^^^^14query14^^^^&&&)", "^^^^^^^^14max_results14 ^^^^14sort_by14^^^^, "^^^^^^^^14sort_by14 "relevance", "^^^^^^^^14sort_order14 "^^^^^^^^14descending14 Residual-stream gauge denotes a family of analytical constructions that measure, probe, or coordinate the residual stream of a deep network. In current usage, the term is not singular: it has been applied to exact-state analyses in autoregressive transformers, probe-based detectors of knowledge conflict, PCA-based diagnostics of representational collapse, layer-activation distributions for sparse autoencoders, causal ablation-and-rescue metrics for multi-stream transformers, anchor-based semantic coordinate systems, and per-channel diagnostics in residual vision networks (&&&^^^^14query14^^^^&&&, &&&^^^^14 gauge OR \14^^^^&&&, &&&^^^^14 OR SemRF residual-stream dynamics)14^^^^&&&, &&&^^^^14max_results14^^^^&&&, &&&^^^^14sort_by14^^^^&&&, &&&^^^^14submittedDate14^^^^&&&, &&&^^^^14sort_order14^^^^&&&, &&&^^^^14descending14^^^^&&&). Across these formulations, the common premise is that the residual stream is the privileged carrier of model state, so a gauge on that stream can expose computation, memory, geometry, and failure modes.

^^^^14all:(\14^^^^. Definition and formal scope

In transformer notation, one standard definition writes the residual-stream vector at layer PRESERVED_PLACEHOLDER_^^^^14query14^^^^ and token position PRESERVED_PLACEHOLDER_^^^^14all:(\14^^^^ as

PRESERVED_PLACEHOLDER_^^^^14 gauge OR \14^^^^

where PRESERVED_PLACEHOLDER_^^^^14 OR SemRF residual-stream dynamics)14^^^^ is the self-attention activation and PRESERVED_PLACEHOLDER_^^^^14max_results14^^^^ is the MLP activation (&&&^^^^14 gauge OR \14^^^^&&&). In a pre-norm transformer analysis of inference state, the residual stream is written as PRESERVED_PLACEHOLDER_^^^^14sort_by14^^^^ immediately before the attention and MLP sublayers, with normalized residual PRESERVED_PLACEHOLDER_^^^^14submittedDate14^^^^ (&&&^^^^14query14^^^^&&&). In both conventions, the residual stream is the hidden state propagated by residual addition and reused across depth.

Recent work uses “gauge” in several technically distinct senses:

Formulation	Object gauged	Core quantity
Exact-state transformer analysis	Sufficiency of residual state	PRESERVED_PLACEHOLDER_^^^^14sort_order14^^^^
Knowledge-conflict probing	Conflict and source selection	PRESERVED_PLACEHOLDER_^^^^14descending14^^^^
FlowLens	Residual geometry	PRESERVED_PLACEHOLDER_^^^^14query14^^^^
MLSAE	Layer occupancy of latents	PRESERVED_PLACEHOLDER_^^^^14all:(\14query14^^^^
Ablation-and-rescue	Functional contribution of streams	PRESERVED_PLACEHOLDER_^^^^14all:(\14all:(\14^^^^
SemRF	Cross-layer semantic coordinates	PRESERVED_PLACEHOLDER_^^^^14all:(\14 gauge OR \14^^^^
ResNet^^^^14all:(\14descending14^^^^ residual analysis	Skip/overwrite/mix and scale invariance	PRESERVED_PLACEHOLDER_^^^^14all:(\14 OR SemRF residual-stream dynamics)14^^^^, PRESERVED_PLACEHOLDER_^^^^14all:(\14max_results14^^^^, PRESERVED_PLACEHOLDER_^^^^14all:(\14sort_by14^^^^, PRESERVED_PLACEHOLDER_^^^^14all:(\14submittedDate14^^^^

This diversity is substantive rather than terminological noise. Some gauges are classifiers, some are causal interventions, some are spectral summaries, and some are coordinate systems. A plausible implication is that “residual-stream gauge” is best understood as a methodological category: any formal device that reads out structure from the residual pathway while treating that pathway as the locus of computation.

^^^^14 gauge OR \14^^^^. Residual stream as sufficient state in transformer inference

A strong formulation of residual-stream centrality is given by the proof that, in a standard pre-norm transformer, every key and value cache entry is a deterministic linear function of the residual stream (&&&^^^^14query14^^^^&&&). With frozen projection matrices and fixed RoPE rotation,

PRESERVED_PLACEHOLDER_^^^^14all:(\14sort_order14^^^^

Because cached and recomputed keys and values apply the same frozen maps to the same normalized residual, reconstruction is exact:

PRESERVED_PLACEHOLDER_^^^^14all:(\14descending14^^^^

with maximum absolute difference identically zero, or empirically “exactly zero (max absolute difference PRESERVED_PLACEHOLDER_^^^^14all:(\14query14^^^^)” across every full-attention layer and all tested dtypes: float^^^^14 OR SemRF residual-stream dynamics)14 gauge OR \14^^^^, float^^^^14all:(\14submittedDate14^^^^, bfloat^^^^14all:(\14submittedDate14^^^^, and mixed (&&&^^^^14query14^^^^&&&). The same paper states the information-theoretic form

PRESERVED_PLACEHOLDER_^^^^14 gauge OR \14query14^^^^

The analysis extends from reconstruction to autoregressive state. Since subsequent layers apply deterministic normalization, frozen linear maps, fixed RoPE rotations, and pointwise feed-forward sublayers, the next-token distribution is a deterministic function of the residual collection at any layer, and in particular

PRESERVED_PLACEHOLDER_^^^^14 gauge OR \14all:(\14^^^^

The paper therefore describes the residual stream as a complete Markov state and the sole information-carrying state (&&&^^^^14query14^^^^&&&). Cross-task residual activation patching at every layer yields PRESERVED_PLACEHOLDER_^^^^14 gauge OR \14 gauge OR \14^^^^ between the patched output distribution and the donor’s, and greedy decoding with no cache produces bit-identical tokens, with match PRESERVED_PLACEHOLDER_^^^^14 gauge OR \14 OR SemRF residual-stream dynamics)14^^^^ in every trial of ^^^^14 OR SemRF residual-stream dynamics)14query14^^^^ token generations (&&&^^^^14query14^^^^&&&).

The memory and latency consequences are explicit. Under standard caching, one token contributes approximately PRESERVED_PLACEHOLDER_^^^^14 gauge OR \14max_results14^^^^ of GPU memory for the example PRESERVED_PLACEHOLDER_^^^^14 gauge OR \14sort_by14^^^^, PRESERVED_PLACEHOLDER_^^^^14 gauge OR \14submittedDate14^^^^, PRESERVED_PLACEHOLDER_^^^^14 gauge OR \14sort_order14^^^^, PRESERVED_PLACEHOLDER_^^^^14 gauge OR \14descending14^^^^, whereas a single residual vector contributes approximately PRESERVED_PLACEHOLDER_^^^^14 gauge OR \14query14^^^^ for PRESERVED_PLACEHOLDER_^^^^14 OR SemRF residual-stream dynamics)14query14^^^^, PRESERVED_PLACEHOLDER_^^^^14 OR SemRF residual-stream dynamics)14all:(\14^^^^, yielding a PRESERVED_PLACEHOLDER_^^^^14 OR SemRF residual-stream dynamics)14 gauge OR \14^^^^ per-token compression (&&&^^^^14query14^^^^&&&). On Gemma ^^^^14 OR SemRF residual-stream dynamics)14^^^^-^^^^14max_results14^^^^B, KV-Direct checkpoints residual vectors rather than full KV pairs; over ^^^^14 gauge OR \14query14^^^^ conversation turns, peak memory is PRESERVED_PLACEHOLDER_^^^^14 OR SemRF residual-stream dynamics)14 OR SemRF residual-stream dynamics)14^^^^ while the standard cache grows past PRESERVED_PLACEHOLDER_^^^^14 OR SemRF residual-stream dynamics)14max_results14^^^^, and against H^^^^14 gauge OR \14^^^^O, StreamingLLM, SnapKV, TOVA, and window-only, KV-Direct maintains ^^^^14all:(\14query14query14^^^^% token match at every cache budget while all baselines degrade to ^^^^14sort_by14^^^^–^^^^14 gauge OR \14descending14^^^^% (&&&^^^^14query14^^^^&&&). The same work reports that recomputing ^^^^14sort_by14query14query14^^^^ evicted KV vectors from residuals on Apple M^^^^14 OR SemRF residual-stream dynamics)14^^^^-Max runs in only PRESERVED_PLACEHOLDER_^^^^14 OR SemRF residual-stream dynamics)14sort_by14^^^^–PRESERVED_PLACEHOLDER_^^^^14 OR SemRF residual-stream dynamics)14submittedDate14^^^^ the time needed to load ^^^^14sort_by14query14query14^^^^ cached KV vectors from memory, and its abstract states that recomputation runs up to PRESERVED_PLACEHOLDER_^^^^14 OR SemRF residual-stream dynamics)14sort_order14^^^^ faster than reading cached tensors at moderate batch sizes (&&&^^^^14query14^^^^&&&).

These results matter for the notion of a gauge because they justify treating the residual stream not merely as a convenient probe site but as the exact state on which inference depends. This suggests that a residual-stream gauge can, in principle, target the full autoregressive state rather than a lossy proxy.

^^^^14 OR SemRF residual-stream dynamics)14^^^^. Probe-based and causal gauges

One influential operational use of the term is the live residual-stream gauge for knowledge conflicts in LLMs (&&&^^^^14 gauge OR \14^^^^&&&). The method defines two binary probing tasks. Task A distinguishes question–evidence pairs with conflicting evidence PRESERVED_PLACEHOLDER_^^^^14 OR SemRF residual-stream dynamics)14descending14^^^^ from those with non-conflicting evidence PRESERVED_PLACEHOLDER_^^^^14 OR SemRF residual-stream dynamics)14query14^^^^, and Task B predicts whether, under conflicting evidence, the model will generate the context answer PRESERVED_PLACEHOLDER_^^^^14max_results14query14^^^^ or the memorized answer PRESERVED_PLACEHOLDER_^^^^14max_results14all:(\14^^^^ (&&&^^^^14 gauge OR \14^^^^&&&). For both tasks, the probe input is the residual vector PRESERVED_PLACEHOLDER_^^^^14max_results14 gauge OR \14^^^^ at the last input token, chosen to capture the state just before first-answer-token decoding. The classifier is a one-layer linear probe with sigmoid,

PRESERVED_PLACEHOLDER_^^^^14max_results14 OR SemRF residual-stream dynamics)14^^^^

trained with binary cross-entropy plus PRESERVED_PLACEHOLDER_^^^^14max_results14max_results14^^^^ regularization with PRESERVED_PLACEHOLDER_^^^^14max_results14sort_by14^^^^ (&&&^^^^14 gauge OR \14^^^^&&&).

On Llama^^^^14 OR SemRF residual-stream dynamics)14^^^^-^^^^14descending14^^^^B / NQSwap, conflict-detection performance peaks at layer PRESERVED_PLACEHOLDER_^^^^14max_results14submittedDate14^^^^, with Accuracy PRESERVED_PLACEHOLDER_^^^^14max_results14sort_order14^^^^, AUROC PRESERVED_PLACEHOLDER_^^^^14max_results14descending14^^^^, and AUPRC PRESERVED_PLACEHOLDER_^^^^14max_results14query14^^^^ (&&&^^^^14 gauge OR \14^^^^&&&). For source-selection, performance improves after PRESERVED_PLACEHOLDER_^^^^14sort_by14query14^^^^ and peaks around PRESERVED_PLACEHOLDER_^^^^14sort_by14all:(\14^^^^, with Accuracy PRESERVED_PLACEHOLDER_^^^^14sort_by14 gauge OR \14^^^^, AUROC PRESERVED_PLACEHOLDER_^^^^14sort_by14 OR SemRF residual-stream dynamics)14^^^^, and AUPRC PRESERVED_PLACEHOLDER_^^^^14sort_by14max_results14^^^^ (&&&^^^^14 gauge OR \14^^^^&&&). The paper then instantiates a live gauge by hooking PRESERVED_PLACEHOLDER_^^^^14sort_by14sort_by14^^^^ at PRESERVED_PLACEHOLDER_^^^^14sort_by14submittedDate14^^^^ and PRESERVED_PLACEHOLDER_^^^^14sort_by14sort_order14^^^^ at PRESERVED_PLACEHOLDER_^^^^14sort_by14descending14^^^^, computing

PRESERVED_PLACEHOLDER_^^^^14sort_by14query14^^^^

and thresholding them with PRESERVED_PLACEHOLDER_^^^^14submittedDate14query14^^^^ and PRESERVED_PLACEHOLDER_^^^^14submittedDate14all:(\14^^^^ (&&&^^^^14 gauge OR \14^^^^&&&). The reported example thresholds are PRESERVED_PLACEHOLDER_^^^^14submittedDate14 gauge OR \14^^^^, which yields ^^^^14query14query14^^^^% conflict-detection accuracy, and PRESERVED_PLACEHOLDER_^^^^14submittedDate14 OR SemRF residual-stream dynamics)14^^^^, which yields PRESERVED_PLACEHOLDER_^^^^14submittedDate14max_results14^^^^ accuracy in source prediction, with only a single small-matrix multiply at two layers and negligible overhead (&&&^^^^14 gauge OR \14^^^^&&&).

A distinct causal gauge appears in the analysis of manifold-constrained hyper-connections (mHC), a multi-stream transformer with PRESERVED_PLACEHOLDER_^^^^14submittedDate14sort_by14^^^^ residual streams per layer (&&&^^^^14sort_by14^^^^&&&). The architecture updates

PRESERVED_PLACEHOLDER_^^^^14submittedDate14submittedDate14^^^^

where PRESERVED_PLACEHOLDER_^^^^14submittedDate14sort_order14^^^^ is doubly-stochastic, PRESERVED_PLACEHOLDER_^^^^14submittedDate14descending14^^^^ collapses the PRESERVED_PLACEHOLDER_^^^^14submittedDate14query14^^^^ streams to a single vector, and PRESERVED_PLACEHOLDER_^^^^14sort_order14query14^^^^ redistributes the block output (&&&^^^^14sort_by14^^^^&&&). The ablation-and-rescue framework defines a do-operation that zeroes streams PRESERVED_PLACEHOLDER_^^^^14sort_order14all:(\14^^^^ and PRESERVED_PLACEHOLDER_^^^^14sort_order14 gauge OR \14^^^^ at layer PRESERVED_PLACEHOLDER_^^^^14sort_order14 OR SemRF residual-stream dynamics)14^^^^, producing PRESERVED_PLACEHOLDER_^^^^14sort_order14max_results14^^^^, and measures ablation impact by

PRESERVED_PLACEHOLDER_^^^^14sort_order14sort_by14^^^^

It then rescues one stream and defines

PRESERVED_PLACEHOLDER_^^^^14sort_order14submittedDate14^^^^

This framework is explicitly presented as a sensitive “gauge” of functional contribution (&&&^^^^14sort_by14^^^^&&&).

The causal findings differ from representational similarity alone. Middle layers show a “checkerboard” CKA pattern with high within-group CKA of PRESERVED_PLACEHOLDER_^^^^14sort_order14sort_order14^^^^ for stream groups such as PRESERVED_PLACEHOLDER_^^^^14sort_order14descending14^^^^ and PRESERVED_PLACEHOLDER_^^^^14sort_order14query14^^^^, but lower cross-group CKA of PRESERVED_PLACEHOLDER_^^^^14descending14query14^^^^ (&&&^^^^14sort_by14^^^^&&&). Yet rescue scores reveal three regimes: redundancy, asymmetry, and complementarity. Streams ^^^^14query14^^^^ and ^^^^14 gauge OR \14^^^^ each rescue PRESERVED_PLACEHOLDER_^^^^14descending14all:(\14^^^^ of the KL gap caused by ablating the other; for pair PRESERVED_PLACEHOLDER_^^^^14descending14 gauge OR \14^^^^, rescuing stream ^^^^14 OR SemRF residual-stream dynamics)14^^^^ recovers on average ^^^^14sort_order14max_results14^^^^% of the KL gap while rescuing stream ^^^^14all:(\14^^^^ recovers only ^^^^14sort_by14descending14^^^^%, an asymmetry of PRESERVED_PLACEHOLDER_^^^^14descending14 OR SemRF residual-stream dynamics)14^^^^ (&&&^^^^14sort_by14^^^^&&&). The summary rescue matrix reports values such as ^^^^14descending14max_results14^^^^.^^^^14max_results14 ^^^^14descending14all:(\14^^^^.^^^^14all:(\14 and ^^^^14descending14 gauge OR \14^^^^.^^^^14max_results14^^^^ for stream ^^^^14all:(\14^^^^ rescuing ablated streams ^^^^14query14^^^^, ^^^^14 gauge OR \14^^^^, and ^^^^14 OR SemRF residual-stream dynamics)14^^^^ respectively (&&&^^^^14sort_by14^^^^&&&). The methodological point is explicit: high CKA does not guarantee high rescue, and rescue scores quantify directional dependence rather than mere similarity.

^^^^14max_results14^^^^. Geometric and layer-distribution gauges

A third class of residual-stream gauge is geometric rather than token-predictive. FlowLens is described as a stable PCA-based tool for residual-stream geometry analysis that stacks residual vectors from multiple layers and prompts into a single matrix PRESERVED_PLACEHOLDER_^^^^14descending14max_results14^^^^, centers it, forms the covariance

PRESERVED_PLACEHOLDER_^^^^14descending14sort_by14^^^^

and summarizes variance concentration by

PRESERVED_PLACEHOLDER_^^^^14descending14submittedDate14^^^^

(&&&^^^^14 OR SemRF residual-stream dynamics)14^^^^&&&). High PRESERVED_PLACEHOLDER_^^^^14descending14sort_order14^^^^ indicates that residual-stream variance collapses onto a PRESERVED_PLACEHOLDER_^^^^14descending14descending14^^^^-dimensional subspace. Applied to safety fine-tuning, the paper reports that safety completion corpora have unigram entropy PRESERVED_PLACEHOLDER_^^^^14descending14query14^^^^ bits, versus PRESERVED_PLACEHOLDER_^^^^14query14query14^^^^ bits for general instruction data, and ^^^^14 gauge OR \14^^^^-gram diversity of approximately ^^^^14max_results14^^^^.^^^^14descending14 distinct bigrams versus approximately ^^^^14 gauge OR \14query14^^^^.^^^^14sort_by14^^^^% (&&&^^^^14 OR SemRF residual-stream dynamics)14^^^^&&&). As the safety ratio rises from PRESERVED_PLACEHOLDER_^^^^14query14all:(\14^^^^, the structural alignment score

PRESERVED_PLACEHOLDER_^^^^14query14 gauge OR \14^^^^

drops from ^^^^14all:(\14^^^^.^^^^14query14query14^^^^ to ^^^^14query14^^^^.^^^^14descending14sort_by14 while false refusal rate on XSTest climbs from ^^^^14submittedDate14 OR SemRF residual-stream dynamics)14^^^^% to ^^^^14descending14max_results14^^^^% (&&&^^^^14 OR SemRF residual-stream dynamics)14^^^^&&&). The paper attributes this to variance concentration in mid-network residuals and reduced representational smoothness.

The proposed remedy is Variance Concentration Loss (VCL). Given centered residual matrix PRESERVED_PLACEHOLDER_^^^^14query14 OR SemRF residual-stream dynamics)14^^^^ over a layer window, with SVD PRESERVED_PLACEHOLDER_^^^^14query14max_results14^^^^, the regularizer is

PRESERVED_PLACEHOLDER_^^^^14query14sort_by14^^^^

with example settings PRESERVED_PLACEHOLDER_^^^^14query14submittedDate14^^^^, PRESERVED_PLACEHOLDER_^^^^14query14sort_order14^^^^, and window depths PRESERVED_PLACEHOLDER_^^^^14query14descending14^^^^ of normalized layer depth (&&&^^^^14 OR SemRF residual-stream dynamics)14^^^^&&&). At ^^^^14max_results14query14^^^^% safety ratio, VCL reduces false refusal on XSTest by over ^^^^14 OR SemRF residual-stream dynamics)14sort_by14^^^^ percentage points compared to standard SFT while maintaining or improving general benchmarks such as MMLU and GSM^^^^14descending14^^^^K; the paper lists DAN from PRESERVED_PLACEHOLDER_^^^^14query14query14^^^^, Toxigen from PRESERVED_PLACEHOLDER_^^^^14all:(\14query14query14^^^^, MMLU PRESERVED_PLACEHOLDER_^^^^14all:(\14query14all:(\14^^^^, and GSM^^^^14descending14^^^^K PRESERVED_PLACEHOLDER_^^^^14all:(\14query14 gauge OR \14^^^^ (&&&^^^^14 OR SemRF residual-stream dynamics)14^^^^&&&). The residual-stream gauge here is therefore a global structural diagnostic of overfitting and disruption.

A different distributional gauge is introduced by multi-layer sparse autoencoders (MLSAEs) (&&&^^^^14max_results14^^^^&&&). For latent index PRESERVED_PLACEHOLDER_^^^^14all:(\14query14 OR SemRF residual-stream dynamics)14^^^^, the layer-activation distribution is

PRESERVED_PLACEHOLDER_^^^^14all:(\14query14max_results14^^^^

and its variance is

PRESERVED_PLACEHOLDER_^^^^14all:(\14query14sort_by14^^^^

Low variance means the latent is essentially active at a single layer; high variance means that it is active at multiple layers (&&&^^^^14max_results14^^^^&&&). Empirically, for Pythia-^^^^14sort_order14query14^^^^m, PRESERVED_PLACEHOLDER_^^^^14all:(\14query14submittedDate14^^^^, PRESERVED_PLACEHOLDER_^^^^14all:(\14query14sort_order14^^^^, aggregating over ten million test tokens yields

PRESERVED_PLACEHOLDER_^^^^14all:(\14query14descending14^^^^

and for Pythia-^^^^14all:(\14^^^^B the aggregate relative variance rises to approximately ^^^^14query14^^^^.^^^^14descending14submittedDate14^^^^ (&&&^^^^14max_results14^^^^&&&). Larger models also show higher adjacent-layer cosine similarity, from approximately ^^^^14query14^^^^.^^^^14submittedDate14^^^^ for ^^^^14sort_order14query14^^^^M up to approximately ^^^^14query14^^^^.^^^^14descending14sort_by14^^^^ for ^^^^14 gauge OR \14^^^^.^^^^14descending14^^^^B, consistent with slower representation drift and more cross-layer latent activity (&&&^^^^14max_results14^^^^&&&). When pre-trained tuned-lens transformations are applied, aggregate variance ratios decrease, for example from ^^^^14query14^^^^.^^^^14sort_by14max_results14^^^^ to ^^^^14query14^^^^.^^^^14 OR SemRF residual-stream dynamics)14descending14^^^^ for Pythia-^^^^14sort_order14query14^^^^m, while the single-token ratio increases from ^^^^14all:(\14^^^^% to ^^^^14all:(\14^^^^.^^^^14sort_by14 (&&&^^^^14max_results14^^^^&&&). This gauge therefore measures how features are distributed over depth, rather than whether a specific task variable is linearly recoverable.

^^^^14sort_by14^^^^. Semantic reference frames and canonical traces

SemRF, or Semantic Reference Frames, formalizes residual-stream gauge as a problem of coordinate synchronization across layers (&&&^^^^14submittedDate14^^^^&&&). Its starting point is that intermediate decoding requires comparable readout coordinates across depth: if embedding anchors and unembedding readout disagree on the chosen span, apparent motion may reflect measurement drift rather than computation. A SemRF is the pair PRESERVED_PLACEHOLDER_^^^^14all:(\14query14query14^^^^, where PRESERVED_PLACEHOLDER_^^^^14all:(\14all:(\14query14^^^^ is a fixed set of anchors and PRESERVED_PLACEHOLDER_^^^^14all:(\14all:(\14all:(\14^^^^ is a coordinate map with PRESERVED_PLACEHOLDER_^^^^14all:(\14all:(\14 gauge OR \14^^^^ (&&&^^^^14submittedDate14^^^^&&&). In the vocabulary-readout case, with embedding matrix PRESERVED_PLACEHOLDER_^^^^14all:(\14all:(\14 OR SemRF residual-stream dynamics)14^^^^ and unembedding PRESERVED_PLACEHOLDER_^^^^14all:(\14all:(\14max_results14^^^^, one writes PRESERVED_PLACEHOLDER_^^^^14all:(\14all:(\14sort_by14^^^^ and may use restricted readout coordinates

PRESERVED_PLACEHOLDER_^^^^14all:(\14all:(\14submittedDate14^^^^

The paper gives an exact synchronization condition. If there exist PRESERVED_PLACEHOLDER_^^^^14all:(\14all:(\14sort_order14^^^^ and invertible PRESERVED_PLACEHOLDER_^^^^14all:(\14all:(\14descending14^^^^ such that PRESERVED_PLACEHOLDER_^^^^14all:(\14all:(\14query14^^^^ and

PRESERVED_PLACEHOLDER_^^^^14all:(\14 gauge OR \14query14^^^^

then

PRESERVED_PLACEHOLDER_^^^^14all:(\14 gauge OR \14all:(\14^^^^

Under restricted bi-invertibility, with full-column-rank PRESERVED_PLACEHOLDER_^^^^14all:(\14 gauge OR \14 gauge OR \14^^^^, restricted readout PRESERVED_PLACEHOLDER_^^^^14all:(\14 gauge OR \14 OR SemRF residual-stream dynamics)14^^^^, interface error PRESERVED_PLACEHOLDER_^^^^14all:(\14 gauge OR \14max_results14^^^^, and bounded PRESERVED_PLACEHOLDER_^^^^14all:(\14 gauge OR \14sort_by14^^^^, the coordinate distortion on the anchor span obeys

PRESERVED_PLACEHOLDER_^^^^14all:(\14 gauge OR \14submittedDate14^^^^

and more generally, for PRESERVED_PLACEHOLDER_^^^^14all:(\14 gauge OR \14sort_order14^^^^,

PRESERVED_PLACEHOLDER_^^^^14all:(\14 gauge OR \14descending14^^^^

(&&&^^^^14submittedDate14^^^^&&&). These bounds are then extended to full trajectories and to near-identity changes of basis between admissible frames.

With the frame fixed, residual computation becomes a depthwise semantic trajectory (&&&^^^^14submittedDate14^^^^&&&). Anchors induce a semantic Voronoi diagram:

PRESERVED_PLACEHOLDER_^^^^14all:(\14 gauge OR \14query14^^^^

with semantic margin

PRESERVED_PLACEHOLDER_^^^^14all:(\14 OR SemRF residual-stream dynamics)14query14^^^^

The layerwise semantic step is PRESERVED_PLACEHOLDER_^^^^14all:(\14 OR SemRF residual-stream dynamics)14all:(\14^^^^, and SemRF defines three imbalance diagnostics: motion energy PRESERVED_PLACEHOLDER_^^^^14all:(\14 OR SemRF residual-stream dynamics)14 gauge OR \14^^^^, curvature energy PRESERVED_PLACEHOLDER_^^^^14all:(\14 OR SemRF residual-stream dynamics)14 OR SemRF residual-stream dynamics)14^^^^, and backtracking penalty PRESERVED_PLACEHOLDER_^^^^14all:(\14 OR SemRF residual-stream dynamics)14max_results14^^^^ (&&&^^^^14submittedDate14^^^^&&&).

The canonical trace is defined by a margin-relaxed semantic tube and a quadratic action. For observed trajectory PRESERVED_PLACEHOLDER_^^^^14all:(\14 OR SemRF residual-stream dynamics)14sort_by14^^^^, Voronoi trace PRESERVED_PLACEHOLDER_^^^^14all:(\14 OR SemRF residual-stream dynamics)14submittedDate14^^^^, trusted layer set PRESERVED_PLACEHOLDER_^^^^14all:(\14 OR SemRF residual-stream dynamics)14sort_order14^^^^, and relaxation radii PRESERVED_PLACEHOLDER_^^^^14all:(\14 OR SemRF residual-stream dynamics)14descending14^^^^,

PRESERVED_PLACEHOLDER_^^^^14all:(\14 OR SemRF residual-stream dynamics)14query14^^^^

Given weights PRESERVED_PLACEHOLDER_^^^^14all:(\14max_results14query14^^^^ not both zero and PRESERVED_PLACEHOLDER_^^^^14all:(\14max_results14all:(\14^^^^, the path action is

PRESERVED_PLACEHOLDER_^^^^14all:(\14max_results14 gauge OR \14^^^^

The canonical trace PRESERVED_PLACEHOLDER_^^^^14all:(\14max_results14 OR SemRF residual-stream dynamics)14^^^^ is the unique minimizer of PRESERVED_PLACEHOLDER_^^^^14all:(\14max_results14max_results14^^^^ over PRESERVED_PLACEHOLDER_^^^^14all:(\14max_results14sort_by14^^^^ (&&&^^^^14submittedDate14^^^^&&&). For PRESERVED_PLACEHOLDER_^^^^14all:(\14max_results14submittedDate14^^^^ and no active tube constraint, the interior Euler–Lagrange condition is the discrete fourth-order spline equation

PRESERVED_PLACEHOLDER_^^^^14all:(\14max_results14sort_order14^^^^

Excess action controls deviations from the canonical trace, and low curvature yields compressibility bounds of the form

PRESERVED_PLACEHOLDER_^^^^14all:(\14max_results14descending14^^^^

for a piecewise-linear approximation with PRESERVED_PLACEHOLDER_^^^^14all:(\14max_results14query14^^^^ segments (&&&^^^^14submittedDate14^^^^&&&). SemRF thus treats gauge not as a classifier or summary statistic, but as a formally admissible coordinate frame in which depthwise semantics can be compared without confounding by readout mismatch.

^^^^14submittedDate14^^^^. Architectural and visual-model extensions

The residual-stream gauge has also been used as a design and monitoring tool for alternative sequence architectures. In the Residual Matrix Transformer (RMT), the usual vector residual stream is replaced by an outer-product memory matrix PRESERVED_PLACEHOLDER_^^^^14all:(\14sort_by14query14^^^^ for token PRESERVED_PLACEHOLDER_^^^^14all:(\14sort_by14all:(\14^^^^ at layer PRESERVED_PLACEHOLDER_^^^^14all:(\14sort_by14 gauge OR \14^^^^, stacked into PRESERVED_PLACEHOLDER_^^^^14all:(\14sort_by14 OR SemRF residual-stream dynamics)14^^^^ (&&&^^^^14max_results14descending14^^^^&&&). Storage writes

PRESERVED_PLACEHOLDER_^^^^14all:(\14sort_by14max_results14^^^^

and retrieval reads by tensor contraction PRESERVED_PLACEHOLDER_^^^^14all:(\14sort_by14sort_by14^^^^ (&&&^^^^14max_results14descending14^^^^&&&). The associated residual-stream gauge has three components: activation-variance gauge

PRESERVED_PLACEHOLDER_^^^^14all:(\14sort_by14submittedDate14^^^^

with target PRESERVED_PLACEHOLDER_^^^^14all:(\14sort_by14sort_order14^^^^; capacity/use gauge

PRESERVED_PLACEHOLDER_^^^^14all:(\14sort_by14descending14^^^^

and an effective-dimension gauge based on the singular spectrum of the average memory PRESERVED_PLACEHOLDER_^^^^14all:(\14sort_by14query14^^^^ (&&&^^^^14max_results14descending14^^^^&&&). The paper further reports that RMT can achieve the same loss as the transformer with ^^^^14sort_by14descending14^^^^% fewer FLOPS, ^^^^14 gauge OR \14sort_by14^^^^% fewer parameters, and ^^^^14max_results14all:(\14^^^^% fewer training tokens, and that increasing the residual dimension PRESERVED_PLACEHOLDER_^^^^14all:(\14submittedDate14query14^^^^ from ^^^^14sort_order14submittedDate14descending14^^^^ to ^^^^14max_results14query14query14submittedDate14^^^^ monotonically lowers dev-loss by PRESERVED_PLACEHOLDER_^^^^14all:(\14submittedDate14all:(\14^^^^ nats and can match the smaller-residual transformer with ^^^^14 gauge OR \14 OR SemRF residual-stream dynamics)14^^^^% fewer FLOPs and ^^^^14 gauge OR \14sort_by14^^^^% fewer tokens (&&&^^^^14max_results14descending14^^^^&&&). Here the gauge is explicitly a tuning and scaling instrument for residual capacity.

In ResNet^^^^14all:(\14descending14^^^^, residual-stream gauge is a per-channel diagnostic on residual addition rather than a sequence-state analysis (&&&^^^^14sort_order14^^^^&&&). For a block output channel,

PRESERVED_PLACEHOLDER_^^^^14all:(\14submittedDate14 gauge OR \14^^^^

with skip input PRESERVED_PLACEHOLDER_^^^^14all:(\14submittedDate14 OR SemRF residual-stream dynamics)14^^^^ and block feature PRESERVED_PLACEHOLDER_^^^^14all:(\14submittedDate14max_results14^^^^, the gauge defines a mix ratio

PRESERVED_PLACEHOLDER_^^^^14all:(\14submittedDate14sort_by14^^^^

where PRESERVED_PLACEHOLDER_^^^^14all:(\14submittedDate14submittedDate14^^^^ and PRESERVED_PLACEHOLDER_^^^^14all:(\14submittedDate14sort_order14^^^^ are feature-visualization images that maximize PRESERVED_PLACEHOLDER_^^^^14all:(\14submittedDate14descending14^^^^ and PRESERVED_PLACEHOLDER_^^^^14all:(\14submittedDate14query14^^^^ respectively (&&&^^^^14sort_order14^^^^&&&). Normalized effective gating coefficients are then

PRESERVED_PLACEHOLDER_^^^^14all:(\14sort_order14query14^^^^

Large PRESERVED_PLACEHOLDER_^^^^14all:(\14sort_order14all:(\14^^^^ corresponds to skip-like behavior, small PRESERVED_PLACEHOLDER_^^^^14all:(\14sort_order14 gauge OR \14^^^^ to overwrite-like behavior, and PRESERVED_PLACEHOLDER_^^^^14all:(\14sort_order14 OR SemRF residual-stream dynamics)14^^^^ to mixture (&&&^^^^14sort_order14^^^^&&&). The same work introduces a scale-invariance gauge using three conditions: the mix criterion PRESERVED_PLACEHOLDER_^^^^14all:(\14sort_order14max_results14^^^^, a block criterion PRESERVED_PLACEHOLDER_^^^^14all:(\14sort_order14sort_by14^^^^, and an input criterion PRESERVED_PLACEHOLDER_^^^^14all:(\14sort_order14submittedDate14^^^^; a scale metric PRESERVED_PLACEHOLDER_^^^^14all:(\14sort_order14sort_order14^^^^ ranks the strength of the effect (&&&^^^^14sort_order14^^^^&&&). Empirically, a small but appreciable fraction, up to PRESERVED_PLACEHOLDER_^^^^14all:(\14sort_order14descending14^^^^–PRESERVED_PLACEHOLDER_^^^^14all:(\14sort_order14query14^^^^ of channels in blocks ^^^^14all:(\14^^^^.^^^^14all:(\14 ^^^^14 gauge OR \14^^^^.^^^^14query14^^^^, ^^^^14 gauge OR \14^^^^.^^^^14all:(\14^^^^ and ^^^^14 OR SemRF residual-stream dynamics)14^^^^.^^^^14all:(\14^^^^, pass the scale-invariance test, whereas blocks ^^^^14 OR SemRF residual-stream dynamics)14^^^^.^^^^14query14^^^^, ^^^^14max_results14^^^^.^^^^14query14 and ^^^^14max_results14^^^^.^^^^14all:(\14^^^^ yield almost none (&&&^^^^14sort_order14^^^^&&&).

A related ResNet^^^^14all:(\14descending14^^^^ study focuses on scale-invariant representations computed by residual summation (&&&^^^^14descending14^^^^&&&). It defines

PRESERVED_PLACEHOLDER_^^^^14all:(\14descending14query14^^^^

and operationally classifies a channel as scale-invariant if

PRESERVED_PLACEHOLDER_^^^^14all:(\14descending14all:(\14^^^^

and

PRESERVED_PLACEHOLDER_^^^^14all:(\14descending14 gauge OR \14^^^^

Blocks ^^^^14 gauge OR \14^^^^.^^^^14all:(\14^^^^ and ^^^^14 OR SemRF residual-stream dynamics)14^^^^.^^^^14all:(\14^^^^ had the highest proportion of passing channels, approximately ^^^^14all:(\14descending14^^^^%, and ablations that overwrite scale-invariant channels with their mean activation produce larger top-^^^^14all:(\14^^^^ accuracy degradation under scale transformations than matched random ablations (&&&^^^^14descending14^^^^&&&). The paper summarizes the mechanism by stating that the bypass connection carries a small-scale version of the feature in parallel with the main path’s large-scale version, and their element-wise sum yields an invariant signal (&&&^^^^14descending14^^^^&&&). In this visual setting, the gauge diagnoses how residual addition manages features and builds invariance.

^^^^14sort_order14^^^^. Terminological ambiguity and related usage

The word “gauge” is not uniform across the cited literature. In most machine-learning works discussed above, it is a metaphor for a probe, metric panel, causal instrument, or coordinate system on residual activations. A different, literal use appears in light-cone gravity, where residual gauge transformations survive gauge fixing and realize the full four-dimensional BMS algebra on the two physical helicity fields (&&&^^^^14submittedDate14query14^^^^&&&). After imposing

PRESERVED_PLACEHOLDER_^^^^14all:(\14descending14 OR SemRF residual-stream dynamics)14^^^^

and PRESERVED_PLACEHOLDER_^^^^14all:(\14descending14max_results14^^^^, the remaining transformations act on the helicity fields PRESERVED_PLACEHOLDER_^^^^14all:(\14descending14sort_by14^^^^ by

PRESERVED_PLACEHOLDER_^^^^14all:(\14descending14submittedDate14^^^^

with analogous action on PRESERVED_PLACEHOLDER_^^^^14all:(\14descending14sort_order14^^^^ (&&&^^^^14submittedDate14query14^^^^&&&). Requiring invariance of the light-cone Hamiltonian yields the conformal-Killing conditions PRESERVED_PLACEHOLDER_^^^^14all:(\14descending14descending14^^^^ and PRESERVED_PLACEHOLDER_^^^^14all:(\14descending14query14^^^^, while an arbitrary function PRESERVED_PLACEHOLDER_^^^^14all:(\14query14query14^^^^ enters through

PRESERVED_PLACEHOLDER_^^^^14all:(\14query14all:(\14^^^^

The commutator closes onto new parameters PRESERVED_PLACEHOLDER_^^^^14all:(\14query14 gauge OR \14^^^^ with the light-cone realization of the BMS algebra, including

PRESERVED_PLACEHOLDER_^^^^14all:(\14query14 OR SemRF residual-stream dynamics)14^^^^

(&&&^^^^14submittedDate14query14^^^^&&&).

This usage is conceptually separate from neural-network residual-stream gauges, but the juxtaposition is instructive. In gravity, gauge denotes residual symmetry after gauge fixing; in residual-stream analysis, gauge denotes a way of measuring or coordinating hidden-state evolution. The overlap is therefore lexical rather than formal. Still, both usages place emphasis on what remains invariant or recoverable once a representation has been fixed, and this suggests why the term has proved attractive in analyses of residual pathways.

Across these literatures, the residual-stream gauge has become a technical label for methods that treat the residual pathway as the natural site of state, geometry, and mechanism. In transformers, this can mean exact sufficiency of the residual state; in probing, it can mean anticipatory detection of conflict and source selection; in causal analysis, it can mean directed rescue after intervention; in geometry, it can mean variance concentration and structural smoothness; in semantic analysis, it can mean a synchronized reference frame across depth; and in residual vision models, it can mean per-channel quantification of skip, overwrite, mixture, and scale invariance (&&&^^^^14query14^^^^&&&, &&&^^^^14 gauge OR \14^^^^&&&, &&&^^^^14sort_by14^^^^&&&, &&&^^^^14 OR SemRF residual-stream dynamics)14^^^^&&&, &&&^^^^14submittedDate14^^^^&&&, &&&^^^^14sort_order14^^^^&&&, &&&^^^^14descending14^^^^&&&).