Borrowed Geometry: Computational Reuse of Frozen Text-Pretrained Transformer Weights Across Modalities

Abstract: Frozen Gemma 4 31B weights pretrained exclusively on text tokens, unmodified, transfer across modality boundaries through a thin trainable interface. (1) OGBench scene-play-singletask-task1-v0: $+4.33$pt over published GCIQL at $n=3$ with std 0.74 -- a published-SOTA win on a robotic manipulation task the substrate has never seen. (2) D4RL Walker2d-medium-v2: Decision-Transformer parity ($76.2 \pm 0.8$, $n=3$) at $0.43\times$ DT's trainable count, with the frozen substrate compressing to a 5L slice ($+1.66$pt over the 6L baseline at $n=3$). (3) Associative recall as the cleanest pretraining-load-bearing case: the frozen slice + a 113K-parameter linear interface reaches L30 best-checkpoint per-bit error 0.0505 ($n=2$); a 6.36M-parameter from-scratch trained transformer at matched capacity ($1/\sqrt{d_k}$ scaling, two seeds, LR sweep) cannot solve the task at all under the protocol (best L30 = 0.4395), an $8.7\times$ advantage. Architecture-alone falsifications: a frozen random transformer with correct $1/\sqrt{d_k}$ scaling stays at random-chance loss for 50k steps; a random-init Gemma slice fails OGBench cube-double-play-task1 entirely (0.89% across $n=3$ where pretrained reaches 60%). A dual-measurement protocol -- text-activation probing on 95 English sentences plus task-ablation on a non-language target -- names individual heads independently identifiable on both protocols: head L26.28 scores $3.7\times$ the slice mean for English token-copying and is the #2 most-critical head for binary copy ablation ($Δ$ L30 $= +0.221$); three further heads (L27.28, L27.2, L27.3) classify by the same protocol. The mechanism is single-model and the cross-modality results are single-task within their respective benchmarks; cross-model replication is structurally constrained because Gemma 4 31B is the only model on the small-scale Pareto frontier as of April 2026.

• The paper demonstrates that frozen Gemma 4 31B layers, paired with interfaces as small as 113K trainable parameters, transfer to robotic manipulation, locomotion control, and sequence tasks, including 97.33% scene-play success and a 76.2 Walker2d score.
• The paper identifies four attention heads reused from English token-copying for non-language computation, with associative recall achieving an 8.7× advantage over a matched-capacity transformer trained from scratch and controls failing at chance.
• The paper finds that transfer supports token matching and bounded pattern completion but not stack-like state accumulation, 2D relational reasoning, smooth regression, or environment-level out-of-distribution recovery, highlighting the need for broader replication.

Overview and motivation

"Borrowed Geometry: Computational Reuse of Frozen Text-Pretrained Transformer Weights Across Modalities" (2605.00333) asks whether frozen, unmodified text-pretrained weights can serve as a general computational substrate for tasks outside their pretraining distribution. The substrate is Gemma 4 31B — 60 layers, hidden dimension 5376, 32 heads per layer — pretrained exclusively on text tokens. The canonical experiments freeze layers L24–L29 (~2.93B parameters) in bf16 with zero gradients, wrap them in a thin trainable interface (a linear encoder into the 5376-d input space, a calibrated input-statistics matcher aligned to English-prose activation statistics, and a linear decoder), and train only that interface on non-text tasks: bit strings, state-action sequences, joint kinematics, and continuous-control feedback.

The paper frames this setup as a pretrained generalization of reservoir computing: computation rides on a fixed high-dimensional substrate while only a small readout is learned. Theoretically it positions itself against the Platonic Representation Hypothesis (PRH), which predicts representational convergence across modalities under multitask scaling, capacity, and simplicity bias; if representations converge, frozen transferability should be a property of the converged geometry. The term "exaptation," borrowed from evolutionary biology, labels the central mechanistic claim: attention heads shaped by one selective pressure (text-copying during language modeling) are recruited, frozen, for analogous computation in a non-language domain.

Cross-modality transfer results

Three headline results use the same frozen weights across three modalities.

OGBench scene-play-singletask-task1-v0: GemmaIQL — V/Q/π heads over a single frozen layer L24 (488M frozen) — reaches 97.33% ± 0.74 at n=3 seeds, beating published GCIQL (93%) by +4.33 points, with all three seeds independently above 96%. The random-init Gemma control (NC1) reaches 86.89% ± 8.70, so the pretrained advantage over matched architecture is +10.44 points. The authors note honestly that scene-play is a regime where IQL plus sufficient capacity does most of the work.

OGBench cube-double-play-singletask-task1-v0: this is the cleanest substrate-isolation measurement. A random-init Gemma slice fails entirely (0.89% across n=3 / 30 evals) where pretrained Gemma reaches 42–60%, a +59-point pretrained-vs-architecture gap at L26 (+41 at L24). Absolute performance nonetheless loses to GCIQL (−13.78 points at L26 mean). The L26 result is bimodal: two seeds reach 88–96% while one collapses from a 24% peak to 0%; the isolation claim holds because the random control stays at zero on all seeds, but the authors flag n ≥ 5 replication as the appropriate stress test.

D4RL Walker2d-medium-v2: Gemma-DT replaces Decision Transformer's GPT-2 body with frozen L24–L29, reaching 76.2 ± 0.8 normalized score at n=3 — all three seeds exceed Chen et al.'s DT reference of 74.0 — using 521K trainable parameters, roughly 0.43× DT's count. A layer-drop sweep shows the signal compresses: dropping L24 yields a 5-layer slice (2.45B frozen) that beats the 6-layer baseline by +1.66 points with tighter variance. Notably, Walker2d is the only D4RL task that transfers; HalfCheetah and Hopper fail, attributed respectively to fine-continuous-control precision demands and compounding-error recovery demands.

A consistent layer–task interaction emerges: cube-task1 recruits L26 (which contains the named head L26.28), while scene-task1 and Walker2d recruit L24, which lies outside the named-head triple identified by the supervised ablation protocol. The paper explicitly does not claim its mechanism explains the scene and Walker2d results.

Mechanism: named heads under dual measurement

The cleanest pretraining-load-bearing case is associative recall (AR): a 113K-parameter linear interface around the frozen L25–L27 slice reaches best-checkpoint per-bit error 0.0505 (n=2), while a 6.36M-parameter from-scratch transformer at matched capacity cannot solve the task at all under the protocol (best L30 = 0.4395, indistinguishable from the 0.5 random-chance error) — an 8.7× advantage. This establishes that pretraining, not capacity or architecture, carries the result.

Two falsification controls close confounds. A frozen 6-layer GPT-2-initialized transformer with correct $1/\sqrt{d_k}$ scaling never escapes random-chance loss on CA Rule 90 across two seeds (BCE stuck at 0.693 for 50k steps), closing the architecture-alone confound raised in prior critiques of Frozen Pretrained Transformers. Separately, adapter-capacity ablations show NTM memory-slot count is irrelevant (N=1, 4, 128 indistinguishable) and a 113K linear interface reproduces the 6.1M pipeline — evidence the authors argue converts the claim from feature reuse into computation reuse, since the trainable interface is demonstrably thin.

The dual-measurement protocol names individual heads via two independent procedures:

• Text-activation probing: each of 192 heads scored on English attention patterns over 95 sentences (TxtCopy, Induction, PrevToken primitives).
• Task ablation: zeroing each head's output projection in trained non-language task models and measuring per-bit-error increase.

Head L26.28 scores 3.7× the slice mean for English token-copying (4th of 192) and ranks #2 most critical for binary copy ablation (Δ = +0.221). Three further heads classify by both protocols: L27.28 (#1 critical on copy and addition, 2.6× TxtCopy), L27.2 (#1 on AR, 1.7×), and L27.3 (#1 on CA Rule 90, 1.7×). The same physical atoms of weight space are picked out by unrelated measurements on different input distributions performing the same kind of computation.

The paper applies notable base-rate honesty here: under naive independence, three-of-three named heads classifying as TxtCopy has base rate ≈ 6.9% — distinguishable from chance but not overwhelming. Across the full 141-pair sweep, 47.5% of critical pairs have a single dominant language function, with TxtCopy/Induction/PrevToken accounting for 90% of those classifications, consistent with superposition-theory predictions about which features crystallize versus remain superposed.

Coverage boundary

The negative results are as carefully characterized as the positives. Dyck-2 exhibits a plateau specific to frozen-LM mid-band weights: five extraction surfaces and three stack depths all cluster at L30 ∈ [0.024, 0.036], while a 0.56M LSTM reaches 0.0037 and a matched-capacity trained transformer reaches 0.0011 — so the plateau is not architectural. Two candidate accounts (coincidence vs. an "anti-prior" whereby language pretraining shapes weights away from explicit-stack structure) remain undistinguished because neither an un-pretrained matched-depth control nor a controlled fine-tune was run. Other failures include 2D relational computation (Game of Life, even with direct access to attention matrices as graph adjacencies), smooth continuous regression (NARMA, Mackey-Glass, Lorenz — where classical ESNs also win), environment-OOD generalization (Sokoban held-out solve rate 0%), and unbounded state accumulation. The unified statement: the geometry supports token-matching primitives and pattern completion with bounded length-OOD radius, and does not support state accumulation, 2D relational computation, smooth regression, or OOD recovery.

Supporting analyses reinforce the picture: per-head SVD shows ~50% of singular values needed for 90% spectral energy, and truncating output projections to top 256 collapses performance to chance; layer-shuffle within the slice is less destructive than row-shuffle within matrices, indicating the geometry lives in row arrangement. Distillation experiments show circuit layout predicts strategy — AR, whose criticality is distributed across layers, requires multi-hint alignment (DS-PROC multi-hint closes to within 24% of online Gemma at zero inference cost), whereas copy, localized at L27.28, needs only single-hint.

Limitations and open questions

The paper is unusually explicit about its constraints. All results are single-model: Gemma 4 31B is the only model on the small-scale Pareto frontier as of April 2026, making cross-model replication structurally constrained at independent-research compute; a null result on a less-compressed sub-Pareto substrate like Qwen 3 32B would not falsify the mechanism claim under PRH simplicity bias. Several key results rest on n=1 seeds, including the standalone-student distillation reaching 79.97 on Walker2d (beating its teacher at zero Gemma inference cost) and the multi-task shared-adapter observation. Task selection partially overlaps the NTM lineage from which the text-copying/induction/prev-token primitives derive, though Dyck-2's differential failure within that lineage and the out-of-lineage RL results mitigate this concern. The best-checkpoint protocol is mandatory rather than optional: training loss (~0.04) is decoupled from eval scores (0–77) across identical configurations, and final checkpoints varied 0.1–61.9 across same-config seeds. Finally, cross-modality head-ablation on the RL bodies — zeroing L24's heads on Walker2d, L26's on cube-task1 — is identified as the immediate next experiment needed to unify the mechanism claim, and is explicitly not claimed here.

Conclusion

This paper demonstrates that frozen text-pretrained transformer weights, wrapped in interfaces as thin as 113K trainable parameters, solve robotic manipulation, offline locomotion control, and algorithmic sequence tasks they were never trained for — including a +4.33-point win over published GCIQL at n=3 and an 8.7× advantage over matched-capacity from-scratch transformers on associative recall — while architecture-alone controls fail entirely. Its mechanistic contribution is the identification of four specific attention heads exapted from English token-copying to non-language token prediction, named independently by text probing and task ablation. The claims are bounded carefully: single-model, single-task within benchmarks, with the RL-side mechanism unverified and several key results awaiting multi-seed replication.