Chaos and bifurcations in continuous language-model feedback

Abstract: Feeding a pretrained LLM the expected embedding of its own next-token distribution turns it into a discrete-time dynamical system with continuous states. Across Qwen2.5, OLMo-2 and Llama-3 models from 0.5B to 72B parameters, temperature and activation steering move this loop between repeating text and chaos, with positive finite-time Lyapunov exponents. The loop's derivative factorizes into the inverse temperature, the covariance of token embeddings under the next-token distribution (a cross-covariance for untied embeddings) and the network's Jacobian. In one model studied in detail, along recorded orbits, the growth that steering adds comes mainly from the readout, where next tokens compete. The periodic windows show the quantitative signatures of low-dimensional maps, from saddle-node scaling and Feigenbaum's constant to shrimp-shaped domains; passage times predicted in advance from the model's derivatives came within one return-map iterate of the measured passage times in 42 of 48 cases over two prospective stages (24 and 18 of 24), and interval arithmetic proves attracting periodic orbits of the full transformer map. Numerical reproducibility depends on the dynamical regime: changing only the batch layout changed the text of 173 of 192 chaotic float64 loops in the smallest model, compared with only 7 of 192 periodic ones. These results show that bifurcation theory can make quantitative predictions about continuous language-model feedback, with rigorous guarantees for selected periodic orbits of the full transformer.

• The paper studies the dynamical behavior of transformer feedback maps, revealing periodic windows and chaotic regimes through non-linear dynamics techniques on various models from 0.5B to 72B parameters.
• The exact Jacobian factorization of the feedback map decomposes how temperature, token competition, and steering influence the map's dynamics, leading to qualitative predictions of periodic and chaotic behaviors.
• The analysis shows that chaos is a generic feature of transformer models and is sensitive to numerical execution, affecting the reproducibility of periodic and chaotic trajectories.

Problem formulation and scope

“Chaos and bifurcations in continuous language-model feedback” (2610.31692) studies a specific inference construction in which a pretrained transformer receives, at every iteration, the expected embedding of its own next-token distribution. With a fixed prompt and a one-token feedback window, this procedure defines an autonomous discrete-time map on the embedding space rather than an ordinary autoregressive process with growing context. The distinction is essential: the paper’s dynamical claims concern repeated application of a fixed transformer operator, not conventional sampled generation.

For a model with input embedding matrix $E$, logits $z_\alpha(x)$, temperature $T$, and an internal steering intervention of strength $\alpha$, the map is

$$F_{T,\alpha}(x) = E^\top \operatorname{softmax}\!\left(\frac{z_\alpha(x)}{T}\right).$$

The state $x$ is therefore continuous, even though the model ultimately defines a distribution over discrete tokens. Since the output is always a convex combination of token embeddings, every trajectory enters the compact convex hull of the embedding vectors after one iteration. For dense transformers, the resulting map is real analytic under the stated architectural assumptions. These properties make standard tools from nonlinear dynamics applicable: fixed points, periodic orbits, multipliers, Lyapunov exponents, continuation of bifurcation curves, center-manifold normal forms, and interval-arithmetic validation.

The empirical study spans Qwen2.5, OLMo-2, and Llama-3 models from 0.5B to approximately 72B parameters. The detailed bifurcation analysis is concentrated on Qwen2.5-0.5B-Instruct, with additional confirmation in Qwen2.5-1.5B and larger models. Temperature and activation steering act as control parameters. The central claim is not merely that feedback can produce sensitivity, but that the full transformer feedback map exhibits the organized parameter-space structures associated with low-dimensional nonlinear maps: periodic windows, saddle-node bottlenecks, period-doubling cascades, Feigenbaum scaling, and shrimp-shaped domains.

Derivative factorization and the mechanism of instability

The paper’s principal analytical result is an exact decomposition of the feedback-map Jacobian. Differentiating the softmax feedback gives

$$DF(x) = \frac{1}{T} E^\top \left(\operatorname{diag}(p)-pp^\top\right) J_z(x),$$

where $p$ is the next-token distribution and $J_z$ is the Jacobian of the transformer’s logits with respect to the fed-back embedding. For tied input and output embeddings, the transformer logits have the form $z=E\tilde h$, yielding

$z_\alpha(x)$0

The matrix $z_\alpha(x)$1 is the covariance of token embeddings under the model’s own next-token distribution. It is large when several tokens with substantially different embeddings compete, and it contracts toward zero when the distribution concentrates on a single token. For untied embeddings, the covariance is replaced by a generally nonsymmetric cross-covariance between input embeddings and output vectors.

This factorization separates two sources of local amplification. The network factor $z_\alpha(x)$2 describes how perturbations to the fed-back embedding propagate through attention, MLPs, normalization, residual connections, and steering. The readout factor $z_\alpha(x)$3 describes how perturbations in the final hidden state alter the probability-weighted embedding. The factorization was numerically verified at a period-8 orbit: its directional derivatives agreed with forward-mode differentiation to relative errors no larger than $z_\alpha(x)$4, while the best finite-difference estimates were more than four orders of magnitude less accurate.

The decomposition also establishes useful limiting behavior. At sufficiently high temperature, an embedding-diameter bound and a supremum bound on the transformer Jacobian imply contraction of the entire compact state set. Consequently, the map has a unique globally attracting fixed point above a rigorous, though extremely conservative, temperature threshold. In the 0.5B model, the estimated thresholds were at least $z_\alpha(x)$5 across sampled steering strengths and reached $z_\alpha(x)$6 at one strength, far above the observed chaotic temperatures.

At low temperature, the inverse-temperature factor alone does not imply instability. Away from logit ties, the covariance decays exponentially faster than $z_\alpha(x)$7 grows, and the map approaches one-token greedy feedback. On invariant sets with a uniform gap between the leading and runner-up logits, this excludes chaos in the low-temperature limit. The qualification is important: convergence is not uniform near token ties, where the readout gain can remain large at small positive temperatures. The observed chaotic regime therefore lies between the high-temperature contraction regime and the low-temperature concentration regime.

The paper further shows that the transformer component is necessary for the tied-embedding system to support nontrivial recurrent dynamics. If the network is replaced by the identity, the map becomes

$z_\alpha(x)$8

which is the update rule of a modern Hopfield network. Its Jacobian is symmetric positive semidefinite, and the map admits a strictly decreasing energy functional. This rules out attracting cycles of period greater than one and chaotic motion. Thus, in the tied-embedding case, the transformer Jacobian breaks an otherwise energy-dissipating structure. The result does not extend unchanged to untied embeddings, where the network-free cross-covariance need not be symmetric.

The steering analysis attributes most of the additional growth observed in Qwen2.5-0.5B to the readout rather than to the network Jacobian. Across a scan, chaotic steered cells exhibited a readout gain increase of $z_\alpha(x)$9 nats per step relative to nearly unsteered cells, with a bootstrap interval of $T$0, whereas the corresponding network-factor change was $T$1 nats per step with interval $T$2. Chaotic cells also had $T$3 nats higher next-token entropy at matched temperature. Orbit-preserving crossover experiments produced a $T$4-nat-per-step change when only the steered readout was substituted, compared with a $T$5 change when only the steered network factor was substituted.

These comparisons are conditional rather than causal decompositions of a separately rerun dynamical system. Steering changes the orbit as well as both derivative factors, and the hybrid products are assembled along recorded trajectories. The evidence supports the narrower claim that, on the studied Qwen2.5-0.5B trajectories, steering-induced amplification is predominantly associated with increased competition at the readout.

Chaotic regimes, periodic windows, and model-scale generality

The paper labels a finite-horizon trajectory chaotic when its largest Lyapunov exponent exceeds $T$6 and no period up to a specified cutoff is detected. These labels are explicitly operational and do not establish that the corresponding attractor satisfies stronger asymptotic definitions. Nevertheless, multiple checks support their numerical validity: tangent-vector and twin-trajectory exponents agreed within $T$7 across 448 sampled cells, and longer runs retained most labels. In two principal float64 scans, 176 and 143 of 192 chaos-labeled cells remained nonperiodic with positive exponent for $T$8 steps; the remaining cases were mostly long chaotic transients that eventually settled onto periodic orbits.

The parameter scans reveal extensive interleaving of chaos and periodicity. In one $T$9 scan of Qwen2.5-0.5B, more than half the cells satisfied the finite-horizon chaos criterion, while $\alpha$0 of a random sample remained chaotic for $\alpha$1 iterations. Periodic windows of periods 2 through 32 were embedded in the chaotic region. In a finer scan, approximately $\alpha$2 of cells met the chaos criterion. The same qualitative organization appeared in all three dense model families, and unsteered temperature scans produced chaos with periodic windows in 11 of 14 tested models.

The phenomenon was not confined to small models. A steered Llama-3.3-70B-Instruct scan contained 3,410 chaotic cells out of 16,384, or $\alpha$3, with a largest exponent reaching $\alpha$4 per iteration. Qwen2.5-14B was chaotic at more tested temperatures than Qwen2.5-7B across eight fixed prompts, while Qwen2.5-72B also exhibited a confirmed chaotic band. The result is therefore not a monotonic “larger models are more stable” effect. However, the model-scale comparison remains heterogeneous: architectures, prompts, temperatures, steering layers, and intervention directions differ, and the paper does not provide a uniform scaling law.

Prompt and steering direction affect the location of chaotic regions substantially. Across 19 prompt conditions, the overlap between the base prompt’s chaotic cells and those of other prompts ranged from 1% to 37% by Jaccard index. Nonetheless, every tested prompt produced chaos and periodic windows somewhere in the scanned plane. Random directions, individual MLP-neuron interventions, contrastive concept vectors, and sparse-autoencoder features all generated chaotic regions. This supports the interpretation that chaos is a generic possibility of the feedback architecture, while the prompt and intervention determine which attractors and bifurcation structures are encountered.

A major practical consequence is sensitivity to numerical execution. Changing only the GPU batch layout caused 190 of 192 chaotic float64 loops to separate beyond relative distance $\alpha$5, whereas 182 of 192 periodic loops remained within that threshold for 1,500 steps. In the 0.5B model, the most probable tokens eventually differed in 173 of 192 chaotic cells but only 7 of 192 periodic cells. In lower precision, including bfloat16, even periodic trajectories commonly separated within a few iterations. The implication is regime-dependent reproducibility: periodic attractors suppress small rounding perturbations, while chaotic trajectories amplify them at approximately their Lyapunov rate.

The paper connects this sensitivity to downstream answers but reports it primarily in settings where the underlying procedure already performs poorly or consumes its computational budget. For selected two-digit additions, changing batch layout altered 55–56% of Soft Thinking answers in particular temperature regimes; on GSM8K, 21% of answers changed under a fixed design, although overall accuracy was only 22–30% in that pipeline. When the feedback tokens were replaced by a reference trajectory, the changed-answer rate fell sharply. Among reasoning models, most changes occurred on examples reaching a thinking or answer budget. The study therefore demonstrates numerical instability of the continuous feedback process, but it does not establish that chaotic feedback systematically degrades ordinary task accuracy.

Saddle-node bottlenecks and quantitative prediction

The most detailed bifurcation analysis concerns a period-2 orbit whose leading-token loop is “is it.” At fixed $\alpha$6, the attracting 2-cycle meets an unstable 2-cycle in a saddle-node bifurcation at $\alpha$7. The critical multiplier of the second iterate equals $\alpha$8 to within $\alpha$9, while the next multiplier has modulus $$F_{T,\alpha}(x) = E^\top \operatorname{softmax}\!\left(\frac{z_\alpha(x)}{T}\right).$$0.

The center-manifold normal form is

$$F_{T,\alpha}(x) = E^\top \operatorname{softmax}\!\left(\frac{z_\alpha(x)}{T}\right).$$1

with computed coefficients $$F_{T,\alpha}(x) = E^\top \operatorname{softmax}\!\left(\frac{z_\alpha(x)}{T}\right).$$2 and $$F_{T,\alpha}(x) = E^\top \operatorname{softmax}\!\left(\frac{z_\alpha(x)}{T}\right).$$3, hence $$F_{T,\alpha}(x) = E^\top \operatorname{softmax}\!\left(\frac{z_\alpha(x)}{T}\right).$$4. These coefficients were obtained from the transformer’s first and second derivatives rather than fitted to trajectory data. Independent checks using branch multipliers recovered $$F_{T,\alpha}(x) = E^\top \operatorname{softmax}\!\left(\frac{z_\alpha(x)}{T}\right).$$5, and a separate implementation reproduced the critical parameter and coefficients to near machine precision.

Beyond the fold, the vanished periodic orbit leaves a bottleneck or “ghost.” Trajectories move slowly through this channel, producing type-I intermittency: long laminar passages resembling the former periodic motion separated by bursts. The normal form predicts passage durations without fitted parameters. For four distances from the fold, full passages measured 144, 71, 34, and 14 iterations, and the iterated normal form produced exactly the same counts. Midpoint and transversely displaced starts also agreed to within one iteration.

The paper then tests the prediction prospectively at four additional folds. Across two stages and 48 passage forecasts, the continuous normal-form integral was within one iterate in 42 cases. In the second stage, where it was the primary forecast, it succeeded for 18 of 24 cases. The six failures all occurred at a period-6 fold and were 3.1–5.2% shorter than predicted. These results are unusually strong for a full transformer map, but their interpretation is appropriately local: the normal form is truncated, the center manifold is not explicitly constructed, and the global reinjection mechanism is not predicted.

The same bottleneck accounts for the accumulation of periodic windows. Stripe windows with periods $$F_{T,\alpha}(x) = E^\top \operatorname{softmax}\!\left(\frac{z_\alpha(x)}{T}\right).$$6 approach the fold, and their distances scale approximately as $$F_{T,\alpha}(x) = E^\top \operatorname{softmax}\!\left(\frac{z_\alpha(x)}{T}\right).$$7, where $$F_{T,\alpha}(x) = E^\top \operatorname{softmax}\!\left(\frac{z_\alpha(x)}{T}\right).$$8 counts repeated token-pair segments in the attractor text. The measured log-log slope was $$F_{T,\alpha}(x) = E^\top \operatorname{softmax}\!\left(\frac{z_\alpha(x)}{T}\right).$$9. A free exponent fitted jointly with an offset varied from 2.13 to 2.45 depending on the fitting range, so the exponent is consistent with but not rigorously fixed to the ideal asymptotic value. Using parameters frozen before testing, the predicted locations of higher-period fold edges were within 3% of the measured distances.

The periodic orbit itself retains textual structure. The period-8 orbit, for example, writes a repeated pattern containing “This,” “there is,” and “is a.” Yet the fed-back state is not a sequence of discrete tokens; each iteration uses the full probability-weighted embedding. The text identifies only the most probable token at each state and therefore underdetermines the actual continuous orbit.

Period-doubling cascades and hidden chaos

A separate period-3 window undergoes a period-doubling cascade. At each flip, a real multiplier reaches $x$0, the original periodic orbit loses stability, and a stable orbit of twice the period emerges. Along one fixed-temperature cut, the cascade was followed through nine flips, from period 3 to period 1,536. The finite-order scaling ratios converge toward Feigenbaum’s constant $x$1; the last reported ratio was $x$2, within $x$3.

The cascade is strongly dissipative in the transverse directions. From the fifth flip onward, the next multiplier modulus was below approximately $x$4, near the numerical precision limit for products of hundreds of $x$5 Jacobians. This supports an effectively one-dimensional critical direction near the cascade, but it does not establish the full hypotheses of renormalization theory or prove a one-dimensional reduction of the transformer map. The authors explicitly classify the result as consistency with the universal quadratic class rather than a theorem about the network.

The paper also identifies banded chaos in which the emitted text is exactly periodic while the continuous state is chaotic. In one hostility-steered example, the leading-token sequence had period seven for all 2,000 measured steps, but the largest Lyapunov exponent was $x$6 per step. The state occupied seven separated bands visited cyclically, and a scalar return map sampled once per text period had an estimated exponent within 2% of the full-map exponent after rescaling. Thus, monitoring only the discrete text can miss positive-entropy dynamics in the underlying feedback state. This is a direct consequence of the non-generating nature of the most-probable-token partition on the observed attractor.

Shrimp-shaped windows and coexistence

The paper examines a period-6 window whose parameter-domain geometry resembles the shrimp structures known from low-dimensional maps. The authors require three properties before applying this classification: a primitive periodic orbit of the full transformer state, verified stability, and a boundary organized by continued fold and flip curves. The period-6 orbit satisfies these criteria, and its continued curves form a body, legs, a kite, and a cusp.

The body is bounded by a fold curve on which the period-6 orbit is created or destroyed and by flip curves leading to period-doubled shells. Two flip curves cross near the lower-left region, but this is a pseudo-crossing: the curves belong to two distinct coexisting period-6 orbits rather than to a single orbit with a simultaneous degeneracy. A fold branch turns at a cusp with coordinates approximately $x$7. At this point, the quadratic coefficient of the fold normal form vanishes while the cubic coefficient remains nonzero, and the two-parameter unfolding is numerically regular.

The full map differs from canonical two-dimensional shrimp models in an important respect. The “superstable” lines of a one-dimensional map are replaced by curves where the leading multiplier changes sign, becomes complex, or exchanges dominance with another multiplier. No true one-dimensional head was found. The candidate lines approach and turn away in a Type-B-like geometry, and the paper does not claim an exact two-dimensional reduction. The shrimp classification is therefore organizational and bifurcational, not a statement that the $x$8-dimensional map is dynamically equivalent to a standard two-dimensional model.

The same window exhibits basin coexistence. At one parameter region, two attracting period-6 orbits coexist, and a straight boundary in a fixed-start scan separates which orbit is reached. Near this boundary, trajectories approach a saddle period-6 orbit with unstable multiplier $x$9. The observed closest approach and escape times scale with the perturbation from the boundary as predicted by the saddle’s stable and unstable exponents. Changing the initial state removes the same visible boundary, confirming that it is a basin-selection feature rather than a bifurcation curve. This distinction is central throughout the paper: scans from one fixed start reveal only attractors whose basins contain that start.

Coupled loops and synchronization

The paper extends the analysis to two identical feedback loops coupled through their soft outputs. Each copy receives a convex combination of its own and the other copy’s next state. The synchronized manifold is invariant, and a transverse perturbation evolves approximately according to

$$DF(x) = \frac{1}{T} E^\top \left(\operatorname{diag}(p)-pp^\top\right) J_z(x),$$0

If $$DF(x) = \frac{1}{T} E^\top \left(\operatorname{diag}(p)-pp^\top\right) J_z(x),$$1 is the largest Lyapunov exponent of the uncoupled loop, the predicted transverse exponent is

$$DF(x) = \frac{1}{T} E^\top \left(\operatorname{diag}(p)-pp^\top\right) J_z(x),$$2

Synchronization should therefore become typical above

$$DF(x) = \frac{1}{T} E^\top \left(\operatorname{diag}(p)-pp^\top\right) J_z(x),$$3

At three chaotic cells with single-loop exponents $$DF(x) = \frac{1}{T} E^\top \left(\operatorname{diag}(p)-pp^\top\right) J_z(x),$$4, $$DF(x) = \frac{1}{T} E^\top \left(\operatorname{diag}(p)-pp^\top\right) J_z(x),$$5, and $$DF(x) = \frac{1}{T} E^\top \left(\operatorname{diag}(p)-pp^\top\right) J_z(x),$$6 per step, the predicted thresholds were $$DF(x) = \frac{1}{T} E^\top \left(\operatorname{diag}(p)-pp^\top\right) J_z(x),$$7, $$DF(x) = \frac{1}{T} E^\top \left(\operatorname{diag}(p)-pp^\top\right) J_z(x),$$8, and $$DF(x) = \frac{1}{T} E^\top \left(\operatorname{diag}(p)-pp^\top\right) J_z(x),$$9. Measured transverse growth followed the prediction with maximum discrepancy $p$0 per step. Observed synchronization thresholds generally increased with the single-loop exponent, although the operational threshold depended strongly on tolerance, run length, initial conditions, and the attractor reached.

Near threshold, the coupled system displayed on-off intermittency. In one case, synchronized intervals were interrupted by bursts despite negative typical transverse growth, consistent with attractor bubbling caused by transversely unstable periodic orbits embedded in a chaotic attractor. Some far-start pairs instead synchronized onto a coexisting periodic orbit or settled into periodic joint dynamics without synchronizing. These results validate the local transverse-stability calculation but do not establish global synchronization for arbitrary initial conditions.

Limitations and open questions

The paper’s strongest conclusions concern an exactly specified, fixed-prefix, fixed-window map. They do not directly characterize ordinary autoregressive generation with growing context, token sampling, or systems whose prompt state changes across turns. The authors also study mostly one prompt in the detailed 0.5B analysis, and the model-family and scale comparisons are not controlled as a single factorial experiment.

Chaos is generally inferred from finite-horizon Lyapunov estimates and period searches. Some chaos labels correspond to long transients, and the chaotic attractors are not computer-assisted proofs. By contrast, interval arithmetic certifies three periodic-orbit results: a locally attracting primitive period-8 orbit and coexisting attracting period-2 and period-3 orbits. The certificates rely on stored weights, exact binary64 parameter interpretations, interval-enclosure software, and saved computational artifacts; they establish existence and local attraction in specified small boxes, not the global organization of the surrounding chaotic sets.

Several structural interpretations remain numerical. Feigenbaum convergence is observed only at finite cascade depth. The $p$1 window law is supported by fits but affected by finite-distance corrections and localization uncertainty. The shrimp geometry is consistent with low-dimensional bifurcation anatomy, yet no exact low-dimensional invariant reduction is established. Similarly, the attribution of steering-induced growth to the readout is based on conditional trajectory decompositions rather than an intervention that holds the network dynamics and attractor fixed.

The downstream answer experiments also leave open whether chaotic feedback itself causes task failures. Most changes occurred in regimes with poor baseline performance or budget exhaustion, and accuracy associations disappeared after controlling for difficulty-related covariates. A specific unresolved question is whether the bifurcation and Lyapunov structure can predict task-relevant reliability in feedback pipelines that complete within their computational budgets, rather than merely predicting state divergence or changes in emitted answers under numerical perturbations.

Conclusion

The paper demonstrates that expected-embedding feedback through a pretrained transformer constitutes a nonlinear dynamical system with continuous state and model-dependent bifurcation structure. Its exact Jacobian factorization explains how temperature, token competition, and transformer sensitivity combine to produce contraction, periodicity, and chaos. Across several model families and scales, the maps exhibit periodic windows, saddle-node intermittency, period-doubling cascades with ratios approaching Feigenbaum’s constant, shrimp-shaped parameter domains, and numerically certified attracting periodic orbits. The results establish that nonlinear-dynamics methods can make quantitative predictions about full-transformer feedback maps, while also showing that their conclusions depend critically on the feedback architecture, initialization, precision, and finite-horizon diagnostic used.