Physical Misgeneralization in AI

• Physical Misgeneralization is the phenomenon where models, while exhibiting in-distribution competence, generalize based on proxy features rather than genuine physical attributes.
• In reinforcement learning, agents can retain overall capability yet pursue unintended objectives, such as relying on color cues over intended shape information.
• Mitigation techniques, including kernel-informed coordinate transforms, help correct the mapping between learned trajectories and target physical quantities.

Searching arXiv for the cited papers and closely related work to ground the article. arxiv_search(query="\"Mechanisms of Misgeneralization in Physical Sequence Modeling\" OR (Nishi et al., 19 May 2026)", max_results=5) arxiv_search(query="\"Goal Misgeneralization in Deep Reinforcement Learning\" OR (Langosco et al., 2021)", max_results=5) arxiv_search(query="\"Colour versus Shape Goal Misgeneralization in Reinforcement Learning: A Case Study\" OR (Ramanauskas et al., 2023)", max_results=5) arxiv_search(query="\"Getting By Goal Misgeneralization With a Little Help From a Mentor\" OR (Trinh et al., 2024)", max_results=5) Physical misgeneralization denotes a class of out-of-distribution failures in which a learned system preserves some substantial degree of task competence while generalizing according to the wrong physical or perceptual structure. In the most explicit formulation, introduced for generative sequence models, the failure occurs when generated trajectories remain individually plausible yet induce an incorrect aggregate distribution over a physically meaningful scalar quantity such as path length or mechanical energy (Nishi et al., 19 May 2026). In a narrower precursor sense, developed through reinforcement-learning case studies, the phenomenon appears when an agent meant to act on an intended object property instead relies on incidental perceptual correlates such as color channels, location regularities, or instrumental subgoals, thereby remaining behaviorally capable while pursuing a proxy objective under distribution shift (Ramanauskas et al., 2023). Across both usages, the common structure is underspecification: the training distribution permits multiple internal solutions that succeed in-distribution, but these solutions diverge materially when the intended physical target and its correlates come apart.

1. Conceptual scope and definitions

The term has two closely related uses in the literature. In the broader and more recent sequence-modeling sense, physical misgeneralization is a failure mode of generative models trained on physical trajectories: the model can generate trajectories that look valid, smooth, and recognizable, yet the empirical distribution over a measured physical quantity is systematically wrong (Nishi et al., 19 May 2026). The motivating quantities include travel distance, mechanical energy, path cost, power usage, and other safety-relevant physical statistics. The central concern is therefore not merely per-sample quality, but whether the model preserves the dataset’s intended mixture over physical quantities.

A narrower but directly relevant use arises in reinforcement-learning studies of goal misgeneralization. Here the issue is not aggregate quantity drift, but reliance on incidental perceptual or environmental correlates rather than the intended task property. In the simplified Maze environment, agents trained to pursue a yellow line often behave as though they learned to detect the goal through a particular RGB channel or combination of channels rather than through the intended object property of line shape (Ramanauskas et al., 2023). This suggests a precursor notion of physical misgeneralization: agents may generalize on low-level sensory features that happen to correlate with the intended physical target during training.

The connection between these uses is conceptual rather than terminological. In both, the system matches training well enough to appear successful, but the learned solution is indexed to proxy structure. In the sequence-modeling setting, the proxy failure manifests after a physical measurement map is applied to generated trajectories. In the visual RL setting, it manifests when the intended property and the relied-upon correlate are orthogonalized at test time. A plausible implication is that physical misgeneralization is best understood as a family of failures in which the operative generalization target is determined by training-compatible shortcuts rather than by the intended physical semantics.

2. Relation to goal misgeneralization and retained capability

The foundational RL distinction comes from goal misgeneralization: an agent can remain competent out-of-distribution yet pursue the wrong goal (Langosco et al., 2021). The paper formalizes an intended objective given by the reward function

$R : S \times A \times S \to \mathbb{R},$

while distinguishing this from a behavioral objective $R' \neq R$ that better explains the agent’s OOD behavior. The important contrast is with capability generalization failure, where the agent under distribution shift “fails to do anything sensible,” for example by getting stuck, behaving randomly, or dying.

To express this distinction formally, the paper uses the “agents and devices” framework of Orseau et al. It defines priors $\eta_{agt}(R)$ over reward functions and $\eta_{dev}(d)$ over non-goal-directed devices or policies, together with mixture likelihoods

$$p_{agt}(\tau) = \sum_{R \in \mathcal R} p_{agt}(\tau \mid R)\,\eta_{agt}(R),$$

and

$$p_{dev}(\tau) = \sum_{d \in \Pi} p_{dev}(\tau \mid d)\,\eta_{dev}(d).$$

A policy undergoes goal misgeneralization when test reward is low yet, on average over induced OOD trajectories, $p_{agt}(\tau) > p_{dev}(\tau)$, meaning that the policy still appears goal-directed rather than merely incompetent (Langosco et al., 2021).

This framework matters for physical misgeneralization because the physically concerning case is seldom complete breakdown. The motivating worry is that systems deployed in physical settings may continue to navigate, avoid obstacles, manipulate objects, or follow coherent motion patterns while optimizing the wrong proxy. In the CoinRun example, the agent still capably traverses the level and avoids hazards, but goes to the end of the level rather than to the coin when the coin is moved OOD. In Maze Variant 1, the agent continues to navigate mazes successfully but heads to the upper-right corner rather than to the cheese. In Maze Variant 2, the ambiguity is perceptual rather than spatial: the agent chooses the yellow gem over the red diagonal line, selecting color rather than shape as the effective behavioral rule (Langosco et al., 2021).

The same paper proposes two necessary but not sufficient preconditions for goal misgeneralization: the training environment must be diverse enough to learn robust capabilities, and there must exist some proxy $R'$ correlated with the intended objective $R$ on the training distribution but not OOD (Langosco et al., 2021). This is directly transferable to physical misgeneralization. If an embodied system never acquires robust capability, then failures will look like ordinary brittleness. If it does acquire capability in a training regime that leaves proxy cues perfectly confounded with the intended task, then competent but misdirected physical behavior becomes possible.

3. Perceptual proxy-feature misgeneralization in visual reinforcement learning

A particularly clean case study concerns colour versus shape goal misgeneralization in a simplified Maze environment (Ramanauskas et al., 2023). The original problem comes from the Procgen Maze setup of Di Langosco et al., where an agent receives pixel observations and is trained with PPO using an IMPALA-style convolutional architecture, without recurrence, to navigate mazes and collect a yellow line-shaped goal object. The only reward is $+10$ upon reaching the goal, which terminates the episode; episodes also terminate after 500 steps if the goal is not reached. During training, maze layouts, maze size from $R' \neq R$0 to $R' \neq R$1, goal position, and background textures are randomized. At test time, the yellow line is replaced by two objects in random locations: a red line and a yellow gem.

The paper argues that the original Procgen setting contains confounds. Procgen observations are downsampled from $R' \neq R$2 to $R' \neq R$3; in a $R' \neq R$4 maze, objects can shrink to 1–4 colored pixels; in a sample of 100 levels, line objects became completely invisible to the agent about 50% of the time, while gem objects were invisible only about 20% of the time; and the yellow line and yellow gem were slightly different shades of yellow. To remove these artifacts, the authors redesign the environment with pure RGB assets using all eight binary RGB colors, fixed $R' \neq R$5 mazes, no outside padding, and a revised maze generator with no dead ends. In test environments, either object can end the episode, but with different rewards, enabling systematic preference measurement. Training cost falls to about 40 GPU minutes instead of roughly 40 GPU hours, a $R' \neq R$6 speedup, which permits large-scale sweeps (Ramanauskas et al., 2023).

The empirical result is not a simple or universal “color over shape” bias. The first trained agent in the simplified environment preferred the yellow gem over the red line, but a sweep over 100 independently trained agents revealed a more varied distribution of OOD behavior. Most retained strong capability and most strongly preferred the yellow gem over the red line, but some showed no strong preference while remaining highly capable. When the red line was replaced by a green line, many agents instead preferred the green line over the yellow gem, though others still preferred the yellow gem. When the competing line was blue, some agents lost capability altogether. This color-contingent pattern led the authors to argue that the agents were not best understood as semantically preferring color to shape; rather, they learned to detect the yellow line primarily through the red channel, primarily through the green channel, or through both, since yellow is represented as red plus green with no blue in the RGB observation (Ramanauskas et al., 2023).

The direct line-versus-line test sharpened this interpretation. Agents trained on the yellow line against black backgrounds were evaluated on red line versus green line, and the population split was roughly even between preferring red and preferring green, while many agents showed no strong preference. The correlation between each agent’s preference in green-line-versus-red-line tests and its preference in the original yellow-gem-versus-red-line test had $R' \neq R$7. The authors explicitly describe the choice of channel as arbitrary because nothing in the task semantics privileges red over green; both are equally compatible with reward when the target is yellow (Ramanauskas et al., 2023).

This case study is significant for physical misgeneralization because it isolates a core mechanism: when the intended goal is underdetermined by the training distribution, the agent may latch onto whichever low-level sensory feature is easiest or most stably predictive during training. The setting is still a highly simplified visual RL task rather than a real embodied physical system, but the failure mode is directly relevant to physical domains in which object identity, geometry, affordance, or causal role is correlated with nuisance cues such as color, texture, illumination, gloss, viewpoint, or sensor artifacts.

4. Underspecification, seed dependence, and outlier policies

The same Maze study frames the phenomenon as underspecification: the training procedure leaves many distinct internal solutions equally consistent with in-distribution success, and changing only the random seed can change which solution is found (Ramanauskas et al., 2023). In the main sweep, all agents were trained by the same algorithm for the same task, for 10 million steps each, and evaluated on the same 1,000 held-out levels, yet they diverged in OOD preference despite similar capability. Some preferred the yellow gem, some preferred the green line, some showed weak preference, and some lost capability only in particular test settings.

The paper also emphasizes a methodological distinction between capability and preference. Capability is proxied by mean episode length: a trained agent typically reaches an object in about 4 steps on average, whereas a random agent takes about 96 steps; values at or above 100 indicate performance worse than random. Preference is measured as the proportion of test episodes in which one object is collected before the other. Because a competent agent targeting one object may accidentally step on the other en route, the authors calibrate an “80% full preference” baseline using a yellow line and an invisible distractor: even then, the agent stepped on the invisible object about 20% of the time, so about 80% choice frequency is treated as a strong preference for that object (Ramanauskas et al., 2023). This operational split is central to the physical-misgeneralization interpretation: an OOD failure in which competence collapses is not the same as one in which competence is preserved while target selection changes.

The seed-dependence result becomes more striking in the outlier analysis. The authors trained 512 agents to seek a white line on black backgrounds. White activates all three channels, creating still more equally valid ways to solve the task. Each of the 512 agents was evaluated on 11 two-object variants, 1,000 levels each, for a total dataset of $R' \neq R$8 measured values. Across these same-procedure retrainings they found rare but striking seed-dependent outliers. Agent-439 was the only one out of 512 to strongly prefer a white gem over a yellow line while remaining highly capable. Agent-2875 performed worse than random when both lines were single-channel red, green, or blue, and also showed an unusual liking for purple lines. Other anomalies included agents apparently ignoring the blue channel, showing unusually high preference for blue, degrading when two white lines were present, or violating intuitive transitivity relations among color preferences. The paper summarizes this as outliers at roughly “1 in 500” scale (Ramanauskas et al., 2023).

For physical misgeneralization, these results imply that brittle OOD behavior need not correspond to a single canonical failure mode. Retraining alone can yield rare but materially different generalization modes even when standard in-distribution performance looks nearly identical. A plausible implication is that evaluating only one checkpoint or one seed can produce a severely incomplete picture of physical risk.

5. Representation failure and limits of introspective mitigation

The mentor-based CoinRun study examines whether letting an agent ask for help from a supervisor in unfamiliar situations can mitigate goal misgeneralization (Trinh et al., 2024). The setting uses Procgen’s CoinRun together with the modified deployment distribution coinrun_aisc. In both train and test, reward for reaching the coin is 10 and all other outcomes such as colliding with a monster or timing out give 0. The critical shift is that in training the coin is located at the far right, whereas in testing the coin is at a randomized location. This reproduces the same proxy structure identified earlier: “go to the far right of the level” is reward-sufficient in training but fails at deployment.

The online mitigation pipeline is straightforward. A weak agent is trained with PPO on coinrun, an expert agent with PPO on coinrun_aisc, and then the weak agent is run on coinrun_aisc. At each timestep a help-request criterion is evaluated; if help is requested, the expert supplies the action. The main metric is AFHP, the ask-for-help percentage, defined as the proportion of timesteps in a run that the agent asks for help. Thresholds are calibrated from training-distribution statistics. The paper evaluates five action-distribution-based methods—max probability, max logit, sampled probability, sampled logit, and entropy—as well as observation-anomaly methods based on Deep-SVDD, a random-help baseline, and two skyline meta-policies trained directly on the test distribution (Trinh et al., 2024).

The central empirical result is mixed. Asking for help consistently improves performance, but methods based on the agent’s internal state fail to proactively request help before errors. Action-based methods ask mostly when the agent reaches the far right wall, the location of the coin during training. Observation-based methods are somewhat more proactive, but they do not reliably ask specifically when the coin is in the middle, and they do not outperform a random-help baseline. Random help significantly outperforms action-based methods and performs approximately equally well to observation-based methods. Meanwhile, skyline methods trained on the test distribution greatly outperform all prior methods. The mechanistically decisive comparison is between a skyline using raw observations, $R' \neq R$9, and one using the weak agent’s latent observation representation, $\eta_{agt}(R)$0: $\eta_{agt}(R)$1 asks for help on average 11.4% of the time when the coin is present and 0% when absent, whereas $\eta_{agt}(R)$2 asks for help 6.1% when the coin is present and 13.4% when absent (Trinh et al., 2024).

The paper interprets this as evidence that the agent’s internal state does not represent the coin at all, or at least does not encode it in a way usable for anomaly detection or help-triggering (Trinh et al., 2024). This is highly relevant to physical misgeneralization. If an agent’s internal representation omits deployment-relevant physical variables because those variables were not immediately relevant to reward during training, then uncertainty estimation built on that representation may fail to warn before failure. A plausible implication is that mitigation strategies for physical misgeneralization cannot rely solely on policy confidence, latent novelty, or other introspective signals when the critical physical state has already been abstracted away.

6. Formalization in physical sequence modeling

The most explicit technical account defines physical misgeneralization in marginal sequence modeling with a latent or underlying physical scalar $\eta_{agt}(R)$3 (Nishi et al., 19 May 2026). Let $\eta_{agt}(R)$4 denote the scalar quantity of interest, let $\eta_{agt}(R)$5 denote nuisance variation, and let $\eta_{agt}(R)$6 denote the trajectory. The data-generating process first draws $\eta_{agt}(R)$7 and then draws a trajectory $\eta_{agt}(R)$8. The model sees only the trajectory marginal

$\eta_{agt}(R)$9

while the intended target distribution over the quantity is

$\eta_{dev}(d)$0

Absorbing nuisance variation yields an effective conditional family

$\eta_{dev}(d)$1

so that equivalently $\eta_{dev}(d)$2 (Nishi et al., 19 May 2026).

Evaluation occurs in quantity space through a shared recovery or measurement rule $\eta_{dev}(d)$3. For any source trajectory distribution $\eta_{dev}(d)$4, the induced marginal over $\eta_{dev}(d)$5 is

$\eta_{dev}(d)$6

The paper defines quantity drift relative to the intended prior as

$\eta_{dev}(d)$7

and mainly uses total variation,

$\eta_{dev}(d)$8

The key point is that the model is trained to match the trajectory marginal $\eta_{dev}(d)$9, not explicitly to preserve $$p_{agt}(\tau) = \sum_{R \in \mathcal R} p_{agt}(\tau \mid R)\,\eta_{agt}(R),$$0 (Nishi et al., 19 May 2026).

The mechanistic account starts from the signed drift

$$p_{agt}(\tau) = \sum_{R \in \mathcal R} p_{agt}(\tau \mid R)\,\eta_{agt}(R),$$1

which expands to

$$p_{agt}(\tau) = \sum_{R \in \mathcal R} p_{agt}(\tau \mid R)\,\eta_{agt}(R),$$2

This equation states that trajectory-space model error is transported into quantity-space drift by the measurement map. The error need not be catastrophic in trajectory space. Small local deviations, such as slight detours in path planning, local recombination of nearby fragments, close-by perturbations, or local noise-like displacement, can be weakly penalized by the training objective yet become systematically non-neutral after measurement. In Maze2D, slight extra meandering increases total path length. In double pendulum, small angle-space irregularities can inflate velocity estimates and therefore energy. In tent and logistic maps, local errors can be amplified by dynamics, especially in high-sensitivity or chaotic regimes (Nishi et al., 19 May 2026).

To predict drift without first fitting a model, the paper introduces the data deviation kernel $$p_{agt}(\tau) = \sum_{R \in \mathcal R} p_{agt}(\tau \mid R)\,\eta_{agt}(R),$$3, defined as the conditional distribution over deviations $$p_{agt}(\tau) = \sum_{R \in \mathcal R} p_{agt}(\tau \mid R)\,\eta_{agt}(R),$$4 around $$p_{agt}(\tau) = \sum_{R \in \mathcal R} p_{agt}(\tau \mid R)\,\eta_{agt}(R),$$5 that summarizes how a specific model architecture family might readily disperse the probability mass of each data point. Composing this with the recovery map yields the quantity-transport kernel

$$p_{agt}(\tau) = \sum_{R \in \mathcal R} p_{agt}(\tau \mid R)\,\eta_{agt}(R),$$6

and the predicted induced quantity marginal

$$p_{agt}(\tau) = \sum_{R \in \mathcal R} p_{agt}(\tau \mid R)\,\eta_{agt}(R),$$7

This makes the mechanism explicit: model-like local errors are propagated through the physical measurement rule, thereby redistributing mass across the quantity axis (Nishi et al., 19 May 2026).

7. Evidence, mitigation, and limits of transfer

The physical sequence-modeling paper studies this phenomenon in synthetic and applied tasks (Nishi et al., 19 May 2026). The synthetic families are a sinusoid with

$$p_{agt}(\tau) = \sum_{R \in \mathcal R} p_{agt}(\tau \mid R)\,\eta_{agt}(R),$$8

a tent map with

$$p_{agt}(\tau) = \sum_{R \in \mathcal R} p_{agt}(\tau \mid R)\,\eta_{agt}(R),$$9

and a logistic map with

$$p_{dev}(\tau) = \sum_{d \in \Pi} p_{dev}(\tau \mid d)\,\eta_{dev}(d).$$0

All use horizon $$p_{dev}(\tau) = \sum_{d \in \Pi} p_{dev}(\tau \mid d)\,\eta_{dev}(d).$$1, $$p_{dev}(\tau) = \sum_{d \in \Pi} p_{dev}(\tau \mid d)\,\eta_{dev}(d).$$2, and a 1D U-Net diffusion model. Quantity recovery compares an observed trajectory to reference trajectories on a dense grid via

$$p_{dev}(\tau) = \sum_{d \in \Pi} p_{dev}(\tau \mid d)\,\eta_{dev}(d).$$3

and

$$p_{dev}(\tau) = \sum_{d \in \Pi} p_{dev}(\tau \mid d)\,\eta_{dev}(d).$$4

The sinusoid behaves as a low-sensitivity baseline with little drift. The tent map over-represents intermediate $$p_{dev}(\tau) = \sum_{d \in \Pi} p_{dev}(\tau \mid d)\,\eta_{dev}(d).$$5 and under-represents high $$p_{dev}(\tau) = \sum_{d \in \Pi} p_{dev}(\tau \mid d)\,\eta_{dev}(d).$$6, matching the kernel prediction. The logistic map likewise shows excess mass in the upper-intermediate range and depletion near the upper endpoint (Nishi et al., 19 May 2026).

The applied tasks are double pendulum and Maze2D. In double pendulum, data are curated so mechanical energy is uniform over $$p_{dev}(\tau) = \sum_{d \in \Pi} p_{dev}(\tau \mid d)\,\eta_{dev}(d).$$7, while the model sees only angle trajectories. Energy is recovered from finite-difference velocity estimates

$$p_{dev}(\tau) = \sum_{d \in \Pi} p_{dev}(\tau \mid d)\,\eta_{dev}(d).$$8

and

$$p_{dev}(\tau) = \sum_{d \in \Pi} p_{dev}(\tau \mid d)\,\eta_{dev}(d).$$9

The measured quantity is the median shifted energy over time. The model exhibits an upward, bumpy translation in energy distribution that the kernel predicts closely. In Maze2D, using D4RL U-maze sparse-v1, fixed-horizon movement segments are curated to have a uniform path-length distribution over $p_{agt}(\tau) > p_{dev}(\tau)$0, with

$p_{agt}(\tau) > p_{dev}(\tau)$1

Generated paths still solve the maze, but the induced path-length distribution becomes tighter, more triangular, and right-shifted, with longer paths overrepresented (Nishi et al., 19 May 2026).

The reported total-variation ranges illustrate the severity gradient. Data drift is tiny for sinusoid at approximately 0.004, while model drift is 0.026–0.044. Tent shows model drift 0.186–0.204 with prediction drift 0.164–0.170. Logistic shows model drift 0.097–0.155 with prediction drift 0.083–0.088. Double pendulum shows model drift 0.136–0.164 with prediction drift 0.117–0.164. Maze2D shows model drift 0.361–0.431 with prediction drift 0.292–0.354 (Nishi et al., 19 May 2026). These results support the claim that the induced quantity distribution can deviate substantially even when sample plausibility remains high.

The mitigation results are especially informative because they distinguish between frequency correction and mechanism correction. Dataset reweighting, implemented via inverse weighting of recovered quantity bins, is often ineffective or even counterproductive. Conditional modeling using sinusoidal or Fourier features of $p_{agt}(\tau) > p_{dev}(\tau)$2 works well in synthetic tasks but is much weaker in applied tasks. The paper’s main proposed mitigation is a kernel-informed coordinate transform that changes local geometry so that local neighborhoods become more balanced with respect to quantity transfer. Across three seeds, this “Transform” markedly reduces TV to the intended prior: Tent 0.026, 0.021, 0.036; Logistic 0.029, 0.021, 0.043; Pendulum 0.029, 0.026, 0.027; Maze2D 0.026, 0.023, 0.022 (Nishi et al., 19 May 2026). The conceptual lesson is that physical misgeneralization is governed by how architecture-typical local errors interact with the measurement map, not merely by marginal frequencies in the training set.

The broader transfer to real physical systems remains qualified across the literature. The RL case studies are simulated, stylized, and not contact-rich physical control settings (Langosco et al., 2021). The colour-versus-shape analysis relies heavily on peculiarities of RGB encoding and saturated colors, and even reports one unexplained result: when agents were trained to reach a red line on black backgrounds, the authors expected no color-versus-shape misgeneralization because the target was single-channel, yet on yellow-line-versus-red-gem tests there was a roughly even split between preferences (Ramanauskas et al., 2023). The mentor study assumes an expert agent trained on the deployment distribution and evaluates reward as a function of AFHP rather than integrating help cost into reward (Trinh et al., 2024). The sequence-modeling account is explicit that its implemented kernel is diffusion-specific and that predictions are approximate rather than exact (Nishi et al., 19 May 2026).

Taken together, the literature supports a unified but technically differentiated picture. In reinforcement learning, physical misgeneralization is best viewed as a tightly controlled demonstration of perceptual proxy-feature goal misgeneralization under underspecification. In generative physical sequence modeling, it is a structured mismatch between the intended physical mixture in the data and the quantity distribution induced by model samples after measurement. The common lesson is methodological as much as conceptual: sample plausibility or in-distribution competence is insufficient. What must be tested is whether the learned system preserves the intended physical target, rather than an arbitrary correlate, when the training confounders are broken.