Principled resolution of undesirable CIP maximizers

Develop a principled objective-level resolution for undesirable maximizers of Controllable Information Production, including behaviors such as freely spinning pendulum joints or uncontrolled thrashing in locomotion environments, without relying on environment-level modifications such as joint damping.

Background

Forward CIP can reward dynamically unstable behavior, but the maximizers of the objective are not always behaviorally desirable. In the pendulum systems, an agent may spin freely to accumulate reward, while in the hopper it may obtain high reward through uncontrolled thrashing.

The authors mitigate these behaviors experimentally by adding joint damping, which limits indiscriminate velocity accumulation while preserving rewards for configuration-dependent instability. Because damping changes the environment rather than the intrinsic objective, the paper identifies the development of a principled objective-level solution as unresolved.