Scaling abstract world-model representations across domains and modalities
Develop scalable methods for learning compact, abstract world-model representations that discard irrelevant details and generalize across arbitrary domains and modalities, rather than relying on approaches that preserve complete observation information via reconstructable latent representations or raw data.
References
Although psychology and cognitive science suggest that human mental models rely on compact representations that discard irrelevant details, how to scale approaches capable of learning such abstract representations to arbitrary domains and modalities is still unclear.
Aligning such representations with the code world's state, and adapting pretrained video models to use them, remain open problems.
Future work includes scaling up the pre-training data to explore whether this compact representation is able to generalize to open-world environments.