Dream to Control: Learning Behaviors by Latent Imagination
This presentation explores Dreamer, a breakthrough model-based reinforcement learning agent that learns complex visual control tasks by imagining trajectories in a compact latent space. By training a world model from pixel observations and then learning behaviors entirely within this model's imagination using analytic gradients, Dreamer achieves state-of-the-art performance on continuous control benchmarks while using 20 times less data than model-free methods and training in just 3 hours per million steps.Script
Most reinforcement learning agents learn by trial and error in the real world, but what if they could learn by dreaming? Dreamer masters complex robotic control tasks by training entirely within its imagination, using 20 times less real-world experience than traditional methods.
The agent operates through three concurrent processes: it learns a world model from past visual experiences, trains an actor and critic purely on imagined trajectories within that model's latent space, and finally executes learned actions in the real environment.
Here's where it gets clever. Starting from any real latent state, Dreamer unrolls imagined trajectories 15 steps into the future using its learned transition model, predicting rewards along the way. The actor learns by backpropagating gradients through these imagined dynamics, a computation that's vastly more efficient in compact latent space than in raw pixel space.
On 20 continuous control tasks from the DeepMind Control Suite, Dreamer achieves an average score of 823, outperforming the model-free baseline's 786, while consuming only 5 million environment steps compared to 100 million. The entire training completes in roughly 3 hours on a single GPU.
The value function is essential for long-horizon reasoning. Without it, the agent is shortsighted, limited strictly by its finite imagination horizon. By learning to estimate rewards beyond what it can directly imagine, Dreamer maintains stable performance even when the imagination horizon is shortened, solving tasks that require extended credit assignment.
By compressing visual complexity into a 30-dimensional latent space and learning entirely within that imagination, Dreamer bridges model-based data efficiency with model-free asymptotic performance. Visit EmergentMind.com to explore this paper in depth and create your own research videos.
