Convergence under the deadly triad
Establish general convergence guarantees or characterize necessary and sufficient stability conditions for reinforcement learning algorithms that concurrently employ function approximation, bootstrapping, and off-policy learning (the deadly triad), including deep neural network–based methods.
References
Each is desirable in isolation. Their interaction, known as the deadly triad \citep[Ch.~11]{sutton2018}, is the central open problem in reinforcement learning theory.
Several directions remain open. First, while there is supporting intuition, our analysis does not yet explain the empirical acceleration of dueling Q-learning over standard Q-learning. A related question is whether value sharing can yield sharper problem-dependent or sample-complexity guarantees. Second, extending the analysis of to function approximation is an important direction.