Provably Optimal Learning Algorithms for Assistance Games

Abstract: This paper studies an online variant of the assistance games framework, where an informed agent and an uninformed agent repeatedly interact over $T$ timesteps to optimize a common reward function. While the informed agent (the human) observes a latent state of the world, the uninformed agent (the assistant) observes only the human's actions. We provide the first provably efficient learning algorithms for repeated assistance games. We introduce the notion of assistance regret: the gap between the cumulative utility of interactions and that of the optimal joint policies in hindsight, which map latent states to action pairs. We present decentralized algorithms for both the human and the assistant that achieve a $(1-1/e)$-approximate assistance regret rate of $\widetilde{O}(T<sup>{3/4})$, with runtime polynomial in the size of the action and state spaces. These algorithms are general; in particular, they accommodate any no-regret algorithm for the assistant. We prove that achieving a regret approximation factor better than $(1-1/e)$ is computationally intractable. Furthermore, we demonstrate how these generic no-regret algorithms can be tailored to a pseudo-decentralized setting -- using a shared random string -- to achieve a rate of $\widetilde{O}(T<sup>{1/2})$, optimal up to logarithmic factors.

• The paper introduces a two-stage reduction using online submodular maximization and decentralized decomposition to achieve a (1 - 1/e)-approximate assistance regret.
• It presents polynomial-time algorithms that manage the exponential policy space by balancing stability with adaptivity, ensuring sublinear cumulative regret.
• The study establishes that surpassing the (1 - 1/e) approximation is computationally intractable, setting explicit complexity lower bounds.

Provably Optimal Learning Algorithms for Assistance Games: Technical Overview

Problem Setting and Formal Model

The paper "Provably Optimal Learning Algorithms for Assistance Games" (2607.08012) develops a rigorous online-learning theory for assistance games, a framework for modeling cooperative interaction between an informed human and an uninformed assistant agent with a shared utility but asymmetric information. Specifically, the latent state (e.g., user preferences) is observed only by the human and changes arbitrarily over rounds, while the assistant observes only the human’s actions and the realized reward. The objective is to design learning dynamics for both agents that jointly maximize cumulative reward, even as latent states drift nonstationarily and neither agent observes the other's internal information.

Policy spaces are exponentially large: the human’s policy maps $\Theta$ (latent states, size $N$) to $\mathcal{A}_H$ (human actions, size $M_H$) and the assistant’s policy maps $\mathcal{A}_H$ to $\mathcal{A}_A$ (assistant actions, size $M_A$). Feedback is asymmetric and partial: the human receives full-information feedback, while the assistant observes only realized rewards and cannot reconstruct counterfactuals.

Performance is measured by $\alpha$-approximate assistance regret: the (scaled) gap between cumulative reward and the reward achievable by the best joint policy in hindsight. The aim is to design decentralized, computationally efficient online algorithms that achieve sublinear assistance regret (i.e., converge to optimal joint play).

Main Algorithmic Contributions

The paper's central algorithmic advance is a two-stage reduction:

1. Centralized reduction to online submodular maximization. Joint optimal policy computation is cast as online maximization of a submodular utility function under partition matroid constraints. Specifically, the space of joint policies forms a partition matroid, and the joint reward as a function of assistant's (and thus joint) policies is a monotone submodular function, specifically a weighted threshold potential. This structure enables tractable online learning over an exponentially large policy space.
2. Decentralized decomposition (stable-adaptive reduction). The transition from centralized to decentralized learning is handled by decomposing the regret into terms associated with the centralized algorithm’s external regret, its policy switch count (stability), and the assistant’s (tracking) regret relative to a moving target (adaptivity). This modular decomposition clarifies how to break up the global coordination challenge into independent stability and tracking components for the two agents.

Centralized Learning Algorithm

Through the structural reduction, the centralized online assistance game admits polynomial-time algorithms (using, for instance, Randomized-Augmented OCO over concave relaxations of the reward structure) that achieve sublinear $(1-1/e)$-approximate assistance regret of the form:

$\mathrm{Regret}_T \le O(M_H \sqrt{T \log M_A}).$

This leverage of submodular maximization under matroid constraints is crucial: it renders the otherwise intractable joint policy optimization problem feasible.

Decentralized Decoupling and Assistance Regret Bounds

In the actual online interactive protocol, learning must be decentralized. The stable-adaptive decomposition allows the design of two classes of algorithms with strong provable guarantees:

• General decentralized algorithms achieve $N$0-approximate assistance regret $N$1, with runtime polynomial in $N$2, $N$3, $N$4, and $N$5. These exploit any no-regret assistant learning algorithm, so are highly modular.
• Improved decentralized algorithms (pseudo-decentralized setting), where agents share a pre-established communication protocol (a random string encoding policy switch messages), achieve $N$6-approximate assistance regret $N$7, matching the optimal minmax regret up to logarithmic factors.

Notably, in both cases, the best achievable approximation factor is $N$8, and no polynomial-time algorithm can guarantee uniform sublinear regret at any $N$9 unless $\mathcal{A}_H$0 (via reduction from the hardness of submodular maximization exceeding the $\mathcal{A}_H$1-approximation threshold).

Tightness, Optimality, and Hardness

Approximation Guarantees and Complexity Separations

The $\mathcal{A}_H$2 factor is tight for computational tractability: achieving sublinear $\mathcal{A}_H$3-approximate regret for any $\mathcal{A}_H$4 would violate established complexity lower bounds for (offline) $\mathcal{A}_H$5-maximum coverage and related submodular maximization problems. The reduction argument is explicit and formalized in the paper.

Information-Theoretic Barriers

If the underlying latent state is selected adaptively based on past interaction, rather than being fixed in advance (the "oblivious" setting), then it is information-theoretically impossible for any learning protocol (even computationally unbounded) to achieve sublinear regret. This is shown explicitly for simple signaling games via combinatorial lower bounds.

Algorithmic Techniques: Stability and Adaptivity

The paper introduces an insightful decomposition of the communication challenge: coordination costs are bounded in terms of the centralized learner’s number of policy switches (stability) and the assistant’s tracking regret (adaptivity). By controlling each of these using advances from online convex optimization (lazy OCO for few switches; black-box tracking regret minimization), the decentralized regret is jointly minimized.

• The assistant’s tracking regret is handled by running $\mathcal{A}_H$6 independent copies of a black-box tracking regret minimizer (e.g., Fixed-Share), indexed by observed human actions.
• The human leverages the centralized optimizer, but, to maintain stability, only switches rarely (sublinear number of times).

With a shared randomness protocol, coordination can be made nearly perfect (logarithmic communication cost for each policy switch), driving the rate down to the minimax-optimal regret in $\mathcal{A}_H$7.

Numerical Guarantees and Strong Claims

• No computationally efficient algorithm achieves sublinear $\mathcal{A}_H$8-approximate assistance regret for $\mathcal{A}_H$9 (unless $M_H$0).
• The optimal regret scaling in $M_H$1 is $M_H$2 modulo logarithmic factors, matching lower bounds from online learning theory.
• The algorithms' runtime is polynomial in the parameters of the game, despite the exponential size of the joint policy space.

Theoretical and Practical Implications

The results demonstrate that exploiting submodularity and matroid structure is a powerful formal tool in cooperative multi-agent learning under asymmetric information. The separation of stability and adaptivity as generic design goals further guides algorithm construction. Practically, these methods provide the first principled, provably efficient learning rules for assistive AI agents operating with bandit feedback, as in preference elicitation or human-AI collaboration settings.

Open Problems and Future Directions

• Tightening the dependence of regret bounds on $M_H$3 and $M_H$4 remains unresolved.
• Minimizing coordination overhead in the absence of pre-shared communication protocols and extending techniques to richer models (e.g., long-horizon POMDPs or two-sided partial observability) are promising directions.
• Generalizing the approach to other classes of asymmetric-information cooperative games or extending to learning under richer feedback models presents significant opportunities.

Conclusion

This paper closes an important theoretical gap in online assistance games by providing the first decentralized, computationally efficient learning algorithms with provable optimality guarantees, matching lower bounds both in approximation factor and regret rate. The technical approach—leveraging submodular maximization reductions and a modular decomposition of coordination challenges—establishes a rigorous, extensible foundation for the design of learning algorithms in cooperative AI systems with asymmetric information.