Repeated Assistance Games Overview

• Repeated assistance games are sequential coordination models where informed humans guide uncertain assistants to achieve shared goals.
• These games incorporate multi-principal and two-agent frameworks, blending social choice mechanisms with online learning for improved preference inference.
• The repeated interaction structure enables advanced algorithms to balance exploration and immediate rewards, enhancing incentive alignment and scalable planning.

Repeated assistance games are sequential or online variants of assistance games, also known as cooperative inverse reinforcement learning games, in which an informed human principal and an uninformed assistant repeatedly interact to optimize a shared objective while the assistant must act under uncertainty about the relevant reward parameter or private reward functions (Fickinger et al., 2020). In the contemporary literature, the term covers both repeated multi-principal settings, where a robot learns from the actions of several humans over time, and online two-agent settings, where a human observes a latent state and an assistant repeatedly responds to human actions without observing that latent state directly (Ananthakrishnan et al., 9 Jul 2026). The central technical theme is that repetition changes both inference and incentives: it yields more observations for preference learning, but it also makes demonstrations consequential, strategic, and potentially manipulable. Recent work treats this repeated structure not as a complication to be abstracted away, but as the main design resource for preference revelation, social choice, regret minimization, and scalable planning (Fickinger et al., 2020).

1. Formal foundations

Assistance games are built on an MDP without a reward function,

$M\setminus R = \langle S, A, P, \mu_0, T\rangle,$

together with a human principal whose reward is private and a robot that must infer that reward from human behavior. In the classical single-principal case, the human has a private reward function $R^*: S \to \mathbb{R}$, the robot maintains uncertainty over $R^*$, and the mechanism maps observed trajectories to a policy that is good for the human’s true reward (Fickinger et al., 2020). Trajectory space is

$\Xi = (S\times A)^{T-1}\times S,$

and the paper formalizes an IRL algorithm as a mapping

$IRL:\mathcal{P}(\Xi)\to \mathbb{R}^S,$

or more generally to a distribution over rewards.

The multi-principal generalization replaces the single reward with $N$ private reward functions,

$$R_h^* : S \to \mathbb{R}, \qquad h \in \{1,\dots,N\},$$

and introduces a social welfare function

$W(R_1^*,\dots,R_N^*).$

The robot’s objective is then to maximize the induced aggregate reward $R^* = W(R_1^*,\dots,R_N^*)$. Canonical examples are utilitarian aggregation,

$W_U(R_1^*,\dots,R_N^*)(s) = \sum_{h=1}^N R_h^*(s),$

and egalitarian aggregation,

$R^*: S \to \mathbb{R}$0

A mechanism

$R^*: S \to \mathbb{R}$1

maps observed human trajectories to a possibly stochastic robot policy (Fickinger et al., 2020).

A complementary online formalization models a repeated two-agent interaction over $R^*: S \to \mathbb{R}$2 rounds. There is a latent preference or state space $R^*: S \to \mathbb{R}$3, human and assistant action spaces $R^*: S \to \mathbb{R}$4 and $R^*: S \to \mathbb{R}$5, and a common reward function

$R^*: S \to \mathbb{R}$6

In each round, nature fixes $R^*: S \to \mathbb{R}$7, the human observes $R^*: S \to \mathbb{R}$8, chooses $R^*: S \to \mathbb{R}$9, the assistant observes $R^*$0 but not $R^*$1, then chooses $R^*$2, and both receive $R^*$3. Human policies are mappings $R^*$4, assistant policies are mappings $R^*$5, and the benchmark is the best fixed joint policy pair in hindsight (Ananthakrishnan et al., 9 Jul 2026).

These formulations share the same structural asymmetry: the human’s action is both payoff-relevant and informational, while the assistant’s task is to convert repeated observation of human behavior into high-welfare action under latent-goal uncertainty.

2. Repeated interaction architectures

The main repeated model in multi-principal assistance is the multi-principal bandit apprentice (MPBA). The environment has a finite set of arms $R^*$6, rounds $R^*$7, and $R^*$8 principals with deterministic arm-specific utilities

$R^*$9

Social welfare per arm is

$\Xi = (S\times A)^{T-1}\times S,$0

At each round, the robot chooses who has control next, either a human $\Xi = (S\times A)^{T-1}\times S,$1 or itself, the controlling agent pulls an arm $\Xi = (S\times A)^{T-1}\times S,$2, the robot observes $\Xi = (S\times A)^{T-1}\times S,$3 but not the realized individual reward, and the realized welfare is $\Xi = (S\times A)^{T-1}\times S,$4. The robot’s objective is social-welfare regret minimization,

$\Xi = (S\times A)^{T-1}\times S,$5

This is explicitly a repeated assistance game: the robot repeatedly allocates control, infers preferences from actions, and trades off exploration and exploitation (Fickinger et al., 2020).

The non-strategic baseline assumes a softmax human policy,

$\Xi = (S\times A)^{T-1}\times S,$6

with $\Xi = (S\times A)^{T-1}\times S,$7 corresponding to almost deterministic best-arm play and finite $\Xi = (S\times A)^{T-1}\times S,$8 to noisy optimality. The repeated structure then turns empirical arm frequencies into a preference signal.

A second repeated architecture appears in the collegial-mechanism formulation of multi-principal apprenticeship learning. There, the environment is a finite-horizon MDP $\Xi = (S\times A)^{T-1}\times S,$9, each human $IRL:\mathcal{P}(\Xi)\to \mathbb{R}^S,$0 has a private reward $IRL:\mathcal{P}(\Xi)\to \mathbb{R}^S,$1, each human provides one trajectory $IRL:\mathcal{P}(\Xi)\to \mathbb{R}^S,$2, and the robot maps the trajectory profile to a stochastic trajectory via

$IRL:\mathcal{P}(\Xi)\to \mathbb{R}^S,$3

The repeated aspect exists both within each trajectory and, in the collaboration mechanism, across many rounds of stateless interaction: at each time step the robot chooses a human $IRL:\mathcal{P}(\Xi)\to \mathbb{R}^S,$4, the human chooses an action $IRL:\mathcal{P}(\Xi)\to \mathbb{R}^S,$5, the robot decides whether to allocate the corresponding reward now, and eventually chooses a final action $IRL:\mathcal{P}(\Xi)\to \mathbb{R}^S,$6 whose social welfare is evaluated (Fickinger et al., 2020).

These models differ in state structure and observability, but they share a common temporal logic: human work is repeated, directly consequential, and simultaneously instrumental for current reward and future preference inference.

3. Strategic incentives and social choice under repetition

Multi-principal assistance is also a social choice problem. In one-shot apprenticeship-style mechanisms, Gibbard–Satterthwaite-type impossibility results apply. The literature defines a human’s total utility as

$IRL:\mathcal{P}(\Xi)\to \mathbb{R}^S,$7

and defines straightforward mechanisms by dominant strategies for each human. Under small $IRL:\mathcal{P}(\Xi)\to \mathbb{R}^S,$8, the only straightforward deterministic mechanisms are duple mechanisms or dictatorships, and the nondeterministic case reduces to mixtures of duple mechanisms and unilateral mechanisms. In general multi-principal assistance, truthful revelation via demonstrations is therefore impossible to guarantee on universal preference domains (Fickinger et al., 2020).

Repeated assistance changes this conclusion because demonstrations become costly. In MPBA, a human may choose suboptimal arms to mislead the robot, but doing so sacrifices immediate reward. Truthfulness is quantified by

$IRL:\mathcal{P}(\Xi)\to \mathbb{R}^S,$9

and the smallest personal gap between the best arm and any suboptimal arm is

$N$0

For an explore-then-commit robot strategy, the key theorem states that given $N$1, if

$N$2

then any strategy $N$3 such that $N$4 is strictly dominated by a truthful strategy. The longer the exploration phase, the more expensive manipulation becomes relative to its future benefit (Fickinger et al., 2020).

The same paper compares “voting by demonstrating” with direct announcement. In plurality voting with random tiebreak with 3 voters and 3 alternatives, the set of manipulable profiles by demonstrating is included in the set of manipulable profiles by announcing, and the proportion of manipulable profiles by announcing but not by demonstrating is $N$5. Repetition therefore reduces manipulability in a precise comparative sense.

This logic yields explicit repeated social-choice mechanisms. In plurality voting with shared control,

$N$6

For any utility domain

$N$7

if

$N$8

then plurality voting with shared control is non-dictatorial, not limited to two alternatives, and dominant-strategy incentive-compatible on $N$9. The repeated, consequential nature of demonstrations changes the game form enough that the standard impossibility no longer applies in the same way (Fickinger et al., 2020).

4. Collegiality and repeated preference revelation

The collegial-mechanism program formalizes the same idea with a direct reward term on demonstrations. Human $$R_h^* : S \to \mathbb{R}, \qquad h \in \{1,\dots,N\},$$0 solves

$$R_h^* : S \to \mathbb{R}, \qquad h \in \{1,\dots,N\},$$1

where $$R_h^* : S \to \mathbb{R}, \qquad h \in \{1,\dots,N\},$$2 measures how much direct consequences of work matter relative to downstream robot behavior. Collegiality means that humans obtain a sufficiently large fraction of their rewards directly through their own actions (Fickinger et al., 2020).

In the integer-valued case, if $$R_h^* : S \to \mathbb{R}, \qquad h \in \{1,\dots,N\},$$3 and $$R_h^* : S \to \mathbb{R}, \qquad h \in \{1,\dots,N\},$$4 for all $$R_h^* : S \to \mathbb{R}, \qquad h \in \{1,\dots,N\},$$5, then Proposition 1 states: if $$R_h^* : S \to \mathbb{R}, \qquad h \in \{1,\dots,N\},$$6, every mechanism is straightforward. In the real-valued case, define

$$R_h^* : S \to \mathbb{R}, \qquad h \in \{1,\dots,N\},$$7

and let $$R_h^* : S \to \mathbb{R}, \qquad h \in \{1,\dots,N\},$$8. Proposition 2 states: if

$$R_h^* : S \to \mathbb{R}, \qquad h \in \{1,\dots,N\},$$9

then every mechanism is straightforward. The proofs compare the direct reward loss from distorting one’s trajectory with the bounded gain obtainable by manipulating the robot’s behavior.

In the apprenticeship subclass, the robot uses feature matching IRL: $W(R_1^*,\dots,R_N^*).$0 with

$W(R_1^*,\dots,R_N^*).$1

and the first-order condition

$W(R_1^*,\dots,R_N^*).$2

Under sufficiently large $W(R_1^*,\dots,R_N^*).$3, the demonstrated feature counts straightforwardly reveal preferences because deviations away from direct reward maximization are not profitable (Fickinger et al., 2020).

The same framework gives an approximately efficient repeated mechanism with distortion

$W(R_1^*,\dots,R_N^*).$4

and Proposition 4 proves

$W(R_1^*,\dots,R_N^*).$5

This combines incentive compatibility induced by collegiality with computational-social-choice-style welfare guarantees.

5. Learning-theoretic and scalable solvers

Recent work studies repeated assistance games directly as online learning problems. The central metric is $W(R_1^*,\dots,R_N^*).$6-assistance regret: $W(R_1^*,\dots,R_N^*).$7 where the comparator is the best fixed human–assistant policy pair in hindsight. This paper gives the first provably efficient learning algorithms for repeated assistance games. In the decentralized setting, it achieves a $W(R_1^*,\dots,R_N^*).$8-approximate assistance regret rate of $W(R_1^*,\dots,R_N^*).$9, and in a pseudo-decentralized setting with a shared random string it achieves $R^* = W(R_1^*,\dots,R_N^*)$0, optimal up to logarithmic factors. It also proves that achieving a regret approximation factor better than $R^* = W(R_1^*,\dots,R_N^*)$1 is computationally intractable (Ananthakrishnan et al., 9 Jul 2026).

The algorithmic core is a reduction from assistance policy optimization to online submodular maximization under a partition matroid, together with a decomposition of assistance regret into centralized external regret, assistant tracking regret, and a stability term counting policy switches. This yields a formal interpretation of repeated assistance as co-adaptive signaling: the human adapts how preferences are expressed, the assistant adapts how those signals are decoded, and performance is measured against the best joint signaling-decoding protocol in hindsight.

A separate line addresses scalable planning in complex state spaces. AssistanceZero formulates assistance games as two-player Markov games with state space $R^* = W(R_1^*,\dots,R_N^*)$2, human and assistant action spaces $R^* = W(R_1^*,\dots,R_N^*)$3 and $R^* = W(R_1^*,\dots,R_N^*)$4, reward parameters $R^* = W(R_1^*,\dots,R_N^*)$5, and shared return

$R^* = W(R_1^*,\dots,R_N^*)$6

The assistant policy is history-dependent because it cannot observe $R^* = W(R_1^*,\dots,R_N^*)$7, while the human policy conditions on $R^* = W(R_1^*,\dots,R_N^*)$8 and $R^* = W(R_1^*,\dots,R_N^*)$9. AssistanceZero extends AlphaZero with a neural network that predicts human actions and rewards, enabling planning under uncertainty. It is applied to a Minecraft-based assistance game with over $W_U(R_1^*,\dots,R_N^*)(s) = \sum_{h=1}^N R_h^*(s),$0 possible goals, outperforms model-free RL algorithms and imitation learning, and in a human study significantly reduces the number of actions participants take to complete building tasks in Minecraft (Laidlaw et al., 9 Apr 2025).

Together, these results show two complementary algorithmic routes: one treats repeated assistance as an online coordination-and-regret problem with explicit approximation guarantees, and the other treats it as a partially observed planning problem in a large sequential environment.

6. Implications, limitations, and directions

Repeated assistance games alter three properties of the original assistance-game framework. First, they improve identifiability because the assistant sees more behavior over time. Second, they change incentives because demonstrations directly affect current payoffs. Third, they convert preference learning into a social choice problem whenever multiple humans or heterogeneous interests are involved (Fickinger et al., 2020).

For AI alignment, the main implication is that a robot cannot naively treat multiple humans’ demonstrations as truthful inputs to IRL. In single-shot multi-principal assistance learning, low-stakes demonstrations are manipulable and subject to Gibbard-type impossibility. In repeated assistance games such as MPBA, where human actions directly affect immediate payoffs and the robot keeps humans in control long enough, misrepresentation becomes costly and honest behavior can become approximately dominant. This is the “natural mechanism design” interpretation: incentive alignment can arise from repeated, consequential interaction rather than monetary transfers (Fickinger et al., 2020).

The main limitations are explicit. Existing repeated multi-principal models largely assume linear or otherwise simple social welfare aggregation, deterministic arm-specific or state-action rewards, full observability of actions, and specific time structures such as explore-then-commit. The bandit analysis does not treat richer MDPs in which actions affect state transitions, and the collegial analysis assumes rational expected-utility maximizers with known or bounded $W_U(R_1^*,\dots,R_N^*)(s) = \sum_{h=1}^N R_h^*(s),$1, stationary rewards, and tractable feature representations (Fickinger et al., 2020). Online-learning results still benchmark against the best fixed joint policy pair in hindsight, and scalable planning work such as AssistanceZero relies on learned human models and fixed within-episode goals rather than user adaptation across episodes (Ananthakrishnan et al., 9 Jul 2026).

The open problems follow directly from these assumptions. The literature highlights extending mechanisms and incentive results beyond bandits to richer MDPs, handling more complex social welfare functions such as fairness and equity constraints, modeling richer strategic reasoning and multi-round manipulation, integrating Bayesian CIRL more fully with multi-principal repeated interaction, and designing repeated assistance systems that remain tractable under partial observability and large hypothesis spaces (Fickinger et al., 2020). A plausible implication is that future repeated assistance game models will need to combine all three strands now present separately in the literature: social-choice-aware incentive design, online no-regret coordination, and scalable belief-state planning.