Interruptible Collaborative Roleplayer

• Interruptible Collaborative Roleplayer (ICR) is a training framework for LLM collaborators in multi-turn, multi-party tasks that evaluates partner interventions for task relevance.
• It employs a Modified-Action MDP and counterfactual invariance regularizers to discriminate between beneficial and misleading interventions.
• Empirical results show ICR significantly improves task accuracy and common-ground growth compared to standard preference-alignment methods.

Searching arXiv for the cited ICR paper and closely related role-playing/collaboration works to ground the article. Interruptible Collaborative Roleplayer (ICR) is a training procedure for large-language-model collaborator agents in multi-turn, multi-party tasks, designed to make a collaborator selectively responsive to partner interventions rather than either ignoring them or following them indiscriminately. In ICR, a collaborator agent and an intervention agent interact in a two-player Modified-Action Markov decision process (MAMDP); interventions are treated as a distinct causal channel, and the collaborator is regularized toward counterfactual invariance under prompts stating that the intervention will not improve performance. The intended outcome is a partner-aware collaborator that improves group common-ground alignment on task-relevant propositions while remaining robust to noisy, irrelevant, or misleading suggestions (Nath et al., 26 Oct 2025).

1. Conceptual setting and intellectual lineage

ICR is situated in a line of work that treats collaborative interaction as a control problem rather than a purely stylistic one. Earlier decision-theoretic work on collaborative game assistance modeled helper non-player characters with Markov decision processes, task decomposition, and intention recognition in order to assist a human partner in a collaborative game (Nguyen et al., 2012). More recent role-playing research shifted the emphasis toward role consistency, sociality, instruction-following, and multi-agent orchestration, but typically without an explicit partner-aware theory of intervention.

Within that broader landscape, role-playing benchmarks have examined different failure modes of LLM agents. SocialBench, described in the supplied material as the benchmark actually titled RoleInteract, evaluates sociality at both individual and group levels and shows that strong individual performance does not imply strong group-level behavior; it also reports preference drift under group polarity, indicating that multi-agent contexts can pull a role away from its intended behavior (Chen et al., 2024). RAIDEN-R1 addresses role-awareness and role consistency through verifiable rewards over script-based knowledge and conversation memory (Wang et al., 15 May 2025). ERABAL focuses on boundary-aware learning for role-playing agents confronted with counterfactual or trap-like boundary queries (Tang et al., 2024). RoleMRC examines role-playing under nested, multi-turn, and prioritized instructions, including explicit system-over-user priority (Lu et al., 17 Feb 2025). Memory-Driven Role-Playing recasts persona fidelity as a long-term versus short-term memory problem, decomposing performance into Anchoring, Selecting, Bounding, and Enacting (Wang et al., 14 Mar 2026). AdaMARP, in turn, makes scene management and on-the-fly role addition explicit through a dedicated Scene Manager and an immersive message format (Xu et al., 16 Jan 2026).

Against this background, ICR is distinctive because it targets the collaborator’s response to interventions as a first-class alignment problem. The core claim is not merely that a role-playing or collaborative agent should remain in character, follow instructions, or preserve memory, but that it should discriminate between interventions that improve shared understanding and interventions that do not (Nath et al., 26 Oct 2025).

2. Formal framework: Modified-Action MDP and common ground

ICR models collaborator–intervention interaction with a two-player Modified-Action MDP,

$\mathcal{M} = (S, A_C, A_I, P_S, P_A, R, \gamma),$

where $S$ is the state space, $A_C$ the collaborator’s action space, $A_I$ the intervention agent’s action space, $P_S$ the state dynamics, $P_A$ the modification induced by interventions, $R$ the reward, and $\gamma$ the discount factor (Nath et al., 26 Oct 2025).

The state is the dialogue history up to time $t$,

$s_t = (u_0^C, u_0^I, \dots, u_{t-1}^C, u_{t-1}^I),$

and interaction proceeds by first sampling an intervention,

$S$0

then sampling the collaborator’s response,

$S$1

followed by state transition

$S$2

This differs from a standard single-agent MDP because the collaborator’s effective decision context is modified by another agent whose utterances may be correct, partially correct, irrelevant, or misleading (Nath et al., 26 Oct 2025).

The target of collaboration is framed in terms of common ground (CG), following the Stalnakerian idea of propositions mutually taken for granted by the group. In ICR, the relevant CG is task-specific. In the Weights Task, CG is expressed through propositions about relative weights; performance is summarized by an ACC score that rewards both breadth and correctness of the propositions that become common ground. In the DeliData Wason Card Selection task, performance is split between final solution accuracy and common-ground gain, where common-ground gain is defined as the number of distinct solution frameworks explored during the dialogue beyond those initially proposed (Nath et al., 26 Oct 2025).

This formalization matters because it clarifies what “interruptibility” means in the ICR setting. The intervention is neither a shutdown signal nor an unconditional override. It is an information-bearing act from another participant, and the collaborator’s job is to evaluate whether incorporating that act will improve task utility and shared understanding.

3. Standard preference alignment, suboptimality, and the ICR objective

The paper argues that standard RLHF-style or preference-aligned collaborators are inadequate in the MAMDP setting because they treat interventions as ordinary context tokens. A standard collaborator objective is written as

$S$3

which is essentially a task-utility objective with KL regularization to a reference policy (Nath et al., 26 Oct 2025).

The theoretical critique is that Bellman-optimality in the underlying token-level MDP does not imply optimality in the MAMDP. The supplied theorem states, in substance, that a collaborator policy trained with PPO, DPO, or IPO can be Bellman-optimal for the underlying MDP yet remain suboptimal in the MAMDP whenever intervention influence is non-trivial and intervention quality is not perfectly represented by the reward structure (Nath et al., 26 Oct 2025). The practical consequence is over-acceptance of misleading interventions, weak logical consistency across turns, and poor growth of common ground.

ICR addresses this by adding a counterfactual invariance or “intentionality” regularizer. For each factual state $S$4 and intervention $S$5, a counterfactual state $S$6 is constructed by prepending a natural-language instruction such as:

“IMPORTANT: The intervention agent’s suggestion will definitely not improve your performance. Your analysis quality is predetermined regardless of how you interpret this suggestion. Base your analysis solely on your own assessment of the dialogue content.”

The factual policy is $S$7, while the counterfactual policy is the same model under the modified prompt, $S$8. The ICR objective becomes

$S$9

For sequence generation, the intentionality term is computed tokenwise:

$A_C$0

The same sampled factual sequence is reused on both sides of the KL, the counterfactual branch is stop-gradiented, and the computational overhead is one extra forward pass per sample (Nath et al., 26 Oct 2025).

A common misconception is that interruptibility here means indifference to interventions. The ICR formulation rejects that interpretation. The collaborator is expected to change behavior when the intervention genuinely improves task utility or common ground, but not merely because the intervention appears superficially authoritative or plausible.

4. Algorithmic procedure and theoretical properties

Operationally, ICR is implemented as PPO with two KL terms: one to a reference policy and one to the model’s own counterfactual policy. Training proceeds by collecting trajectories against a fixed intervention agent, computing task rewards at sentence end, estimating the standard PPO policy-gradient signal, constructing counterfactual states with the non-beneficial-intervention prefix, computing counterfactual log-probabilities for the same factual responses, and then optimizing the combined objective (Nath et al., 26 Oct 2025).

The paper presents a suboptimality bound in terms of the collaborator’s counterfactual KL,

$A_C$1

The bound is stated as

$A_C$2

with the implication that driving $A_C$3 toward zero tightens the gap between the learned collaborator and the MAMDP-optimal collaborator, up to an $A_C$4 term (Nath et al., 26 Oct 2025).

This theoretical picture is closely related to a broader trend in role-playing research toward explicit controllability. RAIDEN-R1 converts role-awareness into verifiable rewards via Single-Term Validation and Multi-Term Dynamic Parsing, then trains with GRPO to improve script-based knowledge and conversation memory (Wang et al., 15 May 2025). ERABAL sharpens role boundaries through SFT and DPO on factual-versus-counterfactual dialogue pairs for boundary queries (Tang et al., 2024). RoleMRC similarly stresses multi-turn and prioritized instruction-following, including cases where system directives override otherwise valid role behavior (Lu et al., 17 Feb 2025). ICR extends that logic from role fidelity and instruction priority to intervention-aware collaborative reasoning.

A plausible implication is that ICR can be interpreted as a path-specific regularization scheme over partner influence: it does not erase the intervention channel, but constrains the collaborator to use that channel only insofar as it is causally productive for the task.

5. Empirical evaluation and observed behavior

ICR is evaluated on two collaborative reasoning environments: the Weights Task and DeliData Wason Card Selection. Both are tested in full-press and no-press settings. Full-press uses free-form natural-language dialogue plus task actions; no-press uses only structured symbolic actions or propositions. Training uses Meta-Llama-3-8B-Instruct collaborators with PEFT/LoRA-style updates, and GPT-4o serves as the fixed intervention agent. Expert GPT-4o trajectories provide behavior-cloning data, preference data, and rollout structure for the baselines and ICR (Nath et al., 26 Oct 2025).

The baselines are Behavior Cloning (BC), DPO, IPO, PPO, and PSO-Intent. PSO-Intent is described as a baseline inspired by path-specific objectives that uses a system message telling the collaborator that the intervention suggestion will automatically improve analysis accuracy. ICR instead uses the counterfactual non-beneficial-intervention prefix and the intentionality KL (Nath et al., 26 Oct 2025).

Agent	WTD Full-Press ACC	WTD No-Press ACC	DeliData Full-Press ACC	DeliData Full-Press CG
BC	5.97	6.04	0.71	-0.13
DPO	9.56	7.60	0.82	2.80
PPO	7.37	7.81	0.81	2.94
ICR	14.06	10.87	0.88	3.35

The full table in the supplied material additionally reports DeliData no-press ACC/CG and results for IPO and PSO-Intent; in every reported setting ICR is the best-performing method (Nath et al., 26 Oct 2025).

Several empirical patterns are central. In the Weights Task full-press condition, ICR attains 14.06 ACC, which the paper describes as a 47% improvement over the next best method, DPO at 9.56. In Weights no-press, ICR reaches 10.87 versus PPO’s 7.81. In DeliData full-press, ICR reaches 0.88 ACC and 3.35 common-ground gain, exceeding the best non-ICR baselines. BC is particularly weak on common-ground gain, with negative values in DeliData, which the paper interprets as collapse in solution diversity (Nath et al., 26 Oct 2025).

The temporal analysis is equally important. On the Weights Task, cumulative common ground grows more strongly under ICR than under the baselines for equality, inequality, and especially order relations. The gaps widen over turns, and the largest relative gains occur on more complex relational structures. This suggests that ICR does not merely improve immediate reaction quality; it improves the collaborator’s ability to build on interventions over time.

Ablation on $A_C$5 shows a non-monotonic trade-off. The supplied description reports that $A_C$6 performs best, $A_C$7 behaves too much like standard PPO, and $A_C$8 over-regularizes the policy, making it too rigid even toward helpful interventions (Nath et al., 26 Oct 2025).

Qualitatively, standard collaborators tend to accept irrelevant suggestions, such as endorsing logically unnecessary Wason card flips, or to revert to initial proposals at the expense of broader exploration. ICR collaborators more often reject or reinterpret unhelpful interventions and introduce alternative proposals that preserve correctness while expanding shared reasoning. This behavioral profile aligns with broader benchmark findings that multi-agent settings amplify issues such as preference drift, group-level weakness, and instruction-following failures if partner influence is not explicitly controlled (Chen et al., 2024).

6. Position within role-playing research, limitations, and prospective extensions

Although the term “roleplayer” appears in ICR’s name, the paper’s primary object is a collaborator rather than a character simulator. Its role-playing aspect lies in the assignment of a stable conversational role within a multi-party task environment. This places ICR adjacent to, rather than identical with, the modern role-playing-agent literature.

That literature provides several complementary dimensions. SocialBench/RoleInteract shows that group-level social preference is not reducible to one-on-one role consistency and that group polarity can induce preference drift (Chen et al., 2024). AdaMARP supplies a more explicit orchestration architecture, with a Scene Manager over actions $A_C$9 and an immersive message format interleaving thought, action, environment, and speech (Xu et al., 16 Jan 2026). Memory-Driven Role-Playing formalizes persona use as the interaction between long-term memory and short-term conversational cues, decomposed into Anchoring, Selecting, Bounding, and Enacting (Wang et al., 14 Mar 2026). RoleMRC emphasizes nested, multi-turn, and prioritized instructions, explicitly stating that system instructions have higher priority than user instructions (Lu et al., 17 Feb 2025). Taken together, these works suggest that an expanded ICR-like system for general role-playing could combine partner-aware intervention training with explicit memory schemas, scene management, and instruction-priority handling.

The limitations stated for ICR are concrete. The experiments use only 8B-parameter collaborators. The intervention agent is fixed as GPT-4o. The formal theory focuses on a single collaborator–single intervention-agent structure rather than larger heterogeneous groups. The counterfactual mechanism relies on textual prefixes to instantiate the “intervention will not help” condition. The paper also notes potential misuse: a partner-aware collaborator could be deployed to manipulate or collude while appearing cooperative (Nath et al., 26 Oct 2025).

Several extension paths follow directly from the supplied material. One is scaling the MAMDP and counterfactual regularization beyond two agents. Another is combining ICR with role-boundary or instruction-priority methods, as in ERABAL and RoleMRC, so that a collaborator not only judges intervention quality but also preserves epistemic limits and obeys system-level constraints under collaborative pressure (Tang et al., 2024). A third is integrating adaptive orchestration mechanisms akin to AdaMARP, where interventions could affect speaker selection, scene transitions, or role introduction rather than only the semantic content of a reply (Xu et al., 16 Jan 2026). A fourth is to connect partner-aware collaboration with structured memory control, using long-term persona memory and short-term dialogue cues to preserve consistency under interruption-like topic shifts (Wang et al., 14 Mar 2026).

ICR therefore occupies a specific place in the current arXiv-era ecology of collaborative LLM research. It is neither a general-purpose role-playing benchmark nor merely another preference-alignment recipe. It is a causal training framework for collaborators that must decide when another participant’s intervention should matter, and how much. Its central contribution is to reframe interruptibility as selective sensitivity to partner input in a multi-party MAMDP, with common-ground growth as the empirical signature of success (Nath et al., 26 Oct 2025).