Action-Influence in Interactive Systems

• Action-Influence is defined as the systematic capacity of actions to modify both immediate states and latent strategies in interactive systems.
• Formal frameworks such as MOMDPs, causal influence diagrams, and conditional mutual information quantify how actions steer future behaviors and equilibria.
• Applications in human–robot interaction, network science, and quantum dynamics demonstrate the practical impact of strategic actions on long-term outcomes.

Action-Influence refers to the capacity of actions—executed by an agent, subsystem, or component—to systematically alter the state, strategy, or choices of another entity so as to yield favorable long-term outcomes within interactive or dynamical systems. The term carries precise technical meaning across robotics, decision theory, multi-agent learning, causal inference, quantum dynamics, and network science. Across these domains, action-influence quantifies how present choices can intentionally modulate the future behaviors, responses, or internal variables of other entities, transcending mere immediate effects to encompass the shaping of latent strategies, equilibria, or behaviors on extended horizons.

1. Formal Characterization and Mathematical Frameworks

Action-influence is rigorously formalized in multiple domains, with the human–robot interaction (HRI) setting providing a paradigmatic example. Here, an action-influential robot selects actions $a^R_t$ with dual objectives: (i) to achieve task-aligned changes in the observable joint state $s_t$ and (ii) to steer the latent decision strategy of its human counterpart so that future human actions $a^H_{t+1}, a^H_{t+2}, \dots$ become more favorable to the robot's long-term reward.

The technical formalism represents interaction dynamics as

$s_{t+1} = f(s_t, a^R_t, a^H_t)$

with human policy modeled via a latent two-layer dynamics:

$$\begin{aligned} a^H_t &= \pi_H(s_t, z_t) \ z_{t+1} &= g_s(s_t, a^R_t, a^H_t, z_t, \phi_t) \ \phi_{t+1} &= g_l(s_t, a^R_t, a^H_t, \phi_t) \end{aligned}$$

where $z_t$ denotes a short-term latent strategy (e.g., aggressiveness in driving) and $\phi_t$ parameterizes the longitudinal adaptation of that strategy over repeated episodes.

From the robot's perspective, this yields an augmented state $x_t = (s_t, z_t, \phi_t)$, but only $s_t$ is directly observed, making the control problem a mixed-observability Markov Decision Process (MOMDP). The robot's optimal action policy $\pi^R(b_t, s_t)$ seeks to maximize expected cumulative reward, subject to belief updates over hidden human latents $b_t(z, \phi)$:

$$\max_{\pi^R} \mathbb{E}_{b_0}\Bigl[\sum_{t=0}^T r_R(s_t, a^R_t)\Bigr]$$

subject to the system dynamics above (Sagheb et al., 18 Mar 2025).

In other settings, such as causal influence diagrams, influence is captured through the existence of directed (or d-connected) graphical paths by which interventions on an action variable can alter downstream utility or state nodes—a criterion that supports automated detection of possible observation and control incentives (Everitt et al., 2019).

2. Action-Influence in Human–Robot and Human–Agent Interaction

In interactive multi-agent scenarios, action-influence occurs when one agent (autonomous or robotic) deliberately modulates another's internal strategy, not just their immediate behavior, thereby manipulating future choices to be advantageous for the influencer's objectives. This is distinct from reactive, myopic, or mere “nudge” approaches.

Within HRI, action-influence is divided into:

• Short-term influence: Strategies that seek immediate responses (e.g., Stackelberg game formulations where the human’s policy is assumed fixed or fully observable);
• Long-term influence: Policies that model and purposely maintain influence over sequences of repeated encounters, explicitly capturing how human strategies adapt to recurrent robot behaviors via latent variables $(z_t, \phi_t)$.

Regulating influence over time requires explicit belief maintenance over latent adaptation variables, interleaving reward-maximizing and information-gathering actions to sustain the robot's ability to shape human behavior as adaptation occurs (Sagheb et al., 18 Mar 2025).

3. Quantification of Causal Action Influence

Causal action influence is operationalized via pointwise measures of conditional mutual information (CMI) between an agent’s action and components of the next state, conditioned on the current state:

$$C^j(s) = I(S'_j ; A | S = s) = \mathbb{E}_{a\sim\pi}[D_{\mathrm{KL}}(P_{S'_j|S=s,A=a} \parallel P_{S'_j|S=s})]$$

This score quantifies, for each factor $S'_j$ of the next state, the extent to which the agent's action can exert causal influence. When $C^j(s) \approx 0$, the agent has negligible causal control over $S'_j$ in state $s$. This formulation underpins data-driven detection of controllable versus uncontrollable factors, as in counterfactual data augmentation and intrinsic motivation for exploration in RL and robotics (Urpí et al., 2024, Yuan et al., 2 Feb 2025).

In multi-agent deep RL contexts, action influence can be computed as the average KL divergence between another agent’s action distribution conditioned on an agent’s chosen action versus the marginal, yielding an influence reward formally equivalent to mutual information between agents' actions (Jaques et al., 2018):

$$c^k_t = \sum_{j\neq k} D_{\mathrm{KL}}\left[ p(a^j_t | a^k_t, s^j_t, u^j_t) \parallel p(a^j_t | s^j_t, u^j_t) \right]$$

4. Algorithmic and Optimization Techniques

The practical computation and optimization of action-influential policies draw on several paradigms:

• MOMDPs for Latent Human-Influence Modeling: As in the unified robot framework, action-selection is reframed as planning under partial observability over human adaptation parameters, with approximations including entropy-regularized one-step lookahead and QMDP-like simplifications (Sagheb et al., 18 Mar 2025).
• Influence Maximization in Networks: In networked populations, influence is maximized by selecting subsets of agents (seed sets) whose initial actions “force” favorable equilibria (PSNE) throughout the network, subject to positive/negative weights and possibly repeated actions (Irfan et al., 2013, Yu et al., 2017). Theoretical guarantees for influence ability are provided for submodular, knapsack, or cardinality-constrained maximization settings.
• Causal Influence-Based Intrinsic Rewards: For RL in high-dimensional or sparse interaction regimes, causal action influence serves as an intrinsic bonus guiding exploration toward controllable aspects of the environment, driving sample-efficient behavior acquisition (Urpí et al., 2024, Yuan et al., 2 Feb 2025).
• Multi-Agent Coordination via Influence: Influence rewards, specifically those based on mutual information or counterfactuals, systematically encourage coordinated and communicative behavior in MARL without requiring extrinsic coordination signals (Jaques et al., 2018).

5. Empirical and Theoretical Evidence

Action-influence has been demonstrated empirically and theoretically across diverse domains:

• Human–Robot Experiments: Unified controllers maintain positive long-term influence over human behavior across simulated driving, manipulation, and drone experiments, outperforming short-term baselines under repeated human adaptation. For example, in simulated highway driving, long-term action-influence reduced human lane progress by 35% and collisions by 40% versus Stackelberg controllers; in user studies, unified policies maintained behavioral influence over 60 trials and were rated less predictable and more influential by humans (Sagheb et al., 18 Mar 2025).
• Offline RL and Data Augmentation: CAIAC-based augmentation doubles the support coverage of the state space and yields up to 80% success rates in out-of-distribution goal-conditioned tasks where vanilla learners fail. This effect is robust to spurious correlations and withstands severe data scarcity (Urpí et al., 2024).
• Network Influence Games: In large-scale real-world influence games (e.g., the US Senate), minimal seed sets capable of enforcing desired outcomes are efficiently identified with near-optimality, and the computational framework handles positive and negative multimodal influences (Irfan et al., 2013).
• Sensorimotor and Linguistic Action-Influence: Neurophysiological studies using dynamic causal modeling show that affordance signals in premotor cortex causally modulate activity in temporal and parietal semantic hubs during action-language comprehension, providing a mechanistic account of action-influence in human cognition (Bordoloi et al., 4 Dec 2025).
• Open Quantum System Dynamics: Influence action functionals encode the entire effect of an external environment on a system, yielding stochastic equations whose solutions exhibit steady-state energy flow and unique nonequilibrium steady states, even with nonlinearities (Hsiang et al., 2014, Yang et al., 2020, Hsiang et al., 2020).

6. Extensions and Broader Relevance

The action-influence formalism generalizes well beyond HRI and RL. In network science, action-influence encapsulates both positive and negative behavioral propagations, enabling analysis of stable behavioral patterns, strategic reversals, and equilibria in heterogeneous populations (Irfan et al., 2013, Rendsvig, 2017). In quantum dynamics, “influence action” formalizes the effect of a subsystem’s actions on the future dynamical evolution of coupled degrees of freedom (Hsiang et al., 2014, Boyanovsky, 2016, Hsiang et al., 2020). In causal inference, action-influence diagrams and counterfactual reasoning provide principled tools for auditing and designing decision systems to achieve or suppress certain incentives (Everitt et al., 2019).

Influence techniques also enable sample-efficient learning in scenarios with sparse feedback, complex causal dependencies, or distributional shifts, and can be systematically combined with prior knowledge, hierarchical control, and graph-based representations to address tractability and generalization.

7. Summary Table: Core Formalisms of Action-Influence

Domain	Formalism	Influence Mechanism
HRI / RL	Latent dynamics, MOMDP	Actions modify human policy adaptation $(z,\phi)$ (Sagheb et al., 18 Mar 2025)
Multi-Agent RL	Causal MI / Reward bonus	Agent's action shapes others' policies/rewards (Jaques et al., 2018, Yuan et al., 2 Feb 2025)
Counterfactual Data	Conditional MI (CMI)	Detect controllable state-factors for augmentation (Urpí et al., 2024)
Networks	Influence games, PSNE	Actions force system-wide equilibria (Irfan et al., 2013, Yu et al., 2017)
Open Quantum Sys.	Influence action functional	System actions steer environment, encode memory and noise (Hsiang et al., 2014, Hsiang et al., 2020)

These formalisms provide a unified conceptual and computational toolkit for quantifying and optimizing deliberate long-term influence of actions on the strategies, dynamics, or cognitive states of other entities, with demonstrable impact in robotics, learning, neuroscience, network science, and quantum physics.