Embedded Interaction Graph (EIG)

• Embedded Interaction Graph is a sparse, time-dependent subgraph representation that captures essential spatiotemporal interactions between agents.
• It is learned via a dual-stage architecture employing multi-head cross-attention for edge selection and a pose decoder for next-state prediction.
• EIG enables effective cross-embodiment imitation by transferring key interaction semantics and adapting relational structures to diverse morphologies.

Embedded Interaction Graph (EIG) is a sparse, learned, time-dependent subgraph representation for physical interaction imitation across morphologically distinct agents. Introduced in the BuddyImitation framework, EIG compresses a dense, fully connected interaction graph into a small, task-relevant graph that captures key spatiotemporal relations between two interacting bodies and then reuses that representation as an imitation objective for control learning in physics-based simulation (Li et al., 28 Jul 2025). In this formulation, the central problem is not merely motion retargeting, but the transfer of interaction semantics—who influences whom, through which body parts, and at what time—across embodiments such as humanoids, quadrupedal robots with manipulators, and mobile manipulators.

1. Conceptual role and motivation

EIG is motivated by limitations in several existing approaches to interaction imitation. Pure kinematic retargeting methods usually assume similar body structures, so they do not generalize well to robots with very different morphology. Full interaction graphs can encode pairwise relations across all joints, but the paper characterizes them as too dense, often redundant, difficult to transfer across embodiments, and liable to cause collapse or poor motion when applied naively to robots with different size, limb count, or joint types (Li et al., 28 Jul 2025). Handcrafted descriptors or task-specific constraints are described as brittle and not scalable across interaction types such as handshake, sparring, dancing, or circling, while diffusion-based interaction generators can synthesize realistic human interactions but do not provide a directly transferable, interpretable structure for downstream imitation in simulation.

Within this setting, EIG is introduced to answer two coupled questions: what the essential semantic structure of an interaction is, and how that structure can be adapted to a new embodiment for policy learning. The paper’s answer is a sparse interaction graph that highlights only the most informative inter-agent relations at each time step. This gives EIG the status of an interaction bottleneck: it removes irrelevant edges, retains semantically dominant relations, and produces a compact representation intended to be transferable across embodiments.

The representation is explicitly described as sparse, time-dependent, task-adaptive, and transferable. Those properties are not auxiliary implementation details; they define the reason EIG is proposed in the first place. In the paper’s framing, a useful interaction representation must preserve semantic structure while discarding embodiment-specific redundancy.

2. From the full Interaction Graph to the embedded form

EIG is defined relative to a denser precursor called the Interaction Graph. The underlying graph is a spatial graph $G(V,E)$ whose nodes are placed on salient joints or body points of each character and whose edges connect joints across characters (Li et al., 28 Jul 2025). Each edge $\phi^{ij} \in \mathbb{R}^6$ encodes relative spatial information between joint $i$ of one character and joint $j$ of the other, including relative position and midpoint in world coordinates. The time-$t$ graph feature is the collection of all such edge features.

A key formal property of the full Interaction Graph is that, due to kinematic structure, the full interaction graph plus one agent’s pose can reconstruct all agents’ joint positions. This makes the original graph geometrically sufficient, not merely descriptive. The difficulty, however, is that the same completeness makes it large and embodiment-specific.

The Embedded Interaction Graph is therefore defined as a sparse learned subgraph of the full graph:

$G^{\text{emb}}_t \subseteq G.$

Time is explicit because the important interaction edges can change over the course of an interaction. The paper’s central claim is that, at each time step, only a small subset of edges matters for predicting future motion and preserving interaction semantics (Li et al., 28 Jul 2025).

This distinction between full Interaction Graph and EIG is fundamental. The full graph provides exhaustive cross-body relational information; EIG seeks the minimal subset that remains predictive of the next motion state. A plausible implication is that EIG is best understood not as a generic graph reduction, but as a task-conditioned compression of inter-agent relational structure.

3. Learning objective and model architecture

EIG learning is formulated as a Markovian next-pose prediction problem. The paper assumes that the future state depends only on the current state, and for one character $k$, the future pose depends on that character’s current pose together with the current interaction graph state (Li et al., 28 Jul 2025). The goal is to replace the full graph feature with the embedded graph while preserving predictive power.

The optimization problem is posed as sparse graph selection for future-pose reconstruction. The learned embedding or edge-selection function $\xi$ produces the embedded graph under a target edge budget $\bar n$, while the pose prediction model $\Psi$ uses the current pose and the embedded graph to predict the next pose. This makes EIG learning a constrained predictive compression problem: the graph should be as small as specified by the edge budget, but still sufficient for accurate next-state prediction.

The architecture has two stages. First, edge selection is performed by multi-head cross-attention. The query is the encoded current pose of a character, the keys are latent embeddings of each edge in the full graph, and the values are the original edge features. Sparsity is enforced by hard attention: each attention head selects one edge, and the number of selected edges equals the number of heads. The embedded graph therefore has exactly $\phi^{ij} \in \mathbb{R}^6$0 edges if there are $\phi^{ij} \in \mathbb{R}^6$1 heads (Li et al., 28 Jul 2025).

Second, the selected sparse graph is used for pose prediction. The embedded graph is encoded with a graph encoder, concatenated with the current pose, and fed into a neural pose decoder that predicts the next pose. The same model is applied to each human in the interaction, allowing the system to learn the overall interaction dynamics.

To encourage temporal stability, the paper adds a graph consistency loss:

$\phi^{ij} \in \mathbb{R}^6$2

where $\phi^{ij} \in \mathbb{R}^6$3 is the variance of the attention or Gumbel-Softmax outputs across time (Li et al., 28 Jul 2025). Low variance is interpreted as stable edge selection, reducing redundancy and preserving temporal coherence.

The pose decoder used in this stage is pretrained as a Motion Variational Autoencoder. The MVAE takes the current pose and a latent variable $\phi^{ij} \in \mathbb{R}^6$4, reconstructs the next pose, and regularizes the latent space to a normal distribution. Scheduled sampling is used, and the KL weight is set to $\phi^{ij} \in \mathbb{R}^6$5 as a tradeoff between generalization and motion quality.

4. Semantics, sparsity, and temporal adaptation

The paper emphasizes that EIG is not an arbitrary sparse graph. Its selected edges concentrate on body regions and relations that are task-relevant. In handshaking and rock-paper-scissors, the learned graph focuses on root positions and right-arm relations. In sparring, the graph shifts dynamically between right-arm, left-arm, and root relations depending on who is attacking or defending. In circling, it captures mirrored hand positions and circular coordination (Li et al., 28 Jul 2025).

A heatmap analysis divides the body into Root, Upper, Lower, and Arms, and reports that the most influential edges are usually between the root and arm regions of both characters. The paper states that this supports the interpretation that EIG captures the “causal core” of the interaction. That phrase is significant because it positions EIG as a semantic abstraction, not just a compact encoding.

The interaction semantics captured by EIG are also explicitly phase-dependent. The representation can change across motion phases and across changes in interaction role. This temporal adaptability is one of the main distinctions between EIG and a static handcrafted interaction descriptor. At the same time, graph consistency regularization prevents the learned graph from fluctuating needlessly, so the model is designed to balance adaptation with stability.

This yields a specific notion of interpretability. EIG is interpretable not because it is symbolic in a strict logical sense, but because it restricts the interaction description to a small number of semantically meaningful cross-body edges. The interpretability claim is therefore inseparable from the sparsity mechanism.

5. Cross-embodiment transfer and control objective

After EIG is learned from human demonstrations, it becomes the core imitation signal for interaction transfer. Policy learning is carried out in physics-based simulation with reinforcement learning, and the imitation objective is the difference between the embedded graph of the demonstration and the embedded graph induced by the target agent’s current state (Li et al., 28 Jul 2025). The paper refers to this as interaction consistency reward.

Because the target agent may have a very different morphology, the method first constructs a corresponding embedded interaction graph for the new embodiment. Vertex correspondence is assigned automatically by a positional alignment heuristic: it starts from end-effectors, compares root-to-end-effector vectors, matches the new character’s end-effector with the highest inner product, and then assigns the remaining vertices by relative sequence. This is intended to preserve edge relations while adapting node identities to the robot’s body structure.

The paper defines three graph-distance terms for transfer: a length metric, a root edge direction metric, and a center-point height metric. The length term measures normalized edge-length difference between robot and reference; the root edge direction term compares the direction of the root-to-root connection projected onto the $\phi^{ij} \in \mathbb{R}^6$6 plane; and the center-point height term aligns the vertical placement of interaction elements. These are combined into the final interaction consistency reward:

$\phi^{ij} \in \mathbb{R}^6$7

Here, $\phi^{ij} \in \mathbb{R}^6$8 is a far-distance fallback term, included to prevent reward collapse early in training (Li et al., 28 Jul 2025).

The reinforcement-learning objective is standard discounted return,

$\phi^{ij} \in \mathbb{R}^6$9

with $i$0 including the EIG-based interaction consistency reward plus regularizers. The regularization terms are root height regularization, lateral orientation regularization, and torque regularization. The transfer architecture is a centralized hierarchical policy with one high-level policy that sees observations from all characters and outputs a latent control vector, together with multiple low-level policies that generate motor commands for specific agents. Low-level controllers are initialized by motion primitive pretraining following Imitate-and-Repurpose.

In this formulation, EIG functions in two roles simultaneously: as a predictive interaction model during representation learning, and as a control objective during policy optimization.

6. Evaluation, empirical profile, and scope

The empirical setup uses the Genesis physics simulator and demonstrations from InterHuman. The paper selects 25 interaction scenes of 10–20 seconds each and evaluates four embodiments: a humanoid, a child, Go2Ar, and Stretch (Li et al., 28 Jul 2025). The interaction categories include collaborative dancing, competitive sparring, social handshaking, and, in the embedding analysis, rock-paper-scissors and circling.

EIG itself is evaluated through future pose prediction with rollout length 120 frames, corresponding to 2 seconds, over 500 trials, using mean squared error between predicted and ground-truth poses. Baselines and ablations include a random graph with the same edge count (ER), no graph ($i$1), a single-edge graph ($i$2), the default 4-edge embedded graph ($i$3), an 8-edge graph ($i$4), and a fully connected graph ($i$5) with 484 edges. The paper reports that learned EIG significantly outperforms the random graph, that sparse graphs outperform no-graph and overly dense graphs, that $i$6 provides a good tradeoff between accuracy and efficiency, and that the fully connected graph can overfit and be harder to process.

Transfer performance is assessed both qualitatively and through a user study. Example behaviors include Go2Ar learning sparring and dancing from human demonstrations, Stretch learning handshaking while adapting arm height via prismatic extension and base rotation, and single-arm robots emulating bimanual motions by switching reference limbs dynamically. In the user study, participants were asked to identify interaction scenarios from learned motion videos and to rate semantic alignment with demonstrations on a 0-to-5 scale. Compared with inverse-kinematics retargeting, the EIG-based method improved recognition accuracy and obtained higher semantic alignment ratings (Li et al., 28 Jul 2025).

The paper also states several limitations. Experiments are mainly on two-character interactions. Each character setting and interaction scenario is handled separately, with no unified universal policy. Transfer still depends on hand-designed correspondence between embedded graph vertices and the target embodiment. Some motions, especially sharp or highly dynamic turns, cannot be fully replicated by constrained robots such as Go2Ar. The method is also imitation-focused and does not yet provide deeper strategic reasoning or adaptation in competitive settings.

Related arXiv work uses “interaction graph” language in substantially different senses, including explicit word-word interaction graphs for intrinsically interpretable NLP classifiers (Sekhon et al., 2023) and a graph–hypergraph taxonomy centered on focal versus non-focal interaction domains (Moutuou, 4 Mar 2026). EIG belongs to neither of those settings directly. Its specific domain is cross-embodiment physical interaction imitation, where the graph is a sparse spatiotemporal representation of inter-agent body relations rather than a text-layer adjacency structure or a general dynamical taxonomy.

Taken together, the EIG formulation defines a particular research program: learn interaction semantics from human demonstrations as a compact graph, adapt that graph across embodiments, and use it as the operational target for physics-based control. Within the scope demonstrated so far, its distinguishing feature is not merely graph sparsification, but the coupling of sparse interaction representation, temporal adaptation, and embodiment transfer into a single imitation pipeline.