InterSyn: Interleaved Learning for Motion Synthesis

• The paper presents InterSyn, a unified diffusion-based framework that synthesizes continuous human motion trajectories by integrating both solo and interactive segments.
• InterSyn utilizes a two-stage process—Interleaved Interaction Synthesis (INS) and Relative Coordination Refinement (REC)—to ensure smooth transitions and precise mutual coordination between agents.
• The method employs text-controllable diffusion, geometric normalization, and SLERP interpolation to generate lifelike, context-aware motion sequences with strong quantitative performance.

Interleaved Learning for Motion Synthesis (InterSyn) is a framework for generating realistic human motion that fluidly transitions between solo and multi-person interaction dynamics, targeting the synthesis of lifelike behavior in unconstrained, real-world scenarios (Ma et al., 14 Aug 2025). Distinct from prior approaches that treat individual and interactive motion as separate modeling tasks, InterSyn captures the inherently continuous nature of human behavior, where actions such as walking, pausing, conversing, and gesturing organically blend and adapt in the presence of other agents. Central to InterSyn is the principle that motion synthesis must not only span semantic diversity and context-dependent transitions, but also support fine-grained mutual adaptation between characters to yield physically plausible, contextually aligned, and text-controllable motion sequences.

1. Problem Context and Motivation

Most existing motion synthesis systems partition their modeling pipelines into either single-person “self-expressive” motion or explicit “social/interaction” motion, often utilizing separate datasets (e.g., HumanML3D for solos, InterHuman for interactions). This bifurcation is misaligned with the structure of real-world motion: practical scenarios involve frequent and seamless transitions between individual and group behaviors, requiring smooth switching and coordinated adaptation. Conventional models, by training on isolated regimes, are empirically limited in their ability to synthesize composite sequences that alternate between solo and social segments or to synchronize the spatiotemporal interplay of multiple characters. This leads to artifacts such as discontinuities at mode-switch boundaries, unnatural interpersonal spacing, and lack of response to others' presence or actions.

2. Methodological Framework: INS and REC

InterSyn addresses these challenges through an interleaved learning paradigm with two core modules:

Interleaved Interaction Synthesis (INS):

INS models the joint distribution of solo and interaction motion using a unified conditional motion diffusion model (CMDM). Training sequences are constructed via an interleaving operator, which fuses solo and interaction segments (from $p_s$ and $(p_x,p_y)$, with alignment and temporal offsets $t_s,t_i$) into composite trajectories and concatenated textual prompts. The CMDM, implemented as a Transformer-based denoising diffusion process, learns both the solo and interaction phases in a single latent space, explicitly conditioned on the temporal structure of interleaving. This allows the generator to produce sequences that, for instance, begin with individual movement, transition into synchronized social interaction, and smoothly return to solo trajectories—all in response to composite textual inputs.

The INS module’s loss function consists of:

• $\mathcal{L}_{\text{rec}}$: MSE reconstruction of clean motion.
• $\mathcal{L}_{\text{smooth}}$: velocity penalties in a local window around interleaving boundaries to ensure physical and temporal continuity.

Relative Coordination Refinement (REC):

Following initial INS generation, the REC module applies an iterative refinement scheme to maximize mutual coordination and interaction realism. This is formalized as a relative conditioning process: for a two-character sequence, person A’s motion is refined conditioned on B, and reciprocally. The coordinator network minimizes the divergence between original and refined outputs while further minimizing a masked joint distance map loss drawn from prior work. This second stage is designed to address phenomena such as response timing, spatial alignment, and mutual anticipation—critical for actions where synchronized or physically constrained interaction is necessary (e.g., handshakes, object passing).

The total REC loss blends:

• $\mathcal{L}_{\text{rela}}$: $L_2$ difference between refined and original motion.
• $\mathcal{L}_{\text{dm}}$: masked joint distance loss, targeting spatial consistency under occlusions or partial matches.

3. Motion Representation, Alignment, and Interleaving Construction

The motion representation is defined as a “motion bucket," an array $$u = (u_x,u_y) \in \mathbb{R}^{2 \times T \times K \times C}$$ encapsulating keypoints over time for both characters. The fusion operator $U(\cdot)$ aligns solos and interactions in temporal and spatial domains, leveraging explicit geometric normalization: forward kinematics, rotation transfer via quaternion products, bone-length rescaling, and translation/orientation matches. SLERP interpolation is employed for smooth transitions at segment boundaries, while cross-product/dot-product tests prevent rotational flips. The composite motion/text sequence is then provided to the generator alongside frame indices for segment starts, allowing text conditions to specify both action semantics and social context. This design enables the framework to learn from and generate arbitrary interleaved regimes, including multiple alternations of solo and interaction segments.

4. Training Pipeline and Implementation Details

InterSyn’s training operates in two sequential phases:

• Stage 1: INS is trained on data composed by interleaving HumanML3D and InterHuman samples, normalizing both to a unified motion representation. Alternating exposure to pure solo, pure interaction, and interleaved sequences is used for robust space coverage. The optimizer (learning rate $(p_x,p_y)$0, 2000 epochs, batch of 256, one H100 GPU) employs DDIM sampling and uses CLIP ViT-L/14 encodings as text features.
• Stage 2: The REC coordinator is trained (with INS frozen) on the output of the first stage, targeting further refinement of multi-character synchronization and spatial relations via the relative and masked joint distance losses.

The hyperparameters for loss balancing are set as $(p_x,p_y)$1, $(p_x,p_y)$2 (INS), and $(p_x,p_y)$3, $(p_x,p_y)$4 (REC).

5. Evaluation Protocol and Quantitative Results

The evaluation of InterSyn covers both the InterHuman (interaction-only) and an interleaved benchmark synthesizing human motion from both HumanML3D and InterHuman datasets. Performance is measured using:

• FID (Fréchet Inception Distance): motion realism.
• R-Precision: text-motion alignment (Top-1, Top-2, Top-3).
• MM Dist: motion-text feature distance.
• Diversity and MModality: sample and prompt-wise motion variability.

InterSyn achieves state-of-the-art performance on InterHuman, with Top-1 R-Precision of 0.335, Top-2 of 0.479, Top-3 of 0.584, and FID of 6.332. Compared to FreeMotion and InterGen, InterSyn yields higher textual alignment, lower FID, improved diversity, and better multimodal coverage. On the interleaved benchmark, further gains are reported in R-Precision, FID, MM Dist, and MModality, demonstrating that the unified modeling strategy enhances both semantic and physical plausibility across blended action types.

6. Ablation Studies and Empirical Insights

Extensive ablations underscore critical design decisions:

• Data interleaving patterns: Simpler interleaving schedules (e.g., single alternations) outperform frequent or complex alternations, as over-fragmenting segments truncates semantic phases and reduces coherence.
• Timestep offsets: Randomized interaction segment starts ($(p_x,p_y)$5, random $(p_x,p_y)$6) yield optimal alignment and diversity, supporting the importance of temporal variety in interleaving placement.
• Loss components: Removing the coordinator or relative loss drastically degrades alignment and increases FID/MM Dist, while omitting the smoothness term introduces transition artifacts. This confirms the essential role of REC and boundary-aware objectives for synthesis quality.

7. Qualitative Behavior and Theoretical Significance

Generated sequences from InterSyn exhibit smoother, more naturalistic transitions between solo and interactive motion phases, stronger adherence to text prompts, and improved mutual adaptation in multi-agent contexts. Visualizations show avoidance of posture discontinuity, collision, and stiff or implausible interactions even in challenging interleaved scenarios. The iterative REC refinement is extendable to multi-agent settings via repeated coordination steps.

The principal theoretical advance of InterSyn is the operationalization of “interleaved learning” for motion: rather than separating motion synthesis tasks by social granularity, a unified diffusion-based generator encompassed by refinement modules is capable of capturing the spectrum of human behavioral dynamics as a continuous latent space. This approach underpins the transition from specialist pipeline architectures to integrated agents capable of flexible, contextual, and temporally adaptive motion synthesis across social contexts (Ma et al., 14 Aug 2025).

---

References:

• "InterSyn: Interleaved Learning for Dynamic Motion Synthesis in the Wild" (Ma et al., 14 Aug 2025)
• Related frameworks: Uni-Inter (Liu et al., 17 Nov 2025), OmniMoGen (Bu et al., 22 Dec 2025), Adaptive Interpolation-Synthesis (Raël et al., 4 May 2026), Programmatic Concept Learning (Kulal et al., 2022)