Interactive Program Synthesis for Modeling Collaborative Physical Activities from Narrated Demonstrations

Abstract: Teaching systems physical tasks is a long standing goal in HCI, yet most prior work has focused on non collaborative physical activities. Collaborative tasks introduce added complexity, requiring systems to infer users assumptions about their teammates intent, which is an inherently ambiguous and dynamic process. This necessitates representations that are interpretable and correctable, enabling users to inspect and refine system behavior. We address this challenge by framing collaborative task learning as a program synthesis problem. Our system represents behavior as editable programs and uses narrated demonstrations, i.e. paired physical actions and natural language, as a unified modality for teaching, inspecting, and correcting system logic without requiring users to see or write code. The same modality is used for the system to communicate its learning to users. In a within subjects study, 20 users taught multiplayer soccer tactics to our system. 70 percent (14/20) of participants successfully refined learned programs to match their intent and 90 percent (18/20) found it easy to correct the programs. The study surfaced unique challenges in representing learning as programs and in enabling users to teach collaborative physical activities. We discuss these issues and outline mitigation strategies.

• The paper presents an interactive synthesis framework that allows non-programmers to model collaborative physical activities using narrated demonstrations.
• It employs a hierarchical finite state machine integrated with LVLM and domain-specific APIs to represent complex, context-dependent actions.
• Empirical evaluations show significant improvements in program completeness and usability through iterative user feedback and correction.

Interactive Program Synthesis for Collaborative Physical Activities from Narrated Demonstrations

Introduction and Motivation

This work addresses the challenge of enabling non-programmers to teach systems complex, collaborative physical activities—tasks that require modeling not only individual actions but also the dynamic, contingent reasoning about teammates’ and opponents’ intentions. The authors propose an interactive program synthesis framework that leverages narrated demonstrations—paired physical actions and natural language explanations—to synthesize interpretable, correctable programs representing collaborative behaviors. The system is designed to support iterative teaching, inspection, and correction, all without requiring users to interact with raw code.

System Architecture and Program Representation

The system architecture is centered on the use of narrated demonstrations in mixed reality (MR) as the primary teaching modality. Users perform physical actions in an MR environment while providing concurrent verbal explanations. The system records RGB video, action annotations, and timestamped transcriptions, which are then processed by a large vision-LLM (LVLM) to synthesize a programmatic representation of the demonstrated behavior.

Behavioral logic is modeled as a hierarchical, interrupt-driven finite state machine (FSM), where states correspond to actions and transitions are gated by preconditions—Boolean predicates over continuous world variables. This abstraction enables the encoding of both discrete sequencing and contingent, context-dependent behavior.

Figure 1: Hierarchical FSM representation of collaborative behaviors, with zoom-in on a substate modeling interrupt-driven transitions.

The FSM structure supports both hierarchical and primitive states, allowing for modular decomposition of complex behaviors. Interrupt-driven transitions enable the system to model reactive, context-sensitive human actions, a critical requirement for collaborative physical activities.

Domain-Specific APIs and Constraint Composition

The system’s expressivity is grounded in a domain-specific API library, which includes both action APIs (e.g., MoveTo, Pass, Shoot) and constraint APIs for logical and spatial reasoning. Constraint APIs can return Boolean values or spatial fields, and can be composed using logical operators (AND, OR, NOT) to yield complex, probabilistic spatial distributions over the workspace.

Figure 2: Logical composition of spatial constraints to generate a probability distribution for agent positioning in a collaborative soccer scenario.

This compositional approach allows the system to capture nuanced spatial and logical requirements, such as “move to either side of the defender to receive a pass,” by combining multiple constraints into a single, sampleable distribution.

Program Synthesis via LVLMs and Probabilistic Programming

The core program synthesis is performed by an LVLM, which receives as input the grounded, multimodal demonstration data and the API library. The LVLM is prompted to generate a Scenic probabilistic program that encodes the FSM-based behavioral logic, including both action sequences and spatial/temporal constraints.

Figure 3: Example Scenic program output by the LVLM, with learned constraints and embedded narration.

A key feature is the integration of a Speak() API, enabling the synthesized program to narrate its intent during execution. This temporally aligned narration enhances interpretability and supports user understanding of the system’s reasoning at each decision point.

Interactive Inspection, Feedback, and Repair

The system supports user-in-the-loop correction through two complementary interfaces:

1. Decision Flow Visualization: The synthesized FSM is visualized as a decision flow chart, with nodes and edges annotated by natural language descriptions. Users can inspect, annotate, and provide feedback on specific states or transitions without interacting with code.
2. Program Execution with Narration: The program is executed in MR, with the avatar performing the learned behavior and narrating its actions. Users can pause, annotate the environment, and provide situated feedback, which is then used to localize and repair errors in the program.

Figure 4: User-in-the-loop feedback via decision flow inspection and narrated program execution, supporting structured and experiential correction.

Feedback is processed by the LVLM, which is prompted to make localized edits to the candidate program, leveraging the multimodal feedback stream and the original demonstration context.

Empirical Evaluation: User Study and Results

A within-subjects study with 20 non-programmer participants (all with competitive soccer experience) was conducted to evaluate the system’s efficacy in modeling collaborative soccer tactics. Participants taught the system using narrated demonstrations in MR, inspected the synthesized programs, and provided corrective feedback.

Key quantitative findings include:

• Initial alignment: Pre-feedback correctness and completeness scores exhibited high variance (mean correctness 70.28%, SD 24.02%; mean completeness 76.57%, SD 28.85%), reflecting differences in teaching quality and explicitness of rationale in demonstrations.
• Post-feedback improvement: After one round of feedback, completeness improved significantly by 16.68% (p=0.03), while correctness showed a positive but non-significant trend (mean +6.87%, p=0.10).
• Subjective ratings: Likert ratings for perceived alignment and teachability improved significantly post-feedback.

Figure 5: Distributions of user scores for intuitiveness and ease of use across system components.

Figure 6: Box plots of correctness and completeness scores before and after user feedback, demonstrating significant improvement in program completeness.

Figure 7: Mean Likert ratings for perceived alignment of decision flow and teachability, before and after feedback.

Qualitative analysis identified several sources of misalignment and repair failure, including insufficient API expressivity, context loss during repair, and program deadlocks due to unsatisfiable preconditions. Notably, 90% of participants found the correction process easy, and 95% found the narrated demonstration interface intuitive, though some requested richer feedback tools and improved agent realism.

Generalizability and Domain Transfer

The architecture is designed to be domain-agnostic, with domain-specificity encapsulated in the API library and environment modeling. The authors demonstrate applicability to other collaborative domains (e.g., manufacturing) by swapping the API set and environment model, while retaining the narrated demonstration and program synthesis loop.

Implications, Limitations, and Future Directions

This work demonstrates that interactive program synthesis from narrated demonstrations can enable non-programmers to externalize and iteratively refine complex collaborative behaviors as interpretable, executable programs. The FSM-based abstraction, combined with probabilistic programming and LVLM-based synthesis, provides a flexible and generalizable framework.

However, several limitations are identified:

• API expressivity: The system’s ability to model user intent is fundamentally constrained by the provided API library. Expanding and iteratively refining the API set is necessary for broader applicability.
• Debugging and deadlock detection: Lack of explicit deadlock detection and explanation mechanisms impedes user debugging. Future work should incorporate program analysis routines to surface and explain unsatisfiable conditions.
• Instructional scope: Modeling only the user’s behavior may be insufficient for fully capturing collaborative intent, especially when environment agents’ behaviors are pre-scripted and not user-editable.
• Program minimality and interpretability: As program complexity grows, decision flow diagrams can become difficult to interpret. Imposing parsimony constraints and supporting user queries about ambiguous terms are promising directions.

The authors suggest that integrating active elicitation of tacit knowledge—potentially via Theory of Mind-inspired dialogue—could further improve alignment and robustness, shifting the interaction from passive demonstration to collaborative teaching.

Conclusion

This work presents a comprehensive framework for interactive program synthesis of collaborative physical activities from narrated demonstrations, validated in a challenging multi-agent domain. The approach enables non-programmers to teach, inspect, and correct system behavior via interpretable program abstractions, with strong evidence for usability and iterative refinement. The architecture’s modularity and reliance on domain-specific APIs position it as a promising foundation for end-user development of AI systems in diverse collaborative settings. Future research should focus on expanding API expressivity, enhancing debugging support, and developing mechanisms for active knowledge elicitation and multi-agent instruction.