Action Dynamics Task Graphs (ADTG)

• Action Dynamics Task Graphs (ADTG) are a formalism for encoding procedural tasks as directed graphs with temporal, causal, and spatial dependencies.
• They integrate multimodal feature learning by mapping video and narration to semantic embeddings that capture pre- and post-conditions of actions.
• ADTG supports efficient planning, task tracking, and automation across various domains such as cooking and robotics, demonstrating significant performance gains.

Action Dynamics Task Graphs (ADTG) are a formalism for representing and learning the structure of procedural tasks as directed graphs over semantic action nodes, providing a data-driven, temporally-aware substrate for tracking, planning, and automated execution. ADTG encompasses both supervised action-centric models for perceptually-grounded tasks (notably from video and narration) and formal, modular ontologies for domains such as cooking, where recipes are codified as temporal action graphs. In both conceptualizations, the core objective is to encode tasks by their dispositive actions and the dynamical dependencies—temporal, causal, and spatial—between them, thereby enabling downstream applications in task tracking, recommendation, planning, and automation (Mao et al., 2023, Kumbhakern et al., 4 Sep 2025).

1. Formal Representations and Graph Structures

Procedural tasks are encoded as directed graphs $G = (V, E)$ or $G = (N, E, T)$, where nodes correspond to semantic actions or entities and edges represent temporal or material dependencies.

• In the perceptual ADTG model (Mao et al., 2023), each node $a \in V$ corresponds to a durative action observed in at-home task demonstrations, with an edge $(a_i \rightarrow a_j) \in E$ present if action $a_j$ immediately follows $a_i$ in any demonstration. The weight $w_{ij}$ is set as $1$ if such adjacency is observed, $0$ otherwise, and co-occurrence counts $N(a_i \rightarrow a_j)$ may be mapped to Markovian transition probabilities.
• In the Action-Graph DSL for structured domains (Kumbhakern et al., 4 Sep 2025), $G = (N, E, T)$0 is partitioned into node types: Ingredients $G = (N, E, T)$1, Process actions $G = (N, E, T)$2, Transfer actions $G = (N, E, T)$3, and Plate (assembly) nodes $G = (N, E, T)$4. Edges are further divided into material flow ($G = (N, E, T)$5), hard precedence ($G = (N, E, T)$6), and concurrency/merge edges ($G = (N, E, T)$7). Temporal constraints $G = (N, E, T)$8 incorporate both absolute and relative timing.

Table 1 summarizes node and edge types in the structured recipe graph model.

Node Type	Description	Key Attributes/Edges
Ingredient (I)	Input material	name, quantity, unit, env
Process (P)	State change	tech, tool, tempProfile, duration
Transfer (R)	Spatial change	sourceEnv, targetEnv
Plate (L)	Final assembly	inputs

Edges: $G = (N, E, T)$9 (material provenance), $a \in V$0 (strict order), $a \in V$1 (concurrent merge/sync) (Kumbhakern et al., 4 Sep 2025).

2. Action Embedding and Dynamics

ADTG integrates action-centric feature learning by encoding each action as a transformation between “pre-condition” and “post-condition” semantic vectors derived from video, language, or other context.

• For each segment $a \in V$2 with temporal boundaries $a \in V$3, “pre-condition” $a \in V$4 and “post-condition” $a \in V$5 features are extracted from high-dimensional representations (e.g., CrossTask features, $a \in V$6). A shared encoder $a \in V$7 maps these windows to $a \in V$8.
• Each action node is associated with a learned embedding $a \in V$9, and a transformation predictor $(a_i \rightarrow a_j) \in E$0 predicts the post-condition vector given $(a_i \rightarrow a_j) \in E$1.
• Supervised losses include a discriminative component ensuring the predicted post-condition matches the ground-truth and a contrastive margin enforcing separation from negatives; see:

$(a_i \rightarrow a_j) \in E$2

$(a_i \rightarrow a_j) \in E$3

with $(a_i \rightarrow a_j) \in E$4 the cosine distance and $(a_i \rightarrow a_j) \in E$5. Overall, $(a_i \rightarrow a_j) \in E$6 (Mao et al., 2023).

A plausible implication is that such embeddings generalize to unseen variations and enable task-centric perceptual modeling.

3. Temporal Dependencies, Concurrency, and Modularity

ADTG models capture a wide range of temporal phenomena:

• Material and Temporal Flow: $(a_i \rightarrow a_j) \in E$7 edges formalize provenance (e.g., output of one node as input to another). $(a_i \rightarrow a_j) \in E$8 encodes strict finish-before-start relations.
• Concurrency: Disjoint subgraphs can execute in parallel if no $(a_i \rightarrow a_j) \in E$9 path connects them. Merge nodes with $a_j$0 enforce that subsequent actions proceed only after all concurrent branches have begun or completed.
• Relative Timing and Interjection: Nodes can be inserted with relative start constraints (e.g., start at a fraction $a_j$1 of parent’s duration), including explicit “interrupts” or “interleaves” semantics.
• Modularity: Entire graphs $a_j$2 can be nested as submodules via attaching root and plate nodes, supporting scalable and hierarchical task description (Kumbhakern et al., 4 Sep 2025).

4. End-to-End Pipeline: From Perception to Structured Graphs

ADTG frameworks support either weakly-to-fully supervised graph induction from multimodal input:

• Perceptual ADTG: Video demonstrations and narrations are automatically segmented, features extracted, and actions localized via RNN/MLP architectures over the learned embeddings (Mao et al., 2023).
• Recipe Graph Parsing: Structured graph construction from text proceeds in three stages:
  1. Simplification: Text is split into atomic steps.
  2. Standardization: Steps are mapped to canonical ingredient, technique, and measurement vocabularies; ambiguous descriptors are normalized.
  3. Structured IE/Graph Construction: For each atomic step, NER+SRL techniques identify entities and actions; process and transfer nodes are instantiated, $a_j$3 and $a_j$4 edges assembled, and type/acyclicity constraints checked. Pseudocode is provided for this stage (Kumbhakern et al., 4 Sep 2025).

A plausible implication is that advances in LLM-driven parsing and semantic alignment will further streamline robust ADTG induction from natural language sources.

5. Applications: Task Tracking, Planning, and Automation

ADTG enables a range of downstream tasks via its explicit graphical structure and action-centric embeddings:

• Real-Time Task Tracking: At each timestep, the current action is inferred by maximizing the task-tracking score over observed features and action histories. This is analogous to HMM filtering but employs discriminatively trained MLP scorers.

• Next-Action Recommendation: For a localized action $a_j$5, its graph edges enumerate admissible next actions, which are then scored and selected by the next-action module.
• Multi-Step Planning: Planning is achieved via autoregressive beam search over the graph, rolling out next actions to generate full or partial plans, computing cumulative log-likelihoods, and stopping at end-of-sequence or terminal nodes (Mao et al., 2023).
• Robotic/Cooking Execution: In the recipe DSL, acyclicity, explicit environments, and process/transfer nodes support partial-order scheduling, opportunistic parallelism, resource contention/resolution, modular subgraph invocation, and sensor-driven outcome predicates, facilitating automated execution in robotic or orchestration contexts (Kumbhakern et al., 4 Sep 2025).

6. Empirical Results and Comparative Performance

On the CrossTask benchmark (18 procedural tasks, $a_j$6 videos):

• Task-Tracking Accuracy (Excluding Nulls): ADTG achieves 55.7% vs. NTG (Neural Task Graph) at ≈24.7%, an improvement of +30.1 percentage points.
• Next-Action Accuracy: ADTG reaches 52.3% vs. NTG at 32.3%, a gain of +20.0 percentage points.
• Planning: Complete plan generation accuracy is 19.0%, plan after prefix is 29.4%, and mean Intersection-over-Union (mIoU) is ≈0.63.
• Qualitative: ADTG beam search produces reasonable plans even with non-canonical action orders; most errors derive from initial localization failures that propagate (Mao et al., 2023).

These results provide evidence for the superiority of action-centric, dynamics-aware graph formalisms over strictly sequence-based or weakly-supervised alternatives in procedural reasoning tasks.

7. Scalability, Extensibility, and Future Considerations

ADTG frameworks address scalability by minimizing redundant nodes (via implicit representation of intermediate artifacts), supporting subgraph modularity, and permitting extensible vocabularies for techniques, tools, and environments. Resource constraints such as container occupancy and mutual exclusion are explicitly modeled, allowing sophisticated scheduling and execution strategies.

The ontological extensibility and compatibility with perceptual, language, and structural data position ADTG as a foundation for future research in automated task understanding, culinary robotics, and instructional support. Continued advances in semantic parsing, representation learning, and task graph induction are likely to further enhance the effectiveness and breadth of Action Dynamics Task Graphs (Mao et al., 2023, Kumbhakern et al., 4 Sep 2025).