Video-to-Behavior Trees

Updated 12 July 2026

Video-to-BT is a hierarchical framework that extracts structured, executable behavior trees from human demonstration videos to facilitate robotic assembly tasks.
It combines high-level cognitive planning through vision-language models with a low-level reactive control layer that dynamically updates a symbolic world state.
Experimental results demonstrate high planning accuracy and robust recovery under external disturbances, underscoring its practical impact on real-world robotic assembly.

Video-to-BT is a hierarchical framework for generating reactive Behavior Trees from human demonstration videos for robotic assembly. In its 2025 formulation, the system takes a human demonstration video $V$ and object set $O$ , uses a Vision-LLM to extract an ordered sequence of subtasks $\{\tau_i\}$ , synthesizes a structured Behavior Tree $\Pi$ , and executes that tree through a low-level reactive layer that continuously updates a symbolic world state $\omega^t=(P^t,R^t)$ from perception; replanning is invoked when execution fails (Zhao et al., 20 Sep 2025). The label is not globally unique across video research: in adjacent work, “Video-to-BT” also denotes conversion of generative video outputs into Bradley–Terry ratings, and “BT” can denote a Branching Temporal representation for video conversation models (Inbasekar et al., 5 May 2026, Liu et al., 2023).

1. Terminological scope and research context

The principal use of Video-to-BT in robotics is the framework titled “Video-to-BT: Generating Reactive Behavior Trees from Human Demonstration Videos for Robotic Assembly,” where BT means Behavior Tree. In nearby video literature, however, the same abbreviation is overloaded.

Source	Meaning of “BT”	Video-to-BT transformation
“Video-to-BT: Generating Reactive Behavior Trees from Human Demonstration Videos for Robotic Assembly” (Zhao et al., 20 Sep 2025)	Behavior Tree	Human demonstration video $\rightarrow$ structured BT $\Pi$ for execution
“WorldJen: An End-to-End Multi-Dimensional Benchmark for Generative Video Models” (Inbasekar et al., 5 May 2026)	Bradley–Terry rating	Generative video outputs $\rightarrow$ pairwise wins $W_{ij}$ $\rightarrow$ BT ratings
“BT-Adapter: Video Conversation is Feasible Without Video Instruction Tuning” (Liu et al., 2023)	Branching Temporal	Raw video $O$ 0 Branching-Temporal representation for video–LLM conversation

A common misconception is that “BT” in video research unambiguously denotes Behavior Trees. The available literature does not support that assumption. In the robotic assembly framework, BT refers to the control formalism used for structured planning and execution. In WorldJen, BT refers to Bradley–Terry scores fitted from human or VLM-derived pairwise comparisons. In BT-Adapter, BT refers to a temporal adapter architecture attached to a frozen CLIP backbone. The acronym therefore requires local interpretation from the surrounding research problem.

Within that landscape, the robotic Video-to-BT framework is specifically concerned with extracting executable task structure from demonstrations rather than learning video embeddings or ranking model outputs. Its distinguishing property is the coupling of VLM-based task interpretation with reactive symbolic execution in a real-world assembly setting.

2. Hierarchical architecture and problem formulation

The Video-to-BT system is organized into two principal layers. The High-Level Cognitive Planning Layer takes as input a human demonstration video $O$ 1 and object set $O$ 2, processes the video via a Vision-LLM to extract a sequence of subtasks $O$ 3, generates a structured Behavior Tree $O$ 4, and provides occasional human-in-the-loop verification of VLM outputs. The Low-Level Reactive Control Layer contains a Perception Module, which builds and continuously updates the symbolic world state $O$ 5 by fusing 2D segmentation from SAM 2 and 6D-pose estimation from FoundationPose, and an Execution Module, which executes the BT $O$ 6 by ticking it at high frequency, monitors for condition failures, and invokes the Planning Module to replan when needed (Zhao et al., 20 Sep 2025).

This architecture is explicitly hierarchical rather than monolithic. Planning produces a BT as a structured representation of intended task logic, while execution remains reactive to the current state of the scene. The design therefore separates long-horizon decomposition from low-level state monitoring. A plausible implication is that the framework uses the BT both as an interpretable plan representation and as the runtime control graph, which is consistent with the paper’s description that BTs serve both as the structured output of planning and as the governing structure for execution.

The world state $O$ 7 is symbolic. $O$ 8 stores predicates and object properties, while $O$ 9 stores relations. This state is not static: it is refreshed through perception during execution and is the basis for condition checking, failure detection, rollback, and replanning. That continuous symbolic maintenance is central to the claim of reactivity in dynamic environments.

3. Behavior Tree semantics

The framework formalizes a Behavior Tree $\{\tau_i\}$ 0 as a set of nodes partitioned into control-flow nodes $\{\tau_i\}$ 1 and execution nodes $\{\tau_i\}$ 2. Every node $\{\tau_i\}$ 3 is associated with a tick function

$\{\tau_i\}$ 4

which returns the node status in world state $\{\tau_i\}$ 5 (Zhao et al., 20 Sep 2025).

A Sequence node $\{\tau_i\}$ 6 executes its children left-to-right. It returns Running if some child $\{\tau_i\}$ 7 is Running and all earlier children have succeeded, Failure if some child $\{\tau_i\}$ 8 fails after all earlier children succeeded, and Success only if all children succeed: $\{\tau_i\}$ 9

A Fallback or Selector node $\Pi$ 0 returns success as soon as one child succeeds or is running; it fails only if all children fail: $\Pi$ 1

A Condition node $\Pi$ 2 evaluates a boolean predicate over $\Pi$ 3: $\Pi$ 4

An Action leaf $\Pi$ 5 invokes a skill $\Pi$ 6 on objects $\Pi$ 7, executes asynchronously, returns Running while the robot executes, returns Success upon completion with $\Pi$ 8, and returns Failure if an error occurs: $\Pi$ 9

These semantics make the BT an executable state machine with explicit failure propagation. In the context of robotic assembly, that matters because failure is not terminal; it is a signal used to trigger subtree replacement or rollback to earlier subtasks.

4. VLM-based video interpretation and BT synthesis

The video interpretation stage uses three inputs: key frames $\omega^t=(P^t,R^t)$ 0 manually selected to cover each major step, an audio track converted to a speech-to-text transcript $\omega^t=(P^t,R^t)$ 1, and an object vocabulary $\omega^t=(P^t,R^t)$ 2 with skill symbols $\omega^t=(P^t,R^t)$ 3 (Zhao et al., 20 Sep 2025). The VLM is prompted to extract an ordered subtask list

$\omega^t=(P^t,R^t)$ 4

and to list all binary inter-object constraints

$\omega^t=(P^t,R^t)$ 5

The returned $\omega^t=(P^t,R^t)$ 6 and $\omega^t=(P^t,R^t)$ 7 are then human-verified in one short HITL cycle.

BT synthesis proceeds subtask by subtask. For a current state $\omega^t=(P^t,R^t)$ 8, Stage A is action decomposition via LLM: given subtask $\omega^t=(P^t,R^t)$ 9, domain vocabulary $\rightarrow$ 0, constraints $\rightarrow$ 1, and state $\rightarrow$ 2, the model proposes a primitive action sequence

$\rightarrow$ 3

Stage B is subtree generation via LLM: using $\rightarrow$ 4, $\rightarrow$ 5, $\rightarrow$ 6, and $\rightarrow$ 7, the model generates a Behavior Tree subtree $\rightarrow$ 8 that “(1) checks preconditions, (2) invokes each $\rightarrow$ 9, (3) asserts postconditions.” Each subtree is human-verified, then virtually ticked to update the state: $\Pi$ 0 The overall BT is represented as $\Pi$ 1.

Instantiation is direct. Any domain predicate $\Pi$ 2 or relation $\Pi$ 3 becomes a Condition node, while action parameters $\Pi$ 4 are embedded directly as Action leaves. This gives the generated BT a mixed symbolic-operational character: predicates encode what must be true, and action leaves encode what the controller should do.

5. Reactive execution, world-state maintenance, and replanning

Execution interleaves tree ticking with world-state updates. For each subtree $\Pi$ 5, the execution loop extends it with previous relations $\Pi$ 6, forming

$\Pi$ 7

then repeatedly applies

$\Pi$ 8

at frequency $\Pi$ 9 until the subtree succeeds or fails (Zhao et al., 20 Sep 2025).

Failure handling is structured. If the failed node $\rightarrow$ 0 corresponds to a prior relation $\rightarrow$ 1 with $\rightarrow$ 2, the system rolls back to subtask $\rightarrow$ 3, replans $\rightarrow$ 4 in the current state $\rightarrow$ 5, and resumes from that point. If the failed node lies inside the current subtree $\rightarrow$ 6, only $\rightarrow$ 7 is replanned. This is a finer-grained mechanism than restarting the entire task: rollback is localized to the earliest invalidated relation or the active subtree.

In parallel, a world-state maintenance thread updates $\rightarrow$ 8 using the current action and perceptual modules. Three update rules are specified. Position invariance: if $\rightarrow$ 9 detects that any uninvolved object $W_{ij}$ 0 moved, its mask and position in $W_{ij}$ 1 are updated. Relation validity: $W_{ij}$ 2 Pose consistency: if $W_{ij}$ 3 reports an unexpected 6D change for $W_{ij}$ 4, the predicate $W_{ij}$ 5 is removed from $W_{ij}$ 6 until updated.

This execution model makes the BT reactive in a precise sense: conditions are reevaluated under a continuously refreshed symbolic state, and execution failures can invalidate earlier relational assumptions. A plausible implication is that the framework treats environmental disturbances as first-class runtime events rather than as off-distribution exceptions outside the control logic.

6. Experimental validation and reported performance

The reported experiments use gear-set assembly tasks with 1, 3, and 5 subtasks on a Franka Panda robot, together with three classes of external disturbances: (I) moving current target object, (II) moving future objects, and (III) invalidating a previously reached relation (Zhao et al., 20 Sep 2025).

Planning performance is evaluated on 25 test videos using Task Decomposition Accuracy (TDA), Logical Coherence Rate (LCR), and Syntactic Validity Rate (SVR). Average results across three VLMs are as follows:

VLM	Initial	After 1 HITL pass
GPT-4o	TDA 98%, LCR 99%, SVR 100%	TDA 100%, LCR 100%, SVR 100%
Gemini-1.5	TDA 81%, LCR 61%, SVR 99%	TDA 94%, LCR 88%, SVR 100%
Qwen-2.5	TDA 76%, LCR 80%, SVR 100%	TDA 79%, LCR 100%, SVR 100%

These numbers indicate that a single HITL pass materially changes planning quality, particularly for TDA and LCR. The strongest result is the GPT-4o setting, which reaches 100% on all three metrics after one HITL pass.

Perception is evaluated through mask-to-name matching accuracy over 15 trials with the GPT-4o pipeline, reaching 100% perfect matches within 5 human-assisted refinements. This result pertains specifically to the symbolic grounding layer that connects visual instances to the object vocabulary used by the planner and BT.

Execution is reported over 15 trials for each setting. Without disturbance, Completion Rate (CR) is at least 90% and Task Success (TS) is at least 80%. Under disturbances I–III, CR is at least 75%, TS is at least 60%, and Disturbance Recovery Rate (DRR) is approximately 75–100%, degrading with longer horizons. The paper interprets these results as evidence of high planning reliability, robust reactive BT execution under real-world perturbations, and strong generalization across varied demonstration styles, object sets, and environmental disturbances.

In WorldJen, “Video-to-BT” denotes a process for converting generative video outputs into a Bradley–Terry rating rather than a Behavior Tree. The pipeline begins with prompt curation from a large human-authored pool, followed by a blind human preference study and a VLM-as-a-judge evaluation engine. The human study uses 50 curated prompts, $W_{ij}$ 7 models, 300 videos, all 15 pairwise comparisons among the 6 models for each prompt, and 2,696 votes with 100% coverage. The VLM pipeline generates 10 targeted Likert questions per applicable dimension, produces 47,160 scored responses, converts averaged per-prompt scores into pairwise wins $W_{ij}$ 8, and fits a Bradley–Terry model with worth parameters $W_{ij}$ 9, where

$\rightarrow$ 0

Human BT produced three clear tiers among 6 models, and VLM BT reproduced the exact same three-tier partition, with Spearman $\rightarrow$ 1 and Kendall $\rightarrow$ 2 (Inbasekar et al., 5 May 2026).

In BT-Adapter, BT means Branching Temporal, not Behavior Tree or Bradley–Terry. The method adds a learned temporal branch to a frozen OpenAI-CLIP (ViT-L/14) visual encoder, runs the branch in parallel to the last $\rightarrow$ 3 CLIP layers with $\rightarrow$ 4, uses temporal self-attention over spatially aligned patch “tubes,” applies asymmetric token masking only in the branch at mask ratio $\rightarrow$ 5, and jointly optimizes video–text contrastive, masked branching token alignment, and masked branching cross-modal alignment losses. After pretraining, CLIP and BT-Adapter are frozen and the combined encoder can replace the image-only CLIP in CLIP $\rightarrow$ 6Q-former $\rightarrow$ 7LLM pipelines such as LLaVa, MiniGPT-4, and InstructBLIP (Liu et al., 2023).

Taken together, these usages show that “Video-to-BT” is not a single method class but a family of video-to-structure mappings whose output structure depends on the surrounding field: executable symbolic controllers in robotics, ranking models in evaluation, or temporal representations in multimodal modeling. The shared pattern is conversion from raw video or generated video into an explicitly structured intermediate object, but the semantics of that object differ substantially across domains.