VineLM: Robotics & LLM Workflow Control

• VineLM is a term covering two distinct systems: one for vision-language-action robotics and one for agentic LLM workflows, each addressing unique planning challenges.
• In robotics, VineLM employs a hierarchical reinforcement learning framework with a 2D scene-graph and failure-aware planning to utilize both successful and failed demonstrations.
• In LLM workflows, VineLM uses a trie-based control method to perform fine-grained, per-invocation model selection, optimizing cost, latency, and accuracy.

“VineLM” designates two distinct systems in recent arXiv literature. In robotics, VINE—explicitly described as “a.k.a. VineLM”—is a hierarchical vision-language-action model that uses both successful and failed teleoperation trajectories to perform failure-aware planning over a 2D scene-graph abstraction (Park et al., 3 Dec 2025). In LLM systems research, VineLM is a workflow manager for agentic pipelines that performs fine-grained, per-invocation model selection over an annotated trie of feasible executions, with checkpointing, cascade profiling, and runtime replanning under request-level cost or latency objectives (Pagonas et al., 9 Apr 2026). The shared name obscures a substantive divergence: one formulation addresses embodied manipulation under a hierarchical RL formalism, while the other addresses orchestration of configurable LLM stages in refinement-heavy workflows.

1. Terminological scope and disambiguation

The term “VineLM” is not used uniformly. One usage appears in the paper “Hierarchical Vision Language Action Model Using Success and Failure Demonstrations,” where the system is introduced as VINE and identified in the accompanying technical description as “a.k.a. VineLM” (Park et al., 3 Dec 2025). A second usage appears in the paper “VineLM: Trie-Based Fine-Grained Control for Agentic Workflows,” where VineLM names a workflow manager for agentic execution (Pagonas et al., 9 Apr 2026).

Usage	Domain	Core abstraction
VINE / VineLM	Vision-language-action robotics	Hierarchical planning over 2D scene-graph nodes and subgoal edges
VineLM	Agentic LLM workflows	Annotated trie of model-choice prefixes

This naming collision matters because the two systems solve different optimization problems, operate over different state spaces, and define “planning” differently. In the robotics usage, planning selects symbolic subgoal transitions that are then grounded into continuous control. In the workflow usage, planning selects LLM models for successive stage invocations as execution unfolds. A plausible implication is that citations to “VineLM” require immediate domain disambiguation to avoid conflating failure-aware manipulation planning with trie-based workflow control.

2. VineLM in robotics: hierarchical VLA with failure-aware planning

In the robotics formulation, VINE is a hierarchical vision-language-action model that “separates high-level reasoning (System 2) from low-level control (System 1) under a hierarchical reinforcement learning formalism,” with the explicit purpose of making failed demonstrations “usable as a structured learning signal rather than noisy supervision” (Park et al., 3 Dec 2025). System 2 operates in an abstract space of 2D scene-graph nodes $n \in \mathcal{N}$ and subgoal edges $e \in \mathcal{E}$. It performs a brief look-ahead tree search “à la MCTS” using a learned world model to predict successor nodes and a learned value head $V_\theta(n \mid \text{context})$ to score feasibility. Once it selects a highest-value plan $\tau^* = (n_0, e_0, \ldots, n_K)$, it passes only the next edge $e_0$ and predicted successor node $n_1$ to System 1.

System 1 functions as an intra-option controller. Given $(s_t, \ell, e_k, n_{k+1})$, it executes a chunk of continuous joint actions until a learned “done” signal $\beta_\theta$ triggers, after which control returns to System 2 or advances to the next edge in $\tau^*$. The paper frames this decomposition as a way to inject failure-aware reasoning “without retraining the low-level skill policy.”

The formal substrate is a semi-Markov decision process over the high-level sequence $(n_k)$. Each edge $e \in \mathcal{E}$0 defines an option

$e \in \mathcal{E}$1

with initiation set $e \in \mathcal{E}$2, policy $e \in \mathcal{E}$3, termination rule $e \in \mathcal{E}$4, and projection $e \in \mathcal{E}$5. Mixed-quality trajectories $e \in \mathcal{E}$6 are collected, where $e \in \mathcal{E}$7 at success-terminal and $e \in \mathcal{E}$8 otherwise, and nodes are defined by a 2D scene graph $e \in \mathcal{E}$9. The high-level reward is sparse: $V_\theta(n \mid \text{context})$0 when $V_\theta(n \mid \text{context})$1 hits a goal set $V_\theta(n \mid \text{context})$2, and $V_\theta(n \mid \text{context})$3 otherwise; episodes also terminate upon entering a failure set $V_\theta(n \mid \text{context})$4. The first-exit return is

$V_\theta(n \mid \text{context})$5

so that

$V_\theta(n \mid \text{context})$6

Under policy $V_\theta(n \mid \text{context})$7, the Bellman-style relation is

$V_\theta(n \mid \text{context})$8

This formulation places feasibility, rather than imitation of only successful rollouts, at the center of the decision loop. The paper’s stated contribution is therefore not merely hierarchical decomposition, but the conversion of negative experience into an explicit planning signal.

3. Scene-graph abstraction, subgoal generation, and feasibility-guided search

The robotics VineLM uses a 2D scene-graph abstraction $V_\theta(n \mid \text{context})$9 composed of object nodes, a gripper node, and edge relations (Park et al., 3 Dec 2025). Object nodes are represented by bounding boxes from GroundingDINO and semantic labels via a VLM, specifically Gemini-2.5. The gripper node is a projected 2D position obtained from known robot extrinsics. Edge relations are coarse spatial predicates such as “inside” and “left-of,” inferred geometrically or by the VLM. The graph is serialized into text for the LM head.

Nodes $\tau^* = (n_0, e_0, \ldots, n_K)$0 are defined at keyframes detected by gripper state changes, including grasp start or end. Edges $\tau^* = (n_0, e_0, \ldots, n_K)$1 are symbolic subgoals, with examples including “push box,” “grasp spoon,” and “insert salmon in drawer,” aligned with the transition $\tau^* = (n_0, e_0, \ldots, n_K)$2. During tree search, the LM head autoregressively proposes next-node text $\tau^* = (n_0, e_0, \ldots, n_K)$3 and then edge text $\tau^* = (n_0, e_0, \ldots, n_K)$4, grounded in the existing graph and instruction $\tau^* = (n_0, e_0, \ldots, n_K)$5.

The look-ahead tree is denoted $\tau^* = (n_0, e_0, \ldots, n_K)$6. At each expansion step, a batch $\tau^* = (n_0, e_0, \ldots, n_K)$7 of frontier nodes is selected by current mean $\tau^* = (n_0, e_0, \ldots, n_K)$8-values. For each selected parent $\tau^* = (n_0, e_0, \ldots, n_K)$9, the system uses $e_0$0-beam decoding to propose candidate edges and greedy decoding to predict a child node $e_0$1. Each child is immediately scored by the learned value head:

$e_0$2

The value is then backed up along the visited path:

$e_0$3

Low-feasibility branches thus receive low $e_0$4 and are rarely re-expanded.

The returned plan is the trajectory traced from the leaf with largest $e_0$5:

$e_0$6

Proposal selection blends exploitation and exploration through

$e_0$7

where $e_0$8 is the LM proposal probability and $e_0$9 mixes the two terms.

This design makes feasibility scoring the central planner primitive. Rather than treating the LM purely as a generator of subgoals, the system uses the value head to reject brittle branches before execution. A plausible implication is that the architecture is less concerned with broad hypothesis generation than with failure-aware branch ranking under sparse terminal supervision.

4. Integration of success and failure demonstrations in robotics

The robotics paper is explicit that prior VLA models are “typically trained on teleoperated successful demonstrations, while discarding numerous failed attempts,” and that these failures encode where policies are fragile (Park et al., 3 Dec 2025). VINE’s contribution is to train on mixed-quality data so that failure examples directly shape the high-level value function.

The offline dataset for System 2, denoted $n_1$0, contains tuples

$n_1$1

drawn from both successful trials, which terminate in $n_1$2, and failed trials, which terminate in $n_1$3. The language-model loss for generating nodes and edges is

$n_1$4

The first-exit bootstrapped value target is

$n_1$5

The value objective is an Expectile-TD loss:

$n_1$6

with

$n_1$7

The description states that this is used “to discourage overestimation,” and that failures act as negative examples that push down $n_1$8 on brittle branches.

System 2 is fine-tuned from a merged backbone $n_1$9 with LoRA adapters, optimizing $(s_t, \ell, e_k, n_{k+1})$0 and using an EMA target for $(s_t, \ell, e_k, n_{k+1})$1. Training is tuned to convergence, “on the order of 10–20 epochs,” with “exact LR $(s_t, \ell, e_k, n_{k+1})$2.” System 1 is trained only on successful demonstrations $(s_t, \ell, e_k, n_{k+1})$3. Its action expert uses a flow-matching objective

$(s_t, \ell, e_k, n_{k+1})$4

and the done expert uses focal loss

$(s_t, \ell, e_k, n_{k+1})$5

The joint loss is

$(s_t, \ell, e_k, n_{k+1})$6

with learning rates “$(s_t, \ell, e_k, n_{k+1})$7” and training for “10–15 epochs.”

The empirical setup includes two teleoperation datasets: plug insertion, with “450 human trajectories covering all insertion orders (six 3-socket, two 2-socket), naturally yielding $(s_t, \ell, e_k, n_{k+1})$8 failure,” and drawer packing, with “240 teleop demos across three strategies (Pick/Push/Leave) and two distractor types, with $(s_t, \ell, e_k, n_{k+1})$9–$\beta_\theta$0 success depending on object.” Both are split into seen and unseen configurations involving “novel colors, shapes, placements.”

Quantitatively, Table 1 reports that unified VLAs such as $\beta_\theta$1 average “$\beta_\theta$2,” while “$\beta_\theta$3+reward cond. (failure data) drops to $\beta_\theta$4.” VLM-as-planner baselines using GPT-4o and Gemini-2.5 average “$\beta_\theta$5 (no failure examples)” up to “$\beta_\theta$6 (with failure in-context).” The VINE variants are reported as “VINE-Chain (no tree, no failures) at $\beta_\theta$7,” “VINE-Tree (tree only) at $\beta_\theta$8,” and “VINE-Full (tree + failure value) at $\beta_\theta$9,” corresponding to “a $\tau^*$0 relative boost over the best baseline in plug insertion and $\tau^*$1 in drawer packing.” On unseen splits, “VINE-Full” reaches “$\tau^*$2, versus $\tau^*$3 for the best baselines.”

Ablations isolate the role of failure data and tree search. “VINE-Tree vs. VINE-Full” lifts unseen success “in plug insertion from $\tau^*$4 and in drawer packing from $\tau^*$5.” “VINE-Chain $\tau^*$6 VINE-Tree” improves unseen drawer “from $\tau^*$7.” Test-time scaling with beam width “$\tau^*$8 from $\tau^*$9” raises unseen plug insertion “from $(n_k)$0 at a roughly linear compute cost.” The paper also states that “MCTS-style backup outperforms fixed-order DFS both in success rate and inference speed,” and that interpolating $(n_k)$1 and PaliGemma weights at “$(n_k)$2” yields the best average performance.

5. VineLM in agentic workflows: trie-based fine-grained control

In the workflow formulation, VineLM addresses agentic workflows that interleave tool stages and configurable LLM stages, often with retries or refinement loops (Pagonas et al., 9 Apr 2026). Existing coarse-grained controllers, exemplified by Murakkab, choose one model per stage template up front, reuse that model across all loop iterations, and never revisit the choice at runtime. The paper argues that such control fails to mix models across loop iterations and cannot react to realized cost or latency deviations.

VineLM’s alternative is fine-grained, per-invocation control. The workflow template $(n_k)$3 contains configurable stages at which the system may choose among models $(n_k)$4. If $(n_k)$5 denotes the sequence of configurable-stage invocations along one run, a workflow path is

$(n_k)$6

where $(n_k)$7 and different terminating paths may have different lengths $(n_k)$8.

All feasible executions are represented as an annotated trie $(n_k)$9. The root is the empty prefix $e \in \mathcal{E}$00. Each node $e \in \mathcal{E}$01 corresponds to a prefix of a feasible path, and edges connect legal one-step extensions. Each node is annotated with expected cumulative metrics:

$e \in \mathcal{E}$02

$e \in \mathcal{E}$03

$e \in \mathcal{E}$04

Here, $e \in \mathcal{E}$05 indicates whether the workflow up to prefix $e \in \mathcal{E}$06 returns the correct answer on request $e \in \mathcal{E}$07; monetary cost is

$e \in \mathcal{E}$08

with zero cost for unreached stages; and latency is conditioned on $e \in \mathcal{E}$09 “to avoid undercounting latency on the requests that actually reach that stage.”

The trie satisfies monotonicity along root-to-leaf paths:

$e \in \mathcal{E}$10

whenever $e \in \mathcal{E}$11 prefixes $e \in \mathcal{E}$12. This supports request-level objectives such as maximizing $e \in \mathcal{E}$13 subject to $e \in \mathcal{E}$14, maximizing $e \in \mathcal{E}$15 subject to $e \in \mathcal{E}$16, or minimizing $e \in \mathcal{E}$17 subject to $e \in \mathcal{E}$18. The oracle offline search is

$e \in \mathcal{E}$19

The paper notes that monotonicity allows large subtrees to be pruned during search.

The central contribution is therefore not merely profiling candidate workflows, but representing execution as a prefix tree over model choices and controlling that tree at invocation granularity rather than at the level of an entire workflow configuration.

6. Cascade profiling, checkpointing, runtime rerooting, and evaluation

Estimating the trie annotations exactly would require exhaustive profiling over all requests and leaf paths, which the paper characterizes as infeasible because $e \in \mathcal{E}$20 can reach “hundreds of thousands of runs” (Pagonas et al., 9 Apr 2026). VineLM therefore combines cascade profiling with checkpointing and runtime replanning.

Cascade sampling proceeds by selecting a profiling query $e \in \mathcal{E}$21 at random, sampling a random depth-1 model $e \in \mathcal{E}$22 and executing stage 1, stopping if it succeeds, otherwise sampling $e \in \mathcal{E}$23 at depth 2, and continuing until success or maximal depth. When a run succeeds at prefix $e \in \mathcal{E}$24, VineLM marks “$e \in \mathcal{E}$25 and all its descendants as successful at no extra cost.” The paper identifies an MNAR bias in naive column means at deeper nodes, because only hard queries reach deep levels. It then decomposes success recursively. Writing $e \in \mathcal{E}$26, the recursion is

$e \in \mathcal{E}$27

$e \in \mathcal{E}$28

$e \in \mathcal{E}$29

At deep levels, where conditional estimates become noisy, VineLM fits “a rank-1 SVD on the depth-$e \in \mathcal{E}$30 conditional-accuracy matrix” and projects back, exploiting “a single latent ‘difficulty’ axis.” At “$e \in \mathcal{E}$31 of the full offline LLM profiling cost,” the cascade estimator achieves “$e \in \mathcal{E}$32 column-mean MAE,” whereas “standard matrix-completion or simple averaging remain at $e \in \mathcal{E}$33–$e \in \mathcal{E}$34 MAE.”

Checkpointing is defined at trie nodes: after completing a prefix $e \in \mathcal{E}$35, the system serializes the workflow state, including “all intermediate artifacts,” “random seeds,” and “loop counters and retry budgets.” To profile a deeper prefix $e \in \mathcal{E}$36 extending $e \in \mathcal{E}$37, the profiler deserializes the checkpoint at $e \in \mathcal{E}$38, executes only the suffix, and re-serializes at $e \in \mathcal{E}$39. The stated benefits are “no redundant reruns of shared prefixes,” “lower variance in shared-prefix measurements,” and containment of failures because checkpoints live briefly in isolated subprocesses.

At serving time, VineLM re-roots the trie after each invocation. Starting from prefix $e \in \mathcal{E}$40, it subtracts $e \in \mathcal{E}$41 and $e \in \mathcal{E}$42 from the request’s budget or SLO, solves the same constrained search problem over the remaining subtree $e \in \mathcal{E}$43, chooses the first next model $e \in \mathcal{E}$44 on the selected path, executes the next stage, then updates the prefix and observed cost or latency. The paper reports a load-aware variant that augments offline latencies using engine-specific slowdowns $e \in \mathcal{E}$45:

$e \in \mathcal{E}$46

Its per-invocation replanning cost is reported as “$e \in \mathcal{E}$47 ms,” which is “$e \in \mathcal{E}$48 of typical LLM call latency.”

The evaluation covers three workflows: “NL2SQL-8,” with “8 candidate models, one Generation + up to 2 Repairs (depth 3, 584 paths)”; “NL2SQL-2,” with “2 models, one Generation + up to 3 Repairs (depth 4, 30 paths)”; and “MathQA-4,” with “4 models, self-reflection up to 6 rounds (depth 6, $e \in \mathcal{E}$49 theoretical, but constrained by stop logic).” Baselines include Murakkab, “VINELM (full)” with exhaustive checkpointed profiling, and “VINELM (sparse)” with “0.2–2% offline profiling cost with cascade + low-rank estimator.”

At fixed cost budget, the reported accuracy gains over Murakkab are “up to $e \in \mathcal{E}$50” on NL2SQL-8, “up to $e \in \mathcal{E}$51” on NL2SQL-2, and “up to $e \in \mathcal{E}$52” on MathQA, with sparse VineLM still achieving “$e \in \mathcal{E}$53,” “$e \in \mathcal{E}$54,” and “$e \in \mathcal{E}$55,” respectively. In profiling cost, checkpointed exhaustive profiling is “$e \in \mathcal{E}$56–$e \in \mathcal{E}$57 cheaper than naive full,” while sparse VineLM is “$e \in \mathcal{E}$58 cheaper on NL2SQL-2, $e \in \mathcal{E}$59 on NL2SQL-8, $e \in \mathcal{E}$60 on MathQA.” Figure 1 reports “VINELM $e \in \mathcal{E}$61 MAE versus $e \in \mathcal{E}$62 for naive averaging” at $e \in \mathcal{E}$63 coverage. Figure 2 reports that per-invocation replanning “cuts violations by up to $e \in \mathcal{E}$64 vs Murakkab,” with further reduction from the load-aware variant.

The paper’s discussion identifies several limits and extensions: distribution shift and drift detection through monitoring deviations between observed and offline statistics; resource-aware integration with higher-level hardware provisioning; automatic accounting of context-prefill overhead in $e \in \mathcal{E}$65 and $e \in \mathcal{E}$66; possible incorporation of query-difficulty signals; joint multi-timescale control with autoscaling; and extension from trie-structured workflows to “general DAG workflows.”

Taken together, the two VineLM usages share a structural interest in hierarchical or prefix-conditioned control, but they are otherwise unrelated. The robotics system uses failure-aware value learning to rank embodied subgoal trajectories without online rollouts, whereas the workflow system uses trie annotations, sparse profiling, and runtime rerooting to improve the cost-latency-accuracy frontier of agentic LLM workflows (Park et al., 3 Dec 2025, Pagonas et al., 9 Apr 2026).