DIRECT: When and Where Should You Allocate Test-Time Compute in Embodied Planners?

Abstract: Vision-LLMs (VLMs) are increasingly deployed as high-level planners for embodied agents, with an emerging strategy of scaling test-time compute to improve capability. However, we observe that doing so increases latency, token usage, and FLOPs while yielding uneven, often diminishing gains in downstream success, limiting where embodied agents can be deployed. We argue that choosing when and where to spend test-time compute is central to bringing frontier performance to the real world. We introduce DIRECT, a routing framework that uses multimodal scene context to allocate compute per prompt, improving the success--cost Pareto frontier over fixed model selection. Across three dominant scaling axes, namely chain-of-thought depth, model size, and memory history, our experiments on VLABench and RoboMME show that test-time compute is not a uniform lever: different axes yield qualitatively distinct capability gains. We validate these insights on a physical Franka arm in a DROID setup spanning zero-shot manipulation and long-horizon chaining, where our router matches or exceeds a stronger model's success rate at up to 65% lower average latency. Ultimately, our results show that naively scaling test-time compute is wasteful, and that DIRECT can provide frontier-level embodied planning in robotic systems at a fraction of the cost. Project page can be found at jadee-dao.github.io/direct/.

• The paper introduces DIRECT, a dynamic inference router that selectively allocates test-time compute to achieve near-frontier task success with up to 65% lower latency.
• It evaluates multiple planning axes—chain-of-thought depth, model size, and memory augmentation—using embedding-based features for optimal compute routing.
• Empirical results from VLABench and RoboMME show that DIRECT outperforms static, uniform compute scaling by invoking high-cost planners only when necessary.

Test-Time Compute Allocation in Embodied Planners: The DIRECT Framework

Introduction

The increasing deployment of vision-LLMs (VLMs) as high-level planners for hierarchical embodied agents raises urgent questions about the efficient allocation of test-time compute. Recent practice often allocates compute uniformly by scaling model size, chain-of-thought (CoT) depth, or memory horizon, aiming to bridge capability gaps for complex tasks. However, these strategies entail significant increases in latency and resource demands, particularly when deployed in robotics scenarios where such inefficiencies become operationally prohibitive. This paper introduces DIRECT (Dynamic Inference Router for Embodied Compute Tradeoffs), a selective routing framework that dynamically allocates test-time compute for each inference, optimizing the tradeoff between task success and compute/latency.

Problem Context and Motivation

While VLMs serve effectively as instruction decomposers—mapping high-level language instructions and scene observations to executable skill sequences—indiscriminate upscaling of planner compute leads to wasted resources without consistent downstream gains. Latencies from deliberate reasoning (e.g., CoT prompting) or the use of large models can constrain system applicability in real-world settings, especially in closed-loop robotic control. Baseline model routing approaches (e.g., FrugalGPT, RouteLLM) are limited by their reliance on linguistic features alone and do not consider the multimodal complexity of robotic planning, such as physical scene difficulty or temporal dependencies.

Empirical Analysis: Non-Uniformity of Compute Scaling

The paper presents a systematic investigation into the impact of different test-time scaling axes in embodied planning:

• Chain-of-Thought Depth: Deliberate planning through CoT can resolve subtler constraints, but many tasks do not benefit. Notably, on VLABench, 44% of cases with Qwen3-VL 8B Instruct matched or outperformed CoT-enabled planners at <2% of the latency, establishing that most tasks can be addressed efficiently without recourse to heavy test-time reasoning.
• Model Size: Empirical results across Qwen3-VL models (2B–235B) show no monotonic relationship between size and either performance or latency; smaller models sometimes outproduce more expensive ones due to verbosity rather than parameter count. Crucially, gains from larger models concentrate on tasks requiring broader skill coverage, not uniformly across all inputs.
• Memory Horizon/Augmentation: On RoboMME, lightweight token/frame selection strategies outperform sophisticated memory-augmented variants on easy tasks, while architectures like MemER and GroundSG excel only on the hardest tasks requiring long-horizon recall. This reveals the absence of a dominant memory scheme and substantiates the necessity for per-task adaptation.

The DIRECT Routing Framework

Router Architecture

DIRECT formulates embodied test-time compute allocation as a selection problem over a fixed pool $\mathcal{M} = \{m_1, \ldots, m_K\}$, each a VLM planner with distinct quality–cost characteristics. For each input $(I, \ell)$—scene observation and instruction—DIRECT encodes multimodal context with a frozen SigLIP-family vision encoder and BGE-M3 text encoder, concatenating these to obtain task features $\phi(x)$. The router is instantiated via efficient architectures (Linear, KNN/PR-KNN, MLP, k-means, OVR), keeping inference overhead negligible relative to any planner call.

Figure 1: DIRECT routes tasks by predicting the planner with the best quality–cost tradeoff given scene and instruction context.

Data Collection and Synthetic Training

For each planner, quality ($Q$) and inference cost ($C$) are measured across all training tasks. For real hardware, data collection leverages synthetic task generation and LLM-based scoring to avoid combinatorial cost of full-trajectory robot executions.

Routing Objective

Routing is formulated through utility functions $U(q, c)$ combining quality and cost per planner, supporting regression heads for direct prediction and flexible objective specification (including harmonic mean efficiency metrics to compare routers across cost-quality frontiers).

Evaluation and Deployment

Routers are compared using mean task success and either latency (for autoregressive axes such as CoT reasoning) or FLOPs (for memory/configuration axes). The router predicts the optimal planner for each new task independently, enabling dynamic compute allocation with minimal prediction overhead.

Experimental Results

Chain-of-Thought Routing

DIRECT achieves superior routing efficiency $\eta$ compared to baselines, consistently matching or exceeding the success rate of costly CoT planners at 30–65% less latency across VLABench, hardware, and real-world robotic evaluations.

Figure 2: DIRECT selectively escalates planner compute, invoking expensive reasoning only where necessary, and routes to minimal planners where adequate, matching frontier-level performance at a fraction of the cost.

Figure 3: Main results corroborate that only a subset of tasks benefit from heavyweight planners across all compute axes.

Direct per-step routing on temporally extended tasks (e.g., grocery bagging) matches high-reasoning planner performance while eliminating unneeded computation on simple steps—a crucial result for real-world robotic viability.

Model Size Routing

DIRECT seamlessly composes capabilities from multiple model sizes. While the standalone performance curve of individual planners is non-monotonic, cumulative routing over size variants produces monotonic, strictly Pareto-optimal frontiers. The router leverages smaller models on basic tasks, escalating to larger models only when necessary for skill breadth.

Memory Routing

On RoboMME, DIRECT traces a success–cost Pareto frontier above all individual memory architectures, spending only as much on memory augmentation as each task requires. No single memory model is optimal across all difficulty tiers or task suites; routing discovers complementary decompositions for optimal efficiency.

Figure 4: DIRECT's routed Pareto frontier for memory routing on RoboMME outperforms all static memory architectures, highlighting the inefficacy of uniform large-memory deployment.

Figure 5: Enlarged memory-routing frontiers split by difficulty and task suite; routing tracks or surpasses the best specialist at reduced TFLOP budgets.

Numerical Highlights and Contradictory Claims

• In multiple regimes, DIRECT attains the strongest model's average success with up to 65% less latency, which directly contradicts the prevailing assumption that uniform compute scaling is the sole viable path to peak performance.
• For memory architectures, DIRECT's curve sits strictly above the best individual memory system, indicating that the diversity of induced computations is critical and single-model specialization is suboptimal.
• Static deployment of large planners results in substantial resource waste on tasks where lightweight planners suffice without degradation in success rate.

Implications and Future Directions

This work demonstrates that effective task-conditioned routing based on multimodal context can routinely subsume the performance of any single planner configuration at a fraction of the cost across multiple axes. The strong results validate embedding-based feature routing for complex embodied planning and indicate that, as diversity in planner architectures grows, router-based dynamic selection is a necessary system design for real-time robotic control.

In practical terms, DIRECT enables deployment of frontier-level VLM-based planners in latency-sensitive, resource-constrained domains by ensuring that high-compute planners are invoked exclusively on tasks with demonstrated marginal utility. Theoretically, this reframes the role of model variety and scaling in embodied AI: planner diversity and dynamic selection should be considered primary control levers, not just brute-force scaling.

A major limitation is that the router operates on a fixed planner pool, requiring retraining for pool changes, and does not synthesize new policies. Future work should investigate continuous planner space traversal via policy interpolation, meta-routing over evolving model families, and tight theoretical bounds on routing regret in embodied task distributions.

Conclusion

DIRECT operationalizes multimodal context-aware test-time compute allocation in embodied planning, attaining near-frontier success metrics at dramatically lower computational and latency costs. Empirical results advocate for embodied AI systems to cast high-level planner selection as a dynamic, context-sensitive optimization, rather than as a single static choice. This work suggests that embedding-based router architectures, lightweight and efficient, are sufficient to realize most of the available gains in cost-aware planning and underscores the need for continued research in dynamic system orchestration for real-time robotics.

Figure 6: Held-out test-set Pareto frontiers with DIRECT-selected routers, marking cost–quality tradeoffs that approach the oracle upper bound.