FORGE: Towards Functional Tool-Use Generalization via Keypoint Trajectory Reasoning

Abstract: While humans readily repurpose a book, a stone, or a shoe to drive a nail, robots trained on specific tools fail to transfer the same function to novel ones -- a gap we formalize as functional generalization. Such tools share a common functional intent that is visually recognizable, yet this perceptual similarity does not carry over to action space, where each tool demands an entirely different motor pattern. To bridge this gap, we explore intermediate representations including affordance images, human video prompts, and 2D keypoint trajectories, finding that keypoint trajectories best balance functional expressiveness and action groundability. Building on this, we propose FunctiOnal Reasoning and Grounded Execution (FORGE), a two-stage policy that decouples functional reasoning from action execution: predicting generalizable keypoint trajectories from action-free data, then grounding them into robot actions with limited demonstrations. On a seven-tool hitting-function benchmark, FORGE consistently outperforms state-of-the-art methods on unseen tools in both simulation and the real world, achieving over 2X improvement in average success rate.

• The paper presents FORGE, a two-stage policy that decouples functional reasoning and action execution using 2D keypoint trajectories.
• It achieves over a two-fold improvement in success rates on unseen tools in both simulated and real-world hitting tasks compared to baselines.
• FORGE leverages large action-free datasets with minimal action-labeled data to robustly generalize tool-use across diverse object geometries.

Functional Tool-Use Generalization via Keypoint Trajectory Reasoning: An Analysis of FORGE

Problem Setting: Functional Generalization in Tool-Use

Conventional approaches in robotic manipulation predominantly target scene-level and category-level generalization, focusing on robustness to visual variations and transfer across similar object categories (Figure 1). In contrast, functional generalization necessitates that an agent perform a fixed function with entirely novel tools, requiring adaptation of both contact region and motion patterns to accommodate diverse geometries and affordances. Critically, while functionally equivalent tools may share visually recognizable structures, their corresponding optimal actions are typically dissimilar—implying a significant perception-to-action gap.

Figure 1: Comparison of generalization paradigms—(a) scene generalization, (b) category generalization, and (c) functional generalization requiring transfer of function across unseen tools.

Functional generalization presents a stricter generalization regime than those previously addressed in robotics literature, as it demands the derivation and application of generalizable representations that capture function-relevant attributes rather than superficial visual or categorical similarity.

Methodological Framework: The FORGE Pipeline

The authors introduce FunctiOnal Reasoning and Grounded Execution (FORGE), a two-stage policy that decouples functional reasoning from action execution, employing carefully selected intermediate representations as a bridge. The methodological insight is that suitable intermediate representations should capture both the expressive details of a function and be readily groundable into executable robot actions, even for tools unseen during training.

The authors systematically evaluate candidate representations: affordance images (static spatial cues), human video prompts (dense motion, highly entangled with tool appearance), and 2D keypoint trajectories. Empirically, 2D keypoint trajectories best satisfy the dual criteria of functional expressiveness and action groundability, enabling explicit description of the relevant contact region and its intended motion while remaining invariant to tool-specific visual features.

FORGE proceeds in two distinct stages (Figure 2):

1. System-2 – Functional Reasoning: A keypoint motion predictor is trained on action-free data to forecast future function-relevant keypoint trajectories conditioned on current observations. This enables scalable utilization of large, cheaply-available action-free datasets.
2. System-1 – Grounded Execution: Given predicted keypoint plans, an execution policy is trained using limited action-labeled data to ground these trajectories into concrete motor actions. The policy is robustified to moderate keypoint prediction errors through noise-augmented training.

   Figure 2: Schematic overview—(a) selection of the intermediate representation, (b) System-2 predicts keypoint trajectories from observations, (c) System-1 grounds these into robot actions.

Training leverages the asymmetry between abundant action-free and scarce action-labeled data, such that generalizable function priors are learned at scale and only minimal robot-execution data are required for each new tool.

Experimental Protocol: Simulated and Real-World Functional Hitting

The evaluation benchmark comprises both simulation and real-world settings instantiated as a hitting task, involving seven distinct tools (Figure 3). In simulation, the agent is trained on a subset (four) of the tools and tested on the remaining (three) completely unseen ones. In the real world, a Franka robot is tasked with striking targets using both familiar and novel objects with diverse designated hitting regions.

Figure 3: (a) Simulated benchmark: seven tools, diverse initializations. (b) Real-world: Franka robot, three objects tested across hitting patterns.

Baselines span end-to-end policies (Flow-Matching, Diffusion Policy) and keypoint-conditioned policies (ATM), controlling for use of structured representations versus purely raw visual input.

Main Results: Quantitative and Qualitative Insights

Across both simulation and real-world settings, FORGE achieves a greater than two-fold improvement in average success rate compared to all baselines on unseen tools. In simulation, FORGE attains a $0.36$ mean success rate versus $0.17$ (Diffusion Policy), $0.09$ (Flow-Matching), and $0.08$ (ATM). Real-world experiments confirm this trend, with FORGE substantially outperforming end-to-end approaches in reliably aligning function-relevant regions for striking (Figures 4 and 5).

Figure 4: Predicted keypoint-based functional plans on unseen tools—successes (left) and representative failure modes (right, including spatial misalignment and tool dropping).

Figure 5: Real-world performance on the unseen “book” object; FORGE achieves consistently higher accuracy across different hitting patterns than baseline.

Ablations reveal that only 2D keypoint trajectories consistently enable generalization; affordance images and human video prompts contribute, but lack the temporal/geometric specificity. Additionally, the degree of action-labeled tool diversity during training correlates positively with generalization success, underscoring the practical trade-off between demonstration expense and extrapolative capacity.

ATM’s keypoint tracker baseline, while leveraging keypoints, fails to encode function-aware motion, and thus underperforms FORGE. Failure cases for FORGE involve limitations in 2D-only representation for fine-grained spatial alignment and occasional jerky executions, indicating directions for future improvement.

Implications and Future Directions

The empirical findings substantiate that explicitly modeling function-relevant keypoint dynamics facilitates robust policy transfer to truly novel tools, a capability largely unattainable by end-to-end or naive keypoint-tracking methods—even when scaling action-free data.

From a practical robotics perspective, FORGE provides a scalable recipe for learning tool-use: large volumes of unlabelled video or play data can be mined to acquire functional priors, and only modest annotation of robot actions per function is required for robust grounding on new hardware or with new manipulanda.

Theoretically, this work motivates further exploration of intermediate representations for skill generalization, especially those that abstract away object identity in favor of function, geometric affordance, or physical intent. Automatic discovery of function-relevant keypoints (potentially via vision-LLMs), extension to 3D and temporal relational constraints, and unified grasping-motion planning all represent promising extensions.

Conclusion

The FORGE framework formalizes and benchmarks functional generalization in robotic tool-use, introducing a two-stage pipeline that leverages function-aware keypoint trajectories as an intermediate representation. Through systematic experiments, the authors demonstrate that decoupling reasoning and execution via such representations is both necessary and sufficient for strong generalization to unseen tools. While some operational gaps remain—such as automated keypoint discovery and more abstract functional reasoning—FORGE establishes a new standard and practical paradigm for scalable robotic tool-use transfer (2607.05780).