CollaBot: Vision-Language Collaborative Manipulation

• CollaBot is a vision-language-guided framework for simultaneous collaborative manipulation that coordinates multiple mobile robots to move large objects using natural language instructions.
• It integrates advanced perception techniques like SEEM-based segmentation and LoGNet for grasp synthesis, fused with multi-view point cloud data to plan and execute coordinated grasps.
• The framework employs a two-stage motion planning process and foundation-model reasoning to select robot teams and ensure stable, load-distributed manipulation in complex environments.

Searching arXiv for papers on “CollaBot” and closely related collaborative robot frameworks to ground the article. Using the available arXiv materials already supplied, the entry below centers on the named CollaBot framework and situates it within closely related collaborative robotics work. CollaBot is a vision-language-guided framework for simultaneous collaborative manipulation (SCM), in which multiple mobile manipulators grasp and move the same large object concurrently under high-level language guidance. It is designed for large-object tasks in factory or home environments, such as moving tables, where single-robot manipulation is inadequate and the manipulated object induces a closed kinematic chain across several end-effectors. The framework combines SEEM-based scene segmentation, fused multi-view point clouds, collaborative grasp planning, and a two-stage motion-planning stack, and reports a 52% success rate across different numbers of robots, objects, and tasks (Song et al., 5 Aug 2025).

1. Scope, task model, and problem formulation

CollaBot addresses a class of manipulation problems that differ sharply from conventional single-arm grasping. The target objects are large, extend across a broad workspace, and require several robots to support weight and resist torque simultaneously. In this setting, grasping is not a local decision made independently at each end-effector; it is a coupled geometric and wrench-distribution problem in which the relative poses of all grasping end-effectors must remain constant while the object is transported (Song et al., 5 Aug 2025).

A subtask is represented as

$$\mathcal{T} = [O, \mathbf{p}_{\text{init}}, \mathbf{p}_{\text{end}}],$$

where $O$ is the manipulated object and $$\mathbf{p}_{\text{init}}, \mathbf{p}_{\text{end}} \in SE(3)$$ are its initial and desired final poses in a world frame. For a set of $N$ available robots, the framework selects a subset $I_u(\mathcal{T})$ of size $M$, and casts SCM as minimizing the number of engaged robots subject to grasp feasibility, object-pose boundary conditions, closed-chain constraints, and task-level motion constraints inferred from language and vision (Song et al., 5 Aug 2025). In the paper’s formulation, those constraints include

$$\dot{\mathbf{F}}(\mathbf{e}_i,\mathbf{e}_j,t) = \mathbf{0}$$

for all engaged robot pairs, enforcing constant relative end-effector poses during manipulation, and

$\mathbf{G}(\mathbf{p}(t)) \le 0,$

encoding task-specific object-motion constraints such as keeping a tabletop horizontal (Song et al., 5 Aug 2025).

This formulation places CollaBot between classical multi-robot manipulation and recent foundation-model-driven robotics. Classical SCM literature typically assumes known object geometry, fixed grasps, and a small fixed team, often two robots. By contrast, CollaBot uses foundation models to infer the manipulated object, the required number of robots, admissible object-motion constraints, and globally coordinated grasp points directly from RGB-D observations and natural-language instructions (Song et al., 5 Aug 2025).

2. Vision-language grounding and scene perception

The perception stack begins with language understanding. From a high-level instruction, a language or vision-LLM identifies the target object category and supplies the text token used for segmentation. The same family of models is used to infer motion constraints $\mathbf{G}(\mathbf{p}(t))$, such as maintaining object orientation relative to the world frame, and to predict the required number of robots $M \le N$ given the task, object, and scene context (Song et al., 5 Aug 2025).

Object extraction is performed with SEEM, used as a text-conditioned segmentation model. Each robot provides an RGB-D image and a calibrated camera pose. SEEM returns an object mask for the queried category, and the masked depth observations are back-projected into per-view object point clouds $O$0. These are then transformed into the world frame and fused into a global object point cloud

$O$1

which is the geometric substrate for collaborative grasp synthesis (Song et al., 5 Aug 2025).

The role of the VLM is not limited to semantic grounding. It also interprets task context in ways that are hard to express through hand-coded heuristics. In the reported system, VLM prompting is used to infer how many robots are needed for a given object and task, and to translate natural-language constraints such as “without spilling the cups” into pose restrictions on the manipulated object. This suggests a broadening of foundation-model use from semantic recognition to geometric task structuring, though the downstream planner remains explicitly model-based (Song et al., 5 Aug 2025).

3. Collaborative grasping: local grasp synthesis and global coordination

CollaBot decomposes collaborative grasping into two layers: local grasp pose generation and global collaboration. The local layer produces a feasible 6-DoF grasp pose near a candidate contact region. The global layer decides which contact regions should be used jointly by the team so that the resulting multi-contact configuration is stable and collaboration-friendly (Song et al., 5 Aug 2025).

For the local layer, the framework introduces LoGNet, a PointNet-based network trained for large-object grasp synthesis. Training data are generated from ShapeNet large indoor object models and the DA$O$2 dual-arm grasp dataset. For each object and randomized camera view, a local point cloud patch around a reference contact point is extracted as

$O$3

then centered and downsampled to $O$4 (Song et al., 5 Aug 2025). The network outputs a translation $O$5, an approach vector $O$6, and an in-plane angle $O$7, jointly representing a 6-DoF grasp pose in $O$8. Training uses

$O$9

with $$\mathbf{p}_{\text{init}}, \mathbf{p}_{\text{end}} \in SE(3)$$0 (Song et al., 5 Aug 2025).

For the global layer, collision-free candidate contact points are first extracted from the fused object point cloud. The system assumes the fingertip normal should be approximately parallel to the local surface normal, and collision checks discard infeasible placements. The remaining points are clustered with K-means into $$\mathbf{p}_{\text{init}}, \mathbf{p}_{\text{end}} \in SE(3)$$1 candidate grasp points, where $$\mathbf{p}_{\text{init}}, \mathbf{p}_{\text{end}} \in SE(3)$$2, and these cluster centers are projected into a single RGB image and labeled numerically (Song et al., 5 Aug 2025).

A VLM then selects the final $$\mathbf{p}_{\text{init}}, \mathbf{p}_{\text{end}} \in SE(3)$$3 grasp points using a prompt that includes explicit evaluation criteria, chain-of-thought grounding, and example outputs. The stated criteria are: collaborative support relative to the center of mass, gripper-versus-external-force alignment, and local stability. In particular, the prompt encourages grasp selections for which the center of mass lies near the convex hull of the chosen grasp points and for which the grasp directions reduce the tangential friction required to counter gravity (Song et al., 5 Aug 2025). Once the numbered grasp points are selected, local point patches around those points are passed through LoGNet to recover full 6-DoF grasp poses.

The framework evaluates collaborative grasps using several static metrics. One is a relaxed force-closure quantity

$$\mathbf{p}_{\text{init}}, \mathbf{p}_{\text{end}} \in SE(3)$$4

where $$\mathbf{p}_{\text{init}}, \mathbf{p}_{\text{end}} \in SE(3)$$5 is the grasp matrix, and smaller $$\mathbf{p}_{\text{init}}, \mathbf{p}_{\text{end}} \in SE(3)$$6 is უკეთter. A second is the minimum singular value of $$\mathbf{p}_{\text{init}}, \mathbf{p}_{\text{end}} \in SE(3)$$7, with larger values indicating more symmetric wrench capability. A third is the normalized maximum contact force $$\mathbf{p}_{\text{init}}, \mathbf{p}_{\text{end}} \in SE(3)$$8, obtained by solving a minimum-norm equilibrium problem under friction-cone constraints and taking

$$\mathbf{p}_{\text{init}}, \mathbf{p}_{\text{end}} \in SE(3)$$9

Smaller $N$0 indicates that the load is more evenly and efficiently distributed across contacts (Song et al., 5 Aug 2025).

4. Two-stage planning and execution

After selecting the robot subset and collaborative grasps, CollaBot plans in two stages. The first stage moves each participating robot from its random initial configuration to its grasp configuration $N$1. This is performed with RRT* in each robot’s configuration space, with collision checks against the environment, the object, and other robots. The treatment is sequential and deliberately simple; the paper does not frame stage 1 as the principal contribution (Song et al., 5 Aug 2025).

The second stage plans the actual collaborative transport. First, an object-level RRT* trajectory $N$2 in $N$3 is generated from $N$4 to $N$5, subject to the task-level constraints $N$6. This planner reasons at the object level and does not initially encode full robot kinematics or inter-robot collision structure (Song et al., 5 Aug 2025).

Once an object path is found, the system computes desired end-effector trajectories by composing the object trajectory with each robot’s fixed object-relative grasp transform: $N$7 This preserves the closed-chain relation implicitly, since each end-effector follows a pose trajectory rigidly tied to the same object motion (Song et al., 5 Aug 2025).

Tracking is then handled by NEO, a reactive optimization-based controller used independently for each robot. NEO produces joint motions that track the desired end-effector trajectory while respecting joint limits and performing collision avoidance. If the induced robot motions turn out to be infeasible for parts of the object path, those regions are marked collision-prone and the object trajectory is replanned. The result is not a single monolithic optimization over all robots and the object, but an iterative coupling between object-space planning and robot-level feasibility checking (Song et al., 5 Aug 2025).

Notably, the framework emphasizes geometric feasibility rather than explicit force control. There is no active force-distribution layer, torque-sharing controller, or impedance coordination policy. Stability is achieved through collaborative grasp selection, object-level path constraints, and feasible end-effector tracking. This makes the system operationally simpler, but also delimits its robustness under disturbance or model mismatch (Song et al., 5 Aug 2025).

5. Empirical evaluation, prompt ablations, and failure modes

The paper evaluates CollaBot along three axes: local grasp quality, collaborative grasp quality, and end-to-end task success. For local grasping, LoGNet is compared qualitatively against GraspNet. The reported outcome is that GraspNet, trained on small objects, often proposes grasps that collide or ignore the geometry of large objects, whereas LoGNet produces collision-free local grasps on large-object patches (Song et al., 5 Aug 2025).

For collaborative grasping, four VLMs are assessed: Qwen-VL-MAX, DeepSeek-R1, GPT 4.1, and Gemini 2.5 Pro. The evaluation includes Grounding Success (GS), which measures whether the model correctly grounds the numbered grasp points in the image, along with the static force metrics $N$8, minimum singular value, and $N$9. Qwen-VL-MAX and DeepSeek-R1 frequently fail GS, while GPT 4.1 and Gemini 2.5 Pro achieve GS and produce successful collaborative grasp selections across translation and rotation tasks involving tables and chairs (Song et al., 5 Aug 2025).

The paper also reports that increasing robot count can reduce per-contact force burden. In the comparison between two-robot and three-robot table manipulation, $I_u(\mathcal{T})$0 drops substantially for the three-robot case, consistent with the intuition that additional robots distribute the load and reduce peak force requirements on each contact (Song et al., 5 Aug 2025).

Prompt ablations examine three prompt components: explicit evaluation metrics, chain-of-thought reasoning, and example outputs. Removing all three yields poor collaborative grasps and low success. Adding only metrics improves performance somewhat. The best results are obtained when all three are present, with the largest gains attributed to the combination of evaluation criteria and chain-of-thought grounding; examples further stabilize the output format and reasoning style (Song et al., 5 Aug 2025).

End-to-end evaluation spans five tasks over five views each: TABLE, CHAIR, Flip TABLE, Align, and TABLE-3. Success counts are 3/5, 3/5, 1/5, 4/5, and 2/5 respectively, for an overall success rate of 13/25 = 52% (Song et al., 5 Aug 2025). The paper notes that ordinary two-robot translation or horizontal-rotation tasks achieve about 66.7% success, while vertical flipping is much harder. Reported failure attribution is approximately 50% due to collaborative grasp issues, 33.3% due to motion-planning failures, and 16.7% due to perception failures such as SEEM failing to recover the object point cloud (Song et al., 5 Aug 2025).

A representative timing breakdown for the Align task reports 25.9 s for VLM inference, 96.3 s for local grasp pose generation, 21.1 s for collaborative grasp selection, 6.6 s for stage-1 motion planning, and 28.0 s for stage-2 planning (Song et al., 5 Aug 2025). This places the current implementation firmly in the offline-planning regime.

6. Position within collaborative robotics research

CollaBot occupies a distinct position within collaborative robotics. Earlier cobot systems in Industry 4.0 settings emphasized natural-language interaction, enterprise integration, and computer vision for pick-and-place or quality-control tasks, but did not address simultaneous multi-robot closed-chain manipulation of large objects (Pazienza et al., 2024). Similarly, CoSTAR used behavior trees, perception, and symbolic predicates to let non-experts author robust collaborative-robot tasks, but its focus was end-user task specification rather than vision-language-guided SCM with arbitrary robot counts (Paxton et al., 2016).

The significance of CollaBot lies in its attempt to fuse three lines of work that had largely remained separate: foundation-model-based semantic reasoning, geometric grasp and motion planning, and simultaneous multi-robot manipulation. Its VLM component does not merely identify objects; it is tasked with inferring robot count, motion constraints, and globally coordinated grasp structure. A plausible implication is that foundation models can function as heuristic priors over difficult combinatorial choices in multi-robot manipulation, while model-based planners retain responsibility for kinematic and collision feasibility (Song et al., 5 Aug 2025).

The framework remains limited by VLM grounding errors, SEEM failures, the absence of explicit force control, and the decoupled nature of its planner. The paper explicitly points to replanning and improved grasp adjustment as future directions (Song et al., 5 Aug 2025). Even so, the reported results establish CollaBot as an early generalist architecture for large-object simultaneous collaborative manipulation, and as a concrete reference point for later systems that may combine better perception, stronger global grasp optimization, and more integrated closed-chain planning.