A ROS 2 Wrapper for Florence-2: Multi-Mode Local Vision-Language Inference for Robotic Systems

Abstract: Foundation vision-LLMs are becoming increasingly relevant to robotics because they can provide richer semantic perception than narrow task-specific pipelines. However, their practical adoption in robot software stacks still depends on reproducible middleware integrations rather than on model quality alone. Florence-2 is especially attractive in this regard because it unifies captioning, optical character recognition, open-vocabulary detection, grounding and related vision-language tasks within a comparatively manageable model size. This article presents a ROS 2 wrapper for Florence-2 that exposes the model through three complementary interaction modes: continuous topic-driven processing, synchronous service calls and asynchronous actions. The wrapper is designed for local execution and supports both native installation and Docker container deployment. It also combines generic JSON outputs with standard ROS 2 message bindings for detection-oriented tasks. A functional validation is reported together with a throughput study on several GPUs, showing that local deployment is feasible with consumer grade hardware. The repository is publicly available here: https://github.com/JEDominguezVidal/florence2_ros2_wrapper

• The paper presents a ROS2 wrapper that integrates the Florence-2 vision-language model with ROS2 topics, services, and actions for flexible robotic perception.
• It details a multi-mode inference architecture—including continuous, service, and action modes—with validated performance across varied GPU hardware.
• The study discusses practical deployment, reproducibility challenges, and future extensions for enhanced message typification and inference optimization.

A ROS 2 Wrapper for Florence-2: Enabling Unified Vision-Language Inference in Robotic Systems

Introduction

The integration of foundation vision-LLMs (VLMs) into robotic systems has significantly enhanced semantic perception capabilities, enabling more flexible and compositional scene understanding. Despite rapid advances in VLM architectures, translation of such models into reproducible and idiomatic robot middleware remains an unsolved bottleneck. This paper, "A ROS 2 Wrapper for Florence-2: Multi-Mode Local Vision-Language Inference for Robotic Systems" (2604.01179), presents a ROS 2 software component that exposes the multi-capability Florence-2 VLM through interfaces standard in robotics: topics, services, and actions. The wrapper emphasizes local execution, modularity, and practical deployment, enabling new application paradigms in collaborative and embodied robotics.

Related Work and Context

A convergence is observed across three research threads: the proliferation of unified foundation models for vision (e.g., Florence, Florence-2, CLIP, Grounding DINO), the instantiation of these models for embodied intelligence (PaLM-E, RT-2, OpenVLA, VoxPoser), and the advancement of model wrappers and middleware bridges within the ROS/ROS 2 ecosystem. While frameworks for ASR (Whisper), segmentation (SAM), and generic LLM integration are increasingly common, Florence-2 lacked a robust, publicly available ROS 2 wrapper prior to this work. This paper fills that gap, yielding a modular component for multi-task, prompt-based inference on local robots, addressing both practical integration and middleware constraints.

System Architecture and Interfaces

The wrapper's architecture encapsulates model management, image I/O, prompt handling, inference execution, and diverse output formats within a single ROS 2 node. Its design exposes three interaction modes:

• Continuous mode: Automatic semantic processing of all incoming frames, optimal for real-time streaming perception.
• Service mode: Synchronous, on-demand inference with immediate response.
• Action mode: Asynchronous execution suitable for long-running tasks, supporting intermediate feedback and cancellation.

This multi-modal interface enables flexible integration into heterogeneous robotic architectures without imposing an interaction semantic on upstream modules.

Figure 1: Overview of the Florence-2 ROS 2 wrapper architecture, illustrating how the node interfaces with ROS 2 topics, services, and actions to manage image ingestion, inference, and result publication.

A hybrid message design is adopted. Generic, structured JSON outputs support arbitrary Florence-2 tasks, while object-detection outputs exploit the ROS-native vision_msgs/Detection2DArray type for downstream composability. The architecture explicitly preserves prompt-based task abstraction, allowing task selection at runtime and maximizing component reuse.

Implementation, Deployment, and Reproducibility

Implementation leverages Python, rclpy, PyTorch, and Hugging Face transformers, with image handling via cv_bridge. The repository is organized into interface definitions (services and actions) and the executable node, promoting decoupling and downstream extension. Support for both native (virtualenv-based) and containerized (Docker) deployment is emphasized, including CUDA-optimized variants for self-contained GPU execution. This addresses dependency management—a crucial pain point in deploying VLMs in ROS-based systems. Example clients for both service and action invocation are provided, clarifying the interaction contract and reducing onboarding friction.

Experimental Validation

Functional Confirmation Across Modes

The wrapper is validated for all three interaction paradigms. In continuous mode, images are processed and results published with minimal latency. Service mode correctly handles synchronous requests, sourcing either a provided or the most recent image. Action mode supports progress feedback and eventual result delivery. Action cancellation is implemented at the server level but is limited by blocking behaviour in model inference, which should be accounted for in latency-sensitive applications.

Throughput and Hardware Profiling

Empirical throughput evaluation focuses on end-to-end wrapper performance under continuous operation. Across a GPU range (GTX 1060 Mobile, RTX 3060 Mobile, RTX 3080 Ti Desktop), base model inference rates span from approximately 5.8 FPS (GTX 1060) up to 26 FPS (RTX 3080 Ti). The large Florence-2 model achieves roughly half the throughput on equivalent hardware. Empirically, the RTX 3060 Mobile achieves $\sim$10 FPS for object detection, providing actionable guidance for typical robotic hardware selection.

Limitations and Future Developments

Several practical constraints are identified:

• Only detection-style outputs have strongly typed ROS 2 messages; other vision-language tasks revert to generic JSON.
• Action cancellation granularity is bounded by the underlying model inference mechanism.
• Integration validation is prioritized over task-by-task precision/recall evaluation.
• Throughput is inherently hardware dependent and not guaranteed on resource-constrained systems.

Future effort will focus on extending typed message support beyond detection to additional VLM tasks (e.g., OCR, dense captioning), optimizing inference via quantization and alternate back-ends, and incorporating the wrapper into closed-loop robot pipelines encompassing navigation and HRI modules.

Conclusion

This work delivers a modular, production-grade FLORENCE-2 ROS 2 wrapper positioned for rapid adoption across advanced robotics. By supporting multi-mode interaction and unifying multiple VLM task archetypes in a single, prompt-driven node, it significantly lowers the integration barrier for rich semantic perception on ROS 2. This architecture is poised to facilitate more generalizable, deployable, and composable multimodal intelligence in practical collaborative, mobile, and assistive robots, with immediate relevance to the emerging paradigm of foundation model–powered robotics.