VisioFirm: AI-Assisted Image Annotation

• VisioFirm is an open-source, cross-platform web application for AI-assisted image annotation, employing a hybrid human–AI approach for generating and refining annotations.
• It integrates multiple AI models such as YOLOv10, Grounding DINO, CLIP, and SAM2, achieving up to a 90% reduction in manual labeling effort.
• Its robust architecture supports offline operation, multiple export formats, and extensibility, making it ideal for various computer vision workflows.

Searching arXiv for the specified paper and closely related model papers to ensure citations are current. arxiv_search(query="(Ghazouali et al., 4 Sep 2025) VisioFirm", max_results=5) VisioFirm is an open-source, cross-platform web application for AI-assisted image annotation in computer vision. It is designed to streamline image labeling through AI-assisted automation while reducing the human effort required for bounding boxes, oriented bounding box estimation, and instance segmentation. The system adopts a hybrid human–AI approach: it generates high-recall pre-annotations via a cascade of AI models set to a low confidence threshold, verifies and filters these proposals with CLIP embeddings, clusters redundant detections with an IoU-graph algorithm, and presents editable proposals for user refinement through an HTML5-Canvas interface. It also provides on-the-fly segmentation powered by Segment Anything accelerated through WebGPU, supports multiple export formats, and can operate entirely offline after model caching (Ghazouali et al., 4 Sep 2025).

1. Motivation and problem setting

In modern computer vision projects, high-quality ground-truth annotations are indispensable but notoriously labor-intensive. Traditional tooling forces annotators to draw every shape by hand, leading to long turn-around times on large batches of images, subjective or inconsistent label boundaries, high costs when scaling to thousands of images, and difficulty handling complex shapes such as remote-sensing oriented objects or novel categories not covered by standard datasets (Ghazouali et al., 4 Sep 2025).

VisioFirm addresses these issues by adopting a hybrid human–AI approach rather than fully automating annotation or fully relying on manual labeling. In this formulation, the system generates high-recall pre-annotations via a cascade of AI models set to a low confidence threshold, ensuring few true objects are missed; verifies and filters these proposals with CLIP embeddings to correct mis-labels; clusters redundant detections with an IoU-graph algorithm; and presents the user with editable proposals. Annotators can accept, reject, or refine shapes via an interface supporting rectangles, oriented boxes, and polygons. When detectors underperform, click-guided segmentation through the Segment Anything Model is available on demand (Ghazouali et al., 4 Sep 2025).

A central implication of this design is that annotation is treated as proposal generation followed by controlled refinement rather than as fully manual drawing. This suggests a workflow optimized for recall at the proposal stage and precision during user-assisted correction.

2. System architecture and execution model

VisioFirm is implemented as a single-page web application with a Flask/Python backend and an HTML/CSS/JavaScript frontend. The backend manages project metadata, user authentication, and a local SQLite database of images and annotations; hosts REST API endpoints for pre-annotation, filtering, and export; and loads AI models, including detectors, CLIP, Grounding DINO, and SAM2, caching them upon first use. The frontend provides an interactive annotation canvas with pan/zoom, keyboard shortcuts, and multiple drawing modes, together with dashboard visualizations via Plotly.js, including a completion pie chart, a class distribution bar chart, and an annotation-per-image histogram (Ghazouali et al., 4 Sep 2025).

A distinct architectural feature is the coupling of browser-side and backend-side execution. On-the-fly SAM2 segmentation is powered by WebGPU for sub-second mask responses in supporting browsers, while model inference can be executed on CPU or, if available, CUDA/GPU for faster batch processing. Upon first run, VisioFirm downloads and caches all selected AI model weights locally, including Ultralytics YOLOv10, Grounding DINO, OpenAI-CLIP, and SAM2. After caching, it can operate entirely offline (Ghazouali et al., 4 Sep 2025).

This architecture positions VisioFirm as both a local annotation environment and a model-assisted preprocessing system. A plausible implication is that reproducibility and data locality are emphasized by the use of a local SQLite schema and offline-capable cached models.

3. AI-assisted annotation pipeline

VisioFirm’s pre-annotation stack is orchestrated by the VFPreAnnotator as a hybrid cascade. For COCO-aligned classes, Ultralytics YOLOv10x is invoked with a low confidence threshold $\lambda$ such as $\lambda = 0.1$ to maximize recall. For user-defined or domain-specific classes not in COCO, Grounding DINO operates in open-vocabulary mode from text prompts, also with the same low threshold. These models therefore serve complementary roles: one for common classes and one for custom labels (Ghazouali et al., 4 Sep 2025).

For each initial box proposal $b_k$, VisioFirm crops the corresponding image region $I_k$ and applies CLIP for semantic verification. It computes a normalized image embedding $\mathbf{f}(I_k)$ and text embeddings $\mathbf{g}(l_j)$ for each target label $l_j$, then evaluates the similarity score

$s_j = \mathbf{f}(I_k)\cdot \mathbf{g}(l_j).$

These scores are converted to probabilities via a softmax with temperature $\tau$,

$$p_j = \frac{\exp(s_j/\tau)}{\sum_m \exp(s_m/\tau)},$$

and the assigned label is

$\lambda = 0.1$0

In the stated workflow, CLIP is used to confirm or correct assigned classes and refine false positives (Ghazouali et al., 4 Sep 2025).

When either detectors underperform or the user activates “Magic Mode,” a single-click prompt invokes SAM2 in the browser via WebGPU. The resulting mask can be immediately converted to a polygon, rectangle, or oriented box. Polygons from SAM2 are simplified with Ramer–Douglas–Peucker using $\lambda = 0.1$1 perimeter to reduce point count (Ghazouali et al., 4 Sep 2025).

This pipeline does not treat detector outputs as final annotations. Instead, the detector stage is intentionally permissive, and the subsequent CLIP verification and user-assisted editing stages provide the principal mechanisms for disambiguation and correction.

4. Filtering, clustering, and interactive refinement

The annotation workflow proceeds in four explicit stages. First, for the target label set $\lambda = 0.1$2 and low-confidence threshold $\lambda = 0.1$3, all detections $\lambda = 0.1$4 with score $\lambda = 0.1$5 are collected from YOLOv10 and Grounding DINO. Second, each proposal $\lambda = 0.1$6 produces a crop $\lambda = 0.1$7 that is re-scored by CLIP to confirm or correct the assigned class (Ghazouali et al., 4 Sep 2025).

Third, VisioFirm applies IoU-graph connected components clustering. It constructs an undirected graph $\lambda = 0.1$8 in which each node corresponds to a proposal $\lambda = 0.1$9, and an edge $b_k$0 is placed whenever

$b_k$1

with clustering linkage threshold $b_k$2. Connected components $b_k$3 are then identified. If all proposals within a component share the same label, the one with highest confidence is kept. If labels conflict, the system computes the union box

$b_k$4

crops $b_k$5, re-runs CLIP, and assigns a final label (Ghazouali et al., 4 Sep 2025).

Fourth, pre-annotations are overlaid on the canvas for user-assisted refinement. The annotator may accept or reject any box or polygon, drag corners to refine rectangles or oriented boxes, switch to Polygon Mode for pixel-precise segments, or click-invoke SAM2 to generate or refine masks. Through this sequence, VisioFirm reduces human-in-the-loop effort without eliminating the verification role of the annotator (Ghazouali et al., 4 Sep 2025).

A recurrent misconception in annotation systems is that AI assistance implies full automation. VisioFirm is explicitly not framed in that way. Rather than fully automating annotation, which risks missing rare objects, the system prioritizes high-recall proposal generation followed by filtering and human correction.

5. Performance characteristics and benchmark results

The reported benchmarks were performed on a 100-image subset of COCO val, IoU-filtered for common classes. Under this setup, VisioFirm reports up to 90\% reduction in pure manual drawing time versus a manual-only baseline through hybrid pre-annotation and interactive refinement (Ghazouali et al., 4 Sep 2025).

Inference latency is reported for both CPU and GPU execution. For YOLOv10x at $b_k$6, latency is 7.43 s/image on CPU, totaling 12 min 23 s, and 2.53 s/image on GPU, totaling 4 min 13 s, corresponding to a 2.9× speedup. At $b_k$7, YOLOv10x latency is 2.38 s/image on CPU, totaling 3 min 58 s, and 0.14 s/image on GPU, totaling 14 s, corresponding to a 17× speedup. For Grounding DINO (Tiny) at $b_k$8, latency is 29.86 s/image on CPU and 5.28 s/image on GPU, a 5.7× speedup. At $b_k$9, Grounding DINO latency is 9.03 s/image on CPU and 2.18 s/image on GPU, a 4.1× speedup (Ghazouali et al., 4 Sep 2025).

On COCO-aligned classes with default YOLOv10x, pre-annotation recall exceeds 95\%, and post–IoU-graph and CLIP filtering yields a precision near 90\% ([email protected]). For zero-shot detection, Grounding DINO is reported to achieve 52.5 AP zero-shot on COCO out of the box, rising to approximately 63 AP after fine-tuning, which is presented as making it suitable for custom labels (Ghazouali et al., 4 Sep 2025).

These results characterize VisioFirm as a system whose effectiveness depends on the combination of permissive initial detection and structured filtering. The benchmark framing suggests that the reduction in manual effort is tied not only to detector quality but also to the post-processing and interface design.

6. Interoperability, workflow, and extensibility

VisioFirm exports to YOLO TXT/YAML, COCO JSON, Pascal VOC XML, and CSV, with the CSV representation described as one row per shape with coordinates and label. Imported annotations in COCO or YOLO format can be merged into an existing project, with external files parsed into the SQLite schema for unified refinement. The stated bidirectional compatibility is intended to ensure integration with LabelStudio, CVAT, Roboflow, or proprietary pipelines (Ghazouali et al., 4 Sep 2025).

The operational workflow begins with Python 3.8+ and pip, followed by installation via: $I_k$1 The local server is started with: $I_k$2 On first launch, models including YOLOv10, Grounding DINO, CLIP, and SAM2 download to ~/.cache/visiofirm/models/. Runtime flags allow CPU-only or GPU (CUDA) inference. GUI settings specify target classes, low-confidence threshold $I_k$0, and toggles for CLIP verification and SAM2 fallback. After caching, the system continues to operate fully offline. During annotation sessions, users create or import a project, define labels, cycle through images with keyboard shortcuts, accept, refine, or re-generate pre-annotations, use Magic Mode for click-guided masks, monitor progress through the project dashboard, and export packaged annotations for downstream CV model training (Ghazouali et al., 4 Sep 2025).

VisioFirm is described as open-source and released under the MIT license. Its plugin-style design is presented as straightforward to extend, with examples including Detectron2 instance-segmentation backends, video-frame tracking, or multi-modal captioning in future releases. This suggests an architecture intended not only for immediate annotation workloads but also for adaptation to broader labeling scenarios (Ghazouali et al., 4 Sep 2025).