Online Annotation Platform

• The online annotation platform is a web-based system that enables collaborative, scalable annotation of digital artifacts such as text, images, and videos with quality controls.
• It integrates multi-tiered architectures, containerization, and machine-in-the-loop automation to streamline tasks and boost annotation efficiency.
• The platform supports diverse modalities across domains like NLP, computer vision, and biomedicine while ensuring traceable versioning and standardized data exports.

An online annotation platform is a web-based system facilitating structured, collaborative, and often multimodal annotation of digital artifacts—such as text, images, documents, or videos—at scale and with quality controls appropriate for data-centric machine learning or domain-specific analyses. These platforms integrate user-facing annotation interfaces, project management, workflow coordination, data export, and increasingly, machine-in-the-loop automation and real-time collaboration. The rapid proliferation of these platforms has been driven by the demands of large curated datasets in natural language processing, computer vision, document analysis, and biomedicine, as well as by requirements for transparent provenance and reproducibility in data-centric AI.

1. System Architectures and Deployment Patterns

Online annotation platforms exhibit distinct architectural patterns determined by modality, scalability, and integration requirements:

• Three-tiered microservices are prevalent for high-throughput and collaborative annotation. For example, Callico (Kermorvant et al., 2024) employs a Python/Django backend orchestrating user, campaign, and task logic, plus Celery/Redis for asynchronous task queuing, PostgreSQL for metadata, and MinIO/S3 for storage. Client-side rendering uses a single-page Vue.js app that communicates via REST APIs.
• Containerization and modularity are baseline features for scalable, reliable deployment. Docker Compose and Kubernetes enable horizontal scaling—especially for parallel background tasks like export or inference. MOSaiC (Mazellier et al., 2023) demonstrates a fully containerized, cloud-native deployment tailored for medical video annotation, supporting per-component scaling of API servers, structure tracking, and workers.
• File-based, browser-only stacks (e.g., EEVEE (Sorensen et al., 2024), TAG (Forbes et al., 2017), BRIMA (Lahtinen et al., 2021)) achieve maximum simplicity and zero setup by running client-side only, often as static assets on any server, sometimes as browser extensions or single-page applications without a backend.
• On-device, privacy-preserving deployments (e.g., DOCMASTER (Nguyen et al., 2024)) are implemented using microservices (React+PDF.js frontend, FastAPI/Python backend, local SQL DB), with all data processing—including annotation, training, and inference—restricted to local containers for regulatory or privacy reasons.

2. Annotation Modalities and Task Types

Modern platforms support diverse annotation types aligned with application domains:

Platform	Supported Modalities	Notable Features
Callico	Document image: transcription, layout, NER, key-value	Dual-display, model-in-loop, hierarchical struct
DocSpiral	Layout, OCR, semantic tables/formulas/figures	Human-in-the-spiral, model retrain, metric dash
MOSaiC	Video: spatio-temporal, keyframes, forms	Real-time collab, ontologies, multi-group modes
HistoColAi	Digital pathology images	Deep zoom, crowdsourcing, AI-first annotation
EEVEE	NLP: sequence, spans, text class, seq2seq	TSV-based, browser-only, multi-tasking
TAG	Complex text relations (e.g., graphs, events)	Semantic hypergraphs, recursive relations
VisioFirm	Image, vision: detection, segmentation, OBB	AI-assisted, CLIP, SAM2, WebGPU, offline
TASSY, TeamTat	NLP, survey/text mix	Survey integration, span annotation, IAA metrics

Task types range from basic classification and labeling of tokens, regions, or spans to hierarchical structure building, event graph construction, and argument graph modeling (SenTag (Loreggia et al., 2021), TAG (Forbes et al., 2017), Textarium (Proff et al., 16 Sep 2025)). Machine-in-the-loop is increasingly standard, enabling pre-annotation and progressive model refinement (Callico, DocSpiral, VisioFirm).

3. Collaboration, Quality Control, and Workflow Coordination

Annotation platforms encode complex team workflows and manage quality through hierarchical roles, conflict resolution, and agreement statistics:

• Roles and permissions: Typical roles include contributor/annotator, moderator/reviewer, and manager/curator. Role-based dashboards and invitation controls (Callico) enable both open volunteer campaigns and tightly restricted projects (Kermorvant et al., 2024).
• Versioning and audit trails: Full annotation version control underpins traceability and adjudication (Callico, EXACT (Marzahl et al., 2020)), with storage of all diffs and audit logs for governance.
• Blind and consensus annotation: Many platforms support independent (blind) annotation rounds followed by reconciliation; e.g., TeamTat (Islamaj et al., 2020) and MOSaiC allow assignment and conflict resolution workflows. Inter-annotator agreement (IAA) is quantified via κ, α, or F₁ statistics; quality dashboards highlight bottlenecks and discrepancies.
• Real-time collaboration: MOSaiC uses WebSockets for live update of spatio-temporal annotations, while other platforms offer synchronous or batched communication for collaborative projects.
• Automated consensus and review: Advanced tools implement algorithmic consensus, such as the candidate-cluster mapping in COREFI (Bornstein et al., 2020), or dual-agent (Annotator/Reviewer) reflective pipelines in LinguistAgent (Li, 5 Feb 2026), where reviewer intervention measurably boosts F₁.

4. Automation, Machine-in-the-Loop, and Extensibility

Modern annotation systems increasingly integrate automation and extensibility:

• Model-in-the-loop annotation: Pre-annotation suggestions from OCR, HTR, or deep-learning detectors decrease manual effort—e.g., Callico’s pre-fill halves annotation time (R ≈ 2.3) (Kermorvant et al., 2024). DocSpiral quantifies reduction in annotation time per page (≥41%) per spiral feedback loop (Sun et al., 6 May 2025).
• Pipeline extensibility: Plugin frameworks underpin algorithmic extensibility (EXACT plugin API, Callico’s Python subclass interface, TAG import templates). RESTful APIs and model hooks facilitate on-the-fly inference (Callico, VisioFirm, LinguistAgent (Li, 5 Feb 2026)).
• Active learning and hybrid pipelines: VisioFirm fuses YOLO, Grounding DINO, and CLIP in a two-stage pipeline (low-threshold recall, CLIP verification, graph-based clustering), reducing manual labeling up to 90% (Ghazouali et al., 4 Sep 2025).
• Evaluation dashboards: Real-time metrics (annotation throughput, latency, CER, F₁, satisfaction) are trended per campaign or model iteration. DocSpiral, LinguistAgent, and MOSaiC implement central dashboards with worker progress, error statistics, and time reduction ratios.

5. Data Export, Standards, and Integration

Supporting downstream workflows, annotation platforms implement extensive export and integration facilities:

• Standardized exports: CSV, XLSX, JSON (custom schemas), COCO, BioC, Arkindex, and domain-specific formats are supported (Callico, VisioFirm, TeamTat, EXACT, etc.).
• Data provenance and schema enforcement: Schema-driven annotation tightly couples project creation with data model (SenTag via user-uploaded XSD, Notitia/Sansu-Wolke via Linked Data vocabularies) (Nithya et al., 2014, Loreggia et al., 2021).
• API integration: REST endpoints enable submission, review, prediction, model training, and bulk export (Callico, DocSpiral, VisioFirm, LinguistAgent).
• CMS and knowledge-graph linkage: Platforms such as semantify.it (Kärle et al., 2017) automate the injection of semantic annotations into websites via plugins, aligning with schema.org or LOD vocabularies, and supporting analytics for usage and consistency validation.

6. Performance, Evaluation, and Comparative Results

Systematic evaluation guides optimization and research adoption:

• Throughput and efficiency metrics: Callico defines annotation throughput ($\lambda = N/T$) and time reduction ratio ($R = T_{\text{baseline}}/T_{\text{Callico}}$) (Kermorvant et al., 2024). DocSpiral tracks mAP/F₁ over spiral cycles with quantified annotation time savings (Sun et al., 6 May 2025).
• Domain-specific deployment: MOSaiC, HistoColAi, and EXACT document successful use in large-scale clinical, biomedical, and histopathological annotation (cases of 3,000+ hours, up to 700,000 pages, or 100,000+ vector annotations).
• Extensibility and scaling: Platforms like DOCMASTER highlight seamless scale-out, on-device model training/inference for regulatory environments, and up to 7× throughput improvement for QA on complex forms (Nguyen et al., 2024).
• Comparative assessment: Feature matrices and empirical case studies unambiguously quantify benefits of integrated, modular workflow (e.g., DocSpiral versus Label Studio/PAWLS/COCO Annotator for modality integration; VisioFirm’s manual effort reduction versus prior tools), supporting adoption decisions.

7. Trends and Research Directions

Contemporary research and development in online annotation platforms emphasize:

• Unified, multi-modal annotation (Callico: integrated transcription, layout, NER, key-value, grouping (Kermorvant et al., 2024); DocSpiral: simultaneous layout, OCR, figure/formula/table understanding (Sun et al., 6 May 2025)).
• Automation and AI-augmented workflows (pre-annotation, dual-agent review, human-in-the-spiral iterative model retraining).
• Privacy-aware, on-premise deployment for regulated domains (DOCMASTER (Nguyen et al., 2024)).
• Extensibility via plugin APIs and configuration-driven schema, empowering rapid adaptation to new domains (EXACT, Callico, TAG).
• Data-centric AI methodologies, with an emphasis on annotation quality, inter-annotator agreement, and workflow analytics.

Ongoing developments aim to extend inter-annotator agreement analysis, ML integration, prediction-based triage, active learning, and modularization for new modalities and project scales (Kermorvant et al., 2024, Sun et al., 6 May 2025, Li, 5 Feb 2026).

---

References

• "Callico: a Versatile Open-Source Document Image Annotation Platform" (Kermorvant et al., 2024)
• "DocSpiral: A Platform for Integrated Assistive Document Annotation through Human-in-the-Spiral" (Sun et al., 6 May 2025)
• "VisioFirm: Cross-Platform AI-assisted Annotation Tool for Computer Vision" (Ghazouali et al., 4 Sep 2025)
• "MOSaiC: a Web-based Platform for Collaborative Medical Video Assessment and Annotation" (Mazellier et al., 2023)
• "EXACT: A collaboration toolset for algorithm-aided annotation of images with annotation version control" (Marzahl et al., 2020)
• "TeamTat: a collaborative text annotation tool" (Islamaj et al., 2020)
• "LinguistAgent: A Reflective Multi-Model Platform for Automated Linguistic Annotation" (Li, 5 Feb 2026)
• "Text Annotation Graphs: Annotating Complex Natural Language Phenomena" (Forbes et al., 2017)
• "DOCMASTER: A Unified Platform for Annotation, Training, & Inference in Document Question-Answering" (Nguyen et al., 2024)
• "Tag: a Web-based Tool for Semantic Annotation of Textual Documents" (Loreggia et al., 2021)
• "TASSY -- A Text Annotation Survey System" (Spinde et al., 2021)
• "Semantic Annotation and Search for Educational Resources Supporting Distance Learning" (Nithya et al., 2014)
• "semantify.it, a Platform for Creation, Publication and Distribution of Semantic Annotations" (Kärle et al., 2017)
• "PAGAN: Video Affect Annotation Made Easy" (Melhart et al., 2019)
• "BRIMA: low-overhead BRowser-only IMage Annotation tool" (Lahtinen et al., 2021)
• "IndiTag: An Online Media Bias Analysis and Annotation System Using Fine-Grained Bias Indicators" (Lin et al., 2024)
• "Textarium: Entangling Annotation, Abstraction and Argument" (Proff et al., 16 Sep 2025)