Modular Data Annotation Strategy

• Modular data annotation strategy is a systematic method that decomposes annotation pipelines into distinct, interoperable modules with formal interfaces and decision rules.
• The approach enhances scalability, reproducibility, and domain portability through clear module specifications, standardized protocols, and configurable APIs.
• It integrates human, model, and hybrid annotations by leveraging modules for active learning, bias correction, and budget-aware resource allocation.

A modular data annotation strategy is a systematized methodology in which the annotation pipeline is decomposed into distinct, interoperable modules, each responsible for a specific aspect of data selection, label generation, quality control, or adaptation to constraints. By enforcing strict interfaces and decision rules between stages, such frameworks ensure reproducibility, scalability, and domain portability. They are characterized by formal input/output definitions, mathematical decision rules for annotation assignment and post-processing, and explicit support for ambiguity and annotator bias. This concept has been instantiated in various recent works across biomedical imaging, interactive annotation for NLP and vision, hierarchical protocols, active learning, and budget-aware resource allocation (Schmarje et al., 2023, Huang et al., 2024, Wolf et al., 2020, Kadir et al., 2024, Tejero et al., 2023, Huang et al., 2024, Jäger et al., 2019, Lynnette et al., 2020, Ji et al., 16 Oct 2025).

1. Modular Pipeline Architectures

Modular annotation frameworks explicitly divide the workflow into sequential or parallel modules, each with fixed inputs and outputs. Examples include the five-module strategy of Schmarje et al. (Schmarje et al., 2023):

• Definition of task and data partition (“What?”)
• Annotator qualification/training (“Who?”)
• Annotation method selection (“How?”—manual vs. model-guided)
• Annotation process (collection of votes/labels)
• Post-processing (de-biasing labels to obtain soft/hard targets)

Systems such as LOST (Jäger et al., 2019) and HUMAN (Wolf et al., 2020) compose pipelines as acyclic graphs or state machines, whose nodes correspond to functional modules such as datasource management, proposal generation, annotation interfaces, or looped retraining. Each node adheres to a stubbed Python interface or a JSON-defined protocol, and modules are chained or branched according to the underlying annotation protocol.

For active learning, modular architectures like MedDeepCyleAL (Kadir et al., 2024) separate components into microservices—annotation tool, controller, data manager, and active learning backend—with RESTful APIs for inter-module communication. These boundaries provide extensibility, permitting plug-in of new deep models, transformation pipelines, or acquisition functions via configuration files rather than code changes.

2. Module Specification and Decision Rules

Each module is formally defined by:

• Explicit inputs (e.g., image subset $X_u$, model state, candidate annotators)
• Outputs (e.g., qualified annotators, annotation tally $T_x$, post-processed label distributions)
• Internal logic/algorithms and mathematical decision criteria

An example from (Schmarje et al., 2023) is the post-processing module:

• Input: raw vote counts $T_x$, confusion matrix $c$, bias estimate $\delta$
• Algorithm: Class Blending and Bias Correction to compute de-biased label distributions

$$P_{\mathrm{blend}}(L^x) = (1-\alpha)\frac{T_x}{\sum T_x} + \alpha c_{\cdot}$$

• Label confidence intervals and required annotation numbers are analytically derived,

$$P(\hat L^x=k) \pm Z_{0.975}\sqrt{p_k(1-p_k)/A},\ A = 4 Z_{0.975}^2 p_k(1-p_k)/W^2$$

Proposals for guided annotation are adopted if empirical speedup $S > 3$ (ratio of baseline to proposal-accelerated annotation time) or bias is deemed acceptable.

Budget-aware frameworks, as in (Tejero et al., 2023), formalize resource allocation with Gaussian process surrogates to maximize test-set performance $f(n_s,n_c)$ under cost constraints: $$\text{maximize}_{n_s,n_c\in\mathbb{N}}\ f(n_s,n_c)\quad\text{subject to}\ c_s n_s + c_c n_c \leq B$$ The sequential algorithm adaptively sets the split between full and weak (e.g., segmentation vs. classification) labels across rounds to optimize an acquisition function (expected improvement).

3. Integration of Human, Model, and Hybrid Annotations

Modern modular pipelines leverage both human and machine contributions, applying explicit allocation or integration mechanisms:

• Proposal-guided annotation: Models generate suggestions, which are accepted or corrected by humans; a bias-speedup trade-off governs if/when proposal guidance is used (Schmarje et al., 2023).
• Analogical reasoning and error-aware integration (ARAIDA): Final label suggestions are computed as

$F(x) = \lambda(x)f(x) + (1-\lambda(x))g(x)$

where $f(x)$ is a model prediction, $g(x)$ is a KNN-based analogical label, and $\lambda(x)\in[0,1]$ is produced by an error-estimation network (Huang et al., 2024).

• Selective annotation with triage: SANT (Huang et al., 2024) employs error-aware triage to route hard examples to experts and easy examples to the model, optimizing a joint loss over the model, AL, and error-prediction modules. The bi-weight score for each sample at time $t$ is

$$d_t^{\text{bi}}(x) = (d_t^{\text{AL}}(x))^{\eta(t)} \cdot d_t^{\text{EAT}}(x)$$

dynamically shifting the emphasis between AL “informativeness” and predicted error risk as labeling proceeds.

Frameworks such as LOST (Jäger et al., 2019) and Cross-Model (Lynnette et al., 2020) exploit active learning uncertainty, annotation-assistance modules (reference hierarchies, reference images), and quality-controlled handoffs between automatic proposals and manual correction.

4. Workflow Adaptation: Active Learning, Budget-Constraint, and Protocol Extension

Modular annotation strategies are built for adaptation to changing task requirements, new models, or resource limitations:

• Active learning cycles are implemented as explicit control flows, orchestrating model retraining, acquisition, and human labeling (Kadir et al., 2024, Jäger et al., 2019, Lynnette et al., 2020).
• Selection of annotation type (strong/weak, segmentation/classification) is determined per-batch based on estimated gains via GP models (Tejero et al., 2023).
• Dynamic reweighting between human and model annotation, as with SANT's EAT and bi-weight mechanisms, enables cost-quality trade-offs in real time (Huang et al., 2024).
• High-level architecture and protocols are configured declaratively (YAML/JSON), lowering the coding burden and promoting rapid extension to new data modalities or annotation schemas (Kadir et al., 2024, Wolf et al., 2020).
• Interoperable plugin APIs in open-source frameworks provide mechanisms for domain-specific extension, model adaptation, or specialized task integration (Jäger et al., 2019, Lynnette et al., 2020).

5. Empirical Validation and Performance Metrics

Empirical studies consistently demonstrate major efficiency and quality benefits of modular annotation strategies:

• Schmarje et al. validated on 3,761 vertebral images (≈250,000 annotations), finding optimal macro F1 for humans in the 0.62–0.65 range and demonstrating that DC3 + balanced class blending + bias correction minimizes KL-divergence to the human “consensus” for soft label estimation (Schmarje et al., 2023).
• In ARAIDA, the integration of analogical (KNN) and model-based predictions reduced human correction labor by 11.02% across four tasks; gains are especially pronounced for weak base models (Huang et al., 2024).
• Adaptive budget allocation outperforms any fixed annotation scheme, routinely tracking within 1–2% of the optimal split between strong and weak labels across multiple datasets and cost ratios (Tejero et al., 2023).
• SANT outperforms both random triage and strong LLM-based annotation (ChatGPT, CoT) across sentiment, KG, and multi-label tagging tasks, achieving +0.5–4.9 percentage points accuracy/HR@10 improvements for model-annotated data at medium/high budgets. Its modularity supports plug-in of new AL/error modules as needed (Huang et al., 2024).
• LOST’s two-stage active learning pipeline delivered ≈2× speed-up with no measurable loss in annotation precision on Pascal VOC (Jäger et al., 2019).
• MedDeepCyleAL’s extensible microservice structure supported plug-in of new deep architectures and AL strategies, with performance logs enabling per-stage diagnostics (Kadir et al., 2024).

6. Best Practices and Portability Guidelines

Generalizable principles extracted from literature for effective deployment of modular annotation strategies include:

• Define task and annotator modules prior to execution, ensuring explicit thresholds and coverage requirements (Schmarje et al., 2023).
• Enforce rigorous annotator qualification on gold sets, with operational thresholds for accuracy/F1 (Schmarje et al., 2023).
• Modularize each pipeline step—annotation UI, modeling, active learning, data management—to allow independent updates and ensure future extensibility (Kadir et al., 2024, Wolf et al., 2020, Jäger et al., 2019).
• Use uncertainty and error estimation not only for sample acquisition but also to allocate the right annotation method (machine/human) per instance (Huang et al., 2024, Huang et al., 2024).
• Document all process parameters (thresholds, bias estimates, confusion matrices, annotation counts) for reproducibility (Schmarje et al., 2023).
• Configure protocols via centralized YAML/JSON files and expose well-defined APIs for module integration (Kadir et al., 2024, Wolf et al., 2020, Lynnette et al., 2020).
• Portability to new domains/tasks is achieved by redefining data interfaces, module logic, and minimal protocol extension without wholesale architecture change (Schmarje et al., 2023, Wolf et al., 2020, Kadir et al., 2024, Ji et al., 16 Oct 2025, Jäger et al., 2019).

7. Comparative Features and Limitations

Framework	Human-Model Hybrid	Protocol Extensibility	Task/Modality Generality
Schmarje et al.	Consensus & Proposals with Bias Correction	Yes (modular pipeline, explicit module configs)	Image (classification, biomedical); adaptable
ARAIDA	Error-aware analogical fusion	Full (swap modules)	Text, sequence, vision
HUMAN	Pre-labeling, active learning API	State machine, JSON protocols	Text, sequence, image
LOST	Proposal/MIA/SIA/active loop	Plugin API (Python)	Image, video, clustering, custom UIs
MedDeepCyleAL	Prelabeling AL loop	Config-file microservice modules	Image (2D/3D)—customizable
SANT	Model triage + EAT, budget optimization	Any AL/error-modules	NLP, vision, multi-label
Cross-Model	Uncertainty-based agent/human routing	Adapters for models, APIs	Vision multi-model annotation
Full-vs-Weak	Adaptive strong/weak label split	Hyperparameter/algorithm	Segmentation/classification allocation

A plausible implication is that modularity not only accelerates deployment and adaptation, but also facilitates robust empirical analysis of annotation pipelines, as reproducibility and auditability are preserved through standardized module boundaries and config-driven workflows. However, certain frameworks rely on lightweight annotators for efficiency, assuming constant per-instance costs and not accounting for model computation overhead; extension to very large or cost-sensitive models remains an open limitation in some cases (Huang et al., 2024). Extensions to richer “hardness” signals (e.g., OOD detection) or hierarchical/graph-based annotation taxonomies are practical future directions.

• (Schmarje et al., 2023) Annotating Ambiguous Images: General Annotation Strategy for High-Quality Data with Real-World Biomedical Validation
• (Huang et al., 2024) ARAIDA: Analogical Reasoning-Augmented Interactive Data Annotation
• (Wolf et al., 2020) HUMAN: Hierarchical Universal Modular Annotator
• (Kadir et al., 2024) Modular Deep Active Learning Framework for Image Annotation: A Technical Report for the Ophthalmo-AI Project
• (Tejero et al., 2023) Full or Weak annotations? An adaptive strategy for budget-constrained annotation campaigns
• (Huang et al., 2024) Selective Annotation via Data Allocation: These Data Should Be Triaged to Experts for Annotation Rather Than the Model
• (Jäger et al., 2019) LOST: A flexible framework for semi-automatic image annotation
• (Lynnette et al., 2020) Cross-Model Image Annotation Platform with Active Learning
• (Ji et al., 16 Oct 2025) A Generalizable Rhetorical Strategy Annotation Model Using LLM-based Debate Simulation and Labelling