4-Layer Annotation Scheme Overview

• 4-layer annotation scheme is a multi-dimensional framework that segregates linguistic, argumentative, and factual phenomena into distinct, hierarchically dependent layers.
• It employs vertical layering and facet decomposition to isolate semantic, coreference, veracity, and attack dimensions with clear label taxonomies.
• The approach enhances corpus creation and model accuracy through standardized guidelines, reliable metrics, and integration with tools like UIMA and Inception.

A four-layer annotation scheme structures linguistic, argumentative, and factual phenomena into hierarchically or sequentially dependent annotation layers. Each layer isolates a distinct aspect of meaning, reasoning, or referential behavior, facilitating precise downstream modeling and specialized corpus creation. The following entry synthesizes major published schemes in paraphrase annotation (Kanerva et al., 2021), cross-document entity coreference (Vogel, 2023), fake news annotation (Murayama et al., 2022), and argumentative attack modeling (Mim et al., 2022), clarifying layer definitions, practical guidelines, label taxonomies, schema conventions, and reliability assessments.

1. Structural Classification of Four-Layer Annotation Schemes

Four-layer annotation generally refers to either (a) stratified assessment of linguistic items (paraphrase, coreference, veracity, attack type), or (b) sequential passage through increasingly granular or semantically deep analytic categories. In practice, papers adopt one of two organizational logics:

• Vertical Layering: Layers form a stack, with dependencies between them (e.g., entity first detected, then identity chains built, then relational semantics assigned, then aggregated or linked globally).
• Facet Decomposition: Layers target orthogonal aspects (e.g., paraphrase granularity, intent, harm, audience), allowing different interpretations or analytic paths.

Table 1 summarizes canonical interpretations from recent corpora.

Corpus/Task	Layered Aspects	Scheme Reference
Paraphrase annotation (Finnish)	Substitutability, context dependence, topical relation, flags	(Kanerva et al., 2021)
Cross-document coreference	Mention, identity, relational links, global entity mapping	(Vogel, 2023)
Fake news annotation (Japanese)	Veracity, disseminator intent, social harm, audience	(Murayama et al., 2022)
Argumentative attack (LPAttack)	Attack mode, logic pattern, presupposition, value-judgment	(Mim et al., 2022)

These layered frameworks foster analytic reproducibility, fine-grained modeling, and facilitate adjudication and reliability quantification.

2. Label Taxonomies and Formal Criteria by Layer

Each four-layer scheme defines explicit label sets, tests, and rules for each layer. Notable design choices include:

• Turku Paraphrase Corpus (Kanerva et al., 2021):
  • Layer 1: Base semantic relatedness scale (1=unrelated, 2=related-not-paraphrase, 3=contextual, 4=context-independent/"universal", x=skipped cases).
  • Layer 2: Positive paraphrase subcategory flags — subsumption (> or <), style (s), minor deviation (i).
  • Layer 3: Contextual substitution formalized as $\forall C$: meaning$_C$(u₁) = meaning$_C$(u₂) for label 4; existential contexts diverge for label 3.
  • Layer 4: Annotation decision rules — always prefer higher paraphrase grade unless precise flag suffices.
• Diverse Cross-Document Coreference (Vogel, 2023):
  • Layer 1: Mention detection, entity typing (PER, ORG, GRP, GPE, LOC, OBJ).
  • Layer 2: Intra-document identity relations (ID).
  • Layer 3: Six semantic relation types — metonymy (MET), meronymy (MER), class/subclass (CLS), spatio-temporal function (STF), definition (DEC), bridging (BRD), prioritized ID>MET>MER>CLS>DEC>BRD in ambiguous cases.
  • Layer 4: Cross-document cluster linkage and Wikidata URI anchoring.
• Fake News Annotation (Murayama et al., 2022):
  • Layer 1: Veracity levels (True, Half-True, Inaccurate, Misleading, False, Pants-On-Fire, Unknown Evidence, Suspended Judgement).
  • Layer 2: Disseminator intention (definitely/probably knows = disinformation, does not know = misinformation), with mechanism/question subcodes (fabricated, manipulated, trusted, misunderstood).
  • Layer 3: Social harm quantification (real-valued 0…5), type (harmless, confusion, health, prejudice, conspiracy, etc.).
  • Layer 4: Target audience extraction (free-text noun phrases affected).
• LPAttack Argumentation (Mim et al., 2022):
  • Layer 1: Attack mode (nullify, limit, acknowledgement).
  • Layer 2: Logic pattern (stance, causal/promote/suppress, rationale/condition, contradiction relations).
  • Layer 3: Presupposition relations (rationale/condition, contradiction).
  • Layer 4: Value-judgment (comparative “more important/severe/weighty than”).

3. Annotation Guidelines and Decision Rules

Annotation must follow precise operationalization criteria:

• Semantics-first orientation (Turku Paraphrase, LPAttack): Assign highest possible label when ambiguity exists; use flags only for "universal" pairs if differences are semantic trivialities or stylistic.
• Layer dependencies (Cross-Document Coreference): Entity-detection precedes relation annotation; identity chains local before cross-document clustering.
• Evidence hierarchy (Fake News): Copy fact-check rating for veracity; if veracity is True/Half-True, skip subsequent intention/harm/target annotation.
• Slot-filling templates and explicit span selection (LPAttack): Minimal text span for atomic concepts; logic patterns must connect stances and reasons; attack relations between IA and CA.

Figure 1 in (Vogel, 2023) visualizes the incremental annotation passage (mention→identity→relation→global cluster). Decision rules are strictly defined for ambiguous cases: e.g., prefer higher coreference type in entity annotation, or universal substitution for paraphrase.

4. Data-Format Schemas and Tool Integration

Adoption of schema and workflow conventions is critical for scalable annotation.

• UIMA/CAS format, JSON-style schema (Cross-Document Coreference): Explicit begin-end offsets and label features for entities and relations.
• Inception Tool: Layer-structured annotation UI, field drop-downs for entity types and relations, GUI-driven mention linking, project import/export for reproducibility (Vogel, 2023).
• Flag combinations (Paraphrase Corpus): Orthogonal flag stacking allowed (e.g., 4<s i) except for conflicting arrows, which default to contextual paraphrase (label 3).
• Free-text slot extraction (Fake News, Target Audience): Significant in assessing real-world impact.

Table 2 characterizes schema conventions.

Feature Type	Example Layer	Format/Tool
Entity mention	Coreference Layer 1	JSON: { begin, end, type }
Relation link	Coreference Layer 2/3	JSON: { label, target }
Composite flagging	Paraphrase Layer 2	Numeric+symbolic (4<s i)
Harm scoring	Fake News Layer 3	Numeric (0..5), label

5. Inter-Annotator Agreement and Reliability Assessment

Scheme fidelity and reproducibility require assessment of annotation agreement.

• Turku Paraphrase Corpus: No formal kappa values reported; guidelines developed iteratively (Kanerva et al., 2021).
• Cross-Document Coreference: No quantitative IAA; quality control by document-lock adjudication and suggestion review (Vogel, 2023).
• Fake News Scheme: Fleiss’ κ > 0.80 for veracity and disseminator intention; moderate agreement (κ ≈ 0.62) for harm type; calculated via standard formula

$\kappa = \frac{\bar{P} - P_e}{1 - P_e}$

as defined in (Murayama et al., 2022).

• LPAttack: Moderate reliability (Cohen’s κ=0.63 for relations/attributes at markable level, κ=0.49 for full annotation string) (Mim et al., 2022).

Pilot annotation and ongoing joint training are common mechanisms for increasing consensus in ambiguous cases, especially for boundary phenomena (subsumption direction, harm typology, slot spans).

6. Applications, Corpus Statistics, and Empirical Observations

Four-layer annotation enables fine-grained corpus construction for tasks including paraphrase mining, cross-document entity resolution, deception detection, and computational argumentation.

• Turku Paraphrase Corpus: >100,000 Finnish paraphrase pairs, analyzed via context-independence and stylistic/semantic flags (Kanerva et al., 2021).
• Cross-Document Coreference Corpus: Entity clusters mapped onto Wikidata, facilitating media bias analysis and knowledge graph integration (Vogel, 2023).
• Japanese Fake News Dataset: 307 annotated news items, with mean harm scores clustered in 1–3 range, 87% misinformation, 13% disinformation, bot participation <10% (Murayama et al., 2022).
• LPAttack Corpus: Annotates >90% of debate attacks using base-pattern, causal, comparative, and attack-mode relations; moderate IAA ensures feasibility for computational argument mining (Mim et al., 2022).

Layered annotation structure directly supports downstream clustering, prioritization in fact-checking, explainable model construction, and cross-lingual corpus transfer. These schemes yield empirically robust datasets suitable for advanced inference tasks and rich semantic modeling.