CHAMP: Cluster Hierarchy Annotation Tool

• CHAMP is an annotation tool that incrementally builds a directed forest of text clusters, ensuring transitivity at both clustering and hierarchy levels.
• It integrates cluster formation and hierarchy construction in a single workflow, allowing for seamless reassignment and automatic hierarchy updates.
• CHAMP supports multi-annotator adjudication via a two-stage consolidation process, improving annotation quality and reducing the burden of pairwise decisions.

CHAMP, short for Cluster Hierarchy Annotation for Multiple Participants, is an open-source annotation tool for constructing and adjudicating a specific global NLP structure: a hierarchy whose nodes are themselves clusters of text items. It is designed for tasks in which annotators must simultaneously group semantically equivalent items into disjoint clusters and organize those clusters into a directed hierarchy, with each cluster node having at most one parent. The resulting structure is a directed forest over clusters, and CHAMP’s central contribution is an incremental workflow that constructs the cluster partition and the hierarchy in lockstep while preserving transitivity at both levels (Cattan et al., 2023).

1. Formal object and representational scope

CHAMP formalizes annotation over an input set of spans or statements

$S=\{s_1,\ldots,s_n\}.$

Annotators must partition these inputs into disjoint clusters of semantically equivalent spans

$C=\{C_1,\ldots,C_k\},$

and construct a directed forest

$G=(C,E),$

where each node is a cluster $C_i$, each node has no more than one parent, and an edge $e_{ij}$ represents a hierarchical relation

$C_i > C_j,$

meaning that $C_i$ is a child of $C_j$. The intended semantics is specificity/generalization: child clusters are more specific than their parents (Cattan et al., 2023).

This representation is narrower than an arbitrary graph annotation formalism. CHAMP does not target unrestricted relation graphs; it targets a hierarchy of clusters with a single-parent constraint. That restriction yields a directed forest, i.e., a DAG in which every node has in-degree at most 1. The restriction is consequential because many target tasks require global consistency conditions rather than isolated local labels. If two items are equivalent to the same third item, clustering must respect equivalence transitivity. If one cluster lies under another, and that one lies under a third, the hierarchy must respect transitivity as well. CHAMP encodes those constraints directly in the annotation object rather than repairing them after pairwise judgments (Cattan et al., 2023).

The paper explicitly contrasts this structure with both flat clustering and pairwise edge annotation. A flat cluster inventory is too coarse for tasks that also require broader/narrower relations among clusters, while isolated pairwise judgments are too unstructured to guarantee global consistency. A plausible implication is that CHAMP is best understood as a tool for annotating globally constrained relational objects rather than a generic labeling interface.

2. Target NLP tasks and demonstrated use cases

CHAMP is motivated by NLP tasks whose outputs are naturally hierarchies over clusters rather than flat labels. The paper names entailment graph construction, hierarchical cross-document coreference resolution, and event/subevent annotation as motivating examples. In each of these, the annotation target combines identity-like equivalence relations with broader/narrower structural relations, and both must remain globally coherent (Cattan et al., 2023).

Two concrete use cases are emphasized.

Dataset	Input type	Annotation target
SciCo	Paragraphs from computer science papers with highlighted mentions	Hierarchical cross-document coreference resolution
THINKP	Concise opinion statements called key points	Key point hierarchies for opinion summarization

In SciCo, the inputs are paragraphs from computer science papers with highlighted mentions of scientific concepts, especially tasks and methods. Annotators cluster mentions that refer to the same concept and then infer a referential hierarchy over clusters. The paper gives “categorical image generation” $>>$ “class-conditional image synthesis” as an equivalence example, and then “categorical image generation” $\rightarrow$ “image synthesis” as a hierarchy example. In THINKP, nodes group statements expressing the same opinion, such as “the cleaning crew is great!” $C=\{C_1,\ldots,C_k\},$0 “housekeeping is fantastic,” while edges encode specification/generalization, as in “housekeeping is fantastic” $C=\{C_1,\ldots,C_k\},$1 “the personnel is great” (Cattan et al., 2023).

The paper further notes that CHAMP was used with “minimal effort” for both SciCo and THINKP through a simple JSON configuration scheme, despite the fact that one task used standalone statements and the other used concept spans in context. This suggests that the decisive requirement for CHAMP is not domain uniformity but structural uniformity: the output must be a hierarchy of clusters.

3. Incremental joint construction of clusters and hierarchy

CHAMP’s core methodological contribution is its incremental annotation procedure. Rather than asking annotators to judge all pairs, or first cluster all items and only afterward build a hierarchy over fixed clusters, CHAMP presents spans one by one and lets the annotator jointly update both structures. At initialization, the first span is automatically assigned to the first cluster $C=\{C_1,\ldots,C_k\},$2 and to a corresponding hierarchy node. For each subsequent span $C=\{C_1,\ldots,C_k\},$3, the annotator first decides its cluster assignment: either assign it to an existing cluster or create a new one. If a new cluster is created, a new hierarchy node is automatically inserted under the root; the annotator can then drag it to its proper place in the current hierarchy (Cattan et al., 2023).

This design makes cluster formation and hierarchy construction proceed in lockstep. If the current span belongs to an existing cluster, it augments that node. If it starts a new cluster, it immediately appears in the hierarchy and can be positioned relative to other nodes. The paper explicitly contrasts this with a naive two-stage workflow—first clustering all items, then building a hierarchy over the fixed clusters—and argues that such separation is inflexible and cognitively awkward because many real decisions lie at the boundary between equivalence and hierarchical relatedness (Cattan et al., 2023).

The efficiency claim is central. Pairwise annotation scales quadratically, so for 50 items all possible pairs require

$C=\{C_1,\ldots,C_k\},$4

binary decisions, before hierarchy inference or consistency checking. By contrast, CHAMP’s incremental method asks the annotator to place each new span into an evolving clustered hierarchy. The paper reports that annotating or consolidating a hierarchy of clusters for 50 statements takes approximately one hour in its experiments, whereas collecting all pairwise annotations “would obviously take much more than one hour” (Cattan et al., 2023).

The consistency claim is equally central. Because every action directly edits the current cluster partition and the current hierarchy, CHAMP preserves transitivity by construction rather than by post hoc closure. The paper states that, unlike the pairwise annotation approach, the incremental method “guarantees that the resulting annotation will respect the global constraint of transitivity” at the cluster and hierarchy levels.

4. Reassignment logic and automatic hierarchy updates

A defining feature of CHAMP is that it supports revision of earlier assignments without forcing manual repair of the hierarchy. The paper emphasizes that reassignment is essential because annotators frequently revise earlier decisions, and those revisions may affect both the cluster partition and the hierarchy (Cattan et al., 2023).

CHAMP therefore defines a hierarchy update procedure for re-assigning a previously assigned span $C=\{C_1,\ldots,C_k\},$5, with three cases:

1. From a singleton cluster $C=\{C_1,\ldots,C_k\},$6 to a cluster $C=\{C_1,\ldots,C_k\},$7: $C=\{C_1,\ldots,C_k\},$8 is added to $C=\{C_1,\ldots,C_k\},$9, and $G=(C,E),$0’s children move under $G=(C,E),$1.
2. From a non-singleton cluster $G=(C,E),$2 to a cluster $G=(C,E),$3: $G=(C,E),$4 is added to $G=(C,E),$5, but $G=(C,E),$6’s children stay under $G=(C,E),$7.
3. From a cluster $G=(C,E),$8 to a new singleton cluster: a new node $G=(C,E),$9 is created in the hierarchy and is initially placed as a sibling of $C_i$0; the annotator can then drag it to the intended location.

The paper adds a footnote clarifying the rationale for the third case: when annotators reassign $C_i$1 to a standalone cluster, the intention is typically not to delete the hierarchical relation between $C_i$2 and its parent cluster, so the new singleton begins as a sibling rather than in an unrelated position (Cattan et al., 2023).

This automatic hierarchy update procedure is what makes the “hierarchy of clusters” directly annotatable as one unified task. Without it, any cluster revision would require cumbersome manual restructuring. A common misconception is to treat clustering and hierarchy as separable annotation layers; CHAMP is built around the opposite premise, namely that they are operationally entangled and must therefore be revised together.

5. Consolidation and adjudication across multiple annotations

CHAMP’s second major contribution is a consolidation mode for adjudicating multiple hierarchy-of-cluster annotations. The paper argues that globally constrained structures benefit especially from adjudication because disagreement is harder to resolve than in ordinary classification settings. Standard aggregation methods such as majority vote were developed mainly for independent labels, and COREFI’s reviewing mode covered only single clustering annotations rather than multiple full cluster hierarchies (Cattan et al., 2023).

The main technical difficulty is that different annotators may not induce the same node set. If they cluster spans differently, then their hierarchy nodes are different objects, so parent-link comparison is ill-defined. CHAMP addresses this by decoupling adjudication into two consecutive stages: first clustering decisions, then hierarchy decisions.

In the clustering adjudication stage, the adjudicator is shown spans sequentially together with the original annotators’ cluster assignments. CHAMP adopts COREFI’s reviewing logic separately for each original annotation. When the reviewer is shown a mention $C_i$3, the candidate clusters implied by an original annotation are the set of clusters in the current reviewer configuration that include at least one of the previously annotated antecedents of $C_i$4 according to the original annotation. CHAMP presents, for each source annotation, a set of candidate clusters; in the interface these candidates are displayed in purple at the bottom. If annotators disagree, the adjudicator chooses the correct assignment (Cattan et al., 2023).

A crucial detail is that changing a clustering decision for one annotator requires updating that annotator’s hierarchy as well. CHAMP therefore applies the same hierarchy update procedure used during annotation. This synchronizes the modified annotations so that, after clustering adjudication, they all contain exactly the same node set $C_i$5. Only then does the hierarchy adjudication stage become well-defined. In that second stage, a hierarchy disagreement is simply a case where a node $C_i$6 has a different direct parent in different annotations. CHAMP surfaces such cases with a dedicated procedure: the adjudicator can click “Go To Next Disagreement,” the disputed node $C_i$7 is highlighted in blue and its direct parent in violet in all compared hierarchies, and the adjudicator decides which parent relation is correct and manually updates the other hierarchies accordingly (Cattan et al., 2023).

The paper does not provide pseudocode, but it describes an effective adjudication algorithm: normalize clustering across annotations via sequential review; propagate hierarchy updates after each clustering change; once all annotations induce the same node set $C_i$8, iterate over nodes whose direct parents differ; surface each disagreement with synchronized highlighting; choose the correct parent assignment; and submit the final consolidated hierarchy once disagreements are exhausted. This design turns the otherwise difficult problem of comparing non-isomorphic hierarchical clusterings into two manageable subtasks.

6. Interface, implementation, empirical profile, and limits

The user interface is organized around the incremental workflow. Figure 1 in the paper shows the current item to annotate highlighted, a hierarchy view in the lower portion of the window, and a cluster bank in the footer. Existing clusters appear as hierarchy nodes, and when an item is assigned to a cluster it is concatenated into the display text for that node, separated by semicolons. The interface includes controls such as RESET and SORT, and a special reject bucket for unusable items: “No clear stance, Multiple ideas, Wrong polarity, Unclear or not self-contained.” The interface also supports quick keyboard-centric interaction inherited from COREFI and drag-and-drop placement for newly created hierarchy nodes. An alternate interface mode displays spans within context, as used for SciCo (Cattan et al., 2023).

On implementation, CHAMP is built on top of COREFI and implemented in Vue.js. It is released as open source under the MIT License, with the repository at https://github.com/ariecattan/champ. The tool is released as a WebComponent, so it can be embedded in arbitrary HTML pages, including crowdsourcing platforms such as Amazon Mechanical Turk. Both annotation and adjudication use the same general interface and are configured through a simple JSON schema. The authors also provide an annotation portal where users can upload configuration files, perform annotation online, and download the resulting files afterward. The portal also supports uploading multiple annotation files and computing inter-annotator agreement (Cattan et al., 2023).

The empirical evidence reported in the system paper is practical rather than exhaustive. The main quantitative findings are that annotating or consolidating a hierarchy of clusters for 50 statements takes about one hour, that a pairwise approach for 50 items would require at least 1225 pair decisions on average, and that the consolidation mode improves annotation quality by 5–6 F1 points, citing results from Cattan et al. (2023). At the same time, the paper does not provide a fuller table of metrics, a detailed error analysis, or a controlled usability study with statistical significance tests. It also does not spell out exact JSON field names or a full serialization schema. These omissions define the paper’s scope: CHAMP is introduced primarily as a reusable annotation environment and workflow for globally constrained cluster hierarchies, not as a formal evaluation study of annotation ergonomics (Cattan et al., 2023).

A final conceptual clarification follows from these limits. CHAMP is specialized rather than universal: it is intended for tasks “over any type of texts” only insofar as the output object is a hierarchy of clusters. Within that scope, its contribution is to make the global structure directly editable and adjudicable, replacing large numbers of independent local decisions with direct manipulation of the evolving global object.