---
title: 'Co-DETECT: Collaborative Edge Case Discovery'
url: https://www.emergentmind.com/topics/co-detect
type: topic
---

# Co-DETECT: Collaborative Edge Case Discovery

Searching arXiv for the Co-DETECT paper and closely related annotation/codebook work to ground the article with current citations.
arxiv_search.query({"search_query":"ti:\"Co-DETECT: Collaborative Discovery of Edge Cases in Text Classification\" OR all:\"Co-DETECT Collaborative Discovery of Edge Cases in Text Classification\"","max_results":5,"sort_by":"submittedDate","sort_order":"descending"})
arxiv_search.query({"search_query":"all:\"Co-DETECT: Collaborative Discovery of Edge Cases in Text Classification\"","max_results":10,"sort_by":"relevance","sort_order":"descending"})
Co-DETECT is a mixed-initiative annotation framework for text classification that integrates human expertise with automatic annotation guided by large language models (LLMs). Introduced as “Collaborative Discovery of Edge cases in TExt ClassificaTion,” it is designed for settings in which a domain expert can provide only a sketch-level codebook and a representative corpus, while many decisive phenomena remain poorly articulated as explicit annotation rules. The framework uses LLM-generated labels, confidence signals, and natural-language explanations to identify challenging instances, induce high-level descriptions of edge cases, and support iterative codebook refinement through compact, generalizable handling rules [2507.05010].

## 1. Problem setting and motivation

Co-DETECT addresses two coupled difficulties in text classification. The first is codebook development: researchers must specify task definitions, label sets, and guidelines for nuanced cases such as borderline hate speech or implicit emotions. The second is large-scale annotation: nuanced, context-heavy tasks require trained annotators and substantial cognitive effort. In such settings, the most consequential failures often arise not from prototypical instances but from edge cases, namely examples that are poorly covered by the current codebook [2507.05010].

The framework is motivated by the observation that ambiguity, subjectivity, and tacit expert knowledge make exhaustive guideline construction intrinsically difficult. The data explicitly situate this under Polanyi’s paradox: experts “know more than they can tell,” so their practical judgment often outstrips what they can initially encode as rules. Traditional workflows identify edge cases manually through pilot annotation and inspection, but this is slow and incomplete. Recent LLM-based annotation methods, by contrast, often assume that a well-developed, stable codebook already exists, offering limited support for co-evolving the codebook with the annotation process [2507.05010].

Edge cases matter because they drive inter-annotator disagreement and lower reliability. They also often carry important substantive content, and models trained without explicit handling rules for such cases may misclassify them, overfit spurious patterns, and become less robust and fair. Co-DETECT therefore formalizes edge-case discovery as a systematic activity rather than an ad hoc one, with the explicit aim of transforming collections of challenging instances into high-level, reusable annotation rules [2507.05010].

## 2. Mixed-initiative architecture and workflow

The framework combines human domain experts, LLM-based annotation, and a visual interface. Human experts provide the task definition, the initial codebook, and the unlabeled corpus; they then inspect, edit, accept, or reject candidate edge-case rules. LLMs provide the first-pass annotations, self-report confidence, explain low-confidence cases, and help abstract specific ambiguities into higher-level edge-case types [2507.05010].

At initialization, the expert supplies three core inputs: a task definition, a classification label set, and a task ID. The initial corpus is uploaded as a CSV file containing **500–1000 target texts**. This scale is described as large enough for representative coverage of edge cases while remaining manageable in cost and latency. The initial codebook is intentionally incomplete: it is expected to contain a draft task definition and labels, not a fully elaborated rule system [2507.05010].

The workflow then proceeds in an iterative loop. A non-reasoning LLM, specifically GPT-4.1, labels each item according to the current codebook and produces a verbal confidence estimate. When confidence is low, it also explains why the example is difficult and proposes a handling rule in the form “when [Case Description], do [Action].” These item-level outputs are then passed through a backend discovery pipeline that embeds, clusters, abstracts, and merges them into candidate edge-case types. The front-end dashboard exposes all annotated examples, uncertainty cues, edge-case clusters, and editable codebook sections; after expert revision, the updated guidelines are used for a new round of annotation [2507.05010].

This design is explicitly expert-in-the-loop: LLMs propose, humans decide. The framework stores earlier codebook versions under “Previous Guidelines,” while the active version appears as “Current Guidelines.” Iteration continues until the expert is satisfied, which is operationally described as the point at which new iterations yield few genuinely novel edge-case types and the codebook appears comprehensive and clear [2507.05010].

## 3. Edge-case induction pipeline

The central algorithm transforms many item-specific ambiguity descriptions into a compact set of high-level edge-case types. At the item level, for each text instance \(x_i\), GPT-4.1 outputs a label \(\hat{y}_i\), a confidence \(c_i \in [0,1]\) in verbalized form, and, when confidence is low, a pair
\[
e_i = (\text{CaseDescription}_i, \text{Action}_i).
\]
These \(e_i\) are deliberately treated as over-specific and instance-bound rather than directly reusable as codebook rules [2507.05010].

The next stage embeds the \(\text{CaseDescription}_i\) strings using OpenAI `text-embedding-3-large`, mapping each description to a vector \(\mathbf{v}_i \in \mathbb{R}^d\). The vectors are then partitioned by constrained KMeans into clusters \(C_1,\dots,C_K\) under the size constraint
\[
10 \le |C_k| \le 20.
\]
The stated rationale is pragmatic: clusters larger than 20 may induce hallucination or abstraction failures in the reasoning LLM, whereas clusters smaller than 5 tend to remain too narrow and specific [2507.05010].

For each cluster, DeepSeek-R1 receives all case descriptions and corresponding actions, and produces an abstracted cluster-level description together with a more general handling rule:
\[
E_k = (\text{ClusterCaseDescription}_k, \text{ClusterAction}_k).
\]
A second DeepSeek-R1 pass then merges overlapping or inconsistent cluster-level outputs into a smaller set
\[
\{E'_1,\dots,E'_{K'}\},
\]
which constitutes the dashboard’s “Suggested Edge Cases.” The framework does not formalize edge-ness through entropy or disagreement scores; instead, low LLM confidence serves as the operational signal of edge-ness, while semantic similarity of natural-language explanations supplies the grouping mechanism [2507.05010].

This architecture distinguishes between “non-reasoning” and “reasoning” LLM roles. GPT-4.1 is used as the fast annotator that answers directly, whereas DeepSeek-R1 is used for long chain-of-thought reasoning in cluster abstraction and merge operations. A plausible implication is that the system treats annotation throughput and conceptual synthesis as separable computational roles rather than assigning both to a single model [2507.05010].

## 4. Codebook evolution and human supervision

The output of the induction pipeline is not a finalized ontology but a set of candidate natural-language rules. In the interface, each suggested edge case is presented with a high-level description and representative examples. The expert may accept a rule, edit it to better match domain understanding, or reject it. Accepted rules are placed in the “Edge Case Handling” section of the codebook [2507.05010].

Conceptually, a rule is expressed as
\[
\text{Rule}_j:\quad \text{WHEN } \text{Condition}_j \;\text{THEN }\text{Action}_j,
\]
where \(\text{Condition}_j\) is a natural-language description of an edge-case type and \(\text{Action}_j\) specifies the appropriate label or decision criterion. The framework does not impose a formal logic syntax; natural language is retained for readability and expert control [2507.05010].

The user interface has two main pages. The input page contains fields for task definition, labels, task ID, CSV upload, and controls such as “Load Demo Data” and “Send.” The analysis dashboard contains “Current Guidelines” and “Previous Guidelines”; a scatter plot of all examples using an embedding-based 2D projection with color indicating label and point size indicating uncertainty or edge-ness; an “All Examples” list with labels, confidence, and edge-case explanations; a scatter plot of edge-case candidates only; a “Suggested Edge Cases” panel with high-level descriptions and representative instances; and an “Edge Case Handling” area where the expert edits rules. An “Iterate” button triggers re-annotation with the revised codebook [2507.05010].

The significance of this arrangement lies in the coupling of annotation and codebook development. Unlike workflows in which a codebook is treated as fixed infrastructure, Co-DETECT operationalizes codebook evolution as a versioned, inspectable process. This suggests a form of annotation governance in which guideline changes are first-class objects rather than informal side effects of pilot coding.

## 5. Evaluation and empirical findings

Evaluation was conducted through a user study, a human validity assessment of edge-case descriptions, and a direct test of whether an improved codebook increases LLM annotation quality [2507.05010].

The user study involved **10 domain experts** from social science, computational linguistics, and interdisciplinary backgrounds, with mixed levels of prior experience in qualitative coding and LLM-assisted annotation. Five participants were independent, and five were co-authors who were unfamiliar with the workflow before the study. The procedure consisted of a pre-interaction survey, an approximately **45-minute** interaction with Co-DETECT on each participant’s own dataset and task, and a post-interaction survey covering usability, interpretability, edge-case quality, iterative refinement, and satisfaction [2507.05010].

A quantitative human evaluation assessed whether cluster-level edge-case descriptions accurately covered sampled examples. Three participants optionally evaluated **41** randomly selected samples, judging whether the cluster-level description fit each instance. Of these, **33 (80.5%)** were judged to be well described by their cluster description, and experts were reported to make these coverage judgments in **<5 seconds per sample**, indicating low supervisory overhead [2507.05010].

The codebook-improvement experiment compared GPT-4.1 annotation using the original first-iteration codebook against the second-iteration codebook after one Co-DETECT run. The evaluation used **GabHateCorpus** for hate speech detection and **GoEmotions** for positive and negative emotion detection, with F1 defined as
\[
F1 = 2 \cdot \frac{\text{precision}\cdot\text{recall}}{\text{precision}+\text{recall}}.
\]

| Dataset | 1st Iter. | 2nd Iter. |
|---|---:|---:|
| GabHateCorpus | 0.2144 | 0.2523 |
| GoEmotions-Positive | 0.0300 | 0.3297 |
| GoEmotions-Negative | 0.2823 | 0.3046 |

The reported changes were an improvement from **0.2144** to **0.2523** on GabHateCorpus, from **0.0300** to **0.3297** on GoEmotions-Positive, and from **0.2823** to **0.3046** on GoEmotions-Negative. The interpretation given in the source is that augmenting the codebook with edge-case handling rules can substantially improve LLM annotation performance without changing the model or training regime [2507.05010].

Survey findings reinforce this interpretation. In pre-interaction responses, **80%** reported manually reviewing subsets of data for edge cases, and **70%** already knew some edge cases in their dataset. After interaction, **80%** found the workflow easy to follow, **60%** agreed that Co-DETECT clearly identifies relevant edge cases, **90%** believed it can help discover new edge cases beyond prior knowledge, and **80%** found the iterative feature helpful for refining annotation guidelines. At the same time, **40%** worried that some edge cases may still be missed, and some participants requested richer visualizations and export features [2507.05010].

## 6. Position within the literature, limitations, and misconceptions

Co-DETECT sits at the intersection of LLM-assisted annotation, human–AI collaborative annotation tools, and goal-driven clustering. Its distinctive emphasis is not merely automatic labeling, but the co-evolution of codebooks and annotations. The framework treats edge-case discovery as a structured induction problem: low-confidence item-level explanations are clustered semantically, abstracted by a reasoning LLM, and translated into natural-language rules that become part of the operative codebook [2507.05010].

The system’s strengths are presented in four terms: mixed-initiative codebook development, systematic edge-case discovery, reasoning-augmented clustering described as “goal-driven clustering,” and a practical demonstration through both expert study and F1 improvements. Its limitations are equally explicit. Dependence on GPT-4.1 confidence and explanation quality may skew discovery if calibration is poor or outputs are biased. DeepSeek-R1 may hallucinate or over-generalize if clusters are too large or too small. Some edge cases may remain undetected because the framework is driven by low-confidence samples and clusters of LLM explanations. The rules remain in natural language rather than formal predicates, which improves readability but may reduce determinism and cross-system consistency. Evaluation focused primarily on social-science-style text classification, so scalability to very large corpora or very many label dimensions remains underexplored [2507.05010].

Several future directions are implied directly by these limitations: improving the coverage and coherence of edge-case suggestions, integrating richer visualizations and export features, addressing potential “Clever Hans” issues that might reinforce model-specific biases, and exploring more formal rule representations across a broader range of domains [2507.05010].

A common source of confusion is nomenclature. Co-DETECT in this setting refers to collaborative discovery of edge cases in text classification, not to the visually grounded open-vocabulary object detection framework “CoDet” [2310.16667], nor to other uses of “co-detection” in network, lexical, or multi-image analysis. This distinction matters because the present framework is fundamentally a mixed-initiative annotation and codebook-refinement system rather than a detection model in computer vision or graph mining.

Source: https://www.emergentmind.com/topics/co-detect