Hybrid Confirmation Trees

• Hybrid Confirmation Trees (HCTs) are sequential aggregation strategies that combine independent human and AI judgments, deferring to a second human only when the first human and AI disagree, ensuring human agency.
• HCTs consistently achieve or surpass the accuracy of three-person human majority voting with substantially less human input, particularly when human and AI accuracies are comparable and their errors are uncorrelated.
• Empirical evaluations across six diverse domains, including skin cancer detection and deepfake detection, demonstrate HCT's up to 10.4 percentage points higher accuracy and 28 to 44% lower human cost compared to majority voting.

The hybrid confirmation tree (HCT) is a sequential aggregation strategy for binary decision-making that systematically integrates independent human and AI judgments, operationalizing a robust human-in-the-loop protocol that preserves human agency. The HCT’s central procedure first compares independent votes from one human and one AI, deferring to a second human only in instances of disagreement. Analytical, probabilistic, and empirical investigations have established the framework’s ability to meet or exceed the accuracy of three-person human majority voting, especially when human and AI accuracies are comparable and their errors exhibit low correlation, all while substantially reducing necessary human input (Berger et al., 2 Feb 2026).

1. Decision Rule and Sequential Workflow

The HCT operates on a streamlined, three-step protocol. For each binary-labeled case $x$:

1. A first human ($\mathrm{H}_1$) and the AI model ($M$) independently provide labels.
2. If $\mathrm{H}_1 = M$, their joint label is accepted as final and only one human is consulted.
3. If $\mathrm{H}_1 \neq M$, a second human ($\mathrm{H}_2$) adjudicates, and their label becomes the output, requiring two human inputs.

Pseudocode formalization: $\mathrm{H}_1 \neq M$1 In all scenarios, the output is sanctioned by at least one human, ensuring the retention of human agency at the point of final annotation.

2. Mathematical Formalism and Notation

Assume binary labels ("0" vs "1") and indicator random variables for correctness. Let:

• $p_h$: Probability the human is correct.
• $p_a$: Probability the AI is correct.
• $\rho$: Correlation between the correctness indicators $I_h$ (human) and $\mathrm{H}_1$0 (AI).

Define

• $\mathrm{H}_1$1 (joint correct agreement).
• $\mathrm{H}_1$2 (total agreement, both correct or both wrong).

HCT accuracy ($\mathrm{H}_1$3) is then given by:

$\mathrm{H}_1$4

This form encompasses scenarios wherein both agree correctly, or one corrects the other via the tiebreaking step.

3. Comparative Accuracy Analysis

Three-person majority voting, using independent humans each with accuracy $\mathrm{H}_1$5, achieves:

$\mathrm{H}_1$6

Under independence ($\mathrm{H}_1$7), HCT accuracy matches or exceeds $\mathrm{H}_1$8 whenever $\mathrm{H}_1$9. The region where $M$0 is maximal when $M$1 and $M$2 is small, with complementarity—outperformance relative to both majority vote and AI alone—peaking in this regime.

4. Cost of Human Input

Expected human consultation cost in HCT:

$M$3

For majority voting:

$M$4

Replacement of majority voting with HCT yields up to 33% reduction in expected human effort, even at equivalent accuracy. Human-input savings are given by:

$M$5

A maximal gain is realized when $M$6 and $M$7.

5. Flexibility in True/False Positive Trade-Offs

With probabilistic AI outputs ($M$8), the decision threshold $M$9 for the model’s positive classification induces a continuum of true and false positive rates, generating a spectrum of operating points analogous to a shifted AI ROC curve. At $\mathrm{H}_1 = M$0, HCT reduces to “polyarchy” (final positive if either human or AI is positive); at $\mathrm{H}_1 = M$1, to “hierarchy” (both must be positive). Intermediate thresholds allow for Pareto-optimal navigation of the true-/false-positive frontier, in contrast to fixed setting heuristics which afford only single trade-off points.

$\mathrm{H}_1 = M$2 value	Reduction to	Decision Rule Example
$\mathrm{H}_1 = M$3	Polyarchy	Output positive if $\mathrm{H}_1 = M$4 OR $\mathrm{H}_1 = M$5 is positive
$\mathrm{H}_1 = M$6	Hierarchy	Output positive if $\mathrm{H}_1 = M$7 AND $\mathrm{H}_1 = M$8 are positive
$\mathrm{H}_1 = M$9	Flexible Pareto frontier	Operating point set by $\mathrm{H}_1 \neq M$0

6. Empirical Evaluation Across Six Domains

The HCT procedure has been empirically evaluated on six real-world data regimes, each entailing crowds of human annotators and a relevant AI model:

Domain	Data Scale / Methods	HCT Advantage over Majority Vote
Skin cancer (dermoscopic)	100 images × 157 dermatologists × CNN (ISIC)	Up to +10.4 pp accuracy, 28–44% less cost
Skin cancer (nondermatologist)	100 images × nondermatologists × same CNN	+8.5 pp accuracy gain
Deepfake detection	54 videos × 132 humans × CNN (AUC=0.95)	+7.9 pp accuracy gain
Criminal rearrest	1,000 cases × 400 MTurk workers × logistic regression (AUC=0.72)	+2.3 pp accuracy gain
Hybrid Forecasting Competition	52 events × 111 humans × time-series model	+10.4 pp accuracy gain
ForecastBench	422 events × 500 humans × leading LLM	+4.5 pp accuracy gain

In all settings, improvements over majority voting were statistically credible (Bayesian 95% highest density intervals exclude zero; probability of practical significance >99%), with cost reductions consistently in the 28–44% range. Cross-validation confirmed out-of-sample persistence of these gains. HCT outperformed majority vote even for simulated human crowds up to size 15.

7. Significance and Applicability

The hybrid confirmation tree provides a transparent, practical, and general aggregation protocol for hybrid collective intelligence, maintaining explicit human agency over final decisions. It systematically outperforms both human-majority votes and AI-alone baselines under realistic conditions—specifically, when human and AI accuracies are similar and their errors are uncorrelated. Furthermore, HCT enables fine-grained control over decision operating points via threshold adjustment, addressing flexibility limitations inherent in static human-only heuristics such as hierarchies and polyarchies. These characteristics collectively position the HCT as a robust standard for cost-efficient, high-accuracy, human-in-the-loop decision aggregation (Berger et al., 2 Feb 2026).