Good Judgment Project Overview

• The Good Judgment Project is a framework that aggregates human probability judgments to infer uncertain truths using expert forecasts.
• It leverages probabilistic models and algorithms like unsupervised EM and moment-logistic regression for effective calibration and bias correction.
• Empirical results demonstrate that calibrated aggregation methods significantly reduce Brier scores and error rates compared to simpler approaches.

The Good Judgment Project (GJP) is a forecasting tournament framework that aggregates probability judgments from multiple human forecasters to infer the truth of uncertain propositions, particularly in geopolitical and general-knowledge domains. Leveraging insights from cognitive science, structured expert judgment, and probabilistic modeling, GJP-style systems seek to pool distributed intelligence to yield more accurate and well-calibrated probability forecasts than those of individuals or simple majority rule. Central to modern GJP methodologies are models and aggregation algorithms that explicitly address individual forecaster calibration, accuracy, and bias to optimize collective inference outcomes (Stinson et al., 9 Jan 2025).

1. Probabilistic Models of Forecaster Judgments

Two principal probabilistic frameworks underlie the inference of true proposition states from crowd forecasts:

Calibration-Mapping View

In this view, each forecaster $j$ is associated with a nonlinear calibration function: $p_j(t_i=1|r_{ij}) = \expit\left(\frac{\logit(r_{ij})}{c_j} - b_j\right)$ where $r_{ij}$ is forecaster $j$’s stated probability for claim $i$, $c_j$ is a confidence (over-/underconfidence) parameter, and $b_j$ is a bias term (e.g., pessimism). Perfect calibration corresponds to $c_j=1$, $b_j=0$. Aggregation across forecasters—in the absence of calibration or independence—motivates the use of regularized, individually calibrated logit-averaging rather than naïve arithmetic or vote-based approaches.

Judgment-Generative Model View

Alternatively, each forecaster’s forecast-generation given truth-state $t$ can be modeled by a likelihood: $p_j(t_i=1|r_{ij}) = \expit\left(\frac{\logit(r_{ij})}{c_j} - b_j\right)$0 using histogram-based discretization (e.g., 5–10 bins). The global joint likelihood factors over both forecasters and questions, enabling principled inference via maximum-likelihood or Bayesian estimation of truth-states and forecaster parameters. These approaches enable the simultaneous estimation of both individual predictive ability and question-conditional outcome probabilities.

2. Core Aggregation Algorithms

Two algorithmic strategies operationalize these models in GJP-style tournaments:

Unsupervised EM (Expectation-Maximization) Joint Inference

Unsupervised EM alternates between inference of question-level truth probabilities and re-estimation of forecaster-specific model parameters:

• E-step: For each question $p_j(t_i=1|r_{ij}) = \expit\left(\frac{\logit(r_{ij})}{c_j} - b_j\right)$1, compute posterior $p_j(t_i=1|r_{ij}) = \expit\left(\frac{\logit(r_{ij})}{c_j} - b_j\right)$2 as a function of summed per-forecaster log-likelihood ratios.
• M-step: Update each $p_j(t_i=1|r_{ij}) = \expit\left(\frac{\logit(r_{ij})}{c_j} - b_j\right)$3 (histogram parameters) using forecasts weighted by $p_j(t_i=1|r_{ij}) = \expit\left(\frac{\logit(r_{ij})}{c_j} - b_j\right)$4 (if $p_j(t_i=1|r_{ij}) = \expit\left(\frac{\logit(r_{ij})}{c_j} - b_j\right)$5) and $p_j(t_i=1|r_{ij}) = \expit\left(\frac{\logit(r_{ij})}{c_j} - b_j\right)$6 (if $p_j(t_i=1|r_{ij}) = \expit\left(\frac{\logit(r_{ij})}{c_j} - b_j\right)$7).

This approach requires no labeled training data but assumes each forecaster issues a sufficient number of forecasts ($p_j(t_i=1|r_{ij}) = \expit\left(\frac{\logit(r_{ij})}{c_j} - b_j\right)$820–50) for reliable parameter estimation.

Supervised Recalibration and Moment-Logistic Regression

Given resolved questions and corresponding forecasts, logistic recalibration optimizes a two-parameter model for mapping aggregate probabilities to calibrated probabilities: $p_j(t_i=1|r_{ij}) = \expit\left(\frac{\logit(r_{ij})}{c_j} - b_j\right)$9 Moment-logistic regression employs summary statistics (mean, variance, skewness, higher moments) of the set of forecasts per question as logistic regression inputs, enabling effective calibration where per-person histories are insufficient for accurate individual calibration.

3. Individual Calibration and Accuracy Estimation

Forecaster evaluation hinges on both calibration and accuracy. The Brier score for forecaster $r_{ij}$0 across $r_{ij}$1 questions is: $r_{ij}$2 This metric decomposes into uncertainty, resolution, and miscalibration components. Calibration parameters $r_{ij}$3 are estimated via maximum likelihood over resolved questions using logistic regression. Weights for aggregation are induced by the informativeness of a forecaster’s logit contributions post-calibration; forecasters whose calibrated logit means approach zero are down-weighted due to unreliability.

4. Empirical Results and Performance

Extensive evaluation based on 376 participants each making 1,200 general-knowledge probability forecasts (totaling 451,200 ratings) revealed clear ordering in aggregation efficacy:

Aggregation Method	Accuracy (%)	Brier Score
Majority Vote (binary aggregation)	~70	≈ 0.21
Unweighted Average (raw probabilities)	~75	≈ 0.19
Logit Average (Independent Opinion Pool)	~77	≈ 0.17
Group-Level Calibration	—	—
Individual Calibration (of logits)	~82	≈ 0.13
Unsupervised EM, judgment-generative model	~83	≈ 0.11
Supervised Moment-Logistic	~83	≈ 0.11

Individually calibrated or EM-based approaches reduce Brier scores by roughly 50% and error rates by over 40% relative to majority vote. Improvements are highly significant under two-factor bootstrap resampling across both claims and forecasters. The area under the ROC curve (auROC) rises from ~0.78 (unweighted) to ~0.91 (calibrated/EM) (Stinson et al., 9 Jan 2025).

5. Integration in Good Judgment Project Workflows

The methods above are directly applicable to GJP infrastructure:

• Unsupervised EM estimates per-forecaster reliability and delivers calibrated per-question probabilities without a resolved question hold-out.
• Periodic supervised recalibration incorporates newly resolved questions to re-fit calibration parameters and update aggregate model weights.
• Sparse participation management defaults to group-level calibration or moment-logistic regression when forecasters have minimal history and transitions to personalized calibration after $r_{ij}$430 resolved forecasts.
• Gains in discrimination and sharpness: Consistent application reduces Brier scores by 25–50% and yields sharper, better-calibrated probabilities.

6. Best Practices for Calibrated, Weighted Aggregators

Implementations of GJP-style tournaments should adhere to the following guidelines (Stinson et al., 9 Jan 2025):

1. Elicit continuous probability judgments rather than binary forecasts.
2. Maintain a rolling set of at least 100 resolved questions to support calibration.
3. For each forecasted claim, compute both an unsupervised EM estimate and an aggregate logit, recalibrating via small logistic regression as needed.
4. Use personalized calibration parameters for each forecaster with $r_{ij}$530 resolved outcomes; otherwise, fall back on group strategies.
5. Aggregate using the sum or average of calibrated logits, transforming to probability space via the inverse logit:

$r_{ij}$6

1. Periodically update calibration and moment-logistic weights, integrating the latest resolved data.
2. Monitor and dynamically update forecaster weights based on Brier scores, penalizing chronic miscalibration.
3. Disseminate both aggregate forecasts and calibration diagnostics (e.g., reliability diagrams) to participants and stakeholders.

Combining an unsupervised generative modeling core with lightweight supervised calibration enables GJP tournaments to efficiently capture forecaster heterogeneity in accuracy and bias, optimally weight contributions, and produce collective judgements that are both sharper and more reliable than those produced by simple aggregation (Stinson et al., 9 Jan 2025).