Agentic Forecasting using Sequential Bayesian Updating of Linguistic Beliefs

Abstract: We present BLF (Bayesian Linguistic Forecaster), an agentic system for binary forecasting that achieves state-of-the-art performance on the ForecastBench benchmark. The system is built on three ideas. (1) A linguistic belief state: a semi-structured representation combining numerical probability estimates with natural-language evidence summaries, updated by the LLM at each step of an iterative tool-use loop. This contrasts with the common approach of appending all retrieved evidence to an ever-growing context. (2) Hierarchical multi-trial aggregation: running $K$ independent trials and combining them using logit-space shrinkage with a data-dependent prior. (3) Hierarchical calibration: Platt scaling with a hierarchical prior, which avoids over-shrinking extreme predictions for sources with skewed base rates. On 400 backtesting questions from the ForecastBench leaderboard, BLF outperforms all the top public methods, including Cassi, GPT-5, Grok~4.20, and Foresight-32B. Ablation studies show that the structured belief state is almost as impactful as web search access, and that shrinkage aggregation and hierarchical calibration each provide significant additional gains. In addition, we develop a robust back-testing framework with a leakage rate below 1.5\%, and use rigorous statistical methodology to compare different methods while controlling for various sources of noise.

• The paper introduces the BLF system that uses a Bayesian linguistic belief state updated sequentially in an agent loop to achieve state-of-the-art performance on ForecastBench.
• The methodology integrates hierarchical aggregation and Platt scaling to reduce overconfidence and improve calibration through ensemble rollouts.
• The system combines web search, tool use, and crowd priors with rigorous date-leakage mitigation, outperforming competitors in binary event forecasting.

Agentic Forecasting with Sequential Bayesian Updating of Linguistic Beliefs

System Overview

The paper introduces BLF (Bayesian Linguistic Forecaster), an agentic forecasting system for binary event prediction that achieves state-of-the-art performance on the ForecastBench benchmark (2604.18576). The system is grounded in three innovations: the maintenance of a semi-structured linguistic belief state, hierarchical multi-trial aggregation with logit-space shrinkage, and hierarchical calibration via Platt scaling with per-source offsets. BLF leverages web search, tool use, and crowd/empirical priors while enforcing rigorous date-leakage controls to preserve backtesting validity. Experimentation is conducted on a comprehensive evaluation suite, including 400 ForecastBench questions and 113 AIBQ2 questions.

Linguistic Belief State and Agentic Loop

The core methodological advancement is the Bayesian linguistic belief state, which encapsulates numerical probability estimates, confidence levels, evidential summaries, and ongoing queries. Unlike naive evidence accumulation (growing text context) or batch search-and-reason strategies, BLF updates the belief state stepwise in an agent loop, where the LLM jointly proposes an action (e.g., web search, tool use) and revises its belief. Each tool call is date-filtered and leak-audited, ensuring temporal integrity.

Multi-trial stochasticity is addressed by running $K=5$ independent agentic rollouts per question, aggregating predicted logits with tuned shrinkage based on empirical variance. The agentic harness selectively enables tools per question type, with domain-specific statistical models on sources such as DBnomics bypassing LLM reasoning where appropriate.

Figure 1: Evolution of belief state across agent rollouts demonstrates high inter-trial variance, motivating ensemble aggregation.

Hierarchical Aggregation and Calibration

BLF exploits logit-space mean aggregation; when variance is high, shrinkage toward $p=0.5$ reduces overconfidence, particularly for hard questions. Empirical analysis shows statistical significance of ensemble aggregation, with higher $K$ consistently improving proper scoring rule metrics (Brier Index, MBS). Hierarchical Platt scaling is adopted for post-hoc calibration, with per-source intercepts to avoid global over-shrinkage on sources with skewed base rates.

Ablation studies validate that belief state structure yields comparable gains to web search access. Removal of sequential search, belief tracking, or tools individually degrades Brier Index by ~3–5 points, with crowd priors boosting market-specific accuracy.

Figure 2: Comparative Brier Index performance across models, with BLF outperforming external baselines on both market and dataset questions.

Rigorous Evaluation and Leakage Mitigation

Backtesting uses datasets strictly post-cutoff relative to LLM training. Leakage is minimized via a four-stage defense: search engine date filters, LLM classifiers on result content, algorithmic date-clamping in data tools, and blacklist enforcement on resolution URLs. Empirical leakage rate is held below 1.5%, far beneath noise from question difficulty or stochastic LLM behavior.

Figure 3: ForecastBench leaderboard screenshot highlights BLF's performance relative to agentic and zero-shot baselines.

Empirical Results

On 400 ForecastBench questions, BLF achieves Brier Index #1{73.8} (market #1{85.2}, dataset #1{62.4}), significantly outperforming all public baselines (paired bootstrap $p<0.001$). Only BLF surpasses the crowd baseline on market questions—a result not replicated by Cassi, Grok~4.20, Foresight-32B, or GPT-5. Difficulty-adjusted BI matches the superforecaster median on the leaderboard.

Multi-trial aggregation, logit mean, and calibrated hierarchical Platt scaling together produce robust forecast distributions, showing no degradation with additional trials and preserving extremity where appropriate. Source-specific empirical priors are used for dataset questions, providing competitive baselines especially on sources with extreme base rates (e.g., ACLED, Wikipedia vaccine questions).

Figure 4: Reliability diagram demonstrates calibration effectiveness for BLF’s predictions.

Behavioral and Ablation Analysis

Detailed trace analysis reveals agentic reasoning paths with high inter-trial divergence, justifying ensemble methods. Tool usage and action sequences are source-specific; DBnomics questions are solved with statistical KNN models, bypassing LLM inference, while Polymarket and Wikipedia questions leverage historical data extraction and search.

Ablations show that removing the structured belief representation, switching to batch approaches, or using lower-capacity models (Kimi, Flash) yields significant drops in BI. Ensembling multiple base models only improves performance when diversity is genuine, else aggregation of weaker models actively hurts accuracy (JSD analysis confirms low cross-model forecast diversity on FB, moderate on AIBQ2).

Figure 5: Mean tool calls per question, illustrating selective use of domain-specific tools and LLMs.

Figure 6: Agent steps per question, showing runtime efficiency and learned stopping behavior.

Implications and Future Directions

BLF establishes agentic tool-use and sequential linguistic belief updating as essential to SOTA probabilistic forecasting with LLMs. The results suggest that structured belief tracking is as critical as web search access, and that hierarchical calibration is required to avoid miscalibration on sources with heterogeneous priors. The system’s modular harness supports the integration of domain-specific statistical models, reinforcing the importance of combining LLMs with expert system tools.

Practically, BLF demonstrates that beating crowd baselines on market questions is currently only feasible via agentic reasoning and belief state tracking; zero-shot prompting, even with crowd information, does not suffice. Theoretically, the Bayesian approach supports further integration with RL fine-tuning and online learning for live environments such as FutureX and ProphetArena.

Future work should extend to categorical and numerical forecasting, dynamic weighting of prior signals based on market metadata, adaptive meta-controller learning, RL fine-tuning for harness optimization, and continuous live evaluation. Human-AI ensemble strategies may also offer substantial gains over silicon-only aggregation.

Conclusion

The Bayesian Linguistic Forecaster system demonstrates that structured sequential belief updating, agentic tool use, and hierarchical Bayesian aggregation are collectively necessary for state-of-the-art forecasting with LLMs. The empirical results, rigorous ablation analysis, and robust controls against information leakage collectively establish BLF as a new benchmark for automated probabilistic forecasting, with strong implications for the integration of LLMs in high-stakes prediction domains.