Future-as-Label: Scalable Supervision from Real-World Outcomes

Abstract: Time creates free supervision: forecasts about real-world events resolve to verifiable outcomes. The passage of time provides labels that require no annotation. To exploit this structure, we extend reinforcement learning with verifiable rewards to real-world prediction over time. We train LLMs to make probabilistic forecasts from causally masked information, using proper scoring rules as the reward function once events resolve. Learning is driven entirely by realized outcomes, enabling scalable outcome-based supervision in open-world prediction. On real-world forecasting benchmarks, Qwen3-32B trained using Foresight Learning improves Brier score by 27% and halves calibration error relative to its pretrained baseline, and outperforms Qwen3-235B on both constructed future-event prediction tasks and the Metaculus benchmark despite a 7x parameter disadvantage.

• The paper introduces outcome-based supervision that uses post-event outcomes as labels, overcoming delayed and sparse feedback in forecasting.
• It distinguishes between a predictor and a resolver, using Group Relative Policy Optimization to assign credit and reduce variance in forecast training.
• Empirical evaluations demonstrate a 27% Brier score improvement and superior calibration on both synthetic benchmarks and external datasets.

Future-as-Label: Outcome-Based Supervision for Real-World Prediction

Problem Setting and Motivation

Forecasting real-world events with LLMs fundamentally suffers from the absence of immediate, contemporaneously available supervision. In many domains—such as judicial decisions, geopolitical events, corporate actions—the ground truth lag renders conventional supervised learning inapplicable, as labels are only available post hoc. Standard RL with verifiable rewards (RLVR) methods rely on reward signals that can be computed contemporaneously with prediction, naturally limiting them to domains with closed-form, automatable verification (e.g., mathematics, code, or formal proofs).

The paper "Future-as-Label: Scalable Supervision from Real-World Outcomes" (2601.06336) formalizes and operationalizes outcome-based supervision derived retroactively from the actualization of future events. This formulation generalizes RLVR to settings with open-world uncertainty, sparse feedback, and intrinsic delays between prediction and resolution, establishing a framework for supervised policy optimization aligned with the true causal structure of real-world forecasting.

Methodology

Causal Structure and Roles

The learning protocol distinguishes between a "predictor" and a "resolver":

• Predictor: The trainable LLM, which, at time $t$, only observes a temporally masked information state (all data timestamped no later than $t$). It emits a probabilistic prediction $p\in(0,1)$.
• Resolver: An external, frozen process (in this work, a frozen LLM) with full access to post-$t$ information, tasked exclusively with resolving ground truth $y\in\{0,1\}$ after event actualization at some $s>t$.

There is strict causal separation: the predictor receives no outcome information during prediction or learning, and the resolver cannot access predictor outputs, ensuring no leakage or endogenous reward shaping.

Objective and Optimization

Supervision is derived exclusively from post-resolution outcomes, with the per-episode terminal reward given by the log-score (proper scoring rule):

$R(p, y) = y\log p + (1-y)\log(1-p)$

While this superficially resembles supervised likelihood optimization, the formulation is fundamentally RL-style: the predictor samples reasoning trajectories under masked state, credit is attributed by the terminal outcome with no intermediate rewards, and optimization proceeds via policy gradient, specifically Group Relative Policy Optimization (GRPO), which reduces estimator variance by groupwise subtraction of the within-group mean reward.

Training and Data Generation Protocol

• Dataset Construction: Events are paired with all information available up to a strict cutoff $t$, prohibiting any post hoc leakage. Outcomes are resolved automatically using a frozen, post-hoc LLM given exhaustive post-$t$ sources. Training, test, and evaluation sets are strictly partitioned in time.
• Masking: All model inputs are timestamp-masked—to enforce the predictor only sees what would be knowable as of the forecast moment.
• Evaluation: Metrics include log-score, Brier score, and expected calibration error (ECE), across both a synthetic, tightly controlled future-event benchmark and an independent Metaculus dataset.

Empirical Findings

The Qwen3-32B model trained with Foresight Learning under outcome-based supervision demonstrates:

• 27% improvement in Brier score and a halving of calibration error relative to the base pretrained model when evaluated on held-out synthetic event benchmarks.
• Consistent outperformance relative to both model scale and sampling-based baselines: Qwen3-32B with Foresight Learning exceeds the accuracy and calibration of the much larger Qwen3-235B model, indicating that gains stem from the training objective rather than scale or ensembling.
• Generalization to external benchmarks: On independently sourced Metaculus questions, Foresight-trained models retain strong calibration and accuracy, despite domain shift.
• Variance ablation: Pure ensembling/generation of multiple probabilistic outputs does not close the gap, underscoring the learning benefit of outcome-aligned supervision.

Significance and Implications

This work operationalizes RL-style training under real-world, outcome-based feedback, bridging the full temporal gap between forecast and resolution—a regime that is unavoidable in open-world deployment. The demonstrated gains in both accuracy and (notably) calibration highlight the centrality of outcome-centric policy shaping for reliable forecasting, as proper scoring rules effectively penalize overconfidence and reward uncertainty quantification.

From a theoretical perspective, Foresight Learning establishes a framework for scalable RL in environments with delayed, sparse, and exogenous feedback. By leveraging group-relative credit assignment, it mitigates the sample inefficiency that plagues naive policy gradients under sparse rewards. The method is robust to scale and outperforms sampling heuristics, suggesting that—even in large-model regimes—training objective selection can have impacts exceeding those from model scaling alone.

Limitations and Future Work

• Offline training: All feedback is derived from pre-resolved events. Extending to online and continually updating forecasters that autonomously collect and resolve labels would require integrating deployment-time feedback.
• Automated Question/Resolution Biases: The fully automated pipeline for event and outcome specification, while reproducibility-friendly, is subject to data coverage and resolver error risks.
• Outcome Space: The work is restricted to binary outcomes; extension to continuous, categorical, or complex structured labels (e.g., distributions, free text) remains open.

Conclusion

The paper provides a formal and empirical advance in outcome-aligned model training, showing that direct supervision from temporally resolved, real-world outcomes yields superior accuracy and calibration over both scaling and sampling-based SFT alternatives. This paradigm is essential for scaling LLMs into open-world domains, and points toward the broader adoption of outcome-based RL objectives for robust, deployable future-event prediction models. Further extension to richer event structures and online learning scenarios is an important next step for the field.