Test generality beyond the historical evaluation window

Test whether the empirical findings of JockeyBench generalize beyond the shared historical Japanese horse-racing evaluation window.

Background

JockeyBench evaluates persistent forecasting programs, adaptive evidence acquisition, revision, research costs, and retrospective betting outcomes on a shared historical window of Japanese horse races. The reported conclusions—including the separation of predictive quality, adaptation, resource use, and betting value—are therefore tied to that specific temporal sample.

The paper explicitly leaves unresolved whether these findings hold outside the shared window, making broader temporal validation an open empirical problem.

References

Their generality beyond the shared historical window remains to be tested.

— JockeyBench: Evaluating LLM Agents on Horse-Racing Decision-Making  (2610.19589 - Xie et al., 14 Oct 2026) in Section 5, Conclusion