Determine causes of Elo discrepancy against humans vs. bots on Lichess

Ascertain the precise factors that cause the observed discrepancy between the Lichess blitz Elo ratings achieved by the 270M-parameter transformer action-value policy when playing exclusively against human opponents versus when playing against bots, rigorously evaluating the extent to which resignation behavior, rating-pool miscalibration between humans and bots, and differential exploitation of occasional tactical mistakes by bots contribute to the difference.

Background

The authors report a substantial difference between the Lichess blitz Elo achieved by their strongest model when playing exclusively against humans versus when playing primarily against bots. They note that the reasons for this discrepancy are not fully understood and offer several hypotheses that could explain it.

Understanding and quantifying the causes of this discrepancy would clarify how evaluation conditions (human vs. bot opponents), resignation behavior, rating-pool calibration, and opponent-specific exploitation of weaknesses affect measured playing strength, thereby informing more robust evaluation protocols for search-free chess agents.