Identify the mechanism of candidate-set optimization failure

Determine why large language models fail to select the cheapest feasible housing listing reliably as the number of candidate listings increases, distinguishing attention dilution, numeric-comparison instability, position effects, persistent default preferences, and output-stage execution errors.

Background

The audit finds that recommendation quality deteriorates as the number of feasible listings increases: the cheapest listing appears in 93.7% of responses with ten candidates but only 53.5% with eighty, while the rent gap rises and then plateaus. The study establishes that the failure is not resolved by clearer preference instructions, but it does not identify the underlying mechanism.

The authors identify several competing explanations—attention dilution, unstable numerical comparison, position effects, a default preference surviving into generation, and errors in executing an internally correct ranking. They propose order randomization, forced full-ranking before selection, pre-sorted inputs, and tool-based sorting as experiments that could distinguish these explanations.