Direct measurement of AI consumer capability preferences

Construct and apply a survey or other preference-elicitation instrument, such as forced-choice experiments, conjoint analysis, or willingness-to-pay measurement, to determine whether the consumer need weights assigned to the reasoning, coding, knowledge, safety, communication, and agentic capabilities accurately represent users’ relative preferences.

Background

Consumer satisfaction in the simulation is computed from provider capability vectors and researcher-assigned need-weight vectors for 17 use-case profiles. The weights are informed by sector-level evidence about professional AI use and concerns, but the exact numerical values are imposed by the researchers rather than directly elicited from users.

The authors explain that the classification of benchmarks as widening or shrinking can change depending on whose satisfaction is used and on the distribution of consumer needs. Direct preference measurement is therefore needed to test the judgment embedded in the model’s consumer-satisfaction function.