Validate GPS against human judgment

Determine whether the Governance Policy Simulator (GPS) tracks human judgment when ranking or evaluating AI-governance policies, using a human-subject evaluation study.

Background

The paper evaluates GPS on 15 decision sets by comparing its policy rankings with the plurality choices of five single-lens LLM experts. Although GPS agrees with that LLM panel on 14 of 15 sets, the authors explicitly characterize this comparison as limited because both sides are language-model systems rather than human experts.

The unresolved issue is whether GPS's recommendations correspond to human judgment. Resolving it requires a human study that evaluates the system's policy rankings or recommendations against judgments from human domain experts or other suitably defined human evaluators.

References

Whether it tracks human judgment needs a human study we have not run, the result's central limitation.

— GPS-Bench: A Governance Policy Benchmark for Automating Policy Analysis  (2609.03553 - Le et al., 3 Sep 2026) in Appendix, Section “The Precursor System,” subsection “Does GPS Recommend Sensible Policies?”