User preference for models fine-tuned on the moral graph

Determine whether users prefer interacting with a language model fine-tuned on the moral graph alignment target, as compared to their current interactions with existing systems.

Background

The authors outline several ways to train models on the moral graph, including generating datasets for RLHF-like pipelines and training reward models on wisdom upgrades. They discuss the need to deduce moral contexts throughout a dialogue and to rate completions by adherence to retrieved values cards.

Despite these proposals, the authors explicitly state uncertainty regarding user preference for models fine-tuned using the moral graph and indicate an ongoing effort to fine-tune a larger model to address this question.