Develop a training algorithm for optimizing the moral graph alignment target

Develop a concrete algorithm for training large language models to optimize the moral graph alignment target produced by Moral Graph Elicitation, converting the moral graph into an objective function suitable for post-training and demonstrating its effectiveness relative to existing alignment methods.

Background

The paper decomposes "aligning to human values" into three stages: eliciting values, reconciling those values into an alignment target, and training the model to optimize that target. The work focuses on the first two stages and introduces Moral Graph Elicitation (MGE) and the moral graph as a new alignment target.

While several options for using the moral graph in training are sketched later, the authors explicitly leave the final stage—creating and validating a training algorithm—open for future work.