Detailed characterization of value-stability measures

Develop more detailed methods for exploring mean-level, rank-order, and ipsative value stability in language models.

Background

The paper introduces three ways to quantify how expressed values vary across contexts and demonstrates that they yield different assessments of model stability. However, the comparison is limited to a small set of models and textual-format contexts.

The authors explicitly identify the need for more detailed exploration of these stability types. Such methods would support more systematic analysis of how language-model values change under expected and unexpected context variations.

References

These results open many future research directions into models' value stability. Examples of open questions include: "Which types of stability are more important for which types of usecases?", "Are different models specialized in different types of stabilty needed, or can one model by highly stable in all types?", "How can we explore these types of stability in more detail?", and many more.

— Large Language Models as Superpositions of Cultural Perspectives  (2307.07870 - Kovač et al., 2023) in Appendix Section 'Systematic comparison of models on different types of value stability (Mean-level, rank-order, ipsative),' paragraph following Table 4