Importance of different value-stability dimensions

Determine which types of value stability—mean-level, rank-order, or ipsative stability—are most important for different use cases.

Background

The paper evaluates six LLMs using three distinct forms of value stability: mean-level change, rank-order stability, and ipsative stability. The results show that the model appearing most stable depends on which stability measure is used.

The authors therefore leave unresolved which stability dimension should matter most in a given application. Answering this question would help determine how model stability should be evaluated for different practical use cases.

References

These results open many future research directions into models' value stability. Examples of open questions include: "Which types of stability are more important for which types of usecases?", "Are different models specialized in different types of stabilty needed, or can one model by highly stable in all types?", "How can we explore these types of stability in more detail?", and many more.

— Large Language Models as Superpositions of Cultural Perspectives  (2307.07870 - Kovač et al., 2023) in Appendix Section 'Systematic comparison of models on different types of value stability (Mean-level, rank-order, ipsative),' paragraph following Table 4