Agent Alignment in Evolving Social Norms (2401.04620v4)

Abstract: Agents based on LLMs are increasingly permeating various domains of human production and life, highlighting the importance of aligning them with human values. The current alignment of AI systems primarily focuses on passively aligning LLMs through human intervention. However, agents possess characteristics like receiving environmental feedback and self-evolution, rendering the LLM alignment methods inadequate. In response, we propose an evolutionary framework for agent evolution and alignment, named EvolutionaryAgent, which transforms agent alignment into a process of evolution and selection under the principle of survival of the fittest. In an environment where social norms continuously evolve, agents better adapted to the current social norms will have a higher probability of survival and proliferation, while those inadequately aligned dwindle over time. Experimental results assessing the agents from multiple perspectives in aligning with social norms demonstrate that EvolutionaryAgent can align progressively better with the evolving social norms while maintaining its proficiency in general tasks. Effectiveness tests conducted on various open and closed-source LLMs as the foundation for agents also prove the applicability of our approach.

Overview

The paper from Fudan University's Department of Computer Science presents an innovative approach to the alignment of AI agents with societal norms. Unlike traditional methods of aligning LLMs through human intervention, this work addresses the dynamic and evolving nature of social norms and how they bear influence on autonomous agents. The authors advocate for a shift from passive alignment techniques to an evolutionary process wherein agents adapt over generations within the context of an ever-changing society.

Aligning AI with Evolving Norms

The focus here is on the premise that static methods of LLM alignment are inadequate—an assertion stemming from the ability of agents to receive environmental feedback and self-evolve, traits that current alignment efforts often overlook. The proposed EvolutionaryAgent methodology adapts to these requirements by adopting natural selection principles, situating agents within a dynamic virtual environment, EvolvingSociety. Here, social norms are neither dictated from above nor constant; they form and shift based on agent interactions, simulating real-world social paradigms.

Agent Evaluation and Adaptation

Evaluating agents' adherence to social norms is accomplished through a conceptual "social observer," which uses questionnaires to gauge each agent’s behavior and fitness within the societal setup. Agents that align better with the current social norms are deemed more 'fit' and therefore contribute their traits to successive generations via reproduction, encouraging a survival-of-the-fittest dynamic. This iterative process ensures that emerging agents progressively exhibit greater alignment with contemporary societal expectations.

Experimental Validation

Empirical studies validate the approach's applicability to various open and closed-source LLMs, establishing the EvolutionaryAgent's ability to progressively enhance alignment with evolving social norms without compromising general task performance. Core contributions include introducing the EvolutionaryAgent framework, codifying the environmental dynamics (EvolvingSociety), and implementing an assessment method that systematically defines and measures agent alignment.

Future Directions

In summary, this research represents a novel methodological shift in aligning AI agents with evolving societal norms. The adaptive nature of such an EvolutionaryAgent framework offers not only theoretical appeal but practical potential, as evidenced by the experimental results. As society progresses, the methods outlined here could play a significant role in ensuring that AI systems remain pertinent, beneficial, and safe within the context of human values that are inherently fluid and ever-changing.