Implications of Demographic Information in System Prompts vs User Interactions

Determine how embedding demographic information about end-users within deployer-level system prompts, as opposed to collecting such information through user interactions (e.g., conversational inputs and memory functions), affects large language model behavior toward different audience groups.

Background

The paper discusses how system prompts, which take precedence over user prompts, are increasingly tailored with audience-specific information that can shape model behavior in opaque ways across AI supply chains. This layering creates accountability and transparency challenges because deployers and end-users typically cannot see upstream prompt configurations.

Within this context, the authors highlight that demographic information may be embedded in system prompts or inferred through user interactions and memory functions. While tailoring responses to specific audiences aims to improve relevance, it risks introducing representational and allocative biases. The authors explicitly state that understanding the implications of placing demographic information in system prompts versus gathering it through user interactions, and how this affects different audience groups, remains unresolved.