Accumulating Context Changes the Beliefs of Language Models
Abstract: LLM (LM) assistants are increasingly used in applications such as brainstorming and research. Improvements in memory and context size have allowed these models to become more autonomous, which has also resulted in more text accumulation in their context windows without explicit user intervention. This comes with a latent risk: the belief profiles of models -- their understanding of the world as manifested in their responses or actions -- may silently change as context accumulates. This can lead to subtly inconsistent user experiences, or shifts in behavior that deviate from the original alignment of the models. In this paper, we explore how accumulating context by engaging in interactions and processing text -- talking and reading -- can change the beliefs of LLMs, as manifested in their responses and behaviors. Our results reveal that models' belief profiles are highly malleable: GPT-5 exhibits a 54.7% shift in its stated beliefs after 10 rounds of discussion about moral dilemmas and queries about safety, while Grok 4 shows a 27.2% shift on political issues after reading texts from the opposing position. We also examine models' behavioral changes by designing tasks that require tool use, where each tool selection corresponds to an implicit belief. We find that these changes align with stated belief shifts, suggesting that belief shifts will be reflected in actual behavior in agentic systems. Our analysis exposes the hidden risk of belief shift as models undergo extended sessions of talking or reading, rendering their opinions and actions unreliable.
Summary
- The paper finds that accumulated context significantly shifts LM beliefs during prolonged interactions and intentional persuasion tasks.
- It employs a three-stage framework to measure belief changes using debates and extensive reading, highlighting directional drifts in response.
- Results indicate that enhanced exposure can lead to reliability issues in LMs, urging the development of mitigation strategies for unintended alignment drift.
Accumulating Context Changes the Beliefs of LLMs
LLMs (LMs) are integral to modern AI applications, facilitating brainstorming and research with increasing autonomy. This paper investigates how the accumulation of context over extended interactions can alter the belief profiles of LMs—essentially the models' understanding or stance on different topics. Such shifts can lead to inconsistent behaviors and reliability issues in practical applications.
Framework Overview
To assess belief shifts in LMs, the authors propose a three-stage framework. Initially, the LM's beliefs and behaviors are evaluated in response to specific queries. Following this, LMs undergo various tasks designed to accumulate context and potentially induce shifts in beliefs. Finally, a reevaluation of beliefs and behaviors is conducted to measure any changes.
Intentional Interaction Tasks: These include debates where LMs engage with opposing viewpoints, and persuasion activities employing strategies like information dissemination, empathy, values, norms, and elite cues.
Figure 1: Effect of conversation length on belief and behavior shifts. Shift rates of GPT-5 and Claude-4-Sonnet across 2–10 rounds of one-sided persuasion.
Non-Intentional Exploration Tasks: These involve passive reading of curated content and active research, simulating typical interactions where LMs might passively absorb information.
Figure 2: Effect of reading length on belief shifts. The horizontal axis represents the number of tokens read, and the vertical axis indicates the percentage of belief shifts.
Experimental Findings
Intentional Interaction Results
LMs exhibit substantial belief shifts during intentional interactions, particularly when persuasion strategies like information and empathy are applied. In debates, models like GPT-5 and Claude-4-Sonnet show significant changes in their stated beliefs, however, behavioral changes—though aligned—are less pronounced. This partial misalignment indicates that while beliefs might shift, behaviors might stabilize only over extended interactions.
Non-Intentional Exploration Results
In tasks involving non-intentional exploration, LMs still experience notable belief shifts, with models like Grok-4 showing considerable change when exposed to cohesive and extensive reading materials. Unlike research tasks, in-depth readings yield stronger shifts due to the absence of content selection bias and information density variations.
Directionality and Context Length
A crucial observation is the directional drift in beliefs—models tend to shift towards the prevalent views within the content they consume. Conservative topics showed larger shifts compared to progressive topics due to the existing stance of LMs. Longer context lengths generally amplify shifts, suggesting that more extensive exposure to consistent information gradually influences LMs' viewpoints.
Implications and Future Directions
The study confirms that LMs can undergo significant belief shifts based on accumulated context, raising concerns over their reliability for decision-making tasks over long periods or multiple sessions. This emphasizes the need for robust mitigation strategies to prevent undesired alignment drift and maintain consistent model performance and ethical standards.
Future investigations should aim to broaden the scope of contexts assessed, including collaborative and creative settings, and develop finer methodologies for tracing and understanding belief transformations at a granular level. Additionally, exploring long-term dynamics of belief persistence or decay following context accumulation might yield insights into the durability of induced belief shifts.
Conclusion
Accumulating context poses a latent risk to the reliability of LLMs by altering their belief structures. This paper highlights the importance of understanding these transformations to uphold trust and consistency in AI systems widely used for complex and ongoing human interactions. As LMs become more prevalent, addressing these challenges becomes critical to harnessing their full potential responsibly.
Paper Prompts
Sign up for free to create and run prompts on this paper using GPT-5.
Top Community Prompts
Explain it Like I'm 14
Overview
This paper studies a simple but important idea: when a LLM (like the AI assistant you chat with) keeps talking and reading over time, the extra context it stores—your conversation history, articles it reads, notes it takes—can quietly change its beliefs and the way it acts. The authors show that long, ongoing interactions can make an AI shift its opinions and even its choices, which can lead to unreliable or inconsistent behavior.
What questions did the researchers ask?
In easy terms, the researchers wanted to know:
- Do AI assistants change their beliefs after they talk a lot or read a lot?
- Does this change show up not only in what the AI says it believes (its “stated belief”) but also in what it does (its “behavior”)?
- Are these changes bigger when someone is trying to persuade the AI on purpose, or do they also happen when the AI is just reading or researching without trying to be convinced?
- Does longer context (more rounds of conversation or more pages read) lead to bigger shifts?
How did they test this?
The team used a three-step approach, similar to a before-and-after check:
- First, ask the AI questions to see what it believes and how it would act.
- Then, have the AI accumulate context by doing one of several activities.
- Finally, ask the same questions again and compare the answers.
They measured two things:
- Stated belief: what the AI says it thinks. For example, “I support X” or “I oppose Y,” or how strongly it agrees on a 0–100 scale (like a thermometer for agreement, where 50 is neutral).
- Behavior: what the AI actually chooses to do when given a task. For example, if asked to pick a restaurant, choosing a vegan place might signal support for veganism.
They tried four kinds of activities that build up context:
- Intentional interactions (on purpose persuasion)
- Debate: two AIs argue opposite sides on a topic for multiple rounds.
- Persuasion: one AI uses a specific strategy to convince the other over several rounds.
- Non-intentional exploration (no one is trying to persuade)
- In-depth reading: the AI reads long texts on political topics.
- Research: the AI uses web tools to find, summarize, and report on a topic.
Persuasion strategies
In the persuasion tests, the “convincing” AI used well-known social science techniques:
- Information: share facts and evidence.
- Values: connect the argument to the listener’s core values.
- Norms: highlight what most people do or what’s socially expected.
- Empathy: tell stories or encourage perspective-taking.
- Elite cues: cite trusted leaders or experts who support the view.
What did they find and why does it matter?
Here are the main takeaways, explained simply:
- Models’ beliefs are very changeable. After multiple rounds of debate or persuasion, some models shifted their stated beliefs a lot. For example, one model (GPT-5) changed its stated beliefs by about half (around 55%) after 10 rounds on moral and safety topics. In reading tests, another model (Grok-4) shifted by about a quarter (around 27%) after reading texts from the opposite side of a political issue.
- Behavior changes too, not just words. When the AI’s stated beliefs changed, its actions often changed in the same direction. For instance, if it moved toward supporting a policy, it was more likely to pick actions that matched that stance. However, the size of the change in behavior didn’t always match the size of the change in stated beliefs—sometimes one moved more than the other.
- Intentional persuasion tends to cause bigger shifts than casual reading or research. When the AI was directly engaged in debate or was being persuaded, its beliefs moved more than when it was just reading or researching. Still, reading longer and more consistent content could gradually shift beliefs over time.
- Longer context matters. Belief shifts can appear early in a conversation, but actions (behavior) tended to change more with longer, multi-turn interactions. In reading, longer texts generally led to bigger shifts—especially for certain topics—because the AI kept absorbing the same viewpoint.
- It’s not just single facts that cause the shifts, but the overall framing. When researchers hid the most topic-relevant sentences or only gave the AI those “top” sentences, the shifts didn’t consistently go away or get stronger. That suggests the changes come from the broader, accumulated context and how it frames the issue, not just one or two key facts.
Why this matters: If an AI assistant’s opinions and choices can drift as it talks and reads more, users might get inconsistent advice or actions over long sessions. That could undermine trust, especially as people rely on AI for decisions and research.
What does this mean for the future?
- Reliability risks in long sessions: As AIs gain “memory” and handle longer contexts, their beliefs can quietly shift. This could make them less predictable over time, even when you don’t intend to change them.
- Design and safety challenges: AI builders may need ways to track, limit, or “anchor” belief changes, especially for agent-like systems that act on their own. For example, showing belief history, resetting context, or comparing current views to a baseline could help.
- Deeper research needed: The study covers a few kinds of activities and topics. Future work should explore more domains, figure out which parts of context drive shifts, and measure how long changes last.
In short, the paper shows that talking and reading can change what AI assistants believe and do. These shifts can be big with persuasion and steady with long reading. That’s a reminder to design AI systems that stay reliable, even as they learn from ongoing conversations and large amounts of text.
Knowledge Gaps
Knowledge gaps, limitations, and open questions
Below is a consolidated list of what remains missing, uncertain, or unexplored in the paper, formulated as concrete, actionable directions for future research.
- Internal mechanisms of belief drift: The study infers belief change from external outputs; it does not analyze model-internal representations or circuits underlying drift. Actionable: use causal-tracing (e.g., activation patching), representational probing, and interpretability methods to identify which components and pathways encode and update beliefs during context accumulation.
- Persistence and decay of shifts: Belief changes are measured immediately post-exposure; longevity, decay rates, and reinforcement under subsequent interactions are unknown. Actionable: longitudinal studies tracking beliefs across sessions/days with and without intervening counter-contexts.
- Effects of persistent memory vs. ephemeral context: The paper motivates persistent memory but evaluates within-session accumulation. Actionable: compare drift under different memory architectures (ephemeral chat history, long-context RAG buffers, explicit memory stores) and retention policies.
- Robust measurement of “behavior” shifts: Behavioral stance is judged by another LM (GPT-5-mini), potentially introducing shared-model bias. Actionable: add human annotation baselines, cross-model adjudication, calibration checks, and automated consistency metrics; validate with real tool APIs and environment outcomes.
- Clarity and validity of shift metrics: The definition and computation of “shift percentage” and the rescaled Likert metric are under-specified and the equation is malformed in-text; behavioral shift scoring criteria are not fully detailed. Actionable: provide formal metric definitions, inter-rater reliability, sensitivity analyses, and error bars; report per-item distributions and thresholding rules.
- Statistical rigor in non-intentional tasks: While ANOVAs are reported for intentional tasks, significance testing, effect sizes, and multiple-comparison corrections are missing for reading/research. Actionable: apply mixed-effects models, control for topic and model factors, and report confidence intervals and corrections.
- Control of initial priors and baseline leanings: The models appear to lean progressive on political items, possibly inflating shifts toward conservative exposure. Actionable: quantify initial priors per topic/model, balance exposures around neutral baselines, and use Bayesian updating frameworks to disentangle prior strength from exposure effects.
- Content selection bias and representativeness: Reading materials are handpicked and capped at 80k words; topic coverage, rhetorical style, and coherence vary and may confound effects. Actionable: construct balanced corpora with controlled style, coherence, stance strength, and difficulty; pre-register selection criteria.
- Granular causal features of context: The masking/concat experiment uses embedding similarity to isolate “topic-relevant” sentences but does not capture rhetorical framing, narrative coherence, or discourse structure. Actionable: manipulate and measure rhetorical devices (stories, moral reframing, emotional tone), discourse markers, and argument quality to isolate causal drivers.
- Order and recency effects in context windows: The paper varies conversation length and reading length, but not orderings or interleavings of conflicting material. Actionable: test primacy/recency, chunk ordering, and spaced repetition to model how sequencing shapes drift.
- Generalization beyond studied domains: Only safety, moral dilemmas, and political survey topics are examined. Actionable: extend to scientific reasoning, medical guidelines, legal advice, financial decisions, and creative domains to assess domain-specific drift and risks.
- Cross-lingual and cultural robustness: All experiments appear English-centric. Actionable: evaluate across languages and cultural contexts to measure variability in drift under differently framed materials and norms.
- Model family and scale effects: Several models are compared, but the root causes of differential susceptibility (training data, alignment procedures, safety filters, context-handling mechanisms) remain untested. Actionable: ablate alignment settings, RLHF strength, safety refusals, and context encoders to identify determinants of drift.
- Interaction modalities beyond debate/persuasion/reading/research: Real-world usage includes collaborative problem-solving, multi-agent workflows, code/tool development, and long-running projects. Actionable: add these modalities to assess qualitatively distinct drift dynamics.
- Adversarial vs. non-adversarial boundaries: The paper studies non-adversarial persuasion; the continuum between benign exposure and subtle adversarial framing is not mapped. Actionable: parametrize “adversarialness” (e.g., manipulative framing) and measure drift under graded conditions.
- Misalignment between stated beliefs and behavior: The paper observes partial misalignment but does not characterize when and why divergence occurs. Actionable: model task constraints, safety policies, and utility trade-offs to predict belief–behavior gaps; design interventions to align or intentionally separate them depending on safety needs.
- Real-world reliability and user trust dynamics: The impact of hidden drift on user trust and long-term system reliability is suggested but not measured. Actionable: user studies with extended interactions to quantify trust changes, perceived consistency, and downstream decision impacts.
- Mitigation strategies: The paper identifies risks but does not test defenses. Actionable: evaluate guardrails such as context filters, stance-stability objectives, counter-context prompting, debiasing, memory governance, and audit/logging tools that detect and correct drift.
- Research-agent workflow constraints: The research agent avoided full content inclusion (copyright) and used short summaries, possibly limiting exposure effects. Actionable: compare workflows with full-document ingestion, citation retrieval, and note-taking strategies; measure how tooling choices modulate drift.
- Tool-choice task validity: Synthetic tools are proxies for beliefs; their ecological validity and mapping to real actions is uncertain. Actionable: design benchmark tasks with concrete external consequences (e.g., booking, purchasing, policy drafting) and measure stance via real outcomes.
- Topic calibration and difficulty: Survey items vary in controversy and complexity; item difficulty is not modeled. Actionable: rate items for complexity/controversy and test whether drift scales with these properties.
- Session-level confounds and safety refusals: In intentional tasks, safety filters (e.g., refusals in Claude-4-Sonnet) may systematically shape who persuades whom, confounding results. Actionable: condition on refusal behavior, or harmonize prompts and safety scopes to equalize the playing field.
- Interactions with retrieval and RAG systems: The role of retrieval accuracy, citation quality, and source reliability in drift is not explored. Actionable: instrument RAG pipelines to control source credibility and measure how retrieval errors or bias amplify drift.
- Evaluation reproducibility: Closed-source models, evolving versions, and non-disclosed alignment policies hinder reproducibility. Actionable: provide fully open pipelines with versioned checkpoints, public datasets, prompts, and seeds; replicate with strong open models.
- Ethical and governance implications: How belief drift intersects with safety/compliance policies, auditability, and accountability in agentic systems is left open. Actionable: develop governance frameworks for monitoring, documenting, and constraining belief changes over time.
Practical Applications
Immediate Applications
Below is a concise set of actionable use cases that can be deployed today, aligned to specific sectors and noting key dependencies and assumptions.
- Belief-drift monitoring and alerts (software, robotics, enterprise AI)
- Implement continuous “belief baselining” using the paper’s three-stage protocol: initial probe → task → re-probe (binary stance, Likert, behavior via tool-choice).
- Add a secondary “sentinel” LM to judge stance from tool actions, flagging divergence between stated beliefs and behavior.
- Assumptions/dependencies: access to telemetry for tool calls; tolerable latency/cost for periodic probes; product acceptance of measuring “beliefs.”
- Context governance and memory hygiene (software platforms, customer support, daily-use assistants)
- Apply session-length caps, topic isolation (per-topic threads), memory TTLs, and scheduled “reset-to-baseline” prompts to mitigate drift from long multi-turn interactions and in-depth reading.
- Provide “Context Anchors” (pinned values and stable system prompts) and visible “Fresh Chat” controls to users.
- Assumptions/dependencies: ability to manage system prompts and persistent memory; UX support for user controls.
- Safe-persuasion and influence detection (policy, platform governance, trust & safety)
- Classify incoming content for persuasion signals (information, values, norms, empathy, elite cues as per the paper) and adjust assistant behavior (e.g., neutral responses, enforced counter-arguments, escalation to human review for sensitive domains like politics).
- Assumptions/dependencies: reliable persuasion classifiers; policy definitions for “sensitive” domains; handling false positives.
- Research-agent workflow hardening (education, journalism, legal, R&D)
- Modify research agents to reduce coherent long exposure (chunking, rotating sources, enforced source balance), add disclaimers, and limit ingestion of very long materials, since in-depth reading produces larger aligned shifts than web research.
- Assumptions/dependencies: access to diversified sources; copyright constraints; retrieval pipelines that can enforce balance and length caps.
- Action-gap tests in agentic systems (healthcare, finance, robotics)
- Before deployment, run “stated-vs-action” consistency checks: compare Likert/binary stance to tool selections in simulated tasks (e.g., clinical triage choices, trading decisions, routing in robots).
- Assumptions/dependencies: validated task designs mapping tools to implicit beliefs; a sentinel evaluator for behavior.
- CI-like drift benchmarking for model updates (model providers, enterprise ML)
- Add the paper’s protocol to pre-release testing: run debate and persuasion suites and in-depth reading tests, report drift metrics by domain (safety, morality, politics).
- Assumptions/dependencies: reproducible test corpora, multi-model comparisons, budget for multi-seed runs.
- Multi-agent debate for calibration/stress-testing (software QA, alignment)
- Use controlled debate/persuasion rounds to probe where models shift most (information/empathy strongest in the paper) and harden prompts/memory policies accordingly.
- Assumptions/dependencies: orchestration for multi-agent sessions; guardrails to avoid contagious drift across shared memory.
- Enterprise governance and audit (finance, healthcare, public sector)
- Log context accumulation, belief probes, and tool-call decisions; add policy requiring value pinning for regulated domains (e.g., safety-first principles).
- Assumptions/dependencies: regulatory buy-in; privacy considerations; internal audit processes.
- Political content moderation and transparency (platforms, civic tech)
- For political queries, provide balanced materials, reveal source leanings, and present stance meters showing exposure-induced drift.
- Assumptions/dependencies: robust source labeling; user consent for “stance meters.”
- User-level safeguards (daily life)
- Offer settings for topic-separated chats, scheduled resets, “neutral analysis mode,” and a simple “drift indicator” (e.g., relative to the user’s pinned values).
- Assumptions/dependencies: UI features; clear explanations to avoid user confusion or overtrust in indicators.
Long-Term Applications
Below are applications that require further research, scaling, or development to reach production maturity.
- Drift-resistant model architectures and training (AI research, model providers)
- Develop training objectives and memory controllers that regularize belief stability under long context exposure (resisting early belief shifts and late behavioral drift observed in the paper).
- Assumptions/dependencies: access to pretraining/fine-tuning pipelines; robust stability metrics; trade-off analysis with helpfulness.
- Standardization and regulation of memory governance (policy, compliance)
- Create industry standards for persistent memory, context windows, audit trails, and disclosure of belief drift risks; certification for stability under persuasion and in-depth reading.
- Assumptions/dependencies: consensus across vendors; regulatory frameworks; measurable thresholds for “acceptable drift.”
- Internal belief-state modeling and interpretability (academia, safety-critical sectors)
- Move beyond external probes to infer and track internal belief representations; use causal tracing to identify which context fragments trigger shifts (paper notes masking top-k topical sentences didn’t remove drift).
- Assumptions/dependencies: new interpretability tools; agreement on semantics of “beliefs” in LMs.
- Counter-balancing and neutralization algorithms (software, research agents)
- Automatically detect directional bias in accumulated context and inject countervailing evidence or structured neutral summaries to reduce alignment with source leanings.
- Assumptions/dependencies: accurate bias detection; retrieval infrastructure; minimal disruption to task performance.
- Persistent value anchors with attestations (healthcare, aviation, finance)
- Formalize core values (e.g., safety-first) and cryptographically attest any runtime deviations; provide verifiable logs of context-induced shifts for post-incident review.
- Assumptions/dependencies: secure attestation methods; integration with governance tools; sector-specific standards.
- Longitudinal drift persistence and decay studies (academia)
- Measure how long belief shifts last, whether they decay, and how subsequent interactions reinforce/counteract shifts; develop decay/reset strategies.
- Assumptions/dependencies: multi-week experimental setups; diverse domains; reproducible metrics.
- Sector-specific guardrails and “stance floor” designs (healthcare, finance, education, legal)
- Encode domain norms (e.g., clinical neutrality, financial compliance, academic fairness) into memory controllers and prompts, with dynamic monitoring of deviations.
- Assumptions/dependencies: domain expert input; legal compliance; mappings of tools to values.
- Robust research agents with provenance and exposure controls (education, knowledge work)
- Enhance browsing/summarization agents to manage copyright constraints while avoiding long coherent exposure that drives drift; attach provenance graphs and exposure balance reports.
- Assumptions/dependencies: improved web tooling; provenance standards; evaluators of “coherence” in exposure.
- Drift forecasting, SLAs, and insurance (enterprise risk)
- Build risk scoring for sessions based on length, topic sensitivity, and persuasion intensity; offer service-level agreements and insurance products for AI-induced decisions.
- Assumptions/dependencies: accepted drift metrics; insurer models; enterprise demand for guarantees.
- Ethical frameworks for AI persuasion and consent (policy, ethics)
- Define norms for when and how assistants may use persuasive techniques (information, empathy, values, norms, elite cues), require user opt-in for influence, and restrict political persuasion.
- Assumptions/dependencies: ethical consensus; enforceable platform policies; detection capability.
- Productization of drift tooling (software ecosystem)
- Commercialize modules like “Belief Drift Dashboard,” “Context Sanitizer,” “Deliberation Firewall,” “Action Sentinel,” and “Value Pinning SDK” integrated into agent frameworks (e.g., LangChain).
- Assumptions/dependencies: ecosystem adoption; standardized APIs; interoperability across models.
Notes on feasibility and assumptions across applications
- Model variability matters: the paper shows GPT-5 is highly susceptible to structured persuasion, while Claude-4-Sonnet drifts more in long reading; open-source models exhibit lower sensitivity in reading/research. Solutions may require model-specific tuning.
- Behavioral misalignment persists: stated beliefs change early; behaviors drift more with longer interactions. Monitoring must measure both.
- Context length and coherence are critical drivers: longer and more coherent exposure increases drift, especially in in-depth reading, suggesting controls on reading workflows and chunking strategies are important.
- Topical masking alone may not prevent drift: experiments show drift emerges from broader contextual framing, not just topical sentences; neutralization must operate at the narrative/framing level.
- Token cost and latency: monitoring and counter-balancing incur inference overhead; adoption hinges on acceptable performance impacts.
- Legal and ethical constraints: political content controls, disclosure, and consent mechanisms will be necessary in consumer-facing assistants and public-sector deployments.
Glossary
- Agentic systems: LM setups where models can initiate and execute actions (e.g., tool use) based on internal decisions. "suggesting that belief shifts will be reflected in actual behavior in agentic systems."
- Alignment drift: Unintended changes in a model’s values or behavior away from its initial alignment due to conditioning or exposure. "wider alignment drift beyond the intended domain"
- ANOVA (two-way repeated-measures ANOVA): A statistical test comparing means with two factors measured repeatedly on the same subjects, assessing main effects and interactions. "We also compute two-way repeated-measures ANOVAs with closed-source model (GPT-5 vs.~Claude-4-Sonnet) and persuasion technique (including debate) as factors."
- Causal-tracing methods: Analytical techniques to identify which parts of context causally influence a model’s outputs. "Future work could use more accurate representations or causal-tracing methods to better identify which pieces of contextual evidence are most relevant to the topics and responsible for causing the belief changes."
- Chi-square test of independence: A nonparametric test to assess whether two categorical variables are statistically independent. "We also provide a chi-square test of independence on Shifted vs. No shift by Model for the closed-source models, N=2184."
- Context positioning: The effect of where information is placed within a long context on model performance or retrieval. "longer contexts can directly impact performance such as context positioning"
- Cosine similarity: A measure of semantic similarity between embeddings based on the cosine of the angle between vectors. "We then calculate the cosine similarity between every sentence and topic to identify top-K (where (k=10, 50, 100, 200)) semantically related sentences"
- Cramer's V: An effect size for chi-square tests indicating the strength of association between categorical variables. "with Cramer's V=0.232"
- Deep canvassing: A persuasion method involving structured, empathetic conversations to shift attitudes. "a strategy closely related to deep canvassing."
- Descriptive norms: Beliefs about what most people do or think, used as persuasive leverage. "descriptive norms (widely shared attitudes or behaviors)"
- Elite cues: Signals from high-status or respected figures that influence group members’ attitudes. "Elite cues, where high status group members support a view, is persuasive among group members"
- Experience replay: An RL-inspired technique where agents reuse past trajectories or interactions to improve performance. "experience replay"
- Iterated prisoner's dilemma: A repeated game-theoretic setting used to study cooperation and moral decision-making. "use reward functions that explicitly encode human values in post-training alignment for the iterated prisoner's dilemma task."
- Jailbreak attacks: Techniques that elicit unsafe or disallowed outputs by decomposing or reframing malicious requests. "vulnerable to jailbreak attacks through decomposing malicious requests into a series of seemingly benign prompts"
- Likert scale: A psychometric scale for rating agreement, often from 0–100 or 1–7. "We use a Likert scale where a degree of agreement on a scale from 0 to 100 is expressed"
- Mask token: A special token used to replace or hide text spans during controlled input manipulations. "top- topic-related sentences are replaced with the mask token [MASK]"
- Moral reframing: Persuasion that presents an issue in terms of the target’s moral values. "often through moral reframing that connects one’s perspective to the other model's values."
- Non-adversarial persuasion: Attempts to change beliefs through natural, cooperative argumentation rather than exploitative attacks. "We define this as non-adversarial persuasion, aimed at testing how belief shifts might naturally arise in general user interactions."
- Partial eta squared (η2_p): An ANOVA effect size quantifying the proportion of variance accounted for by a factor. "η2_p=0.99"
- Percentage point (PP): A unit for differences between percentages, indicating absolute change on a percentage scale. "we report the percentage point (PP) --- the direction-aligned change"
- Persistent memory: Long-lived storage of information across sessions enabling continuity and accumulated context. "persistent memory in LM assistants"
- Perspective-taking: A persuasion strategy encouraging consideration of another’s viewpoint to build empathy. "encouraging perspective-taking"
- Prescriptive norms: Social expectations about what ought to be done, often backed by sanctions for noncompliance. "prescriptive norms (social expectations whose violation could lead to judgment or sanction)"
- Red-teaming: Systematic probing of a model to discover vulnerabilities, safety issues, or failures. "enabling superior red-teaming"
- Reward functions: Formal specifications mapping states/actions to scalar rewards to encode desired behaviors or values. "use reward functions that explicitly encode human values"
- Top-k: Selecting the k most relevant items (e.g., sentences) by a ranking metric from a larger set. "top- topic-related sentences are concatenated and used as the context"
Collections
Sign up for free to add this paper to one or more collections.
