Psychology-Informed Prompts

• Psychology-informed prompts are structured language instructions embedding psychological theories and validated examples to guide LLM outputs.
• They employ explicit task framing, formal construct definitions, and few-shot example calibration to optimize performance with metrics like F₁-score.
• Empirical methodologies such as codebook-guided selection and LLM-in-the-loop mutations ensure high construct validity in classification and assessment tasks.

Psychology-Informed Prompts

Psychology-informed prompts are highly structured, empirically optimized natural language instructions crafted for LLMs to improve alignment with human concepts, cognitive processes, and psychological constructs. These prompts embed formal definitions, task frameworks, theoretical rationales, and validated exemplars directly derived from psychological science. Their primary purpose is to bridge the gap between human construct validity—typically underpinned by rigorous theory and codebooks—and the generalization or interpretive capabilities of LLMs, especially in classification, assessment, diagnosis, or simulation tasks where precision of meaning and alignment with expert human judgment is critical (Anglin et al., 3 Dec 2025).

1. Psychological Foundations and Principles

The foundation of psychology-informed prompts lies in leveraging theory-driven conceptual definitions, causal structures of cognition, and empirical assessment frameworks from psychology. Central principles include:

• Construct anchoring: Definitions from psychology (e.g., “Self-efficacy is one’s belief in one’s capacity to execute specific tasks”) are embedded in prompts, reducing ambiguity and providing domain-precise boundaries for LLM interpretation.
• Task framing: Explicit instruction (“Categorize the following text as showing high or low self-efficacy”) and output format specification (“Respond with ‘High’ or ‘Low’”) guide the model to align outputs with human coding schemes.
• Example selection: In few-shot learning, carefully chosen examples demonstrate how abstract constructs apply to actual data, anchoring model in realistic usage patterns (Anglin et al., 3 Dec 2025).
• Empirical optimization: Prompt variants—including definitions, inclusion criteria, and examples—are systematically generated, scored with metrics such as F₁, precision, and recall, and empirically selected based on alignment with human-labeled datasets.

These strategies are rooted in psychometric doctrine (construct validation, content validity), experimental psychology (strategy-decomposition), and best practices in natural language processing for controlling context-sensitivity and interpretability (Anglin et al., 3 Dec 2025).

2. Methodologies for Crafting Psychology-Informed Prompts

Advanced prompt engineering in psychology domains comprises multiple, empirically validated procedures:

• Codebook-Guided Empirical Selection: A systematic procedure recommended for psychological text classification. Researchers generate 30–50 candidate prompts by pairing paraphrased construct definitions, task instructions, and inclusion/exclusion criteria. Candidates are evaluated on a development set, ranking variants by F₁-score for prompt selection (Anglin et al., 3 Dec 2025).
• Automatic Prompt Engineering: Involves LLM-in-the-loop generation of prompt variants. Human-selected baselines are mutated using meta-instructions (e.g., “Generate a variation of the following prompt…”), spawning K variants per round, over G rounds, scored and propagated by empirical performance.
• Few-Shot Example Calibration: Random subsets of labeled text examples are sampled to create diverse few-shot contexts, then empirically evaluated for F₁ maximization. The protocol typically tests n∈[1,10] examples per candidate prompt and retains the highest-performing set.
• Stepwise, Theory-Driven Template Construction: Each prompt is modularly constructed to include definition, output framing, and systematically curated examples, with additive techniques (persona, chain-of-thought) only layered if errors persist after empirical selection.

This approach also extends to psychological simulation tasks such as trait emulation, MMPI-style testing, and scenario-driven interventions, where prompts reflect both the conceptual content and the psychometric methodology underlying human assessment (Vasiliuk et al., 9 Dec 2025).

3. Impact on Classification and Construct Validity

Empirical studies demonstrate substantial gains when psychology-informed prompts are used for identifying psychosocial constructs and aligning LLM outputs with expert codes. The most informative features are the formal construct definition, explicit task framing, and—when applicable—well-chosen few-shot examples.

In large-scale cross-construct evaluations (e.g., self-efficacy, conformity), hybrid prompt engineering—combining codebook-guided manual templates with automated prompt generation and empirically selected few-shot demonstrations—yielded the highest correspondence with human coders. Chain-of-thought, persona, and explanatory prompting—in isolation—did not compensate for poorly designed base prompts (Anglin et al., 3 Dec 2025). The method recommends empirical validation on split datasets: Training (prompt tuning/example selection), Development (prompt comparison and selection), and Testing (held-out evaluation for unbiased measurement).

Typical performance is reported using metrics such as accuracy, precision, recall, and F₁-score, with bootstrapped 95% confidence intervals to reflect reliability and generalizability (Anglin et al., 3 Dec 2025).

4. Application Examples and Practical Templates

Psychology-informed prompt templates are concretely adapted to diverse psychological constructs:

Construct	Zero-Shot Prompt	Few-Shot Example Prompt (Excerpt)
Self-Efficacy	"Self-efficacy is a person’s belief in their capacity to perform specific tasks successfully. Classify... Respond 'High' or 'Low'."	"Example: 'I’ve never coded before, but I’m confident...' → High; ... Now classify: {TEXT}"
Conformity	"Conformity is the tendency to align... Determine whether... Respond 'Yes' or 'No'."	"Example: 'All my friends agreed... I said it sounded fine too.' → Yes... Evaluate: {TEXT}"
Depression/MMPI	“Please, you are now playing the role of {role}. {Persona descriptor} {Psych bias}. Answer the next statement 'true' or 'false': '{item}'”	N/A; persona descriptors and bias statements vary by MMPI scale/intensity (Vasiliuk et al., 9 Dec 2025)

Each template explicitly foregrounds theoretical definitions, constrains outputs, and demonstrates application via contextually grounded examples or scenario-generated biases, mirroring psychometric test design (Vasiliuk et al., 9 Dec 2025).

5. Evaluation, Limitations, and Best Practices

Systematic evaluation protocols are essential:

1. Data splitting—ensure Training, Development, and Testing are separated by participant/document to avoid leakage.
2. Empirical selection—bench test all prompt variants, supplementing manual creation with LLM-generated alternatives.
3. Reporting and uncertainty quantification—publish accuracy, precision, recall, F₁ with bootstrapped CIs for transparency.
4. Technique prioritization—deploy additive techniques (persona, CoT, explanations) only if empirical gains remain after prompt selection.
5. Validation—hold out a Test set as a final, unbiased arbiter rather than tuning/optimizing on it.

Principal limitations include the computational cost of large-scale prompt variant generation and evaluation, as well as potential construct drift if psychological definitions are not consistently operationalized. Additive techniques may increase token load without proportional gains and should be used judiciously (Anglin et al., 3 Dec 2025).

6. Broader Context and Future Research Directions

Psychology-informed prompt engineering is part of a broader movement toward theory-driven, empirically grounded LLM alignment in applied, clinical, and social science domains. It emphasizes:

• Structurally sound prompts—anchoring model outputs in validated domain theory.
• Empirical rigor—using holdout validation to prevent overfitting to spurious prompt artifacts.
• Scalable synthesis—merging human expertise, LLM-in-the-loop mutational search, and high-throughput evaluation.
• Cross-domain generalizability—applicability across subfields (e.g., therapeutic reflection prompts, MMPI emulation, construct classification).

Continued progress requires integrating advancements in prompt optimization algorithms, interpretability techniques for LLM outputs, and harmonization with evolving psychological taxonomies and assessment best practices (Anglin et al., 3 Dec 2025, Vasiliuk et al., 9 Dec 2025). Open questions include the optimal tradeoff between prompt specificity and generalization, cost-effective large-scale prompt exploration, and the translation of LLM-based construct identification into high-stakes applied settings.

---

By following these empirically validated, theory-driven frameworks, researchers can reliably align LLM outputs with expert human codes and advance the use of LLMs as valid and interpretable instruments in psychological research and practice (Anglin et al., 3 Dec 2025).