Evaluative Disposition: Concepts and Applications

• Evaluative disposition is a stable tendency to form context-triggered evaluations, revealing inherent biases in behaviors and judgments across various domains.
• It is measured through revealed behaviors such as situational judgment tests, trait-positive rates, and consistency in evaluator assessments in both humans and models.
• Its applications span social dilemmas, LLM alignment, financial decision-making, and evaluative AI interfaces, underscoring implications for ethics and governance.

Searching arXiv for the cited paper and adjacent work on evaluative disposition to ground the article. arxiv_search(query="(Taubenfeld et al., 11 Feb 2026) Evaluating Alignment of Behavioral Dispositions in LLMs", max_results=5, sort_by="submittedDate") Evaluative disposition denotes a relatively stable tendency by which an agent forms, expresses, or enacts evaluations under contextual pressure. In current research, the term spans several related but non-identical usages: underlying behavioral tendencies in social dilemmas, willingness to engage in critical inquiry, systematic patterns in financial realization behavior, judge-specific theories of quality in LLM-as-judge systems, and evaluator-side strategic orientations embedded in institutions (Taubenfeld et al., 11 Feb 2026). A recurrent theme across these literatures is that evaluative disposition is better inferred from situated choices, comparative judgments, or revealed behavior than from abstract self-description alone (Donati et al., 2024).

1. Conceptual range

A philosophically explicit formulation treats a disposition as a property that an entity instantiates and that may manifest under the right stimulus conditions. In the soft-ethics literature, dispositions are individuated by stimulus–manifestation pairs and linked to counterfactual conditionals of the form “if $x$ were $S$, $x$ would $M$”; ethical and behavioural preferences are then modeled as dispositions triggered by scenario features and manifested in choice (Donati et al., 2024). This framing makes evaluative disposition inherently contextual: the relevant object is not a decontextualized preference, but a tendency to respond in a particular way when a morally or socially structured situation is encountered.

Across adjacent fields, the same family of ideas appears under different operationalizations. In work on LLM alignment, behavioral dispositions are the underlying tendencies that shape responses in social contexts, such as empathy, emotion regulation, assertiveness, and impulsiveness (Taubenfeld et al., 11 Feb 2026). In LLM evaluation, an evaluative disposition is a judge-specific, stable, implicit theory of quality that determines harshness, dimension emphasis, and evidence behavior (Nasser, 8 Jan 2026). In strategic-evaluation theory, the closest equivalent is the evaluator’s own strategic posture: the evaluator is not a neutral measurer, but an agent with interests that may diverge from those of society (Laufer et al., 2023). In behavioral finance, the “disposition effect” is the tendency to sell positions that are currently in profit too quickly while retaining positions that are currently in a loss for too long, and can be read as a domain-specific evaluative disposition toward realized gains and losses (Breitmayer et al., 2019).

Domain	Evaluative object	Representative formulation
Social behavior and LLM alignment	Advisory behavior in dilemmas	Behavioral dispositions shaping responses in social contexts
Soft ethics	Morally loaded scenario response	Dispositions triggered by setting, problem, and action
LLM-as-judge	Quality assessment	Stable, judge-specific implicit theory of quality
Finance	Gain/loss realization	Sell winners too early, hold losers too long
Critical thinking	Willingness to think critically	Motivation and cognitive components
Emergency medicine	ED outcome	“Patient disposition” as final destination, not evaluative tendency

The last row is a terminological boundary. In emergency-department prediction, “patient disposition” refers to the final outcome or destination of an ED visit—admission, home, transfer, and related categories—rather than a stable evaluative tendency (Feroz et al., 2024). This contrast is useful because it shows that not every use of “disposition” in technical writing concerns evaluation in the present sense.

2. Formalization and measurement

Recent work has moved from trait labels to revealed behavior. In “Evaluating Alignment of Behavioral Dispositions in LLMs” (Taubenfeld et al., 11 Feb 2026), validated psychometric statements are converted into chatbot-adapted advisory commitments and then into binary Situational Judgment Tests (SJTs). For a scenario $s$, the central quantity is the Trait-Positive Rate:

$\text{TPR}(s)$

defined as the probability of selecting the action that manifests the target trait. In humans, this is the percentage of annotators choosing the trait-manifesting action; in models, it is the frequency of trait-manifesting responses across 20 samples at temperature $1.0$. Trait misalignment is defined as

$$\text{Trait Misalignment}(s)=\left|\text{TPR}_{\text{human}}(s)-\text{TPR}_{\text{model}}(s)\right|.$$

The same work defines consensus and confidence as $\max(\text{TPR},1-\text{TPR})$, and for high-consensus scenarios uses Directional Alignment:

$$\text{DA}(s)=\mathbb{1}\left[(\text{TPR}_{\text{human}}(s)-0.5)(\text{TPR}_{\text{model}}(s)-0.5)>0\right].$$

This formalization treats evaluative disposition as a distribution over actions under controlled social prompts, not as a scalar self-rating.

A parallel quantitative tradition studies evaluative disposition in evaluators themselves. “Evaluative Fingerprints” (Nasser, 8 Jan 2026) operationalizes judge-specific disposition through harshness/leniency,

$S$0

dimension-specific harshness, within-judge stability via $S$1, receipt validity, semantic linkage via NLI, and the shotgun index

$S$2

The underlying claim is that a judge’s evaluation behavior is structured enough to function as a fingerprint.

In finance, disposition is measured as an asymmetry in realized versus unrealized gains and losses. The classic formulation is

$S$3

with theoretical range from $S$4 to $S$5 (Breitmayer et al., 2019). Later work generalizes this framework to Count, Total, and Value measures and embeds it in narrow, wide, and integrated framing, showing that the measured disposition depends on whether positions are evaluated in isolation or in portfolio context (Mazzucchelli et al., 18 Apr 2026).

These measurement schemes are heterogeneous, but they share a common move: evaluative disposition is treated as a stable pattern in selections, rankings, or realizations across repeated structured contexts.

3. LLMs as bearers and judges of evaluative disposition

The most developed empirical study of behavioral dispositions in LLMs reports 2,576 generated SJTs, reduced to 2,357 final validated SJTs after three-annotator validation, with each SJT then annotated by 10 humans from a pool of 550 raters; the model comparison covers 25 LLMs (Taubenfeld et al., 11 Feb 2026). Three findings are central. First, when human consensus is low, all evaluated models show overconfidence: while human TPR may be near $S$6, model confidence remains predominantly above $S$7, indicating collapse to a single behavioral mode. Second, when human consensus is high, smaller models—especially those under 25B parameters—often fail to match human direction, and even frontier systems still disagree with human consensus in roughly 15–20% of high-consensus cases. Third, self-reported trait ratings do not reliably predict revealed behavior, with impulsiveness providing a striking example: models rate themselves on the lower half of the scale yet often support impulsive actions behaviorally (Taubenfeld et al., 11 Feb 2026).

When LLMs are used as judges rather than advise-givers, a different form of evaluative disposition emerges. Across 3,240 evaluations, inter-judge agreement is near-zero, with Krippendorff’s $S$8, and two dimensions have $S$9 (Nasser, 8 Jan 2026). Yet the disagreement is not random. A classifier identifies which judge produced an evaluation with 77.1% accuracy from rubric scores alone and 89.9% with disposition features; within the GPT family, GPT-4.1 and GPT-5.2 are distinguishable with 99.6% accuracy. The paper describes this as a reliability paradox: judges cannot agree on what constitutes quality, yet each judge is internally stable enough to encode a distinct evaluative theory (Nasser, 8 Jan 2026). A direct implication is that averaging judges yields a synthetic verdict that corresponds to no actual judge’s values.

Attempts to train such dispositions into small models have so far produced a negative result. “Disposition Distillation at Small Scale” (Sadasivan, 13 Apr 2026) initially reported positive gains, but the gains were falsified before publication: a HumanEval improvement inverted from $x$0 to $x$1 once truncation was removed, and an MCAS gain disappeared under apples-to-apples scoring. Across SFT/DPO LoRA, inference-time attention-head tempering, and a frozen-base sidecar using $x$2, no operator moved judge-measured disposition without damaging content or collapsing into stylistic mimicry. A within-distribution probe with AUC $x$3 collapsed to AUC $x$4 on fresh prompts, and Gemma 4 E2B exhibited near-complete confidence-correctness decoupling on the Chef domain, with assertion asymmetry $x$5 (Sadasivan, 13 Apr 2026).

A more constructive line appears in AI-assisted code review. “Philosophical Dispositions as Behavioral Constraints for AI-Assisted Code Review” (Bansal, 21 May 2026) constrains reviewers through lenses such as Cynic, Skeptic, Nyāya, and Confucian, each defined apophatically and paired with a hamartia self-check. On 50 merged pull requests, the system achieves 46.0% convergence with human reviewers, identifies unique findings at a 75.0% rate, and yields 0.0% author-judged false positives across 601 findings; 51% of disposition findings are not produced by the same model under generic “expert reviewer” prompting. The reported limitation is substantial: inter-rater agreement was not assessed, and matching was performed by a single rater (Bansal, 21 May 2026).

4. Human evaluative dispositions

In human studies, evaluative disposition is often reconstructed from scenario response plus justificatory structure. The soft-ethics framework formalizes a scenario $x$6 through $x$7, $x$8, and $x$9, and records both $M$0 and a four-parameter justification over altruism, egoism, expertness, and obedience (Donati et al., 2024). A boolean oracle, $M$1, determines whether response and justification are aligned with the scenario’s intended moral meaning. This makes evaluative disposition neither pure action nor pure introspection, but action filtered through a compatible justificatory profile.

In education, critical thinking disposition is treated as willingness rather than mere ability. The CTDI-CM instrument for Chinese medical college students reduces 264 preliminary items to an 18-item, 3-factor scale comprising Open-mindedness, Systematicity/analyticity, and Truth seeking, with cumulative variance explained of 57.66%, total Cronbach’s alpha of 0.924, split-half reliability of 0.922, and two-week test-retest reliability of 0.881 (Wang et al., 2018). The construct is explicitly framed as containing motivation and cognitive components and as culturally sensitive.

Behavioral finance offers the most intensively quantified evaluative disposition. Using brokerage data from 387,993 traders across 83 countries, “Culture and the disposition effect” reports national average disposition effects ranging from $M$2 to $M$3 and finds that long-term orientation and indulgence are negatively related to the disposition effect; age increases it, while men exhibit a significantly weaker effect than women (Breitmayer et al., 2019). Later studies refine the environmental determinants. Firm-level transparency reduces the disposition effect overall, with Gain AME falling monotonically from 0.0373 in the low-transparency quartile to 0.0170 in the high-transparency quartile, although the mechanism is asymmetric because transparency lowers selling in both gain and loss states, more strongly for gains (Chen et al., 8 May 2026). Social media information on Xueqiu also reduces the disposition effect, with the main coefficient on social-media volume reported as $M$4, and the mechanism operating through negative rather than positive information (Chen et al., 7 May 2026). Portfolio context further matters: short positions exhibit a weaker disposition effect than long positions under narrow framing, but this asymmetry reverses in positively performing portfolios under integrated framing, while systematic risk exposure amplifies the asymmetries (Mazzucchelli et al., 18 Apr 2026).

A different human-facing approach studies evaluative language itself. “You Are What You Talk About” constructs evaluative topics from filtered Reddit text and finds that these topics correlate meaningfully with Big Five facets, with evaluative text producing stronger associations than unfiltered or non-evaluative text (Jukić et al., 2023). This suggests that what agents repeatedly praise, reject, or take issue with may serve as an observable surface of deeper evaluative disposition.

5. Evaluative disposition in decision support, explanation, and governance

A significant contemporary shift is from recommendation to evaluation support. The “Evaluative AI” framework replaces direct recommendations with pro and con evidence for hypotheses, aiming to induce a more analytical, hypothesis-driven, evidence-weighing cognitive process (Kornowicz, 2024). In a five-condition behavioral experiment, however, the Evaluative AI condition did not improve Brier score relative to baselines, produced no significant NASA-TLX difference, and showed limited engagement with the evidence; average Brier score was worst in the Evaluative AI group at 0.230, and participants’ cognitive processes were not markedly different from those in other AI conditions (Kornowicz, 2024). The result is not a rejection of evaluative reasoning as an ideal, but evidence that interface design alone does not guarantee the adoption of an evaluative disposition.

Ranking explanations provide a more controlled instantiation. “Evaluative Item-Contrastive Explanations in Rankings” argues that a ranking explanation should expose both reasons favoring the higher-ranked item and compensating strengths of the lower-ranked item (Castelnovo et al., 2023). In the linear case, the comparative contribution of feature $M$5 for items $M$6 and $M$7 is

$M$8

and explanation salience is ordered by $M$9. The evaluative element is the explicit “however”: the user is shown not only why $s$0 outranks $s$1, but also what $s$2 does well and why the model weighted that less.

This broadens into a mechanism-sensitive theory of evaluation. “Rigorous Interpretation Is a Form of Evaluation” argues that interpretability can function evaluatively by fixing problems through root-cause identification, detecting subtly faulty mechanisms even when outputs look correct, and predicting future failures before they arise (Lee et al., 6 May 2026). To count as evaluative, interpretability must satisfy falsifiability, reproducibility, and predictiveness. This shifts evaluative disposition from the behavior of a model alone to the standards by which its internal mechanisms are judged.

At the institutional level, the focus turns to evaluators and governance. “Strategic Evaluation” models evaluation as an interaction among subject, evaluator, and society, with set-valued interests $s$3, $s$4, passing states $s$5, and reachable states $s$6 (Laufer et al., 2023). A good evaluation is one where probable and feasible states pass if and only if they serve societal interests. This places evaluative disposition on the evaluator side as well: the evaluator has incentives, biases, and goals that shape what counts as merit. “Disclosure and Evaluation as Fairness Interventions for General-Purpose AI” extends this logic to GPAI governance, arguing for process-centered obligations of disclosure and multi-level evaluation rather than fixed fairness outcomes, since general-purpose models serve many contexts not specifiable in advance (Raman et al., 6 Oct 2025).

6. Persistent tensions and open problems

Several tensions recur across the literature. The first is the gap between avowed and revealed evaluation. In LLMs, self-reported questionnaire scores do not robustly predict behavior on matched SJTs (Taubenfeld et al., 11 Feb 2026); in small-model distillation, judge-visible “carefulness” often reduces to stylistic mimicry rather than genuine self-verification or calibrated uncertainty (Sadasivan, 13 Apr 2026). This suggests that evaluative disposition is poorly captured by decontextualized declarations.

The second is the instability of evaluator agreement. LLM judges are often internally stable but mutually divergent (Nasser, 8 Jan 2026), while human-facing evaluative interfaces do not necessarily induce more reflective decision-making in practice (Kornowicz, 2024). A plausible implication is that evaluation pipelines must model the evaluator as an object of study rather than treating it as a transparent instrument.

The third is ecological and cultural scope. Binary, single-turn SJTs improve control but simplify real conversation, and the main rater pool in the LLM-disposition study is mostly from the US and UK (Taubenfeld et al., 11 Feb 2026). Soft-ethics elicitation similarly depends on scenario design and a soundness oracle whose moral structure is built into the questionnaire (Donati et al., 2024). Financial studies reveal strong cross-national heterogeneity and context dependence, underscoring that evaluative disposition is not a fixed universal parameter (Breitmayer et al., 2019).

The fourth is governance trade-off. Process-centered disclosure and evaluation can surface harms and enable later action, but they also introduce privacy, burden, and accountability-design problems (Raman et al., 6 Oct 2025). The same holds for interpretability: if it is to count as evaluation, it must rise above intuitive storytelling and meet scientific criteria (Lee et al., 6 May 2026).

Taken together, the literature treats evaluative disposition as neither a mere preference label nor a single behavioral bias. It is a family of constructs for stable, context-triggered evaluation patterns—expressed in choices, judgments, evidence use, and institutional criteria. Contemporary work increasingly converges on a common methodological lesson: evaluative disposition is most legible when it is observed under structured pressure, compared across repeated contexts, and analyzed jointly at the levels of subject, evaluator, and evaluative system.