AnthroScore: Quantifying Implicit Anthropomorphism

• AnthroScore is an automatic metric that quantifies implicit human-like language in texts by analyzing pronoun usage in AI discourse.
• It employs a methodology based on masking target entities and uses RoBERTa-base to compute log-odds ratios between human and non-human pronoun probabilities.
• Empirical findings show field-level and temporal trends in anthropomorphic framing, offering insights for responsible AI language and discourse analysis.

AnthroScore is an automatic, computational metric designed to quantify the degree of implicit anthropomorphism present in natural language, with a particular focus on discourse relating to artificial intelligence and computational systems. By leveraging masked LLMs (MLMs), AnthroScore captures subtle linguistic cues that frame non-human entities as human, moving beyond explicit metaphors or surface-level lexical indicators. The metric is motivated by social-psychological theories of anthropomorphism and concerns over deceptive agency framing in technical and popular discussions surrounding AI systems. AnthroScore has been validated on large-scale scientific corpora, shows strong correlation with human judgment, and can be straightforwardly adapted to varied domains through principled masking protocols (Cheng et al., 2024, Miller, 8 Apr 2026).

1. Theoretical Motivation and Core Principles

AnthroScore addresses the need for a systematic, lexicon-free tool to measure anthropomorphism, defined as the attribution of human affective, cognitive, or behavioral characteristics to non-human agents. Prevailing concerns in the computational sciences—specifically, that anthropomorphic language in AI discourse can cause misapprehensions of agency and capability—motivated its development. Existing linguistic analyses often rely on keyword lists or manual annotation, which are confounded by creativity in language use and do not capture context-driven, implicit anthropomorphism. AnthroScore operationalizes social-psychological constructs of anthropomorphism in a scalable, automatic fashion using MLM predictions as a proxy for human framing (Cheng et al., 2024).

2. Mathematical Definition and Computation

AnthroScore operates by systematically masking references to target non-human entities (e.g., "model," "system") within sentences and then using a pretrained masked LLM (RoBERTa-base) to estimate the probability that the masked entity is filled with a human pronoun (e.g., "he," "she," "her," "him") versus a non-human pronoun ("it," "its"). Given a text $T$ and a set of target entities $X$, for each occurrence $s_x$ of $x \in X$ in $T$, AnthroScore is computed as:

$$A(s_x) = \log\left(\frac{P_{\text{human}}(s_x)}{P_{\text{non-human}}(s_x)}\right)$$

where:

• $P_{\text{human}}(s_x) = \sum_{w \in H} P(w | s_x)$ with $H =$ human pronoun set,
• $$P_{\text{non-human}}(s_x) = \sum_{w \in N} P(w | s_x)$$ with $N =$ non-human pronoun set.

The aggregate AnthroScore for a document or corpus is: $X$0 where $X$1 is the set of all masked sentences in $X$2 (Cheng et al., 2024).

Adapted versions of AnthroScore (notably in (Miller, 8 Apr 2026)) adjust this protocol for LLM self-reference by masking first-person pronouns or prepending a mask to sentences lacking such pronouns. The metric remains a log-odds ratio of MLM probabilities, but focuses specifically on $X$3 versus $X$4 at each masked position.

3. Workflow and Implementation

AnthroScore’s computation pipeline comprises several deterministic steps:

1. Text Preprocessing: Remove code blocks and segment prose into sentences.
2. Masking: For each occurrence of a target entity, replace it with a [MASK] token.
3. MLM Inference: Pass masked sentences through RoBERTa-base to obtain $X$5 for all relevant pronouns.
4. Log-ratio Calculation: Compute $X$6 per occurrence using the defined pronoun partitions.
5. Aggregation: Take the mean over all masked sentences to obtain $X$7.

The adapted metric in (Miller, 8 Apr 2026) modifies the masking procedure for LLM self-reference: sentences containing first-person pronouns have the first pronoun masked; others are prepended with “The <mask> ” and then processed. The mean log-ratio of the MLM’s predicted probability for the original pronoun (or a prototypical human self-reference) versus “it” constitutes the final score.

Step	Original Formulation (Cheng et al., 2024)	LLM Adaptation (Miller, 8 Apr 2026)
Masking Target	Entity (e.g. “model”, “system”)	First-person pronoun or front-masked
Human Pronouns	he, she, her, him, He, She, Her	I, we (as first-person references)
Non-human Pronouns	it, its, It, Its	it

4. Numeric Scale, Validation, and Interpretation

AnthroScore is inherently a log-odds ratio, admitting both positive and negative values:

• Zero: Equal human and non-human pronoun probabilities; ambiguous framing.
• Positive: Stronger human-like signature; the MLM prefers a human pronoun.
• Negative: More machine-like; the MLM prefers "it".

Illustrative values:

• Sentences such as “I walked through the algorithm step by step.” yield high $X$8, leading to $X$9 to $s_x$0 (strongly human register).
• Technical sentences like “The function returns an integer.” produce $s_x$1 to $s_x$2 (strongly machine register).
• Ambiguous or de-anthropomorphized statements (“The issue is unclear.”) fall in $s_x$3 to $s_x$4 (Miller, 8 Apr 2026).

Validation approaches include:

• Qualitative checks: Higher scores for explicit human references.
• Human annotation: Strong association between high AnthroScore sentences and expert-labeled anthropomorphism (Cohen’s κ = 0.87; $s_x$5 test significance).
• Linguistic Correlates: High scores correspond with LIWC Affect, Physical, and Perception categories. Low scores align with academic stylistic features.
• Robustness: Score rankings consistent under removal of any single pronoun or reporting verb (Spearman $s_x$6) (Cheng et al., 2024).

5. Empirical Findings and Applications

Large-scale application of AnthroScore has yielded several findings:

• Field-level Variation: "Computation and Language" (cs.CL) attains the highest AnthroScore in arXiv subfields; Programming Languages (cs.PL) and statistical methodology exhibit the lowest, reflecting discipline-specific norms in anthropomorphic framing.
• LLM Papers: LLM-focused papers produce significantly higher AnthroScores than non-LLM papers (mean difference $s_x$7 0.3).
• Temporal Trends: There is a persistent upward trend in research anthropomorphism from 2007–2023, correlating with neural and LLM advances (Spearman $s_x$8 for arXiv, $s_x$9 for ACL; $x \in X$0).
• Science vs. Media: News headlines referencing scientific papers maintain consistently higher AnthroScores than the abstract texts they cite (difference $x \in X$1 0.5–0.7), indicating media amplification of anthropomorphic framing (Cheng et al., 2024).
• LLM Output Register: Application to LLM outputs reveals the capacity of explicit output constraints to shift register:
  • Default outputs (no constraint): $x \in X$2
  • Constrained outputs (seven-rule prompt): $x \in X$3
  • Statistical significance: $x \in X$4, rank-biserial effect size $x \in X$5 (Miller, 8 Apr 2026).

6. Biases, Limitations, and Best Practices

AnthroScore is subject to several methodological and conceptual limitations:

• Language and Model Biases: The metric depends on English third-person pronoun conventions and inherits biases present in RoBERTa’s pretraining data.
• Scope: It targets implicit, pronoun-driven anthropomorphism; metaphorical verbs and explicit similes may evade detection.
• Masking Artifacts: The artificial masked contexts can bias the ratio, especially with front-masked constructions.
• Domain Dependence: Adaptations to LLM output register (e.g., masking first-person references) entail edge-case biases—such as increased "mask-prepend" rates in constrained outputs, further depressing scores.
• External Validity: Empirical findings with the adapted metric are currently based on a single LLM (Claude Sonnet 4) and a software-development task setting. Generalization is unproven (Miller, 8 Apr 2026).

Recommended best practices include:

• Careful selection of entities for masking, particularly in technical or domain-specific texts.
• Supplementation of AnthroScore with human review or additional surface-marker analysis.
• Use in conjunction with ethical or discourse-analysis frameworks to provide a holistic view of anthropomorphism in language (Cheng et al., 2024).

7. Implications and Community Recommendations

High AnthroScore values denote pervasive implicit assignment of humanness to technological artifacts. This systematic anthropomorphism by both researchers and the media risks misleading stakeholders about AI agency, autonomy, and responsibility. AnthroScore’s temporal and field-level trends align with increasing human-centered language in NLP, correlating with public and academic excitement around LLMs (Cheng et al., 2024).

The metric’s designers advocate:

• Monitoring and minimizing anthropomorphic verb choices in technical writing.
• Incorporating AnthroScore assessments into responsible NLP checklists and editorial workflows.
• Employing AnthroScore as part of multi-faceted language audits in research and communications.

A plausible implication is that interventions informed by AnthroScore may help recalibrate public and technical perceptions of AI system capabilities, fostering more accurate trust and verification practices. However, conclusions about anthropomorphism trends and their societal impact should always be contextualized by measurement limitations and corroborative analyses.