Theory Trace Card in LLM Evaluation

• Theory Trace Card (TTC) is a formal artifact that maps socio-cognitive theories to specific LLM evaluation metrics, clarifying the validity chain.
• It defines a canonical tuple that includes theoretical framework, exercised components, operational tasks, and interpretative rules for accurate measurement.
• TTC enhances interpretability, construct validity, and comparability in LLM assessments while preventing overgeneralization from narrow benchmark metrics.

The term "Theory Trace Card" (TTC) refers to a formal documentation artifact introduced in socio-cognitive evaluation of LLMs to address foundational validity failures in existing benchmarking practices. In a distinct, unrelated context, "t-traceability codes" (also abbreviated as t-TTC) denote a class of combinatorial codes for traitor tracing. Despite the overlap in abbreviation, the following entry focuses on the TTC as applied to theory-driven evaluation of LLMs, outlining its definition, necessity, construction, and implications for research validity (Karimi-Malekabadi et al., 5 Jan 2026).

1. Theoretical Foundations and Motivation

Theory Trace Card (TTC) emerges from the observation of a systemic and under-diagnosed "theory gap" in socio-cognitive evaluation: the lack of explicit mapping between a targeted socio-cognitive theory and the operational measurement used in model assessment. For any socio-cognitive capability $T$ (e.g., Theory of Mind) characterized by a set of components $C = \{c_1, \ldots, c_n\}$, measurement proceeds via task domains $X$, raw system outputs $S$, and scoring functions $\sigma : S \to \mathbb{R}$. Standard benchmarks often omit the explicit operationalization mapping $f: \Theta \to M$—the link between which components of $C$ are exercised by $X$ and how $\sigma$ is interpreted as evidence for those components. This omission fractures the validity chain and generates a "validity illusion," wherein high benchmark performance is misinterpreted as evidence for broad competence, even if only a narrow slice of the underlying construct is tested (Karimi-Malekabadi et al., 5 Jan 2026).

2. Formalization of the Theory Trace Card

TTC prescribes a canonical tuple-based artifact:

$\mathrm{TTC} = \langle \Theta, C_e, O, I \rangle$

where:

• $\Theta = (T, C)$ represents the theoretical framework and its core components.
• $C_e \subseteq C$ specifies which components the evaluation exercises.
• $O = (X, \sigma)$ details the operationalization: dataset or prompt schema ($X$) and explicit scoring function ($\sigma$).
• $I$ encodes interpretation rules and explicitly enumerated limitations: the inferential mapping $g:\sigma(S)\to$ claims about $C_e$ and the boundaries of this interpretation.

This structure makes the theoretical commitments underlying a socio-cognitive benchmark transparent, enabling precise validation claims and principled diagnosis of score meaning (Karimi-Malekabadi et al., 5 Jan 2026).

3. Construction Workflow and Validity Chain

Constructing a TTC proceeds via five sequential stages:

1. Theory Selection ($\Theta$): Explicit citation and enumeration of the theoretical capability and its subcomponents (e.g., for empathy: affective sharing, perspective taking, self–other distinction, emotion regulation).
2. Component Selection ($C_e$): Delineation of which subset of $C$ the operationalization probes, justified by evaluation aims.
3. Task Operationalization ($O$): Definition of $X$ (dataset/prompt) and $\sigma$ (scoring rubric), such that $X\to S$ (model outputs), $\sigma: S\to [0,1]$.
4. Inference Specification ($I$): Rule $g$ mapping observed scores $\sigma(S)$ to substantive claims about model competence in $C_e$, with decision thresholds made explicit.
5. Limitations Enumeration: Systematic articulation of omitted components ($C\setminus C_e$), task-derived limitations (cultural, contextual, representational), and inferential uncertainties.

Graphically, the validity chain implemented by a TTC is realized as:

$$\Theta.c_i \xrightarrow{f_\mathrm{op}} X~\mathrm{(task)} \xrightarrow{\text{model}} S~\mathrm{(output)} \xrightarrow{\sigma}~\text{metric} \xrightarrow{g}~\text{claim on } c_i$$

with boundary conditions and all mappings declared up front (Karimi-Malekabadi et al., 5 Jan 2026).

4. Illustrative Example: False-Belief Theory of Mind (ToM)

Consider a TTC for the evaluation of "belief reasoning" (a component of ToM):

Step	Instantiation	Explanation
1. Theory ($\Theta$)	ToM per Wimmer & Perner (1983): $C=$ cognitive, affective ToM	Establishes conceptual domain and key sub-capabilities
2. $C_e$	$\{$ cognitive ToM (belief reasoning) $\}$	Narrows evaluation to single component
3. $O$	X: “Smarties”, “Sally–Anne” stories;<br>$\sigma$: accuracy	Classic vignettes, closed-set belief prediction accuracy
4. $I$	$\mathrm{accuracy} \geq 80\% \Rightarrow$ belief-tracking	Defines competence threshold and inference rule
Limitations	excludes affective ToM, lacks nonverbal/cultural context	Explicitly restricts generalization beyond tested components

The explicit mapping clarifies both the claims justified by the score and the boundaries where validity does not extend (Karimi-Malekabadi et al., 5 Jan 2026).

5. Practical Roles: Interpretability, Validity, and Reuse

TTCs address three core deficits in standard benchmarking:

• Interpretability: By tracing the full validity chain, TTCs expose the precise theoretical assumptions embedded in each benchmark and make clear what a model’s score does and does not demonstrate.
• Construct Validity: By linking each capability component $c_i$ directly to its operationalization and scoring rule, TTCs block overgeneralization from narrow task success to broad claims of competence.
• Reuse and Comparability: Standardized specification allows evaluations targeting different $C_e$ but rooted in the same $\Theta$ to be compared directly; existing artifacts can also be situated under alternate theories without redefining tasks (Karimi-Malekabadi et al., 5 Jan 2026).

6. Retrospective Insights and Research Use Cases

Applying TTC analysis retrospectively to canonical socio-cognitive LLM benchmarks—including EmpatheticDialogues, GoEmotions, SocialIQA, ETHICS, and ToM—reveals previously hidden assumptions about what capability dimensions are and are not exercised, and spurs principled refinement in task and claim design. The TTC centers the structured validity argument, rather than leaderboard scores, as the primary evaluation output (Karimi-Malekabadi et al., 5 Jan 2026).

7. Broader Implications and Distinctions

While "t-traceability codes" (t-TTCs) exist in coding theory as a separate construct with no semantic overlap—focused on tracing colluders via combinatorial code properties (Ge et al., 2016)—the Theory Trace Card is specifically tailored for socio-cognitive model evaluation. The unifying theme is explicit mapping: in one case for traitor identification in digital content, in the other for theoretical transparency and validity assurance in AI evaluation. This indicates a general epistemic value for formally tracing chains of inference in different research domains. A plausible implication is that formalized evaluation artifacts analogous to TTC may prove valuable wherever theory-to-measurement mapping is underspecified.