ACT-FM: Evaluating ACT Fidelity Tool

• ACT-FM is a 25-item scale that quantifies therapist adherence to ACT principles through an aggregate score and four key subscales: consistent stance, open, aware, and engaged.
• It is employed in simulated therapy sessions to compare LLM variants, such as ORPO, SFT, and Instruct, highlighting differences in training methodologies.
• The measure is sensitive to training paradigms and chain-of-thought strategies, with higher fidelity scores correlating with greater therapeutic empathy.

The ACT Fidelity Measure (ACT-FM) is a 25-item scale developed by expert consensus to measure therapist adherence to ACT principles. In the study of LLM-delivered Acceptance and Commitment Therapy reported in "The Thinking Therapist: Training LLMs to Deliver Acceptance and Commitment Therapy using Supervised Fine-Tuning and Odds Ratio Policy Optimization", the ACT-FM served as the primary quantitative tool for assessing how faithfully different post-trained language-model variants enacted ACT in simulated therapy sessions (Tahir, 8 Sep 2025). Within that setting, the measure was used not merely as a descriptive checklist but as the central comparative endpoint for evaluating post-training methodology, explicit chain-of-thought reasoning, and model-to-model differences in therapeutic process.

1. Construct and scope

The study characterizes the ACT-FM as a fidelity instrument aimed at therapist adherence to ACT principles. Its primary analytic target was the ACT-FM total consistency score, defined as the sum of four key subscales: consistent stance, open, aware, and engaged. Higher scores indicate better adherence to ACT principles. In addition to this aggregate measure, the study reports that secondary analyses examined the eight ACT-FM subscales individually (Tahir, 8 Sep 2025).

In this usage, the ACT-FM functions as a process-sensitive measure rather than a symptom outcome measure. The emphasis on consistency subscales indicates that the instrument was treated as a way of quantifying whether a therapist response conforms to the stance and procedural commitments associated with ACT. A plausible implication is that, in the study design, fidelity was operationalized as a measurable property of generated dialogue rather than of long-term clinical outcome.

2. Role in the evaluation framework

The ACT-FM was employed as the primary quantitative tool for comparing six model variants derived from Llama-3.2-3b-Instruct: Instruct (COT), Instruct (no COT), SFT (COT), SFT (no COT), ORPO (COT), and ORPO (no COT). These variants were evaluated in simulated therapy sessions, and each therapist-patient transcript was scored by an LLM judge (Mistral-Large, fine-tuned on human-evaluated data) (Tahir, 8 Sep 2025).

The study therefore embeds the ACT-FM in a fully model-mediated evaluation loop: the therapist is an LLM, the patient is simulated, and the rater is also an LLM calibrated on human-scored material. This design made the ACT-FM the principal instrument for distinguishing the effects of supervised fine-tuning (SFT), odds ratio policy optimization (ORPO), and explicit chain-of-thought (COT) prompting under otherwise comparable conditions.

3. Scoring procedure and statistical treatment

The scoring workflow consisted of generating a therapist-patient session, submitting the transcript to the fine-tuned LLM judge, and obtaining ACT-FM ratings for comparative analysis. The paper reports the principal omnibus result for model differences on the ACT-FM as:

$\chi^2(5) = 185.15,\ p < .001$

This test was derived from linear mixed-effects modeling. The study further reports that model type explained 14.8\% of variance as marginal $R^2$, while total conditional $R^2$ including patient random effects was 28.1\%. The Intraclass Correlation Coefficient (ICC) for patient profile as a random effect was 0.16 (95\% CI [0.11, 0.20]) (Tahir, 8 Sep 2025).

These quantities indicate that the ACT-FM was analyzed not only as a raw score but also within a hierarchical model that explicitly accounted for between-patient variability. This suggests that the study treated fidelity as jointly shaped by model variant and simulated patient profile rather than as a simple fixed-effect comparison.

4. Comparative performance across model variants

The ACT-FM sharply differentiated the six model variants. The reported descriptive statistics for ACT-FM Total (Mean, SD), with $N = 150$ simulated therapy sessions per variant, are as follows (Tahir, 8 Sep 2025):

Model variant	ACT-FM Total Mean (SD)	Δ vs Instruct (COT)
Instruct (COT)	26.87 (6.12)	Reference
Instruct (no COT)	26.26 (5.84)	-0.62
SFT (COT)	24.79 (6.55)	-2.08
SFT (no COT)	22.12 (8.00)	-4.75
ORPO (COT)	29.56 (5.46)	+2.69
ORPO (no COT)	29.48 (5.02)	+2.61

The corresponding mixed-effects estimates against Instruct (COT) were:

• Instruct (no COT): 95\% CI [-1.91, 0.68], Hedges' $g = -0.11$, $p = .579$
• SFT (COT): 95\% CI [-3.37, -0.78], Hedges' $g = -0.36$, $p = .007$
• SFT (no COT): 95\% CI [-6.04, -3.46], Hedges' $g = -0.83$, $p < .001$
• ORPO (COT): 95\% CI [1.40, 3.98], Hedges' $R^2$0, $R^2$1
• ORPO (no COT): 95\% CI [1.32, 3.90], Hedges' $R^2$2, $R^2$3

The principal empirical pattern is unambiguous: ORPO models outperformed both SFT and Instruct counterparts on ACT-FM, regardless of COT, whereas SFT models showed the lowest fidelity, with SFT (no COT) performing worst overall.

5. Sensitivity to training paradigm and chain-of-thought

A central finding is that the ACT-FM was sufficiently sensitive to detect an interaction-like dependence between training paradigm and explicit reasoning. For SFT models, adding COT improved ACT-FM by +2.68 points with $R^2$4. By contrast, for ORPO models, there was no significant difference between COT and non-COT variants ($R^2$5; mean difference = 0.08), and for the Instruct model there was likewise no significant difference with/without COT ($R^2$6) (Tahir, 8 Sep 2025).

The study also reports direct session-level comparisons. ORPO (COT) beat SFT (no COT) in 76.7\% of sessions ($R^2$7), and ORPO (COT) beat Instruct (COT) in 60.0\% of cases ($R^2$8). The authors posit that the superiority of ORPO stems from its ability to learn the therapeutic “process” over imitating “content,” while COT acts as a necessary scaffold for models trained only via imitation. That interpretation is explicitly framed as a theoretical account of why the ACT-FM favored ORPO-trained models.

6. Relationship to empathy and interpretive constraints

The ACT-FM did not operate in isolation. The study reports that ACT-FM consistency subscales were strongly correlated with therapeutic empathy (TES scores), with positive correlations of $R^2$9 to $R^2$0, and negative correlation with inconsistency subscales (Tahir, 8 Sep 2025). This indicates that, within the evaluation regime, higher ACT fidelity co-occurred with higher judged therapeutic empathy rather than trading off against it.

At the same time, the paper identifies several limitations that constrain interpretation of ACT-FM results in this context. First, ratings depended on an LLM judge fine-tuned on synthetic/human data, and the paper notes that such a judge may lack the full nuance and contextual understanding of a human rater, creating the possibility of inflated method variance. Second, the evaluation used a single-session format, whereas real ACT fidelity is ideally measured longitudinally. Third, the study took place in a simulation environment in which both therapist and client agents were simulated, and the ACT-FM, although psychometrically validated, was never designed for LLM-to-LLM interactions, which raises questions about generalizability to real human therapy. Fourth, the study used only 50 synthetic transcript sets and a small model (3B parameters), so the findings may not fully generalize. These limitations bear directly on a common overinterpretation: high ACT-FM scores in simulated, model-rated sessions should not be conflated with demonstrated clinical effectiveness in real-world care.

7. Significance within the study

Within the study’s experimental logic, the ACT-FM was the decisive measure for showing that preference-aligned policy optimization can effectively instill ACT competencies in small LLMs, and that the utility of explicit reasoning is highly dependent on the underlying training paradigm (Tahir, 8 Sep 2025). Its importance lies in three specific functions documented by the results: it provided a quantitatively tractable representation of ACT adherence, it separated the effects of ORPO, SFT, and Instruct baselines with clear effect sizes and confidence intervals, and it exposed the conditional value of COT.

The broader methodological significance is narrower than a claim of clinical validation. The ACT-FM proved effective at differentiating model behavior under controlled simulation, but the paper’s own discussion emphasizes that this is an evaluation of ACT-consistent behavior as scored in an AI-mediated framework, not a direct demonstration of psychotherapy quality in human practice. In that sense, the study establishes the ACT-FM as a sensitive benchmark for this particular class of LLM therapy experiments while also delineating the boundary conditions under which its scores should be interpreted.