---
title: 'ACT-FM: Evaluating ACT Fidelity Tool'
url: https://www.emergentmind.com/topics/act-fidelity-measure-act-fm
type: topic
---

# ACT-FM: Evaluating ACT Fidelity Tool

The **ACT Fidelity Measure (ACT-FM)** is a **25-item scale developed by expert consensus to measure therapist adherence to ACT principles**. In the study of LLM-delivered Acceptance and Commitment Therapy reported in *"The Thinking Therapist: Training Large Language Models to Deliver Acceptance and Commitment Therapy using Supervised Fine-Tuning and Odds Ratio Policy Optimization"*, the ACT-FM served as the **primary quantitative tool** for assessing how faithfully different post-trained language-model variants enacted ACT in simulated therapy sessions [2509.09712]. Within that setting, the measure was used not merely as a descriptive checklist but as the central comparative endpoint for evaluating post-training methodology, explicit chain-of-thought reasoning, and model-to-model differences in therapeutic process.

## 1. Construct and scope

The study characterizes the ACT-FM as a fidelity instrument aimed at **therapist adherence to ACT principles**. Its primary analytic target was the **ACT-FM total consistency score**, defined as the sum of four key subscales: **consistent stance**, **open**, **aware**, and **engaged**. Higher scores indicate better adherence to ACT principles. In addition to this aggregate measure, the study reports that **secondary analyses examined the eight ACT-FM subscales individually** [2509.09712].

In this usage, the ACT-FM functions as a process-sensitive measure rather than a symptom outcome measure. The emphasis on **consistency** subscales indicates that the instrument was treated as a way of quantifying whether a therapist response conforms to the stance and procedural commitments associated with ACT. A plausible implication is that, in the study design, fidelity was operationalized as a measurable property of generated dialogue rather than of long-term clinical outcome.

## 2. Role in the evaluation framework

The ACT-FM was employed as the **primary quantitative tool** for comparing six model variants derived from **Llama-3.2-3b-Instruct**: **Instruct (COT)**, **Instruct (no COT)**, **SFT (COT)**, **SFT (no COT)**, **ORPO (COT)**, and **ORPO (no COT)**. These variants were evaluated in **simulated therapy sessions**, and each therapist-patient transcript was scored by an **LLM judge (Mistral-Large, fine-tuned on human-evaluated data)** [2509.09712].

The study therefore embeds the ACT-FM in a fully model-mediated evaluation loop: the therapist is an LLM, the patient is simulated, and the rater is also an LLM calibrated on human-scored material. This design made the ACT-FM the principal instrument for distinguishing the effects of **supervised fine-tuning (SFT)**, **odds ratio policy optimization (ORPO)**, and explicit **chain-of-thought (COT)** prompting under otherwise comparable conditions.

## 3. Scoring procedure and statistical treatment

The scoring workflow consisted of generating a therapist-patient session, submitting the transcript to the fine-tuned LLM judge, and obtaining ACT-FM ratings for comparative analysis. The paper reports the principal omnibus result for model differences on the ACT-FM as:

$$
\chi^2(5) = 185.15,\ p < .001
$$

This test was derived from **linear mixed-effects modeling**. The study further reports that **model type explained 14.8\% of variance** as **marginal $R^2$**, while **total conditional $R^2$ including patient random effects was 28.1\%**. The **Intraclass Correlation Coefficient (ICC)** for **patient profile as a random effect** was **0.16 (95\% CI [0.11, 0.20])** [2509.09712].

These quantities indicate that the ACT-FM was analyzed not only as a raw score but also within a hierarchical model that explicitly accounted for between-patient variability. This suggests that the study treated fidelity as jointly shaped by model variant and simulated patient profile rather than as a simple fixed-effect comparison.

## 4. Comparative performance across model variants

The ACT-FM sharply differentiated the six model variants. The reported descriptive statistics for **ACT-FM Total (Mean, SD)**, with **$N = 150$ simulated therapy sessions per variant**, are as follows [2509.09712]:

| Model variant | ACT-FM Total Mean (SD) | Δ vs Instruct (COT) |
|---|---:|---:|
| Instruct (COT) | 26.87 (6.12) | Reference |
| Instruct (no COT) | 26.26 (5.84) | -0.62 |
| SFT (COT) | 24.79 (6.55) | -2.08 |
| SFT (no COT) | 22.12 (8.00) | -4.75 |
| ORPO (COT) | 29.56 (5.46) | +2.69 |
| ORPO (no COT) | 29.48 (5.02) | +2.61 |

The corresponding mixed-effects estimates against **Instruct (COT)** were:

- **Instruct (no COT)**: **95\% CI [-1.91, 0.68]**, **Hedges' $g = -0.11$**, **$p = .579$**
- **SFT (COT)**: **95\% CI [-3.37, -0.78]**, **Hedges' $g = -0.36$**, **$p = .007$**
- **SFT (no COT)**: **95\% CI [-6.04, -3.46]**, **Hedges' $g = -0.83$**, **$p < .001$**
- **ORPO (COT)**: **95\% CI [1.40, 3.98]**, **Hedges' $g = +0.47$**, **$p < .001$**
- **ORPO (no COT)**: **95\% CI [1.32, 3.90]**, **Hedges' $g = +0.46$**, **$p < .001$**

The principal empirical pattern is unambiguous: **ORPO models outperformed both SFT and Instruct counterparts on ACT-FM, regardless of COT**, whereas **SFT models showed the lowest fidelity**, with **SFT (no COT)** performing worst overall.

## 5. Sensitivity to training paradigm and chain-of-thought

A central finding is that the ACT-FM was sufficiently sensitive to detect an interaction-like dependence between **training paradigm** and **explicit reasoning**. For **SFT models**, adding COT improved ACT-FM by **+2.68 points** with **$p < .001$**. By contrast, for **ORPO models**, there was **no significant difference between COT and non-COT variants** (**$p = .899$; mean difference = 0.08**), and for the **Instruct model** there was likewise **no significant difference with/without COT** (**$p = .579$**) [2509.09712].

The study also reports direct session-level comparisons. **ORPO (COT) beat SFT (no COT) in 76.7\% of sessions** (**$p < .001$**), and **ORPO (COT) beat Instruct (COT) in 60.0\% of cases** (**$p = .006$**). The authors posit that the superiority of ORPO stems from its ability to learn the therapeutic **“process”** over imitating **“content,”** while COT acts as a necessary scaffold for models trained only via imitation. That interpretation is explicitly framed as a theoretical account of why the ACT-FM favored ORPO-trained models.

## 6. Relationship to empathy and interpretive constraints

The ACT-FM did not operate in isolation. The study reports that **ACT-FM consistency subscales were strongly correlated with therapeutic empathy (TES scores)**, with **positive correlations of $r = .84$ to $.87$**, and **negative correlation with inconsistency subscales** [2509.09712]. This indicates that, within the evaluation regime, higher ACT fidelity co-occurred with higher judged therapeutic empathy rather than trading off against it.

At the same time, the paper identifies several limitations that constrain interpretation of ACT-FM results in this context. First, **ratings depended on an LLM judge fine-tuned on synthetic/human data**, and the paper notes that such a judge **may lack the full nuance and contextual understanding of a human rater**, creating the possibility of **inflated method variance**. Second, the evaluation used a **single-session format**, whereas **real ACT fidelity is ideally measured longitudinally**. Third, the study took place in a **simulation environment** in which both therapist and client agents were simulated, and the ACT-FM, although **psychometrically validated**, **was never designed for LLM-to-LLM interactions**, which raises questions about generalizability to real human therapy. Fourth, the study used **only 50 synthetic transcript sets** and a **small model (3B parameters)**, so the findings may not fully generalize. These limitations bear directly on a common overinterpretation: high ACT-FM scores in simulated, model-rated sessions should not be conflated with demonstrated clinical effectiveness in real-world care.

## 7. Significance within the study

Within the study’s experimental logic, the ACT-FM was the decisive measure for showing that **preference-aligned policy optimization can effectively instill ACT competencies in small LLMs**, and that the utility of explicit reasoning is **highly dependent on the underlying training paradigm** [2509.09712]. Its importance lies in three specific functions documented by the results: it provided a quantitatively tractable representation of ACT adherence, it separated the effects of **ORPO**, **SFT**, and **Instruct** baselines with clear effect sizes and confidence intervals, and it exposed the conditional value of **COT**.

The broader methodological significance is narrower than a claim of clinical validation. The ACT-FM proved effective at differentiating model behavior under controlled simulation, but the paper’s own discussion emphasizes that this is an evaluation of **ACT-consistent behavior** as scored in an AI-mediated framework, not a direct demonstration of psychotherapy quality in human practice. In that sense, the study establishes the ACT-FM as a sensitive benchmark for this particular class of LLM therapy experiments while also delineating the boundary conditions under which its scores should be interpreted.

Source: https://www.emergentmind.com/topics/act-fidelity-measure-act-fm