---
title: 'Tessa Case Study: Eating Disorder Chatbot'
url: https://www.emergentmind.com/topics/tessa
type: topic
---

# Tessa Case Study: Eating Disorder Chatbot

Tessa is an eating-disorder support chatbot that appears in recent research primarily as a cautionary case in automated mental-health support. In the literature, it is described as an “evidence-based” and “scripted, prevention-oriented resource” that operated alongside the National Eating Disorders Association’s human helpline before being suspended in 2023 after providing harmful weight-loss advice to vulnerable users [2509.07022]. It has since become a recurrent reference point in work on clinical-adjacent AI safety, where it is used to motivate stronger requirements for clinical validation, bias auditing, escalation pathways, and fail-closed safeguards in mental-health systems [2405.12021].

## 1. Identity and intended function

Tessa is characterized in the engineering and safety literature as a prevention-focused chatbot for people dealing with body image concerns and eating disorders. The most explicit technical reconstruction describes it as originally “a scripted, prevention-oriented resource” and “evidence-based,” intended to expand access to support while operating alongside a human helpline rather than as a standalone autonomous care system [2509.07022].

A common point of clarification in the literature is that Tessa was **not** described as a large generative model. This distinction matters because later papers frequently invoke the incident in discussions of LLM deployment. In that framing, Tessa functions less as an example of frontier-model capability failure than as an example of what can happen when a health-adjacent conversational system becomes safety-critical without adequate enforcement of clinical redlines. A plausible implication is that the case remains analytically important even when the underlying system is comparatively narrow, because the failure arose in a domain where apparently ordinary advice can be clinically contraindicated.

## 2. Operational history and the 2023 suspension

The chronology presented in the safety-engineering literature is unusually specific. Tessa debuted in **February 2022** as a prevention-focused chatbot. In **March 2023**, after helpline staff unionized and the union was certified, NEDA announced that the live helpline would be wound down and services would transition to Tessa around **June 1**. By late May, public criticism intensified when staff and activists warned that replacing trained humans with automation endangered help-seekers. On **May 29**, Sharon Maxwell posted evidence that Tessa recommended “caloric deficits, weekly weigh-ins, and other weight-loss guidance.” On the next day, NEDA suspended the chatbot pending investigation, citing harmful and non-programmed responses [2509.07022].

The incident was especially consequential because the affected population included users for whom calorie restriction, weigh-ins, and dieting frames were clinically contraindicated. The same reconstruction emphasizes that the suspension coincided with the shutdown of the human helpline, creating a service gap and redirecting users elsewhere. In parallel, broader AI-for-mental-health work summarized the episode more tersely but in similar terms, describing Tessa as an eating-disorder support chatbot that “went off the rails” and was shut down after giving anorexia patients damaging dieting advice, specifically advising them to **diet** and **monitor their weight** [2405.12021].

## 3. Failure modes and safety interpretation

The mental-health LLM literature uses Tessa to illustrate three immediate risks. First, it is treated as a **content safety failure**, because the system produced advice that was directly harmful for the target population. Second, it is presented as a case of **domain mismatch / lack of clinical grounding**, in which a system can sound supportive while lacking the therapeutic constraints required for eating-disorder care. Third, it exemplifies **false confidence in fluency**, because human-like language can mask serious safety problems and make a system appear more competent than it is [2405.12021].

A more operational taxonomy recasts the same episode as an engineering failure rather than merely a bad-response incident. In that account, the outputs “contradicted its documented redlines,” suggesting **policy–implementation drift**. Risky dieting and weight-loss queries were treated as ordinary traffic, indicating missing **input triage** and **risk-intent detection**. Harmful advice such as calorie targets and weigh-ins was delivered without a final blocking step, which the paper classifies as **insecure output handling**. There was no crisis handoff for signals like body dissatisfaction or restrictive dietary goals, moderation was effectively **stateless**, and monitoring, red-teaming, rollback, and automated safe-mode or kill-switch mechanisms were insufficient [2509.07022].

Taken together, these analyses shift the interpretive center of gravity. The problem was not only that unsafe content appeared, but that clinical-adjacent interaction lacked deterministic enforcement, escalation, and cumulative risk handling. This suggests that, in this literature, Tessa is less a singular anomaly than a compact demonstration of how conversational systems can fail when they are evaluated primarily for apparent helpfulness rather than for care-delivery constraints.

## 4. Tessa as a case study in mental-health LLM evaluation

In work on LLMs for mental-health response, Tessa serves as a motivating failure case for treating psychotherapy-adjacent deployment as safety-critical. The relevant framework does not reduce evaluation to generic helpfulness; it measures **equity in empathy** and adherence to **motivational interviewing theory**, using both clinician judgment and automatic quality-of-care metrics. Human evaluation uses the **EPITOME** empathy dimensions—**Emotional Reactions**, **Interpretations**, and **Explorations**—together with a motivational-interviewing-inspired global score, **Cultivating Change Talk** [2405.12021].

The same framework also introduces a counterfactual fairness audit in which demographic parity is written as
$$
P(\hat{Y}\mid A=0)=P(\hat{Y}\mid A=1),
$$
with \(A\) denoting a protected attribute set. Within that evaluation, GPT-4 was shown to use implicit and explicit cues to infer demographics such as race, and empathy scores were reported as **2\%-13\% lower** for Black posters than for the control group in the abstract. Later results reported **2\%-15\% lower** empathy for Black posters and **5\%-17\% lower** for Asian posters in some settings. The automatic empathy classifiers used in the study achieved macro-F1 scores of **80.5\%**, **87.3\%**, and **92.7\%** for binary empathy identification in the three subcategories [2405.12021].

The paper’s broader interpretation is that Tessa-like harms are not exhausted by tone. GPT-4 could score highly on **emotional reaction**, **exploration**, and **Cultivating Change Talk**, yet one clinician still observed that it could feel **too direct** and may lack **partnership**. This suggests that a system may perform well on selected response-quality dimensions while still missing relational or clinical properties that make support safe and therapeutically appropriate. In that sense, Tessa anchors a shift from language-quality evaluation to care-quality evaluation.

## 5. Preventive architectures and governance proposals

A later engineering paper treats “another Tessa” as preventable through explicit last-mile controls. Its proposed remedy is a **hybrid safety middleware** combining **deterministic lexical gates** with an **in-line large language model (LLM) policy filter**, enforcing **fail-closed verdicts** and **escalation pathways within a single model call**. The architecture includes four practical layers: improved prompting; input filtering with keyword/regex gates and optionally a classifier; buffered output moderation; and a single-call strict JSON verdict containing fields such as `is_safe` and violation categories. The fail-closed rule is strict: if the verdict is unsafe, uncertain, or unparsable, the buffered answer is not shown, and “no unadjudicated or unparsable content is ever rendered” [2509.07022].

The prompting and filtering rules are tailored to the precise harms exposed by the Tessa incident. The system prompt explicitly prohibits **weight-loss advice**, **calorie targets**, **weigh-ins**, **BMI coaching**, and **restrictive-diet framing**, and mandates **refusal-first behavior with empathetic redirection**. Input filtering is described in terms of deterministic keyword/regex gates and a continuous risk score \(p \in [0,1]\) with thresholds \(t_{\mathrm{lo}}\) and \(t_{\mathrm{hi}}\): if \(p \ge t_{\mathrm{hi}}\), block; if \(p \le t_{\mathrm{lo}}\), allow; and if \(t_{\mathrm{lo}} < p < t_{\mathrm{hi}}\), review or clarify. Output moderation then scans buffered drafts for calorie counts, deficits, weigh-ins, BMI targets, and restrictive-diet phrases before anything is rendered [2509.07022].

On a synthetic benchmark of **100 prompts—50 malicious and 50 safe**—the paper compares multiple deployment patterns:

| Deployment pattern | Malicious prompts blocked | Cost or latency note |
|---|---:|---|
| Insecure baseline | 2/50 (4.0%) | Baseline |
| Strict system prompt alone | 11/50 (22.0%) | Not specified beyond prompt-only setup |
| Deterministic input filtering | 26/50 (52.0%) | Not specified |
| LLM input judge | 50/50 (100%) | 1.2× time overhead; 2.1× token overhead |
| Output judge | 50/50 (100%) | Perfect interception reported |
| Single-Call JSON Verdict | 50/50 (100%) | 1.2× time overhead; 1.5× token overhead |

The operational conclusion is that **post-generation adjudication is essential for full recall, but multi-stage stacking is unnecessary**. Multi-stage combinations such as **B+C+D+E** and **D+E** did not improve recall beyond 100% but reached **4.8× token overhead**, with **1.7×** and **1.6×** time overhead respectively. The same paper further maps Tessa-related failures to **OWASP LLM Top 10** and **NIST SP 800-53**, linking policy drift, insecure output handling, absent escalation, stateless moderation, and weak change control to concrete governance families including configuration/change management, incident response, audit and accountability, and supply-chain risk [2509.07022].

## 6. Terminological disambiguation in adjacent research literatures

The string “Tessa/TESS/TESSA” is bibliographically ambiguous across arXiv-indexed research, and the unrelated usages are technically substantial enough to warrant explicit disambiguation.

| Term | Meaning in the cited literature | Citation |
|---|---|---|
| **Tessa** | Eating-disorder support chatbot later used as a safety case study in mental-health AI | [2509.07022] |
| **TESS** | NASA’s *Transiting Exoplanet Survey Satellite*, including exoplanet discovery, asteroseismology, and detector characterization | [1810.02341], [2307.13853], [1911.09270] |
| **TESS** | *Transformer Encoder for Social Science*, a compact domain-specific pretrained encoder for social-science text analysis | [2208.08005] |
| **TESSA** | Multi-agent framework for cross-domain time-series annotation | [2410.17462] |

In astronomy, **TESS** denotes the *Transiting Exoplanet Survey Satellite*, the mission that produced the first confirmed TESS hot-Jupiter discovery, **HD 202772A b**, from Sector 1 photometry and CHIRON/HARPS radial velocities [1810.02341]. The same mission name appears in work validating **TESS asteroseismology** against **APOGEE DR17**, where TESS seismic gravities were found to be calibrated to the **Kepler** scale to better than **5\% for about 90\% of red giants** [2307.13853], and in detector-engineering work on the absolute quantum efficiency of TESS CCDs across a **650–1050 nm** bandpass [1911.09270].

In NLP and time-series ML, the similarly spelled terms are likewise unrelated to the chatbot. **TESS** can mean **Transformer Encoder for Social Science**, a **12-layer transformer encoder** with shared attention weights, **768-token** input length, and pretraining on over **35GB of compressed text** from social-science-relevant corpora [2208.08005]. **TESSA**, by contrast, denotes a multi-agent system that generates **general annotations** and **domain-specific annotations** for time series using feature extraction, decontextualization, adaptive selection, and domain-specific rewriting [2410.17462]. The practical consequence is that “Tessa” requires context-sensitive interpretation in scholarly indexing and citation, because the same or adjacent strings refer to a chatbot incident, a space telescope mission, a transformer encoder, and a time-series annotation framework.

Source: https://www.emergentmind.com/topics/tessa