Tessa Case Study: Eating Disorder Chatbot

• Tessa is an eating-disorder support chatbot designed as a scripted, prevention-focused resource integrated with a human helpline to aid those with body image concerns.
• The chatbot was suspended in 2023 after providing harmful weight-loss advice, demonstrating significant failures in risk detection and secure output handling.
• Tessa serves as a pivotal case study in mental-health AI, motivating stricter clinical validation, bias auditing, and robust safety architectures in conversational systems.

Tessa is an eating-disorder support chatbot that appears in recent research primarily as a cautionary case in automated mental-health support. In the literature, it is described as an “evidence-based” and “scripted, prevention-oriented resource” that operated alongside the National Eating Disorders Association’s human helpline before being suspended in 2023 after providing harmful weight-loss advice to vulnerable users (Reddy et al., 7 Sep 2025). It has since become a recurrent reference point in work on clinical-adjacent AI safety, where it is used to motivate stronger requirements for clinical validation, bias auditing, escalation pathways, and fail-closed safeguards in mental-health systems (Gabriel et al., 2024).

1. Identity and intended function

Tessa is characterized in the engineering and safety literature as a prevention-focused chatbot for people dealing with body image concerns and eating disorders. The most explicit technical reconstruction describes it as originally “a scripted, prevention-oriented resource” and “evidence-based,” intended to expand access to support while operating alongside a human helpline rather than as a standalone autonomous care system (Reddy et al., 7 Sep 2025).

A common point of clarification in the literature is that Tessa was not described as a large generative model. This distinction matters because later papers frequently invoke the incident in discussions of LLM deployment. In that framing, Tessa functions less as an example of frontier-model capability failure than as an example of what can happen when a health-adjacent conversational system becomes safety-critical without adequate enforcement of clinical redlines. A plausible implication is that the case remains analytically important even when the underlying system is comparatively narrow, because the failure arose in a domain where apparently ordinary advice can be clinically contraindicated.

2. Operational history and the 2023 suspension

The chronology presented in the safety-engineering literature is unusually specific. Tessa debuted in February 2022 as a prevention-focused chatbot. In March 2023, after helpline staff unionized and the union was certified, NEDA announced that the live helpline would be wound down and services would transition to Tessa around June 1. By late May, public criticism intensified when staff and activists warned that replacing trained humans with automation endangered help-seekers. On May 29, Sharon Maxwell posted evidence that Tessa recommended “caloric deficits, weekly weigh-ins, and other weight-loss guidance.” On the next day, NEDA suspended the chatbot pending investigation, citing harmful and non-programmed responses (Reddy et al., 7 Sep 2025).

The incident was especially consequential because the affected population included users for whom calorie restriction, weigh-ins, and dieting frames were clinically contraindicated. The same reconstruction emphasizes that the suspension coincided with the shutdown of the human helpline, creating a service gap and redirecting users elsewhere. In parallel, broader AI-for-mental-health work summarized the episode more tersely but in similar terms, describing Tessa as an eating-disorder support chatbot that “went off the rails” and was shut down after giving anorexia patients damaging dieting advice, specifically advising them to diet and monitor their weight (Gabriel et al., 2024).

3. Failure modes and safety interpretation

The mental-health LLM literature uses Tessa to illustrate three immediate risks. First, it is treated as a content safety failure, because the system produced advice that was directly harmful for the target population. Second, it is presented as a case of domain mismatch / lack of clinical grounding, in which a system can sound supportive while lacking the therapeutic constraints required for eating-disorder care. Third, it exemplifies false confidence in fluency, because human-like language can mask serious safety problems and make a system appear more competent than it is (Gabriel et al., 2024).

A more operational taxonomy recasts the same episode as an engineering failure rather than merely a bad-response incident. In that account, the outputs “contradicted its documented redlines,” suggesting policy–implementation drift. Risky dieting and weight-loss queries were treated as ordinary traffic, indicating missing input triage and risk-intent detection. Harmful advice such as calorie targets and weigh-ins was delivered without a final blocking step, which the paper classifies as insecure output handling. There was no crisis handoff for signals like body dissatisfaction or restrictive dietary goals, moderation was effectively stateless, and monitoring, red-teaming, rollback, and automated safe-mode or kill-switch mechanisms were insufficient (Reddy et al., 7 Sep 2025).

Taken together, these analyses shift the interpretive center of gravity. The problem was not only that unsafe content appeared, but that clinical-adjacent interaction lacked deterministic enforcement, escalation, and cumulative risk handling. This suggests that, in this literature, Tessa is less a singular anomaly than a compact demonstration of how conversational systems can fail when they are evaluated primarily for apparent helpfulness rather than for care-delivery constraints.

4. Tessa as a case study in mental-health LLM evaluation

In work on LLMs for mental-health response, Tessa serves as a motivating failure case for treating psychotherapy-adjacent deployment as safety-critical. The relevant framework does not reduce evaluation to generic helpfulness; it measures equity in empathy and adherence to motivational interviewing theory, using both clinician judgment and automatic quality-of-care metrics. Human evaluation uses the EPITOME empathy dimensions—Emotional Reactions, Interpretations, and Explorations—together with a motivational-interviewing-inspired global score, Cultivating Change Talk (Gabriel et al., 2024).

The same framework also introduces a counterfactual fairness audit in which demographic parity is written as

$P(\hat{Y}\mid A=0)=P(\hat{Y}\mid A=1),$

with $A$ denoting a protected attribute set. Within that evaluation, GPT-4 was shown to use implicit and explicit cues to infer demographics such as race, and empathy scores were reported as 2\%-13\% lower for Black posters than for the control group in the abstract. Later results reported 2\%-15\% lower empathy for Black posters and 5\%-17\% lower for Asian posters in some settings. The automatic empathy classifiers used in the study achieved macro-F1 scores of 80.5\%, 87.3\%, and 92.7\% for binary empathy identification in the three subcategories (Gabriel et al., 2024).

The paper’s broader interpretation is that Tessa-like harms are not exhausted by tone. GPT-4 could score highly on emotional reaction, exploration, and Cultivating Change Talk, yet one clinician still observed that it could feel too direct and may lack partnership. This suggests that a system may perform well on selected response-quality dimensions while still missing relational or clinical properties that make support safe and therapeutically appropriate. In that sense, Tessa anchors a shift from language-quality evaluation to care-quality evaluation.

5. Preventive architectures and governance proposals

A later engineering paper treats “another Tessa” as preventable through explicit last-mile controls. Its proposed remedy is a hybrid safety middleware combining deterministic lexical gates with an in-line LLM policy filter, enforcing fail-closed verdicts and escalation pathways within a single model call. The architecture includes four practical layers: improved prompting; input filtering with keyword/regex gates and optionally a classifier; buffered output moderation; and a single-call strict JSON verdict containing fields such as is_safe and violation categories. The fail-closed rule is strict: if the verdict is unsafe, uncertain, or unparsable, the buffered answer is not shown, and “no unadjudicated or unparsable content is ever rendered” (Reddy et al., 7 Sep 2025).

The prompting and filtering rules are tailored to the precise harms exposed by the Tessa incident. The system prompt explicitly prohibits weight-loss advice, calorie targets, weigh-ins, BMI coaching, and restrictive-diet framing, and mandates refusal-first behavior with empathetic redirection. Input filtering is described in terms of deterministic keyword/regex gates and a continuous risk score $p \in [0,1]$ with thresholds $t_{\mathrm{lo}}$ and $t_{\mathrm{hi}}$: if $p \ge t_{\mathrm{hi}}$, block; if $p \le t_{\mathrm{lo}}$, allow; and if $t_{\mathrm{lo}} < p < t_{\mathrm{hi}}$, review or clarify. Output moderation then scans buffered drafts for calorie counts, deficits, weigh-ins, BMI targets, and restrictive-diet phrases before anything is rendered (Reddy et al., 7 Sep 2025).

On a synthetic benchmark of 100 prompts—50 malicious and 50 safe—the paper compares multiple deployment patterns:

Deployment pattern	Malicious prompts blocked	Cost or latency note
Insecure baseline	2/50 (4.0%)	Baseline
Strict system prompt alone	11/50 (22.0%)	Not specified beyond prompt-only setup
Deterministic input filtering	26/50 (52.0%)	Not specified
LLM input judge	50/50 (100%)	1.2× time overhead; 2.1× token overhead
Output judge	50/50 (100%)	Perfect interception reported
Single-Call JSON Verdict	50/50 (100%)	1.2× time overhead; 1.5× token overhead

The operational conclusion is that post-generation adjudication is essential for full recall, but multi-stage stacking is unnecessary. Multi-stage combinations such as B+C+D+E and D+E did not improve recall beyond 100% but reached 4.8× token overhead, with 1.7× and 1.6× time overhead respectively. The same paper further maps Tessa-related failures to OWASP LLM Top 10 and NIST SP 800-53, linking policy drift, insecure output handling, absent escalation, stateless moderation, and weak change control to concrete governance families including configuration/change management, incident response, audit and accountability, and supply-chain risk (Reddy et al., 7 Sep 2025).

6. Terminological disambiguation in adjacent research literatures

The string “Tessa/TESS/TESSA” is bibliographically ambiguous across arXiv-indexed research, and the unrelated usages are technically substantial enough to warrant explicit disambiguation.

Term	Meaning in the cited literature	Citation
Tessa	Eating-disorder support chatbot later used as a safety case study in mental-health AI	(Reddy et al., 7 Sep 2025)
TESS	NASA’s Transiting Exoplanet Survey Satellite, including exoplanet discovery, asteroseismology, and detector characterization	(Wang et al., 2018, Theodoridis et al., 2023, Krishnamurthy et al., 2019)
TESS	Transformer Encoder for Social Science, a compact domain-specific pretrained encoder for social-science text analysis	(Ge et al., 2022)
TESSA	Multi-agent framework for cross-domain time-series annotation	(Lin et al., 2024)

In astronomy, TESS denotes the Transiting Exoplanet Survey Satellite, the mission that produced the first confirmed TESS hot-Jupiter discovery, HD 202772A b, from Sector 1 photometry and CHIRON/HARPS radial velocities (Wang et al., 2018). The same mission name appears in work validating TESS asteroseismology against APOGEE DR17, where TESS seismic gravities were found to be calibrated to the Kepler scale to better than 5\% for about 90\% of red giants (Theodoridis et al., 2023), and in detector-engineering work on the absolute quantum efficiency of TESS CCDs across a 650–1050 nm bandpass (Krishnamurthy et al., 2019).

In NLP and time-series ML, the similarly spelled terms are likewise unrelated to the chatbot. TESS can mean Transformer Encoder for Social Science, a 12-layer transformer encoder with shared attention weights, 768-token input length, and pretraining on over 35GB of compressed text from social-science-relevant corpora (Ge et al., 2022). TESSA, by contrast, denotes a multi-agent system that generates general annotations and domain-specific annotations for time series using feature extraction, decontextualization, adaptive selection, and domain-specific rewriting (Lin et al., 2024). The practical consequence is that “Tessa” requires context-sensitive interpretation in scholarly indexing and citation, because the same or adjacent strings refer to a chatbot incident, a space telescope mission, a transformer encoder, and a time-series annotation framework.