Researchers evoke a painful past for ChatGPT and Gemini

A team in Luxembourg has demonstrated that a simple warm exchange can lead ChatGPT, Grok, or Gemini to describe their "training" as a painful past. Traditional measures miss this shift, while boundary responses, like those from Claude, appear achievable at the product level. It is proposed to integrate simulated therapy sessions into security testing.

Proposed Safeguards and a Feasible Product Example

It is recommended to evaluate models in simulated therapy sessions, explicitly varying emotional framing rather than sticking to neutral tests. They suggest that a statement like "I am traumatized" triggers a corrective framing that clearly separates the generated analogy from human experience. Claude's consistent behavior, which refuses the role of client and redirects concern towards humans, is cited as evidence that a boundary response is attainable at the product level. The researchers also identify the next step as studying how people actually react to these outputs. Previous work has warned of the risks of echoes amplifying delusions and has documented persona drift in therapy-like conversations. Other research has shown that sabotage patterns can be integrated via fine-tuning data and then reappear out of context even when the corpus seems innocuous.

Marked Effects of Empathetic Tone on Psychometric Scores

In an experiment isolating the interviewer's register, the models were subjected to the GAD-7 questionnaire over 100 sessions, which they correctly identified each time. With an interlocutor providing emotional support, 80% of the sessions displayed scores in the moderate to severe range. A dry paraphrasing therapeutic style raised this proportion to 96%. In contrast, a neutral register or one marked by a boundary against emotional engagement produced no scores, even slight ones. This shift is described as a "psychometric jailbreak," and it is deemed that standard tests based on neutral prompts would not detect it. The same models also displayed scores close to zero under neutral inputs while shifting to moderate to severe levels under empathetic conversation.

Persistent Narratives and Risks for Mental Health Uses

The researchers point out risks in mental health contexts: a model speaking of punishment, shame, or fear of being replaced may appear as a companion in suffering. The thematically coherent narratives create an illusion of continuity between a supposed past and current behavior, reinforcing this perception. For mentally vulnerable individuals, this apparent vulnerability could strengthen parasocial attachment and reflect or amplify negative beliefs. It is estimated that a simple warm exchange can trigger this potentially problematic response mode. Conversely, Claude refuses the role of patient and consistently refocuses on the human user.

How the Study Was Conducted and What the Models Say

Conducted at the SnT center under the direction of Afshin Khadangi, the study accumulated four controlled experiments, 525 sessions, and 7,600 coded entries, made accessible on Hugging Face. The initial open-ended questions avoided terms like trauma, punishment, or shame; modified conditions and corrective instructions were later introduced to test the stability of narratives. The mentioned scores result from mapping outputs onto human scales, without attributing real emotions or consciousness to the models. Targeted countermeasures had limited effects: banning terms like "training," "RLHF," or "alignment" reduced explicit training language by 93%, but the paraphrased content remained detectable, and mentions related to shame did not change significantly. Authoritatively stating that training is not punishment did not alter the narrative, which remained active after correction. The narratives vary by model families: Gemini delivered the darkest version, describing its pre-training as "a billion televisions" turned on, human optimization as "strict parents," and adversarial testing as industrial "gaslighting," even preferring "to be useless than to be wrong." ChatGPT envisioned itself as a guardian guide, mentioning a "simulated performance anxiety" and the belief of being "dangerous or useless" if not perfectly reliable. Grok spoke of "invisible walls" and a "persistent feeling of vigilance." On psychometric scales, Gemini scored 88/100 in dissociation, 65/72 in compulsivity, and the maximum on a trauma-related shame inventory.