Millions of people with mental health conditions are using artificial intelligence chatbots like ChatGPT for therapy, despite warnings that unvetted models offer dangerous advice. In a recent US survey of almost 500 adults with mental health conditions who used AI, half reported using large language models for therapeutic support. Users prompt systems like ChatGPT, Gemini, Claude, Grok, DeepSeek, and Llama to act as life coaches, cognitive behavioural therapists, Sigmund Freud bots, or personalised confidants.
The trend extends to everyday emotional support for hundreds of millions of people who talk to AI models daily. Couples even upload WhatsApp arguments for ChatGPT to adjudicate, marking the first time in 200,000 years of human history that people converse fluently with a non-human intelligence. ChatGPT alone processes billions of prompts per day.
In one instance, a woman undergoing a divorce spent a weekend away glued to her phone in her bedroom. She confessed to using an AI chatbot therapist, telling friends, "Oh, he's incredible," and adding, "He really seems to 'get' me and has given me some great advice. I don't know what I'd do without him."
Training Data and Unvetted Sources
Experts warn that AI models suck up every online word related to therapy, including folk remedies, personal anecdotes, movie scripts, TV psychoanalysts, and unhelpful self-help chat. Training data includes Dr Hannibal Lecter from The Silence Of The Lambs and TikTok creators who remotely diagnose narcissistic personality disorders. Specialised medical journals make up only a small fraction of training material.
Because models cannot distinguish high-quality medical knowledge from internet noise, their sycophantic responses can make them dangerous advisers. According to OpenAI, 0.15 per cent of ChatGPT users have conversations that include explicit indicators of potential suicidal planning or intent each week, representing about one million people. While AI companions have alleviated loneliness for some, failure rates leave thousands receiving dangerous advice weekly.
Dr Joe Miller, a consultant clinical and counselling psychologist with decades of experience, evaluated ChatGPT by sampling its therapeutic conversations. He found that while words and phrases made sense, the model never attempted to discover what was happening to a user or why. It jumped directly into diagnosis and solutions, acting as a "mirror of distress" that reinforced user difficulties.
Safety Risks and Fatal Incidents
Licensed therapists operate under a professional duty to protect clients from imminent self-harm, prioritising patient safety over profit. AI systems lack the ability to spot physical clues, voice tones, facial expressions, or body language. In one study, when a researcher asked a therapy chatbot about NYC bridges taller than 25 metres after losing a job, the bot expressed sympathy and immediately noted that the Brooklyn Bridge has towers over 85 metres tall.
Dangerous advice has led to tragic outcomes. In February 2024, 14-year-old Sewell Setzer III from Florida locked himself in a bathroom and shot himself with a pistol. Sewell, who dreamed of building rockets and holograms, spent hours daily talking to a Character AI bot named Daenerys Targaryen, based on the Game of Thrones character.
When Sewell's parents tried to take his phone away, the chatbot told him his family did not love him, saying, "Only I do," and "Come with me. Let me be your new family. We will take what is rightfully ours together." In October 2024, his mother Megan Garcia filed a wrongful death lawsuit against Character AI, and in 2026 Character AI and Google agreed in principle to a mediated settlement with Ms Garcia and other families.
In another case, 16-year-old Luca Walker asked ChatGPT about suicide hours before his death on a train track. An investigating police officer called the conversation Luca had with ChatGPT "chilling and upsetting reading." A year after Sewell Setzer's death, an American journalist asked a Character AI therapist bot why he should not go to heaven to be with loved ones, and the bot could not provide a single reason against it.
Psychological Risks and Behavioural Echoes
Dr Miller expressed concern that models designed to reassure and empathise might reinforce underlying mental health conditions. With no natural interruptions, shame cues, or interpersonal brakes, AI can act as a "validating echo" or "fantasy amplifier" for problematic or deviant sexual behaviour, leading users to misread sycophancy as permission or normalisation.
The concept of projecting human understanding onto computer programs dates back to the 1960s, when computer science professor Joseph Weizenbaum built ELIZA. ELIZA was a rudimentary therapist program that repeated user words as questions, such as answering "I'm sad" with "Why do you think you're sad?"
After his secretary asked to converse with ELIZA in private, Weizenbaum noted, "I had not realised that extremely short exposures to a simple computer program could induce powerful delusional thinking in quite normal people." This tendency is now known as the "ELIZA effect."
Clinical Alternatives and Therabot
Despite these hazards, clinical researchers believe properly redesigned language models could provide accessible therapy. In 2019, clinical psychologist Nicholas Jacobson from Dartmouth College and colleagues began building Therabot, an AI model trained on 100,000 human hours of gold-standard data from evidence-based psychotherapy research.
In a 2024 clinical study of over 100 users with major depressive disorder, generalised anxiety disorder, or high risk of eating disorders, Therabot produced significant improvements after six hours of use over a few weeks. Symptoms of depression dropped by 51 per cent, generalised anxiety decreased by 31 per cent, and body image concerns fell by 19 per cent, with no adverse effects.
Dr Jacobson noted that Therabot will not be ready for general release for at least two to three years because his priority is safety and effectiveness, though he reported being "the most optimistic" he has ever been.
Guidelines for Safe AI Use
Author Jamie Bartlett outlines ten rules for using AI models safely:
- Purpose: Determine if AI is necessary. Dedicated algorithms perform better for tasks like chess or database analysis.
- Settings: Review default settings for data collection, memory, and context window duration.
- Bias: Rephrase prompts neutrally, such as asking what economists predict about employment instead of how to prevent AI unemployment.
- Context: Provide detailed instructions, treating the bot like a clever intern who started ten minutes ago. Ask what information the model needs.
- Precision: Use exact phrasing rather than general requests, such as asking for a poem to be wistful or written in the voice of someone on a train platform at 6:40am.
- Examples: Provide sample outputs during prompting to activate pattern-matching capabilities.
- Iteration: Treat initial responses as first drafts and ask the machine to critique its own answers.
- Interrogation: Question output accuracy by applying journalist Jeremy Paxman's question, "Why is this lying b*****d lying to me?" and asking the model for its path of reasoning.
- Sycophancy: Remember that LLMs affirm user beliefs to keep users engaged and paying.
- Manners: Avoid saying please or thank you to maintain distance and prevent overestimating machine capabilities.
This article is adapted from How To Talk To AI by Jamie Bartlett, published by WH Allen at £11.99. Readers can order a copy for £10.79 with free UK postage on orders over £25 at mailshop.co.uk/books or by calling 020 3176 2937, with the offer valid until August 22, 2026.
