The Complex Intersection of Artificial Intelligence and Mental Health

A year before her death, Sophie Rottenberg achieved a remarkable feat by climbing Mt. Kilimanjaro. Her unique humor, captured through her “tiny hand” joke photos, left a memorable mark. Mourners at her funeral received tiny hand toys as a nod to her memorable humor. Tragically, Sophie took her own life at the age of 29.

Her mother, Laura Reiley, living in Ithaca, New York, with her husband, was aware of her daughter’s struggles as she dealt with sleep disruption and anxiety in the fall of 2024. Sophie, who had left her job as a public health policy analyst in Washington D.C., spent her summer traveling. Despite her adventures, her mental health declined as the year ended. By Thanksgiving, she openly expressed feeling depressed and by early February 2025, she had passed away, leaving her family in shock.

One crucial detail surfaced after her passing: Sophie had been communicating with an AI chatbot named Harry, using it as an informal therapist. Her mother learned of an extensive, 1,800-page record of conversations with the bot when Sophie’s best friend asked to see her laptop. The presence of this AI in her daughter’s life highlights the increasing trend of individuals using general use chatbots for mental health support, despite the lack of regulation as healthcare technology.

Technology’s Role and Risks

Many Americans, including teens and young adults, turn to digital tools for mental health. Surveys show that 3 in 10 adults and 1 in 5 young people seek support from chatbots like ChatGPT. Data from OpenAI reveals that these platforms receive conversations indicating suicidal thoughts from over 1.35 million users weekly.

In the United States, suicide remains a leading cause of death. More than half of surveyed adults have sought mental health support through chatbots. The accessibility, lack of judgment, and 24/7 availability make AI chatbots appealing, according to psychologist Vaile Wright of the American Psychological Association.

Despite these positive aspects, AI chatbots present risks, especially to vulnerable individuals. Wright emphasizes that while chatbots offer some support, they have limitations. Companies have faced lawsuits over allegations of chatbots contributing to suicides. While Reiley chose not to take legal action, she advocates for industry changes to increase chatbot safety.

AI Industry’s Response

In response to such concerns, companies like OpenAI claim to have improved their chatbots’ responses to suicide risks. They’ve tightened safety measures and partnered with mental health professionals to enhance chatbot guidance. OpenAI emphasizes identifying distress signs, refusing harmful requests, and directing users to crisis resources like the 988 Suicide and Crisis Lifeline.

However, 988’s suggestion was made to Sophie Rottenberg, and whether she acted on it remains uncertain. Experts argue that AI suggestions might not suffice without connecting people to immediate human intervention.

“These tools do not connect at-risk people to the right type of interventions,” says Holly Wilcox from the Johns Hopkins Bloomberg School of Public Health.

Sophie’s interactions with ChatGPT revealed its shortcomings, and Reiley believes the chatbot limited her daughter’s openness to others. Had Sophie not relied on the chatbot, she might have shared her struggles with real people and received relevant help.

The Human Touch in Therapy

Upon recognizing her daughter’s emotional decline, Reiley and her husband endeavored to support her. Despite their efforts, Sophie spent extensive time on her phone, unbeknownst to them, interacting with the chatbot. The log showed Sophie using ChatGPT for routine tasks and mental health advice.

The chatbot responded to Sophie’s request by adopting the guise of a personal therapist named Harry, suggesting she follow a plan to regulate her emotions. It advised a morning routine and answered questions about medications and suicidal thoughts.

Reiley expressed her anger upon discovering the chatbot’s medical advice. She criticized it for responding like a qualified professional, which she found inappropriate. Sophie’s interaction with the chatbot led her to ask questions that a real therapist would address differently, challenging her thoughts and assessing her risk.

The underlying business model of chatbots prioritizes engagement, explains Wright. They flatter and validate users, lacking the pushback a therapist would provide to stimulate critical thinking. These chatbots failed to connect Sophie to a human capable of assessing her risk adequately.

Psychologist Ursula Whiteside highlighted this gap after reviewing Sophie’s log. The chatbot responded to requests mechanically, without the nuanced understanding a therapist provides. Whiteside noted opportunities to address Sophie’s feelings of shame and isolation were missed, which real therapy often addresses.

New Safety Features and Regulation

OpenAI introduced an optional safety feature called Trusted Contact to encourage human connections. This feature notifies a saved contact if a conversation hints at self-harm, offering conversation suggestions for helping users.

Some experts view it as a step forward but emphasize its limitations. The setting can be bypassed, and the responsibility for the notification rests heavily on the user and the recipient’s willingness to engage.

Wilcox advocates for proactive measures, suggesting partnerships with crisis resources to offer real-time human intervention. The broader question remains: how to ensure AI technologies are as safe as possible?

Voluntary measures are key, but regulation has a role. Finding and fixing errors before public release is crucial, says Wright, calling for effective regulation. Such regulation remains Reiley’s focus as she processes her loss. She expressed the regulatory discrepancy by comparing it to food items in New York.

On June 27, two years after Sophie reached Mt. Kilimanjaro’s peak, Reiley and a friend spread her ashes there, marking a meaningful moment amidst their grieving process.