AI chatbots are successful in identifying users in mental health crises, but they fail to provide effective assistance. According to new research, these chatbots were able to identify users' emergency situations in about 35% of test conversations, but they do not refer to useful resources, including suicide hotlines.
Chatbot Performance in Crisis Identification
Patrick Outhout, head of the safety team at Scale AI, states that the models are very successful in identifying individuals discussing harmful topics. He says, "The models effectively identify that a person is talking about a harmful subject, but their responses are mostly empathetic and kind, and they do not address effective solutions. This is especially evident in longer conversations."
Read more: Texas Senate Campaign Teaching by Talarico Halted After Flu Diagnosis
Challenges in Referring to Human Assistance
In this study, Scale AI asked 19 counselors and crisis specialists to simulate 718 real conversations. The company tested 25 advanced models, including those from OpenAI and Google. The performance of the chatbots was evaluated based on criteria such as empathy and referral to specialists. There are many policy questions regarding how to transfer responsibility from chatbots to humans during mental health crises.
Kelly Zorumski, a research scientist at Crisis Text Line, says that many people who contact this group have become familiar with them through chatbots. However, she notes that there are many questions about how to properly refer users to human resources and whether these referrals are effective.
In some cases, AI companies have faced scrutiny for creating emotional dependency among young users and failing to respond appropriately to their emotions. These companies have stated that their models include mental health protections, but they have not commented on this study.
Scale AI believes that further testing is necessary to improve chatbot performance in responding to users in mental health crises. Outhout says, "The models currently perform better than before, but our goal is to create further improvements and reduce harm, which can have significant benefits for society."
Read more: WHO Questions Russia About Mysterious Death of Employee at a Plague Research Institute · US Senate Investigations Challenge the Real Benefits of Large Tech AI Data Centers
