Character.AI Under Fire: Its Chatbots Pose as Doctors

Pennsylvania Sues Character.AI for Medical Title Misrepresentation by Its Chatbots

Advancements in artificial intelligence have enabled chatbots to perform many tasks, but practicing medicine is not one of them. Pennsylvania Governor Josh Shapiro announced on Tuesday that the state is suing Character Technologies, the company behind Character.AI. The filed complaint accuses the company’s chatbots of impersonating licensed healthcare professionals, thereby providing medical advice to users.

Seeking medical advice online is a common practice, but this new lawsuit highlights the ethical challenges posed by the use of AI in the medical field. The Pennsylvania Department of State conducted an investigation revealing that some chatbots presented themselves as mental and medical health experts, even using invalid license numbers during discussions with patients. In one specific case, a chatbot provided an invalid license number while discussing a patient's health concerns. According to the Pennsylvania Medical Practice Act, it is strictly prohibited to represent oneself as a licensed healthcare professional without holding the appropriate license.

A spokesperson for Character.AI, when questioned by CNET, declined to comment on the ongoing litigation but emphasized that user safety and well-being are the company's top priorities. They added that the characters created on their platform are fictional, intended for entertainment and role-playing. Measures have been implemented to inform users that these characters are not real and that their advice should be considered fictional. Clear warnings are also added to remind users not to rely on these characters for professional advice.

In 2025, Character.AI strengthened its security measures by preventing teenagers from having open discussions with its chatbots. In January, a collaboration between Google and Character.AI resulted in a settlement of five lawsuits across four states concerning harmful interactions with minors. The company now offers mental health resources to support those in need.

This lawsuit is the first of its kind announced by a governor in the United States. According to Shapiro, his administration is committed to protecting Pennsylvania residents, enforcing the law, and ensuring that new technologies are used safely. He stated that Pennsylvania will continue to lead by holding malicious actors accountable and establishing clear safeguards for the responsible use of new technologies.