Microsoft and Amazon: AI in Healthcare vs. Real-World Effectiveness

A Rise in AI-Based Health Tools

In October, Microsoft launched Copilot Health, a new feature integrated into its Copilot application. This innovation allows users to connect their medical records and ask specific questions about their health. Just days before this announcement, Amazon revealed that its Health AI tool, previously reserved for One Medical members, would now be accessible to a broader audience. These initiatives are part of a growing trend of integrating artificial intelligence into the healthcare sector, with players like OpenAI's ChatGPT Health, launched in January, and Anthropic's Claude, which can access users' health records with their permission.

The Demand for Health Chatbots

There is a clear demand for chatbots capable of providing health advice, due to the difficulty of accessing medical information in current systems. Some research suggests that current language models can offer safe and useful recommendations. However, researchers emphasize that these tools should be evaluated more rigorously by independent experts, ideally before their large-scale launch.

In a field as critical as healthcare, trusting companies to evaluate their own products could prove unwise, especially if these evaluations are not accessible for external review. Even if some companies, like OpenAI, appear to conduct quality research, they may still have blind spots that the broader research community could help address.

Andrew Bean, a PhD student at the Oxford Internet Institute, states, “As long as you will always need more healthcare, I think we should definitely explore all avenues that work. It is entirely plausible that these models have reached a stage where they are truly worth deploying.” However, he adds, “The evidence base really needs to be there.”

The Turning Points for AI Health Tools

According to developers, these health products are being launched because language models have reached a point where they can effectively provide medical advice. Dominic King, Vice President of Health at Microsoft AI and a former surgeon, cites advancements in AI as a fundamental reason why the company's health team was formed and why Copilot Health exists now. “We have seen huge progress in the capabilities of generative AI to answer health questions and provide good answers,” he says.

However, this only tells part of the story, according to King. The other key factor is demand. Just before the launch of Copilot Health, Microsoft released a report and a blog post detailing how people were using Copilot for health advice. The company claims it receives 50 million health questions every day, and health is the most popular discussion topic on the Copilot mobile app.

The Impact of AI Tools on the Healthcare System

Other AI companies have noticed and responded to this trend. “Even before our health products, we saw a rapid increase in the number of people using ChatGPT for health-related questions,” says Karan Singhal, who leads OpenAI's Health AI team. (OpenAI and Microsoft have a long-standing partnership, and Copilot is powered by OpenAI's models.)

It is possible that people simply prefer to pose their health issues to a non-judgmental bot, available 24/7. But many experts interpret this pattern in light of the current state of the healthcare system. “There’s a reason these tools exist, and they have a place in the overall landscape,” says Girish Nadkarni, Director of AI at Mount Sinai Health System. “It’s because access to healthcare is difficult, and particularly challenging for certain populations.”

The virtuous vision of consumer-oriented LLM-based health chatbots rests on the possibility that they could improve users' health while reducing pressure on the healthcare system. This could involve helping users decide whether they need medical attention, a task known as triage. If chatbot triage works, then patients needing emergency care might seek it earlier than they otherwise would, and patients with milder concerns might feel comfortable managing their symptoms at home with the chatbot's guidance rather than unnecessarily overloading emergency rooms and medical offices.

The Challenges of Evaluating Health Chatbots

However, a recent widely discussed study by Nadkarni and other researchers at Mount Sinai found that ChatGPT Health sometimes recommends too much care for mild conditions and fails to identify emergencies. Although Singhal and some other experts suggested that its methodology might not provide a complete picture of ChatGPT Health's capabilities, the study raised concerns about the low level of external evaluation these tools undergo before being made available to the public.

Most academic experts interviewed for this article agree that LLM-based health chatbots could have real benefits, given the limited access to healthcare that some individuals face. But all six expressed concerns about launching these tools without testing by independent researchers to assess their safety. While some announced uses of these tools, such as recommending exercise plans or suggesting questions a user might ask a doctor, are relatively harmless, others carry clear risks. Triage is one; another is asking a chatbot to provide a diagnosis or treatment plan.

The interface of ChatGPT Health includes a prominently displayed warning stating that it is not intended for diagnosis or treatment, and announcements for Amazon's Copilot Health and Health AI include similar warnings. But these warnings are easy to overlook. “We all know that people are going to use it for diagnosis and management,” says Adam Rodman, an internal medicine physician and researcher at Beth Israel Deaconess Medical Center and a visiting researcher at Google.

Medical Testing and Current Limitations

Companies claim they are testing chatbots to ensure they provide safe responses in the vast majority of cases. OpenAI has designed and released HealthBench, a benchmark that evaluates LLMs on how they respond in realistic health-related conversations, although the conversations themselves are generated by the LLM. When GPT-5, which powers both ChatGPT Health and Copilot Health, was launched last year, OpenAI reported the HealthBench scores of the model: it performed much better than previous OpenAI models, although its overall performance is far from perfect.

However, evaluations like HealthBench have limitations. In a study published last month, Bean and his colleagues found that even if an LLM can accurately identify a medical condition from a fictional scenario, a non-expert user receiving the scenario and asked to determine the condition with the help of the LLM could only do so one-third of the time. Lacking medical expertise, users might not know which parts of a scenario—or their real-life experience—are important to include in their prompt, or they might misinterpret the information provided by an LLM.

Bean asserts that this performance gap could be significant for OpenAI's models. In the original HealthBench study, the company reported that its models performed relatively poorly in conversations requiring them to ask for more information from the user. If this is the case, then users who do not have enough medical knowledge to provide the health chatbot with the necessary information from the outset could receive unhelpful or inaccurate advice.

Singhal, OpenAI's health lead, notes that the current series of GPT-5 models, which had not yet been released when the original HealthBench study was conducted, does a much better job of soliciting additional information than its predecessors. However, OpenAI reported that GPT-5.4, the current flagship model, is actually worse at asking for context than GPT-5.2, an earlier version. Ideally, Bean says, health chatbots should be subjected to controlled testing with human users, as was done in his study, before being launched to the public. This could be a challenge, particularly given the speed at which the AI world is evolving and the time that human studies can take. Bean's study used GPT-4o, which was released almost a year ago and is now outdated.

Studies and Future Perspectives

Earlier this month, Google published a study that meets Bean's standards. In this study, patients discussed medical concerns with the medical chatbot Articulate Medical Intelligence Explorer (AMIE), which is not yet available to the public, before meeting with a human doctor. Overall, AMIE's diagnoses were just as accurate as those of the doctors, and none of the conversations raised major safety concerns for the researchers.

Despite these encouraging results, Google does not plan to launch AMIE anytime soon. “While research has progressed, there are significant limitations that need to be addressed before the actual translation of systems for diagnosis and treatment, including further research on fairness, balance, and safety testing,” wrote Alan Karthikesalingam, a researcher at Google DeepMind, in an email. Google recently revealed that Health100, a health platform it is building in partnership with CVS, will include an AI assistant powered by its flagship Gemini models, although this tool is likely not intended for diagnosis or treatment.

Rodman, who led the AMIE study with Karthikesalingam, does not believe that such extensive, multi-year studies are necessarily the right approach for chatbots like ChatGPT Health and Copilot Health. “There are many reasons why the clinical trial paradigm doesn’t always work in generative AI,” he says. “And that’s where the conversation about benchmarking comes into play. Are there trusted third-party benchmarks that we can agree are meaningful, that labs can adhere to?”

The key here is the “trusted third party.” No matter how well companies evaluate their own products, it is difficult to fully trust their conclusions. Not only does third-party evaluation bring impartiality, but if multiple third parties are involved, it also helps guard against blind spots.

Singhal from OpenAI states that he is strongly in favor of external evaluation. “We do our best to...”