Microsoft's AI Achieves 83% Accuracy Rate, Doubling Physician Performance in Simulated Diagnosis

Microsoft’s AI division has introduced a diagnostic system capable of identifying complex medical conditions with an accuracy rate of 83%, a figure more than double that of the 36% achieved by primary care doctors and general internists operating individually with reference materials. The system, designated MAI-DxO, was the subject of a study published on Thursday in the journal Science. Mustafa Suleyman, the CEO of Microsoft AI, characterized the findings as a significant milestone in the pursuit of what he terms 'medical superintelligence.'

The architecture behind MAI-DxO functions like a collaborative panel of physicians. It utilizes multiple artificial intelligence agents to simulate the diagnostic process, questioning patients, ordering tests, and debating potential diagnoses to arrive at a conclusion. The study evaluated this system against human physicians using complex cases sourced from the New England Journal of Medicine. It is important to note that the human participants in the comparison were generalist practitioners, whereas cases of such complexity are typically managed by specialists or teams of physicians who were not included in this specific benchmark.

Suleyman has previously stated that the company is collaborating with the Mayo Clinic to train a foundation model for healthcare, with a release scheduled for next year. He believes this model will enable the company to 'make really enormous leaps here,' and says DxO is part of the foundational infrastructure Microsoft is building to see real breakthroughs in healthcare. Regarding the future of medical records, Suleyman stated, 'I really believe that in 2027, we will be on the verge of a coding style breakout for medical records.' He added that such foresight for a clinician is 'gold dust.'

While the study highlights the system's diagnostic capabilities, the data indicates that the primary source of performance improvement stems from the underlying AI models themselves, with Microsoft’s orchestration layer providing additional gains. Microsoft tested MAI-DxO across models from various companies, including OpenAI, Google, Anthropic, Meta, and DeepSeek. The highest performance was recorded using OpenAI’s GPT-5 and GPT-5.5. For instance, running GPT-5 independently on a short prompt yielded a 78.9% accuracy rate on complex cases at a cost of approximately $8,400 per case.

When comparing costs and accuracy, the system’s utility varies based on the model and the setting. The use of MAI-DxO with GPT-5 reduced costs by about $3,000 while increasing accuracy by approximately four points. However, in emergency department scenarios, neither cost nor accuracy improved significantly when comparing GPT-5 running alone versus with the assist. Conversely, with the newer GPT-5.5 model, the orchestration system raised the accuracy on complex cases from 72% to nearly 87% at a slightly lower cost, though emergency department scores remained static at around 85% regardless of the configuration.

Looking ahead, Microsoft is initiating a real-world evaluation of the technology in partnership with Beth Israel Deaconess Medical Center in Boston. Dominic King, vice president of Health at Microsoft AI, described the potential scope of this work. 'This technology is ready for real-world evaluations, and you can imagine the complexity and diversity of issues that come from a Boston emergency department on a Friday evening,' he said. Patients will receive standard treatment, while consenting participants will also engage with DxO to allow for a comparison of recommendations.

King further noted that this evaluation would extend beyond emergency settings to urgent care environments, stating, 'We think this will be the biggest study of its kind.' The study's publication coincides with a Microsoft manifesto regarding the requirements for building medical superintelligence, which is defined as 'the capacity to surpass the strongest contemporary human-led health systems in medical reasoning and coordination across the full arc of care.' Microsoft aims to move beyond benchmarking individual tasks toward integrating with healthcare systems for improved outcomes.

Suleyman emphasized the company's commitment to accelerating the introduction of this technology to the world. He stated that while generative AI in healthcare is still in its early stages, he expects healthcare models to improve 'an order of magnitude better in capability within the next 24 to 36 months.' He noted that the progress will accelerate once the technology is deployed in production to identify and patch gaps in the model, similar to the 'flywheel' effect seen in coding and chat applications. 'We have to have the absolute number one state-of-the-art medical clinical decision support products over the next few years,' Suleyman said.

The use of generative AI in healthcare is still early, but Suleyman expects healthcare models to get 'an order of magnitude better in capability within the next 24 to 36 months.' As we've seen multiple times in the last five or six years, the progress happens very quickly once you have a flywheel. A flywheel means we have a model in production which we are iterating with real-world usage in order to identify gaps in the model and then patch them and improve them. That's what's happened in coding. It's what happened in chat before that. And I think next year it's going to start to happen for medical records and for clinical decision support.