“Real patients don’t come with ABC answer options.”
That line from Mustafa Suleyman, CEO of Microsoft AI, sums up a fundamental problem in medical AI.
While many AI models have impressed by scoring high on multiple-choice medical exams like the United States Medical Licensing Examination (USMLE), the tests don't reflect the real-world challenges of diagnosing human beings.
Microsoft now believes it has taken a meaningful step toward solving that problem.
What’s going on?
Microsoft AI has built a new system called the Medical AI Diagnostic Orchestrator (MAI-DxO), which can take on real-world diagnostic challenges, not just simple multiple-choice questions from a textbook but complicated patient cases that even put experienced doctors in a spot.
And here’s the kicker: it does this better than many human physicians and at a lower cost.
In healthcare, delayed or wrong diagnoses are a huge problem. Billions of people around the world struggle to get correct answers, while healthcare costs keep climbing.
"Across Microsoft’s AI consumer products like Bing and Copilot, we see over 50 million health-related sessions every day. From a first-time knee-pain query to a late-night search for an urgent-care clinic, search engines and AI companions are quickly becoming the new front line in healthcare," the company said in a blog post.
How was it tested?
To see how far medical AI has come, Microsoft tested MAI-DxO on 304 real-world cases published in the New England Journal of Medicine (NJEM), one of the world’s leading medical journals.
These are not routine diagnoses. Each case is like a medical mystery: a patient shows up with symptoms, and clinicians must ask questions, order tests, and reason through possibilities before arriving at a diagnosis.
This step-by-step process, known as sequential diagnosis, is what Microsoft tried to replicate. The company built an interactive challenge from the NEJM cases called the Sequential Diagnosis Benchmark (or SD Bench). AI models and human physicians had to work through each case by making decisions, simulating how a doctor would think through a tough case.
How did AI do?
Microsoft tested several major AI models, including GPT, Claude, Gemini, Llama, Grok, and DeepSeek, and then added a twist: they built MAI-DxO as a sort of AI coordinator.
Instead of relying on a single model, MAI-DxO acts like a virtual panel of doctors, pulling together the strengths of multiple AI systems and deciding how to proceed at each step.
With OpenAI’s model at its core, MAI-DxO correctly diagnosed 85.5 percent of the NEJM cases. For comparison, 21 experienced physicians from the US and the UK, each with at least five years of clinical experience, scored just 20 percent on the same benchmark.
Even more impressive, MAI-DxO made fewer costly test requests.
"Microsoft AI built MAI-DxO to simulate a virtual panel of physicians with different approaches collaborating to find a diagnosis on each case. They also included the ability to set a budget to avoid infinite testing (higher costs, longer wait times, etc.)," Suleyman wrote in a LinkedIn post.
What’s next?
Microsoft says this is just the beginning. The company has already built other healthcare AI tools like RAD-DINO (for radiology workflows) and Dragon Copilot (a voice assistant for clinicians).
"This research is just the first step on a long, exciting journey," Suleyman said.
Microsoft AI is partnering with hospitals, clinicians, and health organisations to further validate the technology in real-world settings. It is also working toward open-sourcing the SD Bench so researchers can test their systems on the same challenging cases.
Discover the latest Business News, Sensex, and Nifty updates. Obtain Personal Finance insights, tax queries, and expert opinions on Moneycontrol or download the Moneycontrol App to stay updated!
