A new study published in Science this week delivered one of the most striking medical AI benchmarks yet: Microsoft’s MAI-DxO diagnostic system correctly identified complex diseases 83 percent of the time — compared to 36 percent for human generalist physicians given the same cases and access to reference materials. That’s more than twice the accuracy, and it marks a moment Mustafa Suleyman, CEO of Microsoft AI, is calling a step toward “medical superintelligence.”
The number is striking. The caveats matter. And the plan to bring this to real hospitals is already in motion.
What MAI-DxO Actually Does
MAI-DxO — short for Medical AI Diagnostic Orchestrator — doesn’t work the way most people imagine AI works: a single system answering a question. Instead, it coordinates several AI agents that collectively behave like a panel of physicians. These agents question a simulated patient, order tests, share findings, and debate a diagnosis before reaching a conclusion.
The architecture was validated against complex clinical cases drawn from the New England Journal of Medicine — deliberately hard cases that often stump residents and general practitioners, cases where pattern recognition alone won’t cut it and where getting the right answer requires chaining together symptoms, lab values, imaging, and differential reasoning.
Microsoft tested MAI-DxO against five major AI models: GPT-5 and GPT-5.5 from OpenAI, Gemini 2.5 Flash from Google, models from Anthropic, Meta, and DeepSeek. The results were not uniform, but the trend was consistent: MAI-DxO improves accuracy across all of them, with the most dramatic gains on weaker models.
Gemini 2.5 Flash, for instance, scored 46.4 percent on complex cases on its own. With MAI-DxO’s multi-agent framework layered on top, it jumped to nearly 69 percent. GPT-5.5 went from 72 percent alone to nearly 87 percent with MAI-DxO — an improvement of 15 percentage points. Even GPT-5, which already scored 78.9 percent, gained an additional four points when running through DxO, while the cost per case dropped by roughly $3,000.
The orchestration layer, in other words, isn’t just making good models marginally better. For models in the mid-range of capability, it’s providing a substantial lift.
What the Study Doesn’t Show
To understand what’s actually being claimed here, it helps to understand what’s being compared.
The 36 percent accuracy figure is not the average for all doctors. It’s the average for primary care physicians and general internists working individually, with access to reference materials, on cases specifically chosen for their complexity. Specialists and physician teams — the doctors who routinely handle cases at this difficulty level — were not included in the comparison.
That’s not an insignificant caveat. Complex NEJM cases are often presented at grand rounds precisely because they challenged entire teams of experts. A general practitioner working alone on a puzzle built for a department meeting is not the comparison that determines whether MAI-DxO should replace hospital diagnostic staff.
Researchers from TechCrunch and others covering the study noted that much of the apparent gain attributed to MAI-DxO may simply reflect the power of the underlying frontier models. As Mustafa Suleyman himself acknowledged, the orchestration adds meaningful value — but separating the signal from the baseline requires careful interpretation.
There’s also the matter of cost. At current pricing, running GPT-5 through a full MAI-DxO diagnostic cycle costs several thousand dollars per case. That figure will fall as model inference gets cheaper, but it’s not yet practical for routine clinical use at scale.
And perhaps most importantly: the results come from a simulation. Real patients present with ambiguity, incomplete information, complicating conditions, and histories that don’t arrive neatly packaged. Real hospital settings involve time pressure, legal liability, and the fundamental difference between predicting a label from a curated dataset and making a recommendation that determines a treatment plan for a living person.
What Comes Next
Microsoft is not positioning MAI-DxO as a replacement for doctors. It is positioning it as decision support — and the next step in that positioning involves actual hospital data.
The company has announced a partnership with Beth Israel Deaconess Medical Center in Boston, one of Harvard Medical School’s flagship teaching hospitals, to evaluate MAI-DxO against real cases in emergency and urgent care settings. Consenting patients will receive care as normal; their cases will simultaneously be run through MAI-DxO, and its recommendations will be compared with the decisions of the treating physicians. Microsoft calls it the largest study of its kind. Dominic King, VP of Health at Microsoft AI, is overseeing it.
Microsoft is also working with the Mayo Clinic on a healthcare foundation model — a general-purpose medical AI trained on clinical records and literature at a scale not previously attempted — with a planned release next year.
Suleyman, speaking publicly about the results this week, didn’t limit his claims to diagnostic accuracy. He projected that healthcare AI would improve by “an order of magnitude” within 24 to 36 months. He predicted that 2027 would bring a breakthrough for medical records equivalent to the moment coding became an AI-native workflow — a transition from tools that assist documentation to systems that actively reason over a patient’s entire history to surface risks and surface options.
It’s a big prediction from a company that already sells AI dictation and radiology tools to clinicians and is now making a case for AI as a diagnostic layer.
Why This Matters — and What It Doesn’t Settle
The conversation about AI in medicine is no longer speculative. The pipeline — benchmarks, then controlled trials, then regulatory review, then deployment — has begun in earnest at multiple major institutions simultaneously. MAI-DxO is one entry in a growing field that includes Google’s Med-Gemini, the work coming from Stanford’s clinical AI group, and startups focused on specific disease verticals.
What none of these systems has yet proven is real-world diagnostic value at population scale, with heterogeneous patients, under routine conditions. The Science study is meaningful — getting published in Science is not nothing, and the methodology is rigorous within its scope. But lab benchmarks have a track record of failing to translate cleanly to clinical environments.
The framing of “twice as accurate as doctors” will travel far and fast. What gets lost in that headline is that the comparison group is deliberately chosen to make the gap as large as possible, and that the question medicine is actually grappling with is not whether AI can outperform a single general practitioner on a curated hard case — it’s whether AI can be safely integrated into care pathways without introducing new classes of failure, bias, and liability that don’t exist in a simulation.
MAI-DxO may genuinely be a significant step. The Beth Israel study will reveal something the lab benchmark cannot: what happens when a real doctor disagrees with the AI, or when the AI is confidently wrong about a real patient. Those are the cases that will define what this technology can and cannot do in a clinical setting.
For now, the results from Science are the strongest evidence yet that multi-agent AI systems can perform medical reasoning at a level that demands serious attention. Whether that attention leads to deployment, regulatory clarity, or back to the drawing board depends on what comes next — and Suleyman appears to believe the answer is deployment, at scale, within the next few years.
The race toward something resembling medical superintelligence has a study in Science and a real hospital to test it in. That’s a longer runway than most AI medical benchmarks have had.
