Google's conversational diagnostic AI, AMIE, now has its first prospective results from a real outpatient clinic. The paper was published in The Lancet on October 8, which Google says is the first time the company has appeared in the journal's main edition. DeepMind CEO Demis Hassabis then publicly congratulated the team on social media.
The study ran in the urgent primary care clinic at Beth Israel Deaconess Medical Center (BIDMC) in Boston. Enrollment took place from April to November 2025. The trial was registered in advance on ClinicalTrials.gov and followed a protocol approved by an ethics board, and patients were told that taking part or not would not affect their subsequent care.
How it worked
Before seeing a doctor, each patient first held a text conversation with AMIE, which took the medical history and condensed it into a summary. A physician watched every conversation in real time, and the study had defined in advance the safety criteria that would force a stop. Eight weeks later, researchers went back through the medical records to confirm each patient's final diagnosis and compared it with the assessment AMIE had given at the time.
The headline numbers:
- 114 patients enrolled, 98 completed the conversation;
- No conversation triggered a safety stop;
- The supervising physicians logged one hallucination, and in five other conversations they added clinical information;
- The final diagnosis appeared in AMIE's differential diagnosis list 90% of the time, and in its top three 75% of the time.
The clinicians' side was also surveyed. Of the 98 cases, primary care physicians filled in a questionnaire for 60. In 44 of those they had read AMIE's conversation transcript before the visit, and in 33 of the 44 they said it helped them prepare for the encounter.
Look at the denominators
With a sample this small, the figures need to be read differently. Both the 90% and the 75% rest on fewer than 100 patients. The “helpful” rating from physicians works out to a denominator of just 44 cases: 33 out of 44 is exactly 75%, and every case added or removed moves the percentage by a little over two points.
The same goes for the zero stops. No problems in 98 conversations, under the commonly used “rule of three” (3 divided by the sample size), only roughly shows that the rate of serious safety events is about 3% or lower. A sample this size cannot rule out lower levels of risk. The researchers themselves describe the work as a feasibility study, and it had no control group.
Another set of numbers is less flattering to AMIE. In a blinded comparison of AMIE's management plans against those of primary care physicians, AMIE lost to the doctors on both whether the plan was practical and whether it was cost-effective. Listing possible conditions and deciding what to do next are tasks of a very different order of difficulty.
Limits and next steps
The limitations listed in the paper include a single site, text-only interaction, roughly one hundred patients and no control group. Google's stated position is that patient-facing AI needs large-scale evaluation, which in turn requires larger clinical trials.
A preprint of the study went up on arXiv on March 9 this year, and Google published a detailed write-up on March 11. What is new this time is peer review and formal publication. From AMIE's 2024 comparison against doctors in simulated consultations to this move into a real clinic, Google has followed a sequence of simulation first, then a small group of real patients, then a large trial. The company has not yet said when or where that third step will begin.
Sources: The Lancet, Google Research Blog, CocoLoop, Beth Israel Deaconess Medical Center; enrollment, completed conversations, safety events and diagnostic hit rates were checked against the paper abstract and the hospital's announcement.