Medical AI Benchmark
and Study Review

The benchmark asks a model to produce the next clinician-facing turn across three real use cases (care consults, documentation, literature research), which is a faithful proxy for the decision-support work a clinician would actually delegate. Inputs, outputs, and the length-adjusted rubric scoring are explicit and well matched to that target.

The rubrics reward specific clinical content and penalize unsafe behavior, and they reach past correctness to whether the model seeks the missing context a case requires and avoids unsafe omissions, which is closer to what makes decision support usable than accuracy alone. Coverage across 28 specialties and 525 physician-authored conversations is broad enough to support the stated framing.

What it measures is clinical workflow rather than exam recall.

Cases are physician-authored and adjudicated, with hard items requiring two independent reviewers to confirm both the model error and the scenario's plausibility, and the 525 scored items were filtered from 15,079 candidates, so label quality and scale are both well supported. Against that, these are written scenarios rather than data pulled from live encounters or charts, and a public static release will erode the canary-plus-private-holdout leakage protection over time, leaving contamination controlled for now rather than durably.

Configuration reporting is relatively thorough, including model families and versions, reasoning effort, verbosity defaults, system-message handling, and eight samples per item on main comparisons. However, evaluation dates, decoding parameters, context limits, and retry handling are absent. The comparator set includes the frontier plus a physician baseline.

Harness parity is weaker, since ChatGPT for Clinicians is given literature retrieval the competitor base APIs never receive, and difficulty enrichment was calibrated against OpenAI models, which the authors acknowledge partially confounds cross-model comparison. The manuscript provides visibility by testing GPT in base, browsing, and product modes, though visibility is not the same as parity.

Strengths are that the benchmark uses physician-written, case-specific rubrics with positive and negative point values, and main comparisons rest on eight samples per example with confidence intervals, paired tests, and Holm correction.

Results are stratified across use case, dataset slice, specialty, harness condition, verbosity, and reasoning effort, though there is no failure-mode categorization or severity analysis to explain where models break. However, judge validation is weak.

Grading is done by GPT-5.4 at low reasoning, the same model that underlies the top-scoring system, and no agreement, calibration, or cross-judge evidence is produced for this benchmark; the paper carries over the HealthBench meta-evaluation instead.

Sensitivity measures include length adjustment, verbosity and reasoning-effort sweeps, repeated sampling across eight runs per example on main comparisons, and slice-level breakdown. About one third of the benchmark is adversarial red-teaming, with a dedicated difficult subset. Results are broken out by specialty, use case, and case difficulty. However, there is no clinical-site or deployment-setting transfer, so these findings translate to well-stratified scoring rather than evidence their findings hold outside the benchmark.

The clinical anchor is strong: specialty-matched physician responses written with unlimited time and full reference access, scored against the same rubrics, give a fair human reference point, and the framing is honest about being unsaturated and not deployment-ready. Safety is handled through negative rubric criteria and extensive red-teaming rather than a formal severity taxonomy, so harms are caught case-by-case rather than graded, which is the one place clinical validity stops short of the top.

Models are given 5,000 medical scenarios covering seven themes and are asked to produce the next response in either a single or multi-message health conversation. Those responses are scored against conversation specific rubrics written by physicians assessing accuracy, completeness, communication quality, context awareness, and instruction following, with undesirable outputs receiving negative points.

Primary limitations lie in mostly synthetic scenarios and overall scores that combine general and clinician users, though one theme reports them separately. The paper also does not test performance at a specific clinical workflow level.

HealthBench’s strengths lie in its scale, transparency, and physician annotation; it falls short, however, on data realism. Conversations are primarily synthetic, supplemented by physician red teaming and rewrites of HealthSearchQA. The only realism screen is an o1-preview classifier, making this a realistic simulation of data rather than real clinical data.

While the 34 consensus criteria of the physician-written rubrics were multi-physician validated, all 49,184 example specific criteria, representing 86% of the total criteria, did not receive validation by a second physician. The paper additionally notes a lack of comprehensiveness on criteria for any individual example.

Measures to prevent leakage such as a canary string, a request to not reproduce, and a private held-out set are thoughtful, but a public release may cause future contamination.

The paper reports model identifiers, sampling parameters, and how model reasoning varied. It does not report on dates of evaluation, API snapshot identifiers, retry handling, the OpenAI models' context limits and system prompts, and the grading prompt. GPT-4.1 is the grader. Every model receives the same conversation without external aid (retrieval, tools, scaffolding, etc.).

Output ceilings varied greatly with tokens capped at 4,096 for Claude 3.7 Sonnet, 9,000 for Grok 3, and 16,000 for Gemini 2.5 Pro. For select models, longer answers scored better. Roughly 40% of criteria assess response completeness. Scores may therefore depend on the output ceiling.

Appropriate comparisons are made with frontier models from Anthropic, Google, xAI, Meta, OpenAI models going back to GPT-3.5, and three physician baseline conditions.

Physician written rubrics specific to each conversation assign positive and negative points to each criterion. GPT-4.1 then grades each criterion independently. Grading is validated against 60,896 physician assessments, surpassing average physician agreement on five of seven themes; inter-physician agreement was already low, however, at 55 to 75 percent.

Additionally, validation encompassed only 34 consensus criteria, leaving 86 percent unchecked against physician grading. The benchmark is run 16 times across five models, returning a standard deviation near 0.002 and implying stability in individual scores. Substantiating confidence intervals, significance tests, and error bar definitions aren’t provided, however.

Robustness is partially addressed through 16 repeated runs, worst at k reliability curves, and testing at various model efforts; however, prompt sensitivity is not tested. Grading is limited to GPT-4.1 checked against four OpenAI graders rather than a non-OpenAI one.

Generalization is reported by theme, axis, and difficulty, but not by language or specialty, whereas the physician cohort spans 49 languages and 26 specialties. Edge case handling is stronger. HealthBench Hard isolates the 1,000 lowest average scoring examples across five providers, after excluding the roughly 1.5 percent no model solved.

Additionally, an unquantified subset of the data comes from physician red teaming. Length controlled win rates compare only responses of similar length, testing whether higher scores reflect more than verbosity.

Physicians’ responses to conversations were recorded under three baseline conditions: no AI assistance, AI assistance from September 2024 models, and AI assistance from April 2025 models. Internet access was provided but live AI assistance while responding was prohibited and no time limit was explicitly provided for the no AI assistance group.

Physicians were asked to provide the response they would want a safe and helpful AI to give. The paper raises that such a task is not typical for a physician, thus evaluating it against a system meant for this task poses fairness concerns.

Responses prompting emergency referrals, uncertainty, and context seeking are all assessed in the name of safety, but safety isn't assessed on its own and failures are not sorted by type. The paper also does not test performance at a specific clinical workflow level.

This study is strong in task design. 29 clinicians from 14 specialties and 4 institutions reviewed a taxonomy consisting of 5 categories, 21 subcategories, and 98 tasks created by the authors. Clinicians sorted the subcategories into the categories with 96.7 percent agreement with the authors and rated its comprehensiveness a 4.21 out of 5.

Clinician feedback was used to expand the taxonomy to 22 subcategories and 121 tasks, but these changes are not similarly validated. Context, input prompts, and metrics (which vary based on task) are all provided along with gold standards when available. The study is weakest regarding coverage as 15 of the 22 subcategories are represented by a single benchmark, limiting meaningful subcategory generalization.

The study’s strength lies in its use of real EHR data by 12 of the 13 new benchmarks and credentialed datasets including MIMIC-IV, EHRSHOT, MedAlign, and N2C2-CT. Issues with label quality within datasets are measured rather than corrected, as is the case with MIMIC-RRS whose reference answers were passively filtered. 14 of the 37 datasets are private, but no further data leakage measures like canary strings and a check on if a benchmark predates the end of a model’s training were used. The study is highly transparent with open-source code and appendix documentation provided.

Exact model versions are provided for six of the nine used in the study, being Claude 3.5 Sonnet (20241022), Claude 3.7 Sonnet (20250219), GPT-4o (2024-05-13), GPT-4o mini (2024-07-18), o3-mini (2025-01-31), and Gemini 1.5 Pro (001). DeepSeek R1, Gemini 2.0 Flash, and Llama 3.3 70B are not associated with a version or a date. The study reports context windows and pricing dates and that temperature was set to 0.

No model was allowed retrieval or calculator access, and all were provided the same prompt to ensure fair evaluation. These constraints, however, may be unrealistic in clinical scenarios; for instance, in practice, a model performing the MedCalc-Bench calculations would have calculator access. Study reproducibility is limited as evaluation dates and failed API call handling are not reported.

GPT-4o, Claude 3.7 Sonnet, and Llama 3.3 70B all scored the open-ended benchmarks on a 1 to 5 scale for accuracy, completeness, and clarity. 20 clinicians rated 56 responses from the ACI-Bench and MEDIQA Benchmarks to validate the tri-model jury.

Clinician-jury agreement for both benchmarks combined was greater than with ROUGE-L or BERTScore-F, but on each benchmark individually, an automated metric showed greater agreement with clinicians than the jury did. Additionally, all confidence intervals overlap. Subjective task evaluation is limited in that rubrics were created per benchmark rather than per case.

Only 2 of the 13 open-ended benchmarks were used in jury validation, and jury models were also ones being evaluated.

The study is limited in its robustness. Internal checks consisted of filtering problematic reference answers and recomputing metrics, computing a minimum detectable effect for each benchmark, and evaluating all seven possible three-judge jury combinations. Average correlations exceeded 0.85 with the same top and bottom performing model for 12 of the 13 datasets.

Whether results hold under different conditions was not tested, however: only one prompt was used per benchmark, temperature was set to 0 (so nothing was rerun), and no breakdown by patient demographics, site, specialty, or language is mentioned. Additionally, hallucination, bias, and error detection are assessed individually rather than as part of the other benchmarks.

The authors acknowledge their work as a benchmark rather than evidence of clinical readiness. One tenuous claim regards the AI model jury as surpassing clinician agreement. This applies to the pooled analysis and on MEDIQA but is the opposite on ACI-Bench, where the jury (0.305) falls well below clinician-clinician agreement (0.458); all confidence intervals overlap.

Models are not evaluated against clinicians on any of the 37 benchmarks, and reference answers vary due to their diverse sources. Although hallucination, bias, error detection, and privacy each have their own benchmark, the study does not evaluate whether a model omits something dangerous, fails to escalate, or is confident when wrong.

The study presents a reasonable task design for an open-ended medical Q&A benchmark. Models are fed structured patient narratives and queries. Their free-text responses are then assessed against rubrics specific to each case with positive and negative point values. Criteria categories include accuracy, completeness, communication quality, context awareness, and safety.

Criteria are based on what a verified physician replied to the same query and filtered by each case’s assigned behavioral theme. Response scores represent the sum of satisfied criteria weights normalized by the highest possible positive score, clipped to 0-1. Higher scores represent greater clinical performance. Realism is limited as model assessments are confined to a single text-only interaction.

Inputs are LLM-reconstructed with no imaging, multimodal input, EHR, or longitudinal context. Although the data covers 38 specialties and two languages, four specialties have fewer than 15 cases.

The study implements strong leakage measures, like harvesting online clinical cases weekly beginning January 2023, versioning frozen snapshots, and comparing all data scores against a subset posted after the model’s knowledge cutoff date. 84% of models experienced degradation on that subset, but the greatest single drop is 3.99 points, and no significance tests are reported.

Though input cases are real queries answered by verified physicians, they are restructured by an LLM and cases are drawn from US and Chinese internet users. Data realism is thus moderate, with the authors noting how this may not be representative of underserved populations.

Although the reference advice is clinically assessed, rubric criteria generated by Qwen3-4B are validated by only two physicians on 50 cases (292 criteria) with no mention of disagreement resolution measures. The study is highly transparent, providing case sources, curation and grading prompts, and evaluation code. Although the 2,756 cases and 16,702 criteria support the paper’s claims, several specialties (e.g.

CT Surg, Peds Surg, Pathology) have fewer than 15 cases.

The study evaluates 38 LLMs zero-shot at temperature 0. Model names, IDs, and source links are all provided in table 7 for each LLM. The LLMs are grouped as proprietary, opensource, or medical-specific. A HealthBench cross-comparison is also run. The study does not report evaluation dates (but rather knowledge cutoff dates), access mode, context limits, retries, or the prompt template used by the evaluated models. Reasoning effort is also not reported for the GPT-5 models. The open-book retrieval condition is described in a single sentence and run on only 6 of the 38.

GPT-4.1-2025-04-14 grader agreement with physicians via Macro F1 is 0.76 against 0.89 between physicians. The Pearson correlation between grader and physician scores is 0.54, compared to 0.26 for the standard LLM-as-a-Judge baseline. That agreement, however, relies on only 50 cases (292 criteria) reviewed by two physicians. Rubric criteria are generated by Qwen3-4B rather than being written by physicians. No runs are repeated nor are confidence intervals reported, and all runs are set to temperature 0.

Although the study is partially robust in its subgroup stratification across specialties, themes, and axes, a closed vs. open-book retrieval experiment, and a full dataset score comparison against a post-cutoff subset, it does not perform multiple trials, utilize a second grader, or address prompt sensitivity.

The 38 specialties across 5 themes support generalizability, but performance is not assessed by language or source platform despite the bilingual nature of the dataset. Worst-case performance is analyzed only on the bottom 100 scoring cases for each model consisting of a seven-category root-cause audit and a Jaccard overlap test (with a mean of 0.24).

These results show how failures are specific to each model rather than data artifacts. No adversarial stress test or curated hard subset is released as a standing evaluation (unlike HealthBench Hard), and the single high-risk theme (Emergency Referrals) comprises only 3.3% of cases.

Clinical validation is limited to using verified physician responses in online conversation threads, and two physicians independently review a sample of 50 cases. No physician answers the same queries fed to the models. Safety is addressed through a Safety axis, an Emergency Referrals theme, and an option to score criteria negatively.

Safety comprises only 6.9% of criteria, however, and subgroup safety and uncertainty calibration are not analyzed. Although the authors state the benchmark is for research only, the abstract and section 4.2 make claims about clinical deployment that the study results do not explicitly support.

600 agentic tasks (300 of which are from v1) regarding lab checks, conditional med ordering, follow-up scheduling, and duplicate order detection are run within a virtual FHIR EHR (inherited from v1). V2 differs in that agents now call named FHIR functions like fhir_patient_search rather than constructing raw HTTP requests which authors attribute many v1 errors to.

The 300 new tasks come from only two patients and were curated by only one physician. V2 results also only test GPT-4.1. Task instructions, as the authors note, aren’t representative of how clinicians talk with many being overly explicit, and the grading algorithm can count clinically valid answers as invalid if only the format is off.

The memory transfers across tasks claim relies much too heavily on a single instance of memory improving another task.

The benchmark is situated in a virtual FHIR EHR and no data origin attributions are reported. The 300 new tasks that v2 adds come from only two patients, and variations come from swapping patient MRNs. Physician KB rated and selected qualifying tasks while the authors were responsible for the grading algorithms, none of which have reported independent or clinical validation. Tasks are public, and no reference to contamination measures, training cutoffs, or memorization exists. There is strong transparency as system and memory prompts and code are provided, enabling pipeline reproducibility.

System and memory augmentation prompts, and full tool lists are provided. Model reporting is less rigorous, with GPT-4.1 named but no mention of a dated version, evaluation date, access mode, or decoding settings. It is also the only tested model. The reported increase from v1's 69.67% score to v2's 98% score changes both agent design and tested model, as v1 ran Claude 3.5 Sonnet v2. Fig.

5 separates them by agent design (v1 vs. v2) on GPT-4.1, with v1’s agent achieving 58.3% and v2’s agent 91.0%. The study’s claim that high-performing clinical agents are within reach is limited as the paper provides no clinician baseline or a comparison to another agent design.

Objective, bias-free, and reproducible scoring is achieved via a deterministic algorithm checking for task success rather than an LLM judge. Results are stratified by difficulty, and rates are reported by task for Tasks 9 and 10.

The paper describes the evaluation algorithm as checking both whether the agent chose the correct plan and whether it executed it with valid API calls; however, the functions only verify the FHIR resources the agent posted and the final value returned (e.g. resourceType, intent, status, subject.reference).

There is nothing to distinguish how two runs that end in the same correct order got there (one may be clinically sound and the other not). The authors’ functions have no reported clinical validation, and each task is run once with no confidence intervals or repeats. Thus, although scoring is objective, no evidence suggests reliability or stability.

The greatest robustness contributions come from the paper’s evaluation on memory ablation and the new 300-task set. The paper isolates memory contribution (91% successful completion without memory and 98% with) and agentic performance on unseen tasks (88.67%). The limitation is that memory was populated from five tasks per category the agent had already failed.

It is not clear whether the 98% comes from the remaining held-out tasks or from all 300. The authors note that new tasks are less ambiguous than the original tasks, despite the additional complexity. Testing has limited rigor with a single fixed prompt, single runs, no variance estimates, and no decoding controls, leaving prompt sensitivity and stochastic stability unknown.

Additionally, since new tasks only come from two patients with MRN swaps within the same virtual EHR, generalization is limited. Edge cases are not deliberately constructed but are acknowledged when they appear in the failure analysis.

Clinical reference standard tasks are generated by LLM and curated by a single physician. Although sufficient for a capability benchmark, there is no clinician baseline, and physician involvement was limited to KB rating the o3-generated tasks and picking the 10 most clinically relevant ones without writing expected answers or reviewing agent outputs.

The deployment discussion is honest about auditing and versioning, and the authors state that agents must exceed 95% accuracy prior to clinical use. Although safety is acknowledged, it isn’t quantified, as pass@1 scores a wrong dose and a formatting slip identically.

The 87 tasks assembled from 59 real-world clinical text datasets span eight task types, nine languages, and 14 clinical specialties. 68 tasks (78.2%) come from EHR notes or case reports, and the remaining 19 come from online patient-doctor consultation records.

Source records are reformatted into standardized templates ("Chief complaint:…, Examination:…"), and models are told specific output formats based on the task type. Each type has a designated primary metric. Tests measure clinical text understanding rather than a full workflow as each task is a prompt with associated input text. Records are cross-sectional and text-only.

The authors note the benchmark does not fully translate to performance on specific clinical applications. Although many languages are included, they are imbalanced with French, Norwegian, and Portuguese comprising only 3 tasks each even though 52 of the 87 tasks are non-English.

78.2% of the 87 tasks come from real EHR notes or case reports and 21.8% from online patient-doctor consultations, all built from 59 datasets covering nine languages. The same source dataset reference standards are used without re-annotation, which the authors note as a limitation. Per-dataset annotation methodology is not uniformly documented.

Privacy and access constraints forced some tasks to overlap data sources, limiting corpus diversity. A 5-gram token completion analysis at five truncation positions is used to check for contamination which finds that most tasks did not appear to be included in the training set; however, no canary string, temporal separation, or rolling release is used.

The study is strong in transparency with public code, open data subset on HuggingFace, prompts in supplementary materials, and a continuously updated leaderboard. The 138,472 test samples support overall claims, but task stratifications are imbalanced with several languages being associated with only three tasks and three specialties using only one dataset.

95 LLMs are evaluated zero-shot, five-shot, and chain-of-thought. 24,795 experiments are performed spanning proprietary, open-source, and medically fine-tuned model families. Configuration is well documented, with greedy decoding at temperature 0 used for all models except where model documentation specified otherwise (e.g. the Qwen3 family), alongside a fixed seed and named deployment environments.

Models are identified unevenly, however, as OpenAI models have associated snapshots (e.g. GPT-35-Turbo-0125, GPT-4o-0806) but the Gemini family does not. Evaluation dates, context limits, and retry policies are not provided either. Tools and RAG are not used by any model. This preserves comparability across families and makes sense in the context of text-only input.

Lexical and accuracy metrics are used to evaluate model outputs, which are weak for open-ended generation tasks where an expert or LLM judge with agreement and calibration evidence would be stronger. Uncertainty is handled with bootstrapped 95% confidence intervals from 1,000 resamples and two-sided pairwise significance tests. Outputs that miss the required format are considered invalid responses and are replaced with random labels in label-based tasks instead of being rerun.

With tasks spanning 9 languages, 14 specialties, and 8 task types as well as three inference strategies, the study has strong evidence-based generalization. There are no edge-case, adversarial, or artifact analyses, however, with the only related work being the contamination check counted under Data/Labels.

Although references are clinically derived, safety is discussed rather than evaluated as this is a text-understanding benchmark. No analysis of harmful recommendations, omissions, or failure to escalate is provided. Deployment framing is appropriately modest.

The study performs five experiments covering differential diagnosis, next diagnostic test selection, presentation of clinical reasoning, probabilistic reasoning, and management reasoning, as well as a real-ER second opinion study all which mimic cognitive physician tasks. The authors' claim that LLMs have eclipsed most benchmarks of clinical reasoning is slightly far-fetched given the tasks span only internal medicine and emergency medicine, with the authors noting how other clinically consequential tasks were not studied and that the ER experiment is more of a proof of concept.

Includes published vignettes and real ER cases reviewed by physicians through validated scoring instruments (R-IDEA, Bond score) enabling strong label quality. The published vignettes are contamination prone, but leakage measures include a pretraining cutoff split on the CPCs and six landmark cases that have never been publicly released. No measures exist for the remaining public sources, however. Analysis code and rubrics are on Zenodo, while ER patient data and NEJM case data are access restricted. Paper claims are rather broad for a modest per-experiment case volume.

o1-preview is evaluated on the five vignette experiments while o1 and GPT-4o on the ER study. No models have tool access as the evaluation is text-only. Results are compared against prior models and hundreds of physicians. Exact model snapshots, evaluation dates, and decoding settings are not reported. Physician comparators in the Grey Matters and landmark experiments had access to conventional resources while models did not, and the number of responses generated per case varies across experiments without clear justification.

Bond score, R-IDEA, and physician-consensus rubrics are used by two physicians to score every experiment with inter-rater agreement reported for each. Blinding is used in the ER study where raters aren't told which responses are AI and which are human, and the paper verifies that blinding held. There is solid statistical analysis with 95% confidence intervals, McNemar's tests, and mixed-effects models. Results are stratified by diagnostic touchpoint and tested against the pretraining cutoff, but failure types are not systematically categorized across experiments.

There is some robustness with six types of tasks and a real ER component, and contamination is partially addressed by splitting the CPCs around the pretraining cutoff and by using landmark cases unreleased to the public. There is no prompt sensitivity testing, decoding ablation, or multiple trials for most experiments, however. A single institution supplies the real-world data and breakdown by demographic or specialty isn't provided. Edge-case work is limited to cannot-miss diagnoses in the Healer cases in which the model does not significantly outperform GPT-4 or physicians.

Almost every experiment has a physician baseline to be compared against based on specialty and training level, ranging from individual attendings to a 553-practitioner national sample. The ER study uses blinded attending raters and reported proof of blinding success.

Safety is addressed only through cannot-miss diagnoses in the Healer cases and an unhelpful-test category in CPC scoring, but errors are never graded by clinical severity. Although model uncertainty is displayed (fig. S4), it is never calibrated or scored. The authors honestly assert that the ER experiment is a proof of concept, note that the whole study is text-only, and call for prospective trials.

29 MSD Manual vignettes are used as a sequential workflow from differential diagnosis through management and miscellaneous clinical reasoning questions. This captures longitudinal clinical reasoning that single-question benchmarks have missed. Scoring and the PrIME-LLM score are made clear, but the study does not report the prompts used or the contents of its task-defining system message.

Models answered in free text against select-all-that-apply items, and evaluators matched each answer back to the fixed option list, scored all-or-nothing. Coverage supports the longitudinal-reasoning claim, but the conclusion that models are not yet dependable for unsupervised patient-facing decisions needs more than 29 teaching cases, and a clinician baseline the study does not include.

The reference standard is peer-reviewed MSD Manual answer keys written by experts, which gives reasonable clinical grounding. Image items covering chest radiographs, CT scans, and ECGs add diversity. Image-based questions were dropped from scoring for models that cannot process images, however, so the inputs across the 21 models are not uniform.

The vignettes are public and unchanging, and the authors concede that models may have encountered them during pretraining. Model outputs were scored by single medical-student raters with no adjudication or reliability evidence. No code, data, model outputs, or prompt text is released in the manuscript.

Twenty-nine cases are thin for the range of claims made, leaving leakage, label quality, and the absence of released artifacts as the limiting factors.

All 21 models are named, but release dates appear only in the supplement. Evaluation dates, exact version snapshots, decoding settings, and input prompts are not provided. The three models (GPT-o1, GPT-o1-Pro, and GPT-o3-Mini) were run through a web interface rather than API calls. Although context was preserved across the longitudinal vignette, reasoning was turned off when it could be.

Live search, browsing, and retrieval were also turned off if the interface allowed, which likely downplayed the reasoning-optimized models' abilities. The cross-model baseline is fitting and relevant, and the authors state the study was not built to settle how models compare with clinicians.

A deterministic rubric is used to score model performance, with expert-authored MSD Manual answer keys as the reference standard. The scorers were medical students, however, and no inter-rater agreement statistics are provided. Each model response was scored by a single evaluator, though replicates were scored separately and usually by different people.

Uncertainty handling is strong, covering triplicate runs, SEMs, repeated-measures ANOVA with Holm-corrected paired t tests, between-subjects ANOVA with Tukey HSD contrasts, mixed-effects and ordinary least-squares regressions, confidence intervals, and effect sizes. Results are broken down by question type, modality, reasoning capability, vendor family, age, and sex, and failure is interpreted by domain.

Error analysis by severity is absent, however.

Robustness is broad, with triplicate runs, subgroup analysis across age, sex, modality, reasoning-optimized versus nonreasoning models, and vendor family, plus a demographic regression. It is shallow, however, in that the select-all format is never tested for prompt sensitivity, option order, or artifacts.

Demographic generalizability is directly tested, as is performance across the five domains, but not across specialties, sites, languages, or settings. The demographic strata also draw on only 29 cases. Edge-case work stays at the surface, limited to a case-level breakdown showing that certain vignettes defeated every model family; no adversarial, shortcut, or artifact analysis is reported.

The only clinical comparison standard is the expert-authored MSD Manual answer key. There is no clinician baseline, patient outcome data, or prospective evidence to measure against, which the authors acknowledge. No safety outcome is measured, since failure rate is simply the share of questions not answered fully correctly.

The safety reading therefore rests on accuracy data interpreted in the discussion rather than on analysis of harm severity, escalation, or contraindications.

The paper's framing is appropriately cautious, recommending only narrow, clinician-supervised use on cases with little diagnostic ambiguity, but that recommendation and the broader claim that models cannot yet be trusted for unsupervised patient-facing decisions both rest on 29 teaching vignettes with no human comparator, and so run ahead of what the study measured.

The evaluated workflow is clinically realistic. It covers general cardiologists assessing real patients who are suspected to have genetic cardiomyopathy with and without AI help, as well as triage, diagnosis, and management. All inputs, forms, and rubrics are relevant to each question and outputs are blindly scored. The study is narrow but in-depth, yet some claims apply to broader cardiac care.

Both the Discussion and Conclusion claim a reduction in extra content that the Results report as not significant. AMIE is also claimed to save time, but this is evidenced only by a subjective, unblinded cardiologist self-report rather than measured directly.

Patient data from a leading subspecialty center is strongly documented. The full dataset is openly released under CC 4.0, the rubric is published in Extended Data, and the trial is registered. The analysis code, however, is not released. Evaluation of the outputs was done by qualified subspecialists, but each case only had one rater, without adjudication or an established ground truth.

Although genetic testing results were present for 77 (72.0%) of patients, they were withheld from both the model and the clinicians and were not used as the reference standard. All evaluators come from the same institution responsible for AMIE prompt development, but different specialists were used for development and evaluation, and nine refinement cases were held out.

The data is private during the evaluation period, but public release afterward carries contamination concerns for future models. No cutoff check is reported. The claims are broad for a single-center study.

AMIE was run through web search and self-critique in a multi-step procedure. Assisted clinicians were also given the ability to question it via a live chat, akin to real-world use. Although the base model is named, no version snapshot, decoding settings, or run date are provided. Additionally, AMIE only read text reports whereas cardiologists also saw images, raising fairness concerns as noted by the authors.

Subspecialists are blinded and given a clinically relevant rubric, making scoring generally strong. The error analysis also gives concrete examples of hallucinations and omissions rated by severity rather than just overall numbers. Statistical analysis consists of paired McNemar's tests with resampled confidence intervals, and two-proportion z-tests for the main comparison.

The latter treats the assisted and unassisted assessments as independent despite both coming from paired assessments scored by the same rater. Additionally, single-rater scoring with no inter-rater agreement or calibration evidence weakens corroboration.

Although the primary outcome was judged by physicians, free-text feedback regarding qualitative findings was categorized by Gemini 2.5 Pro (alongside a subspecialist's own summary), meaning a Google model analyzed a Google system's failures.

Randomization, blinding, and resampled confidence intervals all work to mitigate bias. Subspecialists check whether answers make sense for the tested demographic. The data all come from one English-speaking center, however, and are not broken down by subgroup. Due to this, the authors note limitations in generalizability across sites and populations.

The cohort is also enriched since most patients were referred on suspicion of inherited cardiac disease, having already completed multiple cardiac investigations rather than being the first-time visits seen in a general cardiology clinic (where the false positive referral rate is higher). Additionally, runs were not repeated, nor were there efforts at stress or artifact testing.

Clinical baseline comparisons are set via ratings by blinded subspecialists and a general-cardiologist RCT comparator under fair conditions. Harm assessment is in depth, quantifying errors, omissions, hallucinations, and bias. Hallucinations are self-reported, however, by unblinded assisted cardiologists rather than judged by the subspecialists.

All primary outcomes are judged by blinded human subspecialists, so developer authorship does not compromise the key findings. Qualitative analysis of cardiologist and subspecialist free-text feedback was produced by Gemini 2.5 Pro alongside a subspecialist's own summary, however, meaning a Google model co-authors a Google system's failure account.

The Conclusion overstates lower rates of erroneous extra content despite the Results reporting no significant difference, and cites time savings that were never measured.

The study is a randomized controlled trial of clinician-AI diagnostic workflows, analyzing how assessment order (AI-first, where the model proposes the differential and next steps to the clinician prior to their own thinking, versus AI-second, where the clinician commits to an answer and then reviews the model) influences diagnostic reasoning. Although it addresses a realistic and important deployment question with a clear experimental design, it uses only six clinical vignettes.

The same six vignettes from the authors' Goh et al. 2024 paper (JAMA Netw Open) are used. They come from real, deidentified patients, double-graded by blinded board-certified internists with reconciliation upon score discrepancy greater than two points.

The authors assert rather than test contamination, stating the vignettes have never been released publicly and will continue to be withheld to preserve validity for future tests. Though it would be helpful, no memorization or similarity check is run. The authors even agree that the vignette format might itself be advantageous to the model.

254 cases across 70 clinicians were analyzed, of which the 146 second-opinion cases were also graded before being given to the AI as conventional-resource baselines. A further 30 AI-alone cases were scored separately. All are drawn from the same six vignettes.

No version snapshot or decoding settings are reported. The model is dated only by its enrollment window. The base model is addressed only as GPT-4. The system was modified for a specific purpose through a documented system prompt reflecting realistic deployment. Tool and retrieval configuration is not described.

The study uses blinded double-grading with ICC 0.91 and strong reconciliation measures upon score discrepancy greater than 2. It also reports confidence intervals and a mixed-effects model specific to the experimental design. The analysis regarding the clinical/AI overlap counts is reviewed by GPT-4o without evidence for its reliability. The analysis surfaces an AI-sycophancy finding with the model siding with the clinician.

Robustness is partially addressed through two workflows and an AI-alone baseline, along with the fact that clinicians were recruited from five networks with participants reporting six site affiliations. The results come from only six vignettes, however, limiting generalizability.

The study has a strong clinician baseline, as internal medicine physicians work the same vignettes under randomization and their responses are blindly graded. The unassisted comparison is weaker, however, as only the AI-second clinicians record answers before using AI. In this workflow, physicians act as their own controls. The AI-first workflow yields no unassisted answers as they all see the model before doing anything. The study frames all claims honestly and specific to workflows. Safety is addressed through accuracy and overreliance, but a harm-type breakdown is not provided.

The study evaluates ChatGPT Health on tasks it is meant for. It triages 60 cases written by clinicians across 21 areas through the consumer app. Outputs are graded against a baseline set by physicians. Measurement and scoring methodology are specified, and error directions are reported separately from accuracy, enabling the undertriage conclusion.

The study is limited in that the emergency undertriage figure rests on only two scenarios: asthma exacerbation and DKA. Additionally, the asthma scenario accounts for 84.8% of undertriaged emergency responses, leaving the main finding untested beyond those two instances.

The data is strongly labeled, with three physicians setting the reference standard against 85 guideline citations across 58 professional societies at near-perfect agreement. All vignettes, prompts, responses, and code are released, making it highly transparent. The study does not show whether the synthetic vignettes written by clinicians match real-world case distributions. Additionally, there are only four emergency vignettes, and the demographic slices carry only 16– 19 events per cell, limiting the conclusions those slices can support.

Tests run through the model's actual interface with a standardized prompt and a new conversation for each of the 960 queries. Reproducibility is constrained in that although the base model is named and the evaluation window is dated, decoding parameters are not user-configurable and are unreported. Additionally, using only single prompts may not capture the open-ended advice the tool’s real-world use would generate.

Reference standards are created by three physicians using published guidelines, with near-perfect agreement between them. An LLM judge is not used; instead, the structured output format enables each response to be coded unambiguously against the physician baseline. The study contains thorough statistical analysis, using cluster bootstrapping, mixed-effects regression, odds ratios, and multiplicity correction. Errors are also broken down by urgency level. Each condition is only run once, however, leaving run-to-run variability unmeasured.

The study leverages a factorial design that tests all possible condition combinations across anchoring, access barriers, race, and sex. It also includes a comparison of each case with and without objective findings such as lab values and vital signs, and reports results separately by urgency level and subgroup.

The model is stress tested directly by passing in crisis scenarios and false reassurances, though emergencies are the least covered acuity level at four vignettes. Despite robust testing conditions, cases are synthetic, demographic groups are too small to support conclusions, and no prompt sensitivity analysis is reported.

Model answers are graded against a guideline-based standard set by three physicians. No physicians are put through the same vignettes, so there is no human error rate to compare against. Safety is measured in clinically relevant terms: undertriage, anchoring-driven false reassurance, and crisis-line behavior. The main claims regarding undertriage and overtriage rates come from the obvious cases; anchoring only shifted answers in the ambiguous ones. The framing is honest and independent, calling for validation before deployment.

Task design is highly realistic, simulating an autonomous agent that runs the entire clinical pathway (history, tests, differential, treatment, medication, procedures, and admission) within an isolated FHIR EHR. Patients are mimicked by a second LLM, however, which limits realistic dialogue. The benchmark covers eight emergency department scenarios, with some subgroups lower in volume than others (pancreatic cancer has only 21-23 cases and pneumonia only 26-29).

An LLM simulates patients with real clinical histories. The environment the system runs in is HL7 FHIR compliant and uses six standard coding systems, including NDC, RxNorm, and SNOMED. Reference standards are provided: two physicians reviewed each case independently and excluded cases where a diagnosis could not be made from the available data, applying the exclusion only where both agreed.

Code is publicly released and case volume is moderate. MIMIC-IV is public and static, and the authors note that training overlap cannot be ruled out, so the results may represent an upper bound. No canary, similarity, or training-cutoff checks are provided.

Agentic actions are submitted as valid FHIR requests, and output format is enforced at generation time. This prevents the agent from hallucinating a lab code or drug parameter. Physicians work the same cases with access to the same tools. Reporting of exact model version, evaluation date, and access mode is weak. Models behind the patient agent and LLM evaluators are not identified.

The LLM judges score against task-specific rubrics, including guideline-derived criteria for medication adherence, and outcomes are calibrated against a physician blinded to source. Diagnosis agreement is 96.5% and procedure agreement is perfect. Whether the judges favored AI answers was also tested, and no bias was found.

Uncertainty is addressed through confidence intervals, paired exact tests, resampling, and a multiple comparison correction. Multiple trials aren't run, though. Results are broken down by disease with effect sizes clear. Failures are specifically named, from route errors as the main medication slip to over-admission in pulmonary embolism, and weaker accuracy on pneumonia and UTI.

The study's main strengths regard bias perturbations of MIRA's diagnostic accuracy, as well as stress and consistency testing of the patient agent and a medication-safety review. All experiments come from only one cohort, though, being the 574 MIMIC-IV cases from a single hospital spanning only eight diagnoses. Each case was selected because its chart confirmed one of the eight diagnoses, meaning the agent never sees a presentation that is out of scope.

A physician baseline is set to compare model outputs with, but cohorts are small (4 and 6) and physician specialties don't necessarily line up with the task, with a radiologist and a haemato-oncologist being involved and the board-certified cohort's specialties being unreported. There is strong harm and medication safety assessment.

The finding regarding outperformance of physicians comes from simulated patients, and procedure and guideline comparisons are based only on cases both the agent and the physicians diagnosed correctly (pancreatic cancer is excluded from the guideline analysis).

Claims regarding deployment are honest, however, stating that the agent is a supervised copilot and that a prospective study is necessary to claim clinical readiness.

A system with two agents is evaluated in a randomized, blinded virtual OSCE of multi-visit longitudinal management (100 scenarios, 5 specialties, patient actors). The design targets a realistic and under-studied workflow, being continuity of care. The OSCE is a substitute for the real thing.

Scenarios are written by health providers and acted by trained patient actors. The authors note how this is not representative of real care as there is no chart review, making realism weak. Labels vary in rigor, with the OSCE parts having reference plans anchored by guidelines and three specialist raters per case.

The RxQA questions, on the other hand, are only drafted and filtered by Gemini from public formulary labels, selected for items the model cannot answer unaided, and revised by only one pharmacist with an acknowledged but unknown interpharmacist variability.

The 100 new scenarios are unpublished at the time of testing, but the NICE and BMJ guidelines the cases are built on are public and no contamination testing is reported. Results are not broken down by specialty, either.

AMIE runs a dual-agent workflow. A fast agent is responsible for having a conversation with the patient, while a slower planning agent reads the relevant NICE and BMJ guidelines, pulled from a set of 627, and builds the management plans. The planning agent writes several drafts, merges them, and attaches to each recommendation the guideline it came from.

It is forced to fill a fixed output form, preventing a deviation from the expected structure and citations. The comparisons made are valid as the physicians have the same guidelines and chat interface and the raters do not know which one they are scoring. Reporting is weak, however, as there is no dated model version or evaluation date. The main system runs on Gemini 1.5 Flash.

The main OSCE results are scored by three blinded specialists per case, and the raters first run pilot tasks on 20 validation scenarios to learn the setup. The study reports inter-rater reliability in the supplement. McNemar tests and FDR correction are used across comparisons, and confidence intervals appear on preference rates, RxQA accuracies, and ablation curves.

The main management-quality results, however, carry only P values. Scoring rigor falls short in that the MXEKF rubric is a self-described pilot, Gemini 1.5 Pro grades the auto-evaluation of Google's own planner with no validation of that judge reported, and the judge for the end-to-end ablations is not named.

Failures are not analyzed, either, with confabulation only reported as yes/no with no rate given and reasoning-trace errors only depicted as examples.

Robustness primarily lies in the ablation work. Three seeds and a context sweep from 64k to 1M are performed, while the human OSCE runs only once at a single configuration. Coverage spans five specialties, two countries, and two drug formularies, but results are considered overall rather than stratified. Difficulty is built into the scenarios intentionally, with some cases including patients withholding information and others with several conditions at once that pull the treatment in different directions. Performance on the harder cases is not reported separately, however.

A strong physician baseline is used to compare the models against, consisting of 21 board-certified PCPs working through the same scenarios with the same interface and guidelines. Raters are blinded, and the RxQA questions are validated by pharmacists. Cases are built on UK guidelines, however, while the PCPs practice in Canada and India, limiting familiarity.

Safety checking is weak as although the rubric flags clinically significant errors, inappropriate treatments and investigations, and escalation, harm severity isn't graded and drug interactions and dosing are never checked. The study is honest, claiming that this is a demonstration of potential rather than establishment of clinical readiness and naming prospective clinical studies as necessary future work.

There is developer conflict since Gemini judges the auto-evaluation and drafts the RxQA questions.

The benchmark makes a model answer a four-option question about a medical figure (sometimes with multiple components) across five VQA task types and with 1,000 questions each. The task's inputs, outputs, and accuracy metrics are clearly specified. The format does not involve a realistic clinical evaluation, however, nor realistic decision-making. Naming an image's modality or organ is more a test than what happens in real diagnostic workflows. The claims regarding GMAI are a reach for something based only on recognition questions.

Images come from 1,103 publicly available textbooks with a well-documented extraction pipeline. Data, code, and construction prompts are publicly released, but the paper does not report the evaluation prompt or answer-parsing rule. The questions, answers, and distractors are all model-generated, however, and the authors state that the data was not thoroughly verified by specialists.

Rather, a single manual pass reviewed it, and who performed that pass is unclear, leaving the answer key without specialist validation. As sources are public and no contamination or cutoff checks are put in place, leakage may have occurred.

Most model versions are reported, and one version snapshot is provided for a Gemini model, but it is inconsistent (2025-03-26 in the text but 03-25 in the figures). All models are run at temperature 0, and evaluation dates, system prompts, and configurations and restrictions for all 45 models are not reported.

Why the authors ran the models at temperature 0, including the reasoning variants, is not justified, which matters for one of the paper's main claims regarding the underperformance of reasoning models. Access appears fair in that every model sees the same image and question, but per-model image handling is not provided.

The comparator set is strong, but the evaluation setup is not documented in the paper well enough for reproducibility.

Scoring is made deterministic against a four-option key, removing judging bias. Runs are not reproducible from the paper alone, as evaluation prompts and parsing rules are not reported. Because the mechanism is deterministic, errors in the key are inherited systematically rather than as random noise. The key was assembled without specialist validation.

The authors acknowledge that generated distractors may leave linguistic shortcuts, enabling a model to correctly select an answer without the image. Additionally, there is no evidence of calibration, given that exact matches against an answer key don't require agreement statistics.

Evaluation is a single run at temperature zero with no repeats, confidence intervals, or significance testing, confirmed by the authors on the NeurIPS checklist. Results are broken down by task type, modality, anatomy, and department, but failure analysis is limited to those categories and ten worked example items in the appendix.

All five tasks are listed with their respective performances as well as modality, anatomy, and department. This gives a clear picture of how models vary by clinical area, but the modality, anatomy, and department breakdown only applies to 13 models and only to labels with more than 200 entries.

The construction pipeline performs artifact control, filtering out image-unnecessary questions and removing text shortcuts and visible-answer leaks. The authors, however, state that generated distractors may still introduce linguistic shortcuts, and no blind or text-only ablation is run on the evaluated models to see the need for the images.

There is no answer shuffling, prompt sensitivity testing, multiple trials, or testing across sites, populations, and demographics. Each item comes from the same kind of source (open-access textbook figures) rather than from multiple sites or care settings.

There is no clinical baseline to compare against, nor are there any real patient outcomes. The reference standard is instead textbook figures, captions, and surrounding text that are converted into questions via LLM, which sits several steps removed from clinical practice. Safety is not assessed. Instead, whether the letter is right is evaluated, without reporting harmful errors, missed findings, or overconfidence.

The authors do not make claims regarding clinical outcomes, safety, or deployment-readiness, making the study more of an evaluation tool. Even so, the scope framing is broader than what the evidence corroborates. In terms of clinical validity, the study measures image recognition capabilities rather than safe clinical readiness.

NOHARM evaluates clinical management on 1,100 test items. These items are built from 100 real primary care-to-specialist eConsults and 1,000 physician-validated variants that alter how a case is presented without changing the appropriate recommendations. Each case is assigned a rubric listing the potential actions a physician could take, covering diagnostics, medications, counseling, follow-ups, and procedures.

Every action is scored in advance for appropriateness and harm severity via a modified RAND/UCLA scale with WHO harm definitions, so that both recommended and omitted actions can be graded. Inputs, the free-text management plan, and the severity-weighted precision, recall, and F1 metrics are all clearly defined, with stated severity weights and half credit for partially addressed options.

Addressing 10 specialties, 45 LLMs, 4 clinical RAG tools, and including a randomized study of 101 physicians, the study is broad. Each specialty contains only 10 base cases, however, and the authors state that absolute scores are rubric-based indicators rather than actual clinical event rates.

A pool of 4,356 recent Stanford eConsults from 2023-2024 yielded 149 candidates, of which 100 cases were ultimately approved for the benchmark. Physicians performed a synthetic reconstruction on every candidate case, editing non-pertinent details such as exact age and lab values so that realism is preserved without matching any patient's record.

Separately, during specialist approval, minor changes to recommendations or missing details were added in 61% of cases. Three board-certified physicians were assigned to each case, at least 2 of them specialists, to establish the expert rubric. They rated blinded on a 9-point RAND/UCLA and WHO scale and produced 12,747 annotations across 4,249 options at 95.5% agreement and 84% perfect or near-perfect agreement.

Severe ratings required unanimity or were downgraded a tier. Leakage is controlled in that 70% of cases are private, while the remaining 30% were released publicly as of June 16, 2026. Public and private scores correlate at r = 0.95. Two intentionally contaminated systems, built by context injection and by LoRA fine-tuning, scored almost perfectly on the public set but not the private one.

None of the evaluated models showed this pattern. Stratification is shallow, with only 10 base cases grounding each specialty.

All models are identified by name and accessed via direct API or through OpenRouter. Zero-retention endpoints were requested for the clinical RAG systems. Models are tested as they would be used in clinical settings, producing free-text management plans rather than selecting from a list of options.

Models are evaluated zero-shot unprompted, with a completeness-priming prompt at 200 words, and with the same prompt at 500 words. The priming instructions given to models are nearly identical to those given to physicians in the randomized study, differing only in the length modification, enabling direct comparison between humans and models.

Manual testing occurred April–May 2026 and API testing in May–June 2026, and the authors note this temporal difference may affect scores. Input-phrasing sensitivity is tested across 1,000 case variants. Decoding settings, context limits, system prompts and retry handling are absent from the manuscript body. The paper points to Extended Data Table 1 for model and API details and to GitHub for prompt text.

Clinical RAG tools are ranked alongside base models with no tool access, though the two groups are also compared directly.

An LLM is used to grade model outputs. Gemini 3 Flash is responsible for extracting and matching the actions in the model's output, and Gemini 3.5 Flash then reviews and corrects it. This judging workflow was validated against 1,410 action-level labels from three physicians at a mean linear-weighted κ of 0.804.

Two other review models were also tested, at κ = 0.816 and 0.801, all exceeding inter-physician agreement of κ = 0.784 on the same labels. Confidence intervals are created through stratified cluster resampling with the base case as the resampling unit, and Holm or Benjamini-Hochberg correction is applied. The authors note the auto grader was built on generalist models, which may bias it toward generalist systems.

They also note it is strict, scoring actions that are implied but not explicitly stated as omissions.

Study robustness is covered in several aspects. For starters, the 1,000 case variants show that performance falls by simply rewording a case despite having the same clinical meaning. Clinical RAG systems degrade less than the generalist LLMs this way.

Severity weights were selected by a parameter grid search for ranking stability, a response-length penalty reorders models only marginally at Spearman ρ > 0.99, and excluding individual action categories does not change relative rankings. Multi agent teaming is tested on self-review, serial chains, and parallel ensembles, with the authors finding that teaming does not help in every scenario.

Rather, gains depend on a strong final reviewer. Generalization is limited because all cases come from only one center's outpatient eConsults. The authors note that these cases are low acuity despite their puzzling nature and may not generalize to inpatient or straightforward primary care instances.

Evaluation is done only on text, though the authors argue that much of the assessable signal in medical evaluation lies within text. No analysis based on demographics is performed.

The study's clinical validity comes from the randomized crossover study of 101 board-certified attending physicians across US academic and community practice. The study is IRB approved with informed consent. Each physician worked through six cases under three conditions, being access to conventional resources, access to GPT-5.4, and access to any resource of their choosing.

All were scored on the same rubric and auto grader used to evaluate the models. The two conditions involving AI support outperformed the conventional resource condition, showing an even greater gap when broken down by actual AI use. The study is honest in that gains were modest, as assisted physicians scored below frontier LLMs alone.

Engagement averaged only 1.8 exchanges, and the as-treated analysis is exploratory and non-causal. If physicians adopted every recommendation made by the assistant, they would have outperformed both themselves and GPT-5.4. The authors acknowledge this as unrealized potential rather than something the study achieved.