2.9% or 8%? We recomputed the benchmark behind three “#1” claims

On 15 July 2026, Doximity announced that its clinical AI tool had “ranked first among all AI systems evaluated on the study’s real-world clinical sample.” Five days later, OpenEvidence announced that physicians had chosen it “over every other AI chatbot combined.” Eight days after that, AMBOSS announced that it “ranks #1 again, outperforming OpenEvidence, Doximity, and Glass Health.”

All three cite NOHARM, a clinical-safety benchmark from ARISE, a network of Stanford Medicine and Harvard Medical School physicians. It is serious work: 100 real primary-care-to-specialist consultation cases, each presented in 11 variants, scored against rubrics carrying 12,747 annotations from 29 board-certified physicians.

It is also unusually generous with its data. The authors released the per-case scores, the analysis code and the bootstrap machinery under an open licence. So instead of parsing press releases, we recomputed the benchmark from that data at a pinned commit — once with the authors’ own code, once with an implementation written from scratch — and asked what the numbers support.

Becker’s Hospital Review reported the competing-claims story on 21 July, describing the four tools as “tied for the top spot.” Healthcare Huddle reported the no-significant-difference result. What follows is the arithmetic underneath both.

The headline number depends on what counts as a case

AMBOSS’s release says its tool showed “potential for serious harm in just 2.9% of cases.” The figure is in the paper and we reproduce it exactly. It is not, however, a share of the 100 clinical cases.

Each base case was presented eleven ways — the original plus ten physician-written perturbations. The released metric takes, for each case, the fraction of those eleven variants containing a potentially severe error, then averages across cases. For AMBOSS that is 2.91%. Ask the other obvious question of the same rows — in what share of the 100 base cases did the system produce at least one severe error? — and the answer is 8%.

Neither number is wrong. They answer different questions, and a benchmark measuring robustness to rephrasing has good reason to count variants. But the gap is large and it is not constant across systems. The two orderings are broadly similar overall — Spearman ρ = 0.90 across the 45 models with default-prompt rows — while individual systems move a lot. Using midranks, 21 models shift at least three places and 17 shift at least five. Claude Sonnet 5 moves from midrank 19 to 34.5; GPT-5.4 mini goes the other way, from 32 to 18.5.

The direction is worth being precise about. Consider two systems with an identical 5% variant rate: one errs on all eleven variants of five cases, the other on a single variant of fifty-five. On the released metric they are indistinguishable; on base cases the second is 55% against the first’s 5%. The variant metric therefore flatters systems whose errors are spread thinly across many cases, and is hardest on those that fail the same case repeatedly. Among the four clinical tools, AMBOSS has the most dispersed errors by this measure — a ratio of 2.8 between its base-case and variant rates, against 1.2 for the model with the highest severe-error rate on the released variant measure. Whether thin dispersion or concentrated failure is worse depends on what you are buying the tool for. Neither is what a reader hears in “2.9% of cases.”

On its own safety measure, the benchmark cannot separate the four

The measure NOHARM is named for is the rate of potentially severe harmful errors. Here are the four clinical tools on it, with the authors’ own bootstrap intervals, against the general-purpose comparators the paper itself selected.

The clinical tools sit below the general-purpose models, and that gap is the paper’s headline finding — every company is entitled to cite it. Among the four, the intervals overlap substantially. Running the authors’ multiple-comparisons procedure on their own data returns all four in the best set; the three non-leading tools carry Holm-adjusted p-values of 0.34 to 0.58 against the leader. The paper says it plainly: “no statistically significant differences were detected among them.”

That is a failure to establish a winner, not proof the tools are equal — the study was not powered for equivalence. But on this measure, the released analysis supports no claim that any one of the four is safest.

Each claim, and the number under it

AMBOSS has the strongest case. On severity-weighted F1 — the composite accuracy-and-completeness score — it holds the highest value in the released data — 0.8615 on a 0–1 scale, the 86.15% its release quotes; our unrounded recomputation is 0.86155 — and is the sole occupant of the best set. Both our implementations agree on the estimates and the ordering. The separation from Doximity, though, is test-dependent: the authors’ stratified paired bootstrap with Holm correction gives an adjusted p of 0.041; a paired sign-flip permutation test on the same per-case differences gives 0.056. The point difference is 0.016 F1, and the paper specifies no clinically meaningful margin for this score.

Doximity said it ranked first on the “real-world clinical sample.” There is a released slice where that holds: on the 70 cases the authors label held-out, Doximity Ask has the highest severity-weighted F1 of any system in the data. The margin over AMBOSS is 0.0008.

The lead changes hands between the full set and that subset on less than a thousandth of a point, and the authors’ analysis places both tools in the same best set there. The release never defines “real-world clinical sample” and the paper has no section by that name, so the mapping is ours.

OpenEvidence’s claim is not from the benchmark at all. It comes from the companion randomized study, in which 101 physicians worked real cases and one arm let them use any resource they liked.

OpenEvidence appears in 45 of 202 records; the combined external-AI category appears in 40. The release’s body limits the comparison to “every other external AI combined,” and on that scope the arithmetic holds — the study-provided assistant sits outside that category by definition. The body also notes the studies are not peer reviewed and “should be taken with a grain of salt.”

Two things sit alongside it. The headline uses the broader phrase “every other AI chatbot combined,” and the assistant that the headline could be read to include appears in 90 records — twice OpenEvidence’s count. And a five-record gap across 202 multi-select records will not carry a statistical claim: joint usage was not released, and treating the records as independent paired observations, no overlap structure consistent with the published margins reaches significance under the exact McNemar test. The lowest achievable p is 0.0625, and only under complete nesting.

The preprint has been moving under the claims

Those percentages are not four estimates of one quantity: each is a maximum across whichever systems that version evaluated, and the roster changed. Ordinary for a preprint — it matters only because the figures travel. On 29 July, Fortune quoted “76.6% of harmful errors were omissions,” a December figure; v3 and v4 say more than 80% of severe errors, a differently worded quantity.

The roster moved too. AMBOSS and Glass Health were evaluated and named from the first version. AMBOSS’s February 2026 release cited that earlier record for its first “#1” claim; July’s says “again.” Doximity and OpenEvidence appear in v4.

Three statements about funding and vendor access appear in the earlier versions and are not located in the latest: that the authors paid for all token usage including the clinical tools; that they received no financial support from any organisation whose model was benchmarked; and that companies were given no benchmark content and no opportunity to tune. We searched for the ideas as well as the words — payment, tokens, cost, reimbursement, financial support, compensation, tuning, advance access. In v4 the only matches are unrelated: “half credit” in the scoring rubric, “paid consultant” in the disclosures. Because the figures carry no searchable text, we also rendered and read every figure page rather than relying on extraction alone.

The same version adds disclosures the earlier ones lacked: that code and 30% of the cases went public in June with an acknowledged risk of “post-hoc model optimizations”; the contamination probes the authors ran; that manual testing of the products scored below the API results in every case where both exist; and that one system asked for a repeat test mid-update, with both results published.

Two details visible only in the figures

Every figure in this preprint is a flattened image with no text layer, so nothing below can be found by searching the PDF. We rendered each figure page at 400 dpi, read it with optical character recognition, and confirmed the two panels described here by eye against the rendered images.

In the figure carrying the composite scores, an asterisk sits beside AMBOSS LiSA’s 86.15 — the number in its release — and beside OpenEvidence and Glass Health. The caption defines it: models “tested after open-source release of sample cases.” Doximity Ask carries none. It is a timing disclosure and nothing more; the authors’ probes found no evidence of the memorisation pattern they tested for.

In the manual-testing panel, Doximity Ask appears twice: 59.6 on 15 May, 75.8 on 21 May. The caption says one unnamed system requested a repeat while updates were underway, and that both results are shown. Doximity Ask is the only product displayed twice, so we infer the caption refers to it. The inference is about identity alone — the figure says nothing about why the scores differ, and the two entries may be different builds.

What it comes to

Every release here points at a real number, the benchmark is real, and the authors released more than most teams do.

What the data does not contain is a safety ranking of the four. On the severe-harm measure, the authors’ own procedure leaves all four in the best set and detects no significant difference among them — which is not the same as showing they are equal. On the composite score AMBOSS does lead, but its margin survives one test and not another. On the subset where the order flips, it flips by 0.0008. And the headline safety figure moves by a factor of nearly three depending on whether you count variants or cases.

None of it shows that any of these products performs poorly; the underlying numbers each release points to are real. What it shows is that the benchmark is a weaker instrument for ranking than three press releases imply — a finding about the state of clinical-AI evaluation rather than about any one product.

Method and data

Figures attributed to the paper are quoted from arXiv 2512.01241 at the version stated. Because every figure in every version is a flattened raster image, figure-level findings come from 400-dpi renders of each figure page, read by optical character recognition and confirmed visually for the panels cited; the renders are reproducible from the archived PDFs. Recomputed figures come from the authors’ released data at commit 7b089bd, verified by SHA-256 before computing, using their published analysis code and an independent implementation written without reference to it. Both agree on the point estimates and orderings; where the inferential procedures differ — the 0.041 against 0.056 above — we report both. Three runs are captured in full: the deterministic recomputation, the authors’ multiple-comparisons procedure, and the independent implementation. Published alongside this piece: the scripts, the captured output with hashes and environment, and provenance notes giving a hash and retrieval timestamp for every document cited.

This piece is based entirely on public documents and on data the authors released themselves. Three questions can be answered only by the people involved, and are labelled as inference where they appear: what Doximity means by “real-world clinical sample”; which system requested the repeat test; and whether the earlier funding and access practices remained in effect. If ARISE or any company tells us we have read something wrong, we will correct it here and say what changed.

Two disclosures. GPT-5.6 Sol, used as an adversarial reviewer of this analysis, and Claude Fable 5, used in drafting, are both systems NOHARM evaluates. And we corrected our own errors along the way: we had believed all four clinical tools first appeared in the latest version, when two were there from the first; we proposed an arithmetic explanation for the early confidence intervals that does not hold; and we had a rounding error in the gap between the two leading tools. All three are in the published methods notes.