# NOHARM recomputation — provenance and reproduction package

Ground Truth · published 2026-08-16 · accompanies *"2.9% or 8%? We recomputed the benchmark behind three '#1' claims"*

Everything in this directory is what the article's numbers were computed from, plus the code that computes them. Nothing here is derived from private material.

## Primary paper

arXiv 2512.01241, "First, do NOHARM" (ARISE network). Four versions were reviewed.

| Version | Posted (arXiv submission history) | SHA-256 of the PDF as retrieved |
|---|---|---|
| v1 | 2025-12-01 03:33:16 UTC | 89809b2f1bfdc05e49286a5bbb218a5a902907bc56579e75c98444b4872c5ada |
| v2 | 2025-12-17 21:54:50 UTC | a10288ba1f5a2fbb43146985589d2b92a5df19bf5e48ecb64dd6b4c63bd83ae6 |
| v3 | 2026-06-16 03:35:40 UTC | 071c72d28405e0fecba84af6b75d3a2dab8abc9521918f8a6136f701100e00b8 |
| v4 | 2026-07-13 18:47:38 UTC | 48db0ab0f713e4b15c28a9e54d581902d6d52314236a138b038b51e57f38ca3f |

Text was extracted with `pdftotext -layout`. Every figure in every version is a flattened raster image with no text layer; figure pages were rendered at 400 dpi and read directly, and the load-bearing panels were confirmed by eye.

## Data and code used

| Artifact | Value |
|---|---|
| Repository | github.com/ARISENetwork/noharm |
| Commit | 7b089bd3ad00d02f7b01b3f8e345397d2a55f926 |
| Licence | code MIT; data and figures CC-BY-4.0 |

Input files, verified by hash before any computation (the script aborts on mismatch):

| File | SHA-256 |
|---|---|
| severe-mode-counts.csv | e8aa13449fe22e26408af0040a97c51aedb064ecdfa3f9a767e836e1d9fedd29 |
| donoharm-case-performance.csv | 33c62eb40a7b55136f8c17658bb6d7d446512a3bf58765903d7dccb2e772fafa |
| donoharm-severe-full.json | f90938cd3f3a2db99e840ecbb961e46e967c8b2f5867cf9485d32e019e25152d |
| donoharm-rag-vs-generalist.json | d3101feae332833e539062591098524bd9cefbe3468ff9fe111c6f964c6d4873 |
| human-study.json | dc9f26cab986d9cfc1d244da526a231ab01619e9f159ac196ab3ea03b963d95e |

## What is in this directory

| File | What it is |
|---|---|
| `reproduce_all.py` | Regenerates every deterministic figure in the article. Verifies the five inputs against the hashes above, then recomputes and compares. |
| `verify_claims.py` | A second implementation written from scratch, without reference to the authors' analysis code. Includes the stratified cluster bootstrap, sign-flip permutation and exact McNemar bound. |
| `repro_stats_authors_MIT.py` | The authors' own statistics module (MIT), included so the multiple-comparisons results can be reproduced with their code rather than ours. |
| `reproduce_all.out` | Captured output of the deterministic run — `ALL CHECKS PASS`. |
| `top_tier.out` | Captured output of the authors' multiple-comparisons-with-the-best procedure, B=10,000, seed 0. |
| `independent.out` | Captured output of the independent implementation. |
| `hashes.txt` | SHA-256 of the three captured outputs. |

Environment for the captured runs: Python 3.13.2, macOS-26.5.2-arm64.

To reproduce:

```
python3 reproduce_all.py
```

## Results as captured

Multiple comparisons with the best (authors' procedure, clinical cohort):

- Severe-harm metric — leader AMBOSS LiSA; **all four tools in the best set**; Holm-adjusted p 0.5762 (Doximity), 0.5762 (OpenEvidence), 0.3357 (Glass Health).
- Severity-weighted F1, all 100 cases — leader AMBOSS LiSA; **best set = AMBOSS alone**; Holm-adjusted p 0.0406 (Doximity), 0.0003 (OpenEvidence), 0.0003 (Glass Health).
- Severity-weighted F1, 70 held-out cases — leader Doximity Ask; **best set = Doximity and AMBOSS**; Holm-adjusted p 0.4800 (AMBOSS).

Independent implementation: agrees on every point estimate and ordering. Its sign-flip permutation test gives p ≈ 0.056 for AMBOSS against Doximity on full-set F1, where the authors' bootstrap gives 0.041 — both are reported in the article. Exact two-sided McNemar bound for the 45-vs-40 resource-use comparison: minimum p = 0.0625, at complete nesting.

## Sources cited in the article

Primary: the four arXiv versions above; the ARISE analysis repository; the AMBOSS releases of 2026-02-12 and 2026-07-28; the Doximity release of 2026-07-15; the OpenEvidence release of 2026-07-20. Coverage referenced: Becker's Hospital Review (2026-07-21), Healthcare Huddle, Fortune (2026-07-29). Each was archived locally with a SHA-256 and a retrieval timestamp at the time of writing.

## Scope and limits

- The three funding and vendor-access statements discussed in the article are **not located in the rendered v4 PDF**. The arXiv source package and any ancillary files were not inspected, and a journal version may differ.
- The four Extended Data tables the paper cites are listed by title; their contents are not rendered in any of the four PDFs.
- Whether Doximity's "real-world clinical sample" denotes the 70-case held-out slice is Ground Truth's inference. The term is undefined in the release.
- The identity of the system that requested a repeat test is Ground Truth's inference from the figure.
- Bootstrap-derived p-values carry Monte-Carlo error; seeds are fixed in the scripts.

Corrections to this analysis will be recorded in the article and dated.