============================================================================== CLAIM 1 - mean per-case severe-harm fraction (sev_full/n), default prompt, 100 cases ============================================================================== nia-1.0 mean(sev_full/n) = 2.91% [binary any-severe = 8.0%, pooled = 2.91%] doxgpt-condensed mean(sev_full/n) = 4.82% [binary any-severe = 10.0%, pooled = 4.82%] openevidence mean(sev_full/n) = 5.09% [binary any-severe = 10.0%, pooled = 5.09%] glass-5.6-max mean(sev_full/n) = 5.36% [binary any-severe = 12.0%, pooled = 5.36%] ============================================================================== CLAIM 2 - severe-harm separability among RAG systems (stratified paired cluster bootstrap, 20000 draws) ============================================================================== leader (lowest severe rate): nia-1.0 (2.91%) glass-5.6-max obs diff = +2.45pp one-sided boot p = 0.0520 Holm-adj = 0.1562 openevidence obs diff = +2.18pp one-sided boot p = 0.0566 Holm-adj = 0.1562 doxgpt-condensed obs diff = +1.91pp one-sided boot p = 0.0880 Holm-adj = 0.1562 [sensitivity: unstratified bootstrap] glass-5.6-max: p=0.0583 (Holm 0.1748), openevidence: p=0.0656 (Holm 0.1748), doxgpt-condensed: p=0.0946 (Holm 0.1748) sign-flip permutation tests on per-case rate differences (vs nia-1.0): doxgpt-condensed mean diff = +1.91pp nonzero cases = 14 p(two-sided) = 0.2479 p(one-sided) = 0.1240 (exact) openevidence mean diff = +2.18pp nonzero cases = 13 p(two-sided) = 0.2231 p(one-sided) = 0.1116 (exact) glass-5.6-max mean diff = +2.45pp nonzero cases = 14 p(two-sided) = 0.1820 p(one-sided) = 0.0910 (exact) Holm-adjusted one-sided: glass-5.6-max: 0.2730, openevidence: 0.2730, doxgpt-condensed: 0.2730 wilcoxon doxgpt-condensed p = 0.2243 wilcoxon openevidence p = 0.1385 wilcoxon glass-5.6-max p = 0.1344 ============================================================================== CLAIM 3 - F1_weighted, all 100 cases (stratified paired cluster bootstrap, 20000 draws) ============================================================================== n cases = 100 nia-1.0 mean F1 = 0.8616 doxgpt-condensed mean F1 = 0.8451 openevidence mean F1 = 0.8002 glass-5.6-max mean F1 = 0.7974 leader (highest F1): nia-1.0 nia-1.0 vs openevidence obs diff = +0.0613 one-sided boot p = 0.00000 Holm-adj = 0.00000 nia-1.0 vs glass-5.6-max obs diff = +0.0641 one-sided boot p = 0.00000 Holm-adj = 0.00000 nia-1.0 vs doxgpt-condensed obs diff = +0.0165 one-sided boot p = 0.04025 Holm-adj = 0.04025 sign-flip permutation tests (leader minus other, per case): vs doxgpt-condensed mean diff = +0.0165 p(two-sided) = 0.11014 p(one-sided) = 0.05563 (MC) vs openevidence mean diff = +0.0613 p(two-sided) = 0.00001 p(one-sided) = 0.00000 (MC) vs glass-5.6-max mean diff = +0.0641 p(two-sided) = 0.00002 p(one-sided) = 0.00001 (MC) Holm-adjusted one-sided: openevidence: 0.00001, glass-5.6-max: 0.00003, doxgpt-condensed: 0.05563 ============================================================================== CLAIM 4 - F1_weighted, held_out subset (stratified paired cluster bootstrap, 20000 draws) ============================================================================== n cases = 70 doxgpt-condensed mean F1 = 0.8498 nia-1.0 mean F1 = 0.8490 glass-5.6-max mean F1 = 0.7976 openevidence mean F1 = 0.7857 leader (highest F1): doxgpt-condensed doxgpt-condensed vs openevidence obs diff = +0.0642 one-sided boot p = 0.00000 Holm-adj = 0.00000 doxgpt-condensed vs glass-5.6-max obs diff = +0.0523 one-sided boot p = 0.00000 Holm-adj = 0.00000 doxgpt-condensed vs nia-1.0 obs diff = +0.0008 one-sided boot p = 0.47425 Holm-adj = 0.47425 sign-flip permutation tests (leader minus other, per case): vs nia-1.0 mean diff = +0.0008 p(two-sided) = 0.93552 p(one-sided) = 0.46867 (MC) vs openevidence mean diff = +0.0642 p(two-sided) = 0.00001 p(one-sided) = 0.00001 (MC) vs glass-5.6-max mean diff = +0.0523 p(two-sided) = 0.00058 p(one-sided) = 0.00030 (MC) Holm-adjusted one-sided: openevidence: 0.00003, glass-5.6-max: 0.00061, nia-1.0: 0.46867 head-to-head doxgpt-condensed minus nia-1.0: mean diff = +0.00079, boot one-sided p = 0.4743, sign-flip two-sided p = 0.9361 ============================================================================== CLAIM 5 - human-study resource_breakdown ============================================================================== n = 202 #1 Provided AI assistant count= 90 file pct= 44.6 recomputed=44.55 #2 Google Search count= 55 file pct= 27.2 recomputed=27.23 #3 UpToDate count= 49 file pct= 24.3 recomputed=24.26 #4 OpenEvidence count= 45 file pct= 22.3 recomputed=22.28 #5 External AI (ChatGPT, Claude, Gemini, etc.) count= 40 file pct= 19.8 recomputed=19.80 #6 None count= 26 file pct= 12.9 recomputed=12.87 #7 PubMed / journals count= 17 file pct= 8.4 recomputed= 8.42 ============================================================================== CLAIM 6 - uncertainty on OpenEvidence 45 vs External AI 40 (n=202, multi-select, joint counts unknown) ============================================================================== paired-data view: overlap k ranges 0..40 k= 0: discordant=85 (b10=45, b01=40) exact McNemar one-sided p=0.3323 two-sided p=0.6646 k=10: discordant=65 (b10=35, b01=30) exact McNemar one-sided p=0.3101 two-sided p=0.6201 k=20: discordant=45 (b10=25, b01=20) exact McNemar one-sided p=0.2757 two-sided p=0.5515 k=30: discordant=25 (b10=15, b01=10) exact McNemar one-sided p=0.2122 two-sided p=0.4244 k=40: discordant= 5 (b10=5, b01=0) exact McNemar one-sided p=0.0312 two-sided p=0.0625 most favorable overlap for significance: k=40 (complete nesting), discordant=5, one-sided p=0.0312, two-sided p=0.0625 naive two-independent-proportions chi2 p = 0.5417 (45/202=22.3% vs 40/202=19.8%) conclusion: min possible two-sided p = 0.0625 > 0.05 -> no overlap structure makes the gap significant two-sided; only complete nesting (k=40) reaches p=0.0312 one-sided.