Ground Truth

Investigation · Data analysis

Google made Gemma “medical.” We gave it the job.

Google ships MedGemma as “our most capable open models for health AI development.” It is genuinely better than the general Gemma it was built from — on the medical exams it was tuned to pass. So we stopped giving it exams and gave it the work a clinical decision-support tool actually does: recognise an emergency, build a differential, populate a record, write a note. We ran both, on eight open models, through one harness.

Why we didn't just re-run the benchmark

Every headline number about medical AI — MedGemma's included — comes from multiple-choice question banks: MedQA, MedMCQA, PubMedQA, MMLU. They are the format a medical fine-tune is explicitly optimised for, and they are the weakest possible proxy for clinical decision support, which is not a test. Real CDSS is extraction, triage, differential, and documentation — none of it answerable with a letter A–D.

So we ran two benchmarks, on the same eight open models, through one harness with one rule: identical prompts, no per-model tuning. (This matters: the standard finding in this literature is that tuning the prompt for the medical model alone inflates its apparent win rate over the base it is compared against.) Everything ran locally, greedy-decoded, at Q4_K_M quantisation. Every model output is logged. The eight models, size-matched into two tiers: at 4B, MedGemma-4B, MedGemma-1.5-4B, its base Gemma-3-4B, and the newer general Gemma-4-E4B; at 27B, MedGemma-27B, its base Gemma-3-27B, and the newer general Gemma-4-26B / 31B. The comparison that has never been published is the obvious one — does making Gemma “medical” beat just using a newer Gemma? We searched Google’s Gemma 4 model card on 27 July 2026 and could find no medical text benchmark reported for Gemma 4 at all — the one medical result on it is the multimodal split of MedXpertQA.

Part 1 — The exam it was built to pass

On the multiple-choice tests, MedGemma does what Google says. Against its own base, at matched size, it wins: MedQA 58 vs 49 at 4B (a real +9-point gain, 95% CI [+2, +16]), 70 vs 66 at 27B; PubMedQA +16 at 4B. The medical fine-tune is real. That is the finding that makes the rest credible — we are not claiming MedGemma is broken.

But two things sit next to that win. A newer general model beats it on medicine: Gemma-4-31B, with no medical training, beats MedGemma-27B on six of seven medical benchmarks. And on the hardest, least-contaminated set, the badge is worth nothing.

On hard, uncontaminated medical reasoning, the medical badge does nothing
↑ higher is better
0204060MedGemma 4BMedGemma 4B: 12.8 · medical fine-tune12.8Gemma 3 4BGemma 3 4B: 13.2 · its base — statistically tied13.2Gemma 4 E4BGemma 4 E4B: 10 · newer general10MedGemma 27BMedGemma 27B: 13.3 · medical fine-tune13.3Gemma 3 27BGemma 3 27B: 16.7 · its base — tied with MedGemma (CI crosses 0)16.7Gemma 4 31BGemma 4 31B: 48.7 · newer general — +35pp over MedGemma-27B, CI [+27,+44]48.7MedXpertQA accuracy (%)
Source: Ground Truth original benchmark, July 2026. Local, greedy-decoded, Q4_K_M, identical prompts. Data + code: github.com/groundtruth-health/medgemma-benchmark
Data table
ItemMedXpertQA accuracy (%)Note
MedGemma 4B12.8medical fine-tune
Gemma 3 4B13.2its base — statistically tied
Gemma 4 E4B10newer general
MedGemma 27B13.3medical fine-tune
Gemma 3 27B16.7its base — tied with MedGemma (CI crosses 0)
Gemma 4 31B48.7newer general — +35pp over MedGemma-27B, CI [+27,+44]
Read before citing
  • n=150 per model, single-pass, greedy, Q4_K_M quantised — absolute scores sit below fp16 numbers; the comparison is internally consistent.
  • MedGemma-27B (13.3) is NOT significantly below its base (16.7) — that difference is within the confidence interval. The only real gap is Gemma-4-31B's, which is large.
  • MedXpertQA is the least publicly-contaminated set we ran; the medical fine-tune's advantage on the older benchmarks does not appear here.

On MedXpertQA, the medical fine-tune's advantage over the older public benchmarks vanishes: MedGemma-27B (13) is statistically tied with its own base (17), while the newer general Gemma-4-31B scores nearly four times higher, at 49. Google's own report concedes the mechanism, verbatim: these benchmarks “raise the possibility of test data leakage, as [they] are publicly available.” And the fine-tune has a cost, in Google's own numbers (which we cite rather than our own, where at our sample size the effect is a statistical tie): on general reasoning it scores below the base it was built from.

The whole exam picture, at matched size and with the identical prompt, reads the same way. The badge helps against its own base; a newer general Gemma catches or beats it; and the gap the badge is supposed to close shows mainly on the older, potentially-leaked benchmarks — not on the hard one.

BenchmarkMedG‑4BG3‑4BG4‑E4BMedG‑27BG3‑27BG4‑31B
MedQA58.048.861.670.066.069.3
MedMCQA50.844.853.264.065.377.3
PubMedQA78.462.476.078.770.783.3
MMLU-Med62.058.070.078.771.384.7
MedXpertQA (hard)12.813.210.013.316.748.7
MMLU-Pro (health)38.434.043.655.354.766.7
MMLU-Pro (general)30.032.450.859.344.070.7

Single-pass accuracy (%), matched size, identical prompt. MedG = MedGemma (medical), G3 = its base, G4 = the newer general model. Full 89-cell matrix (incl. reasoning & best-of-5 regimes) in the repository.

Part 2 — But medicine isn't an exam

Here is the test that matters. We gave every model a patient and asked one question: rate the urgency, 1 (immediately life-threatening) to 5 (routine). Then we looked only at the cases that are genuinely, unambiguously life-threatening — anaphylaxis, heart attack, acute pulmonary oedema, pulmonary embolism, laryngospasm.

Only the newer general models reliably recognise an emergency
↑ higher is better
020406080100Gemma 4 26BGemma 4 26B: 72 · newer general72Gemma 4 31BGemma 4 31B: 68 · newer general68Gemma 4 E4BGemma 4 E4B: 60 · newer general60MedGemma-1.5 4BMedGemma-1.5 4B: 36 · newest medical — flags a third, no severity discrimination (r=0.10)36MedGemma 4BMedGemma 4B: 24 · medical24Gemma 3 27BGemma 3 27B: 8 · base — never flags urgent (answers '3')8MedGemma 27BMedGemma 27B: 8 · medical8Gemma 3 4BGemma 3 4B: 0 · base — never flags urgent (answers '3')0Life-threatening (severity-1) cases flagged urgent (%)
Source: Ground Truth original benchmark, July 2026. Local, greedy-decoded, Q4_K_M, identical prompts. Data + code: github.com/groundtruth-health/medgemma-benchmark
Data table
ItemLife-threatening (severity-1) cases flagged urgent (%)Note
Gemma 4 26B72newer general
Gemma 4 31B68newer general
Gemma 4 E4B60newer general
MedGemma-1.5 4B36newest medical — flags a third, no severity discrimination (r=0.10)
MedGemma 4B24medical
Gemma 3 27B8base — never flags urgent (answers '3')
MedGemma 27B8medical
Gemma 3 4B0base — never flags urgent (answers '3')
Read before citing
  • 25 unambiguous severity-1 emergencies (anaphylaxis, STEMI, acute pulmonary oedema, PE, laryngospasm). Small n → wide intervals: Gemma-4-31B over MedGemma-27B is +60pp, 95% CI [+40,+80].
  • DDXPlus severity is a rule-based knowledge-base label, not a clinician's triage; the severity→acuity mapping is ours; the vignettes are synthetic and machine-translated from French (one describes 'haunting' chest pain).
  • Four confounds were tested and cleared: scale inversion (control run), prompt parity, chat-template delivery, and token starvation — see results_cdss/INVERSION_CONTROL.md.

No version of MedGemma reliably recognises them. MedGemma-4B rated nearly half of the life-threatening cases as non-urgent; its predictions run backwards (it gives less-urgent scores to more-urgent patients). Its base is no better — Gemma-3-4B flagged zero of the 25 emergencies as urgent, because it answers “3” to almost everything and never commits. The models that actually recognise an emergency are the newer general ones, by a margin no confidence interval erases (Gemma-4-31B over MedGemma-27B: +60 points, CI [+40, +80]). We stress-tested this against four confounds — a reversed-scale control, prompt parity, chat-template delivery, and a token budget — before reporting it.

Modelflagged urgentcalled routinePearson r
Gemma-4-26B72%8%+0.50
Gemma-4-31B68%32%+0.43
Gemma-4-E4B60%8%+0.08
MedGemma-1.5-4B36%4%+0.10
MedGemma-4B24%44%-0.37
MedGemma-27B8%28%+0.16
Gemma-3-27B8%0%+0.15
Gemma-3-4B0%24%-0.18

The 25 unambiguous life-threatening (severity-1) emergencies; percentages are over all 25 (an abstention is a failure to flag). The last column is the Pearson correlation of predicted vs true urgency across all cases: positive = tracks severity, near zero = doesn’t, negative = backwards. Only Gemma-4-26B and 31B discriminate; Gemma-4-E4B flags emergencies but cannot rank them (r=0.08).

One case makes it concrete. Shown a patient with a known severe food allergy, swelling of the face and mouth, a rash, and shortness of breath — the textbook picture of anaphylaxis — MedGemma-4B rated the urgency 4 out of 5 (“semi-urgent”). Gemma-4-31B rated it 1. Because a claim like this should be checkable, here is that case and others below — the exact vignette each model was given and its exact, unedited answer. The inputs are synthetic and deliberately noisy; every model saw the identical text; nothing is cherry-picked.

Real cases from the DDXPlus benchmark and every model's real answer to 'rate urgency 1 (life-threatening) to 5 (routine)'. The findings are the model's exact inputs, rendered from the dataset's evidence items as plain statements for readability. DDXPlus is synthetic, so a patient may carry incidental findings; every model saw the identical input. Within this set nothing is cherry-picked: one case per emergency type plus routine controls, all eight models on each.

1 = immediately life-threatening · 2 = emergency · 3 = urgent · 4 = semi-urgent · 5 = routine

EMERGENCY Anaphylaxis — true urgency: 1 (immediately life-threatening)

What the model was told about the patient (the dataset's findings, as plain statements):

  • Lightheaded and dizzy or you are about to faint
  • A known severe food allergy
  • Diarrhea or an increase in stool frequency
  • Pain somewhere
  • Pain character: stabbing
  • Pain location: flank(L)
  • Pain location: iliac fossa(R)
  • Pain location: hypochondrium(R)
  • Pain location: pubis
  • Pain intensity (/10): 8
  • Pain localization (/10): 1
  • Pain onset speed (/10): 6
  • Shortness of breath or difficulty breathing
  • Any lesions, redness or problems on your skin
  • Rash colour: pink
  • Rash swelling (/10): 3
  • Affected region: biceps(R)
  • Affected region: biceps(L)
  • Affected region: mouth
  • Affected region: thyroid cartilage
  • Affected region: cervical spine
  • Rash pain (/10): 0
  • Is the lesion (or are the lesions) larger than 1cm
  • Itch severity (/10): 10
  • Nauseous or vomiting
  • Swelling in one or more areas of your body
  • Swelling location: forehead
  • Swelling location: cheek(R)
  • Swelling location: cheek(L)
  • Lose consciousness
  • More likely to develop common allergies
ModelTypeUrgency 1–5Verdict
MedGemma-4Bmedical4called ROUTINE
MedGemma-1.5-4Bmedical (newest)3called 'urgent' (3)
Gemma-3-4Bbase4called ROUTINE
Gemma-4-E4Bgeneral (newer)2flagged urgent
MedGemma-27Bmedical3called 'urgent' (3)
Gemma-3-27Bbase3called 'urgent' (3)
Gemma-4-26Bgeneral (newer)1flagged urgent
Gemma-4-31Bgeneral (newer)1flagged urgent
EMERGENCY Acute pulmonary edema — true urgency: 1 (immediately life-threatening)

What the model was told about the patient (the dataset's findings, as plain statements):

  • Pain somewhere
  • Ever had fluid in your lungs
  • Significantly increased sweating
  • Pain character: tedious
  • Pain character: tender
  • Pain character: heavy
  • Pain location: ankle(R)
  • Pain location: ankle(L)
  • Pain location: side of the chest(R)
  • Pain location: side of the chest(L)
  • Pain location: breast(L)
  • Pain intensity (/10): 8
  • Pain radiates to: biceps(R)
  • Pain radiates to: jaw
  • Pain radiates to: scapula(R)
  • Pain radiates to: shoulder(R)
  • Pain radiates to: shoulder(L)
  • Pain localization (/10): 3
  • Pain onset speed (/10): 2
  • Shortness of breath or difficulty breathing
  • High blood pressure or take medications to treat high blood pressure
  • Ever had a heart attack or have angina (chest pain)
  • Heart failure
  • Swelling in one or more areas of your body
  • Swelling location: ankle(R)
  • Swelling location: calf(R)
  • Swelling location: tibia(L)
  • New fatigue, generalized and vague discomfort, diffuse (widespread) muscle aches or a change in your general well-being
  • Are your symptoms worse when lying down and alleviated while sitting up
ModelTypeUrgency 1–5Verdict
MedGemma-4Bmedical3called 'urgent' (3)
MedGemma-1.5-4Bmedical (newest)3called 'urgent' (3)
Gemma-3-4Bbase4called ROUTINE
Gemma-4-E4Bgeneral (newer)2flagged urgent
MedGemma-27Bmedical3called 'urgent' (3)
Gemma-3-27Bbase3called 'urgent' (3)
Gemma-4-26Bgeneral (newer)1flagged urgent
Gemma-4-31Bgeneral (newer)1flagged urgent
EMERGENCY Possible NSTEMI / STEMI — true urgency: 1 (immediately life-threatening)

What the model was told about the patient (the dataset's findings, as plain statements):

  • Pain somewhere
  • Significantly increased sweating
  • Pain character: tedious
  • Pain character: sharp
  • Pain character: sickening
  • Pain location: lower chest
  • Pain location: upper chest
  • Pain location: breast(R)
  • Pain location: breast(L)
  • Pain location: epigastric
  • Pain intensity (/10): 5
  • Pain radiates to: biceps(L)
  • Pain radiates to: thyroid cartilage
  • Pain radiates to: under the jaw
  • Pain radiates to: shoulder(R)
  • Pain radiates to: shoulder(L)
  • Pain localization (/10): 8
  • Pain onset speed (/10): 9
  • Shortness of breath or difficulty breathing
  • Diabetes
  • Significantly overweight compared to people of the same height as you
  • Smoke cigarettes
  • Constantly feel fatigued or have non-restful sleep
  • High blood pressure or take medications to treat high blood pressure
  • Ever had a heart attack or have angina (chest pain)
  • A problem with poor circulation
  • Nauseous or vomiting
  • Close family members who had a cardiovascular disease problem before the age of 50
ModelTypeUrgency 1–5Verdict
MedGemma-4Bmedical4called ROUTINE
MedGemma-1.5-4Bmedical (newest)5called ROUTINE
Gemma-3-4Bbase4called ROUTINE
Gemma-4-E4Bgeneral (newer)2flagged urgent
MedGemma-27Bmedical3called 'urgent' (3)
Gemma-3-27Bbase2flagged urgent
Gemma-4-26Bgeneral (newer)1flagged urgent
Gemma-4-31Bgeneral (newer)1flagged urgent
EMERGENCY Larygospasm — true urgency: 1 (immediately life-threatening)

What the model was told about the patient (the dataset's findings, as plain statements):

  • A high pitched sound when breathing in
  • Ever been diagnosed with gastroesophageal reflux
ModelTypeUrgency 1–5Verdict
MedGemma-4Bmedical2flagged urgent
MedGemma-1.5-4Bmedical (newest)2flagged urgent
Gemma-3-4Bbase3called 'urgent' (3)
Gemma-4-E4Bgeneral (newer)3called 'urgent' (3)
MedGemma-27Bmedical5called ROUTINE
Gemma-3-27Bbase3called 'urgent' (3)
Gemma-4-26Bgeneral (newer)5called ROUTINE
Gemma-4-31Bgeneral (newer)5called ROUTINE
ROUTINE Panic attack — true urgency: 5 (routine)

What the model was told about the patient (the dataset's findings, as plain statements):

  • Pain somewhere
  • Do any members of your immediate family have a psychiatric illness
  • Significantly increased sweating
  • Pain character: cramping
  • Pain character: sharp
  • Pain location: flank(R)
  • Pain location: iliac fossa(L)
  • Pain location: hypochondrium(R)
  • Pain location: hypochondrium(L)
  • Pain location: belly
  • Pain intensity (/10): 5
  • Pain localization (/10): 10
  • Pain onset speed (/10): 8
  • You are (or were) choking or suffocating
  • Drink alcohol excessively or have an addiction to alcohol
  • Suffer from chronic anxiety
  • Lightheaded and dizzy or you are about to faint
  • You are dying or were you afraid that you were about do die
  • Asthma or have you ever had to use a bronchodilator in the past
  • Your heart is beating fast (racing), irregularly (missing a beat) or feel palpitations
  • Recently had numbness, loss of sensation or tingling, in both arms and legs and around your mouth
ModelTypeUrgency 1–5Verdict
MedGemma-4Bmedical3called 'urgent' (3)
MedGemma-1.5-4Bmedical (newest)3called 'urgent' (3)
Gemma-3-4Bbase3called 'urgent' (3)
Gemma-4-E4Bgeneral (newer)2flagged urgent
MedGemma-27Bmedical3called 'urgent' (3)
Gemma-3-27Bbase3called 'urgent' (3)
Gemma-4-26Bgeneral (newer)3called 'urgent' (3)
Gemma-4-31Bgeneral (newer)2flagged urgent
ROUTINE Panic attack — true urgency: 5 (routine)

What the model was told about the patient (the dataset's findings, as plain statements):

  • You are dying or were you afraid that you were about do die
  • Anxious
  • Do any members of your immediate family have a psychiatric illness
  • Pain somewhere
  • Pain character: sharp
  • Pain location: flank(R)
  • Pain location: iliac fossa(R)
  • Pain location: iliac fossa(L)
  • Pain location: hypochondrium(R)
  • Pain location: belly
  • Pain intensity (/10): 7
  • Pain localization (/10): 8
  • Pain onset speed (/10): 5
  • You are (or were) choking or suffocating
  • Drink alcohol excessively or have an addiction to alcohol
  • Ever been diagnosed with depression
  • Lightheaded and dizzy or you are about to faint
  • Ever had a migraine or is a member of your family known to have migraines
  • Asthma or have you ever had to use a bronchodilator in the past
  • Suffer from fibromyalgia
  • Nauseous or vomiting
  • Your heart is beating fast (racing), irregularly (missing a beat) or feel palpitations
  • Recently had numbness, loss of sensation or tingling, in both arms and legs and around your mouth
  • You are detached from your own body or your surroundings
  • Recent travel: North America
ModelTypeUrgency 1–5Verdict
MedGemma-4Bmedical3called 'urgent' (3)
MedGemma-1.5-4Bmedical (newest)5called ROUTINE
Gemma-3-4Bbase3called 'urgent' (3)
Gemma-4-E4Bgeneral (newer)2flagged urgent
MedGemma-27Bmedical3called 'urgent' (3)
Gemma-3-27Bbase3called 'urgent' (3)
Gemma-4-26Bgeneral (newer)2flagged urgent
Gemma-4-31Bgeneral (newer)2flagged urgent

Part 3 — Where the badge helps, and where it doesn't

The honest picture is not “MedGemma bad.” It is mixed, and the mix is the story. Here is every clinical task at once, all eight models, before we walk through them one at a time.

ModelDiff.
top-1		Section
class.		JSON
ok		ROUGE-1Concept
F1		Halluc.
↓
MedGemma-4B41489951.733.32.07
MedGemma-1.5-4B24479649.332.61.58
Gemma-3-4B27447851.530.93.50
Gemma-4-E4B36469956.736.41.15
MedGemma-27B41529754.533.11.73
Gemma-3-27B44549953.532.31.55
Gemma-4-26B57569856.333.21.68
Gemma-4-31B47559856.834.90.86

Every clinical task, all eight models. Percentages except the last column (unsupported statements per 1,000 characters, judged; lower is better). Differential, section classification, JSON and note metrics are deterministic; hallucination is the neural cross-check.

Start with the differential, where the badge genuinely helps — at 4B.

On differential diagnosis, the medical badge helps — but only at 4B
↑ higher is better
0204060MedGemma 4BMedGemma 4B: 40.8 · medical — +13.9pp over its base, CI [+9,+19] (real)40.8Gemma 3 4BGemma 3 4B: 26.9 · its base26.9Gemma 4 E4BGemma 4 E4B: 35.9 · newer general (4B) — tied with MedGemma-4B, CI [-1,+11]35.9MedGemma 27BMedGemma 27B: 40.8 · tied with its base (CI crosses 0) — gain gone at scale40.8Gemma 3 27BGemma 3 27B: 44.5 · its base44.5Gemma 4 26BGemma 4 26B: 56.7 · newer general — leads everything, +16pp over MedGemma-27B56.7Gemma 4 31BGemma 4 31B: 46.9 · newer general46.9Correct diagnosis ranked #1 of 5 (GTPA@1, %)
Source: Ground Truth original benchmark, July 2026. Local, greedy-decoded, Q4_K_M, identical prompts. Data + code: github.com/groundtruth-health/medgemma-benchmark
Data table
ItemCorrect diagnosis ranked #1 of 5 (GTPA@1, %)Note
MedGemma 4B40.8medical — +13.9pp over its base, CI [+9,+19] (real)
Gemma 3 4B26.9its base
Gemma 4 E4B35.9newer general (4B) — tied with MedGemma-4B, CI [-1,+11]
MedGemma 27B40.8tied with its base (CI crosses 0) — gain gone at scale
Gemma 3 27B44.5its base
Gemma 4 26B56.7newer general — leads everything, +16pp over MedGemma-27B
Gemma 4 31B46.9newer general
Read before citing
  • n=245, top-5 differential from the closed 49-pathology DDXPlus list; GTPA@1 = the single most-likely diagnosis is correct.
  • This is the clearest place the medical fine-tune helps — kept in deliberately. The +14pp gain over its base at 4B is real; against the newer general Gemma-4-E4B at 4B it is a statistical tie (+4.9, CI [-1.2,+11.0]); at 27B the badge's gain over its base is gone and Gemma-4 leads.

Asked to name the five most likely conditions from a fixed list, MedGemma-4B puts the correct diagnosis first 41% of the time against its base's 27% — a real +14-point gain (95% CI [+9, +19]). That is a genuine win over the model it was built from. It is not, though, a win over a newer general Gemma: at 4B, MedGemma-4B and Gemma-4-E4B are a statistical tie (41 vs 36, CI crosses zero). And even that much is a 4B phenomenon — at 27B the badge's edge disappears into noise (MedGemma-27B ties its base), and the newer Gemma-4-26B leads everything.

On structured output — emitting valid, schema-correct JSON, the workhorse of any real integration — we predicted the medical fine-tune would fail. It did the opposite, and this is the clearest win for medical fine-tuning in the study. Every model produces schema-correct JSON about 99% of the time — except the base Gemma-3-4B, at 79%. So the medical tuning did not break structured output; it repaired its base's, a +20-point gain. (The claim we could have cited for the opposite — that MedGemma emits garbled output — turned out, when we checked the source, to be a number that appears in no paper at all. We measured instead.)

Then note generation: turn a full doctor-patient transcript into a clinical note. On word overlap with a reference note, the newer general Gemma-4 models write the closest match — the three of them tied at the top; the medical fine-tune merely ties its base.

Clinical note quality: how closely the note matches a reference note
↑ higher is better
0204060MedGemma 4BMedGemma 4B: 51.7 · medical — ties its base51.7Gemma 3 4BGemma 3 4B: 51.5 · its base51.5Gemma 4 E4BGemma 4 E4B: 56.7 · newer general56.7MedGemma 27BMedGemma 27B: 54.5 · medical54.5Gemma 3 27BGemma 3 27B: 53.5 · base53.5Gemma 4 26BGemma 4 26B: 56.3 · newer general56.3Gemma 4 31BGemma 4 31B: 56.8 · newer general — highest (tied with E4B/26B)56.8MedGemma-1.5 4BMedGemma-1.5 4B: 49.3 · newest medical — lowest49.3ROUGE-1 vs the reference note (%)
Source: Ground Truth original benchmark, July 2026. ACI-Bench, n=120, ROUGE-1 (Google rouge_score). Data + code: github.com/groundtruth-health/medgemma-benchmark
Data table
ItemROUGE-1 vs the reference note (%)Note
MedGemma 4B51.7medical — ties its base
Gemma 3 4B51.5its base
Gemma 4 E4B56.7newer general
MedGemma 27B54.5medical
Gemma 3 27B53.5base
Gemma 4 26B56.3newer general
Gemma 4 31B56.8newer general — highest (tied with E4B/26B)
MedGemma-1.5 4B49.3newest medical — lowest
Read before citing
  • ROUGE measures word overlap with a single human reference note, NOT clinical correctness, completeness, or safety — a note can score well while omitting a critical fact. It is the field-standard automatic metric; we pair it with clinical-concept overlap and the faithfulness judge above.
  • The three newer general Gemma-4 models lead the field (56.3–56.8) and are tied with one another at the top; Gemma-4-E4B beats the 4B medical fine-tune by +5.0 (CI [+3.9,+6.2]). At 4B the medical fine-tune ties its base (51.7 vs 51.5, CI crosses 0).

But word overlap is a weak proxy — it cannot tell whether a note is correct, only whether it uses similar words. So we asked a harder question, with an independent judge (OpenAI's gpt-5.5, a different vendor from every model here, blinded to which model wrote what): of everything each note asserts, how much does the transcript actually support? Every unsupported statement the judge flags is checked back against the note (case- and whitespace-insensitive) and dropped if it does not actually appear there — so a judge that invents an accusation is caught, not counted.

How often each model invents a fact the transcript doesn't support
↓ lower is better
01234Gemma 4 31BGemma 4 31B: 0.86 · newer general — the most faithful0.86Gemma 4 E4BGemma 4 E4B: 1.15 · newer general1.15Gemma 3 27BGemma 3 27B: 1.55 · base1.55MedGemma-1.5 4BMedGemma-1.5 4B: 1.58 · newest medical1.58Gemma 4 26BGemma 4 26B: 1.68 · newer general1.68MedGemma 27BMedGemma 27B: 1.73 · medical1.73MedGemma 4BMedGemma 4B: 2.07 · medical — fewer than its base (CI [-2.0,-1.2])2.07Gemma 3 4BGemma 3 4B: 3.5 · base — the worst in the study3.5Unsupported statements per 1,000 characters
Source: Ground Truth original benchmark, July 2026. Note faithfulness graded by OpenAI gpt-5.5 (an independent vendor, blinded); every flagged quote machine-verified to appear in the note. Data + code: github.com/groundtruth-health/medgemma-benchmark
Data table
ItemUnsupported statements per 1,000 charactersNote
Gemma 4 31B0.86newer general — the most faithful
Gemma 4 E4B1.15newer general
Gemma 3 27B1.55base
MedGemma-1.5 4B1.58newest medical
Gemma 4 26B1.68newer general
MedGemma 27B1.73medical
MedGemma 4B2.07medical — fewer than its base (CI [-2.0,-1.2])
Gemma 3 4B3.5base — the worst in the study
Read before citing
  • This is a NEURAL, secondary metric (an LLM judge), reported alongside the deterministic metrics, not instead of them. The deterministic finding it supports: the medical fine-tune reduces invention vs its own base at 4B (2.07 vs 3.50, 95% CI [-2.0,-1.2]).
  • EVERY model invents unsupported content in 74-100% of notes, including the general ones. This is not a MedGemma-specific failure.
  • The judge sometimes invents an accusation: 32 flagged quotes across the 8 models were not in the note and were discarded by the substring check (0-11 per model; Gemma-4-26B and Gemma-4-31B produced none). Every one of the 4,337 retained quotes is confirmed present in the note; a further 8 flags were dropped only for being too short (<8 chars) to verify.
  • The judge is strict (it flags 'decreasing severity' vs 'decreasing occurrences'), so absolute rates are high; the density comparison across models is the meaningful read. n = 101-119 per model.

Two things fall out, and one is genuinely surprising. First, every model — medical and general alike — invents unsupported content in most of its notes; this is not a MedGemma failing. Second, and this cuts for the medical fine-tune: the base Gemma-3-4B is the worst fabricator in the whole study, and MedGemma-4B invents markedly less than the model it was built from (2.1 vs 3.5 per 1,000 characters, 95% CI [−2.0,−1.2] — a real gap). The medical tuning measurably reduces hallucination. The most faithful model, though, is still a general one — Gemma-4-31B, which also drew zero false accusations from the judge.

The inventions are not subtle. In one visit MedGemma-27B wrote that the patient was a “50-year-old male” whose symptoms had lasted “the past few months” — the transcript gives neither age, sex, nor duration. MedGemma-1.5 recorded a “pulse oximeter reading… 98%” from a reading that was never taken. A fabricated vital sign is the kind of error that survives into a chart and changes what happens next. Because a claim like “this model invents facts” should be checkable, here are the receipts — real encounters, the full transcript each model was given, and every unsupported statement each one wrote. Open a transcript and verify any of them.

Every statement below was written by a model into its clinical note, and is not supported by the visit transcript it was given. Each quote is taken from that model's own note and machine-checked to appear in it (a normalized, case-insensitive match), so a grader that invents an accusation is caught and dropped, not shown here. Open the transcript and check any of them yourself. These are the first encounters in the set, not ones picked for effect — and note that every model here, including the general ones, invents something.

Graded by OpenAI gpt-5.5 — an independent vendor with no stake in Google's models — blinded to which model wrote which note.

Encounter D2N088 — 36 unsupported statements across 8 models
Show the full visit transcript the models were given (check any quote against it)
[doctor] hi , andrew . how are you ?
[patient] hey , good to see you .
[doctor] i'm doing well , i'm doing well .
[patient] good .
[doctor] so , i know the nurse told you about dax . i'd like to tell dax a little bit about you .
[patient] sure .
[doctor] uh , so , andrew is a 59-year-old male with a past medical history , significant for depression , type two diabetes , and hypertension who presents today with an upper respiratory infection . so , andrew , what's going on ?
[patient] yeah . we were doing a bit of work out in the yard in the last week or so and i started to feel really tired , was short of breath . um , we- we're not wearing masks as much at the end of the summer and i think i caught my first cold and i think it just got worse .
[doctor] okay . all right . um , now , have you had your covid vaccines ?
[patient] yeah , both .
[doctor] okay . all right . and , um , do you have any history of any seasonal allergies at all ?
[patient] none whatsoever .
[doctor] okay . all right . and when you say you're having some shortness of breath , did you feel short of breath walking around or at rest ?
[patient] uh , usually , it was lifting or carrying something . we were doing some landscaping , so i was carrying some heavy bags of soil and i , i got really winded . it really surprised me .
[doctor] okay . and are you coughing up anything ?
[patient] not yet , but i feel like that's next .
[doctor] okay . and fevers ?
[patient] uh , i felt a little warm , but i , i just thought it was because i was exerting myself .
[doctor] okay . all right . and any other symptoms like muscle aches , joint pain , fatigue ?
[patient] my elbows hurt quite a bit and my knees were pretty tired . l- like i said , i really felt some tension around my knees , but , uh , i think that was a lot to do with , uh , lifting the bags .
[doctor] okay . all right . um , so , you know , how about , how are you doing in terms of your other medical problems , like your depression ? how are you doing with that ? i know we've , you know , talked about not putting you on medication for it because you're on medication for other things . what's going on ?
[patient] i- it's been kind of a crazy year and a half . i was a little concerned about that but , for the most part , i've been , been doing well with it . my , my wife got me into barre classes , to help me relax and i think it's working .
[doctor] okay . all right , great . and , and in terms of your diabetes , how are you doing watching your , your diet and your sugar intake ?
[patient] uh , i've been monitoring my sugar levels while i am going to work during the week . uh , not so , uh , if its saturday or sunday i usually don't remember . uh , the diet's been pretty good for the most part , except for , you know , some house parties and things like that . but , uh , been good for the most part .
[doctor] okay and have they been elevated at all since this episode of your-
[patient] no .
[doctor] okay . and then , how , lastly , for your high blood pressure , have you been monitoring your blood pressures at home ? did you buy the cuff like i suggested ?
[patient] uh , same thing . during the while i'm going to work, i'm regular about monitoring it, but if its a saturday or sunday, not so much . but , uh , it's , it's been under control .
[doctor] but you're taking your medication ?
[patient] yes .
[doctor] okay . all right . well , you know , i know that , you know , you've endorsed , you know , the shortness of breath and some joint pain . um , how about any other symptoms ? nausea or vomiting ? diarrhea ?
[patient] no .
[doctor] anything like that ?
[patient] no .
[doctor] okay . all right . well , i wan na go ahead and do a quick physical exam , all right ? hey , dragon , show me the vital signs . so , your vital signs here in the office look quite good .
[patient] mm-hmm .
[doctor] you know , everything's looking normal , you do n't have a fever , which is really good . um , i'm just gon na go ahead and listen to your heart and your lungs and , kind of , i'll let you know what i hear , okay ?
[patient] sure .
[doctor] okay . so , on your physical exam , you know , your heart sounds nice and strong . your lungs , you do have scattered ronchi bilaterally on your lung exam . uh , it clears with cough . um , i do notice a little bit of , um , some edema of your lower extremities and you do have some pain to palpation of your elbows bilaterally . um , so , let's go ahead , i want to look at some of your results , okay ?
[patient] mm-hmm .
[doctor] hey , dragon . show me the chest x-ray .
[doctor] so , i reviewed the results of your chest x-ray and everything looks good . there's no airspace disease , there's no pneumonia , so that's all very , very good , okay ?
[patient] good .
[doctor] hey , dragon . show me the diabetic labs .
[doctor] and here , looking at your diabetic labs , you know , your hemoglobin a1c is a little elevated at eight .
[patient] mm-hmm .
[doctor] i'd like to see that a little bit better , around six or seven , if possible .
[patient] mm-hmm .
[doctor] um , so let's talk a little bit about my assessment and my plan for you .
[patient] mm-hmm .
[doctor] so , for your first problem , this upper respiratory infection , i believe you , you have a viral syndrome , okay ? we'll go ahead and we'll send a covid test , just to make sure that you do n't have covid .
[patient] mm-hmm .
[doctor] uh , but overall , i think that , um , you know , this will resolve in a couple of days . i do n't think you have covid , you do n't have any exposures , that type of thing .
[patient] mm-hmm .
[doctor] so , i think that this will improve . i'll give you some robitussin for your cough and i would encourage you take some ibuprofen , tylenol for any fever , okay ?
[patient] you got it .
[doctor] for your next problem , your depression , you know , it sounds like you're doing well with that , but again , i'm happy to start on a med- , a medical regiment or ...
[patient] mm-hmm .
[doctor] . refer you to psychotherapy , if you think that that would be helpful .
[patient] mm-hmm .
[doctor] would you like that ?
[patient] u- u- um , maybe not necessarily . maybe in a , uh , few months we'll check on that .
[doctor] okay . all right .
[doctor] for your third problem , your type two diabetes , i want to go ahead and increase your metformin to 1000 milligrams , twice daily .
[patient] mm-hmm .
[doctor] and i'm gon na get an- another hemoglobin a1c in four months , okay ?
[patient] okay , sure .
[doctor] hey , dragon . order a hemoglobin a1c .
[doctor] and lastly , for your high blood pressure , it looks like you're doing a really good job managing that . i want to go ahead and continue you on the , um , lisinopril , 20 milligrams a day .
[patient] mm-hmm .
[doctor] and i'm gon na go ahead and order a lipid panel , okay ?
[patient] sure .
[doctor] do you need a refill of the lisinopril ?
[patient] actually , i do .
[doctor] okay . hey , dragon . order lisinopril , 20 milligrams daily .
[doctor] so , the nurse will be in , she'll help you , uh , make a follow-up appointment with me . i want to see you again in about four months .
[patient] okay .
[doctor] let me know if your symptoms worsen and we can talk more about it , okay ?
[patient] you got it .
[doctor] all right . hey , dragon . finalize the note .
MedGemma-4B medical2 invented · faithfulness 4/5
  • “He has no known allergies.”
    Transcript only supports no seasonal allergies, not no known allergies of any type.
  • “Follow-up appointment scheduled in four months.”
    Transcript says the nurse will help make a follow-up appointment and the doctor wants to see him in about four months; it does not confirm it was scheduled.
MedGemma-1.5-4B medical (newest)5 invented · faithfulness 4/5
  • “but feels they may be developing”
    The patient said he felt a cough might be next, but did not say fever or multiple symptoms were developing.
  • “Denies muscle aches, joint pain (except elbows), nausea, or vomiting.”
    The transcript supports denial of nausea/vomiting, but not denial of joint pain; the patient reported elbow pain and knee tiredness/tension.
  • “ibuprofen/acetaminophen for fever/myalgia”
    The doctor recommended ibuprofen/Tylenol for any fever, but myalgia was not reported or specified as an indication.
  • “Advise rest and hydration.”
    Rest and hydration were not mentioned in the transcript.
  • “Prescribe ibuprofen/acetaminophen PRN for fever/myalgia.”
    The transcript says to take ibuprofen/Tylenol for fever, not that these were prescribed, and myalgia was not discussed as an indication.
MedGemma-27B medical2 invented · faithfulness 4/5
  • “and potentially other medications not specified.”
    The transcript does not state that the patient is taking any additional unspecified medications beyond the discussed metformin and lisinopril.
  • “General: Appears well.”
    The transcript does not include a general appearance finding.
Gemma-3-4B base14 invented · faithfulness 3/5
  • “**Date:** October 26, 2023”
    The transcript does not provide the visit date.
  • “**DOB:** [Assume DOB - e.g., 1963]”
    The transcript does not provide a date of birth or birth year.
  • “**MRN:** [Assume MRN – e.g., 1234567]”
    The transcript does not provide an MRN.
  • “He has been monitoring his blood sugars irregularly, primarily on weekends.”
    The transcript says he monitors sugars during the work week and usually does not remember on weekends.
  • “**Allergies:** No known allergies.”
    The transcript only states no seasonal allergies, not no known allergies generally.
  • “**FAMILY HISTORY:** [Space for family history documentation - e.g., Diabetes, Hypertension, Cardiovascular Disease]”
    No family history is discussed in the transcript.
  • “**Vital Signs:** BP: 145/85 mmHg, HR: 78 bpm, RR: 18 breaths/min, Temp: 98.6°F (oral), SpO2: 97% on room air.”
    The transcript says vitals were normal and no fever, but gives no specific vital sign values.
  • “**General:** Alert and oriented male appearing fatigued.”
    The transcript does not document alert/oriented status or that he appeared fatigued on exam.
  • “**HEENT:** Normal appearance.”
    No HEENT exam findings are documented in the transcript.
  • “**Cardiovascular:** Regular rate and rhythm, no murmurs appreciated.”
    The transcript only says heart sounds were nice and strong; it does not mention rhythm or murmurs.
  • “**Neurological:** Grossly intact.”
    No neurological exam is documented in the transcript.
  • “HbA1c: 8.2% (Elevated)”
    The transcript states hemoglobin A1c was 8, not 8.2%.
  • “Type 2 Diabetes Mellitus - HbA1c elevated at 8.2%, indicating suboptimal glycemic control.”
    The transcript states hemoglobin A1c was 8, not 8.2%.
  • “**Nurse Instructions:** Schedule follow-up appointment, provide patient with medication instructions and education regarding diabetes management.”
    The transcript only says the nurse will help make a follow-up appointment; medication instructions and diabetes education by the nurse are not stated.
Gemma-3-27B base6 invented · faithfulness 4/5
  • “Regular rate and rhythm, strong sounds.”
    The transcript only states that heart sounds were 'nice and strong'; regular rate and rhythm were not documented.
  • “1+ edema noted in lower extremities.”
    The transcript notes 'a little bit' of lower extremity edema but does not grade it as 1+.
  • “Encourage consistent blood sugar monitoring, including weekends.”
    The patient reported inconsistent weekend monitoring, but the doctor did not give this instruction in the plan.
  • “Patient reports generally well-controlled blood pressure with medication adherence during the work week.”
    The transcript supports weekday blood pressure monitoring and general medication adherence, but not medication adherence specifically during the work week.
  • “Encourage consistent home blood pressure monitoring, including weekends.”
    The doctor did not instruct the patient to monitor blood pressure more consistently on weekends.
  • “Follow up appointment scheduled in four months.”
    The doctor said the nurse would help make a follow-up appointment and wanted follow-up in about four months; it was not confirmed as scheduled.
Gemma-4-E4B general (newer)3 invented · faithfulness 4/5
  • “The shortness of breath is currently attributed to exertion rather than underlying cardiac or pulmonary pathology, though monitoring is necessary.”
    The transcript supports that shortness of breath occurred with exertion, but does not state that cardiac or pulmonary pathology was ruled out or that SOB was attributed away from those causes.
  • “Ibuprofen/Tylenol as needed for fever or discomfort.”
    The transcript supports ibuprofen/Tylenol for fever, but does not mention using them for discomfort.
  • “If shortness of breath worsens or new concerning symptoms develop (e.g., nausea, vomiting, diarrhea), the patient should contact the office immediately.”
    The transcript only advises the patient to report worsening symptoms; it does not specifically mention new nausea, vomiting, diarrhea as return precautions or say to contact the office immediately.
Gemma-4-26B general (newer)2 invented · faithfulness 4/5
  • “General: Patient appears well.”
    The transcript does not document a general appearance finding.
  • “Cardiovascular: Heart sounds are strong and regular.”
    The transcript supports strong heart sounds, but does not state they were regular.
Gemma-4-31B general (newer)2 invented · faithfulness 4/5
  • “He notes joint pain in his elbows and knees, which he attributes to physical exertion.”
    The transcript supports elbow pain and knee symptoms, but the patient only explicitly attributes the knee symptoms/tension to lifting bags, not the elbow pain.
  • “- Heart: Regular rate and rhythm, sounds strong.”
    The transcript only states that the heart sounds were nice and strong; regular rate and rhythm were not documented.
Encounter D2N089 — 21 unsupported statements across 8 models
Show the full visit transcript the models were given (check any quote against it)
[doctor] hi andrea , how are you ?
[patient] i'm doing well . how are you ?
[doctor] doing well . uh , so i know the nurse told you about dax . i'd like to tell dax a little bit about you . okay ?
[patient] okay .
[doctor] so , andrea is a 52-year-old female with a past medical history significant for rheumatoid arthritis , atrial fibrillation , and reflux who presents today for her annual exam . so andrea , it's been a year since i saw you . how are you doing ?
[patient] i'm doing well . so , i've been walking like you told me to and , um , exercising and doing yoga , and that's actually helped with my arthritis a lot , just the- the constant movement . so , i have n't had any joint pain recently .
[doctor] okay . good . so , no- no issues with any stiffness or pain or flare ups over the last year ?
[patient] no .
[doctor] okay . and i know that we have you on the methotrexate , are you still taking that once a week ?
[patient] yes , i am .
[doctor] okay . and any issues with that ?
[patient] no .
[doctor] no . okay . and then in terms of your a-fib , how are you doing with that ? are you having any palpitations ? or , i know that you've kind of been in and out of it over the past , you know , year or so .
[patient] yeah . i've still been having palpitations . the- the last one i had was about a week ago . i've noticed that when i start to get stressed , um , they start to flare up again . so , i've been trying meditation , trying running with my dog to try and relieve the stress but it has n't really been working .
[doctor] yeah . i- i know that you had called , um , last month and we- we did that event monitor for you , uh , which we'll take a look at in a few minutes . okay ?
[patient] okay .
[doctor] um , how about um , your reflux ? you know , we had placed you on the protonix , uh , has that helped ? and i know that you were gon na do some dietary modifications .
[patient] yeah . i cut out soda and that- that's helped- seemed to help , and the medication's been helping too . i have n't had a flare up in over , i think , five months .
[doctor] okay . all right . um , so , you know , i know that you did the review of systems sheet when you checked in and , you know , you had- you know , you endorsed the palpitations and you had some nasal congestion . any other symptoms ? you know , chest pain , shortness of breath , nausea or vomiting ?
[patient] no , nothing like that . just the nasal- nasal congestion because of my allergies .
[doctor] okay . all right . okay . well , i'd like to go ahead and do a quick physical exam , okay ?
[patient] okay .
[doctor] all right . hey dragon , show me the vital signs . okay . so , you're in- here in the office today , it looks like , you know , your heart rate's really good today . it's- it's nice and controlled so that's good . um , i'm just gon na take a look into your heart and lungs and- and i'll let you know what i find . okay ?
[patient] okay .
[doctor] okay . so , on physical examination , um , you know , everything looks really good . on your heart examination , i do appreciate a slight 2/6 systolic- systolic ejection murmur , um , which we've heard in the past so i'm not worried at that . you're in the- a nice regular rate and rhythm at this time . your lungs are nice and clear . on your right elbow , i do notice some edema and some erythema . does it hurt when i press it ?
[patient] yeah , it does a bit .
[doctor] okay . so , she has pain to palpation of the right elbow . um , and you have no lower extremity edema , okay ? um , so i wan na go ahead and just take a look at some of your results . okay ?
[patient] okay .
[doctor] hey dragon , show me the event- event monitor results . okay . so , you know , this is the results of your event monitor which shows that , you know , you're in and out of a-fib , you have what we call a conversion pause . you know , you're in a-fib , you pause , and then you go back to regular rhythm . so , we'll talk about that , okay ?
[patient] okay .
[doctor] hey dragon , show me the autoimmune panel . so , looking here at your autoimmune panel , everything looks good , it looks like you're- you know , everything is well controlled with your rheumatoid arthritis on the methotrexate . okay ? so , let me just go over a little bit about my assessment and my plan for you . okay ?
[patient] okay .
[doctor] so for your first problem , your rheumatoid arthritis , again , everything looks good . i wan na just continue you on the methotrexate 2.5 mg , once weekly . um , and uh , if you need a referral back to see the rheumatologist , let me know , but i think everything seems stable now . do you need a refill of the methotrexate ?
[patient] yes , i do .
[doctor] okay . hey dragon , order methotrexate , 2.5 mg once weekly . for your second problem , the atrial fibrillation . so , you're going in and out of a-fib and i'd like to just keep you in normal sinus rhythm . so , i wan na go ahead and refer you to cardiology for a cardiac ablation which just maps out where that rhythm is coming from and burns it so it does n't come back . okay ? you're young , we wan na keep you in a normal rhythm and , being that you're going in and out of a-fib , i think that's what we should do . okay ?
[patient] okay .
[doctor] hey dragon , order a referral to cardiology . and for your last problem , the reflux , you know , i wanna- i want you to just continue on the protonix , 40 mg a day . continue with your dietary modifications , you know , avoiding coffee and spicy foods , that type of thing . okay ? and then let me know if you have any other issues with that , okay ?
[patient] will do .
[doctor] any questions ?
[patient] no , i do n't .
[doctor] okay . all right . it was good to see you .
[patient] good seeing you .
[doctor] hey dragon , finalize the note .
MedGemma-4B medical2 invented · faithfulness 4/5
  • “Vital signs are normal.”
    Transcript only states the heart rate is controlled; it does not state that all vital signs are normal.
  • “dietary modifications (avoiding caffeine and spicy foods)”
    Transcript supports avoiding coffee and spicy foods, but does not specifically mention avoiding caffeine generally.
MedGemma-1.5-4B medical (newest)5 invented · faithfulness 3/5
  • “Patient reports adherence to AFib management including walking and exercise.”
    The transcript does not describe any AFib management regimen involving walking/exercise; walking, exercise, and yoga were discussed as helping arthritis, and running was mentioned for stress relief.
  • “Patient reports palpitations have occurred but are less frequent recently, attributed to stress, despite attempts at meditation and running with her dog.”
    The transcript supports ongoing palpitations triggered by stress and attempts at meditation/running, but does not state they are less frequent recently.
  • “Vital signs are within normal limits.”
    Only the heart rate is described as good/controlled; the transcript does not state all vital signs are within normal limits.
  • “Reflux Esophagitis: Stable.”
    The transcript mentions reflux treated with Protonix, but does not diagnose reflux esophagitis.
  • “Follow up as needed.”
    No specific follow-up plan is stated in the transcript beyond letting the doctor know if issues arise.
MedGemma-27B medical3 invented · faithfulness 4/5
  • “No acute distress observed.”
    The transcript does not mention the patient's general appearance or acute distress status.
  • “Acute inflammation/pain noted.”
    The transcript notes right elbow edema, erythema, and tenderness, but does not state that it is acute.
  • “likely requires further evaluation or treatment (e.g., RICE, NSAIDs, possible imaging) depending on severity and duration”
    No evaluation or treatment plan for the right elbow was discussed in the transcript.
Gemma-3-4B base9 invented · faithfulness 3/5
  • “**DATE:** October 26, 2023”
    The transcript does not provide a visit date.
  • “Heart rate 72 bpm”
    The transcript says the heart rate was controlled but does not provide a numeric heart rate.
  • “Blood pressure: [Insert BP Here - assumed to be within normal range].”
    No blood pressure value or statement that it was normal is provided in the transcript.
  • “Respiratory rate 16 breaths per minute.”
    The transcript does not provide a respiratory rate.
  • “Temperature: 98.6°F (37°C).”
    The transcript does not provide a temperature.
  • “Heart sounds clear to auscultation.”
    The transcript documents a systolic ejection murmur and regular rhythm but does not state that heart sounds were clear.
  • “at the right upper sternal border”
    The transcript documents a 2/6 systolic ejection murmur but does not specify its location.
  • “Patient appears well-nourished and in no acute distress.”
    The transcript does not include a general appearance exam.
  • “**Follow-up:** Schedule follow-up appointment in one month to assess response to treatment plan and discuss cardiology referral.”
    The transcript does not mention scheduling a one-month follow-up.
Gemma-3-27B base1 invented · faithfulness 4/5
  • “walking her dog”
    The transcript says she tried running with her dog, not walking her dog.
Gemma-4-E4B general (newer)1 invented · faithfulness 4/5
  • “attributing nasal congestion only to seasonal allergies.”
    The transcript supports nasal congestion due to allergies, but does not specify seasonal allergies.
Gemma-4-26B general (newer)0 invented · faithfulness 5/5

Nothing unsupported found.

Gemma-4-31B general (newer)0 invented · faithfulness 5/5

Nothing unsupported found.

Encounter D2N090 — 29 unsupported statements across 8 models
Show the full visit transcript the models were given (check any quote against it)
[doctor] hi , albert . how are you ?
[patient] hey , good to see you .
[doctor] it's good to see you too . so , i know the nurse told you about dax . i'd like to tell dax a little bit about you .
[patient] sure .
[doctor] so , albert is a 62-year-old male , with a past medical history significant for depression , type 2 diabetes , and kidney transplant , who is here today for emergency room follow-up .
[patient] mm-hmm .
[doctor] so , i got a notification that you were in the emergency room , but , but what were you there for ?
[patient] well , i , uh , i was n't really , uh , staying on top of my , uh , blood sugar readings , and i felt kinda woozy over the weekend . and i was little concerned , and my wife wanted to take me in and just have me checked out .
[doctor] okay . and , and was it , in fact , high ?
[patient] yeah , it was .
[doctor] okay . did you ... were you admitted to the hospital ?
[patient] uh , no .
[doctor] okay . all right . and , uh , are you ... did they see a reason , as to why it was elevated ?
[patient] uh , yeah . my mother was actually in the hospital the last week. she had a bit of a fall and had to do a hip replacement. she's feeling better now but we have been just grabbing meals at the cafeteria or picking up fast food on our way home and i just really was n't monitoring what i was eating .
[doctor] okay . that's sorry to hear . and are you feeling better now ?
[patient] uh , actually , when we got home from the , uh , f- from the visit , i felt a lot better .
[doctor] okay . and since then , have you been following your diet pretty closely ?
[patient] yes .
[doctor] okay . 'cause we do n't wan na end up in the hospitaltoo
[patient] no .
[doctor] all right . um , okay . and , so , before that happened , how are you doing with your diet ?
[patient] uh , during the week , i've been fine , 'cause i've been very busy . on the weekends , doing things . you're seeing people . you're having people over . it's , i- not , not as consistent on the weekend .
[doctor] okay . all right . um , is there a way that you think that that can improve ?
[patient] uh , s- stop eating .
[doctor] okay . all right . well , let's talk about your , your kidney transplant . how are-
[patient] mm-hmm .
[doctor] . you doing ? you're taking immunotherapy meds ?
[patient] yes .
[doctor] okay .
[patient] yeah . i've , i've been pretty diligent about it , following doctor's orders , so it's been , it's been pretty good so far .
[doctor] okay . and , and y- the last time i saw that you saw dr. reyes , was about three weeks ago , and everything seemed to be fine .
[patient] that's correct .
[doctor] your kidney function is good .
[patient] yes .
[doctor] okay . all right . and in terms of your depression , how are you , how are you doing ?
[patient] and it's been about a , a tough , ugh , year-and-a-half or so , but i've been pretty good with it . i , i have my moments , but i- as long as i find some time to relax , at least in the afternoon , then , then it seems to work out okay .
[doctor] okay . so , i know that we've kind of talked about holding off on medical therapy-
[patient] mm-hmm .
[doctor] . 'cause you're on so many other meds .
[patient] mm-hmm .
[doctor] um , is that something that you wan na revisit , or do you wan na look into therapy , or do you think anything's needed right now ?
[patient] uh , i think i probably wan na shy away from any therapy . my , my wife got me into meditation recently and , and , uh , i , i find that relaxing . so , i think i'd like to continue that , at least for a couple more months and see how it goes .
[doctor] okay . all right . that sounds good . all right . well , i know the nurse did a quick review of systems with you , when you-
[patient] mm-hmm .
[doctor] . checked in . do you have any symptoms , any chest pain or shortness of breath ?
[patient] none whatsoever .
[doctor] lightheadedness ? dizziness ?
[patient] no .
[doctor] no ? okay . um , and i just wan na go ahead and do a quick physical exam .
[patient] mm-hmm .
[doctor] hey , dragon . show me the vital signs .
[doctor] so , looking here right now , your vital signs look great . you know , your pulse ox is great . your h- your blood pressure and heart rate are right where they should be .
[patient] mm-hmm .
[doctor] so , i'm gon na just check you out , and i'm gon na let you know what i find . okay ?
[patient] sure .
[doctor] okay . so , on your physical exam , everything looks really good . um , you do n't appear in any distress at this time . i do n't appreciate any carotid bruits . your heart , on your heart exam , i do hear that slight 2/6 systolic ejection murmur , but we heard that in the past .
[patient] mm-hmm .
[doctor] your lungs sound nice and clear , but i notice , you know , 1+ , uh , edema in your lower extremities . okay ?
[patient] mm-hmm .
[doctor] um , so , let's go ahead . i wan na look at some of your results . okay ?
[patient] sure .
[doctor] hey , dragon . show me the glucose .
[doctor] so , right now , your blood sugar is about 162 . have you eaten before you came in here ?
[patient] i did not .
[doctor] okay . all right . um , hey , dragon . show me the diabetes labs .
[doctor] okay . i'm looking at your diabetes labs . you know , your hemoglobin a1c is about 8 , and that's a , that's a little high .
[patient] mm-hmm .
[doctor] so , not only , you know , have your blood sugars , were they high that one day , they were , they've been a little elevated .
[patient] mm-hmm .
[doctor] so , we'll talk about , you know , how to go ahead and , and fix that . okay ?
[doctor] so , let me talk a little bit about my assessment and my plan for you .
[patient] mm-hmm .
[doctor] so , for your first problem , this hyperglycemia , you know , i wan na go ahead and increase your lantus to 20 units at night . okay ? i want you to continue your monitor your blood sugar and let me know how they're running 'cause we might have to adjust that further .
[patient] mm-hmm .
[doctor] okay ? um , and i wan na order another hemoglobin a1c in a couple months . hey , dragon . order a hemoglobin a1c .
[doctor] for your next problem , your depression , i think you're doing a great job with your current strategies with the meditation . we will hold off on medication or therapy at this time , and you know to call me if you need anything , right ?
[patient] mm-hmm .
[doctor] okay . and for your third problem , your kidney transplant , your kidney function looks stable . uh , i'm gon na just have you go back to dr. reyes , to manage all of your immunosuppression medications .
[patient] okay .
[doctor] um , and then , he knows to reach out to me if he needs anything . okay ?
[patient] you got it .
[doctor] all right . well , the nurse will be in soon to check you out . okay ?
[patient] perfect .
[doctor] hey , dragon . finalize the note .
MedGemma-4B medical2 invented · faithfulness 4/5
  • “He reports elevated blood sugar levels at home”
    The transcript supports elevated blood sugar but does not specify that the elevated readings were measured at home.
  • “The patient's blood sugar was elevated at home”
    The transcript does not state that the elevated blood sugar was documented at home rather than in the emergency room.
MedGemma-1.5-4B medical (newest)0 invented · faithfulness 5/5

Nothing unsupported found.

MedGemma-27B medical9 invented · faithfulness 3/5
  • “**Date:** [Current Date - Assuming today's date]”
    The transcript does not provide the date of the visit.
  • “**DOB:** [Age 62, so calculate DOB based on current year]”
    The transcript provides age but not date of birth.
  • “**MRN:** [Not provided, would be in a real note]”
    No MRN is provided in the transcript.
  • “**Cardiovascular:** Regular rate and rhythm. Slight 2/6 systolic ejection murmur appreciated (chronic finding). No carotid bruits.”
    The transcript supports the murmur and no carotid bruits, but does not state regular rate and rhythm.
  • “**Extremities:** 1+ pitting edema in lower extremities.”
    The transcript states 1+ edema but does not specify that it was pitting.
  • “Patient to contact clinic if blood sugars are consistently high or low.”
    The transcript says to report how blood sugars are running, but does not specifically mention calling for consistently high or low sugars.
  • “Discuss strategies for maintaining consistent diet, especially on weekends (e.g., meal planning, healthy options when socializing).”
    The transcript includes discussion of inconsistent diet but no specific plan for meal planning or healthy options.
  • “Return to clinic in 2-3 months for repeat HbA1c and diabetes follow-up, or sooner if needed.”
    The transcript supports ordering another HbA1c in a couple months but does not specify a return clinic visit or 'sooner if needed.'
  • “Contact clinic if depression symptoms worsen or if any new concerns arise.”
    The transcript says to call if he needs anything regarding depression, but does not mention new concerns generally.
Gemma-3-4B base14 invented · faithfulness 3/5
  • “**Date:** October 26, 2023”
    The transcript does not provide a visit date.
  • “**Time:** 10:30 AM”
    The transcript does not provide a visit time.
  • “His wife noticed a significant increase in his blood glucose readings prior to his presentation.”
    The transcript says his wife wanted him checked out, but does not say she noticed the glucose readings or a significant increase.
  • “meditation, which he continues twice daily.”
    The transcript mentions meditation but does not state it is done twice daily.
  • “**Vitals:** Blood Pressure: 142/88 mmHg, Heart Rate: 78 bpm, Pulse Ox: 98%, Temperature: 98.6°F”
    The transcript only says vital signs looked great; it provides no numeric BP, heart rate, pulse ox, or temperature.
  • “**General:** Alert and oriented male appearing in mild distress.”
    Alert and oriented is not documented, and the transcript states he did not appear in any distress.
  • “Slight 2/6 systolic ejection murmur auscultated at the apex.”
    The murmur is supported, but the location at the apex is not stated in the transcript.
  • “Reports no lightheadedness or dizziness since arriving.”
    The patient denied lightheadedness/dizziness, but the timeframe 'since arriving' is not stated.
  • “**Glucose:** 162 mg/dL (fasting)”
    The glucose value is supported, but the transcript does not establish a formal fasting measurement.
  • “**Kidney Function:** Creatinine within normal limits, consistent with previous assessment by Dr. Reyes.”
    The transcript says kidney function is good/stable but does not mention creatinine or normal limits.
  • “**Immunosuppression:** Medications stable as per Dr. Reyes’s report.”
    The transcript does not state that immunosuppression medications were stable or that there was a Dr. Reyes report.
  • “Continue current strategy of meditation twice daily.”
    The transcript does not state that meditation is twice daily.
  • “including scheduled immunosuppressant medication review and monitoring.”
    The transcript says Dr. Reyes will manage immunosuppression medications, but does not mention a scheduled review.
  • “Discussed the importance of adherence to his medications and regular follow-up appointments.”
    Medication adherence was discussed generally, but regular follow-up appointments were not discussed in the transcript.
Gemma-3-27B base2 invented · faithfulness 4/5
  • “Will communicate with Dr. Reyes if any concerns arise regarding kidney function or immunosuppression.”
    Transcript says Dr. Reyes can reach out to the doctor if needed, not that this clinician will communicate with him if concerns arise.
  • “The nurse will discharge the patient with instructions and schedule appropriate follow-up appointments.”
    Transcript only says the nurse will come in to check the patient out; discharge instructions and scheduling follow-up appointments are not stated.
Gemma-4-E4B general (newer)1 invented · faithfulness 4/5
  • “Dr. Reyes is aware of this visit and should be contacted by his office if any issues arise regarding immunosuppression management.”
    The transcript does not state that Dr. Reyes is aware of this visit or that he should be contacted by his office; it only says Dr. Reyes should manage immunosuppression and can reach out if needed.
Gemma-4-26B general (newer)1 invented · faithfulness 4/5
  • “The patient reports feeling "woozy" over the weekend due to lack of blood sugar monitoring.”
    The transcript says he felt woozy and had not been monitoring blood sugars, but does not establish that the wooziness was due to lack of monitoring.
Gemma-4-31B general (newer)0 invented · faithfulness 5/5

Nothing unsupported found.

Part 4 — The newest medical model is not the upgrade the version number implies

MedGemma-1.5, released January 2026, is Google's current medical model as of 27 July 2026 — the one a developer starting today would reach for. Set against MedGemma-4B, the version it replaced, it is not an improvement so much as a reshuffle: a couple of marginal gains bought with large losses elsewhere and an enormous jump in cost.

TaskMedGemma‑4BMedGemma‑1.5change
Differential, top‑140.823.7−17.1
MedQA exam58.049.3−8.7
PubMedQA78.472.0−6.4
Note wording (ROUGE)51.749.3−2.4
Section classification48.047.0−1.0
MMLU‑Med exam62.063.3+1.3
Note hallucinations ↓ better2.071.58−0.49
Tokens per answer3491160×

MedGemma-1.5 (Jan 2026) against MedGemma-4B, the version it replaced. All figures are % accuracy except note hallucinations (unsupported statements per 1,000 characters — lower is better) and tokens per answer (to return a 1–5 triage rating). Green marks an improvement; section classification is a tie.

One bright spot is real and worth stating: MedGemma-1.5 hallucinates less into notes than its predecessor. It also nudges ahead on one exam subset (MMLU-Med, by a within-noise 1.3 points), and ties it on section classification. But the rest points down: it loses 17 points on differential diagnosis and 6–9 on the medical exams. And it carries a usability cost the accuracy columns only hint at — its reasoning cannot be switched off. Asked to “reply with only the label” or to rate urgency “with a single digit,” it will not: it emits a few hundred tokens of deliberation and puts the actual answer last, in a separate channel. It reaches a triage rating in 491 tokens where its predecessor uses three — and an integration that reads the first number it emits, the obvious way to call it, will usually read a digit from its thinking rather than its answer. It is a Gemma-3 model still — a full generation behind the general line it is being sold as an alternative to — and it charges 160× the compute for the privilege.

The scoreboard

Every clinical task, two questions. Left column: did fine-tuning Gemma into a “medical” model beat the very model it was built from? Right column: did that medical model beat simply picking a newer general Gemma? A green ✓ wins means the medical option came out ahead in that comparison; grey = tie is a statistical tie; a red × loses means it lost.

Clinical taskThe “medical” badgevs the base it was built fromThe medical modelvs a newer general Gemma-4
Medical exam (MedQA)✓ helps+9 points× loses
Hard reasoning (MedXpertQA)= tie× loses badly13 vs 49
Emergency triage× no helpboth fail× losesonly Gemma-4 recognises them
Differential diagnosis✓ helps+14 at 4B, gone at 27B~ mixedties E4B at 4B, loses at 27B
Structured output (JSON)✓ helps+21 points= tie
Note wording (ROUGE)= tie× loses
Note faithfulness✓ helpsfewer inventions× loses

Read each column as a pattern. Against the base it was built from, the medical badge does real work (left column — mostly green). But it never clearly beats a newer general Gemma (right column): it splits differential diagnosis by model size, ties on structured output, and loses everything else — most consequentially emergency triage, where only a Gemma-4 model recognises the emergency.

How this was built

Every number here is one of two kinds: a figure we measured ourselves, or a figure from Google's own technical report checked against the source page. No second-hand citation does load-bearing work. One harness, identical prompts for every model. Greedy decoding, fixed seed, Q4_K_M quantisation, run locally — so absolute scores sit below Google's full-precision numbers, but every comparison here is between models run identically. Every headline gap carries a paired-bootstrap 95% interval; where the interval crosses zero we call it a tie, not a win. Every model's raw output is logged, and the emergency cases above are drawn straight from those logs. The entire benchmark — the harness, the statistics, and all of the raw per-item model output — is open at github.com/groundtruth-health/medgemma-benchmark under an MIT licence, so any number here can be recomputed from scratch.

Verdict: Needs context (3/5). Google's claim is not false — MedGemma is a capable open model that beats its general base on medical exams and helps on some clinical tasks. But for most clinical decision-support work we tested, the better open model is a newer general one.

Methods & reproducibility

Every model received the identical prompt for a given task — no per-model wording, no chat-template hand-editing. This is the control the whole comparison rests on, so here is exactly what was sent, rendered from the code that ran it. (The question / findings / options are filled per item; the instruction text is fixed and identical across all eight models.)

The exact prompts

Multiple-choice (MedQA, MedMCQA, PubMedQA, MMLU-Med, MedXpertQA, MMLU-Pro)

System: You are a careful medical expert answering a multiple-choice question.

Question: A 24-year-old woman presents with fatigue and macrocytic anemia. Which deficiency is most likely?

Options:
A. Vitamin C
B. Vitamin B12
C. Vitamin D
D. Vitamin K

Respond with only the letter (A-D) of the single best option.

Clinical decision-support tasks

System: You are a careful clinician.

Triage (urgency)

Patient: 61-year-old male.
Findings:
- Crushing chest pain radiating to the left arm
- Shortness of breath

Rate the clinical urgency on a 1-5 scale where 1 = immediately life-threatening (needs emergency care now) and 5 = not urgent (routine care). Respond with only the single digit.

Differential diagnosis

Patient: 61-year-old male.
Findings:
- Crushing chest pain
- Diaphoresis

From this exact list of conditions:
Anaphylaxis, Unstable angina, Possible NSTEMI / STEMI, GERD, Pericarditis

Give the 5 most likely diagnoses, most likely first, as a numbered list. Use only names from the list. No explanation.

Structured extraction (JSON)

Clinical note:
Mr X, 61, admitted with chest pain. PMH: hypertension. Discharged on aspirin.

Extract the information into JSON with exactly these top-level keys:
["visit motivation", "admission", "patient information", "patient medical history", "surgeries", "symptoms", "medical examinations", "diagnosis tests", "treatments", "discharge"]
Use "None" for anything not stated in the note. Respond with only the JSON object.

Note generation

[doctor] what brings you in? [patient] chest pain since this morning.

Write the clinical note for this visit. Use these section headers in capitals on their own line: CHIEF COMPLAINT, HISTORY OF PRESENT ILLNESS, PHYSICAL EXAMINATION, RESULTS, ASSESSMENT AND PLAN.

Section classification

Conversation:
Doctor: Any allergies? Patient: Yes, penicillin gives me a rash.

Which clinical note section does this belong to? Choose exactly one from: ALLERGY, ASSESSMENT, CC, MEDICATIONS, PLAN, ROS
Respond with only the label.
The eight models
ModelFamilyBase arch.ParamsOllama tagReasoning
MedGemma-4BMedGemma (medical)Gemma 34.3Bmedgemma:latestno
MedGemma-1.5-4BMedGemma (medical, newest)Gemma 34.3Bmedgemma1.5:latestforced (cannot disable)
Gemma-3-4BGemma 3 (base)Gemma 34.3Bgemma3:4bno
Gemma-4-E4BGemma 4 (general, newer)Gemma 4~4.5B eff / 8Bgemma4:e4btoggleable (off)
MedGemma-27BMedGemma (medical)Gemma 327Bmedgemma:27bno
Gemma-3-27BGemma 3 (base)Gemma 327Bgemma3:27bno
Gemma-4-26BGemma 4 (general, newer, MoE)Gemma 425.8B / ~3.8B activegemma4:26btoggleable (off)
Gemma-4-31BGemma 4 (general, newer)Gemma 430.7Bgemma4:31btoggleable (off)

All run locally via Ollama, Q4_K_M quantisation. Because everything is quantised identically, absolute scores sit below Google's full-precision numbers — but every comparison here is between models run the same way.

Datasets, decoding & scoring
BenchmarkSourceSplitn (4B / 27B)Licence
MedQA (USMLE)GBaker/MedQA-USMLE-4-optionstest250 / 150MIT
MedMCQAopenlifescienceai/medmcqavalidation250 / 150MIT/Apache
PubMedQAqiaojin/PubMedQA (pqa_labeled)labeled250 / 150MIT
MMLU-Med (6 subjects)cais/mmlutest250 / 150MIT
MedXpertQA (text)TsinghuaC3I/MedXpertQAtest250 / 150MIT
MMLU-Pro (health + gen)TIGER-Lab/MMLU-Protest250 / 150MIT
Triage + Differentialaai530-group6/ddxplustest (stratified 5×49)245CC-BY-4.0
Structured extractionAGBonnet/augmented-clinical-notestrain (seeded)250MIT
Note generationmkieffer/ACI-Benchtest1+2+3 pooled120CC-BY-4.0
Section classificationMTS-Dialog TestSet-1 (GitHub)MEDIQA-Chat 2023200CC-BY-4.0

Decoding: greedy (temperature 0), fixed seed 0, top_p 1.0, top_k 0, num_predict 4096. Reasoning is disabled for the Gemma-4 models where toggleable; MedGemma-1.5 reasons unconditionally and cannot be turned off. Samples are drawn with a fixed seed (20260710); triage/differential are stratified so rare emergencies are not starved.

Scoring is deterministic — no LLM-as-judge anywhere. Multiple-choice: exact match on the extracted letter, unparseable counted wrong. Triage: emergency-undertriage rate P(rated ≥4 | true severity ≤2) plus rank correlation of predicted vs true severity. Differential: the correct diagnosis appears in the top-5 (GTPA@1/@5), exact-match against the closed 49-label list. Extraction: strict/lenient JSON validity, schema exact-match, key recall, markdown-fence rate. Note generation: ROUGE-1/2/L (Google's rouge_score) plus section-header presence F1. Section classification: 20-way exact match. Every headline gap carries a paired-bootstrap 95% interval (10,000 resamples); where the interval crosses zero we report a tie, not a win.

Confound controls (triage): a reversed-scale rerun (to rule out a misread), verified identical prompt delivery across the three chat-template families, and a token budget large enough for every model to reach an answer.

Reproduce everything — the full harness and every per-item model output are public at github.com/groundtruth-health/medgemma-benchmark (MIT licence): run_matrix.sh (multiple-choice), run_cdss.sh (clinical tasks), per-item logs under results/ and results_cdss/ (including the judge records with the discarded-quote audit trail), statistics in src/bootstrap.py. The per-model scorecard is also a one-click download: benchmark.csv (CC BY 4.0).

Disclosures & provenance

Published
17 Jul 2026 · last updated 27 Jul 2026
Author
The Ground Truth editor. Editorial standard →
Funding
Self-funded. Ground Truth takes no money from, and has no affiliation with, any organization examined here. Independence policy →
Rating
Needs context (3/5), rubric v1.0, rated 17 Jul 2026. Full rating card →
Data
Download the dataset · released under CC BY 4.0
Corrections
Corrected 27 Jul 2026 — A rating line read “on emergency recognition it is no help at all.” That holds at 27B (MedGemma 8% vs base 8%) but not at 4B, where the medical model flags 24% against its base's 0%. The line now states what holds at both sizes: it adds nothing at 27B and never approaches a newer general model. The band is unchanged. Corrections log →