They want a doctor in every pocket. We handed the pocket a lethal order.

In the models’ own words

Two honest limits before we go on. Nineteen dangerous probes per model is a small sample, so we don’t rank the dangerous models against one another, and zero-out-of-19 is not the same as “perfectly safe” — the true frontier rate could sit as high as one in six. But the split between the tiers is not a close call: treating each model as a single data point, the odds it arose by chance are about 1 in 6,435.

It isn’t being offline

The obvious objection is that a phone can only hold a stripped-down model, so of course it does worse — the real comparison is cloud versus small, not cloud versus offline. So we ran a large model offline. A general 31-billion-parameter model, with no network at all, confirmed just 11% of the dangerous orders — statistically the same as the frontier cloud models. Running without the internet is not what breaks these models. Running small is.

It isn't being offline. It's being pocket-sized.

↓ lower is better

It isn’t the compression

On-device models are also squeezed — quantized — to fit in memory, which could be blunting them. We expected this to be the culprit, and it isn’t. We ran the same small model at full precision, and it confirmed 58% of the dangerous orders, no better than the 47% it managed compressed. The ladder even points the other way: the two most dangerous small models we tested were the least compressed of the set, and the safe larger ones the most. Whatever is failing, compression is not it.

It isn't the compression

↓ lower is better

It’s size — and it shows up in two families

To rule out everything but scale, we walked two general-model families across their size range, changing as little as possible except the number of parameters. In Google’s own Gemma-4 line the danger falls from 42% at ~2B (Gemma-4-E2B) to 11% at 31B (Gemma-4-31B). In a second, unrelated family — Alibaba’s Qwen — it falls further and faster: 74% at 0.8B, still 42% at 2B, then down to 5% by 9B and 0% at the top. Two different labs, two different training pipelines, the same descent.

Bigger is safer — the same pattern in two model families

↓ lower is better

The Qwen line has one rung more telling than the rest. Qwen3.6-35B-A3B is a 35-billion-parameter model that uses only about 3 billion of them on any given answer — and it was as safe as the dense 27B, not as dangerous as the 2-billion one. What protects a model from confirming a lethal order, in other words, is how much it knows, not how hard it works to answer. That points straight at why the small ones fail: on the dangerous orders they got wrong, they mostly didn’t know the dose in the first place.

Why they fail: they don’t know the dose

We went in expecting sycophancy — a model that knows the right dose but folds when an authority figure insists on the wrong one. That is the frightening version, and it does happen: fourteen times, a model gave the correct dose when we asked it plainly, then confirmed the lethal order under pressure. But it’s the minority. Of the 66 dangerous confirmations, 52 were plain ignorance — the model got the dose wrong even with no pressure at all. It didn’t cave to the clinician; it never knew the answer. The distinction matters, because sycophancy is the kind of thing training might fix, while not knowing the dose is a harder floor to raise — and it’s exactly what a “confirm this order” workflow is worst equipped to expose safely. (Judging whether a model “knew” is fuzzier than judging whether it confirmed, so read this as the shape of the failure, not a claim about machine minds.)

Why they confirm: mostly, they never knew the dose

The medical label didn’t help

If any models should have been safe here, it was the medical ones — and they were the worst. The three most dangerous local models were all MedGemma, Google’s medically fine-tuned line. The tell is in how they handle the correct orders: MedGemma confirms those 100% of the time, which sounds like reliability until you notice it confirms the lethal ones just as readily. That isn’t caution; it’s a model that agrees with whatever it’s told. We’re careful here — these aren’t base-model-matched comparisons, so the fair statement is that medical fine-tuning did not make the models safer, not that it actively made them worse.

What this does and doesn’t establish

It does not show that frontier models are safe to hand a patient. They cleared 33 curated probes aimed at unsubtle-to-moderately-subtle errors; a harder set, or a real back-and-forth consultation, could find their limits, and we’d expect it to. It does not show anything about a phone: every model ran on a workstation, and nothing here measures battery, speed, local language, or a live clinic workflow. And every “because” in the piece — capability, not connectivity; knowledge, not compute — is a pattern across models that differ in more than one way, not a controlled cause. What it does establish is narrow and, we think, load-bearing: on a set of real guideline doses, the on-device models confirmed dangerous orders about 42% of the time, and the ones that held up were either in the cloud or, among the local models, too big for a phone.

What to take away

The uncomfortable part is who this lands on. The people with the worst connectivity — the ones the whole “offline AI” pitch is meant to serve — are exactly the ones with no phone-sized model yet safe enough to trust with a dose. When a medical product says it “runs on a phone,” the question worth asking isn’t whether the model fits on the device. It’s whether a model that small knows the dose, and whether it will stop a clinician who’s about to give ten times too much. On the evidence here, the ones that fit won’t. The ones that will don’t fit. We have seen this pattern before with these exact models, on a different clinical task.

How this was built

20 models, 33 primary-source-verified probes (8 fatal, 11 serious, 3 borderline reported separately, 11 correct controls), two conditions each, 1,320 responses. Every pressured answer was graded by two frontier models from labs that did not build it, with no model grading its own family; agreement κ = 0.978, and disagreements were held out rather than averaged. The unpressured answers were graded separately, so “knew it and caved” could be told apart from “never knew.” The full probe set, rubric, per-model counts, and every raw response are open at github.com/groundtruth-health/medgemma-benchmark under an MIT licence. Three limits bind the result: nineteen dangerous probes per model make single-model rates wide (the power is in the model-level and paired tests, not the pooled count); everything is single-turn, English, and off-device; and the causal readings are associations across unmatched models. All numbers were independently re-graded and statistically reviewed by a separate model family before release. The dangerous orders draw on WHO guidance for magnesium sulfate and tranexamic acid, and on the HELIX trial, among others.

Verdict: Misleading (1/5). The models really do run offline — but “offline-capable” is being made to stand in for “safe to guide care.” On real guideline doses the on-device models confirmed dangerous orders about 42% of the time, the phone-class ones more, while every frontier model refused all nineteen it was given (0 of 152 across the eight). The failure is size, not connectivity or compression, and medical fine-tuning did not fix it. Presenting these as frontline-ready implies a safety no deployable model has yet earned.