How to read a health-chatbot impact claim

Our platform has reached over two million people and exchanged tens of millions of messages.

Look how many people it's helping — this is working.

That reach is a monitoring metric, and monitoring is not evaluation. The WHO guide to monitoring and evaluating digital health interventions draws the line explicitly: monitoring tracks the quality and fidelity of the intervention's inputs; evaluation asks about its outputs and impacts — user satisfaction, process improvements, health outcomes, cost-effectiveness. A count of users or messages sits entirely on the monitoring side. It tells you the intervention was delivered and opened; it says nothing about whether a single health outcome changed. The archetype is the voice-IVR helpline that announces it has reached millions of callers on a hotline — a real, impressive delivery number — with no trial behind it of whether those callers were any healthier than people who never called.

And notice what kind of count it is, because reach numbers are usually inflated by construction. A "two million reached" figure is almost always cumulative — everyone ever registered — not who is still there this month. Cumulative sign-ups, monthly active users, and genuinely-engaged users (people who came back and sent more than a message or two) are three very different numbers that can sit an order of magnitude or more apart, and it is the biggest of the three that reaches the headline. The person who tried it once, found it unhelpful, and left is counted in that total forever.

A randomized trial found statistically significant improvements across almost every domain.

The effects are big, and the tool clearly works.

How small "significant" can be. The PROMPTS cluster RCT — a genuinely rigorous, pre-registered, independently evaluated trial of a maternal-health messaging platform across 40 Kenyan facilities (N = 6,139) — found exactly this shape of result: statistically significant improvements of +0.06 to +0.09 standard deviations on five self-reported indices. The p-values are real, earned by a large sample. But an effect that size is a near-total overlap between the treatment and control groups. A p-value answers "is there any difference?"; it is silent on "how big?" — and here the honest answer to "how big" is "barely visible." The forest plot shows all six domains; the distribution plot shows what a 0.09 SD gap actually looks like.

There is also a plausible reason the effect looks small that is about who was measured, not what the messages did. Participants were recruited by enumerators at health facilities during antenatal care — a sample that had, by definition, already walked into the formal system. In the trial's own baseline, about 80% (4,965/6,139) had already received some antenatal care and nearly 90% owned a mobile phone. It is hard to move "routine care seeking" much in women who are already, demonstrably, seeking routine care: a group that starts near the ceiling leaves little room for an effect to register. That is a standard restricted-range concern, not something this trial sets out to prove — so read it as context for the small numbers, not their demonstrated cause. It is also the context that a reach total or a "twice as likely" headline, drawn from the same pre-selected, care-engaged, phone-owning population, never sets beside the number.

And notice who is in neither arm. Because enrollment happens at an antenatal visit, both the treatment and the control group are made of women who already attend care. Women who never reach antenatal care — often the most isolated and highest-risk — sit outside the trial entirely, so it can say nothing about them. A facility-enrolled tool is measured on the already-connected and stays silent about the group a maternal-health programme most needs to reach; an average effect that reads as "it works" can be an effect only for the people who were easiest to help.

Scroll the chart sideways to read it →

Those curves are an illustration. The trial's own data say the same thing: here are two of the care-seeking domains it measured — antenatal and postnatal visit counts — with the treatment and control arms plotted directly. They sit almost on top of each other.

Scroll the chart sideways to read it →

Mothers in the program knew more about newborn danger signs and sought more care.

The program improved health.

That these were self-reported survey indices, not measured clinical outcomes — and that the study was never built to detect a health outcome in the first place. The PROMPTS authors say so in as many words: "All outcomes were self-reported, and the study was not powered to detect effects on health outcomes." Reporting that you know the danger signs of newborn illness, or that you attended a clinic, is a proxy — one or two steps upstream of the thing that matters: a complication caught, a birthweight, a death averted. It is entirely reasonable to measure proxies; knowledge and care-seeking are on the causal path. The error is reading a proxy as if it were the outcome. The trial's own conclusion calls for future work to establish health-outcome impact — an admission that this study, by design, did not.

SMS reminders improve retention and outcomes — the evidence is clear.

This is a settled, evidence-backed intervention. Fund more of it.

What happens when the same intervention is tested rigorously: it often doesn't move the outcome. In the WelTel PMTCT trial, a Kenyan RCT of weekly interactive text-messaging for mothers in HIV care across six clinics, the messaging arm did no better than control on retention — a risk ratio of 1.02 (95% CI 0.92–1.14, p = 0.697); the authors conclude it "was not associated with improved retention in PMTCT care." The Mobile WACh NEO trial, a two-way maternal-newborn SMS-plus-nurse-chat program, was likewise null on its primary outcomes: "the primary analysis revealed no significant differences in outcomes between the intervention and control groups." Null results are not the field failing. They are what honest measurement looks like when a plausible idea is tested against a real counterfactual.

Users are highly engaged — strong daily activity and thousands of active users.

People love it and keep coming back.

That engagement curves collapse, fast, and an "active users" headline is a snapshot taken at the top of the cliff. The largest independent look at real-world usage — Baumel and colleagues' panel study of 93 popular mental-health apps (median 100,000 installs), measured from actual device usage rather than developer reports — found, across the 59 of the 93 apps (63%) that had retention data, a median 15-day retention of 3.9% and 30-day retention of 3.3%; daily opens fell more than 80% between day 1 and day 10, and across all 93 apps a median of just 4.0% of installers opened the app on any given day. The authors' conclusion is the whole warning: "only a small portion of users actually used the apps for a long period of time." A big install or active-user number, quoted once, hides the shape of the curve underneath it.

Averages hide it too. Every one of those churned users still counts toward "reach," and an "average messages per user" is a mean — a handful of power users pull it up while the typical user sends one or two messages and never returns. In a cumulative total, a person who sent two messages, judged the answers low-quality, and quit looks identical to one who engaged for months. That is why the honest numbers are distributional: the median messages per user, the share who sent more than one, and the share still there after week one.

Scroll the chart sideways to read it →

Our most engaged users had the best health outcomes.

Using the tool more caused better outcomes — so drive engagement and scale it.

That the people who engage are different from the people who don't — so an engagement–outcome correlation measures who they are, not what the tool did. The Mobile WACh NEO engagement analysis is a clean dissection of this claim. Messaging was self-selected: younger, more educated, unmarried, first-time mothers messaged more. The engagement–outcome associations then pointed in opposite directions — more messaging went with a greater rise in newborn danger-sign knowledge (Adj Est 0.39; 95% CI 0.09–0.68), but with lower odds of early breastfeeding (aOR 0.62; 95% CI 0.45–0.86). Contradictory signs are the fingerprint of confounding. The authors decline the causal read directly: "we are unable to conclude that messaging directly impacted any health outcomes ... mothers who engaged more may have sought additional support" — that is, they were already the care-seeking type.

The model was validated / tops a medical benchmark — so it's safe for patients.

It passed a test, so it's safe to answer real patients.

That acing a benchmark is a different test than being safe on open patient questions. When sixteen physicians red-teamed four leading public chatbots on 222 patient-posed medical questions (888 responses), the rate of problematic answers ran from 21.6% to 43.2% across models, and outright unsafe answers — ones a physician judged could lead to harm — from 5.0% to 13.5% — from models that score well on medical-licensing-style exams. The authors' conclusion is blunt: "millions of patients could be receiving unsafe medical advice from publicly available chatbots." A benchmark score, or the word "validated," tells you a model cleared some fixed test; it does not tell you how often it is wrong on the messy questions real people ask, and it says nothing about whether there is a path to a human when it is.

Scroll the chart sideways to read it →

It's low-cost and scalable — cents per user.

It's a cost-effective way to improve health.

That a cost per participant is not a cost per outcome — and here the PROMPTS trial is unusually concrete. The same paper reports Jacaranda's own estimate that the platform "costs 74 cents per participant for their lifetime on the platform," covering message delivery, technical infrastructure, field enrollment and county engagement. That is a genuinely low unit cost, transparently itemised — though it is the program's own figure, and a cost per mother enrolled, not per mother helped. Divide 74 cents by a 0.06–0.09 SD, mostly self-reported effect, and the cost per unit of health actually changed is unknown — which is why the same authors leave cost-effectiveness as an open question rather than a claim. Two things even this transparent unit cost doesn't value: the public-system opportunity cost behind each enrollment — the government nurses' and facilities' time the program leans on, which the figure doesn't price in — and any comparison to what the same money, or the same staff hours, would achieve spent another way. And the wider base of evidence is skewed: reviews find the economic evidence for digital health is "disproportionately concerned with high-income countries and hospital settings", with LMIC cost-effectiveness evidence, in one review's words, "urgently required." A "cents per mother" line borrows the credibility of a cost-effectiveness claim without doing the arithmetic that would earn it.