There is a number I keep coming back to.
When researchers catalogued 1,016 AI-enabled medical devices authorised by the US Food and Drug Administration between 1995 and 2024, they found that 84% of them worked on images — mostly radiology. Devices that draw on electronic health record data, the kind of system that predicts who is at risk of kidney failure or sepsis or a first heart attack, made up 0.4% of the total.
Zero point four percent.
That sits oddly against the volume of conversation. Every health system I speak to is discussing predictive analytics. Every conference has a track on it. The published literature contains hundreds of risk models for single conditions. And yet the regulated, deployed, commercially available body of clinical AI is overwhelmingly about reading pictures.
The gap between that conversation and that number is what my new book is about.
The problem is not the algorithm
I have watched enough of these projects to notice that the ones which fail rarely fail for technical reasons. Nobody’s programme collapsed because they chose gradient boosting when they should have chosen a neural network.
They fail for reasons that sound almost boring when you write them down.
They predict an outcome nobody can act on. A model identifies patients at high risk of readmission, goes live, and the health system discovers it has no readmission-prevention service to offer them. The list is accurate and completely useless — worse than useless, because it now consumes clinician attention, which is the scarcest resource in the building.
They depend on data that does not exist yet at the moment of prediction. A feature looked wonderfully predictive in the training set because, in the historical record, it was populated after the event the model is meant to anticipate. Performance collapses in production and everyone is baffled.
They fire into a system already saturated with noise. A systematic review of more than 570,000 prescriptions found that clinicians override roughly 90% of drug–drug interaction alerts. Those alerts are rule-based, well-established, and frequently correct. They are ignored nine times in ten because there are simply too many of them. Any new AI alert enters that environment as a competitor for attention that has already been fully spent.
And they go live on somebody else’s performance figures. The most instructive published case is a proprietary sepsis prediction model that was, at the time, running at hundreds of hospitals. When a team at the University of Michigan evaluated it independently across 38,455 hospitalisations, it achieved an area under the curve of 0.63. At the recommended threshold it identified 33% of sepsis cases — missing two-thirds — with a positive predictive value of 12%, while generating alerts on 18% of all admissions.
None of that is a modelling failure. It is a failure of everything that surrounds the model.
The most consequential bias I know of came from a single design decision
In 2019, Obermeyer and colleagues published an analysis in Science of a commercial algorithm used to select patients for high-risk care management programmes in the United States. It was applied to roughly 200 million people a year.
Race was not an input. The algorithm was not designed to consider it.
At any given score, Black patients were substantially sicker than White patients with the same score. The authors estimated that correcting the bias would raise the share of Black patients receiving extra help from 17.7% to 46.5%.
The cause was one choice: the algorithm predicted healthcare cost as a proxy for healthcare need. Because less is spent on Black patients at equivalent levels of illness — a consequence of unequal access, not of lower need — the model learned to rank them as healthier.
I find this case clarifying rather than damning. The team was competent. The proxy sounded reasonable and was widely used. The bias was invisible in overall accuracy metrics. It was found only because somebody stratified performance by race and looked.
That is the whole lesson in miniature. The outcome definition mattered more than the algorithm. And no amount of technical sophistication would have caught it — only the discipline of asking the right question at the right moment.
What is in the book
The book is 153 pages, organised into nine parts and 24 chapters. It takes the technical material seriously — there are chapters on feature construction, model selection, discrimination and calibration, and validation design — but it treats them as necessary rather than sufficient. Roughly half the book is about the data foundation underneath the model, the clinical workflow it enters, the equity and safety obligations it carries, and the monitoring that has to continue for as long as it runs.
Some of what it contains:
A thirteen-step development workflow, with the specific failure mode attached to each step. Knowing what goes wrong is more useful than knowing what should happen.
An honest treatment of evaluation. Why accuracy is close to meaningless for rare outcomes. Why sensitivity and specificity belong to the model while positive predictive value belongs to the population — which is why a model transported to a new setting can behave so differently without anything having gone wrong. Why calibration, not discrimination, is usually the property clinical decisions actually depend on.
Five worked case studies — cardiovascular risk, type 2 diabetes, chronic kidney disease progression, sepsis and early deterioration, and cognitive decline. Each one benchmarks against the existing standard of care, because the honest starting position is that a well-built conventional risk equation is a high bar to clear.
The Responsible Healthcare AI Framework: ten questions, applied as sequential gates, covering clinical need, data quality, model performance, clinical validation, explainability, human oversight, equity, privacy and security, scalability, and continuous monitoring.
A nine-phase implementation roadmap with entry and exit criteria for each phase, including the one almost everyone skips — running the model silently in production, generating predictions nobody sees, before a single clinician is shown an output.
Checklists throughout, consolidated into a single comprehensive one at the end, designed to be printed, adapted to your own terminology, and worked through with the relevant people in the room.
Every citation in it was verified against the publisher or an institutional repository. Where I could not confirm a detail, I left it out rather than filling the gap with something plausible. Healthcare AI has suffered from a surplus of confident claims, and I would rather the book be useful than impressive.
Who I wrote it for
Clinicians who want to know what to ask a vendor. Data scientists who can already train a model and need to understand what is different about this domain. Administrators and health-system leaders deciding whether to fund a programme, and what to expect from it. Informatics and public health students who will spend their careers on this.
It assumes no programming background, and it does not condescend to readers who have one.
The argument underneath all of it
The purpose of a clinical prediction model is not to be accurate. It is to help a clinician and a patient make a better decision than they would have made otherwise. Accuracy is instrumental to that — and a model can be highly accurate and completely useless.
Keeping the decision at the centre, rather than the algorithm, is the single habit that most reliably separates programmes that change what happens to patients from programmes that produce papers.
Or, as the book puts it in its closing line:
AI should augment clinical intelligence — not replace clinical responsibility.
AI for Early Disease Risk Detection and Clinical Decision Support: A Practical Framework for Scalable and Responsible Healthcare AI is available now.
