AI’s promise in health care comes with a hidden vulnerability in the training pipeline
Artificial Intelligence (AI) is rapidly becoming part of everyday health care, supporting diagnosis, documentation, triage, treatment planning, and resource allocation. Much of the public discussion has focused on accuracy, privacy, and fairness. In a new paper in the Journal of Medical Internet Research, KI researcher Farhad Abtahi highlights a less-discussed patient-safety issue: data poisoning, where manipulated training data can steer an AI system toward unsafe behaviour.
Importantly, the paper does not claim that health care systems are currently under attack. Instead, it synthesises prior security research and uses hypothetical, technically grounded scenarios to show how poisoning could happen in realistic health care settings, and how it might be prevented.
We spoke with Farhad Abtahi, researcher at the Department of Clinical Science, Intervention and Technology and the corresponding author of the study, who addressed a series of questions related to the research.
AI is often described as a major opportunity for health care. How do you see the potential?
AI can genuinely improve care by helping clinicians interpret complex information, reducing administrative load, and making workflows more efficient. Many of these benefits come from scale: models trained on large datasets can generalise across settings and support clinical work at pace.
What led you to direct your research toward security issues, and in particular, data poisoning?
Because security is also a patient-safety issue, and it is often less visible than performance and privacy. Data poisoning is particularly concerning because it targets the training pipeline. A model can be compromised in a way that looks “normal” during day-to-day use, and problems may only be detected after harm accumulates.
Many people assume that bigger datasets and bigger models reduce bias and make systems more robust. What does your paper add here?
That assumption is widespread: if the dataset is huge, any bad data should be “diluted”. But the evidence we synthesise suggests that large scale is not automatically protective; attack success can depend more on the absolute number of poisoned samples than their fraction of the dataset.
How small can an attack be and still matter?
Across the security literature we review, several studies report that as few as 100–500 poisoned samples can be enough to compromise systems in some settings, with attack success in the summarised studies often reported as ≥60%. We also discuss that detection may be slow in practice, especially in distributed or privacy-constrained environments.
Could you provide a concrete example and clarify what it represents?
One illustrative scenario in the paper is a “medical scribe Sybil attack.” This is hypothetical threat modelling, not a documented incident. The idea is that coordinated actors could introduce poisoned data at the point of creation through apparently normal clinical workflows, for example, by arranging many legitimate-looking visits with scripted histories. If an AI “scribe” transcribes these into the health record, those records can later become part of retraining data for multiple downstream AI systems.
The value of the example is the mechanism: poisoning can enter through trusted documentation, without “breaking into” a system in the traditional sense.
Are you suggesting that this is currently happening in health care?
No. We emphasise that the cases are hypothetical and synthesised from published research to show plausible pathways and make prevention actionable. The goal is to strengthen safeguards before real incidents occur.
Is there a ‘silver bullet’ here — meaning a single defence that actually solves the problem?
No, and that’s one of the main messages. There is no single technical fix that fully addresses poisoning risk. The paper argues for defence in depth, where multiple layers reinforce each other across the system lifecycle.
What does a multilayer defence look like in practice?
We outline four interacting layers:
- Detection and monitoring - continuous auditing; monitoring for anomalies and unexpected behaviour.
- Active technical defences, for example, adversarial testing and robustness techniques.
- Policy and governance - testing protocols, documentation practices, incident response), and
- Architecture and design choices that reduce the attack surface, including stronger provenance and supply-chain controls.
Will the EU AI Act address this risk and how?
It helps, but it won’t solve everything on its own. The EU AI Act has a risk-based structure and includes requirements that are directly relevant to poisoning risk, especially for high-risk AI systems, which must meet obligations around data governance, risk management, logging/record-keeping, and accuracy, robustness, and cybersecurity.
It also introduces lifecycle obligations such as post-market monitoring, actively collecting and analysing performance and compliance data after deployment, and mechanisms for reporting serious incidents. Those processes can make it easier to detect and respond when something goes wrong over time.
What remains harder is prevention: poisoning can be subtle, and problems may only become visible months or even years later, after harm has accumulated. That is why the paper argues for defence-in-depth, building safeguards into data pipelines, model development, deployment, and ongoing monitoring, rather than relying on any single measure or regulation, concludes Farhad Abtahi.
Publication
Journal of Medical Internet Research - Data Poisoning Vulnerabilities Across Health Care Artificial Intelligence Architectures: Analytical Security Framework and Defense Strategies
Abtahi F, Seoane F, Pau I, Vega-Barbas M. J Med Internet Res 2026;28:e87969