An investigation of two widely used health datasets reveals problems of poor provenance and potential reliability of the data, raising concerns about the clinical prediction models created from them and prompting calls for stricter research standards.
Study: Evidence of unreliable and poorly sourced data in clinical prediction model research and clinical practice. Image credit: Andrey_Popov/Shutterstock.com
A new study published in BMC Medicine conducted exploratory analyzes to examine data quality and reporting in two large, publicly available stroke and diabetes datasets widely used in clinical prediction models.
The fast-turnover survey raises concerns about data quality
By 2024, researchers had published approximately 250,000 clinical prediction models to help clinicians diagnose disease, estimate prognosis, and guide treatment decisions. Because these models can directly impact patient care, their reliability depends on both robust analytical methods and high-quality underlying data.
To improve transparency, the Transparent Reporting of a Multivariate Prediction Model for Single Prognosis or Diagnosis (TRIPOD) guidelines were introduced in 2015, providing a framework for reporting prediction model research. The 2024 TRIPOD+AI update expanded these recommendations to include both traditional regression models and machine learning models, placing even greater emphasis on documenting data provenance, metadata that records where the data came from, how it was collected, and whether it can be leveraged and reused.
The increasing availability of large routinely collected health data sets has accelerated the development of clinical prediction models. But it has also fueled what researchers describe as “rapid turnaround” research: quick, standardized studies that prioritize publication volume over substantive scientific advances. According to the authors, this approach can increase the risk of false positives and waste valuable research resources.
Data quality concerns have already prompted some publishers and journals to tighten their editorial policies following the misuse of widely used datasets, including the Global Burden of Disease and National Health and Nutrition Examination Survey databases. Other examples, such as unverifiable cancer cell lines that ultimately led to item recalls, have further highlighted the consequences of poor data sourcing.
Although initiatives such as the Findable, Accessible, Interoperable and Reusable (FAIR) principles encourage better data management, adoption remains inconsistent. Similarly, while repositories such as Kaggle make datasets widely accessible, they do not require users to provide comprehensive source information. The authors argue that without stronger standards for verifying data provenance, unreliable data sets can continue to circulate in the scientific literature, potentially undermining evidence-based medicine.
The study evaluated provenance using TRIPOD+AI standards
Two popular, publicly available health datasets with potentially poor data provenance were selected for their high number of downloads and their relevance to clinical prediction modeling research. One dataset focused on stroke and the other on diabetes, both accessed from Kaggle on 27 August 2025. The current study aimed to highlight data sourcing issues in clinical prediction models using these datasets.
Each data set was evaluated using nine TRIPOD+AI data sources, and exploratory analyzes were conducted to assess authenticity, including controls for simulated data, unexpected correlations between variables, non-normal distributions, and duplicate series. Kaggle’s public discussions about the origin of the data were also reviewed and concerns raised with Kaggle.
Google Scholar was searched to identify peer-reviewed articles that used these datasets for model development or validation, and full texts were screened for inclusion. Exclusions included non-peer-reviewed works and non-English articles. The authors noted that this search strategy likely underestimated the use of these datasets because studies that did not include direct Kaggle links would not have been captured.
Inconsistencies in reporting were documented, along with checks for data source disclosures and revisions to statements about ethical approval and potential clinical use. Policy uptake was examined through Altmetric and Overton. Author relationships by country were analyzed and research volume over time was plotted using OpenAlex.
Questionable data sets support more than 125 studies
An evaluation of two widely used Kaggle health datasets for clinical prediction model research revealed significant concerns about data provenance and authenticity. Of the 653 research results identified, 125 published articles developed or validated clinical prediction models using these datasets.
Evaluation using nine TRIPOD+AI items revealed serious deficiencies in both datasets: neither provided information on when, where, why or how the data was collected, nor was authenticity possible Iindependently verified. Both datasets failed all nine TRIPOD+AI data source assessment items.
The stroke dataset included 5,110 cases, which contained irregular patient identities, improbable blood glucose and age distributions, and unrealistically missing data. Similarly, the diabetes dataset included 100,000 cases containing repeated and abnormal values, artificial correlations, and many duplicate records. Together, these findings indicate that both data sets are likely synthetic, constructed, or otherwise unreliable and thus unsuitable for research or clinical application.
The 125 items included come from 32 countries. However, reference to ethical approval was rare and most articles did not have sufficient information about the origin of the data. Only a small number described their data sources, and the majority did not meet basic transparency standards.
However, these data sets were widely reported and often used to make recommendations for clinical care. Of the 125 studies, three models showed evidence of potential use in practice, one was reported in a medical device patent, and 86 review articles reported on these models.
Some articles described actual or potential use in clinical settings, and 11 studies developed web-based prediction tools or applications with graphical user interfaces, two of which were publicly accessible. None of the studies were reported in policy documents.
The number of publications using these datasets has continued to grow, despite ongoing concerns about the quality and authenticity of the underlying data.
Improving data transparency to protect patient care
The current study highlights the urgent need to address the use of unreliable data in clinical prediction model research. Reliable data and transparent methods are essential to ensure reliable clinical decisions and ensure patient care. The authors recommend action by journals, publishers, repositories, researchers, and clinicians to improve standards and promote responsible research practices.
The authors also pointed out that this study examined only two publicly available Kaggle datasets, and that it remains unclear how widespread similar data provenance issues are in other datasets and repositories.
