Limited Demographic Data Transparency Found in Evidence Supporting FDA-Authorized AI Devices for Alzheimer Treatment
Researchers found limited demographic data transparency in the evidence supporting US Food and Drug Administration (FDA)-authorized artificial intelligence (AI)– and machine learning (ML)–enabled medical devices for the treatment of Alzheimer disease and related dementias (ADRD). This prevented the effective evaluation of the demographic representativeness of the devices’ training and validation datasets, according to a research letter published in JAMA.
“Data transparency—particularly regarding demographic representativeness of training and validation datasets—is essential to understanding performance variability and ensuring appropriate application in intended populations,” wrote Krista Y. Chen, MPH, Division of General Internal Medicine, Johns Hopkins University School of Medicine, Baltimore, Maryland, and study coauthors.
READ>>FDA Approves Modified Titration Dosing for Donanemab-azbt to Reduce ARIA-E Incidence in Early AD
The researchers sought to assess the availability and representativeness of data supporting authorization in 24 FDA-authorized AI– and ML–based devices for ADRD, most of which were indicated for volumetric quantification of brain structures (n=20). Dataset representativeness was determined by extracting study design and demographic composition (disease status, age, sex, race and ethnicity) from FDA approval summaries and peer-reviewed journal articles for each of the devices.
The researchers reported that there was no available information on training or validation datasets for 12 of the devices. Training data were reported in FDA summaries for 10 of the devices and in peer-reviewed articles for 5. Of these devices, 3 and 5 reported patient disease status, 2 and 4 reported age, 4 and 5 reported sex, and 0 and 1 reported race and ethnicity, respectively. For the 2 devices that reported validation data in FDA summaries and the 10 that reported validation data in peer-reviewed articles, 1 and 9 reported patient disease status, 0 and 8 reported age, 1 and 7 reported sex, and 1 and 0 reported race and ethnicity, respectively. No justification was given for the 23 devices that had incomplete reporting across disease status, age, sex, race, and ethnicity.
“Existing disparities in ADRD diagnoses and care among racially and ethnically minoritized groups underscore the risk of unintended harms from underrepresentation in training and validation data,” the authors explained. “Enforcing guidelines for data transparency and device labeling would help regulators identify potential biases and promote safe and effective AI and ML deployment in dementia.”
