16 February 2015

FDA and research quality

'Research Misconduct Identified by the US Food and Drug Administration: Out of Sight, Out of Mind, Out of the Peer-Reviewed Literature' by Charles Seife in (2015) JAMA Intern Medicine comments
Every year, the US Food and Drug Administration (FDA) inspects several hundred clinical sites performing biomedical research on human participants and occasionally finds evidence of substantial departures from good clinical practice and research misconduct. However, the FDA has no systematic method of communicating these findings to the scientific community, leaving open the possibility that research misconduct detected by a government agency goes unremarked in the peer-reviewed literature.
Seife's objectives were
To identify published clinical trials in which an FDA inspection found significant evidence of objectionable conditions or practices, to describe violations, and to determine whether the violations are mentioned in the peer-reviewed literature.
He undertook a
Cross-sectional analysis of publicly available documents, dated from January 1, 1998, to September 30, 2013, describing FDA inspections of clinical trial sites in which significant evidence of objectionable conditions or practices was found. 
For each inspection document that could be linked to a specific published clinical trial, the main measure was a yes/no determination of whether there was mention in the peer-reviewed literature of problems the FDA had identified. 
The main outcomes were -
Fifty-seven published clinical trials were identified for which an FDA inspection of a trial site had found significant evidence of 1 or more of the following problems:
  • falsification or submission of false information, 22 trials (39%); 
  • problems with adverse events reporting, 14 trials (25%); 
  • protocol violations, 42 trials (74%); 
  • inadequate or inaccurate recordkeeping, 35 trials (61%); 
  • failure to protect the safety of patients and/or issues with oversight or informed consent, 30 trials (53%); and 
  • violations not otherwise categorized, 20 trials (35%).
Only 3 of the 78 publications (4%) that resulted from trials in which the FDA found significant violations mentioned the objectionable conditions or practices found during the inspection. 
No corrections, retractions, expressions of concern, or other comments acknowledging the key issues identified by the inspection were subsequently published.
Examples of unreported inspection violations were -
Case 1 
A publication describing a stem cell trial in 26 patients with ischemic limbs stated that “all patients recognized and were aware of major clinical improvements in the treated (more ischemic) leg, despite no significant clinical changes in the control (less ischemic) leg.” However, an FDA document revealed that 1 patient had a foot amputated 2 weeks after administration of the stem cells. 
We found no correction or retraction. 
Case 2 
Eight of 16 FDA inspections of sites involved in a clinical trial of rivaroxaban, a novel anticoagulant, had been rated OAI. These inspections had uncovered evidence of various transgressions, such as “systemic discarding of medical records,” unauthorized unblinding, falsification, and “concerns regarding improprieties in randomisation.” Consequently, the entire study, RECORD 4 (Regulation of Coagulation in Orthopedic Surgery to Prevent Deep-Venous Thrombosis and Pulmonary Embolism 4), was deemed unreliable by the FDA. These problems are not mentioned in the article describing the study’s resultsor in other publications associated with the trial.
Case 3 
A researcher was caught falsifying documents in a number of trials, in part because those falsifications led to the death of a patient undergoing treatment in a clinical trial comparing 2 chemotherapy regimens. The researcher had falsified laboratory test results to hide the patient’s impaired kidney and liver function, and the first dose of the treatment proved to be fatal. The researcher pleaded guilty to fraud and criminally negligent homicide and was sentenced to 71 months in prison. 
Although this episode is described in detail in FDA documents  as well as court documents, none of the publications in the peer-reviewed literature associated with the chemotherapy study in which the patient died have any mention of the falsification, fraud, or homicide. The publications associated with 2 of the 3 other studies for which the researcher falsified documents also do not report on the violations. 
Case 4 
A clinical site in China taking part in a large trial of apixaban, a novel anticoagulant, had apparently altered patient records. If one were to exclude the data from the patients at that site, the claim of a statistically significant mortality benefit disappears. For this reason, among others, the FDA wrestled with whether it was appropriate to allow the manufacturer to claim a mortality benefit. None of this discussion appears in the literature. The claim for the mortality benefit, which has appeared in the literature since 2011, consistently relies on the full data set, including data from the site at which the research misconduct allegedly occurred. This is true even for an article that was published52 nearly 18 months after the alleged research misconduct was discovered. In addition, the mortality benefit analysis of the FDA-approved drug label as of August 31, 2014, is also based on the full data set despite a recommendation from the FDA’s Office of Scientific Investigation that data from not just the problematic site but 23 additional suspect Chinese sites be excluded. 
Despite the fraudulent data, when all the suspect Chinese sites are excluded rather than just the one at which the evidence of alleged research misconduct was found, the mortality benefit becomes statistically significant at the P = .05 level once again. One FDA analyst, commenting on the “data quality issues” in this clinical trial, complained about the agency’s lack of transparency and poor handling of evidence of problems with trial data: Some of the responsibility for the data quality issues rests with us, the FDA: We have approved drugs ignoring similar data quality issues, granting superiority claims, and not discussing in the labels the data quality issues. We must stop doing this.
His conclusions are -
When the FDA finds significant departures from good clinical practice, those findings are seldom reflected in the peer-reviewed literature, even when there is evidence of data fabrication or other forms of research misconduct.