User login
Credit: Darren Baker
A new study suggests that as many as a third of randomized clinical trials (RCTs) could be re-analyzed in ways that modify their conclusions.
The study also indicates that such re-analyses are extremely rare, due to many researchers’ unwillingness to share data.
“There is a real need for researchers to provide access to their raw data for others to analyze,” said John Ioannidis, MD, DSc, of the Stanford Prevention Research Center in California.
“Without this access, and possibly incentives to perform this work, there is increasing lack of trust in whether the results of published, randomized trials are credible and can be taken at face value.”
Dr Ioannidis and his colleagues used the MEDLINE database to evaluate re-analyses of RCTs and detailed their findings in JAMA. A related editorial is also available in the journal.
The team searched for articles written in English describing the re-analysis of raw data used in previously published RCTs. Meta-analyses were excluded from the study, as were studies testing a different hypothesis than the original trial.
The researchers screened nearly 3000 articles of potential interest and read the full text of 226. Of these, 37 were ultimately included in the study. Thirty-two of them had an overlap of at least 1 author from the original paper.
New conclusions
Thirteen of the re-analyses (35% of the total) came to conclusions that differed from those of the original trial with regard to who could benefit from the tested medication or intervention.
Three concluded that the patient population to treat should be different from the one recommended by the original study. One concluded that fewer patients should be treated. And the remaining 9 indicated that more patients should be treated.
The differences between the original RCTs and the re-analyses often occurred because the researchers conducting the re-analyses used different statistical or analytical methods, ways of defining outcomes, or ways of handling missing data.
For example, an RCT on the treatment of bleeding esophageal varices concluded that sclerotherapy reduced mortality but didn’t prevent rebleeding.
The re-analysis, which used a different statistical model of risk, suggested the treatment did prevent rebleeding but didn’t reduce mortality. The new conclusion suggested the intervention would be most appropriate for patients with rebleeding, rather than those at the highest risk of death from the condition.
Another study investigated the best way to deliver an erythropoiesis-stimulating medication to anemia patients by comparing a fixed dose administered once every 3 weeks with weight-based weekly dosing. In the re-analysis, the conclusion changed when investigators used an updated hemoglobin threshold level to determine when therapy should be initiated.
“The high proportion of re-analyses reaching different conclusions than the original papers may be partly an artifact,” Dr Ioannidis said. “By that I mean that, in the current environment, re-analyses that reach exactly the same results as the original would have great difficulty getting published.”
“However, making the raw data of trials available for re-analyses is essential not only for re-evaluating whether the original claims were correct, but also for using these data to perform additional analyses of interest and combined analyses.”
In this way, existing raw data could be used to explore new clinical questions and might occasionally eliminate the need to conduct new trials. 
Credit: Darren Baker
A new study suggests that as many as a third of randomized clinical trials (RCTs) could be re-analyzed in ways that modify their conclusions.
The study also indicates that such re-analyses are extremely rare, due to many researchers’ unwillingness to share data.
“There is a real need for researchers to provide access to their raw data for others to analyze,” said John Ioannidis, MD, DSc, of the Stanford Prevention Research Center in California.
“Without this access, and possibly incentives to perform this work, there is increasing lack of trust in whether the results of published, randomized trials are credible and can be taken at face value.”
Dr Ioannidis and his colleagues used the MEDLINE database to evaluate re-analyses of RCTs and detailed their findings in JAMA. A related editorial is also available in the journal.
The team searched for articles written in English describing the re-analysis of raw data used in previously published RCTs. Meta-analyses were excluded from the study, as were studies testing a different hypothesis than the original trial.
The researchers screened nearly 3000 articles of potential interest and read the full text of 226. Of these, 37 were ultimately included in the study. Thirty-two of them had an overlap of at least 1 author from the original paper.
New conclusions
Thirteen of the re-analyses (35% of the total) came to conclusions that differed from those of the original trial with regard to who could benefit from the tested medication or intervention.
Three concluded that the patient population to treat should be different from the one recommended by the original study. One concluded that fewer patients should be treated. And the remaining 9 indicated that more patients should be treated.
The differences between the original RCTs and the re-analyses often occurred because the researchers conducting the re-analyses used different statistical or analytical methods, ways of defining outcomes, or ways of handling missing data.
For example, an RCT on the treatment of bleeding esophageal varices concluded that sclerotherapy reduced mortality but didn’t prevent rebleeding.
The re-analysis, which used a different statistical model of risk, suggested the treatment did prevent rebleeding but didn’t reduce mortality. The new conclusion suggested the intervention would be most appropriate for patients with rebleeding, rather than those at the highest risk of death from the condition.
Another study investigated the best way to deliver an erythropoiesis-stimulating medication to anemia patients by comparing a fixed dose administered once every 3 weeks with weight-based weekly dosing. In the re-analysis, the conclusion changed when investigators used an updated hemoglobin threshold level to determine when therapy should be initiated.
“The high proportion of re-analyses reaching different conclusions than the original papers may be partly an artifact,” Dr Ioannidis said. “By that I mean that, in the current environment, re-analyses that reach exactly the same results as the original would have great difficulty getting published.”
“However, making the raw data of trials available for re-analyses is essential not only for re-evaluating whether the original claims were correct, but also for using these data to perform additional analyses of interest and combined analyses.”
In this way, existing raw data could be used to explore new clinical questions and might occasionally eliminate the need to conduct new trials.
Credit: Darren Baker
A new study suggests that as many as a third of randomized clinical trials (RCTs) could be re-analyzed in ways that modify their conclusions.
The study also indicates that such re-analyses are extremely rare, due to many researchers’ unwillingness to share data.
“There is a real need for researchers to provide access to their raw data for others to analyze,” said John Ioannidis, MD, DSc, of the Stanford Prevention Research Center in California.
“Without this access, and possibly incentives to perform this work, there is increasing lack of trust in whether the results of published, randomized trials are credible and can be taken at face value.”
Dr Ioannidis and his colleagues used the MEDLINE database to evaluate re-analyses of RCTs and detailed their findings in JAMA. A related editorial is also available in the journal.
The team searched for articles written in English describing the re-analysis of raw data used in previously published RCTs. Meta-analyses were excluded from the study, as were studies testing a different hypothesis than the original trial.
The researchers screened nearly 3000 articles of potential interest and read the full text of 226. Of these, 37 were ultimately included in the study. Thirty-two of them had an overlap of at least 1 author from the original paper.
New conclusions
Thirteen of the re-analyses (35% of the total) came to conclusions that differed from those of the original trial with regard to who could benefit from the tested medication or intervention.
Three concluded that the patient population to treat should be different from the one recommended by the original study. One concluded that fewer patients should be treated. And the remaining 9 indicated that more patients should be treated.
The differences between the original RCTs and the re-analyses often occurred because the researchers conducting the re-analyses used different statistical or analytical methods, ways of defining outcomes, or ways of handling missing data.
For example, an RCT on the treatment of bleeding esophageal varices concluded that sclerotherapy reduced mortality but didn’t prevent rebleeding.
The re-analysis, which used a different statistical model of risk, suggested the treatment did prevent rebleeding but didn’t reduce mortality. The new conclusion suggested the intervention would be most appropriate for patients with rebleeding, rather than those at the highest risk of death from the condition.
Another study investigated the best way to deliver an erythropoiesis-stimulating medication to anemia patients by comparing a fixed dose administered once every 3 weeks with weight-based weekly dosing. In the re-analysis, the conclusion changed when investigators used an updated hemoglobin threshold level to determine when therapy should be initiated.
“The high proportion of re-analyses reaching different conclusions than the original papers may be partly an artifact,” Dr Ioannidis said. “By that I mean that, in the current environment, re-analyses that reach exactly the same results as the original would have great difficulty getting published.”
“However, making the raw data of trials available for re-analyses is essential not only for re-evaluating whether the original claims were correct, but also for using these data to perform additional analyses of interest and combined analyses.”
In this way, existing raw data could be used to explore new clinical questions and might occasionally eliminate the need to conduct new trials.