Strategies to minimise bias in analysis of randomised trial evidence
Professor John Simes, Julius Centre, UMC & NHMRC Clinical Trials Centre, University of Sydney
Randomised trials are designed to provide an unbiased estimate of treatment effect in the population studied but can still be prone to a number of biases both within the individual trial and when considering all relevant randomised evidence.
Important distinctions are made when determining an unbiased estimate for the studied population versus whether this is an unbiased estimate for other patients (generalizability of results; external validation). I wish to concentrate here just on whether treatment effect is unbiased within the given trial setting (internal validity) and particularly the risk of false positive results under the null hypothesis of no treatment effect.
Within a single trial the risk of bias has been well studied (eg: Cochrane Risk of Bias tool). But a special issue arises related to post-hoc decision rules in the analysis due to changes in the trial progress and emerging evidence from other trials. Under what circumstances is it appropriate to modify the planned analysis without biasing the results?
The second issue relates to combining the results of all relevant randomised clinical trials. Over 30 years ago, I demonstrated that a systematic review confined to just published trials may be biased, if the unpublished trials (which were more likely to have been negative) were not included. A systematic review of all trials (published or not) is preferred and, if not feasible, then a review confined to just the prospectively registered trials. This strategy has motivated the efforts internationally to register all trials prospectively with the trial registry being the principle source of trials for inclusion.
A third and related question relates to the post-hoc nature of the systematic review. Often reviews of the current evidence are partly based on knowledge of some or many of the trial results to be included in the review. Further, unlike a single randomised trial where a primary outcome is defined, the same set of trials (or subsets of them) may be used to address many overlapping questions each in a different forum. Hence the problem of multiplicity may be addressed in a single trial but remains a big challenge for combined analyses of trials and especially prone to post-hoc decision rules. Prospective meta-analysis provide a strategy to address these challenges and provides similar strengths to a single large scale clinical trial.
For each of these strategies several examples (over two decades) will be presented to illustrate the approaches as well as some potential pitfalls.
Professor John Simes is currently a Visiting Professor at the Julius Centre, UMC on sabbatical from the University of Sydney.
John Simes is a Senior Principal Research Fellow and Founding Director of the NHMRC Clinical Trials Centre (CTC), University of Sydney, Australia. He is a leading international researcher in clinical trials, with particular interests in clinical trials in cancer, cardiovascular disease, diabetes and neonatal medicine. He serves on many clinical trials research committees, and advisory boards, including cancer cooperative groups and international cardiovascular studies. Professor Simes was also founding Director of the Sydney Catalyst Translational Cancer Research Centre, a virtual centre of cancer researchers in New South Wales, Australia anda founding Director of the Australian Clinical Trials Alliance (ACTA).
As Director of the CTC, he has led a research program for the past 30 years aimed at improving health practice and health outcomes through better use of clinical trials research. This research involved more than 800 researchers and collaborators, 80 000 patients and 140 multicentre trials. He has provided teaching in decision analysis and its use in cost-effectiveness analysis in the Master of Public Health and Clinical Epidemiology courses at the University of Sydney.
Professor John Simes practices as a medical oncologist in neuro-oncology at Royal Prince Alfred Hospital and the Chris O’Brien Lifehouse. He is Professor of Clinical Epidemiology at the Faculty of Medicine, University of Sydney.
Professor Simes’ research interests include clinical trials methodology, quality of life assessment and integrating trial evidence with the goal of improving clinical practice and health outcomes. Professor Simes has contributed to more than 380 peer-reviewed publications with many recent examples having a significant impact on current knowledge and clinical practice including advances treatments in cancer, cardiovascular disease and diabetes. His research has also been important in developing better methods for combining trial evidence.