Less than one third of publicly funded trials managed to recruit according to their original plan often resulting in request for additional funding and/or time extensions. The aim was to identify models which might be useful to a major public funder of randomised controlled trials when estimating likely time requirements for recruiting trial participants. The requirements of a useful model were identified as usability, based on experience, able to reflect time trends, accounting for centre recruitment and contribution to a commissioning decision.
A systematic review of English language articles using MEDLINE and EMBASE. Search terms included: randomised controlled trial, patient, accrual, predict, enrol, models, statistical; Bayes Theorem; Decision Theory; Monte Carlo Method and Poisson. Only studies discussing prediction of recruitment to trials using a modelling approach were included. Information was extracted from articles by one author, and checked by a second, using a pre-defined form.
Out of 326 identified abstracts, only 8 met all the inclusion criteria. Of these 8 studies examined, there are five major classes of model discussed: the unconditional model, the conditional model, the Poisson model, Bayesian models and Monte Carlo simulation of Markov models. None of these meet all the pre-identified needs of the funder.
To meet the needs of a number of research programmes, a new model is required as a matter of importance. Any model chosen should be validated against both retrospective and prospective data, to ensure the predictions it gives are superior to those currently used.
Large scale multicentre randomised controlled trials (RCTs) are regarded as the gold standard in rigorous, robust clinical research. Participants are recruited across a number of centres and are randomly assigned to control or intervention groups. Rates of participant recruitment, however, are not as straightforward as it may seem and this can have an enormous impact on the planning, execution and funding of trials. Failure to recruit sufficient numbers of participants, or extended delays in recruitment can have serious implications for the success or otherwise of the trial.
Research funders usually require applicants to calculate how many participants they require to answer their identified research questions, and how long they will need to recruit those participants into their study. In deciding how long they will require to accrue participants, trialists are often over optimistic - in 2007 Campbell found that less than one third of publically funded trials managed to recruit according to their original plan . In some sectors this problem affects up to 80% of studies . Careful planning and preparation can give better estimates of how realistic a particular accrual target is, and how long it will take to recruit the required number of patients.
As the leading public funder of clinical trials in the United Kingdom, the NIHR Health Technology Assessment (HTA) programme has an interest in optimising its investment in trials - and one way in which it can do this is by ensuring good estimates of timescales for research are available at the time of taking funding decisions.
In order to get good estimates, accurate and realistic patient recruitment predictions are required. Unfortunately, most HTA applicants adopt the unconditional approach  to recruitment prediction - suggesting that all their planned centres will start recruiting to their maximum capacity on day one. These centres tend not to be a pre-existing trial network, but are recruited on an ad-hoc basis for the needs of the trial. Given the difficulties with ethical and research governance approval at individual centres, this rarely happens.
As the application process is competitive - multiple teams compete to be commissioned to answer a single question - it might be expected that applicants overestimate their ability to recruit patients to trials, in order to make their applications appear more attractive. This appears to be borne out by 70% of trials failing to meet their original recruitment targets and requiring either additional time and money, or reconsidering the power of their study .
In order to facilitate the work of the HTA programme, and other research funders, the authors set out to investigate whether it would be feasible to provide a model for applicants intending to carry out pragmatic clinical trials to use to improve the prediction of recruitment to trials. The project to deliver this tool was divided into two phases. The first phase was a systematic literature review to identify currently existing tools and assess whether they met the programmes requirements. The objective of the second phase was to produce and test a predictive model for the HTA programme - through either building on a model found in phase 1, or starting from scratch.
This paper reports phase one - the systematic literature review.
For a predictive recruitment model to be useful to a wide group of stakeholders - including researchers and funders - it must exhibit certain characteristics, i.e. be:
1. Simple to Use and Understand - Applicants should be able to describe their proposed trial, and the model should provide the likely recruitment profile.
2. Can adapt to epidemiological changes - reflecting the research community's previous experience of the ability of studies to recruit patients and centres in the area of the trial.
3. Can adapt to environmental changes - changes affecting the conduct of trials in general for example, changes in ethical approval processes or regulatory requirements which then alter the lead time required to add a centre to a trial.
4. Able to take account of centre recruitment - The promptness of centres starting to recruit patients is a significant driver for overall recruitment in multicentre trials.
5. Able to inform commissioning decisions - The model should provide an expected recruitment period to allow proposals to be compared, rather than a probabilistic analysis.
We set out to identify existing models, and assess them against these criteria.
To identify currently existing models, a systematic literature review was conducted. Medline and Embase were searched in July 2008. The following truncated free text terms were searched in the title and abstract fields for: ((subject$ or patient$ or projected or accrual or fail$ or stop$ or clos$ or shut or predict$ or estimat$) adj5 (recruit$ or particip$ or enlist$ or enrol$)); plus MeSH terms "Patient Selection" or "Sample Size". Model types and statistical methods were also searched for: models, theoretical; models, organizational; models, statistical; Bayes Theorem; Decision Theory; Monte Carlo Method; Stochastic Processes; Clinical Trials Data Monitoring Committees; Data Interpretation, Statistical; Statistics; Poisson Distribution; normal distribution.
The papers identified by the search were reviewed by KB and AC to ensure all appropriate identified studies were included. Non-English language papers were excluded, due to a lack of resource for translation. Papers were excluded if they did not discuss using modelling techniques to predict patient recruitment to individually randomised clinical trials.
In order to identify candidate papers in the grey literature which may not be included in standard databases subsidiary searching was carried out using the Google search engine. No papers were found which were not identified through the formal search strategy (see Figure 1).
Data was extracted from the identified papers using a pre-defined data-extraction form. Data was extracted by KB initially and was then checked by LD and AC.
The categories used in the form were
1. question the identified model was designed to address, where not explicitly stated in the paper this is as determined by the reviewers
2. applicability of the model to the needs of the HTA programme and other pragmatic trial funders (eg to multi-centre studies, ad-hoc centres)
3. Approach of the model - the mathematics of the model, what data was used to populate the model, how it was validated
4. Centre Recruitment - did the model consider recruitment of both centres and individual patients?
Three hundred and twenty six papers were identified in the initial search, none of these were duplicates.
Following the initial screen of abstracts, 306 were excluded as the title indicated they did not address modelling of trial recruitment. Full papers were assessed for 20 studies.
Following the full paper review, 12 studies were excluded as they did not address modelling of trial recruitment.
The 8 studies included are summarised in the summary of study data extraction (Additional file 1.
Of the 8 studies examined, there are five major classes of model discussed: the unconditional model, the conditional model, the Poisson model, Bayesian models and Monte Carlo simulation of Markov models (see Table 1 for a summary of these types of model).
Table 1. Definitions of the different types of models
Carter's 2005 paper  considers the conditional model, a deterministic approach is also discussed by Moussa . The conditional model, is a development of the unconditional model which is commonly used by applicants for NIHR HTA programme grants. The conditional model allows the expected recruitment in any given month to vary, depending on other conditions within the trial such as how many centres are available to recruit. This more closely matches the real life experience of multi-centre trials relying on ad-hoc centre recruitment. Using relatively simple calculations this model can easily be constructed within a spreadsheet.
Three papers considered Poisson modelling [3,7,8]. Of these, two are linked, with one addressing the theory  and the other discussing a practical use of the method . Carter (2005)  presents the effect of these models on a trial protocol in development. As the prediction models become more sophisticated, the predicted accrual periods become longer. It should be noted that the mean predicted accrual periods for the conditional and Poisson approaches are very similar at 23 months and 23.2 month respectively. Anisimov  takes a similar approach to Carter (2005), in that he constructs a Poisson based model, but allows the recruitment rate to vary according to a Gamma distribution. He makes no allowance for varying the number of recruiting centres over time.
Williford considers the relative advantages of the Poisson and Bayesian approaches in monitoring ongoing recruitment within a trial and predicting future goals .
Gajewski develops the Bayesian approach of Williford to include establishing prior probabilities without requiring accrual data from a specific trial . His initial prior probability is derived from either previous experience in similar trials, or from clinical opinion based on epidemiology. As more trial specific accrual data becomes available, the prediction can be refined by bringing this data into the calculation.
Abbas  uses a number of Markov models to explore the maximisation of recruitment of patients in a minimum amount of time. He starts from the assumption of a fixed set of centres. He does present the only approach that deals with depleting willing trial participants from a population and the effect this has on the length of an accrual period. Abbas does not however consider the recruitment of trial centres - which could have an effect on reducing the effect of exhaustion of the trial pool.
Hadich uses a time series approach to analyse an existing set of data retrospectively, to determine if it would have been possible to predict recruitment to a trial. This time series model is not validated prospectively.
The characteristics of the identified models were compared to the criteria initially established, a summary of the comparison is shown in Table 2.
Table 2. Match between identified models with requirements of HTA programme
Comparing the different model types identified in this literature it becomes clear that the unconditional model does not fulfil sufficient of the pre-identified criteria because it is too simple and cannot take account of changes in centre recruitment or regulatory environment. The conditional model has promise because it satisfies all the criteria, however it has no fixed structure so a new model must be constructed for each use. The poisson model is promising, as using Carter 2004's illustrations you can allow lambda to vary for different periods of time in the trial as additional centres start recruiting. This involves simulating and combining data from a number of different poisson distributions which complicates the model slightly. The possion model can be further enhanced by allowing lambda to vary according to a gamma or uniform distribution as illustrated by Carter (2005), Amnisov and Williford. It is not clear from the literature exactly how centre recruitment can be taken into account in the poisson-gamma model and if this additional level of complexity is necessary. Baysian models rely on data being available to establish prior probabilities of recruitment which can be updated over the course of the study, making them more challenging to use for commissioning decisions, as they become more accurate as more trial data becomes available. Therefore, they are of more benefit when taking continuation decisions over the course of a trial. Markov models require structural model changes each time they are used, they are comparatively complex (especially when compared to conditional models) and there are no examples in the papers found of Markov approaches incorporating variations in study centre recruitment.
Five articles used recruitment data from existing trials to illustrate their model or establish the structure and parameters used in their models, whilst three presented analysis based on theoretical data only (Table 3). Only two papers tested the fit of their proposed models using trial recruitment data, Anisomov and Williford [7,8]. Wiliford compared the fit of the poisson to the negative-binomial (poisson-gamma) model, while Anisomov focused solely on the poisson-gamma model. They concluded there is some evidence that the poisson-gamma model provides an acceptable fit for trials with more than 200 participants 8] and more than 20 centres . However, Williford questioned its usefulness for any pre-study prediction unless the distribution of changes in patient intake rates can be estimated and modelled.
Table 3. Detail of Trial Data Reported in Modelling Papers
There are two significant drivers to overall recruitment in pragmatic multicentre randomised controlled trials unable to take advantage of a pre-existing network of recruiting centres: recruitment of patients, and recruitment of centres.
All of the studies identified primarily considered recruitment of patients. Five of the 8 studies did not consider the role of centre recruitment at all. The others discuss this as an issue, but make no explicit effort to include this important parameter in their model.
The HTA programme experience has been that poor centre recruitment (often due more to issues of process than clinician engagement) is a significant issue in causing trials to miss their recruitment targets.
For example, SANAD - a trial comparing various anti-epileptic agents - ran into difficulties in expanding its centres due to competition from other trials in epileptics underway at the same time . The trial managed to attain 90 centres, but not as quickly as was originally anticipated.
PAC-MAN, an investigation into the use of pulmonary artery catheters in intensive care units had difficulty recruiting centres due to the procedure which was then in place for seeking ethics approval . Of 95 intensive care units which expressed an interest in participation, only 79 eventually completed all the required processes to join the trial, and only 65 recruited at least one patient. At one stage the trial dropped to only recruiting 40% of the predicted number of patients.
REFLUX was a trial investigating surgery in gastro-oesophageal reflux disease . Similarly to PAC-MAC  difficulties with ethics resulted in the slow accumulation of centres, resulting in an extension of both time and money in order for the trial to meet its patient recruitment targets.
This review is not without limitation, for example, literature searches may not have identified unpublished or grey literature. While we are confident that we have complete coverage of relevant studies in the formal indexes, it would not be surprising had we failed to identify relevant information in the grey literature - especially anything which was not published in English. However, it seems to us likely that we have identified at least one example of each of the classes of approach which could be used to address this issue.
A further limitation may be the assessment criteria against which we judged the models. These were derived from discussions within the scientific secretariat at the NIHR Evaluation, Trials and Studies Coordinating Centre. This is the group which supports the HTA programme - consequently it is possible that the criteria that we used give emphasis to aspects of models which the HTA programme would particularly value. Had these criteria been developed by another group they may have developed a differing set.
In order for a model to be useful, trialists need to take into account local factors which influence a model's parameters. Examples would include delays in trialist and centre start times caused by variations in local ethics and local governance approvals or the existence of trials competing for the same patient pool. Delays caused by governance processes could be estimated from the previous experience of trialists or programmes making decisions. Patient competition must be estimated from knowledge of ongoing or planned trials, potentially run or funded by other groups and estimates of those factors could be used to adjust the parameters used in the predictive model - for example by reducing the expected number of patients recruited while a competitive trial is underway.
It is in the nature of pragmatic trials that an ad-hoc set of centres is required for each trial. The topics commissioned by the programme are so varied that a standing trial network would not usually be useful. There are advantages to be gained from a standing network - experience, habituation to trial recruitment and a pool of identified potential trial participants being examples. It is hoped that the NIHR research networks will in future aid with recruitment to trials in the UK- fulfilling in part the role of a standing trial network.
There appears to be a gap for a tool to predict recruitment to clinical trials of ad-hoc networks, which would account for both centre and patient recruitment, and recognise that one will drive the other. Such a tool might be built on the conditional, Poisson or Poisson-gamma models - although the model comparison in Table 3 suggest that the conditional model is the leading candidate. Bayesian models, however, show promise for use during the conduct of a trial.
The possible attraction of a more sophisticated model is the confidence interval it can place around the accrual period estimate. The major trade-off would be the increased complexity of use of the Bayesian or Poisson based models compared to the relative simplicity of the deterministic conditional model. Given Carter's (2005) findings that there was very little difference in prediction between the Poisson and conditional models he used, the relative simplicity of the conditional model may be more attractive . It should be noted that all the models are heavily dependent on the parameters used (e.g. estimates of monthly recruitment of participants). The confidence intervals from the Bayesian and Poisson models do not reflect the inaccuracy of the model parameters - but more the distributions described by those parameters. The provided confidence levels may therefore offer a false degree of security in the confidence of the estimates produced by the models.
The ease of implementation of a model would also contribute to which method should be used. A model which could be operated by an applicant on their own computer, or alternatively online, might be preferred over one which required significant statistical expertise to implement and use. Alternatively, recruitment prediction could become a service offered by local Clinical Trials Units, or Research Design Services, which may allow more sophisticated tailoring of models to the particular circumstance which a trialist encounters.
The NIHR research programmes do not currently have a preferred model for clinical trial prediction, consequently most applicants choose the simplest - the unconditional model. The programmes could give consideration to giving guidance on prediction of recruitment to applicants - and especially for them to include centre recruitment in their model. They may wish to suggest appropriate methods to use for recruitment prediction, mandate a particular existing model, or design their own model. As we move towards developing a new model, we will have to consider the conflicting needs for simplicity and estimation - possibly by developing more than one model and testing which gives the more useful predictions. Any model chosen should be validated against both retrospective and prospective data, to ensure the predictions it gives are superior to those currently used.
To meet the needs of a number of research programmes, a new model is required as a matter of importance. Any model chosen should be validated against both retrospective and prospective data, to ensure the predictions it gives are superior to those currently used
CTU: Clinical Trials Unit; HTA: Health Technology Assessment; ICU: Intensive Care Unit; NIHR: National Institute for Health Research; PAC-MAN: Pulmonary Artery Catheters in patient MANagement; RCT: Randomised Controlled Trial; RDS: Research Design Service; REFLUX: The effectiveness and cost-effectiveness of minimal access surgery amongst people with gastro-oesophageal reflux disease - a UK collaborative study; SANAD: StANdard versus new Antiepileptic Drugs.
All three authors are employed by the NIHR Evaluation, Trials and Studies coordinating centre, who manage the NIHR HTA programme.
KB designed and led the literature review, and wrote the first draft of the paper.
LD suggested the categories for the review, reviewed the papers and contributed statistical advice and contributed to the writing of the paper.
AC conceived the need for a recruitment prediction model for the NIHR HTA programme, contributed advice, and reviewed papers for the literature review.
All authors read and approved the final manuscript.
Alison Price for literature searching and article identification
Kevin Campbell for help identifying trials to use as examples
Campbell MK, Snowdon C, Francis D, Elbourne D, McDonald AM, Knight R, Entwistle V, Garcia J, Roberts I, Grant A: Recruitment to randomised trials: strategies for trial enrollment and participation study. The STEPS study.
Health technology assessment (Winchester, England) 2007, 11:iii.
ix-105PubMed Abstract | Publisher Full Text
Marson AG, Appleton R, Baker GA, Chadwick DW, Doughty J, Eaton B, Gamble C, Jacoby A, Shackley P, Smith DF, Tudur-Smith C, Vanoli A, Williamson PR: A randomised controlled trial examining the longer-term outcomes of standard versus new antiepileptic drugs. The SANAD trial.
Health technology assessment (Winchester, England) 2007, 11:iii-iv.
ix-x, 1-134PubMed Abstract | Publisher Full Text
Harvey S, Stevens K, Harrison D, Young D, Brampton W, McCabe C, Singer M, Rowan K: An evaluation of the clinical and cost-effectiveness of pulmonary artery catheters in patient management in intensive care: a systematic review and a randomised controlled trial.
Health technology assessment (Winchester, England) 2006, 10:iii-iv.
ix-xi, 1-133PubMed Abstract | Publisher Full Text
Grant A, Wileman S, Ramsay C, Bojke L, Epstein D, Sculpher M, Macran S, Kilonzo M, Vale L, Francis J, Mowat A, Krukowski Z, Heading R, Thursz M, Russell I, Campbell M: The effectiveness and cost-effectiveness of minimal access surgery amongst people with gastro-oesophageal reflux disease - a UK collaborative study. The REFLUX trial.
Health technology assessment (Winchester, England) 2008, 12:1-181.
iii-ivPubMed Abstract | Publisher Full Text
The pre-publication history for this paper can be accessed here: