Closing the chasm between research and practice: evidence of and for change

The usual answer to how to bridge the gap between research and practice or policy is to disseminate scientific findings more efficiently. Perhaps the question should not be how do we get more and better dissemination and implementation of the existing science to practitioners and policymakers, but instead, how do we ask the right questions in the first place and, in turn, how do we get better adaptation of the research practices into the real world.

Jonathan Lomas, former Director of the Canadian Health Services Research Foundation, illustrates this divide of the two world views of science and policy with the following scenario, describing a brief exchange between a group in a hot air balloon, and a man below.¹

‘Where am I?’ the people in the balloon asked. ‘You’re 30 meters above the ground in a balloon’ was the reply below. ‘You must be a researcher?’ ‘Yes, how did you know?’ ‘Because what you told me is absolutely correct but completely useless.’ From the ground, ‘You must be a policymaker.’ Yes, how did you know?’ ‘Because you don’t know where you are, you don’t know where you’re going, and now you’re blaming me.’

The dialogue could be applied equally if those in the balloon were health promotion program planners or practitioners, because the last phrase is what the practitioners often feel: that researchers do not know where they are going with their research and the researchers blame the practitioners for not faithfully applying the products of research.

Among the various challenges and opportunities, the overarching challenge is to close the gap between the evidence for implementation that policymakers, practitioners and communities need and what they are getting from researchers. The overarching opportunity is for the academic community to reform some of the peer review, editorial, and impact-factor scoring tendencies that have distorted the criteria for grant making, publishing, systematic reviews for practice guidelines, and for academic appointments, promotion and tenure in the health professional schools of most universities.

Opportunities in the conduct of research could extend participatory research principles to work with policy makers, program planners and practitioners in setting priorities on research questions relevant to the community,² and greater use of natural experiments incorporating surveillance, monitoring, and evaluation of programs with quasi-experimental and continuous quality improvement methods.³ The ultimate stretch in blending rigor and reality – internal validity and external validity – would be to combine participatory research with multisite randomised controlled trials (RCT) that would expand the external validity of the results of those trials while retaining internal validity.⁴

The most frequently quoted statistic on the gap or ‘chasm’ between researchers and practitioners is that ‘it takes 17 years to turn 14% of original [clinical] research to the benefit of patients’.⁵ These two estimates were derived from a series of estimated points of attrition in the flow of original research through the pipeline of publication and subsequent vetting in systematic reviews that produce guidelines for practice. For example, studies found that between 76 and 80% of clinical research findings are ultimately lost to attrition over time: 18% of it never gets submitted for publication because of negative results and investigators knowing that editors have a bias against publishing negative results, which is well documented.⁶ Then, as Balas et al. reconstructed the data from previous reviews: 46% of the submitted manuscripts are lost between submission and acceptance.⁵ This is where the peer review process comes particularly into play, along with editorial tendencies, with their biases against negative results, non-randomised trials, small or qualitative studies. The rejection is typically attributed to methodological limitations. Little or nothing is lost between acceptance and publication. We were still losing ~35% between publication and indexing into bibliographic databases by the mid-1990s, although this has probably improved since then with electronic publishing, indexing and cataloguing. Then another 50% were lost between getting the evidence into databases and their systematic review or meta-analysis that would lead to guidelines for evidence-based practice and eventually to textbooks. What might be surprising to most who have railed about practitioners not adopting evidence-based practices is that nothing is lost between the publication of guidelines and textbook summaries of systematic reviews and the practitioner implementation of them. It is simply a matter of the time it takes for 100% of such recommended practices to be applied, probably because the recommended practices based on controlled research requires various adaptations to fit the variety of practice circumstances, populations and needs. So the practitioners should not be blamed for not implementing them, eventually.

The cumulative time lost between the phases of vetting evidence from original research to eventual application is the other part of the quotable ‘17 years to apply 14% of original research.’ Balas et al. further compiled from various studies the time lapses between the foregoing phases.⁵ It takes ~6 months between submission and acceptance of a manuscript by a journal, another 6 months between acceptance and publication, ~3 months between publication and indexing, and about 6--13 years between accumulating and cataloguing electronic databases of published evidence and their eventual systematic review by one of the review commissions (e.g. Cochrane or the US Preventive Services Task Force). The estimate Balas accepted from the early 1990s was 9.3 years for ultimate implementation. We may reasonably assume an improved efficiency of online publication, cataloguing and systematic reviews since these estimates were computed. Yet, we might also expect growing scepticism among policy makers and practitioners about the appropriate fit of much of the research evidence for their varied populations, settings and circumstances, especially with the growing recognition of disparities in the health advances of populations.

The ‘Pipeline Fallacy’, refers to an academic and bureaucratic notion of funding, producing and vetting research that can be delivered to policy makers and practitioners as evidence-based ‘best practice’ guidelines, or even as requirements.⁷ A pipeline that flows one-way from theory to basic and highly controlled experimental research inhibits practical processes of informing research from the experience of practitioners and patients or communities that live with the studied problem in all its varieties on a daily basis. The pipeline process begins with identifying priorities for research funding, then squeezes those ideas into requests for proposals and review of research grants that meet the criteria or personal biases of peer reviewers. Outcomes from the research are then further squeezed into publication priorities and peer review. This is further shaped into systematic reviews of published studies that qualify for inclusion, and research synthesis into guidelines for ‘evidenced-based practice.’ Eventually, coming out the end of the pipe is a relatively small, highly distilled subproduct of the original research that does not take into consideration the implementation problems of the practitioner or policymaker with respect to funding, population needs, time and work demands, local practice or policy circumstances, professional discretion in adjusting interventions to individual needs, and the credibility of the results to those who practice in very different contexts than those in which the evidence was generated.

One driver of this directive approach has been the evidence-based medicine movement, a movement that has been, thankfully, successful in ‘clearing the decks’ of medical practices that did not have adequate evidence to justify them. It seems, however, that as this evidence-based medicine approach is applied to public health and health promotion, it encounters contradictory practicalities of intervention on behaviour, communities, policies and environments, and the methodological differences in conducting randomised trials and in inferring and generalising results from controlled experiments in the context of these interventions.

A few government agencies have recognised this gap and have attempted to bridge the relationship between researchers and practitioners or policymakers and to address the challenge of translating research into practice. This includes the National Health and Medical Research Council⁸ of Australia and the Agency for Healthcare Research and Quality⁹ in the US. These organisations often see a gap as a pothole that needs to be filled, a rut in the road, as it were. So they seek to fill it with their Translation of Research into Practice initiative, with annual conferences, education programs and more varied communications, as they attempt to move that evidence through the pipeline faster and more efficiently. They put some emphasis on clearing the way of the practitioners so they are able and supported in adopting and implementing evidence-based practice guidelines, as well as appreciating the need for some innovation and adaptation in all of that, rather than simply practicing every guideline with slavish ‘fidelity’.¹⁰

This conceptualisation of the problem, unlike the pipeline metaphor, is reminiscent of another fallacy that anthropologists have dubbed the ‘Fallacy of the Empty Vessel’.¹¹ This concept exemplified some health education efforts in the mid-20th century, when many mass media messages were based on the assumption that the public were an empty vessel, implicitly devoid of prior attitudes, values or behavioural constraints. The concept was that if enough information was poured into that empty vessel, it would fill up and good practices would spill over. The community and practitioners, however, are not empty vessels. They are composed of funding opportunities, population needs and demands, practice circumstances and professional discretion issues. To support change, this complexity needs to be recognised, along with the many benefits of practitioners, policy makers and researchers working together.

John Snow is usually given credit for the first success in controlling a cholera epidemic by establishing the evidence of a link between cholera incidence and the Broad Street pump in London, and recommending the removal of the pump handle. Crediting John Snow alone with this public health success, however, underplays the importance of some very important others and their collaborative and parallel approaches to identifying the problem, gathering evidence, and the translation of this evidence into policy and practice, as noted by Hanson et al.¹² Sir Edwin Chadwick, a politician who did not like smells, wanted to clean up all of those neighbourhoods and his campaigns through Parliament were a very large part of the story of policymaker and practitioner receptivity to the evidence John Snow brought on the Broad Street problem. Reverend Henry Whitehead was a community activist and his role is written about increasingly as people try to understand the dynamics that led to the changes. William Farr was a bureaucrat and statistician who kept the records in the Ministry of Health, such as it was at that time.¹³ It was the combination of these several individuals that gave momentum and closure to what we now call the London case study in epidemiology and community intervention, a milestone and benchmark for public health reform¹² and for the importance of practice-based and community-based evidence.^14,15

The prevailing standard of evidence is the RCT, but this has its limitations in community-based programs. The biggest overriding limitation is that it decontextualises most of the evidence it produces. It starts with an intervention tested by comparison of average effects with those of a control condition, i.e. no intervention or an alternative intervention. As shown in Fig. 1, it builds on the evidence and the theories about mediating and moderating variables expected to change based on previous evidence and theory. It then looks for change and outcome variables measured and compared between experimental and control groups. The generalisation of these interventions is problematic, however, because when taken out into the real world one must destandardise it. It needs to be tailored to the population or various subpopulations, and to the circumstances in which it will be applied. The interventions in the experimental trial are reduced to a simplistic form to minimise the confounding variables. Everything else is held constant, which is not the way it is in the real world. The interventionists have no discretion in adjusting the intervention to individual cases or circumstances. They are highly trained and supervised to adhere to strict protocols. The analysis of subgroups is discouraged and often censored by editors because the subgroups are not randomised.^16,17 This is most unfortunate because subgroup differences in response to the intervention would be the most useful evidence to program planners and practitioners in community health promotion as they seek to implement and evaluate the intervention in question.

Fig. 1.  The main characteristics of the randomised-controlled trial that produce limitations on the external validity or generalisation of its results to implementation and effectiveness in other populations, settings and circumstances.

Many alternatives to and variations on the RCT methodology exist for evaluating health promotion interventions in real time and in living communities.^18–20 Opportunities also exist to extend participatory research principles^21,22 from community-based participatory research to participatory research at all levels to work with policymakers and practitioners (all the players) in the conduct of natural experiments. The collection of data from surveillance, monitoring and evaluation of programs and continuous quality improvement efforts can be made much more relevant and in turn useful to policy makers, program planners and practitioners.³ Finally, the concept of combining participatory research with multisite RCT would expand the external validity of the results of those trials.⁴ The often-exclusive preoccupation with internal validity in academically controlled and published research makes external validity the fundamental problem in many if not most of the research translation issues.

Fig. 2 suggests four overlapping spheres of research for community health promotion: community-based,²³ academic-based,²⁴ and participatory research,^21,25 and community-based participatory research. The highly controlled academic research overlaps with community-based research to the extent that it is being done in communities. Participatory research overlaps with community-based participatory research and with other community-based research and practice-based research. So they are not distinct categories but rather overlapping categories. The process of planning and conducting research needs to have the right balance for the right questions that match a community’s needs.

Fig. 2.  Four overlapping spheres of research for community health promotion.

Striking the right balance involves reconciling at least two paradoxes. The internal validity--external validity paradox suggests that the more rigorously controlled a study is, the less reality-based it becomes. So it cannot be taken to a larger scale or generalised to other settings or populations, as it does not have very much to do with those other worlds. An RCT is usually testing the efficacy of an intervention – whether it works under more or less ideal circumstances – not its effectiveness.

The specificity–generalisability paradox suggests that the more relevant and particular a study is made to the local context, the less generalisable it may be to others. This often makes community-based participatory research less generalisable. The counter to this, however, is that when results from a study in one community are taken to another community, they will at least have greater credibility to the practitioners in the other community because the study was done in a real setting, more like their own, under circumstances more similar to their own, than the average RCT conducted in or under the control of academic settings. The homophily–social influence paradox occurs when community health workers such as indigenous aides who communicate well in the community are advanced up the professional hierarchy of the agencies that hire them, they become more socially distanced from their community. This, in turn, may undermine their effectiveness. Career ladder advancement is their right if they are effective, but it may in turn undermine some degree of their effectiveness.

The number one complaint from practitioners about evidence represented in ‘best practice’ guidelines based on systematic reviews of RCT evidence, according to a study of practitioners’ problems with evidence-based practice, was their perceived lack of external validity.¹⁶ They did not use the term ‘external validity’ but that is what they were essentially saying, which raises the issue of how objective and subjective evidence is weighed.

Academic scientists and usually professional practitioners view health and evidence about it through different lenses than the lay public in most communities, as suggested by Fig. 3. The scientists and professionals tend to have greater visual acuity (with their validated instruments) and place more weight on objective indicators of health. The layperson has greater acuity of visual and other senses that place more credence on subjective indicators.²⁶ If behaviours are the main point of the intervention, then subjective indicators are often more important than objective. At the very least more credence should be given to the subjective indicators of health.

Fig. 3.  The relative focus of scientific and public or lay lenses on objective indicators and subjective indicators of health.²⁶

How does this play out in relation to blending science, policy and practice? The tendency to favour internal validity over external validity in the funding, conduct and publication of research, and in the systematic review of research to produce guidelines for practice, means that the strength of evidence is favoured over the weight of evidence.¹⁷ Giving greater credence to a wider range of evidence – from evaluation of community-based practice as well as academically controlled experiments – should produce a more balanced weighing of evidence than the single-minded internal validity criteria favoured in the current funding, conduct and systematic reviews of research evidence.

There have been great advances in public health and health promotion in the 20th century in reduction of cardiovascular deaths and stroke deaths, tobacco control, immunisation, injury control, especially automobile injuries, and in occupational health injuries and death. Much of these advances have been achieved without randomised controlled trials preceding their implementation in policy and practice. If these successes are to continue and to gain traction in other areas such as alcohol, HIV and obesity control, there needs to be a continued recognition of the value of RCTs but also greater appreciation of and blending with other sources of data particularly practice-based evidence from surveillance, monitoring and evaluation, so that there is greater relevance to practice and better adaptation of the evidence into the real world.