Taking the pulse of the health services research community: a cross-sectional survey of research impact, barriers and support

Health services research (HSR) is focused on the relationships between patients’ needs and demands relative to the supply, use and effectiveness of health services.¹ The Health Services Research Association of Australia and New Zealand (HSRAANZ) expands this definition to directly specify the need for multidisciplinary approaches with a focus on service quality dimensions, such as appropriateness and equity (http://www.hsraanz.org/what-is-hsr/, accessed 25 January 2018). To conduct this multidisciplinary work, the HSR community must be inclusive of a wide range of expertise, such as economists, statisticians, health providers and administrators, as well as policy developers. As such, there have been several attempts to report on the critical mass and composition of the HSR community.^2,3 A 2005 audit of the Australian and New Zealand (NZ) HSR community revealed that most health service researchers were part-time and held another professional role, such as teaching or administration; in addition, a minority of respondents had employment contracts of greater than 5 years (32%) and reported working with a critical mass of health service researchers (41%).² The authors of that audit,² along with other international examples^3,4 and an Australian survey of research activities in one health service,⁵ suggested there is a need to build the capacity of the HSR community. However, there are no recent data on the characteristics of the HSR community in Australia and NZ.

The degree to which HSR is funded by federal organisations is variable. Inequitable funding allocation was noted in the 2005 audit of the Australian and NZ HSR community, in which respondents perceived a tokenistic approach to HSR funding with organisations prioritising areas such as biomedical research.² Since 2012, the Australian National Health and Medical Research Council (NHMRC) allocated 6% of the annual budget (A$830 684 572) to HSR and public health.⁶ This distribution was criticised by a major review of research funding, which noted current HSR capacity is ‘fractured’ and ‘inadequate’ as a result of limited investment.⁷ Similarly, the NZ Ministry of Health recognised imbalanced funding distribution between basic, clinical, applied and translational research as a weakness of funding strategies.⁸ The effect of this funding allocation across these fields of research is largely unknown, and there is limited information on the degree to which health service researchers can access the methodological expertise and financial resources needed to conduct robust HSR.

Recent developments suggest a greater focus on HSR through the Medical Research Future Fund in Australia and the 2017–27 New Zealand Health Research Strategy.^8,9 More support for HSR carries responsibility for researchers to demonstrate the value and impact of their work; furthermore, demonstrating previous research impact is now emphasised in several grant applications processes.

Although methods to quantify the broader societal value of HSR are emerging, research impact is typically measured using bibliometric outputs that align with key performance indicators standard to universities.^10–12 However, publication in peer-reviewed journals may not be a priority for service-based researchers, who may focus on other process-based metrics, such as clinical guideline development or implementation, cost-savings within a service or improved patient or professional knowledge or engagement.⁵ An audit of Canadian-funded HSR reported close to one-third of HSR was conducted outside academic settings, such as regional health authorities (15%) and charitable organisations (16%).⁴ These data suggest that much of HSR may be conducted outside academic settings and should be evaluated by metrics beyond publication and citation rates. Without taking a broader inventory of the wide range of HSR impacts for both academic and service-based researchers, the true value of HSR is likely to be underestimated.^3,4

Established in 2001, HSRAANZ is a multidisciplinary association of health services, universities, research centres, government agencies and consumer groups. HSRAANZ focuses on capacity building and aims to understand and support the development and implementation of health services policy through robust evidence and rigorous evaluation. The present study was commissioned by HSRAANZ to describe the HSR community in Australia and NZ, and the impact of this community’s work. This study also reports the barriers experienced when translating HSR into practice and researchers’ access to resources and perceived level of financial, executive (e.g. HSR is embedded in organisation mission statements) or network (e.g. from peers) support.

The aims of this study were to: (1) describe the characteristics of individuals conducting HSR in Australia and NZ, and to provide an overview of completed and ongoing HSR projects; (2) report the perceived accessibility of resources available for HSR; (3) quantify the impact of HSR; and (4) report individuals’ perceptions of the barriers to conducting and translating HSR.

This cross-sectional survey of health service researchers was conducted according to the Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) statement.¹³ The University of Newcastle Human Research Ethics Committee provided ethics approval (H-2016-0397). Data collection occurred from February to May 2017.

A sampling frame was compiled from: (1) HSRAANZ subscription and conference delegate lists from the previous 5 years; (2) investigators who registered non-drug trials with the Australia and New Zealand Clinical Trials Register (ANZCTR) in the past 5 years and included ‘health service OR delivery OR healthcare’ as key terms; and (3) investigators listed on projects included in NHMRC, Australian Research Council (ARC) and Health Research Council of New Zealand (HRCNZ) funding announcements in the previous 5 years with field of research codes inclusive of health service research (e.g. Codes 11 and 17) where available (NHMRC and ARC).

To ensure ANZCTR records and funding announcements were related to HSR projects, projects were screened against inclusion criteria by two independent coders (EF, DR) with a third coder (CP) resolving any discrepancies (for detailed inclusion criteria, see Table S1, available as Supplementary Material to this paper). This inclusion criteria were pilot-tested with a random sample of project listings from each source (NHMRC, ARC, HRCNZ and ANZCTR). In all, 4556 records were screened with the inclusion criteria, with 3796 (83%) being excluded. After removal of duplicates, potential participants (i.e. authors of the published records or scientific contacts of registered trials) received personalised emails with survey links and three email reminders. Participants were able to enter a draw to receive a free HSRAANZ membership.

The survey included an introductory screen including participant information, brief introduction to the study, and the definition of HSR used by HSRAANZ. The survey progressed through four modules focused on respondents’: (1) characteristics; (2) access to methodological expertise for HSR (e.g. item stem ‘Are you able to easily access health economists for health service research’, item response options yes/no) and perceived executive, financial or network support; (3) perceived barriers to conducting and translating HSR; and (4) an option to provide an example(s) of their HSR projects and the impact of this work. Individuals were instructed to ‘Please report on the project that, in your opinion, had or will have the greatest potential to inform health service design, outcomes, or policy.’ Impact was measured using an abbreviated version of the Health Service Research Impact Framework.¹¹ Respondents could specify other impacts and indicate whether any were translated beyond study sites. A copy of the survey is provided in Appendix S1, available as Supplementary Material to this paper.

Summary statistics are provided for respondent characteristics, access to resources and project information. Goodness of fit statistics were used to identify any differences in accessible resources by participants’ affiliation (university or non-university). Participants who did not complete demographic items were removed from analyses. Data analyses were conducted using Stata 11 (StataCorp, College Station, TX, USA).

After duplicates and inactive email addresses were removed, 1918 researchers were invited to participate in the study. Of these, 424 individuals participated (22% response rate).

Most respondents were female (71%) and located in Australia (85%) and metropolitan areas (89%). The most common employers were universities (64%) and health services (17%); 57% of respondents were employed on permanent contracts. Respondents commonly held multiple roles as health service researchers (76%; with 28% holding this role for 10–20 years), educators (34%) and health professionals (19%). Complete demographics are provided in Table S2, available as Supplementary Material to this paper.

In all, 308 HSR projects were entered. Most were ongoing projects (55%) and aimed to determine effectiveness (58%), patient or health professional acceptability (45%) or accessibility (40%). The least common aim was to measure cost-effectiveness (26%). Approximately 35% of projects used qualitative methods and 24% were randomised intervention studies. Approximately 19% used mixed-methods designs and 19% included systematic reviews.

In all, 344 respondents (81%) reported the ease of access to methodological expertise for HSR purposes (Table 1). The least accessible resources were economists (52%), consumer advocates (49%) and practice change experts (34%). There were significant differences in reports of access to statisticians (P < 0.001), economists (P < 0.001), librarians (P = 0.02) and practice change experts (P = 0.02) between individuals affiliated with universities and those employed by non-traditional research settings (i.e. health service or government departments). The perceived difficulties accessing HSR expertise, specifically for service-based respondents, was mirrored in comments:

It needs to be easier to access support and on the ground education…We have a lot of junior doctors willing to drive research but it is very difficult to consistently find people willing and able to review a design and then to help evaluate and do statistics on results. (Participant 1; Australian, metropolitan health service, 5–10 years experience)

Clinicians have great ideas and often know where the gaps are, but don’t have the skills, experience or resources to answer these questions using rigorous methods. (Participant 2; Australian, metropolitan health service, 5–10 years experience)

Table 1.  Resources available to researchers and in nominated project
Unless indicated otherwise, data are given as n (%)

Approximately 22% of respondents reported there was no form of HSR support within their institution; 63% reported network support, 48% indicated executive support and 26% indicated financial support. Many participants noted the interplay of these barriers:

Many things are available to support us at my site but they all cost including statistician support…High-level executive support doesn’t equal funding support. (Participant 4; Australian, metropolitan university, year experience not specified)

I think we sometimes have a ‘cultural’ issue where health services want answers to questions quickly and cheaply but robust research takes time and this leads to a clash of agendas. (Participant 5; Australian, regional university, 5–10 years experience)

Health services research is still seen as a ‘luxury’ that is curtailed as soon as clinical services need boosting. (Participant 6; Australian, metropolitan university, 5–10 years experience)

There were no significant differences in the proportion of participants indicating the three support types between individuals affiliated with universities and those based in non-traditional research settings.

Of the 308 entered projects, most were funded (81%) and competitively awarded (82%). The most common funders were federal agencies (47%), state-based departments (19%) and universities (19%). Although participants indicated a lack of financial support within organisations, the incompatibility of external funding agencies’ priorities with HSR approaches was also a cited barrier:

Finding a balance between designing a study that will successfully attract major funding, and designing a study that might actually influence practice and policy. Real world research is hard to have funded because it’s rare that it lends itself to the [randomized control trial] or other NHMRC design favourites. (Participant 7; Australian, regional university, 5–10 years experience)

If HSR is to have an impact, there needs to be…sequestered funding in NHMRC project grant[s]. Colleagues on panels tell me that clinical/basic applications can always claim the potential for a panacea/magic cure in their research, whereas health services research can mostly only look at incremental benefits making it seem less appealing (despite a greater chance of success)… (Participant 9; Australian, metropolitan university, 10–20 years experience)

Only one-third (33%) of nominated projects included policy analysts and fewer included service administrators (2%). Absence of research end-users was a recurrent theme in comments, with participants noting policy makers should actively seek HSR to inform decisions, while researchers are responsible for driving uptake:

If HSR is to have an impact, there needs to be…a greater recognition on the part of health service funders and policy makers of the importance of implementing evidence rather than trying to fund research to provide evidence for positions that they have already decided on – i.e. this is what I want to do – find me the evidence to support it! (Participant 10; Australian, metropolitan university, 10–20 years experience)

If HSR is to have an impact, there needs to be…health policy makers that are interested and able to implement evidence based policy based on research findings. Researchers that are able to test policy relevant questions and examine their impact beyond the short term are needed. (Participant 11; Australian, metropolitan university, 10–20 years experience)

…Policy makers feel that research is too slow, not timely, too detailed, and too specific. To publish and survive in academia demands skills and a specific focus (detailed, methodical, explicitness) that is considered a barrier and problematic to government and policy makers. To successfully translate information for a very different audience takes a very different skill set and mindset – in my current environment, that ‘different’ skill set is not supported/fostered or even particularly well understood.’ (Participant 12; Australian, metropolitan university, 10–20 years experience)

The most common HSR impacts were conference presentations (94%), peer-reviewed articles (87%), improved health professional knowledge, attitudes, behaviours or outcomes (77%) and policy presentations (73%; Table 2). Compared with participants with ongoing projects (n = 168), a smaller proportion of participants with completed projects (n = 140) selected 15 of the 19 types of impact.

Table 2.  Impact and barriers to health service research (HSR) projects by project status (n = 308)
Data are given as n (%)

Individuals who reported completed projects (n = 140) indicated the most common translation barriers were resource related: limited budgets (49%), lack of time or staff (46%) and limited funds to implement findings after study completion (38%). Participants suggested it was not simply funding amounts, but the need to provide balanced and diverse grant types:

It is difficult to obtain funding for pilot projects/small scale research that could lead to competitive grant applications. (Participant 14; Australian, metropolitan university, 10–20 years experience)

Short-term funding makes it hard to gather momentum, it’s often just chasing funds with little time to focus on dissemination and sustainability post-implementation. (Participant 15; Australian, metropolitan university, 10–20 years experience)

Just over one-third of respondents (35%) reported organisational barriers; these barriers were typically administrative processes:

Research and funding processes are very complex and take ages to learn. Better administrative support and the ability to use research funds to buy some of this would be very useful in allowing researchers to spend more time doing research and doing less administrative work. (Participant 17; NZ, metropolitan university, 2–5 years experience)

This study described the characteristics of 424 health services researchers across Australia and NZ and reported their perceptions of HSR resources, impact and barriers. Although there are considerable barriers, individuals reported that many of the HSR projects contributed to the evidence base in the form of peer-reviewed articles and led to policy development and service improvements.

Collectively, respondents represented a diverse range of expertise, including health professionals, educators and researchers. Inconsistent with a 2005 audit of the HSRAANZ membership,² our results found a relatively greater proportion of health professionals (19% vs 10%) and a relatively lower proportion of policy analysts (16% vs 35%). The increased proportion of health professionals conducting HSR may reflect a shift towards clinician-led research, which is now encouraged via dedicated funding and policy such as the 2017–27 New Zealand Health Research Strategy.⁸ The finding aligns with a 2016 survey of research activity within an Australian health service that reported that 75% of health professionals had research experience.⁵ If health professionals are an increasing subgroup within the critical mass needed to conduct HSR, they must be adequately supported and mentored in this role with executive support through position descriptions and dedicated time.

Collectively, there was a perception of limited access to resources needed to conduct rigorous HSR, such as economists or consumer advocates. With many funding agencies now including economic analysis and consumer representation as grant assessment criteria, these two resources are invaluable. Furthermore, only one-third of nominated projects included a policy analyst and very few included health service administrators (2%). The absence of these research end-users was a common barrier, and participants noted the HSR community is responsible for facilitating research adoption in addition to policy makers taking an evidence-based approach to decision making. Although the data clearly demonstrate a general perception of poor accessibility, it is possible that respondents were unaware of existing resources within their institution. In this case, efforts to promote awareness and encourage collaboration within settings are needed.

Only 26% of respondents indicated financial support was available within their organisations. Coupled with 42% indicating they were not employed on a permanent basis and 52% reporting no executive support, there is concern for the sustainability and growth of the HSR community. This was articulated within participants’ comments, and the perception of inequitable HSR funding relative to other fields remains. Interestingly, there were no differences in the proportion of respondents indicating a lack of financial, executive or peer support between university and non-university settings. This suggests HSR may face an uphill battle in both academic and service-based settings.

A tremendous amount of work is being shared, implemented and translated into policy by the HSR community. Interestingly, the proportion of anticipated impacts for ongoing projects was higher than the proportion of actual impacts for completed projects in all items except policy impacts. This may suggest that researchers overestimate the potential impact of their current work. Although anticipating the impacts of ongoing projects is challenging, a detailed and structured dissemination plan is essential to maximise research utility. However, only 15% of individuals with completed projects indicated that limited dissemination plans were a translation barrier. Based on this finding, qualitative work exploring how researchers form and use disseminations plans is warranted.

First, due to the low response rate (22%) and the selection bias inherent with voluntary surveys, our results may not be fully generalisable to the larger HSR community. Only 16% of HSRAANZ conference delegates responded; because the goal of these international conferences is to bring together health service providers, researchers and policy makers, it is likely that only a proportion were active Australian or NZ researchers. Participation among known researchers (investigators on publicly listed projects) was much higher (34%). Second, researchers were asked to describe the projects they felt had or will have the greatest potential impact. Therefore, the findings may provide ‘best case’ scenarios only and rely on self-reported data. Although self-reported data, including interviews, are commonly used for evaluating impact,¹² the degree to which self-report bias influenced the study results is unclear. In addition, although the HSRI framework was a useful tool to quickly collate a wide range of project impacts specific to HSR, it is yet to be validated, and other frameworks are available, such as the Framework to Assess the Impact from Translational health research (FAIT).¹⁴

Funding for conducting HSR is clearly important. However, funding is also required to develop HSR infrastructure, support collaboration between health services and universities and to combine knowledge of the system with research experience and expertise. Formal training programs for health service staff and researchers, from short courses to PhD programs, will support broader interest and involvement in HSR and should look to fill identified survey gaps (e.g. health economics or implementation). Training courses would also provide opportunities for the development of collaborative relationships. In addition, funded secondments for health services researchers and embedding university-based researchers in health services may be a useful option to provide access to HSR expertise.

With regard to funding, the survey found that most HSR is funded by external bodies, such as the NHMRC, HRCNZ or state health departments. Health service providers have very limited capacity to fund HSR; quarantined research budgets to healthcare institutions with advice on how to manage funds would support and encourage relevant, high-value HSR to be conducted. Furthermore, to maximise this investment, research should be designed alongside dissemination plans to support the sustainable implementation of positive HSR and bolster awareness of valuable HSR.

There is a vibrant Australian and NZ HSR community in universities and health systems that includes a wide range of expertise and experience. This community has undertaken a large number of HSR projects with potential and realised impact on the health and well-being of Australian and NZ communities. Respondents indicated the potential for greater impact subject to addressing some key barriers, which included limited financial resources, methodological expertise, organisational support and opportunities to engage with collaborators. The reported survey of the state of HSR in Australia and NZ has been conducted at an opportune moment, and provides a baseline snapshot before the rollout of the Medical Research Future Fund in Australia and the recent New Zealand Health Research Strategy.

The authors declare that they have no competing interests.