Evaluating a research capacity strengthening program for Aboriginal community-controlled health organisations

Additional keywords: Aboriginal medical service, capacity building, health personnel, health service, health workforce, Indigenous, Indigenist, masterclass, Torres Strait Islander, training.

Across Australia, there are 143 Aboriginal Community-Controlled Health Organisations (ACCHOs) providing primary health care to Aboriginal and Torres Strait Islander (hereafter, Indigenous) communities. An ACCHO is a primary healthcare service ‘initiated and operated by the local Aboriginal community to deliver holistic, comprehensive and culturally appropriate health care to the community which controls it, through a locally elected Board of Management’ (National Aboriginal Community Controlled Health Organisation 2018). In addition to healthcare service provision, ACCHOs are frequently requested to contribute to health research projects undertaken with Indigenous community members and the health workforce. In these instances, ACCHO managers and health staff are approached by external research organisations and invited to act as liaisons between community members and the research team and contribute to research projects in the role of participants or members of the research team.

National guidelines for ethical conduct in Indigenous health research recognise that research ‘often involves unequal power relationships’ (p. 10) (National Health and Medical Research Council 2003). The chronology of medical research in Australia illustrates that Indigenous peoples were at first ignored and then subjected to the whims of non-Indigenous researchers and their research agendas (Thomas et al. 2014). This power imbalance between researchers and the researched is consistent with the historical and contemporary systemic power imbalances facing Indigenous peoples (Sherwood 2013; Watson 2016). It is not surprising then, that Indigenous peoples and organisations have long been critical of the research undertaken in their communities (Humphery 2001) and observe that ‘research has been a source of distress for Indigenous people because of inappropriate methods and practices’ (p. 22) (Cochran et al. 2008). There are also concerns regarding the benefit and impact of health research for over-researched Indigenous communities (Bainbridge et al. 2015).

To address power disjunctions in research and the question of benefit, Indigenous scholars in Australia, New Zealand and North America have developed Indigenous methodologies to re-centre Indigenous peoples and their knowledges in the research process. These methodologies highlight that research involving Indigenous peoples must be Indigenous-led, contribute to a decolonising of Indigenous peoples through privileging Indigenous voices and world views and be guided by and respond to Indigenous community priorities (Rigney 1999; Tuhiwai Smith 1999; Martin and Mirraboopa 2003; Bailey et al. 2006). In the Australian context, Indigenist research is that which is ‘carried out by Indigenous Australians whose primary informants are Indigenous Australians and whose goals are to serve and inform the Indigenous liberation struggle to be free of oppression and to gain power’ (p. 118) (Rigney 1999). In the primary healthcare setting, the implementation of Indigenous methodologies requires ACCHO staff with research literacy, knowledge and capacity so that they can: consult with Indigenous community members in an informed manner; effectively discern the research projects they wish to be involved with; negotiate power dynamics with partnering research organisations; and, ultimately, independently design and conduct research in their services.

One way in which Indigenous research capacity can be strengthened is through appropriately designed and targeted education. This paper describes the evaluation of a Masterclass Program that aimed to strengthen the research capacity of ACCHO personnel.

The Masterclass Program comprised three Masterclasses – Understanding Research, Undertaking Research and Research Evaluation. The Masterclass Program invited expressions of interest from ACCHOs through national advertisements disseminated by ACCHO networks. Interested ACCHOs were then contacted to determine the Masterclass to be delivered and negotiate the preferred date, location and venue. Masterclasses were delivered free-of-charge during 2014–17 by senior researchers, with at least one of them being Indigenous, in a 1-, 2- or 3-day face-to-face format. Participants were provided with Study Guides, provided access to the Masterclass Portal with links to electronic resources and were offered ongoing mentoring to support the implementation and integration of their learning within their organisation and daily work. At the end of each Masterclass, participants were encouraged to complete a short paper-based evaluation form. Feedback suggested the Masterclasses were well received. A formal evaluation to answer the research question ‘Can a short educational intervention strengthen research capacity in the Aboriginal Community-Controlled Organisation sector?’ was undertaken to better understand the reach and impact of these Masterclasses.

The evaluation considered both process and impact of the Masterclass Program. The process evaluation examined the reach of the Program, including the number and locations of Masterclasses delivered across Australia during 2014–17. The impact evaluation had three primary aims: (1) to identify changes in awareness, understanding or behaviour that resulted from attending a Masterclass; (2) to identify and describe outcomes that have been influenced by, or are a direct result of attending a Masterclass, including barriers to or enablers of those outcomes; and (3) to explore ways in which the provision of future Masterclasses and associated resources and support could be improved. Ethics approval was obtained from relevant ethics committees across jurisdictions; the Aboriginal Health Research Ethics Committee (South Australia), St Vincent’s Hospital Melbourne Research Governance Unit Human Research Ethics Committee (Victoria), Aboriginal Health and Medical Research Council Ethics Committee (New South Wales), Menzies Human Research Ethics Committee (Northern Territory) and the Western Australia Aboriginal Health Ethics Committee.

Individuals who had participated in a Masterclass by January 2017 and who had provided a contact email address (n = 210) were sent an online survey that focussed on the three aforementioned evaluation aims. At the end of the survey, respondents were invited to participate in a semi-structured face-to-face or telephone interview. The interview schedule invited discussion regarding the impact of the Masterclass Program in addition to the factors that influenced participant utilisation and implementation of Masterclass learning. Each interview participant provided informed consent before the interview commenced. Interviews were undertaken by two Indigenous members of the research team who had no prior involvement in the Masterclass Program. Digitally recorded and transcribed interviews and the notes taken from one non-recorded interview (accuracy checked by the participant) were de-identified before being uploaded to NVivo (ver. 11; QSR International Pty Ltd, Melbourne, Vic., Australia) for analysis. An inductive analysis was undertaken to identify emergent categories and overlying themes aligning with the three aims of the impact evaluation. The analysis was undertaken by an Indigenous researcher, who was also an interviewer, and then discussed and clarified with the research team comprising predominantly Indigenous researchers.

The Program delivered 27 Masterclasses to 260 participants across seven states and territories of Australia during 2014–17 in predominantly metropolitan locations. Some Masterclasses conducted in metropolitan settings included staff who had travelled from regional and remote ACCHOs. Masterclass participants included 161 (62%) ACCHO personnel, 56 (22%) ACCHO peak body personnel (i.e. staff from metropolitan-based advocacy organisations who represent member ACCHOs in each state and territory), 15 (6%) research institute staff and 28 (11%) attendees of a national Indigenous doctor’s conference that included personnel from mainstream medical services (n = 12), universities (n = 10) and non-government organisations (n = 6). Table 1 describes the distribution of Masterclass participants from metropolitan, regional and remote ACCHOs and ACCHO peak bodies across Australia. The participants were drawn from 31 ACCHOs, including 5 in metropolitan settings, 19 in regional settings and 7 in remote settings. Table 2 outlines the number and locations of Masterclasses undertaken each year. The Understanding Research Masterclass was requested most often (n = 16), followed by the Research Evaluation (n = 7) and Undertaking Research (n = 3) Masterclasses. Although the intended audience was ACCHO personnel, there were three Masterclasses delivered to Indigenous medical doctors, some of whom work in the ACCHO sector, and one Masterclass delivered to Indigenous research institute staff who work closely with the ACCHO sector.

Table 1.  Masterclass participation by Aboriginal Community-Controlled Health Organisations (ACCHOs) and Personnel across Australia
Note: An ACCHO Peak body is the metropolitan-based advocacy organisation representing member ACCHO services in each state or territory. The numbers in bold are totals for that particular state or territory. Qld, Queensland; NSW, New South Wales; ACT, Australian Capital Territory; Vic., Victoria; Tas., Tasmania; WA, Western Australia; NT, Northern Territory

Table 2.  Masterclass Program: timing and locations
Data are presented as n. ACT, Australian Capital Territory; NSW, New South Wales; NT, Northern Territory; Qld, Queensland; SA, South Australia; Vic., Victoria; WA, Western Australia

Forty-five completed online surveys were received (21% response rate). The demographic characteristics of survey participants are described in Table 3. The sample included Aboriginal (62%) and non-Indigenous (38%) participants, the majority of whom (55.6%) had more than 5 year’s experience in the sector and who were employed in diverse clinical, management and other roles. All respondents reported that the Masterclass content was relevant, with 67% reporting it to be very relevant. Nearly all reported that the content was useful and resulted in improved understanding (96%). A total of 80% of participants described an improved willingness to participate in research, whereas 76% reported an increased willingness to undertake their own research. Their suggestions for improvement of the Masterclass Program before, during and after the course are outlined in detail in Table 4.

Table 3.  Demographic characteristics of survey and interview participants

Table 4.  Suggested improvements for the Masterclass Program

Twenty-one survey respondents participated in semi-structured interviews; face-to-face (n = 4) and by telephone (n = 17). The interviews were undertaken within an average of 10 months (range: 3–26 months) following the interviewee’s participation in a Masterclass. The demographic characteristics of interview participants and their role in the ACCHO and research sectors are described in Table 3. The participants who agreed to be interviewed were demographically similar to the sample of participants who completed the anonymous survey. In addition, their ratings of the program (in relation to relevance, understanding and willingness to participate) were not significantly different. As depicted in Fig. 1 and described in detail below, participants described a range of Masterclass features that contributed to a supportive learning environment that led them to develop increased research capacity and become more engaged in research. Some participants also identified barriers to research engagement that limited their ability to participate in research activities.

Fig. 1.  Characteristics of the Masterclass Program and resultant impact on participant research capacity and engagement, as described by interview participants.

The reflections of participants demonstrated that the Masterclasses were delivered in a supportive learning environment that was culturally safe and accessible and delivered by encouraging facilitators in an informal and flexible format with relevant content and useful resources.

Interview participants talked about the style of delivery and referred to it being done ‘proper way’, giving a sense that people felt comfortable and considered the Masterclasses culturally appropriate.

…that’s what I think is a good thing about what they’re doing … it’s that proper way of how we learn … I think the way in which it was all done was deadly, you know. I felt challenged. I felt comfortable, you know. They encouraged you, and I think just being able to do that with your work colleagues, encouraged that collaboration too [IP 11, Indigenous Policy Officer].

The accessibility of the Masterclasses was highly valued by interview participants. The Masterclasses were conveniently held within their own workplaces and offered at no cost to ACCHOs.

…I could recommend it to all of our services – it’s free, I’m sorry, it’s free! How many things that are conducted by research leads and taught with people with such experience that are free?… that’s another big tick I think! [IP 04, Indigenous Manager].

Many participants spoke positively of the facilitators as being encouraging, supportive and inclusive and taking the time to help participants understand the information.

…if you’re given too much information, can’t really take it in and we had, enough breaks and the facilitators were excellent, really encouraging questions or if you didn’t understand something, going through it all, making sure everyone understood before we moved on… [IP 05, non-Indigenous Manager].

Participants also described the delivery as flexible and informal, which created a comfortable environment to learn in.

…I like how them mob, they come to us and, it’s the environment, in the workplace and the way in which their mob deliver it, not intense or overly formal [IP 11, Indigenous Policy Officer].

…there might have been, maybe 10 or 15 people in [the room], and that really made it easy to interact with the facilitators and ask any questions … it did really feel interactive … and it did feel relaxed. It was a good environment to learn in… [IP 17, Indigenous Administration Officer].

All interview participants described the information presented in the Masterclasses as relevant to their work. Some had specific responsibilities within their roles to implement research, whereas others did not have a direct responsibility. Some interview participants acknowledged that research can be a ‘dry’ topic and commented that the facilitators made the content interesting and enjoyable.

…it’s definitely fully relevant … it was [facilitator] who was speaking about the ethics and processes in the master class and, that’s something that I’ve been involved in with new projects, so applying to different ethics committees, and, so that was relevant … every single topic would apply to the projects I’m involved in [IP 16, non-Indigenous Pharmacist].

All interview participants found the Masterclass study guide relevant and useful.

…I found the masterclass great, and in fact the booklet that we were given, the information and the program that we were given, I have it here with me and often refer to it, so the resources I found very valuable… [IP 02, non-Indigenous Project Officer].

The areas for improvement of the Masterclass Program suggested by interview participants (regarding accessibility, format, content and assessment) are presented in Table 4, alongside recommendations from the online survey participants.

Participants described increased research capacity as a result of the Masterclasses in terms of changes to awareness, perceptions, understanding, critical thinking and confidence.

Many of those who did not have a direct responsibility in research talked about the benefits of having a greater awareness of research and identified that there could be opportunity to get more involved.

…so whilst I don’t undertake research myself regularly … I do read it quite a bit, through the networks and through the info that we get, linking in with our team, and the research officer here, because I answer general requests through emails, people are always wanting to link in with us with research, just knowing the ethical stuff and who to forward it onto… [IP 05, non-Indigenous Manager].

Participants talked about the importance of research and understanding how their health services could better use evidence to improve primary health care delivery. Many also talked about a shift in how they thought about their work and how they valued research. There was a sense that people’s perceptions of research had changed.

…in the past research was scary … we need the research from the ground up but just having the confidence in the language and [now] I can break that down a bit further or I actually know what you’re talking about, so they can’t fob you off by using terminology that, you know, sit there and be quiet, they have to engage with that conversation because someone else is aware of what they’re actually talking about… [IP 03, Indigenous Social Worker].

Interview participants had varying levels of understanding of research before attending a Masterclass, ranging from no research experience to practical experience and postgraduate degrees. Most participants said that their understanding of research had increased as a result of the Masterclass; others with advanced understanding described it as a ‘good refresher’.

…it’s facilitated my enjoyment in research … because it’s enhanced my understanding. I think when you begin to understand something, it’s a lot easier to enjoy the process, and definitely, the masterclasses aided in that [IP 10, Indigenous Research Assistant].

Participants also talked about how they were able to apply the critical thinking developed during the Masterclasses to other areas of their lives, such as when making informed decisions about personal matters such as their own health.

…I think it’s also helped in my own personal research so becoming someone who’s, very critically minded … and before making a decision being able to find, the best available evidence before making that decision, has been really great. So even aside from research within my own job, it’s really helped with personal decision making… [IP 10, Indigenous Research Assistant].

There was also a new confidence described by some interview participants to develop and lead their own research within their service. Some indicated that, previously, their services had only partnered with researchers instead of designing and driving the research themselves.

…it’s definitely given me more confidence because the more you learn about research and the processes, the more confident you become and you think one day I might actually design a full research project … the Masterclass did give me that confidence to think about the steps involved in research project planning, and that could be something I could think about doing in the future [IP 17, Indigenous Administration Officer].

Participants described a range of behaviours that had been influenced by their attendance at a Masterclass, which demonstrated enhanced research engagement.

People commented that after attending a Masterclass, they were more willing to be a research participant if the study was meaningful, as they could better understand the process.

…because I’ve been able to see the benefit that this masterclass has had for me, participating in research … sometimes you may not see a benefit for yourself, but it will be a benefit, for the wider population or the greater good, so to say [IP 10, Indigenous Research Assistant].

Some of the interview participants were not only motivated to conduct their own research within their services, but were also inspired to go back to their workplace and motivate colleagues about research.

…I’m not always there and so sometimes these, responsibilities might lie with someone else so I try and, inspire them [laughs] with that confidence. And often they’re quite busy and are not interested. But, you know, I can try and instil that confidence to them and, the significance… [IP 14, non-Indigenous Midwife].

Several interview participants described experiences of empowerment as a result of their new confidence. Instances were described where they had held discussion with researchers, reviewed research proposals and made informed decisions about whether their service should participate in the research.

…there were times when we might get a research proposal and we would actually pick it to pieces which we would never have done before. We would say this isn’t doable in this organisation … there is no way that we can do it, or they are going to have to change their approach, or there is no point. Just because somebody has done all the research doesn’t mean to say they know what Aboriginal people are going to want. But I guess it gave us the confidence to deal with research in a much more constructive way [IP 02, non-Indigenous Project Officer].

Some of those who participated in the Masterclass Program were inspired to further their education through a Bachelor’s degree, Masters or PhD.

Yeah, but I still want to know more. And that’s where I want that next lot of classes because I think the type of organisation we are, we’re heavily invested into trying to do things in a new way, so researchers are a huge part of that. I feel some of the more complex stuff, the research methodologies and all that … I want to know more about [IP 11, Indigenous Policy Officer].

Participants described service-level and system-level challenges that constrained their personal ability and the ACCHOs overall capacity to engage in research.

Limited funding for research activities and time constraints because of demanding workloads were frequently described as service-level barriers to the implementation of research learning in the ACCHO setting.

I’ve got somebody who’s keen, but … unless I get some funding for it, I just feel like they’re not going to look at it and take it seriously until I actually get to that next step… [IP 01, non-Indigenous Health Promotion Officer].

Oh, mate, no, look, at the moment, I’m – I’m flat chat trying to do what I’m supposed to do in the community let alone do anything else [IP 01, non-Indigenous Health Promotion Officer].

There were also system-level challenges described in relation to the ACCHO’s capacity to deliver program evaluations to funders. One participant recommended that governments provide an evaluation framework to ACCHOs at the initiation of program funding, which the ACCHO could then implement during the course of their work.

…it would be good for you to sell this to the government, to say, if you’re going to introduce a program, maybe you should set up an evaluation framework first yourselves, and then we have an understanding of what it is at the start of a program, rather than saying, oh look, we should evaluate this program and what the outcomes are, then making it up and we – that’s really the issue we face a lot of the time [IP 15, non-Indigenous Board Member].

Aboriginal Community-Controlled Health Organisation staff require research literacy and knowledge in order to advocate for and enable local community control of research, promote culturally safe research practice and ensure benefit to their communities in line with the principles of Indigenous health research ethical guidelines (National Health and Medical Research Council 2003; South Australian Health and Medical Research Institute 2014). This evaluation found that the foundation-level Masterclass, ‘Understanding Research’, was requested most often by the ACCHO sector, providing an indication of the sector’s current research literacy needs and that the priority of health services is service delivery and not undertaking or leading research. The nation-wide Masterclass Program described and evaluated here extends previous initiatives to promote research participation and capacity of Indigenous community members and ACCHO staff (Tsey 2001; Bailey et al. 2006; Mayo and Tsey 2009; Kelly et al. 2012). Research capacity development across the ACCHO sector is needed to enable future research to be undertaken by, or in partnership with, Indigenous peoples firmly grounded in primary healthcare practice who can hold this lens in the interpretation of findings and translation of evidence to the practice setting. The Program represented a welcome ‘giving back’ from the research sector to the ACCHO sector, which is in contrast to the taking away of knowledge commonly experienced by Indigenous peoples throughout the history of health research.

This evaluation found personnel from 31 ACCHOs were exposed to the Masterclass Program, representing 22% of the 143 ACCHOs across Australia. Most survey and interview participants had established careers in Indigenous health, which suggests that their developed research capacity is likely to be sustained in the sector. Participants described a developed research understanding and confidence, which led to an increased willingness to participate in research and, importantly, a greater sense of empowerment in interactions with external research partners. These findings suggest that the Masterclass Program may, in some instances, have contributed to a remediation of the power imbalances inherent in research through equipping ACCHO staff with the requisite knowledge to critique and make informed decisions about potential research projects and research partners. The system and service-level barriers to research engagement identified by participants must be addressed in order for more Masterclass participants to utilise what they have learnt.

The sustainability of the Program will inevitably be determined by funding mechanisms. A Program model that includes fee-paying participants is not viable for a sector with limited funding for professional development activities. Long-term investment in research capacity building activities is necessary to equip the ACCHO sector with the knowledge, confidence and skills to meaningfully engage in research. It is only then that quality research outcomes centred upon ACCHO service provision and the priorities of Indigenous communities will be delivered. The positive findings of this evaluation considered in the presence of ongoing requests from the ACCHO sector for further Masterclass training, highlight the need for ongoing funding of this Program to reach those ACCHOs not yet exposed to research capacity development opportunities.

A limitation of this evaluation included that some respondents who participated early in the Masterclass Program, and then the evaluation, found it hard to recall specific details and outcomes of the Program. However, inclusion of these participants enabled the long-term impact of the Program in relation to research capacity and engagement to be explored. The online survey considered the views of approximately one in five (21%) eligible Masterclass participants. The views of non-respondents are, of course, unknown and may differ from the views described here. The anonymous nature of the online survey was selected to reduce the potential for bias but did, however, preclude a comparison between responders and non-responders, which is a limitation of the evaluation design.

We have some confidence in the results as a result of consultations undertaken with ACCHO chief executive officers and ACCHO peak body representatives who spoke highly of the value of the Program to the sector and the need for ongoing Research Masterclass initiatives. They suggested that short videos designed for the ACCHO sector could compliment and strengthen the Masterclass Program. Moving forward, the Masterclass Program will be revised to incorporate recommended improvements identified by participants during this evaluation.

The development of research capacity in Indigenous health staff through a Masterclass Program can promote informed research awareness, confidence and engagement among ACCHO personnel. The capacity of Indigenous ACCHO staff must be developed so that they not only actively participate and partner in research, but so they can also play an active role in identifying research questions and priorities, undertaking research and evaluation activities, and translating findings into practice. The identified barriers to research engagement identified by this evaluation must be addressed so as to strengthen the effectiveness of research capacity building activities.

The authors declare that they have no conflicts of interest.