Critical reflection for researcher–community partnership effectiveness: the He Pikinga Waiora process evaluation tool guiding the implementation of chronic condition interventions in Indigenous communities

Additional keywords: community health: planning, healthcare disparities, Indigenous health services, planning techniques.

Health inequities for chronic, non-communicable diseases between Māori and non-Māori in New Zealand (NZ) are persistent and compelling (Harris et al. 2012; Ministry of Health 2013), consistent with inequities faced by Indigenous people in other countries (Anderson et al. 2016). For example, 7.2% of Māori have diabetes compared with 5.1% of Pākehā (NZ European) (Ministry of Health 2015). Further, Māori have a 1.8-fold greater health burden than non-Māori and a 9-year lower average life expectancy (Ministry of Health 2013). Racism, along with unjust distribution of social determinants of health, are root causes of these inequities (Harris et al. 2012; Harris et al. 2015). Additionally, the lack of commitment by the New Zealand Government towards its obligations under Te Tiriti o Waitangi (The Treaty of Waitangi) is a fundamental driver of the unequal distribution of the determinants of health and inaction in the face of need (Harris et al. 2015).

Health service researchers and community health researchers advocate participatory research approaches (e.g. community-based participatory research, CBPR) as being critical for developing health interventions that improve health equity (Wallerstein et al. 2018). In fact, systematic reviews and meta-analyses demonstrate the positive effects of participatory health research on health outcomes and health equity (Cyril et al. 2015; O’Mara-Eves et al. 2015). Two key explanatory factors for the success of participatory approaches are: (i) the commitment to build trusting and synergistic partnerships; and (ii) the transformation of power hierarchies through a critical-reflective process (Khodyakov et al. 2011; Jagosh et al. 2015; Wallerstein et al. 2018). This critical-reflective process is grounded in the emancipatory philosophy of Paulo Freire; specifically focusing on listening, dialogue and action in an iterative cycle of collective reflection moving towards social justice (Freire 1970). The process is designed to centre Indigenous knowledge and promote power sharing.

Despite the evidence and theories guiding participatory research, not all participatory health interventions address systemic health service issues or even engage with health service providers. To promote transformational improvements in health service delivery for Indigenous communities, the He Pikinga Waiora (HPW) project began in 2016 with the aim of identifying what makes health interventions work for Māori communities. The research team partnered with two community health providers to explore the roles of ‘participatory research’, ‘kaupapa Māori’ (Māori principles) and ‘matauranga Māori’ (Māori knowledge) in the development of sustainable and effective evidence-based interventions to slow the progress of pre-diabetes to diabetes among Māori. With this in mind, the HPW project leaders formed partnerships with two Māori community health organisations: Te Kōhao Health and Poutiri Trust.

The project team co-designed the He Pikinga Waiora Implementation Framework (HPW framework) to guide the successful implementation of evidence-based health interventions for Indigenous communities (Oetzel et al. 2017). The HPW framework’s intention is to apply a holistic and collaborative approach to health research; one that ‘embraces collaborative efforts among community, academic, and other stakeholders who gather and use research and data to build on the strengths and priorities of the community for multilevel strategies to improve health and social equity’ (Wallerstein et al. 2018; p. 3). The HPW framework has Indigenous self-determination at its core, and four elements that are essential to implementation: community engagement, cultural centredness, systems thinking and integrated knowledge translation (Oetzel et al. 2017). Community engagement includes the shared decision-making and communication responsibilities between academic and community members and researchers. Cultural centredness emphasises the involvement of community members in defining and solving problems along with sharing resources for transformation. Systems thinking reflects holism, and the complexity and interrelatedness among various stakeholders related to a health issue. Integrated knowledge translation involves the engagement of end-users (people implementing or using an intervention) throughout the research process to ensure ‘buy-in’.

Thus, HPW operates on the premise that making health interventions work for Indigenous communities requires their involvement from the outset, and throughout all phases of the implementation process including: problem definition, design, implementation and evaluation. To test the efficacy of the HPW framework, the two community partnerships were entrusted with co-designing and implementing an intervention that primarily focuses on reducing the progression of pre-diabetes to diabetes along with other related conditions (e.g. obesity and cardiovascular disease). The specific aim of this current study is to describe and report on the process evaluation conducted during the development of the interventions, as part of the critical-reflection process.

A process evaluation was conducted between December 2017 and January 2018 to get a sense of how well the research–community partnerships were performing according to the framework and to also guide changes to further phases of the project. During the time of the evaluation, the two community partnerships were finalising the details of their interventions before implementing them at the end of March 2018 and throughout the following year. This study shares the results of the process evaluation and discusses the changes made as a part of the reflection process and within the context of the extant literature.

Since launching the HPW project in 2016, a considerable amount of time and resources were vested in planning, but most importantly forging relationships between researchers and Māori communities, and within each group. These included: (i) identifying potential research and community partners; (ii) building and widening a collaborative network; and (iii) developing and formalising respectful relationships.

This study utilised a case study approach using mixed-methods data collection (Yin 2014). Case study methodology is an approach for understanding the holistic elements affecting the development of a health intervention and is frequently used for process evaluation in participatory research projects (e.g. Israel et al. 2013; Wallerstein et al. 2018).

The HPW framework forms the basis of the process evaluation tool (see Box 1), which measures how well the research partners (both community and academic) and the partnership as a whole are following the framework while in the process of developing and implementing an intervention. There are 22 items for community engagement, 22 for cultural centredness, seven for systems thinking, six for integrated knowledge translation and five open-ended interview questions. The items for community engagement and cultural centredness come from scales with established psychometric properties from a study of 2000 CBPR partnerships in the USA (Oetzel et al. 2015). Similarly, the open-ended questions were developed in the same project. The systems thinking items were developed for this study based on systems thinking process evaluation by the Institute of Environmental Science and Research NZ. The integrated knowledge translation items were adapted from a guide developed by the Canadian Institutes for Health Research (CIHR) (2012). The process evaluation tool was developed through an iterative process with the research team, along with consultation with the HPW team’s advisory board.

The evaluation tool was administered in two parts by a member of the HPW core evaluation team (lead author of this paper) who was not directly involved in either project. First, participants were sent an online survey asking them to rate how they felt about each statement (item) using a Likert scale ranging from 1 (not at all) to 6 (complete extent). Second, members were then invited to answer five open-ended questions via telephone, video conference or in person. The interviews were audio/video recorded and then transcribed. The interview questions invited members to express their impressions about the project and to state specific examples that illustrated elements of the HPW framework.

The participants included 10 team members, five from each project (eight in-depth interviews and nine surveys). The entire partnership included 18 members; two researchers (one the evaluator-lead author) and six members of an advisory board, who were not directly involved in the research projects and were not interviewed/surveyed. Participants included eight Māori, seven women, four academic researchers, three community researchers and three members from affiliated community providers. The academic research team consisted of two researchers, one health and social practitioner and one public health physician. The community group consisted of three community researchers, one medical doctor, one community social work manager and one primary health organisation manager.

The analysis involved rating the performance of the partnership, identifying areas of disagreement in perceptions and comparing responses between researchers and community. In analysing the online survey data, we examined the mean, mode and level of variability at three levels: (i) individual items; (ii) sub-category level (average of items in each sub-category of community engagement and cultural centredness); and (iii) broad category level (average of items in each category of community engagement, cultural centredness, systems thinking and integrated knowledge translation). Scores in the >4–6 range indicate high performance on the item or category. Scores in the 3–4 range indicate room for improvement, while scores in the 1–2 range demonstrate problematic areas. We used standard deviation in determining the level of variability within each sub-category.

Interview responses were analysed using framework analysis (Gale et al. 2013). We used the major categories of the HPW framework to organise the analysis and then identified specific themes within these categories. We also noted any specific references to elements or sub-categories of the HPW framework that were cited to be critical to implementation along with the overall success of the project.

Ethics approval was granted by The Waikato Management School Ethics Committee, The University of Waikato. This research was undertaken with appropriate informed consent of participants.

The first aspect of the results is how the process evaluation demonstrates alignment with the HPW framework. The results from the survey are summarised in Table 1, with interview data integrated within the text. Both the researchers and community groups unanimously agreed that the partnership is following the HPW framework with some specific areas for improvement.

Table 1.  Results summary by sub-category and partnership group/total

Community engagement was consistently expressed by both partnership groups as a core strength. Members rated the six sub-categories as high (M = 5; i.e. very great extent), and most had relatively little variability. The willingness and motivation to change, and commitment to engage were strongly evident in the interactions between academic researchers and community organisations. Several important qualities were raised by the community groups about the researchers, including willingness to learn, respect, good communication and accessibility:

Well, I think there’s a number of things that work together well, and I think they work together well because of the people who are facilitating some of the conversation. The first bit was the doctor and yourself coming here hoping to hear how we can do this differently in this type of community. That was always good, a willingness and attitude to bring in a refreshing change has come off the back of new learnings for everyone. I think that principle has given us a platform to talk to each other well [Community social work manager].

The ‘how’s’ more important. Anyone can do the ‘what’ but the ‘how’ is a lot more important. You guys have had a really easy approach to engagement – respectful, understanding and take criticism really well, or critiquing and criticism, you take it on board really well [Community researcher 1].

Members rated the partnership as performing at a medium level under most categories of cultural centredness. One community researcher stated that the power-sharing arrangement between the researchers and the community organisations were a key factor that made the relationship work really well:

I think that the balance between people in the community and then the academic research team has been really good. It’s allowed the research and the development of the intervention to really flow because even though sometimes our focuses may be different, it’s always driving towards the same goal. That’s something that I’ve found really good and has worked really well [Community researcher 2].

However, power sharing and agency in research also had differences of at least a point on the Likert scale between the academic and community researchers. Overall, the academic researchers rated both of these sub-categories a point higher than the community researchers. Thus, even though interviews displayed positive regard for power sharing, there is room for improvement. In fact, some of the community members rated low (i.e. not at all/small extent) in the specific areas of choosing research methods, interpreting study findings, interview and survey design, and presenting at meetings and conferences.

The members rated the partnership as high in both systems thinking and integrated knowledge translation (M = 4.7). The public health physician, who works closely with one of the community groups, was able to articulate ‘how’ both systems thinking and integrated knowledge translation are incorporated into the design of the intervention:

Systems thinking, systems perspective. We totally do that. We totally are high level systems … So, our intervention, includes multiple causes, so poverty, food insecurity, environmental, obesogenic environment, and also systems, what’s in the health care system - so there’s no strong screening and referral pathways for pre-diabetes. Also, at the individual level and the whānau [extended family] level, causes there, and solutions in all of those places…Well, I’d say the solutions, originally, we had the individual, the whānau and the community as our solution, but after kind of working it through with [them] we added another level which is the [health and social] system. We’ve got now like a four-pronged intervention. It’s kind of big and messy, working in this multi-layered zone with multiple perspectives. That’s what I was talking about before. It’s so big and complex and you could easily get totally lost going down rabbit holes. That’s why it’s good to have a framework and really good project management, otherwise you’d get lost pretty easy. [Public health physician].

We also note that the community researchers rated systems thinking a point lower than the academic researchers. In a debriefing meeting about the process evaluation, this was explained because the community members perceived a lot of systems constraints in implementing an innovative intervention and making it sustainable. Further systems thinking and integrated knowledge translation issues are noted in the next section.

Overall, all members felt that the HPW project would be successful/sustainable, but stressed that much of it is dependent on the funding and political contexts, and ‘how’ the intervention is implemented. A community member focussed on the major challenge of getting the funders on side:

What is the measure of success? I think one of its major challenges is going to be successful in the eyes of the funder, and that’s always a challenge at the best of times but particularly in this because I think some of the outcomes and some of the things that will be the most successful about it will be things that the funder may not see that way. I guess it depends on that. I think it is a challenge [Medical doctor].

One academic researcher felt that it was important to see the project sustain itself beyond the funding period as an indicator of success:

The question for me that I have is sustainability beyond this year. The funding will end and what’s going to happen? I’m not sure how optimistic I am about changing the DHBs [District Health Boards] mind. That’s my biggest uncertainty. It’s that long-term change that I’m not sure about. Are we really going to make a difference in health equity? I think we can improve [health for] the people who participate. I don’t have any doubt about that. It’s how to make it beyond this next year, that’s the one I’m a little cautious about. I guess I’m cautiously optimistic [Academic researcher 1].

A community researcher highlighted how volatile the political context can be and the potential effect that it can have on implementing an intervention and it being successful:

Well, there [are] too many factors because don’t forget we’re only at the infancy stage, we’re crawling. I still think a critical factor is going to be general practice and this latest thing that’s just happened with [the primary health organisation]… That’s just a political environment, we can’t control that [Community Researcher 1].

A primary health organisation manager believed that one of the proposed interventions was challenging because it required getting marginalised communities to participate in a lifestyle program and to also access services where they would normally feel shunned. However, she noted that the employment of a ‘Kaiarahi’ (Māori community champion) would be critical to linking communities with key services and hence, the intervention’s success:

So, if you’ve got the right person in that role, I think that will be the success and I think that you’ve got that set up and you’ve got [our organisation] involved as well as a back-up in support of it, you know, I guess for me, all these steps are in place to make it successful. However, you’re dealing with human people here that have been disenfranchised from their community and from help for a lot of them [Primary health organisation manager].

In summary, the participants in this study viewed that the future success of this project and the ability to make sustainable change would depend on convincing funders and responding to a changing political context.

We shared the results of this process evaluation at project team meetings and had conversations to identify ways to improve our partnership. Some of the changes made include inviting community researchers to participate fully in survey design (e.g., health screening questionnaires), conference presentations, grant writing and funding applications. Further, we shifted resources and decision-making to the community groups for the implementation of the intervention. Finally, we increased systems thinking and integrated knowledge translation activities by specifically presenting a business case to potential funders to continue the projects beyond the research period.

The results show good alignment to the HPW framework (Oetzel et al. 2017), as members rated the partnership’s overall performance as high, with a few areas for potential improvement. There were three sub-categories that had at least one point differences between community and academic researchers (with academic higher than community on all three): agency in research, power sharing and systems thinking. The major contribution of this study is to show the usefulness of the evaluation tool in relation to the HPW framework and offer key lessons learnt.

The first lesson learnt is that the performance ratings can be attributed to the groundwork and time needed to build relationships of trust. By the time of the process evaluation, the HPW project was in full operational mode having: secured funding, established a framework and formed strong relationships. These processes and outcomes required building and maintaining trust; an important aspect in strengthening partnership synergy and sustainability (Khodyakov et al. 2011; Jagosh et al. 2015; Oetzel et al. 2018). Further, trust requires active management throughout the course of a partnership, as certain events and challenges can create opportunities for mistrust (Lucero et al. 2018). Consistent interaction and critical reflection helps to create opportunities for understanding and allows for a foundation that can be relied on to help partnerships navigate the challenging times. They also help to overcome historical mistrust created through colonisation of Indigenous communities (Lucero et al. 2018).

Second, research teams working with Indigenous communities to improve primary care health services need to engage in the critical reflection process by conducting process evaluations during intervention development and post implementation. Such reflection is critical for helping the partnership to be reflexive and accountable about power relationships, and to also make adjustments to their processes, practices and intervention (Freire 1970; Dutta 2007; Wallerstein et al. 2018). Research teams could consider establishing milestones early in their project schedule to ensure alignment with principles guiding the research. However, this needs to be considered in light of how much focus should be on relationship/trust building during the initial stages of the project. To balance the demands of funding requirements and building relationships requires careful planning, realistic timeframes, contingency plans and continual critical reflection.

The third lesson learnt is that the research team needs to continue to advocate for resources on behalf of the community. One of the biggest challenges that Indigenous communities face is sustainability of interventions, particularly beyond the research funding period (Gibson et al. 2015). Advocacy aligns with the HPW framework’s focus on systems thinking and integrated knowledge translation (Oetzel et al. 2017). This approach is consistent with a sustainability science perspective in that sustainable intervention development should be regarded as complex systems that target specific health programs and the needs and interests of key stakeholders (Gruen et al. 2008). Our interventions purposively acknowledge the complex systems, and yet there is need for improvement by integrating concrete activities to promote the importance of community-based projects to funders. Health intervention programs for Indigenous communities would be wise to integrate such ideas to address health needs, as addressing the social determinants of health in these communities takes time and sustained efforts (Bailie et al. 2007).

This study is not without limitations. First, it provides evaluation of only two partnerships, and thus, further research is needed to determine if the evaluation tool is useful for others. Second, this study was conducted by a member of the partnership team and hence may reflect a positive bias towards the project. We attempted to mitigate that bias by having a team member not affiliated with either project team conduct the evaluation and provide honest feedback to the team.

Taking primary health to the next level in Indigenous communities should include careful reflection on how intervention/research teams engage with communities as part of the implementation and evaluation process; that is, there should be as much focus on the ‘how’ as the ‘what.’. The HPW implementation framework, along with its process evaluation tool, is an effective means for implementation teams to help guide participatory work for relationship building, trust and synergy along with larger system impact. It also helps teams to critically reflect on their performance but, most importantly, compels them to make adjustments where there are differing perspectives, particularly in a way that privileges the Indigenous community perspective. This process helps to ensure that research partnerships and implementation teams stay true to the principles guiding their work.

The authors declare that they have no conflicts of interest. The lead author plus five of the co-authors are Indigenous people from Aotearoa New Zealand. All of the authors are or have been active in either or all of the design, implementation and evaluation.