Process evaluation of the Albany Physical Activity and Nutrition (APAN) program, a home-based intervention for metabolic syndrome and associated chronic disease risk in rural Australian adults

Key words: behaviour change, chronic disease, obesity, program evaluation, rural and regional health.

Poor diet, physical inactivity, and obesity contribute significantly to the burden of disease in Australia,¹ due to the associated risk of metabolic syndrome, type 2 diabetes mellitus and cardiovascular disease.^2–4 In 2011–2012, only 8% of Australian adults met the Australian Dietary Guidelines for vegetable intake and 51% for fruit, only 43% met the recommendation for moderate- or vigorous-intensity physical activity.¹ Adults living in rural and disadvantaged areas are more likely than their metropolitan counterparts to be physically inactive,⁵ consume inadequate amounts of fruit and vegetables,⁶ and be overweight or obese. Consequently, prevalence of metabolic syndrome, type 2 diabetes mellitus and cardiovascular disease is elevated in these regions,¹ highlighting rural populations as high-risk groups in need of intervention.

Combined diet and physical activity interventions are recommended for individuals at an increased risk of developing metabolic syndrome and associated chronic diseases.⁷ Intervention strategies that incorporate multiple counselling and education sessions⁸ plus goal-setting⁹ have been shown to support behaviour change in these individuals.⁷ Process evaluation of a study conducted in rural New South Wales found that obese individuals, who would probably benefit most from counselling, are more likely than the general population to engage with a telephone service.¹⁰ It was recommended that future interventions targeting rural and disadvantaged adults should combine telephone counselling with online and printed resources,¹¹ and tailored feedback via email.⁹

The Albany Physical Activity and Nutrition (APAN) intervention aimed to encourage improvements in diet and physical activity for rural adults with, or at risk of, metabolic syndrome, to prevent the onset of associated chronic diseases. Participants were provided with printed and online resources, encouraged to set goals, and received tailored feedback via email and telephone. A randomised controlled trial (RCT) was conducted to evaluate the effectiveness of the APAN intervention in terms of dietary and physical activity improvements and their effects on weight management and metabolic syndrome parameters. The study protocol has been published previously, including an overview of the process evaluation plan.¹² The effects of the intervention on physical activity and dietary behaviours, also published elsewhere, included significant improvements in minutes of moderate intensity physical activity per week, fat and fibre intake scores, and serves of vegetables per day.¹³

Process evaluation is conducted to determine intervention effectiveness and to highlight reasons for success or failure.¹⁴ Such reasons often include poor program design or implementation, and failure to reach the required target group numbers.¹⁵ There is also a need to address the gap in understanding the effectiveness of diet and physical activity programs delivered online and via email,¹⁶ and to determine reasons for attrition.⁷ Understanding reasons for withdrawal and refusal to participate is imperative for interventions that target rural and disadvantaged target groups, due to the common issue of social exclusion and isolation.¹⁷ The present study describes the process evaluation of the APAN intervention program to gain an insight into the link between intervention elements and outcomes, based on the process evaluation framework described below.¹⁵

Process evaluation of the APAN study was based on the framework described by Saunders et al.¹⁵ The following components were assessed: a) the extent to which implementation of the intervention occurred as planned (fidelity – quality); b) the number of intended intervention components delivered (dose delivered – completeness); c) the extent to which participants used the resources as intended or recommended (dose received – exposure); d) participants’ satisfaction with the program and staff (dose received – satisfaction); e) the number of participants actively participating in the intervention (reach – participation rate); f) procedures used to attract participants and maintain involvement in the intervention (recruitment); and g) factors that influenced the implementation or study outcomes (context).¹⁵

This RCT of adults with, or at risk of, metabolic syndrome evaluated a physical activity and diet behaviour change intervention for the prevention of chronic disease. It was conducted in Albany, a rural area in Western Australia. The study protocol¹² was approved by the Curtin University Human Research Ethics Committee (approval number HR149_2013). All participants provided informed consent before their involvement in the study. Primary outcome results have been reported elsewhere.^13,18 A total of 401 participants were recruited. They were 50 to 69 years old with, or at risk of, metabolic syndrome based on the International Diabetes Federation criteria.¹⁹ The intervention participants (n = 201) comprised the sample for the process evaluation.

Participants were recruited from towns within a 50 km radius of Albany in Western Australia. Screening and recruitment occurred in three stages, the procedure and participant characteristics have been published in detail previously.¹³ Briefly, screening stage one was conducted via telephone using the Australian Type 2 Diabetes Risk Assessment Tool (AUSDRISK).²⁰ Screening stage two was conducted at a local clinic, where two trained researchers recorded anthropometric measurements. Screening stage 3 occurred at a local pathology laboratory for blood sample analysis to determine metabolic syndrome status. Participants who met the criteria for all three stages were eligible for the trial.

Self-determination theory (SDT) provided the theoretical basis for the intervention,²¹ complemented by motivational interviewing.²² In a health context, SDT is a general human motivation theory that focuses on participants’ perception of the support for their autonomy.²³ The APAN intervention was based on several SDT constructs:²³ a) autonomy supportive climate – participants were encouraged to engage in health behaviours for their own reasons; b) autonomous orientation – participants’ engagement in behaviours was based on personal values and interest; c) intrinsic goals – participants were encouraged to set goals relating to personal growth; d) intrinsic motivation – motivation derived from inherent enjoyment of a particular behaviour.

The intervention consisted of several resources and strategies designed to improve the physical activity and dietary behaviours of participants, adapted from previous studies for a rural context.^24,25 Printed resources included an A4-sized educational booklet, exercise charts, and a nutrition panel wallet card to assist with reading food labels. A password-protected website was also available, which consisted of an interactive progress tracker and links to further physical activity and dietary information. All health information was based on the Australian Dietary Guidelines²⁶ and Australia’s Physical Activity and Sedentary Behaviour Guidelines.²⁷

The motivational interviewing component of the intervention consisted of regular telephone calls to the participants by two trained research assistants. Calls were scheduled for weeks 1, 3, 6, 12, 18, and 24 of the 6-month intervention. Interview schedules were based on the four constructs of SDT listed above, to promote autonomy and ensure behaviour change was intrinsically motivated. Additionally, the telephone calls were used to support goal setting and use of the program resources. A detailed overview of the resources and motivational interviewing component of the intervention has been described previously.^12,13

The process evaluation used a mixed-methods approach, with qualitative and quantitative data collected both during and after the intervention. Instruments comprised an online survey, exit interviews, and notes recorded by recruitment and research staff. The instruments aimed to answer the following key process evaluation questions:¹⁵ a) who participated, who withdrew, and for what reasons? b) To what extent was the intervention delivered and received as intended? c) What were the participants’ experiences and their suggestions for improvements? Table 1 provides an overview of the process evaluation questions and components and their corresponding measurement approaches.

Table 1.  Process evaluation components and measurements

Evaluation of the APAN program resources was conducted at the 3-month point of the intervention via a self-administered online survey. Participants were sent a Survey Monkey (www.surveymonkey.com) link via email to evaluate the educational booklet, exercise chart, wallet card, and website. Questions were adapted from process evaluation instruments used in other studies^25,28 and included both open-ended questions and five-point Likert scales (e.g. very attractive to very unattractive; very suitable to very unsuitable).

Participants were asked to rate the extent to which they: a) found the resources useful; b) were attracted to the resources; c) found the resources suitable for their age; d) were encouraged by the resources to be more physically active; e) were encouraged to practice the program exercises; f) were encouraged to eat more fruit and vegetables; and g) were encouraged to eat less sugar, fat, and salt. Open-ended questions encouraged participants to comment on features they particularly liked or disliked about the resources and asked for suggestions for improvement. Participants were also asked to state whether they used specific components of the resources, such as the monthly activity planner in the booklet and the progress tracker on the website.

Exit interviews were conducted via telephone with a purposefully selected sample of intervention completers (n = 8) and non-completers (n = 8). The sample size was based on a similar study,²⁸ with the literature suggesting that saturation occurs within the first 12 interviews.²⁹ A trained researcher who had no previous contact with participants followed a semistructured interview schedule, which included open-ended and closed questions to identify and evaluate: a) reasons for being involved in the program; b) the design features of the program that encouraged participation; c) the guidance and support provided by motivational interviewers; d) changes in physical activity and dietary attitudes and behaviours during the program; e) how the program might be improved.

During the recruitment stage staff recorded reasons provided by potential participants for not wanting to participate in the study. After the intervention commenced, they recorded why participants withdrew from the study.

Online survey data were downloaded directly from Survey Monkey into SPSS version 22. To facilitate analysis, variables recorded on the five-point Likert scales were collapsed into three levels to minimise respondent ambiguity in the positive and negative response categories.³⁰ For example, ‘attractive’ (‘very attractive’ combined with ‘attractive’) and ‘unattractive’ (‘very unattractive’ combined with ‘unattractive’). Descriptive statistics were used to summarise the demographic characteristics of the sample and the survey results. Responses to open-ended questions and staff notes were transcribed verbatim into a text document and managed by NVivo 11.1. Qualitative data were coded and thematic analysis was performed to identify recurring patterns.

One hundred and forty-five intervention participants were invited to participate in the online survey, 50.3% (n = 73) completed the survey (64.4% female, mean age 61 years, s.d. ± 5.4). Twenty-eight intervention participants were randomly selected for the exit interviews, 57.1% (n = 8 completers; n = 8 non-completers) were successfully contacted and completed the interview (completers: 62.5% female, median age 61.5 years; non-completers: 62.5% male, median age 65 years).

The screening and recruitment stages occurred from October 2014 to December 2015. During screening stage one, 12 723 telephone numbers were called based on postcodes listed in the White Pages for Albany and environs. Of the people contacted via these telephone numbers, 57.6% (n = 7332) agreed to participate in the AUSDRISK screening and 33.4% (n = 4247) were not willing to participate. Some individuals opted to provide a reason for not wanting to participate. Table 2 provides a summary of the call statistics for screening stage one, including reasons for not completing the AUSDRISK questionnaire. The main reasons for non-participation as recorded by recruitment staff were: too busy (n = 71); not interested in the program or research (n = 31); perceived their health to be good (n = 26); health issues (n = 21); work commitments (n = 20); personal issues (n = 19); needle phobia (n = 18); travelling (n = 16); have their own exercise program (n = 11); moving out of the area (n = 9); dieting (n = 5); and unwilling to change (n = 4).

Table 2.  Screening stage one: telephone call statistics (n = 12 723)

Of those who completed the AUSDRISK questionnaire, 15.5% (n = 1134) were eligible for screening stage two, with the majority excluded for being outside the desired age group (57.9%, n = 3586), not residing in the target area (13.3%, n = 824), or having a lower AUSDRISK score than required (12.8%, n = 791). Of the participants who were eligible for screening stage two, 46.4% (n = 526) attended the clinic and 53.6% (n = 608) opted out after receiving the invitation letter. After screening stage three, the final sample eligible for randomisation and included in the study was 401 (intervention (n = 201); control (n = 200)) participants.

After randomisation, all intervention group participants received the printed resources and those with computer access (n = 145, 72.1%) also received the online resources. During the 6-month intervention period, 18.4% of participants (n = 37) withdrew from the program and a further 6.5% (n = 13) were lost to follow-up. Reasons for withdrawal were: health issues (n = 10); personal issues (n = 7); lost interest or changed their mind (n = 7); not willing to provide reason (n = 6); travelling (n = 4); moved out of the area (n = 2); and work commitments (n = 1).

All participants were scheduled to receive six motivational telephone calls from research staff during the intervention. The purpose of the calls was to assist with goal setting and to motivate and encourage participants to use the program resources. Research staff recorded the number of phone call attempts and the number of successful calls over the intervention period. On average, participants completed three telephone contacts each,13.9% of participants (n = 28) refused to participate or were unable to be contacted, 53.7% of participants (n = 108) completed between one and three calls, and 32.3% of participants (n = 65) completed four or more calls.

Most online survey participants stated that the printed resources had encouraged them to be physically active (booklet 60.3%; exercise chart 61.1%), to eat more fruit and vegetables (booklet 68.5%), and to eat less sugar, fat, and salt (booklet 71.2%). Specific dietary changes, as noted by participants during the exit interviews, included smaller portions, reduced intake of sugar, alcohol, fat and processed food, and increased intake of fruit and vegetables, grains, and water. Specific changes to physical activity behaviours included increased stretching and balance exercises, particularly due to the exercise charts, increased gym use, and participation in sports.

The majority of participants did not use the monthly activity planner in the booklet (78.1%) or the progress tracker via the website (65.1%). Reasons given for not using the monthly activity planner in the booklet included: too busy (n = 20); already had a schedule or plan (n = 8); no motivation (n = 8); stopped after a while (n = 7); personal issues and distractions (n = 6); did not see the need (n = 5); forgot to use it (n = 3); and do not like to record things (n = 3). Reasons given for not using the progress tracker via the website included: did not use the website (n = 9); no time (n = 9); not confident with computers (n = 7); happy with current method (n = 7); stopped using it (n = 6); and no motivation (n = 3).

The majority of participants found the printed resources useful (booklet 69.9%; exercise chart 69.4%), attractive (booklet 76.7%; exercise chart 73.6%), and suitable for people their age (booklet 78.1%; exercise chart 79.2%). The website was the least preferred resource.

During the online survey, participants were asked what they particularly liked about the program resources. The main reasons provided were aesthetics, ease of use, and awareness raising of the booklet. Suggestions for improvement included more dietary recommendations and recipes and a reduction in booklet size. The main reason participants liked the exercise chart was because it was easy to follow and had a magnet, making it easy to display on the refrigerator. Suggestions for improvement included having a range of more advanced exercises to try, and including a DVD to demonstrate these.

During the exit interviews, participants were asked to state reasons for being involved in the program. The main response themes were issues with weight, expected health benefits, and benefit to the community. Participants were also asked what motivated them to complete the program. Responses included desired health benefits, to honour the commitment, liking the challenge, and wanting to assist with research.

Participants were asked to comment on the guidance and support they received during the intervention. Reasons given for liking the support included the level of encouragement they received, suggestions for overcoming barriers, guidance to assist with the program resources, and the ability to contact research staff at any time for assistance. The main reason given for disliking the support was the lack of connection between the participant and caller.

Non-completers provided reasons for not continuing the program, which included being too busy, having personal or family issues, and lack of motivation. Suggestions to encourage sustained participation in the future were also provided. Responses included changing the delivery mode to face-to-face (individual or group) contact, having more regular feedback, and providing more incentives to complete the program.

The process evaluation of the APAN program assessed the intervention in terms of participation, reach and participant satisfaction, and highlighted reasons for the program’s success or failure. The outcomes suggest good participant adherence and acceptability of the program for the target group. Participants reported program resources to be attractive, useful and suitable for their age group, which encouraged them to improve their physical activity and dietary behaviours. These findings are supported by the significant improvements in moderate-intensity physical activity, and fat, fibre and vegetable intake for the intervention group.¹³

Recruitment of participants was time-consuming and resource-intensive. Screening stage one required cold calling, a method that relies on accessing publicly listed telephone numbers. Approximately one-third of the successful calls made were terminated due to unwillingness of the telephone contacts to participate. Furthermore, over half of the participants eligible for screening stage two opted out after receiving the study information sheet and letter requesting them to make an appointment at the local health clinic. It is possible that this request might place too great a burden on participants. The number of individuals opting out at this point could be reduced by the research staff making these arrangements.³¹

Although the recruitment method, which comprised several stages, was resource intensive, it reduced self-selection bias introduced through general advertising, thus increasing the likelihood of recruiting a more representative sample.³² An alternative but less robust recruitment approach using AUSDRISK²⁰ to identify participants would be less resource-intensive and suitable for large populations.¹³

Identifying reasons for non-participation in studies that target disadvantaged groups will benefit future projects that aim to be more inclusive.¹⁷ The attrition rate for the APAN intervention group was 18%, which is similar to other studies targeting middle- to older-aged adults.^33,34 The process evaluation identified several reasons for participant withdrawal, with the majority citing health or personal issues or a loss of interest in the program, which are similar to those cited in similar studies.^28,32

Reducing attrition ensures studies remain representative while minimising the risk of bias.³² In order to reduce attrition in lifestyle interventions, a heavier focus on readiness-to-change assessment and understanding the benefits of behavioural change on chronic disease risk are warranted,³⁵ as well as providing tailored feedback and encouraging continued goal-setting.³⁴ APAN participants were encouraged to set goals and received tailored feedback, and were informed of the intervention aims and objectives. However, participants were not provided with their individual risk profile before commencing the intervention, which may help prevent withdrawal.³⁶

Participation in the intervention was adequate, with 74.6% of participants finishing the 6-month program. Usage of individual program resources and support varied. The majority of participants used the education booklet and exercise chart, but most did not find the online component useful. This finding is consistent with a web-based physical activity and nutrition study targeting adults at risk of cardiovascular disease.³⁷ The APAN participants found the online tools to be time consuming and burdensome, which may suggest a design issue rather than an acceptability of mode issue. However, this is not unusual as other research indicates that it is common for participants to disengage with online programs over time.³⁷

Web-based tools are recommended to complement traditional chronic disease intervention methods.³⁸ Therefore, strategies to make online components of interventions more effective and useful to participants should be investigated as suggested by the literature.³⁹ Addressing the determinants of engagement with online components using persuasive design (such as novelty, self-monitoring, aesthetics, reminders) as well as addressing determinants of behaviour change, is an important consideration for the design of websites.⁴⁰ A more valid measure of participant engagement with online components is also recommended, due to the limitation of self-reported use.

The majority of the intervention group participated in the motivational telephone calls, yet the prescribed number of telephone sessions was not completed by all participants. Nevertheless, they were given access to other communication channels such as printed resources and email support. These strategies have demonstrated effectiveness for rural adults when used in conjunction with telephone support.^10,11 Strategies to maintain participation in telephone support services should be further investigated, given the significant positive outcomes of previous diet and physical activity behaviour change interventions delivered via telephone.⁴¹

A major strength of this process evaluation is the mixed-methods approach which incorporated a range of indicators to assess recruitment, dose, and satisfaction. Also, the combination of intervention strategies implemented was effective in changing the physical activity and dietary behaviours of the intervention group. Interventions that focus on goal setting, feedback, and self-monitoring appear to be more effective than programs that do not incorporate any of these strategies.⁴² APAN was based on SDT²¹ complemented by motivational interviewing.²² This combination ensured participant autonomy, which seems to enhance motivation.

A limitation of this study is the lack of a face-to-face component. Face-to-face interventions are generally more effective than interventions using other communication methods;⁴² however, it is important to note that the reach of an intervention and the cost of delivery is often better in distance-based interventions. Interventions tend to be more effective when there is an element of face-to-face contact in addition to an online component.⁴³ APAN participants mentioned this strategy as a potential improvement to the program. However, due to the remoteness of the rural study location, this would be challenging to implement. Perhaps video calling could be worthwhile but the older age of this group may limit its suitability.

The results of this process evaluation provide an overview of recruitment challenges, preferred intervention components, and possible improvements for interventions targeting older adults in rural areas. Carefully planned recruitment strategies have the potential to reduce the burden on resources and improve uptake and understanding reasons for attrition will assist practitioners to support intervention participants in overcoming barriers. Given the participants’ preference for printed resources, and the known effectiveness of these in combination with other strategies, methods for successfully combining them with telephone and online support should be investigated. Being able to deliver a program without a face-to-face component makes the program potentially scalable in the rural context. However, methods to make the recruitment more streamlined should be investigated. Further work is required to determine the most effective intervention components for rural adults who are at risk of chronic disease.